Text Parser Stage

The Plain Text parser can split a text file by lines or consume it into a single document.

Options for treatment of this filetype include:

  • Plain Text Parser Fields

  • Number of header rows to skip

  • Split on line end or not

  • Comment character

  • Skip empty lines

  • Charset

Tip
 When entering configuration values in the UI, use unescaped characters, such as \t for the tab character. When entering configuration values in the API, use escaped characters, such as \\t for the tab character.

Text stage-specific properties

Property Description, Type
charset

Character Set

required

Example: "UTF-8"

type: string

default value: 'detect'
comment

Comment character

Characters at start of line to indicate a comment, default # (hash)

type: string

default value: '#'

minLength: 1
commentField

Comment field

Name of the output field where comment is stored, default 'comment'

type: string

default value: 'comment'

minLength: 1
commentHandling

Comment Handling

How to handle comments: include as-is, ignore (and remove from text), add as field (and remove from text), default include

type: string

default value: 'include'

enum: { ignore include as_field }
enabled

Enable this Parser Stage

type: boolean

default value: 'true'
errorHandling

Error Handling

type: string

default value: 'mark'

enum: { ignore log fail mark }
id

Parser ID

type: string

default value: '2f89a296-9ebe-435f-baf4-869ea804846d'
ignoreBOM

Ignore BOM

required

Ignore Byte-Order Mark (BOM) if present and always use the configured character set. When set to false a valid BOM character set overrides the configured default character set.

type: boolean

default value: 'false'
inheritMediaTypes

use default media types for this Parser Stage

Indicates if parser stage should use the default media types. Unchecking this box means that ONLY the manually configured media types will be parsed by the parser and you then MUST provide at least one media type.

type: boolean

default value: 'true'
maxLength

Maximum length

Maximum number of characters to allow for the body, -1 for unlimited, default 1MB

type: integer

default value: '1048576'

exclusiveMaximum: false

exclusiveMinimum: false

maximum: 2147483647

minimum: 0
maxLineLength

Maximum line length

Maximum number of characters to allow for any single line, default 1MB

type: integer

default value: '1048576'

exclusiveMaximum: false

exclusiveMinimum: false

maximum: 2147483647

minimum: 0
mediaTypes

Media Types for this Parser Stage

type: array of string

outputField

Output field

Name of the output field where text is stored, default 'body'

type: string

default value: 'body'

minLength: 1
outputFieldPrefix

Prefix parsed fields with

Fields extracted by this parser will be prefixed with this string. The remainder of the field name will be as detected in the stream

type: string

maxLength: 20

pattern: ^$|^[A-Za-z_][A-Za-z0-9_\-\.]+$
pathPatterns

File names to parse

Specify a file name or pattern that must be matched for this parser stage to run. Forward slashes ("/") are used to join names of files inside archives with the archive name.

type: array of object

object attributes: {
  pattern : {
    display name: File name or pattern
    type: string
    description : e.g.: "z.txt" or "*.md" or "/a/*/b/f.txt" for glob; "z.txt$" or ".*\.txt$" or "^/a/[^\/]*/b/f.txt$" for regex
    }
  syntax : {
    display name: Pattern type
    type: string
    default value: 'glob'
    description : glob uses bash shell-style wildcards; regex uses Java (PCRE-style) regex
    enum: { glob regex     }

    }
  }
skipEmptyLines

Skip empty lines

Skip any empty lines encountered, default false

type: boolean

default value: 'false'
skipHeaderLines

Skip header lines

Skip a number of header lines, default 0

type: integer

default value: '0'
splitLines

Split lines

Split text into lines to create multiple records, default false

type: boolean

default value: 'false'
trimWhitespace

Trim whitespace

Trim off leading and trailing whitespace from lines, default false

type: boolean

default value: 'false'
type

required

type: string

default value: 'text'

enum: { text }