Product Selector

Fusion 5.12
    Fusion 5.12

    Grok Parser Stage

    The Grok parser stage uses Java Grok and Grok patterns (a specific kind of regex matching) to parse log files and similar text files that have line-oriented, semi-structured data. Parsing a text file with the Grok parser lets you give more structure to semi-structured data and extract more information.

    Whether the Grok stage parses a file

    Before a Grok parser stage parses a file, the file must meet criteria regarding the media type and file name.

    Media type

    The Grok parser stage parses files that have media types that match either the default media types or media types that you specify.

    Select or unselect Use default media types for this parser stage:

    • Selected.* The Grok parser stage parses files that have one of the default media types (text/plain or text/x-log), as well as files that have media types that you specify under Media Types for this Parser Stage.

    • Unselected.* The Grok parser stage only parses files that have one of the media types that you specify under Media Types for this Parser Stage.

    File name

    Optionally, you can specify a file name or file name pattern that a file must match for the Grok parser stage to parse the file.

    Field Description

    Pattern Type

    glob. Use bash shell wildcards. Examples include z.txt, *.md, and /a/*/b/f.txt.

    regex. Use Java regular expressions (PCRE; Perl-compatible regular expressions). Examples include z.txt$, .*\.txt$, and ^/a/[^\/]*/b/f.txt$.

    File Name or Pattern

    Name of the file or a pattern for the file name. The parser parses matching files.

    Grok patterns

    Grok patterns are regular expressions written in the language of the Oniguruma regular expression library, which has this syntax.

    You configure a Grok parsing stage to use predefined Grok patterns (about 300 patterns are available) and/or Grok pattern definitions that you write yourself.

    • Use predefined patterns. Under the Grok Pattern part of the Grok parser stage configuration, specify a single top-level Grok pattern by name, for example, REDISLOG.

    • Write your own Grok pattern definition(s). (optional) Write one or more Grok pattern definitions, and then enter them in the Grok Definition part of the Grok parser stage configuration.

    Parsing rules

    These are rules that affect the results of parsing:

    • Precedence in the event of identical names. If the name of a custom Grok pattern definition that you provide is identical to the name of a predefined pattern definition, then your definition is used.

    • Invalid patterns. If a pattern is not syntactically valid, then the full text of the row being parsed is treated as a single field.

    • Pattern does not match any data. If a pattern does not match any data, then the full text of the row being parsed is treated as a single field.

    • Line by line. Parsing is line by line. If data has a multiline structure, the parser does not capture the relationship between lines.

    When entering configuration values in the UI, use unescaped characters, such as \t for the tab character. When entering configuration values in the API, use escaped characters, such as \\t for the tab character.

    Parses semi structured content using Grok patterns (like Regex, see https://github.com/thekrakken/java-grok). This is often ideal for understanding log files, but can be used for other purposes.

    id - string

    Default: 24d0f024-9f9d-47c1-a7af-97d71dfd774c

    label - string

    A label for this Parser Stage

    <= 255 characters

    enabled - boolean

    Default: true

    mediaTypes - array[string]

    Documents with a media type on this list will be matched by this parser stage. See inheritMediaTypes / use default media types for more.

    inheritMediaTypes - boolean

    Each parser stage has a built-in list of media types it handles by default. If this setting is true, that list will be used along with any optional additional types provided in the mediaTypes list. If this setting is false, this stage will only be selected for media types in the mediaTypes list, and the mediaTypes list becomes a mandatory property which must have at least one valid media type.

    Default: true

    pathPatterns - array[object]

    Specify a file name or pattern that must be matched for this parser stage to run. Forward slashes ("/") are used to join names of files inside archives with the archive name.

    object attributes:{syntax : {
     display name: Pattern type
     type: string
    }    pattern : {
     display name: File name or pattern
     type: string
    }    }

    errorHandling - string

    Default: mark

    Allowed values: ignorelogfailmark

    outputFieldPrefix - string

    Fields extracted by this parser will be prefixed with this string. The remainder of the field name will be as detected in the stream

    <= 20 characters

    Match pattern: ^$|^[A-Za-z_][A-Za-z0-9_\-\.]+$

    charset - stringrequired

    Example: "UTF-8"

    Default: detect

    ignoreBOM - booleanrequired

    Ignore Byte-Order Mark (BOM) if present and always use the configured character set. When set to false a valid BOM character set overrides the configured default character set.

    Default: false

    grokDefinition - string

    Custom Grok definition

    grokPattern - string

    Grok parsing pattern

    type - stringrequired

    Default: grok

    Allowed values: grok