The Grok parser stage uses Java Grok and Grok patterns (a specific kind of regex matching) to parse log files and similar text files that have line-oriented, semi-structured data. Parsing a text file with the Grok parser lets you give more structure to semi-structured data and extract more information.
Whether the Grok stage parses a file
Before a Grok parser stage parses a file, the file must meet criteria regarding the media type and file name.
Media type
The Grok parser stage parses files that have media types that match either the default media types or media types that you specify.
Select or unselect Use default media types for this parser stage:
-
Selected – The Grok parser stage parses files that have one of the default media types (
text/plain
ortext/x-log
), as well as files that have media types that you specify under Media Types for this Parser Stage. -
Unselected – The Grok parser stage only parses files that have one of the media types that you specify under Media Types for this Parser Stage.
File name
Optionally, you can specify a file name or file name pattern that a file must match for the Grok parser stage to parse the file.
Field | Description |
---|---|
Pattern Type |
|
File Name or Pattern |
Name of the file or a pattern for the file name. The parser parses matching files. |
Grok patterns
Grok patterns are regular expressions written in the language of the Oniguruma regular expression library, which has this syntax.
You configure a Grok parsing stage to use predefined Grok patterns (about 300 patterns are available) and/or Grok pattern definitions that you write yourself.
-
Use predefined patterns – Under the Grok Pattern part of the Grok parser stage configuration, specify a single top-level Grok pattern by name, for example,
REDISLOG
. -
Write your own Grok pattern definition(s) (optional) – Write one or more Grok pattern definitions, and then enter them in the Grok Definition part of the Grok parser stage configuration.
Parsing rules
These are rules that affect the results of parsing:
-
Precedence in the event of identical names – If the name of a custom Grok pattern definition that you provide is identical to the name of a predefined pattern definition, then your definition is used.
-
Invalid patterns – If a pattern isn’t syntactically valid, then the full text of the row being parsed is treated as a single field.
-
Pattern doesn’t match any data – If a pattern doesn’t match any data, then the full text of the row being parsed is treated as a single field.
-
Line by line – Parsing is line by line. If data has a multiline structure, the parser doesn’t capture the relationship between lines.
Tip
|
When entering configuration values in the UI, use unescaped characters, such as \t for the tab character. When entering configuration values in the API, use escaped characters, such as \\t for the tab character.
|
Grok stage-specific properties
Property | Description, Type |
---|---|
charset
Character Set required |
Example: "UTF-8" type: default value: ' |
enabled
Enable this Parser Stage |
type: default value: ' |
errorHandling
Error Handling |
type: default value: ' enum: { ignore log fail mark } |
grokDefinition
Grok Definition |
Custom Grok definition type: |
grokPattern
Grok Pattern |
Grok parsing pattern type: |
id
Parser ID |
type: default value: ' |
ignoreBOM
Ignore BOM required |
Ignore Byte-Order Mark (BOM) if present and always use the configured character set. When set to false a valid BOM character set overrides the configured default character set. type: default value: ' |
ignoredMediaTypes
Media Types to ignore |
Documents with a media type on this list will be not be processed by this parser stage. type: |
inheritMediaTypes
Match default media types in this Parser Stage |
Each parser stage has a built-in list of media types it handles by default. If this setting is true, that list will be used along with any optional additional types provided in the mediaTypes list. If this setting is false, this stage will only be selected for media types in the mediaTypes list, and the mediaTypes list becomes a mandatory property which must have at least one valid media type. type: default value: ' |
mediaTypes
Media Types to match |
Documents with a media type on this list will be matched by this parser stage. See inheritMediaTypes / use default media types for more. type: |
outputFieldPrefix
Prefix parsed fields with |
Fields extracted by this parser will be prefixed with this string. The remainder of the field name will be as detected in the stream type: maxLength: 20 pattern: ^$|^[A-Za-z_][A-Za-z0-9_\-\.]+$ |
pathPatterns
File names to parse |
Specify a file name or pattern that must be matched for this parser stage to run. Forward slashes ("/") are used to join names of files inside archives with the archive name. type: object attributes: { } |
type
required |
type: default value: ' enum: { grok } |