Grok
Parser stage configuration specifications
The Grok parser stage uses Java Grok and Grok patterns (a specific kind of regex matching) to parse log files and similar text files that have line-oriented, semi-structured data. Parsing a text file with the Grok parser lets you give more structure to semi-structured data and extract more information.
Whether the Grok stage parses a file
Before a Grok parser stage parses a file, the file must meet criteria regarding the media type and file name.Media type
The Grok parser stage parses files that have media types that match either the default media types or media types that you specify. Select or unselect Use default media types for this parser stage:- Selected. The Grok parser stage parses files that have one of the default media types (
text/plainortext/x-log), as well as files that have media types that you specify under Media Types for this Parser Stage. - Unselected. The Grok parser stage only parses files that have one of the media types that you specify under Media Types for this Parser Stage.
File name
Optionally, you can specify a file name or file name pattern that a file must match for the Grok parser stage to parse the file.| Field | Description |
|---|---|
| Pattern Type | glob. Use bash shell wildcards. Examples include z.txt, *.md, and /a/*/b/f.txt. regex. Use Java regular expressions (PCRE; Perl-compatible regular expressions). Examples include z.txt$, .*\.txt$, and ^/a/[^\/]*/b/f.txt$. |
| File Name or Pattern | Name of the file or a pattern for the file name. The parser parses matching files. |
Grok patterns
Grok patterns are regular expressions written in the language of the Oniguruma regular expression library, which has this syntax. You configure a Grok parsing stage to use predefined Grok patterns (about 300 patterns are available) and/or Grok pattern definitions that you write yourself.- Use predefined patterns. Under the Grok Pattern part of the Grok parser stage configuration, specify a single top-level Grok pattern by name, for example,
REDISLOG. - Write your own Grok pattern definition(s). (optional) Write one or more Grok pattern definitions, and then enter them in the Grok Definition part of the Grok parser stage configuration.
Parsing rules
These are rules that affect the results of parsing:- Precedence in the event of identical names. If the name of a custom Grok pattern definition that you provide is identical to the name of a predefined pattern definition, then your definition is used.
- Invalid patterns. If a pattern is not syntactically valid, then the full text of the row being parsed is treated as a single field.
- Pattern does not match any data. If a pattern does not match any data, then the full text of the row being parsed is treated as a single field.
- Line by line. Parsing is line by line. If data has a multiline structure, the parser does not capture the relationship between lines.
When entering configuration values in the UI, use unescaped characters, such as
\t for the tab character. When entering configuration values in the API, use escaped characters, such as \\t for the tab character.