HTML Parser Stage

This parser stage processes the following HTML elements:

  • <title>

  • <body> (with tags removed)

  • <meta>

  • <a> and <link>

Additionally, you can configure JSoup selectors to extract specific HTML and CSS elements from a document and map them to PipelineDocument fields. For example, you could use this to process navigational DIV elements one way, then process content-ful DIV elements another way.

Note
The HTML Transformation index pipeline stage is deprecated in favor of this parser stage.

HTML and CSS elements can be selected for extraction into new documents or fields:

  • To create new documents from selected elements, configure recordSelector.

  • To create new fields from selected elements, configure mappings.

Title, body, metadata, and links are only populated in the parent document. Both of these parameters support JSoup selectors, which provides a rich syntax for selecting HTML and CSS elements.

Tip
When entering configuration values in the UI, use unescaped characters, such as \t for the tab character. When entering configuration values in the API, use escaped characters, such as \\t for the tab character.
HTML Content Extraction

The Context Extraction setting in the HTML Parser has a special meaning.

By default, when unchecked or false, the HTML parser will essentially attempt to extract the text of the entire HTML page as the text that will be used in the Solr documen.

However, when Content Extract is checked, or true, a set of heuristic rules are applied to attempt to automatically determine what node in the entire page is most likely to be the page content, based on the tree of nodes inside it and the text component of all the sub-nodes.

This may be helpful when you have a variety of different page formats, and also significant text on the pages which is not useful to add to the documents, such as significant text in the page header or footer.

However, since the algorithm is based on heuriistics, it is possible that the results could change as the site is altered. When you need a high degree of certainty, we recommend that you use explicit rules of which nodes to extract to match your local configuration.

HTML stage-specific properties

Property Description, Type
charset

Character Set

required

Example: "UTF-8"

type: string

default value: 'detect'

contentExtractionConfig

Content extraction

Extract rich formatted content and metadata

type: object

object attributes: {
  extractContent : {
    display name: Extract page content
    type: boolean
    default value: 'true'
    }
  extractMetadata : {
    display name: Extract metadata
    type: boolean
    default value: 'true'
    }
  metadataPrefix : {
    display name: Optional prefix for rich content and extracted metadata
    type: string
    }
  preserveContentFormat : {
    display name: Store readable html version of body content (experimental)
    type: boolean
    default value: 'false'
    }
  }

enabled

Enable this Parser Stage

type: boolean

default value: 'true'

errorHandling

Error Handling

type: string

default value: 'mark'

enum: { ignore log fail mark }

extractBodyText

Extract body as a text

Extract body as a text

type: boolean

default value: 'true'

extractHtmlLinks

Extract HTML links

Collect links explicitly declared in HTML document

type: boolean

default value: 'false'

id

Parser ID

type: string

default value: '1cd7af1d-a832-46b5-a5be-35dc028b1691'

inheritMediaTypes

use default media types for this Parser Stage

Indicates if parser stage should use the default media types. Unchecking this box means that ONLY the manually configured media types will be parsed by the parser and you then MUST provide at least one media type.

type: boolean

default value: 'true'

keepParent

Keep Parent Document?

Keep or discard parent document with selected records. Has no effect if Record Selector is not specified.

type: boolean

default value: 'true'

mappings

HTML Mappings

type: array of object

object attributes: {
  attribute : {
    display name: Attribute
    type: string
    }
  field  (required) : {
    display name: Field
    type: string
    }
  multivalued : {
    display name: Multi Valued
    type: boolean
    default value: 'false'
    }
  selectRule  (required) : {
    display name: Select Rule
    type: string
    }
  }

mediaTypes

Media Types for this Parser Stage

type: array of string

metatagsPrefix

Optional prefix for metatags captured html document

Optional prefix to add to metatags (except id and charset) captured from an html document to used as a fields as-is (id and charset get prefix) from document

type: string

default value: 'false'

outputFieldPrefix

Prefix parsed fields with

Fields extracted by this parser will be prefixed with this string. The remainder of the field name will be as detected in the stream

type: string

maxLength: 20

pattern: ^$|^[A-Za-z_][A-Za-z0-9_\-\.]+$

pathPatterns

File names to parse

Specify a file name or pattern that must be matched for this parser stage to run. Forward slashes ("/") are used to join names of files inside archives with the archive name.

type: array of object

object attributes: {
  pattern : {
    display name: File name or pattern
    type: string
    description : e.g.: "z.txt" or "*.md" or "/a/*/b/f.txt" for glob; "z.txt$" or ".*\.txt$" or "^/a/[^\/]*/b/f.txt$" for regex
    }
  syntax : {
    display name: Pattern type
    type: string
    default value: 'glob'
    description : glob uses bash shell-style wildcards; regex uses Java (PCRE-style) regex
    enum: { glob regex     }

    }
  }
recordSelector

Record Selector

type: string

type

required

type: string

default value: 'html'

enum: { html }