Product Selector

Fusion 5.12
    Fusion 5.12

    HTML Parser Stage

    This parser stage processes the following HTML elements:

    • <title>

    • <body> (with tags removed)

    • <meta>

    • <a> and <link>

    Additionally, you can configure JSoup selectors to extract specific HTML and CSS elements from a document and map them to PipelineDocument fields. For example, you could use this to process navigational DIV elements one way, then process content-ful DIV elements another way.

    The HTML Transformation index pipeline stage is deprecated in favor of this parser stage.

    HTML and CSS elements can be selected for extraction into new documents or fields:

    • To create new documents from selected elements, configure recordSelector.

    • To create new fields from selected elements, configure mappings.

    Title, body, metadata, and links are only populated in the parent document. Both of these parameters support JSoup selectors, which provides a rich syntax for selecting HTML and CSS elements.

    When entering configuration values in the UI, use unescaped characters, such as \t for the tab character. When entering configuration values in the API, use escaped characters, such as \\t for the tab character.

    Example using HTML elements

    The following example retrieves all URL values for links with the CSS class resources.

    <p>For more information, consult the following resources:</p><br>
<ul>
  <li><a class="resources" href="https://example.com/books">Books on plants and animals</a></li>
  <li><a class="resources" href="https://example.com/illustrations">Botanical illustrations</a></li>
  <li><a href="https://example.com/mammals">Small mammal identification</a></li>
</ul>

    The following table provides an explanation of the example:

    Parameter Example value Description

    Select rule

    a.resources

    The JSoup parser finds <a> link elements that are assigned the class resources.

    Attribute to map

    href

    The URL of each <a> link element, contained within the href attribute, is parsed.

    Target field

    resourceLinks_ss

    The Fusion field to which values are saved.

    Multi-valued

    false

    If true, the parser will save multiple values for the same element. In this case, the <a> link element can only contain one href value, so the example configuration is false.
    In this example, the last link URL, which is for "Small mammal identification," lacks the resources class. As a result, it is not captured by this configuration.

    HTML Content Extraction

    The Context Extraction setting in the HTML Parser has a special meaning.

    By default, when unchecked or false, the HTML parser will essentially attempt to extract the text of the entire HTML page as the text that will be used in the Solr document.

    However, when Content Extract is checked, or true, a set of heuristic rules are applied to attempt to automatically determine what node in the entire page is most likely to be the page content, based on the tree of nodes inside it and the text component of all the sub-nodes.

    This may be helpful when you have a variety of different page formats, and also significant text on the pages which is not useful to add to the documents, such as significant text in the page header or footer.

    However, since the algorithm is based on heuristics , it is possible that the results could change as the site is altered. When you need a high degree of certainty, we recommend that you use explicit rules of which nodes to extract to match your local configuration.

    Parse html content

    id - string

    Default: 50e5be80-b2b7-432b-b7cb-4e6c0748acec

    label - string

    A label for this Parser Stage

    <= 255 characters

    enabled - boolean

    Default: true

    mediaTypes - array[string]

    Documents with a media type on this list will be matched by this parser stage. See inheritMediaTypes / use default media types for more.

    inheritMediaTypes - boolean

    Each parser stage has a built-in list of media types it handles by default. If this setting is true, that list will be used along with any optional additional types provided in the mediaTypes list. If this setting is false, this stage will only be selected for media types in the mediaTypes list, and the mediaTypes list becomes a mandatory property which must have at least one valid media type.

    Default: true

    pathPatterns - array[object]

    Specify a file name or pattern that must be matched for this parser stage to run. Forward slashes ("/") are used to join names of files inside archives with the archive name.

    object attributes:{syntax : {
     display name: Pattern type
     type: string
    }    pattern : {
     display name: File name or pattern
     type: string
    }    }

    errorHandling - string

    Default: mark

    Allowed values: ignorelogfailmark

    outputFieldPrefix - string

    Fields extracted by this parser will be prefixed with this string. The remainder of the field name will be as detected in the stream

    <= 20 characters

    Match pattern: ^$|^[A-Za-z_][A-Za-z0-9_\-\.]+$

    charset - stringrequired

    Example: "UTF-8"

    Default: detect

    recordSelector - string

    keepParent - boolean

    Keep or discard parent document with selected records. Has no effect if Record Selector is not specified.

    Default: true

    mappings - array[object]

    object attributes:{selectRule required : {
     display name: Select Rule
     type: string
    }    attribute : {
     display name: Attribute
     type: string
    }    field required : {
     display name: Field
     type: string
    }    multivalued : {
     display name: Multi Valued
     type: boolean
    }    }

    extractHtmlLinks - boolean

    Collect links explicitly declared in HTML document

    Default: false

    extractBodyText - boolean

    Extract body as a text

    Default: true

    contentExtractionConfig - Content extraction

    Extract rich formatted content and metadata

    extractContent - boolean

    Default: true

    extractMetadata - boolean

    Default: true

    preserveContentFormat - boolean

    Default: false

    metadataPrefix - string

    metatagsPrefix - string

    Optional prefix to add to metatags (except id and charset) captured from an html document to used as a fields as-is (id and charset get prefix) from document

    Default: false

    type - stringrequired

    Default: html

    Allowed values: html