> ## Documentation Index
> Fetch the complete documentation index at: https://doc.lucidworks.com/llms.txt
> Use this file to discover all available pages before exploring further.

# HTML Parser Stage

export const schema = {
  "type": "object",
  "title": "HTML",
  "description": "Parse html content",
  "required": ["charset", "type"],
  "properties": {
    "id": {
      "type": "string",
      "title": "Parser ID",
      "default": "ea1f6c9b-e5e1-4f15-a887-7d0a52833d23"
    },
    "label": {
      "type": "string",
      "title": "Label",
      "description": "A label for this Parser Stage",
      "maxLength": 255
    },
    "enabled": {
      "type": "boolean",
      "title": "Enable this Parser Stage",
      "default": true
    },
    "mediaTypes": {
      "type": "array",
      "title": "Media Types to match",
      "description": "Documents with a media type on this list will be matched by this parser stage. See inheritMediaTypes / use default media types for more.",
      "items": {
        "type": "string",
        "pattern": "^[^\\/]+\\/[^\\/]+$",
        "format": "rfc2646"
      }
    },
    "inheritMediaTypes": {
      "type": "boolean",
      "title": "Match default media types in this Parser Stage",
      "description": "Each parser stage has a built-in list of media types it handles by default. If this setting is true, that list will be used along with any optional additional types provided in the mediaTypes list. If this setting is false, this stage will only be selected for media types in the mediaTypes list, and the mediaTypes list becomes a mandatory property which must have at least one valid media type.",
      "default": true
    },
    "ignoredMediaTypes": {
      "type": "array",
      "title": "Media Types to ignore",
      "description": "Documents with a media type on this list will be not be processed by this parser stage.",
      "items": {
        "type": "string",
        "pattern": "^[^\\/]+\\/[^\\/]+$",
        "format": "rfc2646"
      }
    },
    "pathPatterns": {
      "type": "array",
      "title": "File names to parse",
      "description": "Specify a file name or pattern that must be matched for this parser stage to run. Forward slashes (\"/\") are used to join names of files inside archives with the archive name.",
      "items": {
        "type": "object",
        "properties": {
          "syntax": {
            "type": "string",
            "title": "Pattern type",
            "description": "glob uses bash shell-style wildcards; regex uses Java (PCRE-style) regex",
            "enum": ["glob", "regex"],
            "default": "glob"
          },
          "pattern": {
            "type": "string",
            "title": "File name or pattern",
            "description": "e.g.: \"z.txt\" or \"*.md\" or \"/a/*/b/f.txt\" for glob; \"z.txt$\" or \".*\\.txt$\" or \"^/a/[^\\/]*/b/f.txt$\" for regex"
          }
        }
      }
    },
    "errorHandling": {
      "type": "string",
      "title": "Error Handling",
      "enum": ["ignore", "log", "fail", "mark"],
      "default": "mark"
    },
    "outputFieldPrefix": {
      "type": "string",
      "title": "Prefix parsed fields with",
      "description": "Fields extracted by this parser will be prefixed with this string. The remainder of the field name will be as detected in the stream",
      "maxLength": 20,
      "pattern": "^$|^[A-Za-z_][A-Za-z0-9_\\-\\.]+$"
    },
    "charset": {
      "type": "string",
      "title": "Character Set",
      "description": "Example: \"UTF-8\"",
      "default": "detect"
    },
    "recordSelector": {
      "type": "string",
      "title": "Record Selector"
    },
    "keepParent": {
      "type": "boolean",
      "title": "Keep Parent Document?",
      "description": "Keep or discard parent document with selected records. Has no effect if Record Selector is not specified.",
      "default": true
    },
    "excludeFilters": {
      "type": "array",
      "title": "Exclude filters",
      "description": "Jsoup-formatted selectors for elements to exclude from the HTML document.",
      "items": {
        "type": "string"
      }
    },
    "filterBeforeMapping": {
      "type": "boolean",
      "title": "Filter before mapping",
      "description": "Apply exclude filters before performing HTML field mapping.",
      "default": false
    },
    "filterBeforeExtractingLinks": {
      "type": "boolean",
      "title": "Filter before extracting links",
      "description": "Apply exclude filters before performing link extraction.",
      "default": false
    },
    "mappings": {
      "type": "array",
      "title": "HTML Element Mappings",
      "items": {
        "type": "object",
        "required": ["selectRule", "field"],
        "properties": {
          "selectRule": {
            "type": "string",
            "title": "Select Rule",
            "description": "A jsoup selection rule, for example 'div#foo' to select '<div id=\"foo\">...</div>'"
          },
          "attribute": {
            "type": "string",
            "title": "Attribute to map",
            "description": "What attribute of the selected element to map. For example 'href' to get the link URL from an '<a>' tag. Special values are '.outerText', '.html', '.outerHtml' and '.data'. If left blank, the text within this element will be mapped."
          },
          "field": {
            "type": "string",
            "title": "Target Field",
            "description": "The field in which to save the mapped element"
          },
          "multivalued": {
            "type": "boolean",
            "title": "Multi-valued",
            "description": "Set to true to map multiple elements if there is more than one match for the select rule",
            "default": false
          }
        }
      }
    },
    "extractHtmlLinks": {
      "type": "boolean",
      "title": "Extract HTML links",
      "description": "Collect links explicitly declared in HTML document",
      "default": false
    },
    "extractBodyText": {
      "type": "boolean",
      "title": "Extract body as a text",
      "description": "Extract body as a text",
      "default": true
    },
    "contentExtractionConfig": {
      "type": "object",
      "title": "Content Extraction (Experimental)",
      "description": "Attempt to extract the content that matters using advanced heuristics. Might not work for your website.",
      "properties": {
        "extractContent": {
          "type": "boolean",
          "title": "Extract page content",
          "default": true
        },
        "extractMetadata": {
          "type": "boolean",
          "title": "Extract metadata",
          "default": true
        },
        "preserveContentFormat": {
          "type": "boolean",
          "title": "Store readable html version of body content",
          "default": false
        },
        "metadataPrefix": {
          "type": "string",
          "title": "Optional prefix for rich content and extracted metadata"
        }
      }
    },
    "metatagsPrefix": {
      "type": "string",
      "title": "Optional prefix for metatags captured html document",
      "description": "Optional prefix to add to metatags (except id and charset) captured from an html document to used as a fields as-is (id and charset get prefix) from document",
      "default": "false"
    },
    "type": {
      "type": "string",
      "enum": ["html"],
      "default": "html"
    }
  },
  "additionalProperties": false,
  "category": "Other",
  "categoryPriority": 1,
  "unsafe": false
};

export const SchemaParamFields = ({schema}) => {
  const sanitize = str => {
    if (typeof str !== "string") return str;
    return str.replace(/^"(.*)"$/s, "$1").replace(/\\/g, "").replace(/"/g, "'");
  };
  const formatDescription = str => {
    const s = sanitize(str);
    return (/[.!?]\)*$/).test(s) ? s : `${s}.`;
  };
  const {description, properties = {}, required: requiredProps = []} = schema;
  const visibleProps = useMemo(() => Object.entries(properties).filter(([, prop]) => !prop.hints?.includes("hidden")), [properties]);
  return <div>
      {description && <p>{formatDescription(description)}</p>}

      {visibleProps.map(([name, prop]) => {
    const isRequired = requiredProps.includes(name);
    const hasDefault = prop.default !== undefined;
    const rawDefault = prop.default;
    const isComplexDefault = hasDefault && (typeof rawDefault === "object" || typeof rawDefault === "string" && (rawDefault.length > 20 || rawDefault.includes('"')));
    const fieldProps = {
      key: name,
      body: prop.title || name,
      type: prop.type,
      ...prop.title && ({
        post: [<><span className="text-stone-400 dark:text-stone-500">API property: </span>{name}</>]
      }),
      ...isRequired && ({
        required: true
      }),
      ...!isComplexDefault && hasDefault ? {
        default: sanitize(String(rawDefault))
      } : {}
    };
    const isObject = prop.type === "object" && prop.properties;
    const isArrayOfObjects = prop.type === "array" && prop.items?.type === "object" && prop.items.properties;
    return <ParamField {...fieldProps}>
            {prop.description && <p>{formatDescription(prop.description)}</p>}

            {isComplexDefault && <div className="flex">
                <p>
                  <strong>Default:</strong>
                </p>
                <pre className="!my-0">
                  <code>
                    {JSON.stringify(rawDefault, null, 2)}
                  </code>
                </pre>
              </div>}

            {isArrayOfObjects && <div className="flex">
              <p>
                <strong>Object attributes:</strong>
              </p>
              <pre className="!my-0">
                <code>
                  {'{\n'}
                  {Object.entries(prop.items.properties).map(([iname, iprop]) => <>
                      {`  ${iname}`}
                      {prop.items?.required?.includes(iname) && <span style={{
      color: 'red'
    }}> required</span>}
                      {`: {\n    display name: ${sanitize(iprop.title || '')}\n    type: ${iprop.type}\n  }\n`}
                    </>)}
                  {'}'}
                </code>
              </pre>
              </div>}

            {isObject && <Expandable title="properties">
                <SchemaParamFields schema={{
      properties: prop.properties,
      required: prop.required
    }} />
              </Expandable>}
          </ParamField>;
  })}
    </div>;
};

export const LwTemplate = ({title = "Key questions to get you started", icon = "sparkles", cta = "Powered by Agent Studio", linkHref = "https://lucidworks.com/demo/?utm_source=docs&utm_medium=referral&utm_campaign=docs_cta_ai"}) => {
  const [isLoaded, setIsLoaded] = useState(false);
  useEffect(() => {
    const timer = setTimeout(() => {
      setIsLoaded(true);
    }, 500);
    return () => clearTimeout(timer);
  }, []);
  return <div className="lw-template-container">
      <Card title={title} icon={icon}>
        {isLoaded && <span dangerouslySetInnerHTML={{
    __html: `<lw-template id="a029c1a9-28be-427e-b0e1-5d918920246a"></lw-template
            >`
  }} />}
        <Link href={linkHref} className="agent-studio-link text-left text-gray-600 gap-2 dark:text-gray-400 text-sm font-medium flex flex-row items-center hover:text-primary dark:hover:text-primary-light group-hover:text-primary group-hover:dark:text-primary-light">Powered by Lucidworks Agent Studio</Link>
      </Card>
    </div>;
};

[localhost link]: http://localhost:3000/docs/5/fusion/reference/config-ref/parser-stages/html-parser

[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/reference/config-ref/parser-stages/html-parser

[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/374

This parser stage processes the following HTML elements:

* `<title>`
* `<body>` (with tags removed)
* `<meta>`
* `<a>` and `<link>`

Additionally, you can configure [JSoup selectors](https://jsoup.org/cookbook/extracting-data/) to extract specific HTML and CSS elements from a document and map them to PipelineDocument fields.
For example, you could use this to process navigational `DIV` elements one way, then process content-ful `DIV` elements another way.

HTML Parser Stage supports jsoup selector methods by allowing the following special attribute values starting with `.` in `HtmlMappingRule`:

* `.text`: [jsoup Element text method](https://jsoup.org/apidocs/org/jsoup/nodes/Element.html#text%28%29)
* `.ownText`: [jsoup Element ownText method](https://jsoup.org/apidocs/org/jsoup/nodes/Element.html#ownText%28%29)
* `.html`: [jsoup Element html method](https://jsoup.org/apidocs/org/jsoup/nodes/Element.html#html%28%29)
* `.outerHtml`: [jsoup Element outerHtml method](https://jsoup.org/apidocs/org/jsoup/nodes/Node.html#outerHtml%28%29)
* `.data`: [jsoup Element data method](https://jsoup.org/apidocs/org/jsoup/nodes/Element.html#data%28%29)

<Note>
  The HTML Transformation index pipeline stage is deprecated in favor of this parser stage.
</Note>

HTML and CSS elements can be selected for extraction into new documents or fields:

* To create new documents from selected elements, configure `recordSelector`.
* To create new fields from selected elements, configure `mappings`.

Title, body, metadata, and links are only populated in the parent document. Both of these parameters support [JSoup selectors](https://jsoup.org/cookbook/extracting-data/), which provides a rich syntax for selecting HTML and CSS elements.

<Tip>
  When entering configuration values in the UI, use *unescaped* characters, such as `\t` for the tab character. When entering configuration values in the API, use *escaped* characters, such as `\\t` for the tab character.
</Tip>

<LwTemplate />

## HTML content extraction

The Context Extraction setting in the HTML Parser has a special meaning.

By default, when unchecked or false, the HTML parser will essentially attempt to extract the text of the entire HTML page as the text that will be used in the Solr document.

However, when Content Extract is checked, or true, a set of heuristic rules are applied to attempt to automatically determine what node in the entire page is most likely to be the page content, based on the tree of nodes inside it and the text component of all the sub-nodes.

This may be helpful when you have a variety of different page formats, and also significant text on the pages which is not useful to add to the documents, such as significant text in the page header or footer.

However, since the algorithm is based on heuristics , it is possible that the results could change as the site is altered. When you need a high degree of certainty, we recommend that you use explicit rules of which nodes to extract to match your local configuration.

<SchemaParamFields schema={schema} />