> ## Documentation Index
> Fetch the complete documentation index at: https://doc.lucidworks.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Underperforming Queries

export const LwTemplate = ({title = "Key questions to get you started", icon = "sparkles", cta = "Powered by Agent Studio", linkHref = "https://lucidworks.com/demo/?utm_source=docs&utm_medium=referral&utm_campaign=docs_cta_ai"}) => {
  const [isLoaded, setIsLoaded] = useState(false);
  useEffect(() => {
    const timer = setTimeout(() => {
      setIsLoaded(true);
    }, 500);
    return () => clearTimeout(timer);
  }, []);
  return <div className="lw-template-container">
      <Card title={title} icon={icon}>
        {isLoaded && <span dangerouslySetInnerHTML={{
    __html: `<lw-template id="a029c1a9-28be-427e-b0e1-5d918920246a"></lw-template
            >`
  }} />}
        <Link href={linkHref} className="agent-studio-link text-left text-gray-600 gap-2 dark:text-gray-400 text-sm font-medium flex flex-row items-center hover:text-primary dark:hover:text-primary-light group-hover:text-primary group-hover:dark:text-primary-light">Powered by Lucidworks Agent Studio</Link>
      </Card>
    </div>;
};

export const InlineImage = ({src, alt = '', height = '2em'}) => {
  return <img src={src} alt={alt} style={{
    display: 'inline',
    verticalAlign: 'start',
    height: height,
    margin: '0'
  }} />;
};

[localhost link]: http://localhost:3000/docs/4/fusion-ai/concepts/query-rewriting/underperforming-queries

[mintlify link]: https://doc.lucidworks.com/docs/4/fusion-ai/concepts/query-rewriting/underperforming-queries

[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/486

The Underperforming Query Rewriting feature:

* Uses signals data to identify underperforming queries
* Suggests improved queries that could produce better conversion rates

When underperforming query rewriting is enabled and an incoming query contains a matching underperforming query term, the original query term is replaced by the improved query term.

Query improvements are applied in the [Text Tagger stage](/docs/4/fusion-server/reference/pipeline-stages/query/text-tagger-query-stage) of the query pipeline.

The [Head/Tail Analysis job](/docs/4/fusion-ai/reference/jobs/head-tail-analysis) automatically creates query improvements based on your AI-generated data. When you navigate to **Relevance** > **Rules**, the application displays the **Query Rewriting** screen. Select **Underperforming Query Rewriting**. The application displays the **Underperforming Query Rewriting** screen.

<Tip>
  When you manually add new query improvements, subsequent job runs use those documents as input for machine learning to improve the job’s output. Unlike job-generated documents, manually-added query rewriting documents are never overwritten by new job output.
</Tip>

<Note>
  Job-generated query improvements are always assigned an initial status of "Pending", never "Auto". Query improvements must be explicitly approved and published in order to be copied to the `_query_rewrite` collection.
</Note>

<img src="https://mintcdn.com/lucidworks/vupE2UCZdg04NdXx/assets/images/4.2/rules-head-tail.png?fit=max&auto=format&n=vupE2UCZdg04NdXx&q=85&s=194331dabc803decd551bfdc8f6665eb" alt="Underperforming Queries screen" width="2880" height="1524" data-path="assets/images/4.2/rules-head-tail.png" />

<LwTemplate />

## Reviewing auto-generated query improvements

Query improvements that are automatically generated by the [Head/Tail Analysis job](/docs/4/fusion-ai/reference/jobs/head-tail-analysis) are assigned the following status value:

* **Auto**\
  All results will be automatically deployed to the `_query_rewrite` collection.\
  No action is required on these results, but you can edit them if you wish.

## Adding new query improvements

You can manually add query improvements in addition to any generated by the [Head/Tail Analysis job](/docs/4/fusion-ai/reference/jobs/head-tail-analysis).

### How to add a query improvement

1. Navigate to **Relevance** > **Rules**. The application displays the **Query Rewriting** screen.
2. Select **Underperforming Query Rewriting**. The application displays the **Underperforming Query Rewriting** screen.
3. At the bottom of the rules list, click the <InlineImage src="/assets/images/4.2/icons/add-rule.png" /> icon.\
   A new query improvement appears at the top of the list:

   <img src="https://mintcdn.com/lucidworks/vupE2UCZdg04NdXx/assets/images/4.2/rules-headtail-add.png?fit=max&auto=format&n=vupE2UCZdg04NdXx&q=85&s=2fa29f01fabb180e29703ae053e42b38" alt="Add a query improvement" width="2562" height="1284" data-path="assets/images/4.2/rules-headtail-add.png" />
4. Enter the underperforming query.
5. Enter one or more query improvements.

<Tip>
  It is not necessary to set a confidence value.
</Tip>

6. Select the query improvement’s status, depending on whether you want to deploy it the next time you publish your changes ("Approved") or save it for further review ("Pending").
7. Click the check mark to save the new query improvement:

   <img src="https://mintcdn.com/lucidworks/vupE2UCZdg04NdXx/assets/images/4.2/rules-headtail-save.png?fit=max&auto=format&n=vupE2UCZdg04NdXx&q=85&s=40391d9eb68b9c2f426f891a57826fdc" alt="Save a query improvement" width="2562" height="1284" data-path="assets/images/4.2/rules-headtail-save.png" />

## Publishing your changes

### How to publish updated query improvements

1. In the **Underperforming Query Rewriting** screen, click the **PUBLISH** button.\
   Fusion prompts you to confirm that you want to publish your changes.
2. Click **PUBLISH**.

<Tip>
  You can un-publish a query rewrite by changing its status to "denied", then clicking **PUBLISH**.
</Tip>

## Query rewrite jobs post-processing cleanup

To perform more extensive cleanup of query rewrites, complete the procedures in **Query Rewrite Jobs Post-processing Cleanup**.

<Accordion title="Query Rewrite Jobs Post-processing Cleanup">
  <Danger>The Synonym Detection job uses the output of the Misspelling Detection job and Phrase Extraction job.  Therefore, post processing *must* occur in the order specified in this topic for the [Synonym detection job cleanup](#synonym-detection-job-cleanup), [Phrase extraction job cleanup](#phrase-extraction-job-cleanup), and [Misspelling detection job cleanup](#misspelling-detection-job-cleanup) procedures.  The [Head-Tail Analysis job cleanup](#head-tail-analysis-job-cleanup) can occur in any order.</Danger>

  ## Synonym detection job cleanup

  Use this job to remove low confidence synonyms.

  ### Prerequisites

  Complete this:

  * **AFTER** the Misspelling Detection and Phrase Extraction jobs have successfully completed.
  * **BEFORE** removing low confidence synonym suggestions generated in the *post processing* phrase extraction cleanup and misspelling detection cleanup procedures detailed later in this topic.

  ### Remove low confidence synonym suggestions

  Use either a [Synonym cleanup method 1 - API call](#synonym-cleanup-method-1-api-call) or the [Synonym cleanup method 2 - Fusion Admin UI](#synonym-cleanup-method-2-fusion-admin-ui) to remove low confidence synonym suggestions.

  #### Synonym cleanup method 1 - API call

  1. Open the `delete_lowConf_synonyms.json` file.
     ```json theme={"dark"}
     {
     "type" : "rest-call",
     "id" : "DC_Large_QR_DELETE_LOW_CONFIDENCE_SYNONYMS",
     "callParams" : {
         "uri" : "solr://DC_Large_query_rewrite_staging/update",
         "method" : "post",
         "queryParams" : {
         "wt" : "json"
         },
         "headers" : { },
         "entity" : "<root><delete><query>type:synonym AND confidence:[0 TO 0.0005]</query></delete><commit/></root>"
     },
     "type" : "rest-call",
     "type" : "rest-call"
     }
     ```
       <Note>
         **REQUEST ENTITY** specifies the threshold for low confidence synonyms. Edit the upper range from 0.0005 to increase or decrease the threshold based on your data.
       </Note>
  2. Enter `<your query_rewrite_staging collection name/update>` in the **uri** field. An example URI value for an app called `DC_Large` would be `DC_Large_query_rewrite_staging/update`.
  3. Change the `id` field if applicable.
  4. Specify the **upper confidence level** in the **entity** field.

     <Note>   The **entity** field specifies the threshold for low confidence synonyms. Edit the upper range to increase or decrease the threshold based on your data.</Note>

  #### Synonym cleanup method 2 - Fusion Admin UI

  1. Log in to Fusion and select **Collections > Jobs**.
  2. Select **Add+ > Custom and Other Jobs > REST Call**.
  3. Enter **delete-low-confidence-synonyms** in the **ID** field.
  4. Enter `<your query_rewrite_staging collection name/update>` in the **ENDPOINT URI** field. An example URI value for an app called `DC_Large` would be `DC_Large_query_rewrite_staging/update`.
  5. Enter **POST** in the **CALL METHOD** field.
  6. In the **QUERY PARAMETERS** section, select **+** to add a property.
  7. Enter **wt** in the **Property Name** field.
  8. Enter **json** in the **Property Value** field.
  9. In the **REQUEST PROTOCOL HEADERS** section, select **+** to add a property.
  10. Enter the following as a **REQUEST ENTITY (AS STRING)**

      `<root><delete><query>type:synonym AND confidence: [0 TO 0.0005]</query></delete><commit/></root>`

      <Note>   **REQUEST ENTITY** specifies the threshold for low confidence synonyms. Edit the upper range from 0.0005 to increase or decrease the threshold based on your data.</Note>

  ### Delete all synonym suggestions

  To delete *all* of the synonym suggestions, enter the following in the **REQUEST ENTITY** section:

  `<root><delete><query>type:synonym</query></delete><commit/></root>`

  <Note>This entry may be helpful when tuning the synonym detection job and testing different configuration parameters.</Note>

  ## Phrase extraction job cleanup

  Use this job to remove low confidence phrase suggestions.

  ### Prerequisites

  Complete this:

  * **AFTER** you complete [Synonym detection job cleanup](#synonym-detection-job-cleanup)

  ### Remove low confidence phrase suggestions

  Use either a [Phrase cleanup method 1 - API call](#phrase-cleanup-method-1-api-call) or the [Phrase cleanup method 2 - Fusion Admin UI](#phrase-cleanup-method-2-fusion-admin-ui) to remove low confidence phrase suggestions.

  #### Phrase cleanup method 1 - API call

  1. Open the `delete_lowConf_phrases.json` file.
     ```json theme={"dark"}
     {
     "type" : "rest-call",
     "id" : "DC_Large_QR_DELETE_LOW_CONFIDENCE_PHRASES",
     "callParams" : {
         "uri" : "solr://DC_Large_query_rewrite_staging/update",
         "method" : "post",
         "queryParams" : {
         "wt" : "json"
         },
         "headers" : { },
         "entity" : " <root><delete><query>type:phrase AND confidence:[0 TO <INSERT VALUE HERE>]</query></delete><commit/></root>"
     },
     "type" : "rest-call",
     "type" : "rest-call"
     }
     ```
  2. Enter `<your query_rewrite_staging collection name/update>` in the **uri** field. An example URI value for an app called `DC_Large` would be `DC_Large_query_rewrite_staging/update`.
  3. Change the **id** field if applicable.
  4. Specify the **upper confidence level** in the **entity** field.

     <Note>   The **entity** field specifies the threshold for low confidence phrases. Edit the upper range to increase or decrease the threshold based on your data.</Note>

  #### Phrase cleanup method 2 - Fusion Admin UI

  1. Log in to Fusion and select **Collections > Jobs**.
  2. Select **Add+ > Custom and Other Jobs > REST Call**.
  3. Enter **remove-low-confidence-phrases** in the **ID** field.
  4. Enter `<your query_rewrite_staging collection name/update>` in the **ENDPOINT URI** field. An example URI value for an app called `DC_Large` would be `DC_Large_query_rewrite_staging/update`.
  5. Enter **POST** in the **CALL METHOD** field.
  6. In the **QUERY PARAMETERS** section, select **+** to add a property.
  7. Enter **wt** in the **Property Name** field.
  8. Enter **json** in the **Property Value** field.
  9. In the **REQUEST PROTOCOL HEADERS** section, select **+** to add a property.
  10. Enter the following as a **REQUEST ENTITY (AS STRING)**

      `<root><delete><query>type:phrase AND confidence: [0 TO <insert value>]</query></delete><commit/></root>`

      <Note>   **REQUEST ENTITY** specifies the threshold for low confidence phrases. Edit the upper range to increase or decrease the threshold based on your data.</Note>

  ### Delete all phrase suggestions

  To delete *all* of the phrase suggestions, enter the following in the **REQUEST ENTITY** section:

  `<root><delete><query>type:phrase</query></delete><commit/></root>`

  <Note>This entry may be helpful when tuning the phrase extraction job and testing different configuration parameters.</Note>

  ## Misspelling detection job cleanup

  Use this job to remove low confidence spellings (also referred to as misspellings).

  ### Prerequisites

  Complete this:

  * **AFTER** you complete [Synonym detection job cleanup](#synonym-detection-job-cleanup) and [Phrase extraction job cleanup](#phrase-extraction-job-cleanup)

  ### Remove misspelling suggestions

  Use either a [Misspelling cleanup method 1 - API call](#misspelling-cleanup-method-1-api-call) or the [Misspelling cleanup method 2 - Fusion Admin UI](#misspelling-cleanup-method-2-fusion-admin-ui) to remove misspelling suggestions.

  #### Misspelling cleanup method 1 - API call

  1. Open the `delete_lowConf_misspellings.json` file.
     ```json theme={"dark"}
     {
     "type" : "rest-call",
     "id" : "DC_Large_QR_DELETE_LOW_CONFIDENCE_MISSPELLINGS",
     "callParams" : {
         "uri" : "solr://DC_Large_query_rewrite_staging",
         "method" : "post",
         "queryParams" : {
         "wt" : "json"
         },
         "headers" : { },
         "entity" : "<root><delete><query>type:spell AND confidence:[0 TO 0.5]</query></delete><commit/></root>"
     },
     "type" : "rest-call",
     "type" : "rest-call"
     }
     ```
  2. Enter `<your query_rewrite_staging collection name/update>` in the **uri** field. An example URI value for an app called `DC_Large` would be `DC_Large_query_rewrite_staging/update`.
  3. Change the **id** field if applicable.
  4. Specify the **upper confidence level** in the **entity** field.

     <Note>   The **entity** field specifies the threshold for low confidence spellings. Edit the upper range to increase or decrease the threshold based on your data.</Note>

  #### Misspelling cleanup method 2 - Fusion Admin UI

  1. Log in to Fusion and select **Collections > Jobs**.
  2. Select **Add+ > Custom and Other Jobs > REST Call**.
  3. Enter **remove-low-confidence-spellings** in the **ID** field.
  4. Enter `<your query_rewrite_staging collection name/update>` in the **ENDPOINT URI** field. An example URI value for an app called `DC_Large` would be `DC_Large_query_rewrite_staging/update`.
  5. Enter **POST** in the **CALL METHOD** field.
  6. In the **QUERY PARAMETERS** section, select **+** to add a property.
  7. Enter **wt** in the **Property Name** field.
  8. Enter **json** in the **Property Value** field.
  9. In the **REQUEST PROTOCOL HEADERS** section, select **+** to add a property.
  10. Enter the following as a **REQUEST ENTITY (AS STRING)**

      `<root><delete><query>type:spell AND confidence: [0 TO 0.5]</query></delete><commit/></root>`

      <Note>   **REQUEST ENTITY** specifies the threshold for low confidence spellings. Edit the upper range from 0.5 to increase or decrease the threshold based on your data.</Note>

  ### Delete all misspelling suggestions

  To delete *all* of the misspelling suggestions, enter the following in the **REQUEST ENTITY** section:

  `<root><delete><query>type:spell</query></delete><commit/></root>`

  <Note>This entry may be helpful when tuning the misspelling detection job and testing different configuration parameters.</Note>

  ## Head-tail analysis job cleanup

  The head-tail analysis job puts tail queries into one of multiple reason categories. For example, a tail query that includes a number might be assigned to the 'numbers' reason category. If the output in a particular category is not useful, you can remove it from the results. The examples in this section remove the numbers category.

  ### Prerequisites

  The head-tail analysis job cleanup does *not* have to occur in a specific order.

  ### Remove head-tail analysis query suggestions

  Use either a [Head-tail analysis cleanup method 1 - API call](#head-tail-analysis-cleanup-method-1-api-call) or the [Head-tail analysis cleanup method 2 - Fusion Admin UI](#head-tail-analysis-cleanup-method-2-fusion-admin-ui) to remove query category suggestions.

  #### Head-tail analysis cleanup method 1 - API call

  1. Open the `delete_lowConf_headTail.json` file.
     ```json theme={"dark"}
     {
     "type" : "rest-call",
     "id" : "DC_Large_QR_HEAD_TAIL_CLEANUP",
     "callParams" : {
         "uri" : "solr://DC_Large_query_rewrite_staging/update",
         "method" : "post",
         "queryParams" : {
         "wt" : "json"
         },
         "headers" : { },
         "entity" : "<root><delete><query>reason_code_s:(\"number\" \"number spelling\" \"number rare-term\" \"question number other-specific\" \"number others\" \"number other-specific\" \"number other-extra\" \"product number other-specific\" \"product number other-extra\" \"product number spelling\" \"product number others\" \"product number rare-term\" \"product question number\" \"product number re-wording\" \"question number other-extra\" \"number re-wording\")</query></delete><commit/></root>"
     },
     "type" : "rest-call",
     "type" : "rest-call"
     }
     ```
  2. Enter `<your query_rewrite_staging collection name/update>` in the **uri** field. An example URI value for an app called `DC_Large` would be `DC_Large_query_rewrite_staging/update`.
  3. Change the **id** field if applicable.

  #### Head-tail analysis cleanup method 2 - Fusion Admin UI

  1. Log in to Fusion and select **Collections > Jobs**.
  2. Select **Add+ > Custom and Other Jobs > REST Call**.
  3. Enter **remove-low-confidence-head-tail** in the **ID** field.
  4. Enter `<your query_rewrite_staging collection name/update>` in the **ENDPOINT URI** field. An example URI value for an app called `DC_Large` would be `DC_Large_query_rewrite_staging/update`.
  5. Enter **POST** in the **CALL METHOD** field.
  6. In the **QUERY PARAMETERS** section, select **+** to add a property.
  7. Enter **wt** in the **Property Name** field.
  8. Enter **json** in the **Property Value** field.
  9. In the **REQUEST PROTOCOL HEADERS** section, select **+** to add a property.
  10. Enter the following as a **REQUEST ENTITY (AS STRING)**
      ```
      <root><delete><query>reason_code_s:("number" "number spelling" "number rare-term" "question number other-specific" "number others" "number other-specific" "number other-extra" "product number other-specific" "product number other-extra" "product number spelling" "product number others" "product number rare-term" "product question number" "product number re-wording" "question number other-extra" "number re-wording")</query></delete><commit/></root>
      ```

  ### Delete all head-tail suggestions

  To delete *all* of the head-tail suggestions, enter the following in the **REQUEST ENTITY** section:

  `<root><delete><query>type:tail</query></delete><commit/></root>`

  <Note>This entry may be helpful when tuning the head-tail job and testing different configuration parameters.</Note>
</Accordion>