> ## Documentation Index
> Fetch the complete documentation index at: https://doc.lucidworks.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Head/Tail Analysis
export const LwTemplate = ({title = "Key questions to get you started", icon = "sparkles", cta = "Powered by Agent Studio", linkHref = "https://lucidworks.com/demo/?utm_source=docs&utm_medium=referral&utm_campaign=docs_cta_ai"}) => {
const [isLoaded, setIsLoaded] = useState(false);
useEffect(() => {
const timer = setTimeout(() => {
setIsLoaded(true);
}, 500);
return () => clearTimeout(timer);
}, []);
return
{isLoaded && `
}} />}
Powered by Lucidworks Agent Studio
;
};
export const InlineImage = ({src, alt = '', height = '2em'}) => {
return ;
};
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-out/query-enhancement/underperforming-queries
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-enhancement/underperforming-queries
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/486
Head/tail analysis is:
* Also known as the Underperforming Query Rewriting feature
* Uses signals data to identify underperforming queries
* Suggests improved queries that could produce better conversion rates
This feature is deprecated in Fusion 5.9.15 and will be removed in a future release.
Lucidworks recommends migrating to [Neural Hybrid Search](/docs/5/fusion/hybrid-search/overview), which achieves superior relevance compared to legacy machine learning methods.
When an incoming query contains a matching underperforming query, the original query is replaced by an improved query.
These improvements can be:
* Suggested by the [Head/Tail Analysis Spark job](/docs/5/fusion/reference/config-ref/jobs/head-tail-analysis) operating on your signals data
* Created manually using the [Rules Editor](/docs/5/fusion/getting-data-out/query-enhancement/rules-editor) or underlying API
Query improvements are applied in the [Text Tagger stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/text-tagger-query-stage) of the query pipeline.
The [Head/Tail Analysis job](/docs/5/fusion/reference/config-ref/jobs/head-tail-analysis) automatically creates query improvements based on your AI-generated data. When you navigate to **Relevance** > **Rules** > **Rewrite** and select the **Head/Tail** tab, the application displays the **Underperforming Query Rewriting** screen.
Job-generated query improvements are always assigned an initial status of "Pending", never "Auto". Query improvements must be explicitly approved and published in order to be copied to the `COLLECTION_NAME_query_rewrite` collection.
The course for **Query Analytics** focuses on how Fusion provides query analytics to detect and improve underperforming queries.
## Reviewing auto-generated query improvements
Query improvements that are automatically generated by the [Head/Tail Analysis job](/docs/5/fusion/reference/config-ref/jobs/head-tail-analysis) are assigned the following status value:
* **Auto**
All results will be automatically deployed to the `_query_rewrite` collection.\
No action is required on these results, but you can edit them if you wish.
## Adding new query improvements
You can manually add query improvements in addition to any generated by the [Head/Tail Analysis job](/docs/5/fusion/reference/config-ref/jobs/head-tail-analysis).
### How to add a query improvement
1. Navigate to **Relevance** > **Rules** > **Rewrite**.
2. Select the **Head/Tail** tab. The application displays the **Underperforming Query Rewriting** screen.
3. At the bottom of the rules list, click the icon.\
A new query improvement appears at the top of the list:
4. Enter the underperforming query.
5. Enter one or more query improvements.
It is not necessary to set a confidence value.
6. Select the query improvement’s status, depending on whether you want to deploy it the next time you publish your changes ("Approved") or save it for further review ("Pending").
7. Click the check mark to save the new query improvement.
## Publishing your changes
### How to publish updated query improvements
1. In the **Underperforming Query Rewriting** screen, click the **PUBLISH** button.\
Fusion prompts you to confirm that you want to publish your changes.
2. Click **PUBLISH**.
You can un-publish a query rewrite by changing its status to "denied", then clicking **PUBLISH**.
For more information about how to use underperforming query rewrites in the [Rules Editor](/docs/5/fusion/getting-data-out/query-enhancement/rules-editor), see **Use Query Rewrites in the Rules Editor**.
This article describes how to use the Rewrites screen in the [Rules Editor](/docs/5/fusion/getting-data-out/query-enhancement/rules-editor).
## Query rewrite rule types
* **Head/Tail.** Replace underperforming queries with queries that produce higher click-through rates. Query improvements always require review before they are published. Schedule the Head/Tail Analysis job to get AI-generated query improvements based on your signals collection. \
\
Manually created Head/Tail rewrites must provide an "Improved Query" in a format appropriate to the type of request the rewrite will be applied to. For Legacy requests, this means a valid Solr query fit for use as the `q` parameter. For DSL requests this means a JSON string representing the main DSL query. \
\
For more information, see:
* [Underperforming Query Rewriting](/docs/5/fusion/getting-data-out/query-enhancement/underperforming-queries)
* [Head/Tail Analysis Jobs](/docs/5/fusion/reference/config-ref/jobs/head-tail-analysis)
* **Misspelling.** Fusion replaces misspelled query terms with their spelling corrections for better search results. Schedule the Token and Phrase Spell Correction job to get AI-generated spelling corrections based on your signals collection. For more information, see [Misspelling Detection](/docs/5/fusion/getting-data-out/query-enhancement/misspelling-detection).
* **Phrase.** Known phrases are boosted when they appear in new queries. Schedule the Phrase Extraction job to get AI-generated phrases based on your signals collection.
For more information, see [Phrase Detection](/docs/5/fusion/getting-data-out/query-enhancement/phrase-detection).
* **Synonym.** Synonyms are used to expand queries so that they include all known synonyms of the query terms. To retrieve AI-generated synonyms based on your signals collection, schedule the [Synonym Detection job](/docs/5/fusion/reference/config-ref/jobs/synonym-detection).
* **Remove Words** - Remove particular phrases from queries. Unlike other rewrites, these should be entered manually and are not generated by a job. For more information, see [Remove Words](/docs/5/fusion/getting-data-out/query-enhancement/remove-words).
## Facets
### Facet menu
To facet the rules list by **Status** or **Published**, click one of the facet items within the facet categories.
To facet the rules by **Confidence**, click and drag either slider to set the range of confidence you would like to see:
Faceting rules by confidence is not available for Head/Tail rewrites.
### Search
Enter a term in the search bar to quickly facet your query rewrite rules.
## Rules
Begin by selecting one of the query rewrite rule types you will create.
### Create a rule
1. Click **+Add**.
2. In the new table row, complete the required fields, and additional fields as needed.
3. Click the checkmark to save, or close to cancel the changes.
### Edit a rule
1. Hold the pointer over the rule to edit.
2. Click the pencil icon.
3. Enter desired changes and click the checkmark to save, or close to cancel the changes.
### Approve/deny rules
{/* // New or altered rules will need to be approved by a user with the appropriate permissions before the rule can be enabled. */}
1. Hold the pointer over a rule.
2. Select the checkbox that displays for that rule.
3. To modify multiple rules, hold the pointer over each rule and select its corresponding checkbox.
4. Click **Approve** or **Deny** as needed.
{/*
This is not an explicit option in the rewrite screen. Is this implicit by approval/denial?
### Enable/disable a rule
1. Hold the pointer over a rule.
2. Select the checkbox that displays for that rule.
3. To modify multiple rules, hold the pointer over each rule and select its corresponding checkbox.
4. Click **Enable** or **Disable** as needed.
*/}
### Publish a rule
1. Hold the pointer over a rule.
2. Select the checkbox that displays for that rule.
3. To publish multiple rules, hold the pointer over each rule and select its corresponding checkbox.
4. Click **Publish Selected** to publish the selected rules.
#### Publish an individual rule
1. Hold the pointer over a rule.
2. Click the button.
3. To confirm, click **Publish**.
### Delete rules
#### Delete a single rule
1. Hold the pointer over a rule.
2. Click the trash can.
3. To confirm, click **Delete**.
#### Delete multiple rules
1. Hold the pointer over a rule.
2. Select the checkbox that displays for that rule.
3. To delete multiple rules, hold the pointer over each rule and select its corresponding checkbox.
4. Click **Delete**.
5. To confirm, click **Delete** again.
### Sort rules
1. Hold the pointer over column titles. A icon displays for some titles to indicate the rules list can be sorted by that column.
2. Click the title column to sort the rules.
The column that is currently sorting the rules list is indicated by an or icon.
### Export rules
To export the rules list, click **Export**. This generates a `.csv` file that contains your rules.
## Columns
### Customize columns
To customize columns:
1. Hold the pointer over the row of column titles.
2. Click the button.
3. In the Customize Columns to Display screen, select the checkboxes for the columns you want to display. Clear the checkboxes for columns you do not want to display.
4. Click **Save**. To exit without saving your changes, click **Cancel**.
## Query rewrite jobs post-processing cleanup
To perform more extensive cleanup of query rewrites, complete the procedures in **Query Rewrite Jobs Post-processing Cleanup**.
The Synonym Detection job uses the output of the Misspelling Detection job and Phrase Extraction job. Therefore, post processing *must* occur in the order specified in this topic for the [Synonym detection job cleanup](#synonym-detection-job-cleanup), [Phrase extraction job cleanup](#phrase-extraction-job-cleanup), and [Misspelling detection job cleanup](#misspelling-detection-job-cleanup) procedures. The [Head-Tail Analysis job cleanup](#head-tail-analysis-job-cleanup) can occur in any order.
## Synonym detection job cleanup
Use this job to remove low confidence synonyms.
### Prerequisites
Complete this:
* **AFTER** the Misspelling Detection and Phrase Extraction jobs have successfully completed.
* **BEFORE** removing low confidence synonym suggestions generated in the *post processing* phrase extraction cleanup and misspelling detection cleanup procedures detailed later in this topic.
### Remove low confidence synonym suggestions
Use either a [Synonym cleanup method 1 - API call](#synonym-cleanup-method-1-api-call) or the [Synonym cleanup method 2 - Fusion Admin UI](#synonym-cleanup-method-2-fusion-admin-ui) to remove low confidence synonym suggestions.
#### Synonym cleanup method 1 - API call
1. Open the `delete_lowConf_synonyms.json` file.
```json theme={"dark"}
{
"type" : "rest-call",
"id" : "DC_Large_QR_DELETE_LOW_CONFIDENCE_SYNONYMS",
"callParams" : {
"uri" : "solr://DC_Large_query_rewrite_staging/update",
"method" : "post",
"queryParams" : {
"wt" : "json"
},
"headers" : { },
"entity" : "type:synonym AND confidence:[0 TO 0.0005]"
},
"type" : "rest-call",
"type" : "rest-call"
}
```
**REQUEST ENTITY** specifies the threshold for low confidence synonyms. Edit the upper range from 0.0005 to increase or decrease the threshold based on your data.
2. Enter `` in the **uri** field. An example URI value for an app called `DC_Large` would be `DC_Large_query_rewrite_staging/update`.
3. Change the `id` field if applicable.
4. Specify the **upper confidence level** in the **entity** field.
The **entity** field specifies the threshold for low confidence synonyms. Edit the upper range to increase or decrease the threshold based on your data.
#### Synonym cleanup method 2 - Fusion Admin UI
1. Log in to Fusion and select **Collections > Jobs**.
2. Select **Add+ > Custom and Other Jobs > REST Call**.
3. Enter **delete-low-confidence-synonyms** in the **ID** field.
4. Enter `` in the **ENDPOINT URI** field. An example URI value for an app called `DC_Large` would be `DC_Large_query_rewrite_staging/update`.
5. Enter **POST** in the **CALL METHOD** field.
6. In the **QUERY PARAMETERS** section, select **+** to add a property.
7. Enter **wt** in the **Property Name** field.
8. Enter **json** in the **Property Value** field.
9. In the **REQUEST PROTOCOL HEADERS** section, select **+** to add a property.
10. Enter the following as a **REQUEST ENTITY (AS STRING)**
`type:synonym AND confidence: [0 TO 0.0005]`
**REQUEST ENTITY** specifies the threshold for low confidence synonyms. Edit the upper range from 0.0005 to increase or decrease the threshold based on your data.
### Delete all synonym suggestions
To delete *all* of the synonym suggestions, enter the following in the **REQUEST ENTITY** section:
`type:synonym`
This entry may be helpful when tuning the synonym detection job and testing different configuration parameters.
## Phrase extraction job cleanup
Use this job to remove low confidence phrase suggestions.
### Prerequisites
Complete this:
* **AFTER** you complete [Synonym detection job cleanup](#synonym-detection-job-cleanup)
### Remove low confidence phrase suggestions
Use either a [Phrase cleanup method 1 - API call](#phrase-cleanup-method-1-api-call) or the [Phrase cleanup method 2 - Fusion Admin UI](#phrase-cleanup-method-2-fusion-admin-ui) to remove low confidence phrase suggestions.
#### Phrase cleanup method 1 - API call
1. Open the `delete_lowConf_phrases.json` file.
```json theme={"dark"}
{
"type" : "rest-call",
"id" : "DC_Large_QR_DELETE_LOW_CONFIDENCE_PHRASES",
"callParams" : {
"uri" : "solr://DC_Large_query_rewrite_staging/update",
"method" : "post",
"queryParams" : {
"wt" : "json"
},
"headers" : { },
"entity" : " type:phrase AND confidence:[0 TO ]"
},
"type" : "rest-call",
"type" : "rest-call"
}
```
2. Enter `` in the **uri** field. An example URI value for an app called `DC_Large` would be `DC_Large_query_rewrite_staging/update`.
3. Change the **id** field if applicable.
4. Specify the **upper confidence level** in the **entity** field.
The **entity** field specifies the threshold for low confidence phrases. Edit the upper range to increase or decrease the threshold based on your data.
#### Phrase cleanup method 2 - Fusion Admin UI
1. Log in to Fusion and select **Collections > Jobs**.
2. Select **Add+ > Custom and Other Jobs > REST Call**.
3. Enter **remove-low-confidence-phrases** in the **ID** field.
4. Enter `` in the **ENDPOINT URI** field. An example URI value for an app called `DC_Large` would be `DC_Large_query_rewrite_staging/update`.
5. Enter **POST** in the **CALL METHOD** field.
6. In the **QUERY PARAMETERS** section, select **+** to add a property.
7. Enter **wt** in the **Property Name** field.
8. Enter **json** in the **Property Value** field.
9. In the **REQUEST PROTOCOL HEADERS** section, select **+** to add a property.
10. Enter the following as a **REQUEST ENTITY (AS STRING)**
`type:phrase AND confidence: [0 TO ]`
**REQUEST ENTITY** specifies the threshold for low confidence phrases. Edit the upper range to increase or decrease the threshold based on your data.
### Delete all phrase suggestions
To delete *all* of the phrase suggestions, enter the following in the **REQUEST ENTITY** section:
`type:phrase`
This entry may be helpful when tuning the phrase extraction job and testing different configuration parameters.
## Misspelling detection job cleanup
Use this job to remove low confidence spellings (also referred to as misspellings).
### Prerequisites
Complete this:
* **AFTER** you complete [Synonym detection job cleanup](#synonym-detection-job-cleanup) and [Phrase extraction job cleanup](#phrase-extraction-job-cleanup)
### Remove misspelling suggestions
Use either a [Misspelling cleanup method 1 - API call](#misspelling-cleanup-method-1-api-call) or the [Misspelling cleanup method 2 - Fusion Admin UI](#misspelling-cleanup-method-2-fusion-admin-ui) to remove misspelling suggestions.
#### Misspelling cleanup method 1 - API call
1. Open the `delete_lowConf_misspellings.json` file.
```json theme={"dark"}
{
"type" : "rest-call",
"id" : "DC_Large_QR_DELETE_LOW_CONFIDENCE_MISSPELLINGS",
"callParams" : {
"uri" : "solr://DC_Large_query_rewrite_staging",
"method" : "post",
"queryParams" : {
"wt" : "json"
},
"headers" : { },
"entity" : "type:spell AND confidence:[0 TO 0.5]"
},
"type" : "rest-call",
"type" : "rest-call"
}
```
2. Enter `` in the **uri** field. An example URI value for an app called `DC_Large` would be `DC_Large_query_rewrite_staging/update`.
3. Change the **id** field if applicable.
4. Specify the **upper confidence level** in the **entity** field.
The **entity** field specifies the threshold for low confidence spellings. Edit the upper range to increase or decrease the threshold based on your data.
#### Misspelling cleanup method 2 - Fusion Admin UI
1. Log in to Fusion and select **Collections > Jobs**.
2. Select **Add+ > Custom and Other Jobs > REST Call**.
3. Enter **remove-low-confidence-spellings** in the **ID** field.
4. Enter `` in the **ENDPOINT URI** field. An example URI value for an app called `DC_Large` would be `DC_Large_query_rewrite_staging/update`.
5. Enter **POST** in the **CALL METHOD** field.
6. In the **QUERY PARAMETERS** section, select **+** to add a property.
7. Enter **wt** in the **Property Name** field.
8. Enter **json** in the **Property Value** field.
9. In the **REQUEST PROTOCOL HEADERS** section, select **+** to add a property.
10. Enter the following as a **REQUEST ENTITY (AS STRING)**
`type:spell AND confidence: [0 TO 0.5]`
**REQUEST ENTITY** specifies the threshold for low confidence spellings. Edit the upper range from 0.5 to increase or decrease the threshold based on your data.
### Delete all misspelling suggestions
To delete *all* of the misspelling suggestions, enter the following in the **REQUEST ENTITY** section:
`type:spell`
This entry may be helpful when tuning the misspelling detection job and testing different configuration parameters.
## Head-tail analysis job cleanup
The head-tail analysis job puts tail queries into one of multiple reason categories. For example, a tail query that includes a number might be assigned to the 'numbers' reason category. If the output in a particular category is not useful, you can remove it from the results. The examples in this section remove the numbers category.
### Prerequisites
The head-tail analysis job cleanup does *not* have to occur in a specific order.
### Remove head-tail analysis query suggestions
Use either a [Head-tail analysis cleanup method 1 - API call](#head-tail-analysis-cleanup-method-1-api-call) or the [Head-tail analysis cleanup method 2 - Fusion Admin UI](#head-tail-analysis-cleanup-method-2-fusion-admin-ui) to remove query category suggestions.
#### Head-tail analysis cleanup method 1 - API call
1. Open the `delete_lowConf_headTail.json` file.
```json theme={"dark"}
{
"type" : "rest-call",
"id" : "DC_Large_QR_HEAD_TAIL_CLEANUP",
"callParams" : {
"uri" : "solr://DC_Large_query_rewrite_staging/update",
"method" : "post",
"queryParams" : {
"wt" : "json"
},
"headers" : { },
"entity" : "reason_code_s:(\"number\" \"number spelling\" \"number rare-term\" \"question number other-specific\" \"number others\" \"number other-specific\" \"number other-extra\" \"product number other-specific\" \"product number other-extra\" \"product number spelling\" \"product number others\" \"product number rare-term\" \"product question number\" \"product number re-wording\" \"question number other-extra\" \"number re-wording\")"
},
"type" : "rest-call",
"type" : "rest-call"
}
```
2. Enter `` in the **uri** field. An example URI value for an app called `DC_Large` would be `DC_Large_query_rewrite_staging/update`.
3. Change the **id** field if applicable.
#### Head-tail analysis cleanup method 2 - Fusion Admin UI
1. Log in to Fusion and select **Collections > Jobs**.
2. Select **Add+ > Custom and Other Jobs > REST Call**.
3. Enter **remove-low-confidence-head-tail** in the **ID** field.
4. Enter `` in the **ENDPOINT URI** field. An example URI value for an app called `DC_Large` would be `DC_Large_query_rewrite_staging/update`.
5. Enter **POST** in the **CALL METHOD** field.
6. In the **QUERY PARAMETERS** section, select **+** to add a property.
7. Enter **wt** in the **Property Name** field.
8. Enter **json** in the **Property Value** field.
9. In the **REQUEST PROTOCOL HEADERS** section, select **+** to add a property.
10. Enter the following as a **REQUEST ENTITY (AS STRING)**
```
reason_code_s:("number" "number spelling" "number rare-term" "question number other-specific" "number others" "number other-specific" "number other-extra" "product number other-specific" "product number other-extra" "product number spelling" "product number others" "product number rare-term" "product question number" "product number re-wording" "question number other-extra" "number re-wording")
```
### Delete all head-tail suggestions
To delete *all* of the head-tail suggestions, enter the following in the **REQUEST ENTITY** section:
`type:tail`
This entry may be helpful when tuning the head-tail job and testing different configuration parameters.
4. Enter the underperforming query.
5. Enter one or more query improvements.
To facet the rules list by **Status** or **Published**, click one of the facet items within the facet categories.
To facet the rules by **Confidence**, click and drag either slider to set the range of confidence you would like to see:
### Create a rule
1. Click **+Add**.
2. In the new table row, complete the required fields, and additional fields as needed.
3. Click the checkmark to save, or close to cancel the changes.
### Edit a rule
1. Hold the pointer over the rule to edit.
2. Click the pencil icon.
3. Enter desired changes and click the checkmark to save, or close to cancel the changes.
### Approve/deny rules
{/* // New or altered rules will need to be approved by a user with the appropriate permissions before the rule can be enabled. */}
1. Hold the pointer over a rule.
2. Select the checkbox that displays for that rule.
3. To modify multiple rules, hold the pointer over each rule and select its corresponding checkbox.
4. Click **Approve** or **Deny** as needed.
{/*
This is not an explicit option in the rewrite screen. Is this implicit by approval/denial?
### Enable/disable a rule
1. Hold the pointer over a rule.
2. Select the checkbox that displays for that rule.
3. To modify multiple rules, hold the pointer over each rule and select its corresponding checkbox.
4. Click **Enable** or **Disable** as needed.
*/}
### Publish a rule
1. Hold the pointer over a rule.
2. Select the checkbox that displays for that rule.
3. To publish multiple rules, hold the pointer over each rule and select its corresponding checkbox.
4. Click **Publish Selected** to publish the selected rules.
#### Publish an individual rule
1. Hold the pointer over a rule.
2. Click the 
3. In the Customize Columns to Display screen, select the checkboxes for the columns you want to display. Clear the checkboxes for columns you do not want to display.
4. Click **Save**. To exit without saving your changes, click **Cancel**.