Misspelling Detection - Lucidworks documentation

The Misspelling Detection feature maps misspellings to their corrected spellings. When Fusion receives a query containing a known misspelling, it rewrites the query using the corrected spelling in order to return relevant results instead of an empty or irrelevant results set. Spelling corrections are applied in the Text Tagger stage of the query pipeline. The Token and Phrase Spell Correction job automatically creates spelling corrections based on your AI-generated data. When you navigate to Relevance > Rules > Rewrite and select the Misspelling tab, the application displays the Misspelling Detection screen. You can:

Use Misspelling Detection the output from the job
Use Misspelling Detection
Use Misspelling Detection from the _query_rewrite_staging collection

Use Misspelling Detection

LucidAcademyLucidworks offers free training to help you get started.The Course for Query Analytics focuses on how Fusion provides query analytics to detect and improve underperforming queries:

Visit the LucidAcademy to see the full training catalog.

Reviewing auto-generated spelling corrections

Spelling corrections that are automatically generated by the Token and Phrase Spell Correction job are assigned one of the following status values:

Auto There are three values for confidence level:
Value Confidence Label
0 low confidence Pending
0.5 median confidence Auto
1 high confidence Auto

No action is required on these results, but you can edit them if you wish.
Pending The confidence level is ambiguous, and the result must be reviewed by a user before it can be deployed. It will only be moved from the _query_rewrite_staging collection to the _query_rewrite collection when its status has changed to “Approved” and it has been published.

Value	Confidence	Label
0	low confidence	Pending
0.5	median confidence	Auto
1	high confidence	Auto

Navigate to Relevance > Rules > Rewrite.
Select Misspelling tab. The application displays the Misspelling Detection screen.
Notice the Status facet on the left. Click Pending to view only the items that need review.
Click the icon next to the spelling correction.
In the Status column, select either “Approved” or “Denied”. Optionally, you can also edit the spelling correction itself.

Adding new spelling corrections

You can manually add spelling corrections in addition to any generated by the Fusion 5.x.x Token and Phrase Spell Correction job release.

Navigate to Relevance > Rules > Rewrite.
Select Misspelling tab. The application displays the Misspelling Detection screen.
At the bottom of the rules list, click the icon. A new spelling correction appears at the top of the list:
Enter the misspelled word or phrase.
Enter one or more spelling corrections.
It is not necessary to set a confidence value.
Select the spelling correction’s status, depending on whether you want to deploy it the next time you publish your changes (“Approved”) or save it for further review (“Pending”).
Click the check mark to save the new spelling correction:

Publishing your changes

How to publish updated spelling corrections

In the Misspelling Detection screen, click the PUBLISH button. Fusion prompts you to confirm that you want to publish your changes.
Click PUBLISH.

You can un-publish a query rewrite by changing its status to “denied”, then clicking PUBLISH.

Tuning the misspelling detection job

The default configuration for the Token and Phrase Spell Correction job, respectively, is designed for high accuracy and works well with most signal datasets, depending on the volume and quality of the signals. If you are seeing too few results, or too many inaccurate results, then you can try tuning the job to achieve better results.To modify job configurations, you must be a Fusion user with one of the following roles or permissions that include access to job configurations:

Query rewrite jobs post-processing cleanup

To perform more extensive cleanup of query rewrites, complete the procedures in Query Rewrite Jobs Post-processing Cleanup.

Use Misspelling Detection

Visit the LucidAcademy to see the full training catalog.

Reviewing auto-generated spelling corrections

Spelling corrections that are automatically generated by the Token and Phrase Spell Correction job are assigned one of the following status values:

Auto There are three values for confidence level:
Value Confidence Label
0 low confidence Pending
0.5 median confidence Auto
1 high confidence Auto

No action is required on these results, but you can edit them if you wish.
Pending The confidence level is ambiguous, and the result must be reviewed by a user before it can be deployed. It will only be moved from the _query_rewrite_staging collection to the _query_rewrite collection when its status has changed to “Approved” and it has been published.

Value	Confidence	Label
0	low confidence	Pending
0.5	median confidence	Auto
1	high confidence	Auto

Navigate to Relevance > Rules > Rewrite.
Select Misspelling tab. The application displays the Misspelling Detection screen.
Notice the Status facet on the left. Click Pending to view only the items that need review.
Click the icon next to the spelling correction.
In the Status column, select either “Approved” or “Denied”. Optionally, you can also edit the spelling correction itself.

Adding new spelling corrections

You can manually add spelling corrections in addition to any generated by the Fusion 5.x.x Token and Phrase Spell Correction job release.

Navigate to Relevance > Rules > Rewrite.
Select Misspelling tab. The application displays the Misspelling Detection screen.
At the bottom of the rules list, click the icon. A new spelling correction appears at the top of the list:
Enter the misspelled word or phrase.
Enter one or more spelling corrections.
It is not necessary to set a confidence value.
Select the spelling correction’s status, depending on whether you want to deploy it the next time you publish your changes (“Approved”) or save it for further review (“Pending”).
Click the check mark to save the new spelling correction:

Publishing your changes

How to publish updated spelling corrections

In the Misspelling Detection screen, click the PUBLISH button. Fusion prompts you to confirm that you want to publish your changes.
Click PUBLISH.

You can un-publish a query rewrite by changing its status to “denied”, then clicking PUBLISH.

Tuning the misspelling detection job

Query rewrite jobs post-processing cleanup

To perform more extensive cleanup of query rewrites, complete the procedures in Query Rewrite Jobs Post-processing Cleanup.

Use Misspelling Detection

Visit the LucidAcademy to see the full training catalog.

Reviewing auto-generated spelling corrections

Spelling corrections that are automatically generated by the Token and Phrase Spell Correction job are assigned one of the following status values:

Auto There are three values for confidence level:
Value Confidence Label
0 low confidence Pending
0.5 median confidence Auto
1 high confidence Auto

No action is required on these results, but you can edit them if you wish.
Pending The confidence level is ambiguous, and the result must be reviewed by a user before it can be deployed. It will only be moved from the _query_rewrite_staging collection to the _query_rewrite collection when its status has changed to “Approved” and it has been published.

Value	Confidence	Label
0	low confidence	Pending
0.5	median confidence	Auto
1	high confidence	Auto

Navigate to Relevance > Rules > Rewrite.
Select Misspelling tab. The application displays the Misspelling Detection screen.
Notice the Status facet on the left. Click Pending to view only the items that need review.
Click the icon next to the spelling correction.
In the Status column, select either “Approved” or “Denied”. Optionally, you can also edit the spelling correction itself.

Adding new spelling corrections

You can manually add spelling corrections in addition to any generated by the Fusion 5.x.x Token and Phrase Spell Correction job release.

Navigate to Relevance > Rules > Rewrite.
Select Misspelling tab. The application displays the Misspelling Detection screen.
At the bottom of the rules list, click the icon. A new spelling correction appears at the top of the list:
Enter the misspelled word or phrase.
Enter one or more spelling corrections.
It is not necessary to set a confidence value.
Select the spelling correction’s status, depending on whether you want to deploy it the next time you publish your changes (“Approved”) or save it for further review (“Pending”).
Click the check mark to save the new spelling correction:

Publishing your changes

How to publish updated spelling corrections

In the Misspelling Detection screen, click the PUBLISH button. Fusion prompts you to confirm that you want to publish your changes.
Click PUBLISH.

You can un-publish a query rewrite by changing its status to “denied”, then clicking PUBLISH.

Tuning the misspelling detection job

Query rewrite jobs post-processing cleanup

To perform more extensive cleanup of query rewrites, complete the procedures in Query Rewrite Jobs Post-processing Cleanup.

When you manually add new spelling corrections, subsequent job runs use those documents as input for machine learning to improve the job’s output. Unlike job-generated documents, manually-added query rewriting documents are never overwritten by new job output.

Misspelled terms are completely replaced by their corrected terms. If you want to expand the query to include all alternative terms, set the synonyms to bi-directional. See Synonym Detection for more information.

To learn how to use misspelling detection in the Rules Editor, see Use Query Rewrites in the Rules Editor.

Use Query Rewrites in the Rules Editor

Visit the LucidAcademy to see the full training catalog.

Introduction to Fusion

Getting Data In

Getting Data Out

Operations

Reference

Developer Docs

Neural Hybrid Search

Release Notes