Skip to main content
Fusion 5.9.12 and later releases use index and query stages to split large documents into smaller, more manageable segments called chunks. For more information about chunking, chunking strategies and setting up chunking, see Chunking. The Chunking Neural Hybrid Query stage performs hybrid lexical-semantic search that combines BM25-type lexical search with KNN dense vector search via Solr. This stage differs from the Neural Hybrid Stage because it supports chunking. Not sure which hybrid query stage is right for you? Read about the differences between the hybrid query stages.
This feature is available in Fusion 5.9.12 and later.Some prefiltering capabilities, such as access to the preFilterKey context property and the VectorPreFilter helper class are available in 5.9.13 and later.
Click Get Started below to see how to enable chunking in Fusion:
Click your use case below to see examples of how the Chunking Neural Hybrid Query Stage processes documents:
In a single document, a manufacturing company indexes its parts catalog, where each record contains product specifications, compatibility notes, reorder thresholds, and supplier lead times.A procurement manager queries: “which hydraulic fittings are compatible with the Model 7 assembly line?”Without chunking, the full parts record competes as one unit and the compatibility section is diluted by pricing and supplier content that has no relevance to the query.With chunking, Fusion isolates the compatibility paragraph as its own chunk and scores it independently, returning the most relevant part records at the top even when the rest of the document covers unrelated logistics details.This pattern is common in B2B catalog search where product records are dense and structured around operational concerns. A single SKU might describe assembly specs, regulatory certifications, hazmat handling, and pricing in the same document.Chunking lets users find the specific detail they need without requiring the entire record to be relevant.

​
About the Lexical Query Squash Factor

The Lexical Query Squash Factor field lets you input a value that squashes the lexical query scores from 0..inf to 0..1. This setting helps prevent the lexical query from dominating the final score, and normalizes the score into a range that works well with vector similarity scores. Additionally, it helps prevent the vanishing gradient problem, which occurs when very high lexical scores are mapped to values extremely close to 1, such as 0.99999999. During the hybrid search calculation, these near-1 values can cause the system to lose sensitivity to subtle differences in lexical relevance, effectively ‘squashing’ the gradient and reducing the impact of lexical scoring. Lucidworks recommends setting the Lexical Query Squash Factor to the inverse of the maximum lexical score observed across your queries. This helps balance the impact of lexical and vector scores, leading to more accurate and nuanced search results.

​
Prefiltering

Prefiltering is a technique that can improve performance and accuracy by filtering documents before applying the algorithm, reducing the number of documents that need to be processed. This is especially effective with the KNN algorithm. Prefiltering is disabled by default. To enable it, uncheck Block pre-filtering in this stage. When prefiltering is enabled, you can configure the filters using one or both of these methods:
  • Security filters
    You can use security filters as prefilters by placing the Graph Security Trimming Stage after this one in the pipeline.
    Then Fusion uses the security trimming filter as a prefilter.
  • JavaScript
    When prefiltering is enabled, this stage adds a preFilterKey object to the Javascript ctx object.
    You can place a Javascript stage after this one and use it to access the preFilterKey object, as in this example:
    You can also use the following example in Fusion 5.9.13 and later for placing pre-filter specific filters:
    The context object of the VectorPreFilter class is not present in Fusion 5.9.12 and earlier. Use the Additional Query Parameters stage example and the 5.9.12 note below.
    In Fusion 5.9.12 and earlier, a non-JavaScript approach is required in addition to the Additional Query Parameters stage example in the next section. In Fusion 5.9.12 the parameter vec_sim_q after the Chunking Neural Hybrid Query stage needs to be altered to include {!knn f=$vec_field v=$vec_q topK=100 preFilter=$vectorPreFilter} where topK is consistent with what your stage sets.
  • Additional Query Parameters stage If you do not want to create a JavaScript stage, you can create additional query parameters to prefilter the documents to be processed by using what the previous JavaScript example adds to the request. This step is required for Fusion 5.9.12. The following example uses a single prefilter:
    The following example uses multiple prefilters:

​
Query pipeline stage condition examples

Stages can be triggered conditionally when a script in the Condition field evaluates to true. Some examples are shown below. Run this stage only for mobile clients:
Run this stage when debugging is enabled:
Run this stage when the query includes a specific term:
Run this stage when multiple conditions are met:
The first condition checks that the request parameter “fusion-user-name” is present and has the value “SuperUser”. The second condition checks that the request parameter “isFusionPluginQuery” is not present.

​
Configuration

When entering configuration values in the UI, use unescaped characters, such as \t for the tab character. When entering configuration values in the API, use escaped characters, such as \\t for the tab character.