Query Pipelines

A Query Pipeline transforms a set of inputs into a Solr query request and it can execute requests and manipulate the Solr response as well, via a set of modularized operations called Query Stages. The objects sent from stage to stage are Request objects and Response objects.

Fusion stores pipeline names and definitions, allowing a pipeline to be reused across applications. Pipeline definitions can be modified, so that as an application evolves, the pipelines used by that application can evolve accordingly. During application development, the Fusion UI can be used to develop and debug a Query Pipeline.

The available stage types allow setting specific parameters for the query, such as the number of results to return or the query parser to use. You can also define facets and recommendations to be returned with the results. If Access Control Lists (ACLs) are in use, you can apply a security-trimming stage to apply user access restrictions to the results.

For details about the available REST APIs, see Query Pipelines API and Query Stages API.

Default Query Pipelines

When you create a new app with a default collection, the collection includes a default query pipeline. When you create a new collection in an existing app, Fusion also creates a default query pipeline for the new collection. The pipeline name is the same as the collection name.

The default query pipeline has the following pre-configured stages:

This stage queries the _signals_aggr collection (typically generated by the _click_signals_aggregation Spark job), using the query parameters configured in the stage, and returns document recommendations with boost weights.

  • Query Fields

    Unresolved directive in query-pipelines.asciidoc - include::/fusion-server/reference-guides/query-pipeline-stages/query-pipeline-stages/search-fields-query-stage.asciidoc[tag=intro]

  • Field Facet

    The Field Facet query pipeline stage is used to add a Solr Field Facet query to the search query pipeline.

  • Apply Rules

  • A Solr Query

    The Solr Query stage transforms the Fusion query pipeline Request object into a Solr query and sends it to Solr.

  • Modify Response with Rules

    Most rules operate on the request, but some rule types, such as banner rules or redirect rules, do their work when the response comes back. The Modify Response with Rules stage applies those rules to the response. For example, a banner rule can add a banner URL to the response before returning it to the client.

Custom Query Pipelines

Using the Query Workbench or the REST API, you can develop custom pipelines to suit any search application. Start with any of Fusion’s built-in query pipelines, then add, remove, and re-order the pipeline stages as needed to produce the appropriate query results.

Asynchronous query pipeline processing

Query pipeline processing performance can be improved by enabling asynchronous processing for certain stages that make requests to secondary collections, external databases, and so on. The following stages support asynchronous processing:

This feature uses the fork-and-join model, where any of the stages above can create a fork on the pipeline. The parallel processes are joined again using the Merge Async Results stage at a later point in the pipeline.

How to enable asynchronous query pipeline processing
  1. In the Query Workbench, open the query pipeline you want to modify.

  2. Click Add a Stage and select a stage that supports asynchronous processing to your pipeline, or select the stage in your pipeline if it has already been added.

    Stages that support this are listed above.

  3. In the stage configuration panel, select Asynchronous Execution Config.

  4. Select Enable Async Execution.

    Enabling asynchronous processing

    Tip
    Fusion automatically assigns an Async ID value to this stage. Change this to a more memorable string that describes the asynchronous stages you are merging, such as signals or access_control.
  5. Copy the Async ID value.

  6. Click Apply.

    Verify that the stage is in the correct position in your pipeline.

  7. Click Add a Stage and select the Merge Async Results stage.

    Add the Merge Async Results stage

  8. In the stage configuration panel, click the Add add icon button next to Async IDs.

  9. Paste the ID from the asynchronous query stage into the new ID field.

    Tip
    When using multiple asynchronous stages, the order in which IDs are entered here is the order in which they are joined back into the pipeline.
  10. Click Apply.

    The Merge Async Results stage is now the first stage in the pipeline.

  11. Drag the Merge Async Results stage down so that it appears immediately before the Solr Query stage:

    Move the Merge Async Results

  12. Click Save.