> ## Documentation Index > Fetch the complete documentation index at: https://doc.lucidworks.com/llms.txt > Use this file to discover all available pages before exploring further. # Fusion stages for AI > Lucidworks AI export const LwTemplate = ({title = "Key questions to get you started", icon = "sparkles", cta = "Powered by Agent Studio", linkHref = "https://lucidworks.com/demo/?utm_source=docs&utm_medium=referral&utm_campaign=docs_cta_ai"}) => { const [isLoaded, setIsLoaded] = useState(false); useEffect(() => { const timer = setTimeout(() => { setIsLoaded(true); }, 500); return () => clearTimeout(timer); }, []); return
{isLoaded && ` }} />} Powered by Lucidworks Agent Studio
; }; [localhost link]: http://localhost:3000/docs/lw-platform/lw-ai/lw-ai-stages/overview [mintlify link]: https://doc.lucidworks.com/docs/lw-platform/lw-ai/lw-ai-stages/overview [old doc.lw link]: https://doc.lucidworks.com/lw-platform/ai/b5uv9z Lucidworks Search and self-hosted Fusion come with built-in Lucidworks AI stages for index pipelines and query pipelines. You can use these stages with the [pre-trained embedding models](/docs/lw-platform/lw-ai/lw-ai-pre-trained-embedding-models) provided by Lucidworks or with [custom embedding models](/docs/lw-platform/lw-ai/lw-ai-custom-embedding-model-training/overview). These topics provide instructions and configuration reference information for using each of the Lucidworks AI stages. For additional configuration details, see the [Lucidworks AI APIs](/docs/lw-platform/lw-ai/lw-ai-apis/overview). See also [Choosing, Implementing, and Troubleshooting NHS Stages](/docs/5/fusion/hybrid-search/hybrid-stage-differences) to learn about the differences between Neural Hybrid Search stages. Make sure you have configured a [Lucidworks AI Gateway](/docs/lw-platform/lw-ai/lw-ai-gateway) integration before you begin. Lucidworks AI Gateway provides a secure, authenticated connection between [Fusion](/docs/5/fusion/overview) and your hosted models. ## Run stages asynchronously Some stages allow for asynchronous processing for work that may take a while, keeping the overall pipeline moving. Fusion does not block the request thread while the stage is waiting. Instead, it waits for the stage’s result right before it needs it to finish the response. ### When to use asynchronous processing Use asynchronous processing for the following: * A stage calls out to a remote system. * The stage reads or writes large files or blobs. * The stage can run in parallel with other stages. Use synchronous processing for the following: * The stage does only local work and typically finishes in under a couple of milliseconds. * The stage coordinates tasks that must happen one at a time. ### Benefits of asynchronous processing Asynchronous processing allows: * Lower latency by overlapping slower calls with other work. * Greater resilience because if one asynchronous call is slow or errors out, Fusion can time it out without stalling the whole pipeline. * Easier error recognition and resolution because timeouts and cancellations are scoped to the stage. Before you can use Lucidworks AI with Lucidworks Platform, you must configure the [Lucidworks AI Gateway](/docs/lw-platform/lw-ai/lw-ai-gateway) to provide a secure, authenticated integration between self-hosted Fusion and your hosted models. This configuration is done through a secret properties file that you can find in the Lucidworks Platform UI. This feature is available starting in Fusion 5.9.5 and in all subsequent Fusion 5.9 releases. Integrations are created for you by the Lucidworks team. But as a workspace owner, you can configure those integrations with Lucidworks AI Gateway. Each account can have its own set of credentials and associated scopes, which define the operations it can perform. If configuration properties are not provided at the account level, default settings are used instead. To configure the Lucidworks AI Gateway, navigate to the megamenu and click **Models**. 1. On the Integrations tab, click your integration. If you don’t see your integration, contact your Lucidworks representative. 2. Download or copy the YAML code and paste it into a file called `account.yaml`. The file for a single integration should look similar to this one: ```yaml theme={"dark"} lwai-gateway: lwai: credentials: | fusion.lwai.default.baseUrl: https://APPLICATION_ID.applications.lucidworks.com fusion.lwai.default.authEndpoint: https://identity.lucidworks.com/oauth2/XXXXXXXXXX/v1/token fusion.lwai.account[0].name: ACCOUNT_NAME fusion.lwai.account[0].scopes: machinelearning.predict,machinelearning.model fusion.lwai.account[0].clientId: ***** fusion.lwai.account[0].clientSecret: ***** ``` For a configuration with multiple integrations, it should look like this: ```yaml theme={"dark"} lwai-gateway: lwai: credentials: | fusion.lwai.default.authEndpoint: https://identity.lucidworks.com/oauth2/XXXXXXXXXX/v1/token fusion.lwai.account[0].baseUrl: https://APPLICATION_ID.applications.lucidworks.com fusion.lwai.account[0].name: ACCOUNT_NAME fusion.lwai.account[0].scopes: machinelearning.predict,machinelearning.model fusion.lwai.account[0].clientId: ***** fusion.lwai.account[0].clientSecret: ***** fusion.lwai.account[1].baseUrl: https://APPLICATION_ID2.applications.lucidworks.com fusion.lwai.account[1].name: ACCOUNT_NAME fusion.lwai.account[1].scopes: machinelearning.predict,machinelearning.model fusion.lwai.account[1].clientId: ***** fusion.lwai.account[1].clientSecret: ***** ``` Non-admin users must have the following permissions to use Lucidworks AI integrations: `PUT,POST,GET:/LWAI-ACCOUNT-NAME/**` where `LWAI-ACCOUNT-NAME` must match the value of `fusion.lwai.account[n].name` in the integration YAML. 3. Apply the file to your Fusion configuration file. For example: ```bash theme={"dark"} helm upgrade KUBERNETES_NAMESPACE lucidworks/fusion -f FUSION_VALUES.yaml ``` These Fusion pipeline stages provide interfaces for configuring your Lucidworks AI models in Fusion pipelines. There are differences for Lucidworks Search versus self-hosted Fusion clients. ## Lucidworks Search These Lucidworks Search pipeline stages provide interfaces for configuring your Lucidworks AI models in Lucidworks Search pipelines. * **Vector search stages**. These are the foundational stages that enable vector search features. * [LWAI Chunker Index stage](/docs/lucidworks-search/09-developer-documentation/config-specs/index-pipeline-stages/lwai-chunker-stage) * [LWAI Vectorize Field Index stage](/docs/lucidworks-search/09-developer-documentation/config-specs/index-pipeline-stages/vectorize-field-via-lucidworks-ai-index-stage) * [LWAI Batch Vectorize Index stage](/docs/lucidworks-search/09-developer-documentation/config-specs/index-pipeline-stages/lwai-batch-vectorize) * [LWAI Vectorize Query stage](/docs/lucidworks-search/09-developer-documentation/config-specs/query-pipeline-stages/vectorize-query-via-lucidworks-ai-query-stage) * **Chunker stage**. This stage asynchronously breaks down large text documents and vectorizes the chunks. * [LWAI Chunker Index stage](/docs/lucidworks-search/09-developer-documentation/config-specs/index-pipeline-stages/lwai-chunker-stage) * **RAG Bridge stage**. This stage extracts relevant snippets from source documents used in RAG responses. * [Chunking RAG bridge query stage](/docs/lucidworks-search/09-developer-documentation/config-specs/query-pipeline-stages/chunking-rag-bridge) * **Generative AI (Gen-AI) stages**. See [Generative AI](/docs/lw-platform/lw-ai/lw-ai-generative-ai) for conceptual information about Gen-AI features. * [LWAI Prediction Index stage](/docs/lucidworks-search/09-developer-documentation/config-specs/index-pipeline-stages/lucidworks-ai-prediction-index-stage) * [LWAI Prediction Query stage](/docs/lucidworks-search/09-developer-documentation/config-specs/query-pipeline-stages/lucidworks-ai-prediction-query-stage) The LWAI Prediction index stage is a Fusion index pipeline stage that enriches your index with [Generative AI](/docs/lw-platform/lw-ai/lw-ai-generative-ai) predictions. It defaults to asynchronous processing, which does not block the pipeline while waiting for a response from Lucidworks AI. For reference information, see [LWAI Prediction index stage](/docs/lucidworks-search/09-developer-documentation/config-specs/index-pipeline-stages/lucidworks-ai-prediction-index-stage). To use this stage, non-admin Fusion users must be granted the `PUT,POST,GET:/LWAI-ACCOUNT-NAME/**` permission in Fusion, which is the Lucidworks AI API Account Name defined in Lucidworks AI Gateway when this stage is configured. To configure this stage: 1. Sign in to Fusion and click **Indexing > Index Pipelines**. 2. Click **Add+** to add a new pipeline. 3. Enter the name in **Pipeline ID**. 4. Click **Add a new pipeline stage**. 5. In the AI section, click **LWAI Prediction**. 6. In the **Label** field, enter a unique identifier for this stage. 7. In the **Condition** field, enter a script that results in true or false, which determines if the stage should process. 8. In the **Account Name** field, select the Lucidworks AI API account name defined in [Lucidworks AI Gateway](/docs/lw-platform/lw-ai/lw-ai-gateway). 9. In the **Use Case** field, select the Lucidworks AI use case to associate with this stage. * To generate a list of the use cases for your organization, see [Use Case API](/docs/lw-platform/lw-ai/lw-ai-apis/lw-ai-use-case-api). * If the **Call Asynchronously?** check box is selected, see available use cases described in [Async Prediction API](/docs/lw-platform/lw-ai/lw-ai-apis/lw-ai-async-prediction-api/overview). * If the **Call Asynchronously?** check box is *not* selected, see available use cases described in [Prediction API](/docs/lw-platform/lw-ai/lw-ai-apis/lw-ai-prediction-api/overview). 10. In the **Model** field, select the Lucidworks AI model to associate with this stage. Your Fusion account name must match the name of the account that you selected in the **Account Name** dropdown.\ For more information about models, see: * [Pre-trained embedding models](/docs/lw-platform/lw-ai/lw-ai-pre-trained-embedding-models) * [Custom embedding model training](/docs/lw-platform/lw-ai/lw-ai-custom-embedding-model-training/overview) * [Generative AI models](/docs/lw-platform/lw-ai/lw-ai-generative-ai) 11. In the **Input context variable** field, enter the name of the context variable to be used as input. Template expressions are supported. 12. In the **Destination field name and context output** field, enter the name that will be used as both the field name in the document where the prediction is written and the context variable that contains the prediction. {/* // tag::lwai-prediction-index-stage[] */} * If the **Call Asynchronously?** check box is selected and a value is entered in this field: * `{destination name}_t` is the full response. * In the document: * `_lw_ai_properties_ss` contains the Lucidworks account, boolean setting for async, use case, input for the call, and the collection. * `_lw_ai_request_count` is the number of GET requests by `predictionId` and `_lw_ai_success_count` is the number of responses without errors. These two fields are used for debugging only. Based on the deployment, the most useful measure is the ratio of `_lw_ai_success_count` divided by `_lw_ai_request_count` and then adjusting as much as possible to achieve 1.0. * `enriched_ss` contains the use case. This can be used as a boolean value to verify if the use case indexed successfully. * If the **Call Asynchronously?** check box is *not* selected and a value is entered in this field: * `{destination name}_t` is the full response. * If *no* value is entered in this field (regardless of the **Call Asynchronously?** check box setting): * `_lw_ai_{use case}_t` is the `response.response` object, which is the raw model output. * `_lw_ai_{use case}_response_s` is the full response. {/* // end::lwai-prediction-index-stage[] */} 13. In the **Use Case Configuration** section, click the **+** sign to enter the parameter name and value to send to Lucidworks AI. The `useCaseConfig` parameter is only applicable to certain use cases. * If the **Call Asynchronously?** check box is selected, `useCaseConfig` information for each applicable use case is described in [Async Prediction API](/docs/lw-platform/lw-ai/lw-ai-apis/lw-ai-async-prediction-api/overview). * If the **Call Asynchronously?** check box is *not* selected, `useCaseConfig` information for each applicable use case is described in [Prediction API](/docs/lw-platform/lw-ai/lw-ai-apis/lw-ai-prediction-api/overview). 14. In the **Model Configuration** section, click the **+** sign to enter the parameter name and value to send to Lucidworks AI. Several `modelConfig` parameters are common to generative AI use cases. * If the **Call Asynchronously?** check box is selected, `modelConfig` information is described in [Async Prediction API](/docs/lw-platform/lw-ai/lw-ai-apis/lw-ai-async-prediction-api/overview). * If the **Call Asynchronously?** check box is *not* selected, `modelConfig` information is described in [Prediction API](/docs/lw-platform/lw-ai/lw-ai-apis/lw-ai-prediction-api/overview). 15. In the **API Key** field, enter the secret value specified in the external model. For: * OpenAI models, `"apiKey"` is the value in the model’s `"[OPENAI_API_KEY]"` field. For more information, see [Authentication API keys](https://platform.openai.com/docs/api-reference/authentication). * Azure OpenAI models, `"apiKey"` is the value generated by Azure in either the model’s `"[KEY1 or KEY2]"` field. For requirements to use Azure models, see [Generative AI models](/docs/lw-platform/lw-ai/lw-ai-generative-ai). * Google VertexAI models, `"apiKey"` is the value in the model’s `"[BASE64_ENCODED_GOOGLE_SERVICE_ACCOUNT_KEY]"` field. For more information, see [Create and delete Google service account keys](https://cloud.google.com/iam/docs/keys-create-delete). 16. To run the API call asynchronously, select the **Call Asynchronously?** check box to specify the stage is to use the [Lucidworks AI Async Prediction API](/docs/lw-platform/lw-ai/lw-ai-apis/lw-ai-async-prediction-api/overview) endpoints. If this is selected, the API call does not block the pipeline while waiting for a response from Lucidworks AI. If the check box is *not* selected, the API call uses the [Prediction API](/docs/lw-platform/lw-ai/lw-ai-apis/lw-ai-prediction-api/overview), which uses the pipeline until a response is received from Lucidworks AI. Performance of other API calls can be impacted. 17. In the **Maximum Asynchronous Call Tries** field, enter the maximum number of times to send an asynchronous API call before the system generates a failure error. 18. Select the **Fail on Error** checkbox to generate an exception if an error occurs while generating a prediction for a document. 19. Click **Save**. ## Additional requirements Additional requirements to use async calls include: * Use a V2 connector. Only V2 connectors work for this task and not other options, such as PBL or V1 connectors. * Remove the `Apache Tika` stage from your parser because it can cause datasource failures with the following error: "The following components failed: \[class com.lucidworks.connectors.service.components.job.processor.DefaultDataProcessor : Only Tika Container parser can support Async Parsing.]" * Replace the `Solr Indexer` stage with the `Solr Partial Update Indexer` stage with the following settings: * `Enable Concurrency Control` set to off * `Reject Update if Solr Document is not Present` set to off * `Process All Pipeline Doc Fields` set to on * `Allow reserved fields` set to on * A parameter with `Update Type`, `Field Name` & `Value` in `Updates` The LWAI Prediction AI query stage is a Fusion pipeline query stage that enriches your search results with [Generative AI](/docs/lw-platform/lw-ai/lw-ai-generative-ai) predictions. For reference information, see [LWAI Prediction query stage](/docs/lucidworks-search/09-developer-documentation/config-specs/query-pipeline-stages/lucidworks-ai-prediction-query-stage). To use this stage, non-admin Fusion users must be granted the `PUT,POST,GET:/LWAI-ACCOUNT-NAME/**` permission in Fusion, which is the Lucidworks AI API Account Name defined in Lucidworks AI Gateway when this stage is configured. To configure this stage: 1. Sign in to Fusion and click **Querying > Query Pipelines**. 2. Click **Add+** to add a new pipeline. 3. Enter the name in **Pipeline ID**. 4. Click **Add a new pipeline stage**. 5. In the AI section, click **LWAI Prediction**. 6. In the **Label** field, enter a unique identifier for this stage. 7. In the **Condition** field, enter a script that results in true or false, which determines if the stage should process. 8. Select **Asynchronous Execution Config** if you want to run this stage asynchronously. If this field is enabled, complete the following fields: 1. Select **Enable Async Execution**. Fusion automatically assigns an **Async ID** value to this stage. Change this to a more memorable string that describes the asynchronous stages you are merging, such as `signals` or `access_control`. 2. Copy the **Async ID** value. 9. In the **Account Name** field, enter your Lucidworks AI API Account Name as defined in the Lucidworks AI Gateway Service. 10. In the **Use Case** field, select the Lucidworks AI use case to associate with this stage. * To generate a list of the use cases for your organization, see [Use Case API](/docs/lw-platform/lw-ai/lw-ai-apis/lw-ai-use-case-api). * The available use cases are described in [Prediction API](/docs/lw-platform/lw-ai/lw-ai-apis/lw-ai-prediction-api/overview). 11. In the **Use Case Configuration** section, click the **+** sign to enter the parameter name and value to send to Lucidworks AI. * The `useCaseConfig` parameter is only applicable to certain use cases. For more information, see the [Async Prediction API](/docs/lw-platform/lw-ai/lw-ai-apis/lw-ai-async-prediction-api/overview) and the [Prediction API](/docs/lw-platform/lw-ai/lw-ai-apis/lw-ai-prediction-api/overview). * The `memoryUuid` parameter is required in the Standalone Query Rewriter use case, and is optional in the RAG use case. For more information, see [Prediction API](/docs/lw-platform/lw-ai/lw-ai-apis/lw-ai-prediction-api/overview). 12. In the **Model** field, select the Lucidworks AI model to associate with this stage.\ If you do not see any model names and you are a non-admin Fusion user, verify with a Fusion administrator that your user account has these permissions: `PUT,POST,GET:/LWAI-ACCOUNT-NAME/**` Your Fusion account name must match the name of the account that you selected in the **Account Name** dropdown.\ For more information about models, see: * [Pre-trained embedding models](/docs/lw-platform/lw-ai/lw-ai-pre-trained-embedding-models) * [Custom embedding model training](/docs/lw-platform/lw-ai/lw-ai-custom-embedding-model-training/overview) * [Generative AI models](/docs/lw-platform/lw-ai/lw-ai-generative-ai) 13. In the **Model Configuration** section, click the **+** sign to enter the parameter name and value to send to Lucidworks AI. Several `modelConfig` parameters are common to generative AI use cases. For more information, see [Prediction API](/docs/lw-platform/lw-ai/lw-ai-apis/lw-ai-prediction-api/overview). 14. In the **Input context variable** field, enter the name of the context variable to be used as input. Template expressions are supported. 15. In the **Destination variable name and context output** field, enter the name that will be used as both the query response header in the prediction results and the context variable that contains the prediction. {/* // tag::lwai-prediction-query-stage[] */} * If a value is entered in this field: * `{destination name}_t` is the full response. * In the context: * `_lw_ai_properties_ss` contains the Lucidworks account, boolean setting for async, use case, input for the call, and the collection. * If *no* value is entered in this field: * `_lw_ai_{use case}_t` is the `response.response` object, which is the raw model output. * `_lw_ai_{use case}_response_s` is the full response. {/* // end::lwai-prediction-query-stage[] */} 16. **Grounding Options** is only used for the RAG use case, and connects model output to the data source. This provides more trustworthy responses in a scalable, cost-efficient manner. If selected, enter appropriate values in the following options: 1. In the **Grounding Documents Location** field, enter the location where response documents are stored for use cases that support grounding via attached documents. 2. In the **Grounding Documents Key** field, enter the key in the context variable that contains the grounding documents. If the value of the **Grounding Document Location** field is `SolrResponse`, the value in this field is ignored and the response documents are used. 3. In the **Number of Grounding Documents** field, enter the number of documents to include in the RAG request. 17. In the **Document Field Mappings** section, enter the **LW AI Document field name** and its corresponding **Response document field name** to map from input documents to the fields accepted by the Prediction API RAG use case. The fields are described in the [Prediction API](/docs/lw-platform/lw-ai/lw-ai-apis/lw-ai-prediction-api/overview). If information is not entered in this section, the default mappings are used. * The `body` and `source` fields are required. * `body` - `description_t` is the document content. * `source` - `link_t` is the URL/ID of the document. * The `title` and `date` fields are optional. * `title` - `title_t` is the title of the document. * `date` - `_lw_file_modified_tdt` is the creation date of the document in epoch time format. 18. Select the **Fail on Error** checkbox to generate an exception if an error occurs during this stage. 19. In the **API Key** field, enter the secret value specified in the external model. For: * OpenAI models, `"apiKey"` is the value in the model’s `"[OPENAI_API_KEY]"` field. For more information, see [Authentication API keys](https://platform.openai.com/docs/api-reference/authentication). * Azure OpenAI models, `"apiKey"` is the value generated by Azure in either the model’s `"[KEY1 or KEY2]"` field. For requirements to use Azure models, see [Generative AI models](/docs/lw-platform/lw-ai/lw-ai-generative-ai). * Google VertexAI models, `"apiKey"` is the value in the model’s `"[BASE64_ENCODED_GOOGLE_SERVICE_ACCOUNT_KEY]"` field. For more information, see [Create and delete Google service account keys](https://cloud.google.com/iam/docs/keys-create-delete). 20. Click **Save**. Solr Vector Search supports semantic search with lexical filtering. Lucidworks recommends that if you plan to use Solr SVS, you first set up Lucidworks AI to vectorize text. The LWAI Vectorize pipeline is a default pipeline that contains the required index stages to set up vector search using [Lucidworks AI](/docs/lw-platform/lw-ai/overview). This feature is currently only available to clients who have contracted with Lucidworks for features related to Neural Hybrid Search and Lucidworks AI. This feature is only available in Lucidworks Search 5.9.x for versions 5.9.6+. This pipeline uses the following stages: * [Field Mapping](#field-mapping) * [Solr Dynamic Field Name Mapping](#solr-dynamic-field-name-mapping) * [LWAI Vectorize Field](#lwai-vectorize-field) * [Solr Indexer](#solr-indexer) ## Configure the pipeline To add the Lucidworks AI (LWAI) Vectorize index pipeline: 1. Sign in to Lucidworks Search and click **Indexing > Index Pipelines**. 2. Select the default **LWAI-vectorize** pipeline. 3. Configure the following stages included in the default pipeline. ## Field Mapping The [Field Mapping](/docs/lucidworks-search/09-developer-documentation/config-specs/index-pipeline-stages/field-mapping) stage customizes mapping of the fields in an index pipeline document to fields in the Solr scheme. To configure this stage for the index pipeline: 1. In the **Label** field, enter a unique identifier for this stage or leave blank to use the default value. 2. In the **Condition** field, enter a script that results in true or false, which determines if the stage should process, or leave blank. 3. Select the **Allow System Fields Mapping?** checkbox to map system fields in this stage. 4. In the **Field Retention** section, enter specific fields to either keep or delete. 5. In the **Field Value Updates** section, enter specific fields and then designate the value to either add to the field, or set on the field. When a value is added, any values previously on the field are retained. When a value is set, any values previously on the field are overwritten by the new value entered. 6. In the **Field Translations** section, enter specific fields to either move or copy to a different field. When a field is moved, the values from the source field are moved over to the target field and the source field is removed. When a field is copied, the values from the source field are copied over to the target field and the source field is retained. 7. Select the **Unmapped Fields** checkbox to specify the operation on the fields *not* mapped in the previous sections. Select the **Keep** checkbox to keep all unmapped fields. This is the only option you need to select for the LWAI-vectorize stage. 8. Click **Save**. ## Solr Dynamic Field Name Mapping The [Solr Dynamic Field Name Mapping](/docs/lucidworks-search/09-developer-documentation/config-specs/index-pipeline-stages/solr-dynamic-field-name-mapping) stage maps pipeline document fields to Solr dynamic fields. 1. In the **Label** field, enter a unique identifier for this stage or leave blank to use the default value. 2. In the **Condition** field, enter a script that results in true or false, which determines if the stage should process, or leave blank. 3. Select the **Duplicate Single-Valued Fields as Multi-Valued Fields** checkbox to enable indexing of field data into both single-valued and multi-valued Solr fields. For example, if this option is selected, the `phone` field is indexed into both the `phone_s` single-valued field and the `phone_ss` multi-valued field. If this option is not selected, the `phone` field is indexed into only the `phone_s` single-valued field. 4. In the **Field Not To Map** section, enter the names of the fields that should not be mapped by this stage. 5. Select the **Text Fields Advanced Indexing** checkbox to enable indexing of text data that doesn’t exceed a specific maximum length, into both tokenized and non-tokenized fields. For example, if this option is selected, the `name` text field with a value of **John Smith** is indexed into both the `name_t` and `name_s` fields allowing relevant search using `name_t` field (by matching to a Smith query) and also proper faceting and sorting using `name_s` field (using John Smith for sorting or faceting). If this option is not selected, the `name` text field is indexed into only the `name_t` text field by default. 6. In the **Max Length for Advanced Indexing of Text Fields** field, enter a value used to determine how many characters of the incoming text is indexed. For example, **100**. 7. Click **Save**. ## LWAI Vectorize Field The [LWAI Vectorize](/docs/lucidworks-search/09-developer-documentation/config-specs/index-pipeline-stages/vectorize-field-via-lucidworks-ai-index-stage) stage invokes a Lucidworks AI model to encode a string field to a vector representation. This stage is skipped if the field to encode doesn’t exist or is null on the pipeline document. If you are using Fusion 5.9.15 or later and have many documents to process, use the [LWAI Batch Vectorize](/docs/lucidworks-search/09-developer-documentation/config-specs/index-pipeline-stages/lwai-batch-vectorize) stage in place of this stage in the pipeline. 1. In the **Label** field, enter a unique identifier for this stage. 2. In the **Condition** field, enter a script that results in true or false, which determines if the stage should process. 3. In the **Account Name** field, select the Lucidworks AI API account name defined in [Lucidworks AI Gateway](/docs/lw-platform/lw-ai/lw-ai-gateway).\ If you do not see your account name or you are unsure which one to select, contact the Lucidworks Search team at Lucidworks. 4. In the **Model** field, select the Lucidworks AI model to use for encoding.\ If you do not see your model name or you are unsure which one to select, contact the Lucidworks Search team at Lucidworks.\ For more information, see: * [Pre-trained embedding models](/docs/lw-platform/lw-ai/lw-ai-pre-trained-embedding-models) * [Custom embedding model training](/docs/lw-platform/lw-ai/lw-ai-custom-embedding-model-training/overview) 5. In the **Source** field, enter the name of the string field where the value should be submitted to the model for encoding. If the field is blank or does not exist, this stage is not processed. Template expressions are supported. 6. In the **Destination** field, enter the name of the field where the vector value from the model response is saved. * If a value is entered in this field, the following information added to the document: * `{Destination Field}_b` is the boolean value if the vector has been indexed. * `{Destination Field}` is the vector field. 7. In the **Use Case Configuration** section, click the **+** sign to enter the parameter name and value to send to Lucidworks AI. The `useCaseConfig` parameter that is common to embedding use cases is `dataType`, but each use case may have other parameters. The value for the query stage is `query`. 8. In the **Model Configuration** section, click the **+** sign to enter the parameter name and value to send to Lucidworks AI. Several `modelConfig` parameters are common to generative AI use cases. For more information, see [Prediction API](/docs/lw-platform/lw-ai/lw-ai-apis/lw-ai-prediction-api/overview). 9. Select the **Fail on Error** checkbox to generate an exception if an error occurs while generating a prediction for a document. 10. Click **Save**. 11. Index data using the new pipeline. Verify the vector field is indexed by confirming the field is present in documents. ## Solr Indexer The [Solr Indexer](/docs/lucidworks-search/09-developer-documentation/config-specs/index-pipeline-stages/solr-indexer) stage transforms a Lucidworks Search pipeline document into a Solr document, and sends it to Solr for indexing into a collection. To configure this stage for the index pipeline: 1. In the **Label** field, enter a unique identifier for this stage or leave blank to use the default value. 2. In the **Condition** field, enter a script that results in true or false, which determines if the stage should process, or leave blank. 3. Select the **Map to Solr Schema** checkbox to select and add static and dynamic fields to map in this stage. 4. Select the **Add a field listing all document fields** checkbox to add the `_lw_fields_ss` multi-valued field to the document, which lists all fields that are being sent to Solr. 5. In the **Additional Date Formats** section, enter date formats to include in this stage. 6. In the **Additional Update Request Parameters** section, enter the parameter names and values to update the request parameters. 7. Select the **Buffer Documents and Send Them To Solr in Batches** checkbox to process the documents in batches for this stage. 8. In the **Buffer Size** field, enter the number of documents in a batch before sending the batch to Solr. If no value is specified, the default value for this search cluster is used. 9. In the **Buffer Flush Interval (milliseconds)** field, enter the maximum number of milliseconds to hold the batch before sending the batch to Solr. If no value is specified, the default value for this search cluster is used. 10. Select the **Allow expensive request parameters** checkbox to allow `commit=true` and `optimize=true` to be passed to Solr when specified as request parameters coming into this pipeline. Document commands that specify commit or optimize are still respected even if this checkbox is not selected. 11. Select the **Unmapped Fields Mapping** checkbox to specify the information for all of the fields *not* mapped in the previous sections. * In the **Source** Field, enter the name of the unmapped field to be mapped. * In the **Target** Field, enter the name of the Solr field to which the unmapped field is mapped. * In the **Operation** field, select how the field is mapped. The options are: * **Add** the unmapped field to the Solr field. * **Copy** the unmapped field to the Solr field and retain the value in the Source field. * **Delete** the unmapped field. * **Keep** the unmapped field and do not map it to a Solr field. * **Move** (replace) the Solr field value with the unmapped field Source value and remove the value from the Source field. * **Set** the value of the unmapped field to the value in the Solr field. 12. Click **Save**. ## Order the stages For the pipeline to operate correctly, the stages must be in the following order: * [Field Mapping](#field-mapping) * [Solr Dynamic Field Name Mapping](#solr-dynamic-field-name-mapping) * [LWAI Vectorize Field](#lwai-vectorize-field) * [Solr Indexer](#solr-indexer) When you have ordered the stages, click **Save**. ## Self-hosted Fusion These Fusion pipeline stages provide interfaces for configuring your Lucidworks AI models in Fusion pipelines. * **Vector search stages**. * [LWAI Vectorize Field Index stage](/docs/5/fusion/reference/config-ref/pipeline-stages/index-stages/lwai-vectorize-field) * [LWAI Batch Vectorize Index stage](/docs/5/fusion/reference/config-ref/pipeline-stages/index-stages/lwai-batch-vectorize-index-stage) * [LWAI Vectorize Query stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/lwai-vectorize-query) * **Chunker stage**. This stage asynchronously breaks down large text documents and vectorizes the chunks. * [LWAI Chunker Index stage](/docs/5/fusion/reference/config-ref/pipeline-stages/index-stages/lucidworks-ai-chunker-index-stage) * **RAG Bridge stage**. This stage extracts relevant snippets from source documents used in RAG responses. * [Chunking RAG bridge query stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/chunking-rag-bridge) * **Generative AI stages**. See [Generative AI](/docs/lw-platform/lw-ai/lw-ai-generative-ai) for conceptual information about Gen-AI features. * [LWAI Prediction Index stage](/docs/5/fusion/reference/config-ref/pipeline-stages/index-stages/lucidworks-ai-prediction-index-stage) * [LWAI Prediction Query stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/lucidworks-ai-prediction-query-stage) The LWAI Prediction index stage is a Fusion index pipeline stage that enriches your index with [Generative AI](/docs/lw-platform/lw-ai/lw-ai-generative-ai) predictions. It defaults to asynchronous processing, which does not block the pipeline while waiting for a response from Lucidworks AI. For reference information, see [LWAI Prediction index stage](/docs/lucidworks-search/09-developer-documentation/config-specs/index-pipeline-stages/lucidworks-ai-prediction-index-stage). To use this stage, non-admin Fusion users must be granted the `PUT,POST,GET:/LWAI-ACCOUNT-NAME/**` permission in Fusion, which is the Lucidworks AI API Account Name defined in Lucidworks AI Gateway when this stage is configured. To configure this stage: 1. Sign in to Fusion and click **Indexing > Index Pipelines**. 2. Click **Add+** to add a new pipeline. 3. Enter the name in **Pipeline ID**. 4. Click **Add a new pipeline stage**. 5. In the AI section, click **LWAI Prediction**. 6. In the **Label** field, enter a unique identifier for this stage. 7. In the **Condition** field, enter a script that results in true or false, which determines if the stage should process. 8. In the **Account Name** field, select the Lucidworks AI API account name defined in [Lucidworks AI Gateway](/docs/lw-platform/lw-ai/lw-ai-gateway). 9. In the **Use Case** field, select the Lucidworks AI use case to associate with this stage. * To generate a list of the use cases for your organization, see [Use Case API](/docs/lw-platform/lw-ai/lw-ai-apis/lw-ai-use-case-api). * If the **Call Asynchronously?** check box is selected, see available use cases described in [Async Prediction API](/docs/lw-platform/lw-ai/lw-ai-apis/lw-ai-async-prediction-api/overview). * If the **Call Asynchronously?** check box is *not* selected, see available use cases described in [Prediction API](/docs/lw-platform/lw-ai/lw-ai-apis/lw-ai-prediction-api/overview). 10. In the **Model** field, select the Lucidworks AI model to associate with this stage. Your Fusion account name must match the name of the account that you selected in the **Account Name** dropdown.\ For more information about models, see: * [Pre-trained embedding models](/docs/lw-platform/lw-ai/lw-ai-pre-trained-embedding-models) * [Custom embedding model training](/docs/lw-platform/lw-ai/lw-ai-custom-embedding-model-training/overview) * [Generative AI models](/docs/lw-platform/lw-ai/lw-ai-generative-ai) 11. In the **Input context variable** field, enter the name of the context variable to be used as input. Template expressions are supported. 12. In the **Destination field name and context output** field, enter the name that will be used as both the field name in the document where the prediction is written and the context variable that contains the prediction. {/* // tag::lwai-prediction-index-stage[] */} * If the **Call Asynchronously?** check box is selected and a value is entered in this field: * `{destination name}_t` is the full response. * In the document: * `_lw_ai_properties_ss` contains the Lucidworks account, boolean setting for async, use case, input for the call, and the collection. * `_lw_ai_request_count` is the number of GET requests by `predictionId` and `_lw_ai_success_count` is the number of responses without errors. These two fields are used for debugging only. Based on the deployment, the most useful measure is the ratio of `_lw_ai_success_count` divided by `_lw_ai_request_count` and then adjusting as much as possible to achieve 1.0. * `enriched_ss` contains the use case. This can be used as a boolean value to verify if the use case indexed successfully. * If the **Call Asynchronously?** check box is *not* selected and a value is entered in this field: * `{destination name}_t` is the full response. * If *no* value is entered in this field (regardless of the **Call Asynchronously?** check box setting): * `_lw_ai_{use case}_t` is the `response.response` object, which is the raw model output. * `_lw_ai_{use case}_response_s` is the full response. {/* // end::lwai-prediction-index-stage[] */} 13. In the **Use Case Configuration** section, click the **+** sign to enter the parameter name and value to send to Lucidworks AI. The `useCaseConfig` parameter is only applicable to certain use cases. * If the **Call Asynchronously?** check box is selected, `useCaseConfig` information for each applicable use case is described in [Async Prediction API](/docs/lw-platform/lw-ai/lw-ai-apis/lw-ai-async-prediction-api/overview). * If the **Call Asynchronously?** check box is *not* selected, `useCaseConfig` information for each applicable use case is described in [Prediction API](/docs/lw-platform/lw-ai/lw-ai-apis/lw-ai-prediction-api/overview). 14. In the **Model Configuration** section, click the **+** sign to enter the parameter name and value to send to Lucidworks AI. Several `modelConfig` parameters are common to generative AI use cases. * If the **Call Asynchronously?** check box is selected, `modelConfig` information is described in [Async Prediction API](/docs/lw-platform/lw-ai/lw-ai-apis/lw-ai-async-prediction-api/overview). * If the **Call Asynchronously?** check box is *not* selected, `modelConfig` information is described in [Prediction API](/docs/lw-platform/lw-ai/lw-ai-apis/lw-ai-prediction-api/overview). 15. In the **API Key** field, enter the secret value specified in the external model. For: * OpenAI models, `"apiKey"` is the value in the model’s `"[OPENAI_API_KEY]"` field. For more information, see [Authentication API keys](https://platform.openai.com/docs/api-reference/authentication). * Azure OpenAI models, `"apiKey"` is the value generated by Azure in either the model’s `"[KEY1 or KEY2]"` field. For requirements to use Azure models, see [Generative AI models](/docs/lw-platform/lw-ai/lw-ai-generative-ai). * Google VertexAI models, `"apiKey"` is the value in the model’s `"[BASE64_ENCODED_GOOGLE_SERVICE_ACCOUNT_KEY]"` field. For more information, see [Create and delete Google service account keys](https://cloud.google.com/iam/docs/keys-create-delete). 16. To run the API call asynchronously, select the **Call Asynchronously?** check box to specify the stage is to use the [Lucidworks AI Async Prediction API](/docs/lw-platform/lw-ai/lw-ai-apis/lw-ai-async-prediction-api/overview) endpoints. If this is selected, the API call does not block the pipeline while waiting for a response from Lucidworks AI. If the check box is *not* selected, the API call uses the [Prediction API](/docs/lw-platform/lw-ai/lw-ai-apis/lw-ai-prediction-api/overview), which uses the pipeline until a response is received from Lucidworks AI. Performance of other API calls can be impacted. 17. In the **Maximum Asynchronous Call Tries** field, enter the maximum number of times to send an asynchronous API call before the system generates a failure error. 18. Select the **Fail on Error** checkbox to generate an exception if an error occurs while generating a prediction for a document. 19. Click **Save**. ## Additional requirements Additional requirements to use async calls include: * Use a V2 connector. Only V2 connectors work for this task and not other options, such as PBL or V1 connectors. * Remove the `Apache Tika` stage from your parser because it can cause datasource failures with the following error: "The following components failed: \[class com.lucidworks.connectors.service.components.job.processor.DefaultDataProcessor : Only Tika Container parser can support Async Parsing.]" * Replace the `Solr Indexer` stage with the `Solr Partial Update Indexer` stage with the following settings: * `Enable Concurrency Control` set to off * `Reject Update if Solr Document is not Present` set to off * `Process All Pipeline Doc Fields` set to on * `Allow reserved fields` set to on * A parameter with `Update Type`, `Field Name` & `Value` in `Updates` The LWAI Prediction AI query stage is a Fusion pipeline query stage that enriches your search results with [Generative AI](/docs/lw-platform/lw-ai/lw-ai-generative-ai) predictions. For reference information, see [LWAI Prediction query stage](/docs/lucidworks-search/09-developer-documentation/config-specs/query-pipeline-stages/lucidworks-ai-prediction-query-stage). To use this stage, non-admin Fusion users must be granted the `PUT,POST,GET:/LWAI-ACCOUNT-NAME/**` permission in Fusion, which is the Lucidworks AI API Account Name defined in Lucidworks AI Gateway when this stage is configured. To configure this stage: 1. Sign in to Fusion and click **Querying > Query Pipelines**. 2. Click **Add+** to add a new pipeline. 3. Enter the name in **Pipeline ID**. 4. Click **Add a new pipeline stage**. 5. In the AI section, click **LWAI Prediction**. 6. In the **Label** field, enter a unique identifier for this stage. 7. In the **Condition** field, enter a script that results in true or false, which determines if the stage should process. 8. Select **Asynchronous Execution Config** if you want to run this stage asynchronously. If this field is enabled, complete the following fields: 1. Select **Enable Async Execution**. Fusion automatically assigns an **Async ID** value to this stage. Change this to a more memorable string that describes the asynchronous stages you are merging, such as `signals` or `access_control`. 2. Copy the **Async ID** value. 9. In the **Account Name** field, select the name of the Lucidworks AI integration defined when the integration was created. 10. In the **Use Case** field, select the Lucidworks AI use case to associate with this stage. * To generate a list of the use cases for your organization, see [Use Case API](/docs/lw-platform/lw-ai/lw-ai-apis/lw-ai-use-case-api). * The available use cases are described in [Prediction API](/docs/lw-platform/lw-ai/lw-ai-apis/lw-ai-prediction-api/overview). 11. In the **Model** field, select the Lucidworks AI model to associate with this stage.\ If you do not see any model names and you are a non-admin Fusion user, verify with a Fusion administrator that your user account has these permissions: `PUT,POST,GET:/LWAI-ACCOUNT-NAME/**` Your Fusion account name must match the name of the account that you selected in the **Account Name** dropdown.\ For more information about models, see: * [Pre-trained embedding models](/docs/lw-platform/lw-ai/lw-ai-pre-trained-embedding-models) * [Custom embedding model training](/docs/lw-platform/lw-ai/lw-ai-custom-embedding-model-training/overview) * [Generative AI models](/docs/lw-platform/lw-ai/lw-ai-generative-ai) 12. In the **Input context variable** field, enter the name of the context variable to be used as input. Template expressions are supported. 13. In the **Destination variable name and context output** field, enter the name that will be used as both the query response header in the prediction results and the context variable that contains the prediction. {/* // tag::lwai-prediction-query-stage[] */} * If a value is entered in this field: * `{destination name}_t` is the full response. * In the context: * `_lw_ai_properties_ss` contains the Lucidworks account, boolean setting for async, use case, input for the call, and the collection. * If *no* value is entered in this field: * `_lw_ai_{use case}_t` is the `response.response` object, which is the raw model output. * `_lw_ai_{use case}_response_s` is the full response. {/* // end::lwai-prediction-query-stage[] */} 14. Select the **Include Response Documents?** check box to include the response documents in the Lucidworks AI request. This option is only available for certain use cases. If this is selected, run the Solr Query stage to ensure documents exist before running the LWAI Prediction query stage.\ Response documents must be included in the RAG use case, which supports attaching a maximum of 3 response documents. To prevent errors, enter all of the entries described in the Document Field Mappings section. 15. In the **Document Field Mappings** section, enter the **LW AI Document field name** and its corresponding **Response document field name** to map from input documents to the fields accepted by the Prediction API RAG use case. The fields are described in the [Prediction API](/docs/lw-platform/lw-ai/lw-ai-apis/lw-ai-prediction-api/overview). If information is not entered in this section, the default mappings are used. * The `body` and `source` fields are required. * `body` - `description_t` is the document content. * `source` - `link_t` is the URL/ID of the document. * The `title` and `date` fields are optional. * `title` - `title_t` is the title of the document. * `date` - `_lw_file_modified_tdt` is the creation date of the document in epoch time format. 16. In the **Use Case Configuration** section, click the **+** sign to enter the parameter name and value to send to Lucidworks AI. * The `useCaseConfig` parameter is only applicable to certain use cases. For more information, see the [Async Prediction API](/docs/lw-platform/lw-ai/lw-ai-apis/lw-ai-async-prediction-api/overview) and the [Prediction API](/docs/lw-platform/lw-ai/lw-ai-apis/lw-ai-prediction-api/overview). * The `memoryUuid` parameter is required in the Standalone Query Rewriter use case, and is optional in the RAG use case. For more information, see [Prediction API](/docs/lw-platform/lw-ai/lw-ai-apis/lw-ai-prediction-api/overview). 17. In the **Model Configuration** section, click the **+** sign to enter the parameter name and value to send to Lucidworks AI. Several `modelConfig` parameters are common to generative AI use cases. For more information, see [Prediction API](/docs/lw-platform/lw-ai/lw-ai-apis/lw-ai-prediction-api/overview). 18. In the **API Key** field, enter the secret value specified in the external model. For: * OpenAI models, `"apiKey"` is the value in the model’s `"[OPENAI_API_KEY]"` field. For more information, see [Authentication API keys](https://platform.openai.com/docs/api-reference/authentication). * Azure OpenAI models, `"apiKey"` is the value generated by Azure in either the model’s `"[KEY1 or KEY2]"` field. For requirements to use Azure models, see [Generative AI models](/docs/lw-platform/lw-ai/lw-ai-generative-ai). * Google VertexAI models, `"apiKey"` is the value in the model’s `"[BASE64_ENCODED_GOOGLE_SERVICE_ACCOUNT_KEY]"` field. For more information, see [Create and delete Google service account keys](https://cloud.google.com/iam/docs/keys-create-delete). 19. Select the **Fail on Error** checkbox to generate an exception if an error occurs during this stage. 20. Click **Save**. Solr Vector Search supports semantic search with lexical filtering. Lucidworks recommends that if you plan to use Solr SVS, you first set up Lucidworks AI to vectorize text. The LWAI Vectorize pipeline is a default pipeline that contains the required index stages to set up vector search using [Lucidworks AI](/docs/lw-platform/lw-ai/overview). For more information,refer to Configure Neural Hybrid Search. This feature is currently only available to clients who have contracted with Lucidworks for features related to Neural Hybrid Search and Lucidworks AI. This feature is available starting in Fusion 5.9.5 and in all subsequent Fusion 5.9 releases. This pipeline uses the following stages: * [Field Mapping](#field-mapping) * [Solr Dynamic Field Name Mapping](#solr-dynamic-field-name-mapping) * [LWAI Vectorize Field](#lwai-vectorize-field) * [Solr Indexer](#solr-indexer) ## Configure the pipeline To add the Lucidworks AI (LWAI) Vectorize index pipeline: 1. Sign in to Fusion and click **Indexing > Index Pipelines**. 2. Select the default **LWAI-vectorize** pipeline. 3. Configure the following stages included in the default pipeline. ## Field Mapping The [Field Mapping](/docs/5/fusion/reference/config-ref/pipeline-stages/index-stages/field-mapper-index-stage) stage customizes mapping of the fields in an index pipeline document to fields in the Solr scheme. To configure this stage for the index pipeline: 1. In the **Label** field, enter a unique identifier for this stage or leave blank to use the default value. 2. In the **Condition** field, enter a script that results in true or false, which determines if the stage should process, or leave blank. 3. Select the **Allow System Fields Mapping?** checkbox to map system fields in this stage. 4. In the **Field Retention** section, enter specific fields to either keep or delete. 5. In the **Field Value Updates** section, enter specific fields and then designate the value to either add to the field, or set on the field. When a value is added, any values previously on the field are retained. When a value is set, any values previously on the field are overwritten by the new value entered. 6. In the **Field Translations** section, enter specific fields to either move or copy to a different field. When a field is moved, the values from the source field are moved over to the target field and the source field is removed. When a field is copied, the values from the source field are copied over to the target field and the source field is retained. 7. Select the **Unmapped Fields** checkbox to specify the operation on the fields *not* mapped in the previous sections. Select the **Keep** checkbox to keep all unmapped fields. This is the only option you need to select for the LWAI-vectorize stage. 8. Click **Save**. ## Solr Dynamic Field Name Mapping The [Solr Dynamic Field Name Mapping](/docs/5/fusion/reference/config-ref/pipeline-stages/index-stages/solr-dynamic-field-name-mapping-index-stage) stage maps pipeline document fields to Solr dynamic fields. 1. In the **Label** field, enter a unique identifier for this stage or leave blank to use the default value. 2. In the **Condition** field, enter a script that results in true or false, which determines if the stage should process, or leave blank. 3. Select the **Duplicate Single-Valued Fields as Multi-Valued Fields** checkbox to enable indexing of field data into both single-valued and multi-valued Solr fields. For example, if this option is selected, the `phone` field is indexed into both the `phone_s` single-valued field and the `phone_ss` multi-valued field. If this option is not selected, the `phone` field is indexed into only the `phone_s` single-valued field. 4. In the **Field Not To Map** section, enter the names of the fields that should not be mapped by this stage. 5. Select the **Text Fields Advanced Indexing** checkbox to enable indexing of text data that doesn’t exceed a specific maximum length, into both tokenized and non-tokenized fields. For example, if this option is selected, the `name` text field with a value of **John Smith** is indexed into both the `name_t` and `name_s` fields allowing relevant search using `name_t` field (by matching to a Smith query) and also proper faceting and sorting using `name_s` field (using John Smith for sorting or faceting). If this option is not selected, the `name` text field is indexed into only the `name_t` text field by default. 6. In the **Max Length for Advanced Indexing of Text Fields** field, enter a value used to determine how many characters of the incoming text is indexed. For example, **100**. 7. Click **Save**. ## LWAI Vectorize Field The [LWAI Vectorize](/docs/5/fusion/reference/config-ref/pipeline-stages/index-stages/lwai-vectorize-field) stage invokes a Lucidworks AI model to encode a string field to a vector representation. This stage is skipped if the field to encode doesn’t exist or is null on the pipeline document. If you are using Fusion 5.9.15 or later and have many documents to process, use the [LWAI Batch Vectorize](/docs/5/fusion/reference/config-ref/pipeline-stages/index-stages/lwai-batch-vectorize-index-stage) stage in place of this stage in the pipeline. 1. In the **Label** field, enter a unique identifier for this stage. 2. In the **Condition** field, enter a script that results in true or false, which determines if the stage should process. 3. In the **Account Name** field, select the Lucidworks AI API account name defined in [Lucidworks AI Gateway](/docs/lw-platform/lw-ai/lw-ai-gateway).\ If your account name does not appear in the list or you are unsure which one to select, check your Lucidworks AI Gateway configuration. 4. In the **Model** field, select the Lucidworks AI model to use for encoding.\ If your model does not appear in the list or you are unsure which one to select, check your Lucidworks AI Gateway configuration.\ For more information, see: * [Pre-trained embedding models](/docs/lw-platform/lw-ai/lw-ai-pre-trained-embedding-models) * [Custom embedding model training](/docs/lw-platform/lw-ai/lw-ai-custom-embedding-model-training/overview) 5. In the **Source** field, enter the name of the string field where the value should be submitted to the model for encoding. If the field is blank or does not exist, this stage is not processed. Template expressions are supported. 6. In the **Destination** field, enter the name of the field where the vector value from the model response is saved. * If a value is entered in this field, the following information is added to the document: * `{Destination Field}_b` is the boolean value if the vector has been indexed. * `{Destination Field}` is the vector field. 7. In the **Use Case Configuration** section, click the **+** sign to enter the parameter name and value to send to Lucidworks AI. The `useCaseConfig` parameter that is common to embedding use cases is `dataType`, but each use case may have other parameters. The value for the query stage is `query`. 8. In the **Model Configuration** section, click the **+** sign to enter the parameter name and value to send to Lucidworks AI. Several `modelConfig` parameters are common to generative AI use cases. For more information, see [Prediction API](/docs/lw-platform/lw-ai/lw-ai-apis/lw-ai-prediction-api/overview). 9. Select the **Fail on Error** checkbox to generate an exception if an error occurs while generating a prediction for a document. 10. Click **Save**. 11. Index data using the new pipeline. Verify the vector field is indexed by confirming the field is present in documents. ## Solr Indexer The [Solr Indexer](/docs/5/fusion/reference/config-ref/pipeline-stages/index-stages/solr-indexer-stage) stage transforms a Fusion pipeline document into a Solr document, and sends it to Solr for indexing into a collection. To configure this stage for the index pipeline: 1. In the **Label** field, enter a unique identifier for this stage or leave blank to use the default value. 2. In the **Condition** field, enter a script that results in true or false, which determines if the stage should process, or leave blank. 3. Select the **Map to Solr Schema** checkbox to select and add static and dynamic fields to map in this stage. 4. Select the **Add a field listing all document fields** checkbox to add the `_lw_fields_ss` multi-valued field to the document, which lists all fields that are being sent to Solr. 5. In the **Additional Date Formats** section, enter date formats to include in this stage. 6. In the **Additional Update Request Parameters** section, enter the parameter names and values to update the request parameters. 7. Select the **Buffer Documents and Send Them To Solr in Batches** checkbox to process the documents in batches for this stage. 8. In the **Buffer Size** field, enter the number of documents in a batch before sending the batch to Solr. If no value is specified, the default value for this search cluster is used. 9. In the **Buffer Flush Interval (milliseconds)** field, enter the maximum number of milliseconds to hold the batch before sending the batch to Solr. If no value is specified, the default value for this search cluster is used. 10. Select the **Allow expensive request parameters** checkbox to allow `commit=true` and `optimize=true` to be passed to Solr when specified as request parameters coming into this pipeline. Document commands that specify commit or optimize are still respected even if this checkbox is not selected. 11. Select the **Unmapped Fields Mapping** checkbox to specify the information for all of the fields *not* mapped in the previous sections. * In the **Source** Field, enter the name of the unmapped field to be mapped. * In the **Target** Field, enter the name of the Solr field to which the unmapped field is mapped. * In the **Operation** field, select how the field is mapped. The options are: * **Add** the unmapped field to the Solr field. * **Copy** the unmapped field to the Solr field and retain the value in the Source field. * **Delete** the unmapped field. * **Keep** the unmapped field and do not map it to a Solr field. * **Move** (replace) the Solr field value with the unmapped field Source value and remove the value from the Source field. * **Set** the value of the unmapped field to the value in the Solr field. 12. Click **Save**. ## Order the stages For the pipeline to operate correctly, the stages must be in the following order: * [Field Mapping](#field-mapping) * [Solr Dynamic Field Name Mapping](#solr-dynamic-field-name-mapping) * [LWAI Vectorize Field](#lwai-vectorize-field) * [Solr Indexer](#solr-indexer) When you have ordered the stages, click **Save**. ## Debugging If your pipeline is not producing the desired result, these debugging tips can help you identify the source of the issue and resolve it. For general pipeline debugging, these procedures are helpful: * [View parameters](#view-parameters) to confirm that the right ones are being passed from one stage to the next. * [Enable **Fail on Error**](#enable-“fail-on-error”) to generate an exception when a pipeline error occurs. For Neural Hybrid Search (NHS) pipelines, these additional procedures apply: * [Verify field definitions](#verify-field-definitions) to confirm that they are accurate. * [Check query parsers](#check-query-parsers) to ensure that the right ones are enables. * [Confirm vector usage in queries](#confirm-vector-usage-in-queries) to make sure that vectors are being generated and used. For self-hosted Fusion, verify that your Fusion administrator has set up Lucidworks AI Gateway and configured user permissions as explained in the [Quickstart](/docs/lw-platform/lw-ai/quickstart-lucidworks-ai-nhs-rag#for-self-hosted-fusion-enable-lucidworks-ai). ### View parameters When debugging a pipeline, it helps to see the parameters that are being passed to or from each stage. There are several ways to exposed those parameters: Follow these steps to add a Logging stage to your pipeline: 1. In Fusion UI, navigate to **Indexing > Index Pipelines** (for index pipelines) or **Querying > Query Pipelines** (for query pipelines). 2. Click **Add a new pipeline stage**. 3. Select **Logging** from the Troubleshooting section. 4. In the **Label** field, enter a descriptive name (for example, "Debug After Vectorize"). 5. Set the **detailed** property to `true` to print the full Request or PipelineDocument object. 6. Place the Logging stage after the stage you want to debug. 7. Click **Save**. 8. Run your pipeline and check the appropriate log file for your pipeline type: * Query pipelines: `https://FUSION_HOST/var/log/api/api.log` * Index pipelines: `https://FUSION_HOST/var/log/api/fusion-indexing.log` Follow these steps to inspect context variables: 1. Add a **JavaScript** stage to your pipeline. 2. Use the `ctx` variable to inspect context data: ```javascript theme={"dark"} function (request, response, ctx) { // View all context variables logger.info("Context: " + JSON.stringify(ctx)); // Check specific variable (for example, vector from LWAI stage) if(ctx.get('vector') !== null) { logger.info("Vector: " + ctx.get('vector')); } } ``` 3. Check logs at `https://FUSION_HOST/var/log/api/api.log`. Follow these steps to debug using Query Workbench: 1. Navigate to **Querying > Query Workbench**. 2. Enter a test query and click **Search**. 3. Click the **Debug** tab. The debug view displays the following information: * Request parameters * Pipeline stage execution details * Response data including `responseHeader` and `debug.explain` 4. Switch to **View As: JSON** to see the full response structure. ### Enable **Fail on Error** The **Fail on Error** setting determines whether silent failures can occur in your pipeline. By enabling **Fail on Error** during development, testing, and troubleshooting, you ensure that configuration issues, authentication problems, or model errors are immediately visible rather than producing incomplete or incorrect data that can be difficult to troubleshoot later. The **Fail on Error** setting controls whether pipeline execution stops when an LWAI stage encounters an error. Follow these steps to configure this setting: 1. In Fusion UI, navigate to your pipeline (Index or Query). 2. Click the LWAI stage you want to configure (for example, **LWAI Vectorize Field** or **LWAI Vectorize Query**). 3. Locate the **Fail on Error** checkbox at the bottom of the stage configuration. 4. Select the checkbox to enable any of the following behaviors: * Stop pipeline processing and throw an exception on errors * Get immediate feedback when LWAI models fail or are misconfigured * Guarantee data quality by preventing indexing of documents without vectors In production environments, keep this feature disabled to ensure that service is not interrupted when errors occur. 6. Click **Save**. During development and testing, enable **Fail on Error** to identify configuration issues early. In production, consider disabling **Fail on Error** if you want resilient pipelines that can handle occasional model failures. For critical vector fields, enable **Fail on Error** to ensure that all documents have required embeddings. Follow these steps to verify your Fail on Error configuration: 1. Trigger an intentional error (for example, use an invalid model name or account). 2. Verify that the pipeline fails (when Fail on Error is enabled) or continues (when Fail on Error is disabled). 3. Review logs at `https://FUSION_HOST/var/log/api/api.log` for error messages. ### Verify field definitions For Neural Hybrid Search pipelines, accurate field definitions are critical. Use the following procedures to verify your field configuration: You can inspect your indexed data using Query Workbench or the API. Accurate field definitions are critical for Neural Hybrid Search to function correctly. To use Query Workbench, follow these steps: 1. Navigate to **Querying > Query Workbench**. 2. Enter a wildcard query: `*:*` 3. In the **fl** (field list) parameter, add your vector field name (for example, `text_v,text_v_b`). 4. Click **Search**. 5. Verify the following conditions: * The vector field contains array values (for example, `[0.123, -0.456, ...]`) * The boolean field `{field}_b` is `true` for vectorized documents To use the API, run the following command: ```bash theme={"dark"} curl -u 'USERNAME:PASSWORD' \ 'https://FUSION_HOST/api/collections/COLLECTION_NAME/select?q=*:*&fl=id,text_v,text_v_b&rows=5' ``` Follow these steps to verify field matching between your query and index pipelines: 1. In your query pipeline, open your **Neural Hybrid Query** or **Hybrid Query** stage. 2. Note the **Vector Query Field** value (for example, `text_v`). 3. In your index pipeline, open your vectorization stage (**LWAI Vectorize Field**, **LWAI Batch Vectorize**, or **Ray/Seldon Vectorize Field**). 4. Verify that the **Destination** field matches the Vector Query Field from your query pipeline. Follow these steps to verify the **Source** field configuration in your vectorization stage: 1. In your vectorization index stage, verify that the **Source** field meets the following requirements: * Points to a valid string field in your documents * Uses correct template expression syntax if needed (for example, ``) 2. Test the configuration by indexing a sample document and confirming that the configured **Source** field has content. If vector fields are not present in your indexed data, follow these steps: 1. Stop the datasource job if it is running. 2. Verify that your index pipeline includes a vectorization stage. 3. Optional: Clear the collection if you want to start fresh. 4. Re-run the datasource job. 5. Verify that vector fields appear in the indexed documents. ### Check query parsers Neural Hybrid Search requires specific query parsers in your Solr configuration. Use the following procedures to verify and configure them: Neural Hybrid Search requires specific query parsers in your Solr configuration. Different hybrid query stages require different parsers, as shown in the following table: | Query Stage | Required Parser | | -------------------------------- | ---------------- | | Chunking Neural Hybrid Query | `_lw_chunk_wrap` | | Neural Hybrid Query | `neuralHybrid` | | Hybrid Query (5.9.9 and earlier) | `xVecSim` | Follow these steps to check for required parsers: 1. In Fusion UI, navigate to **System > Solr Admin**. 2. Select your collection. 3. Click **Files** > **solrconfig.xml**. 4. Search for ` ``` For new collections, these parsers are automatically included. If you work with older collections or migrated data, manually add them if they are missing. If any required query parsers are missing from your Solr configuration, follow these steps: 1. In the solrconfig.xml editor, locate an appropriate insertion point. Typically, you can add parsers after other `` entries or before ``. 2. Add the following snippet: ```xml theme={"dark"} ``` 3. Click **Save**. 4. Reload the collection by completing the following steps: * Navigate to **Collections**. * Select your collection. * Click **Reload**. Follow these steps to verify that your query parsers are configured correctly: 1. Navigate to **Query Workbench**. 2. Run a test query using your hybrid query stage. 3. Click the **Debug** tab and check for any parser-related errors. 4. If the configuration is correct, the `debug.parsedquery` field displays the hybrid query. ### Confirm vector usage in queries To ensure that vectors are being used in your Neural Hybrid Search queries, use the following verification and troubleshooting procedures: Follow these steps to verify that vectors are being used in your Neural Hybrid Search queries: 1. Navigate to **Querying > Query Workbench**. 2. Enter a test query and click **Search**. 3. Click **View As: JSON** to see the full response. 4. Look for the following indicators in the response: * `responseHeader.params.q`: Should contain vector query syntax (for example, `{!neuralHybrid ...}` or `{!knn ...}`) * `debug.explain`: Should reference vector similarity calculations * `debug.parsedquery`: Should show the parsed hybrid query The following example shows the expected query structure in the JSON response: ```json theme={"dark"} { "responseHeader": { "params": { "q": "{!neuralHybrid lexicalQuery='your search' ...}" } } } ``` Follow these steps to view detailed scoring information: 1. In Query Workbench, add the debug parameter: `debug=true` 2. Click **Search**. 3. In the **Debug** tab, examine the **explain** field for each document. 4. Look for the following information: * Vector similarity scores * Neural hybrid score calculations * Both lexical and semantic components The explain output shows the combined score calculation: ``` combined score = (lexicalWeight × lexicalScore) + (vectorWeight × vectorScore) ``` Follow these steps to confirm that vectors are available in the pipeline context: 1. Add a JavaScript query stage after your LWAI Vectorize Query stage. 2. Add the following code to the stage: ```javascript theme={"dark"} function (request, response, ctx) { var vector = ctx.get('vector'); if(vector === null) { logger.error("ERROR: No vector in context!"); } else { logger.info("Vector found: " + vector.substring(0, 50) + "..."); } } ``` 3. Check logs at `https://FUSION_HOST/var/log/api/api.log`. The following table describes common issues and their solutions: | Issue | Possible solutions | | ------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | No vector in response | Verify that the LWAI Vectorize Query stage runs before the hybrid stage. Verify that the **Output context variable** is set (default: `vector`). Ensure that LWAI Gateway is configured and accessible. | | Query uses only lexical search | Verify that the hybrid query stage has the correct **Vector Input** (for example, ``). Verify that the **Vector Query Field** matches your indexed vector field. Ensure that the vectorization stage completed successfully. |