Lucidworks AI provides pre-trained embedding models and supports custom embedding models. For additional details about configuration, see the APIs.
Make sure you have configured a Lucidworks AI Gateway integration before you begin. Lucidworks AI Gateway provides a secure, authenticated connection between Fusion and your hosted models.

Run stages asynchronously

Some stages allow for asynchronous processing for work that may take a while, keeping the overall pipeline moving. Fusion does not block the request thread while the stage is waiting. Instead, it waits for the stage’s result right before it needs it to finish the response.

When to use asynchronous processing

Use asynchronous processing for the following:
  • A stage calls out to a remote system.
  • The stage reads or writes large files or blobs.
  • The stage can run in parallel with other stages.
Use synchronous processing for the following:
  • The stage does only local work and typically finishes in under a couple of milliseconds.
  • The stage coordinates tasks that must happen one at a time.

Benefits of asynchronous processing

Asynchronous processing allows:
  • Lower latency by overlapping slower calls with other work.
  • Greater resilience because if one asynchronous call is slow or errors out, Fusion can time it out without stalling the whole pipeline.
  • Easier error recognition and resolution because timeouts and cancellations are scoped to the stage.
Before you can use Lucidworks AI with Lucidworks Platform, you must configure the Lucidworks AI Gateway to provide a secure, authenticated integration between self-hosted Fusion and your hosted models. This configuration is done through a secret properties file that you can find in the Lucidworks Platform UI.
This feature is available starting in Fusion 5.9.5 and in all subsequent Fusion 5.9 releases.
Integrations are created for you by the Lucidworks team. But as a workspace owner, you can configure those integrations with Lucidworks AI Gateway. Each account can have its own set of credentials and associated scopes, which define the operations it can perform. If configuration properties are not provided at the account level, default settings are used instead.To configure the Lucidworks AI Gateway, navigate to the megamenu and click Models.
  1. On the Integrations tab, click your integration. If you don’t see your integration, contact your Lucidworks representative.
  2. Download or copy the YAML code and paste it into a file called account.yaml. The file for a single integration should look similar to this one:
    lwai-gateway:
     lwai:
      credentials: |
         fusion.lwai.default.baseUrl: https://APPLICATION_ID.applications.lucidworks.com
         fusion.lwai.default.authEndpoint: https://identity.lucidworks.com/oauth2/XXXXXXXXXX/v1/token
         fusion.lwai.account[0].name: ACCOUNT_NAME
         fusion.lwai.account[0].scopes: machinelearning.predict,machinelearning.model
         fusion.lwai.account[0].clientId: *****
         fusion.lwai.account[0].clientSecret: *****
    
    For a configuration with multiple integrations, it should look like this:
    lwai-gateway:
     lwai:
      credentials: |
         fusion.lwai.default.authEndpoint: https://identity.lucidworks.com/oauth2/XXXXXXXXXX/v1/token
         fusion.lwai.account[0].baseUrl: https://APPLICATION_ID.applications.lucidworks.com
         fusion.lwai.account[0].name: ACCOUNT_NAME
         fusion.lwai.account[0].scopes: machinelearning.predict,machinelearning.model
         fusion.lwai.account[0].clientId: *****
         fusion.lwai.account[0].clientSecret: *****
    
         fusion.lwai.account[1].baseUrl: https://APPLICATION_ID2.applications.lucidworks.com
         fusion.lwai.account[1].name: ACCOUNT_NAME
         fusion.lwai.account[1].scopes: machinelearning.predict,machinelearning.model
         fusion.lwai.account[1].clientId: *****
         fusion.lwai.account[1].clientSecret: *****
    
    Non-admin users must have the following permissions to use Lucidworks AI integrations:
    PUT,POST,GET:/LWAI-ACCOUNT-NAME/** where LWAI-ACCOUNT-NAME must match the value of fusion.lwai.account[n].name in the integration YAML.
  3. Apply the file to your Fusion configuration file. For example:
    helm upgrade KUBERNETES_NAMESPACE lucidworks/fusion -f FUSION_VALUES.yaml
    
These Fusion pipeline stages provide interfaces for configuring your Lucidworks AI models in Fusion pipelines. There are differences for Managed Fusion versus self-hosted Fusion clients.

Managed Fusion

These Managed Fusion pipeline stages provide interfaces for configuring your Lucidworks AI models in Managed Fusion pipelines.
The LWAI Prediction index stage is a Fusion index pipeline stage that enriches your index with Generative AI predictions. It defaults to asynchronous processing, which does not block the pipeline while waiting for a response from Lucidworks AI.For reference information, see LWAI Prediction index stage.To use this stage, non-admin Fusion users must be granted the PUT,POST,GET:/LWAI-ACCOUNT-NAME/** permission in Fusion, which is the Lucidworks AI API Account Name defined in Lucidworks AI Gateway when this stage is configured.To configure this stage:
  1. Sign in to Fusion and click Indexing > Index Pipelines.
  2. Click Add+ to add a new pipeline.
  3. Enter the name in Pipeline ID.
  4. Click Add a new pipeline stage.
  5. In the AI section, click LWAI Prediction.
  6. In the Label field, enter a unique identifier for this stage.
  7. In the Condition field, enter a script that results in true or false, which determines if the stage should process.
  8. In the Account Name field, select the Lucidworks AI API account name defined in Lucidworks AI Gateway.
  9. In the Use Case field, select the Lucidworks AI use case to associate with this stage.
    • To generate a list of the use cases for your organization, see Use Case API.
    • If the Call Asynchronously? check box is selected, see available use cases described in Async Prediction API.
    • If the Call Asynchronously? check box is not selected, see available use cases described in Prediction API.
  10. In the Model field, select the Lucidworks AI model to associate with this stage. Your Fusion account name must match the name of the account that you selected in the Account Name dropdown.
    For more information about models, see:
  11. In the Input context variable variable field, enter the name of the variable in context to be used as input. Template expressions are supported.
  12. In the Destination field name and context output field, enter the name that will be used as both the field name in the document where the prediction is written and the context variable that contains the prediction.
    • If the Call Asynchronously? check box is selected and a value is entered in this field:
    • {destination name}_t is the full response.
    • In the document:
      • _lw_ai_properties_ss contains the Lucidworks account, boolean setting for async, use case, input for the call, and the collection.
      • _lw_ai_request_count is the number of GET requests by predictionId and _lw_ai_success_count is the number of responses without errors. These two fields are used for debugging only. Based on the deployment, the most useful measure is the ratio of _lw_ai_success_count divided by `_lw_ai_request_count and then adjusting as much as possible to achieve 1.0.
      • enriched_ss contains the use case. This can be used as a boolean value to verify if the use case indexed successfully.
    • If the Call Asynchronously? check box is not selected and a value is entered in this field:
    • {destination name}_t is the full response.
    • If no value is entered in this field (regardless of the Call Asynchronously? check box setting):
    • _lw_ai_{use case}_t is the response.response object, which is the raw model output.
    • _lw_ai_{use case}_response_s is the full response.
  13. In the Use Case Configuration section, click the + sign to enter the parameter name and value to send to Lucidworks AI. The useCaseConfig parameter is only applicable to certain use cases.
    • If the Call Asynchronously? check box is selected, useCaseConfig information for each applicable use case is described in Async Prediction API.
    • If the Call Asynchronously? check box is not selected, useCaseConfig information for each applicable use case is described in Prediction API.
  14. In the Model Configuration section, click the + sign to enter the parameter name and value to send to Lucidworks AI. Several modelConfig parameters are common to generative AI use cases.
    • If the Call Asynchronously? check box is selected, modelConfig information is described in Async Prediction API.
    • If the Call Asynchronously? check box is not selected, modelConfig information is described in Prediction API.
  15. In the API Key field, enter the secret value specified in the external model. For:
    • OpenAI models, "apiKey" is the value in the model’s "[OPENAI_API_KEY]" field. For more information, see Authentication API keys.
    • Azure OpenAI models, "apiKey" is the value generated by Azure in either the model’s "[KEY1 or KEY2]" field. For requirements to use Azure models, see Generative AI models.
    • Google VertexAI models, "apiKey" is the value in the model’s
    "[BASE64_ENCODED_GOOGLE_SERVICE_ACCOUNT_KEY]" field. For more information, see Create and delete Google service account keys.
  16. To run the API call asynchronously, select the Call Asynchronously? check box to specify the stage is to use the Lucidworks AI Async Prediction API endpoints. If this is selected, the API call does not block the pipeline while waiting for a response from Lucidworks AI. If the check box is not selected, the API call uses the Prediction API, which uses the pipeline until a response is received from Lucidworks AI. Performance of other API calls can be impacted.
  17. In the Maximum Asynchronous Call Tries field, enter the maximum number of times to send an asynchronous API call before the system generates a failure error.
  18. Select the Fail on Error checkbox to generate an exception if an error occurs while generating a prediction for a document.
  19. Click Save.

Additional requirements

Additional requirements to use async calls include:
  • Use a V2 connector. Only V2 connectors work for this task and not other options, such as PBL or V1 connectors.
  • Remove the Apache Tika stage from your parser because it can cause datasource failures with the following error: “The following components failed: [class com.lucidworks.connectors.service.components.job.processor.DefaultDataProcessor : Only Tika Container parser can support Async Parsing.]”
  • Replace the Solr Indexer stage with the Solr Partial Update Indexer stage with the following settings:
    • Enable Concurrency Control set to off
    • Reject Update if Solr Document is not Present set to off
    • Process All Pipeline Doc Fields set to on
    • Allow reserved fields set to on
    • A parameter with Update Type, Field Name & Value in Updates
The LWAI Prediction AI query stage is a Fusion pipeline query stage that enriches your search results with Generative AI predictions.For reference information, see LWAI Prediction query stage.To use this stage, non-admin Fusion users must be granted the PUT,POST,GET:/LWAI-ACCOUNT-NAME/** permission in Fusion, which is the Lucidworks AI API Account Name defined in Lucidworks AI Gateway when this stage is configured.To configure this stage:
  1. Sign in to Fusion and click Querying > Query Pipelines.
  2. Click Add+ to add a new pipeline.
  3. Enter the name in Pipeline ID.
  4. Click Add a new pipeline stage.
  5. In the AI section, click LWAI Prediction.
  6. In the Label field, enter a unique identifier for this stage.
  7. In the Condition field, enter a script that results in true or false, which determines if the stage should process.
  8. Select Asynchronous Execution Config if you want to run this stage asynchronously. If this field is enabled, complete the following fields:
    1. Select Enable Async Execution. Fusion automatically assigns an Async ID value to this stage. Change this to a more memorable string that describes the asynchronous stages you are merging, such as signals or access_control.
    2. Copy the Async ID value.
  9. In the Account Name field, select the name of the Lucidworks AI integration defined when the integration was created.
  10. In the Use Case field, select the Lucidworks AI use case to associate with this stage.
    • To generate a list of the use cases for your organization, see Use Case API.
    • The available use cases are described in Prediction API.
  11. In the Model field, select the Lucidworks AI model to associate with this stage.
    If you do not see any model names and you are a non-admin Fusion user, verify with a Fusion administrator that your user account has these permissions: PUT,POST,GET:/LWAI-ACCOUNT-NAME/**
    Your Fusion account name must match the name of the account that you selected in the Account Name dropdown.
    For more information about models, see:
  12. In the Input context variable variable field, enter the name of the variable in context to be used as input. Template expressions are supported.
  13. In the Destination variable name and context output field, enter the name that will be used as both the query response header in the prediction results and the context variable that contains the prediction.
    • If a value is entered in this field:
    • {destination name}_t is the full response.
    • In the context:
      • _lw_ai_properties_ss contains the Lucidworks account, boolean setting for async, use case, input for the call, and the collection.
    • If no value is entered in this field:
    • _lw_ai_{use case}_t is the response.response object, which is the raw model output.
    • _lw_ai_{use case}_response_s is the full response.
  14. Select the Include Response Documents? check box to include the response documents in the Lucidworks AI request. This option is only available for certain use cases. If this is selected, run the Solr Query stage to ensure documents exist before running the LWAI Prediction query stage.
    Response documents must be included in the RAG use case, which supports attaching a maximum of 3 response documents. To prevent errors, enter all of the entries described in the Document Field Mappings section.
  15. In the Document Field Mappings section, enter the LW AI Document field name and its corresponding Response document field name to map from input documents to the fields accepted by the Prediction API RAG use case. The fields are described in the Prediction API. If information is not entered in this section, the default mappings are used.
    • The body and source fields are required.
    • body - description_t is the contents of the document.
    • source - link_t is the URL/ID of the document.
    • The title and date fields are optional.
    • title - title_t is the title of the document.
    • date - _lw_file_modified_tdt is the creation date of the document in epoch time format.
  16. In the Use Case Configuration section, click the + sign to enter the parameter name and value to send to Lucidworks AI.
    • The useCaseConfig parameter is only applicable to certain use cases. For more information, see the Async Prediction API and the Prediction API.
    • The memoryUuid parameter is required in the Standalone Query Rewriter use case, and is optional in the RAG use case.
    For more information, see Prediction API.
  17. In the Model Configuration section, click the + sign to enter the parameter name and value to send to Lucidworks AI. Several modelConfig parameters are common to generative AI use cases. For more information, see Prediction API.
  18. In the API Key field, enter the secret value specified in the external model. For:
    • OpenAI models, "apiKey" is the value in the model’s "[OPENAI_API_KEY]" field. For more information, see Authentication API keys.
    • Azure OpenAI models, "apiKey" is the value generated by Azure in either the model’s "[KEY1 or KEY2]" field. For requirements to use Azure models, see Generative AI models.
    • Google VertexAI models, "apiKey" is the value in the model’s
    "[BASE64_ENCODED_GOOGLE_SERVICE_ACCOUNT_KEY]" field. For more information, see Create and delete Google service account keys.
  19. Select the Fail on Error checkbox to generate an exception if an error occurs during this stage.
  20. Click Save.
Solr Vector Search supports semantic search with lexical filtering. Lucidworks recommends that if you plan to use Solr SVS, you first set up Lucidworks AI to vectorize text.
The LWAI Vectorize pipeline is a default pipeline that contains the required index stages to set up vector search using Lucidworks AI.
This feature is currently only available to clients who have contracted with Lucidworks for features related to Neural Hybrid Search and Lucidworks AI.
This feature is only available in Managed Fusion 5.9.x for versions 5.9.6+.
This pipeline uses the following stages:

Configure the pipeline

To add the Lucidworks AI (LWAI) Vectorize index pipeline:
  1. Sign in to Managed Fusion and click Indexing > Index Pipelines.
  2. Select the default LWAI-vectorize pipeline.
  3. Configure the following stages included in the default pipeline.

Field Mapping

The Field Mapping stage customizes mapping of the fields in an index pipeline document to fields in the Solr scheme.To configure this stage for the index pipeline:
  1. In the Label field, enter a unique identifier for this stage or leave blank to use the default value.
  2. In the Condition field, enter a script that results in true or false, which determines if the stage should process, or leave blank.
  3. Select the Allow System Fields Mapping? checkbox to map system fields in this stage.
  4. In the Field Retention section, enter specific fields to either keep or delete.
  5. In the Field Value Updates section, enter specific fields and then designate the value to either add to the field, or set on the field. When a value is added, any values previously on the field are retained. When a value is set, any values previously on the field are overwritten by the new value entered.
  6. In the Field Translations section, enter specific fields to either move or copy to a different field. When a field is moved, the values from the source field are moved over to the target field and the source field is removed. When a field is copied, the values from the source field are copied over to the target field and the source field is retained.
  7. Select the Unmapped Fields checkbox to specify the operation on the fields not mapped in the previous sections. Select the Keep checkbox to keep all unmapped fields. This is the only option you need to select for the LWAI-vectorize stage.
  8. Click Save.

Solr Dynamic Field Name Mapping

The Solr Dynamic Field Name Mapping stage maps pipeline document fields to Solr dynamic fields.
  1. In the Label field, enter a unique identifier for this stage or leave blank to use the default value.
  2. In the Condition field, enter a script that results in true or false, which determines if the stage should process, or leave blank.
  3. Select the Duplicate Single-Valued Fields as Multi-Valued Fields checkbox to enable indexing of field data into both single-valued and multi-valued Solr fields. For example, if this option is selected, the phone field is indexed into both the phone_s single-valued field and the phone_ss multi-valued field. If this option is not selected, the phone field is indexed into only the phone_s single-valued field.
  4. In the Field Not To Map section, enter the names of the fields that should not be mapped by this stage.
  5. Select the Text Fields Advanced Indexing checkbox to enable indexing of text data that doesn’t exceed a specific maximum length, into both tokenized and non-tokenized fields. For example, if this option is selected, the name text field with a value of John Smith is indexed into both the name_t and name_s fields allowing relevant search using name_t field (by matching to a Smith query) and also proper faceting and sorting using name_s field (using John Smith for sorting or faceting). If this option is not selected, the name text field is indexed into only the name_t text field by default.
  6. In the Max Length for Advanced Indexing of Text Fields field, enter a value used to determine how many characters of the incoming text is indexed. For example, 100.
  7. Click Save.

LWAI Vectorize Field

The LWAI Vectorize stage invokes a Lucidworks AI model to encode a string field to a vector representation. This stage is skipped if the field to encode doesn’t exist or is null on the pipeline document.
  1. In the Label field, enter a unique identifier for this stage.
  2. In the Condition field, enter a script that results in true or false, which determines if the stage should process.
  3. In the Account Name field, select the Lucidworks AI API account name defined in Lucidworks AI Gateway.
    If you do not see your account name or you are unsure which one to select, contact the Managed Fusion team at Lucidworks.
  4. In the Model field, select the Lucidworks AI model to use for encoding.
    If you do not see your model name or you are unsure which one to select, contact the Managed Fusion team at Lucidworks.
    For more information, see:
  5. In the Source field, enter the name of the string field where the value should be submitted to the model for encoding. If the field is blank or does not exist, this stage is not processed. Template expressions are supported.
  6. In the Destination field, enter the name of the field where the vector value from the model response is saved.
    • If a value is entered in this field, the following information added to the document:
    • {Destination Field}_b is the boolean value if the vector has been indexed.
    • {Destination Field} is the vector field.
  7. In the Use Case Configuration section, click the + sign to enter the parameter name and value to send to Lucidworks AI. The useCaseConfig parameter that is common to embedding use cases is dataType, but each use case may have other parameters. The value for the query stage is query.
  8. The Model Configuration section is not currently available.
  9. The Call Asynchronously? check box is not currently available.
  10. Select the Fail on Error checkbox to generate an exception if an error occurs while generating a prediction for a document.
  11. Click Save.
  12. Index data using the new pipeline. Verify the vector field is indexed by confirming the field is present in documents.

Solr Indexer

The Solr Indexer stage transforms a Managed Fusion pipeline document into a Solr document, and sends it to Solr for indexing into a collection.To configure this stage for the index pipeline:
  1. In the Label field, enter a unique identifier for this stage or leave blank to use the default value.
  2. In the Condition field, enter a script that results in true or false, which determines if the stage should process, or leave blank.
  3. Select the Map to Solr Schema checkbox to select and add static and dynamic fields to map in this stage.
  4. Select the Add a field listing all document fields checkbox to add the _lw_fields_ss multi-valued field to the document, which lists all fields that are being sent to Solr.
  5. In the Additional Date Formats section, enter date formats to include in this stage.
  6. In the Additional Update Request Parameters section, enter the parameter names and values to update the request parameters.
  7. Select the Buffer Documents and Send Them To Solr in Batches checkbox to process the documents in batches for this stage.
  8. In the Buffer Size field, enter the number of documents in a batch before sending the batch to Solr. If no value is specified, the default value for this search cluster is used.
  9. In the Buffer Flush Interval (milliseconds) field, enter the maximum number of milliseconds to hold the batch before sending the batch to Solr. If no value is specified, the default value for this search cluster is used.
  10. Select the Allow expensive request parameters checkbox to allow commit=true and optimize=true to be passed to Solr when specified as request parameters coming into this pipeline. Document commands that specify commit or optimize are still respected even if this checkbox is not selected.
  11. Select the Unmapped Fields Mapping checkbox to specify the information for all of the fields not mapped in the previous sections.
    • In the Source Field, enter the name of the unmapped field to be mapped.
    • In the Target Field, enter the name of the Solr field to which the unmapped field is mapped.
    • In the Operation field, select how the field is mapped. The options are:
    • Add the unmapped field to the Solr field.
    • Copy the unmapped field to the Solr field and retain the value in the Source field.
    • Delete the unmapped field.
    • Keep the unmapped field and do not map it to a Solr field.
    • Move (replace) the Solr field value with the unmapped field Source value and remove the value from the Source field.
    • Set the value of the unmapped field to the value in the Solr field.
  12. Click Save.

Order the stages

For the pipeline to operate correctly, the stages must be in the following order:When you have ordered the stages, click Save.

Self-hosted Fusion

These Fusion pipeline stages provide interfaces for configuring your Lucidworks AI models in Fusion pipelines.
The LWAI Prediction index stage is a Fusion index pipeline stage that enriches your index with Generative AI predictions. It defaults to asynchronous processing, which does not block the pipeline while waiting for a response from Lucidworks AI.For reference information, see LWAI Prediction index stage.To use this stage, non-admin Fusion users must be granted the PUT,POST,GET:/LWAI-ACCOUNT-NAME/** permission in Fusion, which is the Lucidworks AI API Account Name defined in Lucidworks AI Gateway when this stage is configured.To configure this stage:
  1. Sign in to Fusion and click Indexing > Index Pipelines.
  2. Click Add+ to add a new pipeline.
  3. Enter the name in Pipeline ID.
  4. Click Add a new pipeline stage.
  5. In the AI section, click LWAI Prediction.
  6. In the Label field, enter a unique identifier for this stage.
  7. In the Condition field, enter a script that results in true or false, which determines if the stage should process.
  8. In the Account Name field, select the Lucidworks AI API account name defined in Lucidworks AI Gateway.
  9. In the Use Case field, select the Lucidworks AI use case to associate with this stage.
    • To generate a list of the use cases for your organization, see Use Case API.
    • If the Call Asynchronously? check box is selected, see available use cases described in Async Prediction API.
    • If the Call Asynchronously? check box is not selected, see available use cases described in Prediction API.
  10. In the Model field, select the Lucidworks AI model to associate with this stage. Your Fusion account name must match the name of the account that you selected in the Account Name dropdown.
    For more information about models, see:
  11. In the Input context variable variable field, enter the name of the variable in context to be used as input. Template expressions are supported.
  12. In the Destination field name and context output field, enter the name that will be used as both the field name in the document where the prediction is written and the context variable that contains the prediction.
    • If the Call Asynchronously? check box is selected and a value is entered in this field:
    • {destination name}_t is the full response.
    • In the document:
      • _lw_ai_properties_ss contains the Lucidworks account, boolean setting for async, use case, input for the call, and the collection.
      • _lw_ai_request_count is the number of GET requests by predictionId and _lw_ai_success_count is the number of responses without errors. These two fields are used for debugging only. Based on the deployment, the most useful measure is the ratio of _lw_ai_success_count divided by `_lw_ai_request_count and then adjusting as much as possible to achieve 1.0.
      • enriched_ss contains the use case. This can be used as a boolean value to verify if the use case indexed successfully.
    • If the Call Asynchronously? check box is not selected and a value is entered in this field:
    • {destination name}_t is the full response.
    • If no value is entered in this field (regardless of the Call Asynchronously? check box setting):
    • _lw_ai_{use case}_t is the response.response object, which is the raw model output.
    • _lw_ai_{use case}_response_s is the full response.
  13. In the Use Case Configuration section, click the + sign to enter the parameter name and value to send to Lucidworks AI. The useCaseConfig parameter is only applicable to certain use cases.
    • If the Call Asynchronously? check box is selected, useCaseConfig information for each applicable use case is described in Async Prediction API.
    • If the Call Asynchronously? check box is not selected, useCaseConfig information for each applicable use case is described in Prediction API.
  14. In the Model Configuration section, click the + sign to enter the parameter name and value to send to Lucidworks AI. Several modelConfig parameters are common to generative AI use cases.
    • If the Call Asynchronously? check box is selected, modelConfig information is described in Async Prediction API.
    • If the Call Asynchronously? check box is not selected, modelConfig information is described in Prediction API.
  15. In the API Key field, enter the secret value specified in the external model. For:
    • OpenAI models, "apiKey" is the value in the model’s "[OPENAI_API_KEY]" field. For more information, see Authentication API keys.
    • Azure OpenAI models, "apiKey" is the value generated by Azure in either the model’s "[KEY1 or KEY2]" field. For requirements to use Azure models, see Generative AI models.
    • Google VertexAI models, "apiKey" is the value in the model’s
    "[BASE64_ENCODED_GOOGLE_SERVICE_ACCOUNT_KEY]" field. For more information, see Create and delete Google service account keys.
  16. To run the API call asynchronously, select the Call Asynchronously? check box to specify the stage is to use the Lucidworks AI Async Prediction API endpoints. If this is selected, the API call does not block the pipeline while waiting for a response from Lucidworks AI. If the check box is not selected, the API call uses the Prediction API, which uses the pipeline until a response is received from Lucidworks AI. Performance of other API calls can be impacted.
  17. In the Maximum Asynchronous Call Tries field, enter the maximum number of times to send an asynchronous API call before the system generates a failure error.
  18. Select the Fail on Error checkbox to generate an exception if an error occurs while generating a prediction for a document.
  19. Click Save.

Additional requirements

Additional requirements to use async calls include:
  • Use a V2 connector. Only V2 connectors work for this task and not other options, such as PBL or V1 connectors.
  • Remove the Apache Tika stage from your parser because it can cause datasource failures with the following error: “The following components failed: [class com.lucidworks.connectors.service.components.job.processor.DefaultDataProcessor : Only Tika Container parser can support Async Parsing.]”
  • Replace the Solr Indexer stage with the Solr Partial Update Indexer stage with the following settings:
    • Enable Concurrency Control set to off
    • Reject Update if Solr Document is not Present set to off
    • Process All Pipeline Doc Fields set to on
    • Allow reserved fields set to on
    • A parameter with Update Type, Field Name & Value in Updates
The LWAI Prediction AI query stage is a Fusion pipeline query stage that enriches your search results with Generative AI predictions.For reference information, see LWAI Prediction query stage.To use this stage, non-admin Fusion users must be granted the PUT,POST,GET:/LWAI-ACCOUNT-NAME/** permission in Fusion, which is the Lucidworks AI API Account Name defined in Lucidworks AI Gateway when this stage is configured.To configure this stage:
  1. Sign in to Fusion and click Querying > Query Pipelines.
  2. Click Add+ to add a new pipeline.
  3. Enter the name in Pipeline ID.
  4. Click Add a new pipeline stage.
  5. In the AI section, click LWAI Prediction.
  6. In the Label field, enter a unique identifier for this stage.
  7. In the Condition field, enter a script that results in true or false, which determines if the stage should process.
  8. Select Asynchronous Execution Config if you want to run this stage asynchronously. If this field is enabled, complete the following fields:
    1. Select Enable Async Execution. Fusion automatically assigns an Async ID value to this stage. Change this to a more memorable string that describes the asynchronous stages you are merging, such as signals or access_control.
    2. Copy the Async ID value.
  9. In the Account Name field, select the name of the Lucidworks AI integration defined when the integration was created.
  10. In the Use Case field, select the Lucidworks AI use case to associate with this stage.
    • To generate a list of the use cases for your organization, see Use Case API.
    • The available use cases are described in Prediction API.
  11. In the Model field, select the Lucidworks AI model to associate with this stage.
    If you do not see any model names and you are a non-admin Fusion user, verify with a Fusion administrator that your user account has these permissions: PUT,POST,GET:/LWAI-ACCOUNT-NAME/**
    Your Fusion account name must match the name of the account that you selected in the Account Name dropdown.
    For more information about models, see:
  12. In the Input context variable variable field, enter the name of the variable in context to be used as input. Template expressions are supported.
  13. In the Destination variable name and context output field, enter the name that will be used as both the query response header in the prediction results and the context variable that contains the prediction.
    • If a value is entered in this field:
    • {destination name}_t is the full response.
    • In the context:
      • _lw_ai_properties_ss contains the Lucidworks account, boolean setting for async, use case, input for the call, and the collection.
    • If no value is entered in this field:
    • _lw_ai_{use case}_t is the response.response object, which is the raw model output.
    • _lw_ai_{use case}_response_s is the full response.
  14. Select the Include Response Documents? check box to include the response documents in the Lucidworks AI request. This option is only available for certain use cases. If this is selected, run the Solr Query stage to ensure documents exist before running the LWAI Prediction query stage.
    Response documents must be included in the RAG use case, which supports attaching a maximum of 3 response documents. To prevent errors, enter all of the entries described in the Document Field Mappings section.
  15. In the Document Field Mappings section, enter the LW AI Document field name and its corresponding Response document field name to map from input documents to the fields accepted by the Prediction API RAG use case. The fields are described in the Prediction API. If information is not entered in this section, the default mappings are used.
    • The body and source fields are required.
    • body - description_t is the contents of the document.
    • source - link_t is the URL/ID of the document.
    • The title and date fields are optional.
    • title - title_t is the title of the document.
    • date - _lw_file_modified_tdt is the creation date of the document in epoch time format.
  16. In the Use Case Configuration section, click the + sign to enter the parameter name and value to send to Lucidworks AI.
    • The useCaseConfig parameter is only applicable to certain use cases. For more information, see the Async Prediction API and the Prediction API.
    • The memoryUuid parameter is required in the Standalone Query Rewriter use case, and is optional in the RAG use case.
    For more information, see Prediction API.
  17. In the Model Configuration section, click the + sign to enter the parameter name and value to send to Lucidworks AI. Several modelConfig parameters are common to generative AI use cases. For more information, see Prediction API.
  18. In the API Key field, enter the secret value specified in the external model. For:
    • OpenAI models, "apiKey" is the value in the model’s "[OPENAI_API_KEY]" field. For more information, see Authentication API keys.
    • Azure OpenAI models, "apiKey" is the value generated by Azure in either the model’s "[KEY1 or KEY2]" field. For requirements to use Azure models, see Generative AI models.
    • Google VertexAI models, "apiKey" is the value in the model’s
    "[BASE64_ENCODED_GOOGLE_SERVICE_ACCOUNT_KEY]" field. For more information, see Create and delete Google service account keys.
  19. Select the Fail on Error checkbox to generate an exception if an error occurs during this stage.
  20. Click Save.
Solr Vector Search supports semantic search with lexical filtering. Lucidworks recommends that if you plan to use Solr SVS, you first set up Lucidworks AI to vectorize text.
The LWAI Vectorize pipeline is a default pipeline that contains the required index stages to set up vector search using Lucidworks AI.For more information,refer to Configure Neural Hybrid Search.
This feature is currently only available to clients who have contracted with Lucidworks for features related to Neural Hybrid Search and Lucidworks AI.
This feature is available starting in Fusion 5.9.5 and in all subsequent Fusion 5.9 releases.
This pipeline uses the following stages:

Configure the pipeline

To add the Lucidworks AI (LWAI) Vectorize index pipeline:
  1. Sign in to Fusion and click Indexing > Index Pipelines.
  2. Select the default LWAI-vectorize pipeline.
  3. Configure the following stages included in the default pipeline.

Field Mapping

The Field Mapping stage customizes mapping of the fields in an index pipeline document to fields in the Solr scheme.To configure this stage for the index pipeline:
  1. In the Label field, enter a unique identifier for this stage or leave blank to use the default value.
  2. In the Condition field, enter a script that results in true or false, which determines if the stage should process, or leave blank.
  3. Select the Allow System Fields Mapping? checkbox to map system fields in this stage.
  4. In the Field Retention section, enter specific fields to either keep or delete.
  5. In the Field Value Updates section, enter specific fields and then designate the value to either add to the field, or set on the field. When a value is added, any values previously on the field are retained. When a value is set, any values previously on the field are overwritten by the new value entered.
  6. In the Field Translations section, enter specific fields to either move or copy to a different field. When a field is moved, the values from the source field are moved over to the target field and the source field is removed. When a field is copied, the values from the source field are copied over to the target field and the source field is retained.
  7. Select the Unmapped Fields checkbox to specify the operation on the fields not mapped in the previous sections. Select the Keep checkbox to keep all unmapped fields. This is the only option you need to select for the LWAI-vectorize stage.
  8. Click Save.

Solr Dynamic Field Name Mapping

The Solr Dynamic Field Name Mapping stage maps pipeline document fields to Solr dynamic fields.
  1. In the Label field, enter a unique identifier for this stage or leave blank to use the default value.
  2. In the Condition field, enter a script that results in true or false, which determines if the stage should process, or leave blank.
  3. Select the Duplicate Single-Valued Fields as Multi-Valued Fields checkbox to enable indexing of field data into both single-valued and multi-valued Solr fields. For example, if this option is selected, the phone field is indexed into both the phone_s single-valued field and the phone_ss multi-valued field. If this option is not selected, the phone field is indexed into only the phone_s single-valued field.
  4. In the Field Not To Map section, enter the names of the fields that should not be mapped by this stage.
  5. Select the Text Fields Advanced Indexing checkbox to enable indexing of text data that doesn’t exceed a specific maximum length, into both tokenized and non-tokenized fields. For example, if this option is selected, the name text field with a value of John Smith is indexed into both the name_t and name_s fields allowing relevant search using name_t field (by matching to a Smith query) and also proper faceting and sorting using name_s field (using John Smith for sorting or faceting). If this option is not selected, the name text field is indexed into only the name_t text field by default.
  6. In the Max Length for Advanced Indexing of Text Fields field, enter a value used to determine how many characters of the incoming text is indexed. For example, 100.
  7. Click Save.

LWAI Vectorize Field

The LWAI Vectorize stage invokes a Lucidworks AI model to encode a string field to a vector representation. This stage is skipped if the field to encode doesn’t exist or is null on the pipeline document.
  1. In the Label field, enter a unique identifier for this stage.
  2. In the Condition field, enter a script that results in true or false, which determines if the stage should process.
  3. In the Account Name field, select the Lucidworks AI API account name defined in Lucidworks AI Gateway.
    If your account name does not appear in the list or you are unsure which one to select, check your Lucidworks AI Gateway configuration.
  4. In the Model field, select the Lucidworks AI model to use for encoding.
    If your model does not appear in the list or you are unsure which one to select, check your Lucidworks AI Gateway configuration.
    For more information, see:
  5. In the Source field, enter the name of the string field where the value should be submitted to the model for encoding. If the field is blank or does not exist, this stage is not processed. Template expressions are supported.
  6. In the Destination field, enter the name of the field where the vector value from the model response is saved.
    • If a value is entered in this field, the following information is added to the document:
    • {Destination Field}_b is the boolean value if the vector has been indexed.
    • {Destination Field} is the vector field.
  7. In the Use Case Configuration section, click the + sign to enter the parameter name and value to send to Lucidworks AI. The useCaseConfig parameter that is common to embedding use cases is dataType, but each use case may have other parameters. The value for the query stage is query.
  8. The Model Configuration section is not currently available.
  9. The Call Asynchronously? check box is not currently available.
  10. Select the Fail on Error checkbox to generate an exception if an error occurs while generating a prediction for a document.
  11. Click Save.
  12. Index data using the new pipeline. Verify the vector field is indexed by confirming the field is present in documents.

Solr Indexer

The Solr Indexer stage transforms a Fusion pipeline document into a Solr document, and sends it to Solr for indexing into a collection.To configure this stage for the index pipeline:
  1. In the Label field, enter a unique identifier for this stage or leave blank to use the default value.
  2. In the Condition field, enter a script that results in true or false, which determines if the stage should process, or leave blank.
  3. Select the Map to Solr Schema checkbox to select and add static and dynamic fields to map in this stage.
  4. Select the Add a field listing all document fields checkbox to add the _lw_fields_ss multi-valued field to the document, which lists all fields that are being sent to Solr.
  5. In the Additional Date Formats section, enter date formats to include in this stage.
  6. In the Additional Update Request Parameters section, enter the parameter names and values to update the request parameters.
  7. Select the Buffer Documents and Send Them To Solr in Batches checkbox to process the documents in batches for this stage.
  8. In the Buffer Size field, enter the number of documents in a batch before sending the batch to Solr. If no value is specified, the default value for this search cluster is used.
  9. In the Buffer Flush Interval (milliseconds) field, enter the maximum number of milliseconds to hold the batch before sending the batch to Solr. If no value is specified, the default value for this search cluster is used.
  10. Select the Allow expensive request parameters checkbox to allow commit=true and optimize=true to be passed to Solr when specified as request parameters coming into this pipeline. Document commands that specify commit or optimize are still respected even if this checkbox is not selected.
  11. Select the Unmapped Fields Mapping checkbox to specify the information for all of the fields not mapped in the previous sections.
    • In the Source Field, enter the name of the unmapped field to be mapped.
    • In the Target Field, enter the name of the Solr field to which the unmapped field is mapped.
    • In the Operation field, select how the field is mapped. The options are:
    • Add the unmapped field to the Solr field.
    • Copy the unmapped field to the Solr field and retain the value in the Source field.
    • Delete the unmapped field.
    • Keep the unmapped field and do not map it to a Solr field.
    • Move (replace) the Solr field value with the unmapped field Source value and remove the value from the Source field.
    • Set the value of the unmapped field to the value in the Solr field.
  12. Click Save.

Order the stages

For the pipeline to operate correctly, the stages must be in the following order:When you have ordered the stages, click Save.