LWAI Vectorize Field

​

Configure vector search

​

Configure the LWAI Vectorize Field index stage

1. Sign in to Fusion and click Indexing > Index Pipelines.
2. Click the pipeline you want to use.
3. Click Add a new pipeline stage.
4. In the AI section, click LWAI Vectorize Field.
5. In the Label field, enter a unique identifier for this stage.
6. In the Condition field, enter a script that results in true or false, which determines if the stage should process.
7. Select Asynchronous Execution Config if you want to run this stage asynchronously. If this field is enabled, complete the following fields:
   1. Select Enable Async Execution. Fusion automatically assigns an Async ID value to this stage. Change this to a more memorable string that describes the asynchronous stages you are merging, such as signals or access_control.
   2. Copy the Async ID value.
      For detailed information, see Asynchronous query pipeline processing.
8. In the Account Name field, select the Lucidworks AI API account name defined in the Lucidworks AI Gateway service. If you do not see your account name, check that your Lucidworks AI Gateway integration is correctly configured.
9. In the Model field, select the Lucidworks AI model to use for encoding. If you do not see any models names and you are a non-admin Fusion user, check that you have these permissions: PUT,POST,GET:/LWAI-ACCOUNT-NAME/** Your Fusion account name must match the name of the account that you selected in the Account Name dropdown. For more information about models, see:
   • Pre-trained embedding models
   • Custom embedding model training. To use a custom model, you must obtain the deployment ID from the deployments screen.
10. In the Source field, enter the name of the string field where the value should be submitted to the model for encoding. If the field is blank or does not exist, this stage is not processed. Template expressions are supported.
11. In the Destination field, enter the name of the field where the vector value from the model response is saved.

• {Destination Field} is the vector field.
• {Destination Field}_b is the boolean value if the vector has been indexed.

11. In the Use Case Configuration section, click the + sign to enter the parameter name and value to send to Lucidworks AI. The useCaseConfig parameter that is common to embedding use cases is dataType, but each use case may have other parameters. The value for the query stage is query.
12. Optionally, you can use the Model Configuration section for any additional parameters you want to send to Lucidworks AI. Several modelConfig parameters are common to generative AI use cases. For more information, see Prediction API.
13. Select the Fail on Error checkbox to generate an exception if an error occurs while generating a prediction for a document.
14. Click Save.
15. Index data using the new pipeline. Verify the vector field is indexed by confirming the field is present in documents.

​

Configure the LWAI Vectorize Query stage

1. Sign in to Fusion and click Querying > Query Pipelines.
2. Select the pipeline you want to use.
3. Click Add a new pipeline stage.
4. Click LWAI Vectorize Query.
5. In the Label field, enter a unique identifier for this stage.
6. In the Condition field, enter a script that results in true or false, which determines if the stage should process.
7. Select Asynchronous Execution Config if you want to run this stage asynchronously. If this field is enabled, complete the following fields:
   1. Select Enable Async Execution. Fusion automatically assigns an Async ID value to this stage. Change this to a more memorable string that describes the asynchronous stages you are merging, such as signals or access_control.
   2. Copy the Async ID value.
      For detailed information, see Asynchronous query pipeline processing.
8. In the Account Name field, select the name of the Lucidworks AI account.
9. In the Model field, select the Lucidworks AI model to use for encoding. If you do not see any model names and you are a non-admin Fusion user, verify with a Fusion administrator that your user account has these permissions: PUT,POST,GET:/LWAI-ACCOUNT-NAME/** Your Fusion account name must match the name of the account that you selected in the Account Name dropdown. For more information about models, see:
   • Pre-trained embedding models
   • Custom embedding model training
10. In the Query Input field, enter the location from which the query is retrieved.
11. In the Output context variable field, enter the name of the variable where the vector value from the response is saved.
12. In the Use Case Configuration section, click the + sign to enter the parameter name and value to send to Lucidworks AI. The useCaseConfig parameter that is common to embedding use cases is dataType, but each use case may have other parameters. The value for the query stage is query.
13. Optionally, you can use the Model Configuration section for any additional parameters you want to send to Lucidworks AI. Several modelConfig parameters are common to generative AI use cases. For more information, see Prediction API.
14. Select the Fail on Error checkbox to generate an exception if an error occurs during this stage.
15. Click Save.

​

Modify Solr managed-schema (5.9.4 and earlier)

1. Go to System > Solr Config and then click managed-schema to edit it.
2. Comment out <copyField dest="\_text_" source="*"/> and add <copyField dest="text" source="*_t"/> below it. This will concatenate and index all *_t fields.
3. Add the following code block to the managed-schema file:
   Copy

   <fieldType class="solr.DenseVectorField" hnswBeamWidth=“200"
       hnswMaxConnections="45” name="knn_DIM_vector" similarityFunction="cosine"
       vectorDimension="DIM"/>
   <dynamicField docValues="false" indexed="true" multiValued="false" name="*_512v"
         required="false" stored="true" type="knn_DIM_vector"/>
   

   This example uses 512 vector dimension. If your model uses a different dimension, modify the code block to match your model. For example, _1024v. There is no limitation on supported vector dimensions.

​

Configure neural hybrid queries

​

Configure Neural Hybrid Query stage (5.9.10 and later)

1. In the same query pipeline where you configured vector search, click Add a new pipeline stage, then select Neural Hybrid Query.

2. In the Label field, enter a unique identifier for this stage or leave blank to use the default value.
3. In the Condition field, enter a script that results in true or false, which determines if the stage should process, or leave blank.
4. In the Lexical Query Input field, enter the location from which the lexical query is retrieved. For example, <request.params.q>. Template expressions are supported.
5. In the Lexical Query Weight field, enter the relative weight of the lexical query. For example, 0.3. If this value is 0, no re-ranking will be applied using the lexical query scores.
6. In the Lexical Query Squash Factor field, enter a value that will be used to squash the lexical query score. The squash factor controls how much difference there is between the top-scoring documents and the rest. It helps ensure that documents with slightly lower scores still have a chance to show up near the top. For this value, Lucidworks recommends entering the inverse of the lexical maximum score across all queries for the given collection.
7. In the Vector Query Field, enter the name of the Solr field for k-nearest neighbor (KNN) vector search.
8. In the Vector Input field, enter the location from which the vector is retrieved. Template expressions are supported. For example, a value of <ctx.vector> evaluates the context variable resulting from a previous stage, such as the LWAI Vectorize Query stage.
9. In the Vector Query Weight field, enter the relative weight of the vector query. For example, 0.7.
10. In the Min Return Vector Similarity field, enter the minimum vector similarity value to qualify as a match from the Vector portion of the hybrid query.
11. In the Min Traversal Vector Similarity field, enter the minimum vector similarity value to use when walking through the graph during the Vector portion of the hybrid query.
12. When enabled, the Compute Vector Similarity for Lexical-Only Matches setting computes vector similarity scores for documents in lexical search results but not in the initial vector search results. Select the checkbox to enable this setting.
13. If you want to use pre-filtering:
    1. Uncheck Block pre-filtering. In the Javascript context (ctx), the preFilterKey object becomes available.
    2. Add a Javascript stage after the Neural Hybrid Query stage and use it to configure your pre-filter. The preFilter object adds both the top-level fq and preFilter to the parameters for the vector query. You do not need to manually add the top level fq in the javascript stage. See the example below:
       Copy

       var QueryRequestAndResponse = Java.type('com.lucidworks.apollo.pipeline.query.QueryRequestAndResponse');
       if(ctx.hasProperty("preFilterKey")) {
         var preFilter = ctx.getProperty("preFilterKey");
         var wrapper = QueryRequestAndResponse.create(request,response,0)
         preFilter.addFilter(wrapper, 'id:* OR foo_s:bar');
       }
       

14. Click Save.

<!-- FUSION NOTES: These query parsers are used with Solr-based vector search -->
<queryParser name="xvecSim" class="org.apache.solr.lwbackported.XVecSimQParserPlugin"/>
<queryParser name="neuralHybrid" class="org.apache.solr.lw.NeuralHybridQParserPlugin"/>

​

Configure Hybrid Query stage (5.9.9 and earlier)

1. In the same query pipeline where you configured vector search, click Add a new pipeline stage, then select Hybrid Query.

2. In the Label field, enter a unique identifier for this stage or leave blank to use the default value.
3. In the Condition field, enter a script that results in true or false, which determines if the stage should process, or leave blank.
4. In the Lexical Query Input field, enter the location from which the lexical query is retrieved. For example, <request.params.q>. Template expressions are supported.
5. In the Lexical Query Weight field, enter the relative weight of the lexical query. For example, 0.3. If this value is 0, no re-ranking will be applied using the lexical query scores.
6. In the Number of Lexical Results field, enter the number of lexical search results to include in re-ranking. For example, 1000. A value is 0 is ignored.
7. In the Vector Query Field, enter the name of the Solr field for k-nearest neighbor (KNN) vector search.
8. In the Vector Input field, enter the location from which the vector is retrieved. Template expressions are supported. For example, a value of <ctx.vector> evaluates the context variable resulting from a previous stage, such as the LWAI Vectorize Query stage.
9. In the Vector Query Weight field, enter the relative weight of the vector query. For example, 0.7.
10. Select the Use KNN Query checkbox to use the knn query parser and configure its options. This option cannot be selected if Use VecSim Query checkbox is selected. In addition, Use KNN Query is used if neither Use KNN Query or Use VecSim Query is selected.
    1. If the Use KNN Query checkbox is selected, enter a value in the Number of Vector Results field. For example, 1000.
11. Select the Use VecSim Query checkbox to use the vecSim query parser and configure its options. This option cannot be selected if Use KNN Query checkbox is selected.

• Min Return Vector Similarity. Enter the minimum vector similarity value to qualify as a match from the Vector portion of the hybrid query.
• Min Traversal Vector Similarity. Enter the minimum vector similarity value to use when walking through the graph during the Vector portion of the hybrid query. The value must be lower than, or equal to, the value in the Min Return Vector Similarity field.

12. In the Minimum Vector Similarity Filter, enter the value for a minimum similarity threshold for filtering documents. This option applies to all documents, regardless of other score boosting such as rules or signals.
13. Click Save.

​

Perform hybrid searches

​

Troubleshoot inconsistent results

​

NRT replicas and HNSW graph challenges

​

Orphaning (Disconnected Nodes)

​

Identify orphaning

curl -sS -u 'USERNAME:PASSWORD' 'https://FUSION_HOST:FUSION_PORT/api/solrAdmin/default/COLLECTION_NAME/select'\
  --form-string 'fl=id,vecSim:$vecSim' \
  --form-string 'rows=1' \
  --form-string 'q=(*:* -{!knn f=VECTOR_FIELD topK=999999 v=$vec})' \
  --form-string 'vecSim=vectorSimilarity(VECTOR_FIELD,$vec)' \
  --form-string 'vec=COMPATIBLE_VECTOR'

​

Resolving orphans

1. Increase the HNSW Solr schema parameters hnswBeamWidth and hnswMaxConnections per the Suggested values below.
2. Save the schema.
3. Clear the index.
4. Re-index your collection.

Suggested values