Managed Fusion 5.9.5

​

Configure vector search

​

Configure the LWAI Vectorize Field index stage

1. Sign in to Managed Fusion and click Indexing > Index Pipelines.
2. Click the pipeline you want to use.
3. Click Add a new pipeline stage.
4. In the AI section, click LWAI Vectorize Field.

5. In the Label field, enter a unique identifier for this stage.
6. In the Condition field, enter a script that results in true or false, which determines if the stage should process.
7. In the Account Name field, select the Lucidworks AI API account name defined in Lucidworks AI Gateway. If you do not see your account name or you are unsure which one to select, contact the Managed Fusion team at Lucidworks.
8. In the Model field, select the Lucidworks AI model to use for encoding. If you do not see your model name or you are unsure which one to select, contact the Managed Fusion team at Lucidworks. For more information about models, see:
   • Pre-trained embedding models
   • Custom embedding model training. To use a custom model, you must obtain the deployment ID from the deployments screen.
9. In the Source field, enter the name of the string field where the value should be submitted to the model for encoding. If the field is blank or does not exist, this stage is not processed. Template expressions are supported.
10. In the Destination field, enter the name of the field where the vector value from the model response is saved.

• {Destination Field} is the vector field.
• {Destination Field}_b is the boolean value if the vector has been indexed.

11. In the Use Case Configuration section, click the + sign to enter the parameter name and value to send to Lucidworks AI. The useCaseConfig parameter that is common to embedding use cases is dataType, but each use case may have other parameters. The value for the query stage is query.
12. Optionally, you can use the Model Configuration section for any additional parameters you want to send to Lucidworks AI. Several modelConfig parameters are common to generative AI use cases. For more information, see Prediction API.
13. Select the Fail on Error checkbox to generate an exception if an error occurs while generating a prediction for a document.
14. Click Save.
15. Index data using the new pipeline. Verify the vector field is indexed by confirming the field is present in documents.

​

Configure the LWAI Vectorize query stage

1. Sign in to Managed Fusion and click Querying > Query Pipelines.
2. Select the pipeline you want to use.
3. Click Add a new pipeline stage.
4. Click LWAI Vectorize Query.

5. In the Label field, enter a unique identifier for this stage.
6. In the Condition field, enter a script that results in true or false, which determines if the stage should process.
7. Select Asynchronous Execution Config if you want to run this stage asynchronously. If this field is enabled, complete the following fields:
   1. Select Enable Async Execution. Fusion automatically assigns an Async ID value to this stage. Change this to a more memorable string that describes the asynchronous stages you are merging, such as signals or access_control.
   2. Copy the Async ID value.
      For detailed information, see Asynchronous query pipeline processing.
8. In the Account Name field, select the name of the Lucidworks AI account. If you do not see your account name or you are unsure which one to select, contact the Managed Fusion team at Lucidworks.
9. In the Model field, select the Lucidworks AI model to use for encoding. If you do not see any model names and you are a non-admin Fusion user, verify with a Fusion administrator that your user account has these permissions: PUT,POST,GET:/LWAI-ACCOUNT-NAME/** For more information about models, see:
   • Pre-trained embedding models
   • Custom embedding model training
10. In the Query Input field, enter the location from which the query is retrieved.
11. In the Output context variable field, enter the name of the variable where the vector value from the response is saved.
12. In the Use Case Configuration section, click the + sign to enter the parameter name and value to send to Lucidworks AI. The useCaseConfig parameter that is common to embedding use cases is dataType, but each use case may have other parameters. The value for the query stage is query.
13. Optionally, you can use the Model Configuration section for any additional parameters you want to send to Lucidworks AI. Several modelConfig parameters are common to generative AI use cases. For more information, see Prediction API.
14. Select the Fail on Error checkbox to generate an exception if an error occurs during this stage.
15. Click Save.

​

Modify Solr managed-schema (5.9.4 and earlier)

1. Go to System > Solr Config and then click managed-schema to edit it.
2. Comment out <copyField dest="\_text_" source="*"/> and add <copyField dest="text" source="*_t"/> below it. This will concatenate and index all *_t fields.
3. Add the following code block to the managed-schema file:
   Copy

   <fieldType class="solr.DenseVectorField" hnswBeamWidth=“200"
       hnswMaxConnections="45” name="knn_DIM_vector" similarityFunction="cosine"
       vectorDimension="DIM"/>
   <dynamicField docValues="false" indexed="true" multiValued="false" name="*_512v"
         required="false" stored="true" type="knn_DIM_vector"/>
   

   This example uses 512 vector dimension. If your model uses a different dimension, modify the code block to match your model. For example, _1024v. There is no limitation on supported vector dimensions.

​

Configure neural hybrid queries

​

Configure the Neural Hybrid Query stage (5.9.10 and later)

1. In the same query pipeline where you configured vector search, click Add a new pipeline stage, then select Neural Hybrid Query.

2. In the Label field, enter a unique identifier for this stage or leave blank to use the default value.
3. In the Condition field, enter a script that results in true or false, which determines if the stage should process, or leave blank.
4. In the Lexical Query Input field, enter the location from which the lexical query is retrieved. For example, <request.params.q>. Template expressions are supported.
5. In the Lexical Query Weight field, enter the relative weight of the lexical query. For example, 0.3. If this value is 0, no re-ranking will be applied using the lexical query scores.
6. In the Lexical Query Squash Factor field, enter a value that will be used to squash the lexical query score. The squash factor controls how much difference there is between the top-scoring documents and the rest. It helps ensure that documents with slightly lower scores still have a chance to show up near the top. For this value, Lucidworks recommends entering the inverse of the lexical maximum score across all queries for the given collection.
7. In the Vector Query Field, enter the name of the Solr field for k-nearest neighbor (KNN) vector search.
8. In the Vector Input field, enter the location from which the vector is retrieved. Template expressions are supported. For example, a value of <ctx.vector> evaluates the context variable resulting from a previous stage, such as the LWAI Vectorize Query stage.
9. In the Vector Query Weight field, enter the relative weight of the vector query. For example, 0.7.
10. In the Min Return Vector Similarity field, enter the minimum vector similarity value to qualify as a match from the Vector portion of the hybrid query.
11. In the Min Traversal Vector Similarity field, enter the minimum vector similarity value to use when walking through the graph during the Vector portion of the hybrid query.
12. When enabled, the Compute Vector Similarity for Lexical-Only Matches setting computes vector similarity scores for documents in lexical search results but not in the initial vector search results. Select the checkbox to enable this setting.
13. If you want to use pre-filtering:
    1. Uncheck Block pre-filtering. In the Javascript context (ctx), the preFilterKey object becomes available.
    2. Add a Javascript stage after the Neural Hybrid Query stage and use it to configure your pre-filter. The preFilter object adds both the top-level fq and preFilter to the parameters for the vector query. You do not need to manually add the top level fq in the javascript stage. See the example below:
       Copy

       var QueryRequestAndResponse = Java.type('com.lucidworks.apollo.pipeline.query. QueryRequestAndResponse');
       if(ctx.hasProperty("preFilterKey")) {
         var preFilter = ctx.getProperty("preFilterKey");
         var wrapper = QueryRequestAndResponse.create(request,response,0)
        preFilter.addFilter(wrapper, 'id:* OR foo_s:bar');
       }
       

14. Click Save.

<!-- FUSION NOTES: These query parsers are used with Solr-based vector search -->
<queryParser name="xvecSim" class="org.apache.solr.lwbackported.XVecSimQParserPlugin"/>
<queryParser name="neuralHybrid" class="org.apache.solr.lw.NeuralHybridQParserPlugin"/>

​

Configure the Hybrid Query stage (5.9.9 and earlier)

1. In the same query pipeline where you configured vector search, click Add a new pipeline stage, then select Hybrid Query.

2. In the Label field, enter a unique identifier for this stage or leave blank to use the default value.
3. In the Condition field, enter a script that results in true or false, which determines if the stage should process, or leave blank.
4. In the Lexical Query Input field, enter the location from which the lexical query is retrieved. For example, <request.params.q>. Template expressions are supported.
5. In the Lexical Query Weight field, enter the relative weight of the lexical query. For example, 0.3. If this value is 0, no re-ranking will be applied using the lexical query scores.
6. In the Number of Lexical Results field, enter the number of lexical search results to include in re-ranking. For example, 1000. A value is 0 is ignored.
7. In the Vector Query Field, enter the name of the Solr field for k-nearest neighbor (KNN) vector search.
8. In the Vector Input field, enter the location from which the vector is retrieved. Template expressions are supported. For example, a value of <ctx.vector> evaluates the context variable resulting from a previous stage, such as the LWAI Vectorize Query stage.
9. In the Vector Query Weight field, enter the relative weight of the vector query. For example, 0.7.
10. Select the Use KNN Query checkbox to use the knn query parser and configure its options. This option cannot be selected if Use VecSim Query checkbox is selected. In addition, Use KNN Query is used if neither Use KNN Query or Use VecSim Query is selected.
    1. If the Use KNN Query checkbox is selected, enter a value in the Number of Vector Results field. For example, 1000.
11. Select the Use VecSim Query checkbox to use the vecSim query parser and configure its options. This option cannot be selected if Use KNN Query checkbox is selected.
    If the Use VecSim Query checkbox is selected, enter values in the following fields:

• Min Return Vector Similarity. Enter the minimum vector similarity value to qualify as a match from the Vector portion of the hybrid query.
• Min Traversal Vector Similarity. Enter the minimum vector similarity value to use when walking through the graph during the Vector portion of the hybrid query. The value must be lower than, or equal to, the value in the Min Return Vector Similarity field.

12. In the Minimum Vector Similarity Filter, enter the value for a minimum similarity threshold for filtering documents. This option applies to all documents, regardless of other score boosting such as rules or signals.
13. Click Save.

​

Perform hybrid searches

​

Troubleshoot inconsistent results

​

NRT replicas and HNSW graph challenges

​

Orphaning (Disconnected Nodes)

​

Identify orphaning

curl -sS -u 'USERNAME:PASSWORD' 'https://EXAMPLE_COMPANY.b.lucidworks.cloud/api/solrAdmin/default/COLLECTION_NAME/select'\
  --form-string 'fl=id,vecSim:$vecSim' \
  --form-string 'rows=1' \
  --form-string 'q=(*:* -{!knn f=VECTOR_FIELD topK=999999 v=$vec})' \
  --form-string 'vecSim=vectorSimilarity(VECTOR_FIELD,$vec)' \
  --form-string 'vec=COMPATIBLE_VECTOR'

​

Resolving orphans

1. Increase the HNSW Solr schema parameters hnswBeamWidth and hnswMaxConnections per the Suggested values below.
2. Save the schema.
3. Clear the index.
4. Re-index your collection.

Suggested values

​

Overview

​

Identify the cause

2024-05-28T11:49:40.812Z - INFO  [pool-140-thread-3:org.apache.kafka.clients.consumer.internals.Fetcher@1413] - [Consumer clientId=example_Products-irdcsn, groupId=index-pipeline--example_Products--fusion.connectors.datasource-products_S3_Load] Fetch position FetchPosition{offset=6963199, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[fusion5-kafka-0.fusion5-kafka-headless.fusion5.svc.cluster.local:9092 (id: 0 rack: null)], epoch=0}} is out of range for partition fusion.connectors.datasource-products_S3_Load-2, resetting offset

​

Adjust indexing settings

​

Adjust retention parameters

1. The default value for kafka.logRetentionBytes is 1073741824 bytes (1 GB). Try increasing this value to 2147483648 bytes (2 GB) or 3221225472 (3 GB), or larger depending on the size of your documents.
   In Fusion 5.9.5, the default value is increased to 5 GB.
   You can also set this to -1 to remove the size limit. If you do this, be sure to set an appropriate limit for logRetentionHours instead.
2. The default value for kafka.logRetentionHours is 168 (7 days). If you increase kafka.logRetentionBytes by a significant amount (for example, 20 GB), you might need to decrease this setting to prevent running out of disk space. However, because older log entries are deleted when either limit is reached, you should set it high enough to ensure the data remains available until it’s no longer needed.
3. In Fusion, go to Indexing > Datasources and create a new datasource to trigger a new Kafka topic that incorporates these settings.

​

Adjust fetch settings

1. In Fusion, go to Indexing > Datasources and click your datasource.
2. Click the Advanced slider to show more settings.
3. Reduce the number of Fetch Threads.
4. Reduce the Request Page Size.
   This setting might not be available in every connector.

​

Prerequisites

• The plugin zip file and the connector-plugin-standalone JAR.
• A configured connector backend gRPC endpoint.
• Username and password of a user with a remote-connectors or admin role. This step is performed by Lucidworks.
• If the host where the remote connector is running is not configured to trust the server’s TLS certificate, Lucidworks must help configure the file path of the trust certificate collection.

​

Connector compatibility

​

System requirements

• (Managed Fusion 5.9.0-5.9.10) JVM version 11
• (Managed Fusion 5.9.11) JVM version 17
• Minimum of 2 CPUs
• 4GB Memory

​

Enable backend ingress

ingress:
  enabled: false
  pathtype: "Prefix"
  path: "/"
  #host: "ingress.example.com"
  ingressClassName: "nginx"   # Fusion 5.9.6 only
  tls:
    enabled: false
    certificateArn: ""
    # Enable the annotations field to override the default annotations
    #annotations: ""

• Set enabled to true to enable the backend ingress.
• Set pathtype to Prefix or Exact.
• Set path to the path where the backend will be available.
• Set host to the host where the backend will be available.
• In Fusion 5.9.6 only, you can set ingressClassName to one of the following:
  • nginx for Nginx Ingress Controller
  • alb for AWS Application Load Balancer (ALB)
• Configure TLS and certificates according to your CA’s procedures and policies.
  TLS must be enabled in order to use AWS ALB for ingress.

​

Connector configuration example

kafka-bridge:
  target: mynamespace-connectors-backend.lucidworkstest.com:443 # mandatory
  plain-text: false # optional, false by default.  
    proxy-server: # optional - needed when a forward proxy server is used to provide outbound access to the standalone connector
    host: host
    port: some-port
    user: user # optional
    password: password # optional
  trust: # optional - needed when the client's system doesn't trust the server's certificate
    cert-collection-filepath: path1

proxy: # mandatory fusion-proxy
  user: admin
  password: password123
  url: https://fusiontest.com/ # needed only when the connector plugin requires blob store access

plugin: # mandatory
  path: ./fs.zip
  type: #optional - the suffix is added to the connector id
    suffix: remote

​

Minimal example

kafka-bridge:
  target: mynamespace-connectors-backend.lucidworkstest.com:443

proxy:
  user: admin
  password: "password123"

plugin:
  path: ./testplugin.zip

​

Logback XML configuration file example

<configuration>
    <appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{50} [%file:%line] {%mdc} %msg%n</pattern>
        </encoder>
    </appender>
    <appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <file>./connector.log</file>
        <append>true</append>
        <rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
            <!-- rollover daily -->
            <fileNamePattern>./connector-%d{yyyy-MM-dd}.%i.log.gz</fileNamePattern>
            <maxFileSize>50MB</maxFileSize>
            <totalSizeCap>10GB</totalSizeCap>
        </rollingPolicy>
        <encoder>
            <pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{50} [%file:%line] {%mdc} %msg%n</pattern>
        </encoder>
    </appender>
    <root level="INFO">
        <appender-ref ref="CONSOLE"/>
        <appender-ref ref="FILE"/>
    </root>
</configuration>

​

Run the remote connector

java [-Dlogging.config=[LOGBACK_XML_FILE]] \
  -jar connector-plugin-client-standalone.jar [YAML_CONFIG_FILE]

​

Test communication

java -Dstandalone.connector.connectivity.test=true -jar connector-plugin-client-standalone.jar [YAML_CONFIG_FILE]

​

Encryption

​

Egress and proxy server configuration

• Host. The hosts where the proxy server is running.
• Port. The port the proxy server is listening to for communication requests.
• Credentials. Optional proxy server user and password.

​

Password encryption

1. Enter a user name and password in the connector configuration YAML.
2. Run the standalone JAR with this property:
   Copy

   -Dstandalone.connector.encrypt.password=true
   

3. Retrieve the encrypted passwords from the log that is created.
4. Replace the clear password in the configuration YAML with the encrypted password.

​

Connector restart (5.7 and earlier)

​

Recoverable bridge (5.8 and later)

​

Job expiration duration (5.9.5 only)

​

Use the remote connector

​

Enable asynchronous parsing (5.9 and later)