Product Selector

Fusion 5.9
    Fusion 5.9

    LWAI Chunker Index StageIndex pipeline stage configuration specifications

    Table of Contents

    The LWAI Chunker Index Stage asynchronously breaks down large text documents into smaller, semantically meaningful chunks, vectorizes those chunks for Neural Hybrid Search, and stores those vectors in Solr. For more information, see the Lucidworks AI Async Chunking API.

    To use this stage, non-admin Fusion users must be granted the PUT,POST,GET:/LWAI-ACCOUNT-NAME/** permission in Fusion, which is the Lucidworks AI API Account Name defined in Lucidworks AI Gateway when this stage is configured.

    Configuration

    When entering configuration values in the UI, use unescaped characters, such as \t for the tab character. When entering configuration values in the API, use escaped characters, such as \\t for the tab character.

    Pass in large text, get back smaller chunks and associated vectors.

    skip - boolean

    Set to true to skip this stage.

    Default: false

    label - string

    A unique label for this stage.

    <= 255 characters

    condition - string

    Define a conditional script that must result in true or false. This can be used to determine if the stage should process or not.

    accountName - stringrequired

    Lucidworks AI API Account Name as defined in AI Gateway Service. This entry should match the account name set in the AI Gateway.

    chunkingStrategy - stringrequired

    Chunking strategy to use

    modelName - stringrequired

    Lucidworks AI Model as defined in documentation

    inputContextVariable - stringrequired

    Name of the variable in context to be used as input. Supports template expressions.

    outputContextVariable - string

    Note: MUST contain '*_chunk_vector_*' and must be a dense vector field type. The name here is used to populate two things with the prediction results: 1) The field name in the document that will contain the prediction, and 2) The name of the context variable that will contain the prediction.

    outputTextChunks - string

    For example, body_chunks_ss. This field will contain the text chunks that are generated by the chunker.

    chunkerConfig - array[object]

    Additional Chunker keys and values to be sent to Lucidworks AI

    object attributes:{key required : {
     display name: Parameter Name
     type: string
    }    value : {
     display name: Parameter Value
     type: string
    }    }

    modelConfig - array[object]

    Additional Model configuration parameters to be sent to Lucidworks AI

    object attributes:{key required : {
     display name: Parameter Name
     type: string
    }    value : {
     display name: Parameter Value
     type: string
    }    }

    apiKey - string

    Secret associated with the model. For example, for OpenAI models, the value would start with 'sk-'.

    maxTries - integer

    The maximum number of attempts to issue an asynchronous Lucidworks AI API call. Chunking is always asynchronous.

    >= 1

    exclusiveMinimum: false

    Default: 3

    failOnError - boolean

    Flag to indicate if this stage should throw an exception if an error occurs while generating a prediction for a document.

    Default: false