FAQ solution - Lucidworks documentation

The Supervised solution consists of two parts.

1. Train a model

Train a deep learning model using existing query/response pairs by configuring the Smart Answers Supervised Training job. See Train a Smart Answers Supervised Model.

Train a Smart Answers Supervised Model

The Supervised solution for Smart Answers begins with training a model using your existing data and the Smart Answers Supervised Training job, as explained in this topic. The job includes an auto-tune feature that you can use instead of manually tuning the configuration.

Training job requirements

Storage150GB plus 2.5 times the total input data size.Processor and memoryThe memory requirements depend on whether you choose GPU or CPU processing:


GPU	CPU
one core 11GB RAM	32 cores 32GB RAM

If your training data contains more than 1 million entries, use GPU.

Prepare the input data

Format your input data as question/answer pairs, that is, a query and its corresponding response in each row. You can do this in any format that Managed Fusion supports. If there are multiple possible answers for a unique question, then repeat the questions and put the pair into different rows to make sure each row has one question and one answer, as in the example JSON below:
```
[{"question":"How to transfer personal auto lease to business auto lease?","answer":"I would approach the lender that you are getting the lease from..."}
 {"question":"How to transfer personal auto lease to business auto lease?","answer":"See what the contract says about transfers or subleases..."}]
```
Index the input data in Managed Fusion. If you wish to have the training data in Managed Fusion, index it into a separate collection for training data such as model_training_input. Otherwise you can use it directly from the cloud storage.

Configure the training job

In Managed Fusion, navigate to Collections > Jobs.
Select Add > Smart Answers Supervised Training:
In the Training Collection field, specify the input data collection that you created when you prepared the input data.
You can also configure this job to read from or write to cloud storage.
Enter the names of the Question Field and the Answer Field in the training collection.
Enter a Model Deployment Name. The new machine learning model will be saved in the blob store with this name. You will reference it later when you configure your pipelines.
Configure the Model base. There are several pre-trained word and BPE embeddings for different languages, as well as a few pre-trained BERT models. If you want to train custom embeddings, select word_custom or bpe_custom. This trains Word2vec on the provided data and specified fields. It might be useful in cases when your content includes unusual or domain-specific vocabulary. If you have content in addition to the query/response pairs that can be used to train the model, then specify it in the Texts Data Path. When you use the pre-trained embeddings, the log shows the percentage of processed vocabulary words. If this value is high, then try using custom embeddings. The job trains a few (configurable) RNN layers on top of word embeddings or fine-tunes a BERT model on the provided training data. The result model uses an attention mechanism to average word embeddings to obtain the final single dense vector for the content.
Dimension size of vectors for Transformer-based models is 768. For RNN-based models it is 2 times the number units of the last layer. To find the dimension size: download the model, expand the zip, open the log and search for Encoder output dim size: line. You might need this information when creating collections in Milvus.
Optional: Check Perform auto hyperparameter tuning to use auto-tune. Although training module tries to select the most optimal default parameters based on the training data statistics, auto-tune can extend it by automatically finding even better training configuration through hyper-parameter search. Although this is a resource-intensive operation, it can be useful to identify the best possible RNN-based configuration. Transformer-based models like BERT are not used during auto hyperparameter tuning as they usually perform better yet they are much more expensive on both training and inference time.
Click Save.
If using solr as the training data source ensure that the source collection contains the random_* dynamic field defined in its managed-schema.xml. This field is required for sampling the data. If it is not present, add the following entry to the managed-schema.xml alongside other dynamic fields <dynamicField name="random_*" type="random"/> and <fieldType class=“solr.RandomSortField” indexed=“true” name=“random”/> alongside other field types.
Click Run > Start.

After training is finished the model is deployed into the cluster and can be used in index and query pipelines.

Next steps

See A Smart Answers Supervised Job’s Status and Output
Configure The Smart Answers Pipelines
Evaluate a Smart Answers Query Pipeline

The job includes an auto-tuning feature that seeks to identify the optimal configuration for your RNN-based model. Transformer-based models like BERT are not used in the auto-tuning.

2. Create collections in Milvus

In order to use {app_name}-smart-answers pipelines, you need to create collections in Milvus. Please, refer to the Milvus documentation page.

3. Configure the pipelines

The trained model should be used at both index and query time in order to perform dense vector search.

At index time, we provide the {app_name}-smart-answers index pipeline to help generate a dense vector representation of answers.
At query time, we provide a {app_name}-smart-answers query pipeline to conduct run-time neural search. This pipeline transforms the incoming query into a dense vector using the trained model, then compares it with indexed answer dense vectors by computing the cosine distance between them. You can also use a query stage to combine Solr and document vector similarity scores at query time.

See Configure the Smart Answers pipelines. Once your pipelines are configured, see Evaluate a Smart Answers Query Pipeline to test its effectiveness so that you can fine-tune the configuration.

Configure the Smart Answers pipelines

Before beginning this procedure, train a machine learning model using either the FAQ method or the cold start method.Regardless of how you set up your model, the deployment procedure is the same:

Create the Milvus collection.
Configure the smart-answers index pipeline.
Configure the smart-answers query pipeline.

Create the Milvus collection

For complete details about job configuration options, see the Create Collections in Milvus job.

Navigate to Collections > Jobs > Add + and select Create Collections in Milvus.
Configure the job:
1. Enter an ID for this job.
2. Under Collections, click Add.
3. Enter a collection name.
4. In the Dimension field, enter the dimension size of vectors to store in this Milvus collection. The Dimension should match the size of the vectors returned by the encoding model. For example, the Smart Answers Pre-trained Coldstart models outputs vectors of 512 dimension size. Dimensionality of encoders trained by Smart Answers Supervised Training job depends on the provided parameters and printed in the training job logs.
Click Save. The Create Collections in Milvus job can be used to create multiple collections at once. In this image, the first collection is used in the indexing and query steps. The other two collections are used in the example.
Click Run > Start to run the job.

Configure the index pipeline

Open the Index Workbench.
Load or create your datasource using the default smart-answers index pipeline.
Configure the Encode into Milvus stage:
1. change the value of Model ID to match the model deployment name you chose when you configured the model training job.
2. Change Field to Encode to the document field name to be processed and encoded into dense vectors.
3. Ensure the Encoder Output Vector matches the output vector from the chosen model.
4. Ensure the Milvus Collection Name matches the collection name created via the Create Milvus Collection job.
  To test out your settings, turn on Fail on Error in the Encode into Milvus stage and Apply the changes. This will cause an error message to display if any settings need to be changed.
Save the datasource.
Index your data.

Configure the query pipeline

Open the Query Workbench.
Load the default smart-answers query pipeline.
Configure the Milvus Query stage:
1. Change the Model ID value to match the model deployment name you chose when you configured the model training job.
2. Ensure the Encoder Output Vector matches the output vector from the chosen model.
3. Ensure the Milvus Collection Name matches the collection name created via the Create Milvus Collection job.
4. Milvus Results Context Key can be changed as needed. It will be used in the Milvus Ensemble Query Stage to calculate the query score.
In the Milvus Ensemble Query stage, update the Ensemble math expression as needed based on your model and the name used in the prior stage for the storing the Milvus results. You can also set the Threshold so that the Milvus Ensemble Query Stage will only return items with a score greater than or equal to the configured value.
Save the query pipeline.

Pipeline Setup Example

Index and retrieve the question and answer together

To show question and answer together in one document (that is, treat the question as the title and the answer as the description), you can index them together in the same document. You can still use the default smart-answers index and query pipelines with a few additional changes.Prior to configuring the Smart Answers pipelines, use the Create Milvus Collection job to create two collections, question_collection and answer_collection, to store the encoded “questions” and the encoded “answers”, respectively.

Index Pipeline

As shown in the pictures below, you will need two Encode into Milvus stages, named Encode Question and Encode Answer respectively.Encode Question (Encode Into Milvus) stage

Pipeline setup example - Encode Question stage

Encode Answer (Encode Into Milvus) stage

Pipeline setup example - Encode Answer stage

In the Encode Question stage, specify Field to Encode to be title_t and change the Milvus Collection Name to match the new Milvus collection, question_collection.In the Encode Answer stage, specify Field to Encode to be description_t and change the Milvus Collection Name to match the new Milvus collection, answer_collection.

Query Pipeline

Since we have two dense vectors generated during indexing, at query time we need to compute both query to question distance and query to answer distance. This can be set up as the pictures shown below with two Milvus Query Stages, one for each of the two Milvus collections. To store those two distances separately, the Milvus Results Context Key needs to be different in each of these two stages.In the Query Questions stage, we set the Milvus Results Context Key to milvus_questions and the Milvus collection name to question_collection.Query Questions (Milvus Query) stage:

Pipeline setup example - Query Questions stage

In the Query Answers stage, we set the Milvus Results Context Key to milvus_answers and the Milvus collection name to answer_collection.Query Answers (Milvus Query) stage:

Pipeline setup example - Query Answers stage

Now we can ensemble them together with the Milvus Ensemble Query Stage with the Ensemble math expression combining the results from the two query stages. If we want the question scores and answer scores weighted equally, we would use: 0.5 * milvus_questions + 0.5 * milvus_answers. This is recommended especially when you have limited FAQ dataset and want to utilize both question and answer information.Milvus Ensemble Query stage

Pipeline setup example - Milvus Ensemble Query stage

Evaluate the query pipeline

The Evaluate QnA Pipeline job evaluates the rankings of results from any Smart Answers pipeline and finds the best set of weights in the ensemble score.

Detailed pipeline setup

Typically, you can use the default pipelines](/managed-fusion/5.9/mwqude/smart-answers#smart-answers-pipelines-and-stages) included with Managed Fusion. These pipelines now utilize Milvus to store encoded vectors and to calculate vector similarity. This topic provides information you can use to customize the Smart Answers pipelines. See also Configure The Smart Answers Pipelines.


”smart-answers” index pipeline		Encode into Milvus stage
”smart-answers” query pipeline		Query Fields Milvus Query Milvus Ensemble Query Milvus Response Update

Create the Milvus collection

Prior to indexing data, the Create Collections in Milvus job can be used to create the Milvus collection(s) used by the Smart Answers pipelines.

Job ID. A unique identifier for the job.
Collection Name. A name for the Milvus collection you are creating. This name is used in both the Smart Answer Index and the Smart Answer Query pipelines.
Dimension. The dimension size of the vectors to store in this Milvus collection. The Dimension should match the size of the vectors returned by the encryption model. For example, if the model was created with either the Smart Answers Coldstart Training job or the Smart Answers Supervised Training job with the Model Base word_en_300d_2M, then the dimension would be 300.
Index file size. Files with more documents than this will cause Milvus to build an index on this collection.
Metric. The type of metric used to calculate vector similarity scores. Inner Product is recommended. It produces values between 0 and 1, where a higher value means higher similarity.

Index pipeline setup

Stages in the default “smart-answers” index pipeline

Only one custom index stage needs to be configured in your index pipeline, the Encode into Milvus index stage.

The Encode into Milvus Index Stage

If you are using a dynamic schema, make sure this stage is added after the Solr Dynamic Field Name Mapping stage.

The Encode into Milvus index stage uses the specified model to encode the Field to Encode and store it in Milvus in the given Milvus collection. There are several required parameters:

Model ID. The ID of the model.
Encoder Output Vector. The name of the field that stores the compressed dense vectors output from the model. Default value: vector.
Field to Encode. The text field to encode into a dense vector, such as answer_t or body_t.
Milvus Collection Name. The name of the collection you created via the Create Milvus Collection job, which will store the dense vectors. When creating the collection you specify the type of Metric to use to calculate vector similarity. This stage can be used multiple times to encode additional fields, each into a different Milvus collection. See how to index and retrieve the question and answer together.

Query pipeline setup

The Query Fields stage

The first stage is Query Fields. For more information see the Query Fields stage.

The Milvus Query stage

The Milvus Query stage encodes the query into a vector using the specified model. It then performs a vector similarity search against the specified Milvus collection and returns a list of the best document matches.

Model ID. The ID of the model used when configuring the model training job.
Encoder Output Vector. The name of the output vector from the specified model, which will contain the query encoded as a vector. Defaults to vector.
Milvus Collection Name. The name of the collection that you used in the Encode into Milvus index stage to store the encoded vectors.
Milvus Results Context Key. The name of the variable used to store the vector distances. It can be changed as needed. It will be used in the Milvus Ensemble Query Stage to calculate the query score for the document.
Number of Results. The number of highest scoring results returned from Milvus. This stage would typically be used the same number of times that the Encode into Milvus index stage is used, each with a different Milvus collection and a different Milvus Results Context Key.

The Milvus Ensemble Query stage

The Milvus Ensemble Query takes the results of the Milvus Query stage(s) and calculates the ensemble score, which is used to return the best matches.

Ensemble math expression. The mathematical expression used to calculate the ensemble score. It should reference the value(s) variable name specified in the Milvus Results Context Key parameter in the Milvus Query stage.
Result field name. The name of the field used to store the ensemble score. It defaults to ensemble_score.
Threshold- A parameter that filters the stage results to remove items that fall below the configured score. Items with a score at, or above, the threshold will be returned.

The Milvus Response Update Query stage

The Milvus Response Update Query stage does not need to be configured and can be skipped if desired. It inserts the Milvus values, including the ensemble_score, into each of the returned documents, which is particularly useful when there is more than one Milvus Query Stage. This stage needs to come after the Solr Query stage.

Short answer extraction

By default, the question-answering query pipelines return complete documents that answer questions. Optionally, you can extract just a paragraph, a sentence, or a few words that answer the question.

Evaluate a Smart Answers Query Pipeline

Short answer extraction

By default, the question-answering query pipelines return complete documents that answer questions. Optionally, you can extract just a paragraph, a sentence, or a few words that answer the question. See Extract short answers from longer documents.

Extract short answers from longer documents

This topic explains how to deploy and configure the transformer-based deep learning model for short answer extraction with Smart Answers. This model is useful for analyzing long documents and extracting just a paragraph, a sentence, or a few words that answer the question.

This model is trained on the SQuAD2.0 dataset which consists of questions about Wikipedia articles and answers gleaned from those articles. Therefore, this model is most effective with Wikipedia-like content and may produce uneven results when applied to more informal content such as message boards.

The out-of-the-box (OOTB) model only supports English content.

Deploy the model in Fusion

Navigate to Collections > Jobs.
Select New > Create Seldon Core Model Deployment.
Configure the job as follows:
- Job ID. the ID for this job, such as deploy-answer-extractor.
- Model Name. model name of the Seldon Core deployment that will be referenced in the Machine Learning pipeline stage configurations, such as answer-extractor.
- Docker Repository. lucidworks
- Image Name. answer-extractor:v1.1
- Kubernetes Secret Name for Model Repo. (empty)
- Output Column Names for Model. [answer,score,start,end]
Click Save.
Click Run > Start.

Configure the Machine Learning query stage

This model provides the best results when used with one of the question-answering query pipelines. The default query pipeline is called APP_NAME-smart-answers.Starting with one of those pipelines, add a new Machine Learning stage to the end of the pipeline and configure it as described below.How to configure short answer extraction in the query pipeline

Make sure you have performed the basic configuration of your query pipeline.
In the query pipeline, click Add a Stage > Machine Learning.
In the Model ID field, enter the model name you configured above, such as answer-extractor.
In the Model input transformation script field, enter the following:
```
var textFieldToExtract = "answer_t"
var numDocsToExtract = 3
responses = new java.util.ArrayList();

var docs = response.get().getInnerResponse().getDocuments();
for (var i=0; i<numDocsToExtract; i++) {
  responses.add(docs[i].getField(textFieldToExtract))
}

var modelInput = new java.util.HashMap()
modelInput.put("question", request.getFirstParam("q"))
modelInput.put("context", responses)
modelInput.put("topk", 3)
modelInput.put("handle_impossible_answer", 'false')
modelInput
```
Configure the parameters in the script as follows:
- question (Required). The name of the field containing the questions.
  Make sure that the question is provided as it was originally entered by user. If you have previous stages that augments question (like stopwords removing or synonyms expansion), it is better to copy original question and use it for the answer extraction without additional modifications.
- context (Required). A string or list of contexts; by default this is the first num_docs_to_extract documents in the output of the previous stage in the pipeline. If only one question is present with multiple contexts, that question will be applied to every context and vice versa for 1 context and multiple questions. If a list of questions and contexts is passed, a 1:1 mapping of questions and contexts will be created in the order in which they’re passed.
- topk. The number of answers to return (will be chosen by order of likelihood). Default: 1
- handle_impossible_answer. Whether or not to deal with a question that has no answer in the context. If true, an empty string is returned. If false, the most probable (topk) answer(s) are returned regardless of how low the probability score is. Default: True
  Experiment with this parameter to see what value returns the most acceptable answers.
For advanced use cases, you can add the following parameters to the script to override their defaults:
- batch_size. How many samples to process at a time. Reducing this number will reduce memory usage but increase execution time, while increasing it will increase memory usage and decrease execution time to a certain extent. Default: 8
- max_context_len. If set to greater than 0, truncate contexts to this length in characters. Default: 5000
- max_answer_len. The maximum length of predicted answers (for example, only answers with a shorter length are considered). Default: 15
- max_question_len. The maximum length of the question after tokenization. It will be truncated if needed. Default: 64
- doc_stride. If the context is too long to fit with the question for the model, it will be split in several chunks with some overlap. This argument controls the size of that overlap. Default: 128
- max_seq_len. The maximum length of the total sentence (context + question) after tokenization. The context will be split in several chunks (using doc_stride) if needed. Default: 384

In the Model output transformation script field, enter the following:

{/* // Parse raw output from model */}
var jsonOutput = JSON.parse(modelOutput.get("_rawJsonResponse"))

var parsedOutput = {};
for (var i=0; i<jsonOutput["names"].length;i++){
  parsedOutput[jsonOutput["names"][i]] = jsonOutput["ndarray"][i]
}

{/* // Get response documents */}
var docs = response.get().getInnerResponse().getDocuments();
var ndocs = new java.util.ArrayList();

{/* // Add extracted answers to the response docs */}
for (var i=0; i < parsedOutput["answer"].length;i++){
  var doc = docs[i];
  doc.putField("extracted_answer", new java.util.ArrayList(parsedOutput["answer"][i]))
  doc.putField("extracted_score", new java.util.ArrayList(parsedOutput["score"][i]))
  doc.putField("extracted_start", new java.util.ArrayList(parsedOutput["start"][i]))
  doc.putField("extracted_end", new java.util.ArrayList(parsedOutput["end"][i]))
  ndocs.add(doc);
}
response.get().getInnerResponse().updateDocuments(ndocs);

Save the pipeline.

Model output

The model adds the following fields to the query pipeline output:

answer. The short answer extracted from the context. This may be blank if handle_impossible_answers=True and topk=1.
score. The score for the extracted answers.
start. The start index of the extracted answer in the provided context.
end. The end index of the extracted answer in the provided context.

UI tour

Index data

Query data

Metrics and analytics

Improve your queries

Administration

Developer documentation

Machine learning

Neural Hybrid Search

Release notes

FAQs

​1. Train a model

​2. Create collections in Milvus

​3. Configure the pipelines

​Short answer extraction

1. Train a model

2. Create collections in Milvus

3. Configure the pipelines

Short answer extraction