FAQ Solution Part 1Model training (Fusion 4.2 only)

Table of Contents

Prerequisites
Hardware requirements and instance setup
Docker image setup
Running Jobs

The FAQ solution for Smart Answers begins with training a model using your existing data, as explained in this topic. Once you have trained a model, you can deploy it.

FAQ model training

Prerequisites

You need a Docker Hub account with access to the lucidworks/ai-lab repository. Contact Lucidworks AI Labs with your Docker Hub account name to gain access to this repository.

Hardware requirements and instance setup

The Docker training module can run on either a CPU or a GPU.

CPU is sufficient for most use cases.

GPU is 2-8 times faster. GPU is recommended if want to try our model auto-tune feature on a large dataset (e.g. over 20k QA pairs) or if the training data size is over 1 million entries for any of the training scenarios.

You can run the Docker image either on-prem or in the cloud. We recommend a minimum of 32GB RAM and 32 cores for a CPU server or a cloud instance. If you choose a GPU machine, one GPU with 11GB memory is sufficient for most training use cases.

Below are examples of the AWS instance types that we recommend:

CPU:

AMI: Ubuntu Server 18.04 LTS (HVM), SSD Volume Type
Instance Type: c5.9xlarge (36 vCPUs, 72 GB RAM)
Recommended storage volume: Suggested 150GB plus 2.5 times the total input data size
Networking
- Allow inbound port 4440
- Allow inbound port 5550

GPU:

AMI: Deep Learning AMI (Ubuntu) Version 22.0
Instance Type: p2.xlarge (11.75 ECUs, 4 vCPUs, 61 GB RAM, EBS only)
Recommended storage volume: Suggested 150GB plus 2.5 times of total input data size.
Networking
- Allow inbound port 4440
- Allow inbound port 5550

Below are examples of the Azure GPU instance type that we recommend:

Deep Learning Virtual Machine
Linux OS
VM Size: 1x Standard NC6 (6 vcpus, 56 GB memory)
Recommended OS Disk Size: Suggested 150GB plus 2.5 times the total input data size
Networking
- Allow inbound port 4440
- Allow inbound port 5550

Docker image setup

Determine the instance’s external hostname.

Refer to your cloud provider’s documentation for instructions on how to do this.
Create a directory, such as /opt/faq_training, and ensure it meets disk space requirements. Also ensure that it is writable by the docker Linux user account.

The training Docker container requires this directory to read input files, write temporary files, and write the output (deep learning models).
Run docker login and enter your account credentials.
Pull the lucidworks/ai-lab:latest image:
```
docker pull lucidworks/ai_lab:latest
```

Run the Docker image:

For GPU instance:

docker run -d --rm \
		-e "RD_HOSTNAME=<external hostname here>" \
		-e "ENABLE_GPU=1" \
		--name deep_faq \
		-v <working directory here>:/src/DL/working_dir \
		--net=host \
           --runtime=nvidia \
		lucidworks/ai_lab:latest

For CPU-only instance:

docker run -d --rm \
		-e "RD_HOSTNAME=<external hostname here>" \
		--name deep_faq \
		-v <working directory here>:/src/DL/working_dir \
		--net=host \
		lucidworks/ai_lab:latest

This starts the Docker container in the background.

Wait about one minute for the container to initialize.

In your working directory, you should see these directories:
```
conf   data   input   models   output   rundeck
```
- The input directory is where all training data and input files should be placed.
- The output directory contains the models that result from the training.
- All other directories are used by the model training container to save persistent files or temporary files.
  
  You can clear the data folder periodically to save disk space.
Point your browser to http://<external hostname>:4440.

This redirects you to a Rundeck login screen.
Log in with your username and password.

Running Jobs

We use https://www.rundeck.com/open source[Rundeck^] to control job runs.

When Rundeck has started, click on the deep_faq project.

This opens the JOBS panel, where you will see three jobs for different training scenarios and one job for query pipeline evaluation.
- Click on a job to configure job parameters.
- Click Run Job Now to start the job.
- You can monitor the progress of a job by clicking on the running button under the Activity for Jobs section as shown in the screenshot below.

If similar jobs have run before, then Rundeck estimates the total run time and displays a progress bar.

Rundeck Jobs page

You can click on the running job to enter the activity control page, where you can kill the job or watch the log output.

If you want to reuse the previously-configured job, click the recent button under Activity for Jobs.

Recent Jobs

Select a recent job configuration that you want to modify, then click the button to enter the configuration page with previously-entered values.

For more details about Rundeck job control, refer to the Rundeck documentation.

Input data format

FAQ data format

The expected FAQ input is Q&A pairs, that is, a question and its corresponding answer in each row. If there are multiple possible answers for a unique question, then repeat the questions and put the Q&A pair into different rows to make sure each row has one question and one answer. Specify the question and answer field names in the Question field name and Answer field name parameters in the job configuration, respectively.

Be sure that your data has the correct JSON format such as in this example:
```
[{"question":"How to transfer personal auto lease to business auto lease?","answer":"I would approach the lender that you are getting the lease from..."} +
 {"question":"How to transfer personal auto lease to business auto lease?","answer":"See what the contract says about transfers or subleases..."}]
```
Content document input data format

If you choose to generate customized embeddings (as described below), then put the documents from which you want to learn the vocabulary into one or multiple fields and specify the field name in the Field which contains the content documents parameter.

For both methods, we only support Parquet and JSON files. Make sure your data filename ends with either .parquet or .json.

Training scenarios and important configuration parameters

There are three main scenarios supported by three different jobs in the deep_faq project:

If you have FAQ records available for training and want to use our auto-tune feature to automatically find the best model through hyper-parameter search, click on the Training with FAQ using autoTune job.

We recommend using GPU for auto-tune if the FAQ data size is over 20k. GPU training speed for auto-tune can be 2 to 6 times faster than CPU. Specify whether you have GPU using the GPU available parameter.

If you have FAQ records available for training and want to tune the model yourself, click on the Training with FAQ job.

Only one set of hyper-parameters, which are specified in the UI, will be used to run the model. If you have a big dataset but do not have GPU, you can consider running this job using default parameter settings, or run autoTune mode once to help find best parameters, then use the found parameter in this job in future model updates.
If you don’t have FAQ records and want to use our cold start solution, click on the Cold Start solution job.

This job can learn about the vocabulary that you want to search against through Word2vec. And we can combine Solr with Word2vec at query time using our query pipeline.

Input data parameters

For training on FAQ data, these parameters are required:
- QA pair file name
- Question field name
- Answer field name
For cold start solution input, these parameters are required:
- Content documents file name
- Field which contains the content documents

For each of the three scenarios above, there are two vocabulary learning options:

You can use the pre-trained word embeddings shipped with our training module.

In the FAQ solution, these pre-trained word embeddings transfer words into embeddings for initialization of the deep learning model.

In cold start mode, it gets the Word2vec vectors from the pre-trained file directly. The pre-trained word embeddings are built on a large corpus from the Internet, covering common words we use daily.
If there are many domain-specific words or special tokens in your documents, we recommend training Word2vec using your own data.

Set the Generate customized embeddings parameter to true to train from your own data. This uses the data and field specified in Content documents file name and Field which contains the content documents to train Word2vec vectors. This dataset can be the same as the FAQ input. If that’s the case, then set the Content documents file name parameter to blank. Otherwise you can specify a different dataset, especially when the FAQ dataset is small.

When you use the pre-trained embeddings, the log shows the percentage of covered vocabulary words (search for vocab coverage by embeddings in the log). If the number of covered vocabulary words is less than 80%, then you should train Word2vec from your own data.

For all training scenarios, you must provide a Model name for model tracking and version control purposes. If you use the same model name across different runs, the new model will replace the old model with the same name.

Result models

Track the running steps in the log, which provides information on data pre-processing, training steps, evaluations, and model generation in the end.

After training is finished, the final models and their associated files are saved in zip file format, which can be downloaded from http://<external hostname>:5550/{modelName}.

Download the two zipfiles ending with _a_fusion_model_bundle.zip and _q_fusion_model_bundle.zip to be used in the model deployment phase.