Product Selector

Fusion 5.12
    Fusion 5.12

    Cold Start Part 1Model Training (Fusion 4.2 only)

    The cold start solution for Smart Answers in Fusion 4.2 begins with training a model using your existing content and the Word2vec training module in the Docker container. Once you have trained a model, you can deploy it.

    Cold start model training

    See also Advanced Model Training Configuration for Smart Answers.

    Begin collecting signals after your cold-start model is deployed. Once you have accumulated a sufficient number of signals, you can train a model more accurately, using the FAQ solution.

    Prerequisites

    You need a Docker Hub account with access to the lucidworks/ai-lab repository. Contact Lucidworks AI Labs with your Docker Hub account name to gain access to this repository.

    Hardware requirements and instance setup

    The Docker training module can run on either a CPU or a GPU.

    CPU is sufficient for most use cases.

    GPU is 2-8 times faster. GPU is recommended if want to try our model auto-tune feature on a large dataset (e.g. over 20k QA pairs) or if the training data size is over 1 million entries for any of the training scenarios.

    You can run the Docker image either on-prem or in the cloud. We recommend a minimum of 32GB RAM and 32 cores for a CPU server or a cloud instance. If you choose a GPU machine, one GPU with 11GB memory is sufficient for most training use cases.

    Below are examples of the AWS instance types that we recommend:

    CPU:

    • AMI: Ubuntu Server 18.04 LTS (HVM), SSD Volume Type

    • Instance Type: c5.9xlarge (36 vCPUs, 72 GB RAM)

    • Recommended storage volume: Suggested 150GB plus 2.5 times the total input data size

    • Networking

      • Allow inbound port 4440

      • Allow inbound port 5550

    GPU:

    • AMI: Deep Learning AMI (Ubuntu) Version 22.0

    • Instance Type: p2.xlarge (11.75 ECUs, 4 vCPUs, 61 GB RAM, EBS only)

    • Recommended storage volume: Suggested 150GB plus 2.5 times of total input data size.

    • Networking

      • Allow inbound port 4440

      • Allow inbound port 5550

    Below are examples of the Azure GPU instance type that we recommend:

    • Deep Learning Virtual Machine

    • Linux OS

    • VM Size: 1x Standard NC6 (6 vcpus, 56 GB memory)

    • Recommended OS Disk Size: Suggested 150GB plus 2.5 times the total input data size

    • Networking

      • Allow inbound port 4440

      • Allow inbound port 5550

    Docker image setup

    1. Determine the instance’s external hostname.

      Refer to your cloud provider’s documentation for instructions on how to do this.

    2. Create a directory, such as /opt/faq_training, and ensure it meets disk space requirements. Also ensure that it is writable by the docker Linux user account.

      The training Docker container requires this directory to read input files, write temporary files, and write the output (deep learning models).

    3. Run docker login and enter your account credentials.

    4. Pull the lucidworks/ai-lab:latest image:

      docker pull lucidworks/ai_lab:latest

    5. Run the Docker image:

      For GPU instance:

      docker run -d --rm \
		-e "RD_HOSTNAME=<external hostname here>" \
		-e "ENABLE_GPU=1" \
		--name deep_faq \
		-v <working directory here>:/src/DL/working_dir \
		--net=host \
           --runtime=nvidia \
		lucidworks/ai_lab:latest

      For CPU-only instance:

      docker run -d --rm \
		-e "RD_HOSTNAME=<external hostname here>" \
		--name deep_faq \
		-v <working directory here>:/src/DL/working_dir \
		--net=host \
		lucidworks/ai_lab:latest

      This starts the Docker container in the background.

    6. Wait about one minute for the container to initialize.

      In your working directory, you should see these directories:

      conf   data   input   models   output   rundeck

      • The input directory is where all training data and input files should be placed.

      • The output directory contains the models that result from the training.

      • All other directories are used by the model training container to save persistent files or temporary files.

        You can clear the data folder periodically to save disk space.

    7. Point your browser to http://<external hostname>:4440.

      This redirects you to a Rundeck login screen.

    8. Log in with your username and password.

    Running Jobs

    We use Rundeck to control job runs.

    1. When Rundeck has started, click on the deep_faq project.

      This opens the JOBS panel, where you will see three jobs for different training scenarios and one job for query pipeline evaluation.

      • Click on a job to configure job parameters.

      • Click Run Job Now to start the job.

      • You can monitor the progress of a job by clicking on the running button under the Activity for Jobs section as shown in the screenshot below.

    If similar jobs have run before, then Rundeck estimates the total run time and displays a progress bar.

    Rundeck Jobs page

    You can click on the running job to enter the activity control page, where you can kill the job or watch the log output.

    If you want to reuse the previously-configured job, click the recent button under Activity for Jobs.

    Recent Jobs

    Select a recent job configuration that you want to modify, then click the button to enter the configuration page with previously-entered values.

    For more details about Rundeck job control, refer to the Rundeck documentation.

    Training scenarios and configuration

    Click on the Cold Start solution job.

    This job can learn about the vocabulary that you want to search against through Word2vec. And we can combine Solr with Word2vec at query time using our query pipeline.

    Input data parameters

    For cold start solution input, these parameters are required:

    • Content documents file name

    • Field which contains the content documents

    There are two vocabulary learning options:

    • You can use the pre-trained word embeddings included with our training module.

      In cold start mode, it gets the Word2vec vectors from the pre-trained file directly. The pre-trained word embeddings are built on a large corpus from the Internet, covering common words we use daily.

    • If there are many domain-specific words or special tokens in your documents, we recommend training Word2vec using your own data.

      Set the Generate customized embeddings parameter to "true" to train from your own data. This uses the data and field specified in Content documents file name and Field which contains the content documents to train Word2vec vectors.

    When you use the pre-trained embeddings, the log shows the percentage of covered vocabulary words (search for vocab coverage by embeddings in the log). If the number of covered vocabulary words is less than 80%, then you should train Word2vec from your own data.
    You must provide a Model name for model tracking and version control purposes. If you use the same model name across different runs, the new model will replace the old model with the same name.

    See also the Advanced Model Training Configuration for Smart Answers.

    Result models

    Track the running steps in the log, which provides information on data pre-processing, training steps, evaluations, and model generation in the end.

    After training is finished, the final models and their associated files are saved in zip file format, which can be downloaded from http://<external hostname>:5550/{modelName}.

    Download the two zipfiles ending with _a_fusion_model_bundle.zip and _q_fusion_model_bundle.zip to be used in the model deployment phase.