Machine Learning Model APIs

​

Prerequisites

• A Managed Fusion instance with an app and indexed data
• An understanding of Python and the ability to write Python code
• Docker installed locally, plus a private or public Docker repository
• Seldon-core installed locally: pip install seldon-core
• Code editor; you can use any editor, but Visual Studio Code is used in the example
• Model: paraphrase-multilingual-MiniLM-L12-v2 from Hugging Face
• Docker image: example_sbert_model

​

Tips

• Always test your Python code locally before uploading to Docker and then Managed Fusion. This simplifies troubleshooting significantly.
• Once you’ve created your Docker you can also test locally by doing docker run with a specified port, like 9000, which you can then curl to confirm functionality in Managed Fusion. See the testing example below.

​

Local testing example

• multilingual miniLM model
• e5-model

1. Docker command:
   Copy

   docker run -p 127.0.0.1:9000:9000 <your-docker-image>
   

2. Curl to hit Docker:
   Copy

   curl -X POST -H 'Content-Type: application/json' -d '{"data": { "ndarray": ["Sentence to test"], "names":["text"]} }' https://localhost:9000/api/.0/predictions
   

3. Curl model in Managed Fusion:
   Copy

   curl -u $FUSION_USER:$FUSION_PASSWORD -X POST -H 'Content-Type: application/json' -d '{"text": "i love fusion"}' https://<your-fusion>.lucidworks.com:6764/api/ai/ml-models/<your-model>/prediction
   

4. See all your deployed models:
   Copy

   https://<your-fusion-host>/api/ai/ml-models
   

​

Download the model

​

Format a Python class

class MyModel(object):
    """
    Model template. You can load your model parameters in __init__ from a
    location accessible at runtime
    """

    def __init__(self):
        """
        Add any initialization parameters. These will be passed at runtime
        from the graph definition parameters defined in your seldondeployment
        kubernetes resource manifest.
        """
        print("Initializing")

    def predict(self,X,features_names,**kwargs):
        """
        Return a prediction.

        Parameters
        ----------
        X : array-like
        feature_names : array of feature names (optional)
        """
        print("Predict called - will run identity function")
        return X

    def  class_names(self):
        return ["X_name"]

import logging
import os

from transformers import AutoTokenizer, AutoModel
from torch.nn import functional as F
from typing import Iterable
import numpy as np
import torch

log = logging.getLogger()

class mini():
    def __init__(self):
        self.tokenizer = AutoTokenizer.from_pretrained('sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2')
        self.model= AutoModel.from_pretrained('sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2')

    #Mean Pooling
    def mean_pooling(self, model_output, attention_mask):
        token_embeddings = model_output[0] #First element of model_output contains all token embeddings
        input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
        return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)

    def predict(self, X:np.ndarray, names=None, **kwargs):
        #   In Managed Fusion there are several variables passed in the numpy array with the Milvus Query stage,
        #   Encode to Milvus index stage, and Vectorize Seldon index and query stage:
        #   [pipeline, bool, and text]. Text is what variable will be encoded, so that is what will be set to 'text'
        #   When using the Machine Learning stage, the input map keys should match what what is in this file.

        model_input = dict(zip(names, X))
        text = model_input["text"]

        with torch.inference_mode(): # Allows torch to run more quickly
          # Tokenize sentences
          encoded_input = self.tokenizer(text, padding=True, truncation=True, return_tensors='pt')
          log.debug('encoded input',str(encoded_input))
          model_output = self.model(**encoded_input)
          log.debug('model output',str(model_output))

          # Perform pooling. In this case, max pooling.
          sentence_embeddings = self.mean_pooling(model_output, encoded_input['attention_mask'])
          # Normalize embeddings, because Managed Fusion likes it that way.
          sentence_embeddings = F.normalize(sentence_embeddings, p=2, dim=-1)
          # Fixing the shape of the emebbedings to match (1, 384).
          final = [sentence_embeddings.squeeze().cpu().detach().numpy().tolist()]
        return final

    def class_names(self) -> Iterable[str]:
        return ["vector"]

• init The init function is where models, tokenizers, vectorizers, and the like should be set to self for invoking.
  It is recommended that you include your model’s trained parameters directly into the Docker container rather than reaching out to external storage inside init.
• predict The predict function processes the field or query that Managed Fusion passes to the model.
  The predict function must be able to handle any text processing needed for the model to accept input invoked in its model.evaluate(), model.predict(), or equivalent function to get the expected model result. If the output needs additional manipulation, that should be done before the result is returned.
  For embedding models the return value must have the shape of (1, DIM), where DIM (dimension) is a consistent integer, to enable Managed Fusion to handle the vector encoding into Milvus or Solr.

​

Create a Dockerfile

#It is important that python version is 3.x-slim for seldon-core
FROM python:3.9-slim
# Whatever directory(folder)the python file for your python class, Dockerfile, and
# requirements.txt is in should be copied then denoted as the work directory.
COPY . /app
WORKDIR /app

# The requirements file for the Docker container
COPY requirements.txt requirements.txt
RUN pip install --no-cache-dir -r requirements.txt

# Copy source code
COPY . .

# GRPC - Allows Managed Fusion to do a Remote Procedure Call
EXPOSE 5000

# Define environment variable for seldon-core
# !!!MODEL_NAME must be the EXACT same as the python file & python class name!!!
ENV MODEL_NAME mini
ENV SERVICE_TYPE MODEL
ENV PERSISTENCE 0

# Changing active directory folder (same one as above on lines 5 & 6) to default user, required for Managed Fusion
RUN chown -R 8888 /app

# Command to wrap python class with seldon-core to allow it to be usable in Managed Fusion
CMD ["sh", "-c", "seldon-core-microservice $MODEL_NAME --service-type $SERVICE_TYPE --persistence $PERSISTENCE"]

# You can use the following if You need shell features like environment variable expansion or
# You need to use shell constructs like pipes, redirects, etc.
# See https://docs.docker.com/reference/dockerfile/#cmd for more details.
# CMD exec seldon-core-microservice $MODEL_NAME --service-type $SERVICE_TYPE --persistence $PERSISTENCE

​

Create a requirements file

seldon-core
torch
transformers
numpy

pip freeze > requirements.txt

​

Build and push the Docker image

DOCKER_DEFAULT_PLATFORM=linux/amd64 docker build . -t [DOCKERHUB-USERNAME]/[REPOSITORY]:[VERSION-TAG]

docker push [DOCKERHUB USERNAME]/[REPOSITORY]:[VERSION-TAG]

DOCKER_DEFAULT_PLATFORM=linux/amd64 docker build . -t jstrmec/example_sbert_model:0.14; docker push jstrmec/example_sbert_model:0.14

​

Deploy the model in Managed Fusion

1. In Managed Fusion, navigate to Collections > Jobs.
2. Add a job by clicking the Add+ Button and selecting Create Seldon Core Model Deployment.
3. Fill in each of the text fields:

   Parameter	Description
   Job ID	A string used by the Managed Fusion API to reference the job after its creation.
   Model name	A name for the deployed model. This is used to generate the deployment name in Seldon Core. It is also the name that you reference as a model-id when making predictions with the ML Service.
   Model replicas	The number of load-balanced replicas of the model to deploy; specify multiple replicas for a higher-volume intake.
   Docker Repository	The public or private repository where the Docker image is located. If you’re using Docker Hub, fill in the Docker Hub username here.
   Image name	The name of the image with an optional tag. If no tag is given, latest is used.
   Kubernetes secret	If you’re using a private repository, supply the name of the Kubernetes secret used for access.
   Output columns	A list of column names that the model’s predict method returns.

4. Click Save, then Run and Start. When the job finishes successfully, you can proceed to the next section.

​

Apply an API key to the deployment

1. Create and modify a <seldon_model_name>_sdep.yaml file. In the first line, kubectl get sdep gets the details for the currently running Seldon Deployment job and saves those details to a YAML file. kubectl apply -f open_sdep.yaml adds the key to the Seldon Deployment job the next time it launches.
   Copy

   kubectl get sdep <seldon_model_name> -o yaml > <seldon_model_name>_sdep.yaml
   # Modify <seldon_model_name>_sdep.yaml to add
           - env:
             - name: API_KEY
               value: "your-api-key-here"
   kubectl apply -f <seldon_model_name>_sdep.yaml
   

2. Delete sdep before redeploying the model. The currently running Seldon Deployment job does not have the key applied to it. Delete it before redeploying and the new job will have the key.
   Copy

   kubectl delete sdep <seldon_model_name>
   

3. Lastly, you can encode into Milvus.

​

Create a Milvus collection

1. In Managed Fusion, navigate to Collections > Jobs.
2. Click the Add+ Button and select Create Collections in Milvus. This job creates a collection in Milvus for storing the vectors sent to it. The job is needed because a collection does not automatically spawn at indexing or query time if it does not already exist.
3. Name the job and the collection.
4. Click Add on the right side of the job panel. The key to creating the collection is the Dimension text field; this must exactly match the shape value your output prediction has. In our example the shape is (1,384), so 384 will be in the collections Dimension field: The Metric field should typically be left at the default of Inner Product, but this also depends on use case and model type.
5. Click Save, then Run and Start.

​

Configure the Managed Fusion pipelines

1. Create a new index pipeline or load an existing one for editing.
2. Click Add a Stage and then Encode to Milvus.

3. In the new stage, fill in these fields:
   • The name of your model
   • The output name you have for your model job
   • The field you’d like to encode
   • The collection name
4. Save the stage in the pipeline and index your data with it.

1. Create a new query pipeline or load an existing one for editing.
2. Click Add a Stage and then Milvus Query.

3. Fill in the configuration fields, then save the stage.
4. Add a Milvus Ensemble Query stage. This stage is necessary to have the Milvus collection scores taken into account in ranking and to weight multiple collections. The Milvus Results Context Key from the Milvus Query Stage is used in this stage to preform math on the Milvus result scores. One (1) is a typical multiplier for the Milvus results but any number can be used.
5. Save the stage and then run a query by typing a search term.
6. To verify the Milvus results are correct, use the Compare+ button to see another pipeline without the model implementation and compare the number of results.

​

Training job requirements

​

Prepare the input data

1. Format your input data as question/answer pairs, that is, a query and its corresponding response in each row. You can do this in any format that Managed Fusion supports. If there are multiple possible answers for a unique question, then repeat the questions and put the pair into different rows to make sure each row has one question and one answer, as in the example JSON below:
   Copy

   [{"question":"How to transfer personal auto lease to business auto lease?","answer":"I would approach the lender that you are getting the lease from..."}
    {"question":"How to transfer personal auto lease to business auto lease?","answer":"See what the contract says about transfers or subleases..."}]
   

2. Index the input data in Managed Fusion. If you wish to have the training data in Managed Fusion, index it into a separate collection for training data such as model_training_input. Otherwise you can use it directly from the cloud storage.

​

Configure the training job

1. In Managed Fusion, navigate to Collections > Jobs.
2. Select Add > Smart Answers Supervised Training:
3. In the Training Collection field, specify the input data collection that you created when you prepared the input data.
   You can also configure this job to read from or write to cloud storage.
4. Enter the names of the Question Field and the Answer Field in the training collection.
5. Enter a Model Deployment Name. The new machine learning model will be saved in the blob store with this name. You will reference it later when you configure your pipelines.
6. Configure the Model base. There are several pre-trained word and BPE embeddings for different languages, as well as a few pre-trained BERT models. If you want to train custom embeddings, select word_custom or bpe_custom. This trains Word2vec on the provided data and specified fields. It might be useful in cases when your content includes unusual or domain-specific vocabulary. If you have content in addition to the query/response pairs that can be used to train the model, then specify it in the Texts Data Path. When you use the pre-trained embeddings, the log shows the percentage of processed vocabulary words. If this value is high, then try using custom embeddings. The job trains a few (configurable) RNN layers on top of word embeddings or fine-tunes a BERT model on the provided training data. The result model uses an attention mechanism to average word embeddings to obtain the final single dense vector for the content.
   Dimension size of vectors for Transformer-based models is 768. For RNN-based models it is 2 times the number units of the last layer. To find the dimension size: download the model, expand the zip, open the log and search for Encoder output dim size: line. You might need this information when creating collections in Milvus.
7. Optional: Check Perform auto hyperparameter tuning to use auto-tune. Although training module tries to select the most optimal default parameters based on the training data statistics, auto-tune can extend it by automatically finding even better training configuration through hyper-parameter search. Although this is a resource-intensive operation, it can be useful to identify the best possible RNN-based configuration. Transformer-based models like BERT are not used during auto hyperparameter tuning as they usually perform better yet they are much more expensive on both training and inference time.
8. Click Save.
   If using solr as the training data source ensure that the source collection contains the random_* dynamic field defined in its managed-schema.xml. This field is required for sampling the data. If it is not present, add the following entry to the managed-schema.xml alongside other dynamic fields <dynamicField name="random_*" type="random"/> and <fieldType class="solr.RandomSortField" indexed="true" name="random"/> alongside other field types.
9. Click Run > Start.

​

Next steps

1. See A Smart Answers Supervised Job’s Status and Output
2. Configure The Smart Answers Pipelines
3. Evaluate a Smart Answers Query Pipeline

1. Create the Milvus collection.
2. Configure the smart-answers index pipeline.
3. Configure the smart-answers query pipeline.

​

Create the Milvus collection

1. Navigate to Collections > Jobs > Add + and select Create Collections in Milvus.
2. Configure the job:
   1. Enter an ID for this job.
   2. Under Collections, click Add.
   3. Enter a collection name.
   4. In the Dimension field, enter the dimension size of vectors to store in this Milvus collection. The Dimension should match the size of the vectors returned by the encoding model. For example, the Smart Answers Pre-trained Coldstart models outputs vectors of 512 dimension size. Dimensionality of encoders trained by Smart Answers Supervised Training job depends on the provided parameters and printed in the training job logs.
3. Click Save. The Create Collections in Milvus job can be used to create multiple collections at once. In this image, the first collection is used in the indexing and query steps. The other two collections are used in the pipeline setup example.
4. Click Run > Start to run the job.

​

Configure the index pipeline

1. Open the Index Workbench.
2. Load or create your datasource using the default smart-answers index pipeline.
3. Configure the Encode into Milvus stage:
   1. change the value of Model ID to match the model deployment name you chose when you configured the model training job.
   2. Change Field to Encode to the document field name to be processed and encoded into dense vectors.
   3. Ensure the Encoder Output Vector matches the output vector from the chosen model.
   4. Ensure the Milvus Collection Name matches the collection name created via the Create Milvus Collection job.
      To test out your settings, turn on Fail on Error in the Encode into Milvus stage and Apply the changes. This will cause an error message to display if any settings need to be changed.
      
4. Save the datasource.
5. Index your data.

​

Configure the query pipeline

1. Open the Query Workbench.
2. Load the default smart-answers query pipeline.
3. Configure the Milvus Query stage:
   1. Change the Model ID value to match the model deployment name you chose when you configured the model training job.
   2. Ensure the Encoder Output Vector matches the output vector from the chosen model.
   3. Ensure the Milvus Collection Name matches the collection name created via the Create Milvus Collection job.
   4. Milvus Results Context Key can be changed as needed. It will be used in the Milvus Ensemble Query Stage to calculate the query score.
4. In the Milvus Ensemble Query stage, update the Ensemble math expression as needed based on your model and the name used in the prior stage for the storing the Milvus results. You can also set the Threshold so that the Milvus Ensemble Query Stage will only return items with a score greater than or equal to the configured value.
5. Save the query pipeline.

​

Pipeline Setup Example

​

Index and retrieve the question and answer together

​

Index Pipeline

​

Query Pipeline

​

Evaluate the query pipeline

​

Detailed pipeline setup

”smart-answers” index pipeline		Encode into Milvus stage
”smart-answers” query pipeline		Query Fields Milvus Query Milvus Ensemble Query Milvus Response Update

​

Create the Milvus collection

• Job ID. A unique identifier for the job.
• Collection Name. A name for the Milvus collection you are creating. This name is used in both the Smart Answer Index and the Smart Answer Query pipelines.
• Dimension. The dimension size of the vectors to store in this Milvus collection. The Dimension should match the size of the vectors returned by the encryption model. For example, if the model was created with either the Smart Answers Coldstart Training job or the Smart Answers Supervised Training job with the Model Base word_en_300d_2M, then the dimension would be 300.
• Index file size. Files with more documents than this will cause Milvus to build an index on this collection.
• Metric. The type of metric used to calculate vector similarity scores. Inner Product is recommended. It produces values between 0 and 1, where a higher value means higher similarity.

​

Index pipeline setup

​

The Encode into Milvus Index Stage

• Model ID. The ID of the model.
• Encoder Output Vector. The name of the field that stores the compressed dense vectors output from the model. Default value: vector.
• Field to Encode. The text field to encode into a dense vector, such as answer_t or body_t.
• Milvus Collection Name. The name of the collection you created via the Create Milvus Collection job, which will store the dense vectors. When creating the collection you specify the type of Metric to use to calculate vector similarity. This stage can be used multiple times to encode additional fields, each into a different Milvus collection. See how to index and retrieve the question and answer together.

​

Query pipeline setup

​

The Query Fields stage

​

The Milvus Query stage

• Model ID. The ID of the model used when configuring the model training job.
• Encoder Output Vector. The name of the output vector from the specified model, which will contain the query encoded as a vector. Defaults to vector.
• Milvus Collection Name. The name of the collection that you used in the Encode into Milvus index stage to store the encoded vectors.
• Milvus Results Context Key. The name of the variable used to store the vector distances. It can be changed as needed. It will be used in the Milvus Ensemble Query Stage to calculate the query score for the document.
• Number of Results. The number of highest scoring results returned from Milvus. This stage would typically be used the same number of times that the Encode into Milvus index stage is used, each with a different Milvus collection and a different Milvus Results Context Key.

​

The Milvus Ensemble Query stage

• Ensemble math expression. The mathematical expression used to calculate the ensemble score. It should reference the value(s) variable name specified in the Milvus Results Context Key parameter in the Milvus Query stage.
• Result field name. The name of the field used to store the ensemble score. It defaults to ensemble_score.
• Threshold- A parameter that filters the stage results to remove items that fall below the configured score. Items with a score at, or above, the threshold will be returned.

​

The Milvus Response Update Query stage

​

Short answer extraction

​

Configure the training job

1. In Fusion, navigate to Collections > Jobs.
2. Select Add > Smart Answer Coldstart Training.
3. In the Training Collection field, specify the collection that contains the content that can be used to answer questions.
4. Enter the name of the Field which contains the content documents.
5. Enter a Model Deployment Name. The new machine learning model is saved in the blob store with this name. You will reference it later when you configure your pipelines.
6. Configure the Model base. There are several pre-trained word and BPE embeddings for different languages, as well as a few pre-trained BERT models. If you want to train custom embeddings, please select word_custom or bpe_custom. This trains Word2vec on the data and fields specified in Training collection and Field which contains the content documents. It might be useful in cases when your content includes unusual or domain-specific vocabulary. When you use the pre-trained embeddings, the log shows the percentage of processed vocabulary words. If this value is high, then try using custom embeddings. During the training job analyzes the content data to select weights for each of the words. The result model performs the weighted average of word embeddings to obtain final single dense vector for the content.
7. Click Save.
   If using solr as the training data source ensure that the source collection contains the random_* dynamic field defined in its managed-schema.xml. This field is required for sampling the data. If it is not present, add the following entry to the managed-schema.xml alongside other dynamic fields <dynamicField name="random_*" type="random"/> and <fieldType class="solr.RandomSortField" indexed="true" name="random"/> alongside other field types.
8. Click Run > Start.

​

Next steps

1. Configure The Smart Answers Pipelines
2. Evaluate a Smart Answers Query Pipeline

• qna-coldstart-large - this is a large model trained on variety of corpuses and tasks.
• qna-coldstart-multilingual - covers 16 languages. List of supported languages: Arabic, Chinese-simplified, Chinese-traditional, English, French, German, Italian, Japanese, Korean, Dutch, Polish, Portuguese, Spanish, Thai, Turkish, Russian.

​

Deploy a pre-trained cold-start model into Managed Fusion

1. Navigate to Collections > Jobs.
2. Select Add > Create Seldon Core Model Deployment.
3. Enter a Job ID, such as deploy-qna-coldstart-multilingual or deploy-qna-coldstart-large.
4. Enter the Model Name, one of the following:
   • qna-coldstart-multilingual
   • qna-coldstart-large
5. In the Docker Repository field, enter lucidworks.
6. In the Image Name field, enter one of the following:
   • qna-coldstart-multilingual:v1.1
   • qna-coldstart-large:v1.1
7. Leave the Kubernetes Secret Name for Model Repo field empty.
8. In the Output Column Names for Model field, enter one of the following:
   • qna-coldstart-multilingual:[vector]
   • qna-coldstart-large:[vector, compressed_vector]
9. Click Save.
10. Click Run > Start to start the deployment job.

​

Next steps

1. Evaluate a Smart Answers Query Pipeline