Deprecations and Removals

​

Prepare the package

1. Package your app into a single JAR file:
   Copy

   ./app-studio package
   

2. App Studio Enterprise includes a dockerfile. Create the App Studio Enterprise Docker image:
   Copy

   docker build PATH -t APP_NAME
   

   Set or replace APP_NAME with the name of your application. Replace PATH with the path to build from.
3. You can test the Docker image locally with the following command:
   Copy

   docker run -it -p LOCAL_PORT:8080 APP_NAME
   

   Set or replace LOCAL_PORT with the port on your local machine that can access the app. Replace APP_NAME with the ASE application name.

​

Publish the image

1. Tag your container for the registry:
   Copy

   docker tag APP_NAME gcr.io/PROJECT_NAME/APP_NAME
   

2. Push your Docker image to the Google Container Registry:
   Copy

   docker push gcr.io/PROJECT_NAME/APP_NAME
   

3. Verify the image:
   Copy

   gcloud compute instances list
   

​

Deploy the app to a cluster

1. Switch the context to your Fusion 5 cluster:
   Copy

   gcloud container clusters get-credentials CLUSTER_NAME
   

   Replace CLUSTER_NAME with your existing Fusion 5 cluster’s name.
2. Create a deployment in your cluster using the image you published:
   Copy

   kubectl create deployment APP_NAME --image=gcr.io/PROJECT_NAME/APP_NAME:latest
   

​

Create an Ingress resource

1. Use the following command to create a minimal Ingress resource:
   Copy

   cat <<EOF | kubectl apply -f -
   apiVersion: networking.k8s.io/v1beta1
   kind: Ingress
   metadata:
     name: ase-ingress
   spec:
     backend:
       serviceName: $AppName
       servicePort: 80
   EOF
   

2. Verify the Ingress resource:
   Copy

   kubectl get ingress ase-ingress
   

​

Prerequisites

• A Fusion instance with an app and indexed data
• An understanding of Python and the ability to write Python code
• Docker installed locally, plus a private or public Docker repository
• Seldon-core installed locally: pip install seldon-core
• Code editor; you can use any editor, but Visual Studio Code is used in the example
• Model: paraphrase-multilingual-MiniLM-L12-v2 from Hugging Face
• Docker image: example_sbert_model

​

Tips

• Always test your Python code locally before uploading to Docker and then Fusion. This simplifies troubleshooting significantly.
• Once you’ve created your Docker you can also test locally by doing docker run with a specified port, like 9000, which you can then curl to confirm functionality in Fusion. See the testing example below.

LucidAcademyLucidworks offers free training to help you get started.The Course for Intro to Machine Learning in Fusion focuses on using machine learning to infer the goals of customers and users in order to deliver a more sophisticated search experience:

Visit the LucidAcademy to see the full training catalog.

​

Local testing example

• multilingual miniLM model
• e5-model

1. Docker command:
   Copy

    docker run -p 127.0.0.1:9000:9000 <your-docker-image>
   

2. Curl to hit Docker:
   Copy

    curl -X POST -H 'Content-Type: application/json' -d '{"data": { "ndarray": ["Sentence to test"], "names":["text"]} }' https://localhost:9000/api/v1.0/predictions
   

3. Curl model in Fusion:
   Copy

    curl -u $FUSION_USER:$FUSION_PASSWORD -X POST -H 'Content-Type: application/json' -d '{"text": "i love fusion"}' https://<your-fusion>.lucidworks.com:6764/api/ai/ml-models/<your-model>/prediction
   

4. See all your deployed models:
   Copy

    curl -u USERNAME:PASSWORD http://FUSION_HOST:FUSION_PORT/api/ai/ml-models
   

​

Download the model

​

Format a Python class

class MyModel(object):
    """
    Model template. You can load your model parameters in __init__ from a
    location accessible at runtime
    """

    def __init__(self):
        """
        Add any initialization parameters. These will be passed at runtime
        from the graph definition parameters defined in your seldondeployment
        kubernetes resource manifest.
        """
        print("Initializing")

    def predict(self,X,features_names,**kwargs):
        """
        Return a prediction.

        Parameters
        ----------
        X : array-like
        feature_names : array of feature names (optional)
        """
        print("Predict called - will run identity function")
        return X

    def  class_names(self):
        return ["X_name"]

import logging
import os

from transformers import AutoTokenizer, AutoModel
from torch.nn import functional as F
from typing import Iterable
import numpy as np
import torch

log = logging.getLogger()

class mini():
    def __init__(self):
        self.tokenizer = AutoTokenizer.from_pretrained('sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2')
        self.model= AutoModel.from_pretrained('sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2')

    #Mean Pooling
    def mean_pooling(self, model_output, attention_mask):
        token_embeddings = model_output[0] #First element of model_output contains all token embeddings
        input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
        return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)

    def predict(self, X:np.ndarray, names=None, **kwargs):
        #   In Fusion there are several variables passed in the numpy array with the Milvus Query stage,
        #   Encode to Milvus index stage, and Vectorize Seldon index and query stage:
        #   [pipeline, bool, and text]. Text is what variable will be encoded, so that is what will be set to 'text'
        #   When using the Machine Learning stage, the input map keys should match what what is in this file.

        model_input = dict(zip(names, X))
        text = model_input["text"]

        with torch.inference_mode(): # Allows torch to run more quickly
          # Tokenize sentences
          encoded_input = self.tokenizer(text, padding=True, truncation=True, return_tensors='pt')
          log.debug('encoded input',str(encoded_input))
          model_output = self.model(**encoded_input)
          log.debug('model output',str(model_output))

          # Perform pooling. In this case, max pooling.
          sentence_embeddings = self.mean_pooling(model_output, encoded_input['attention_mask'])
          # Normalize embeddings, because Fusion likes it that way.
          sentence_embeddings = F.normalize(sentence_embeddings, p=2, dim=-1)
          # Fixing the shape of the emebbedings to match (1, 384).
          final = [sentence_embeddings.squeeze().cpu().detach().numpy().tolist()]
        return final

    def class_names(self) -> Iterable[str]:
        return ["vector"]

• init: The init function is where models, tokenizers, vectorizers, and the like should be set to self for invoking.
  It is recommended that you include your model’s trained parameters directly into the Docker container rather than reaching out to external storage inside init.
• predict: The predict function processes the field or query that Fusion passes to the model.
  The predict function must be able to handle any text processing needed for the model to accept input invoked in its model.evaluate(), model.predict(), or equivalent function to get the expected model result. If the output needs additional manipulation, that should be done before the result is returned.
  For embedding models the return value must have the shape of (1, DIM), where DIM (dimension) is a consistent integer, to enable Fusion to handle the vector encoding into Milvus or Solr.

​

Create a Dockerfile

#It is important that python version is 3.x-slim for seldon-core
FROM python:3.9-slim
# Whatever directory(folder)the python file for your python class, Dockerfile, and
# requirements.txt is in should be copied then denoted as the work directory.
COPY . /app
WORKDIR /app

# The requirements file for the Docker container
COPY requirements.txt requirements.txt
RUN pip install --no-cache-dir -r requirements.txt

# Copy source code
COPY . .

# GRPC - Allows Fusion to do a Remote Procedure Call
EXPOSE 5000

# Define environment variable for seldon-core
# !!!MODEL_NAME must be the EXACT same as the python file & python class name!!!
ENV MODEL_NAME mini
ENV SERVICE_TYPE MODEL
ENV PERSISTENCE 0

# Changing active directory folder (same one as above on lines 5 & 6) to default user, required for Fusion
RUN chown -R 8888 /app

# Command to wrap python class with seldon-core to allow it to be usable in Fusion
CMD ["sh", "-c", "seldon-core-microservice $MODEL_NAME --service-type $SERVICE_TYPE --persistence $PERSISTENCE"]

# You can use the following if You need shell features like environment variable expansion or
# You need to use shell constructs like pipes, redirects, etc.
# See https://docs.docker.com/reference/dockerfile/#cmd for more details.
# CMD exec seldon-core-microservice $MODEL_NAME --service-type $SERVICE_TYPE --persistence $PERSISTENCE

​

Create a requirements file

seldon-core
torch
transformers
numpy

pip freeze > requirements.txt

​

Build and push the Docker image

DOCKER_DEFAULT_PLATFORM=linux/amd64 docker build . -t [DOCKERHUB-USERNAME]/[REPOSITORY]:[VERSION-TAG]

docker push [DOCKERHUB USERNAME]/[REPOSITORY]:[VERSION-TAG]

DOCKER_DEFAULT_PLATFORM=linux/amd64 docker build . -t jstrmec/example_sbert_model:0.14; docker push jstrmec/example_sbert_model:0.14

​

Deploy the model in Fusion

1. In Fusion, navigate to Collections > Jobs.
2. Add a job by clicking the Add+ Button and selecting Create Seldon Core Model Deployment.
3. Fill in each of the text fields:

   Parameter	Description
   Job ID	A string used by the Fusion API to reference the job after its creation.
   Model name	A name for the deployed model. This is used to generate the deployment name in Seldon Core. It is also the name that you reference as a model-id when making predictions with the ML Service.
   Model replicas	The number of load-balanced replicas of the model to deploy; specify multiple replicas for a higher-volume intake.
   Docker Repository	The public or private repository where the Docker image is located. If you’re using Docker Hub, fill in the Docker Hub username here.
   Image name	The name of the image with an optional tag. If no tag is given, latest is used.
   Kubernetes secret	If you’re using a private repository, supply the name of the Kubernetes secret used for access.
   Output columns	A list of column names that the model’s predict method returns.

4. Click Save, then Run and Start. When the job finishes successfully, you can proceed to the next section.

​

Apply an API key to the deployment

1. Create and modify a <seldon_model_name>_sdep.yaml file.
   In the first line, kubectl get sdep gets the details for the currently running Seldon Deployment job and saves those details to a YAML file. kubectl apply -f open_sdep.yaml adds the key to the Seldon Deployment job the next time it launches.
   Copy

   kubectl get sdep <seldon_model_name> -o yaml > <seldon_model_name>_sdep.yaml
   # Modify <seldon_model_name>_sdep.yaml to add
          - env:
            - name: API_KEY
              value: "your-api-key-here"
   kubectl apply -f <seldon_model_name>_sdep.yaml
   

2. Delete sdep before redeploying the model. The currently running Seldon Deployment job does not have the key applied to it. Delete it before redeploying and the new job will have the key.
   Copy

   kubectl delete sdep <seldon_model_name>
   

3. Lastly, you can encode into Milvus.

​

Create a Milvus collection

1. In Fusion, navigate to Collections > Jobs.
2. Click the Add+ Button and select Create Collections in Milvus.
   This job creates a collection in Milvus for storing the vectors sent to it. The job is needed because a collection does not automatically spawn at indexing or query time if it does not already exist.
3. Name the job and the collection.
4. Click Add on the right side of the job panel.
   The key to creating the collection is the Dimension text field; this must exactly match the shape value your output prediction has.
   In our example the shape is (1,384), so 384 will be in the collections Dimension field: The Metric field should typically be left at the default of Inner Product, but this also depends on use case and model type.
5. Click Save, then Run and Start.

​

Configure the Fusion pipelines

1. Create a new index pipeline or load an existing one for editing.
2. Click Add a Stage and then Encode to Milvus.
3. In the new stage, fill in these fields:
   • The name of your model
   • The output name you have for your model job
   • The field you’d like to encode
   • The collection name
4. Save the stage in the pipeline and index your data with it.

1. Create a new query pipeline or load an existing one for editing.
2. Click Add a Stage and then Milvus Query.
3. Fill in the configuration fields, then save the stage.
4. Add a Milvus Ensemble Query stage.
   This stage is necessary to have the Milvus collection scores taken into account in ranking and to weight multiple collections. The Milvus Results Context Key from the Milvus Query Stage is used in this stage to preform math on the Milvus result scores. One (1) is a typical multiplier for the Milvus results but any number can be used.
5. Save the stage and then run a query by typing a search term.
6. To verify the Milvus results are correct, use the Compare+ button to see another pipeline without the model implementation and compare the number of results.

​

Prerequisites

• A Fusion instance with an app and indexed data
• An understanding of Python and the ability to write Python code
• Docker installed locally, plus a private or public Docker repository
• Seldon-core installed locally: pip install seldon-core
• Code editor; you can use any editor, but Visual Studio Code is used in the example
• Model: paraphrase-multilingual-MiniLM-L12-v2 from Hugging Face
• Docker image: example_sbert_model

​

Tips

• Always test your Python code locally before uploading to Docker and then Fusion. This simplifies troubleshooting significantly.
• Once you’ve created your Docker you can also test locally by doing docker run with a specified port, like 9000, which you can then curl to confirm functionality in Fusion. See the testing example below.

LucidAcademyLucidworks offers free training to help you get started.The Course for Intro to Machine Learning in Fusion focuses on using machine learning to infer the goals of customers and users in order to deliver a more sophisticated search experience:

Visit the LucidAcademy to see the full training catalog.

​

Local testing example

• multilingual miniLM model
• e5-model

1. Docker command:
   Copy

    docker run -p 127.0.0.1:9000:9000 <your-docker-image>
   

2. Curl to hit Docker:
   Copy

    curl -X POST -H 'Content-Type: application/json' -d '{"data": { "ndarray": ["Sentence to test"], "names":["text"]} }' https://localhost:9000/api/v1.0/predictions
   

3. Curl model in Fusion:
   Copy

    curl -u $FUSION_USER:$FUSION_PASSWORD -X POST -H 'Content-Type: application/json' -d '{"text": "i love fusion"}' https://<your-fusion>.lucidworks.com:6764/api/ai/ml-models/<your-model>/prediction
   

4. See all your deployed models:
   Copy

    curl -u USERNAME:PASSWORD http://FUSION_HOST:FUSION_PORT/api/ai/ml-models
   

​

Download the model

​

Format a Python class

class MyModel(object):
    """
    Model template. You can load your model parameters in __init__ from a
    location accessible at runtime
    """

    def __init__(self):
        """
        Add any initialization parameters. These will be passed at runtime
        from the graph definition parameters defined in your seldondeployment
        kubernetes resource manifest.
        """
        print("Initializing")

    def predict(self,X,features_names,**kwargs):
        """
        Return a prediction.

        Parameters
        ----------
        X : array-like
        feature_names : array of feature names (optional)
        """
        print("Predict called - will run identity function")
        return X

    def  class_names(self):
        return ["X_name"]

import logging
import os

from transformers import AutoTokenizer, AutoModel
from torch.nn import functional as F
from typing import Iterable
import numpy as np
import torch

log = logging.getLogger()

class mini():
    def __init__(self):
        self.tokenizer = AutoTokenizer.from_pretrained('sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2')
        self.model= AutoModel.from_pretrained('sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2')

    #Mean Pooling
    def mean_pooling(self, model_output, attention_mask):
        token_embeddings = model_output[0] #First element of model_output contains all token embeddings
        input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
        return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)

    def predict(self, X:np.ndarray, names=None, **kwargs):
        #   In Fusion there are several variables passed in the numpy array with the Milvus Query stage,
        #   Encode to Milvus index stage, and Vectorize Seldon index and query stage:
        #   [pipeline, bool, and text]. Text is what variable will be encoded, so that is what will be set to 'text'
        #   When using the Machine Learning stage, the input map keys should match what what is in this file.

        model_input = dict(zip(names, X))
        text = model_input["text"]

        with torch.inference_mode(): # Allows torch to run more quickly
          # Tokenize sentences
          encoded_input = self.tokenizer(text, padding=True, truncation=True, return_tensors='pt')
          log.debug('encoded input',str(encoded_input))
          model_output = self.model(**encoded_input)
          log.debug('model output',str(model_output))

          # Perform pooling. In this case, max pooling.
          sentence_embeddings = self.mean_pooling(model_output, encoded_input['attention_mask'])
          # Normalize embeddings, because Fusion likes it that way.
          sentence_embeddings = F.normalize(sentence_embeddings, p=2, dim=-1)
          # Fixing the shape of the emebbedings to match (1, 384).
          final = [sentence_embeddings.squeeze().cpu().detach().numpy().tolist()]
        return final

    def class_names(self) -> Iterable[str]:
        return ["vector"]

• init: The init function is where models, tokenizers, vectorizers, and the like should be set to self for invoking.
  It is recommended that you include your model’s trained parameters directly into the Docker container rather than reaching out to external storage inside init.
• predict: The predict function processes the field or query that Fusion passes to the model.
  The predict function must be able to handle any text processing needed for the model to accept input invoked in its model.evaluate(), model.predict(), or equivalent function to get the expected model result. If the output needs additional manipulation, that should be done before the result is returned.
  For embedding models the return value must have the shape of (1, DIM), where DIM (dimension) is a consistent integer, to enable Fusion to handle the vector encoding into Milvus or Solr.

​

Create a Dockerfile

#It is important that python version is 3.x-slim for seldon-core
FROM python:3.9-slim
# Whatever directory(folder)the python file for your python class, Dockerfile, and
# requirements.txt is in should be copied then denoted as the work directory.
COPY . /app
WORKDIR /app

# The requirements file for the Docker container
COPY requirements.txt requirements.txt
RUN pip install --no-cache-dir -r requirements.txt

# Copy source code
COPY . .

# GRPC - Allows Fusion to do a Remote Procedure Call
EXPOSE 5000

# Define environment variable for seldon-core
# !!!MODEL_NAME must be the EXACT same as the python file & python class name!!!
ENV MODEL_NAME mini
ENV SERVICE_TYPE MODEL
ENV PERSISTENCE 0

# Changing active directory folder (same one as above on lines 5 & 6) to default user, required for Fusion
RUN chown -R 8888 /app

# Command to wrap python class with seldon-core to allow it to be usable in Fusion
CMD ["sh", "-c", "seldon-core-microservice $MODEL_NAME --service-type $SERVICE_TYPE --persistence $PERSISTENCE"]

# You can use the following if You need shell features like environment variable expansion or
# You need to use shell constructs like pipes, redirects, etc.
# See https://docs.docker.com/reference/dockerfile/#cmd for more details.
# CMD exec seldon-core-microservice $MODEL_NAME --service-type $SERVICE_TYPE --persistence $PERSISTENCE

​

Create a requirements file

seldon-core
torch
transformers
numpy

pip freeze > requirements.txt

​

Build and push the Docker image

DOCKER_DEFAULT_PLATFORM=linux/amd64 docker build . -t [DOCKERHUB-USERNAME]/[REPOSITORY]:[VERSION-TAG]

docker push [DOCKERHUB USERNAME]/[REPOSITORY]:[VERSION-TAG]

DOCKER_DEFAULT_PLATFORM=linux/amd64 docker build . -t jstrmec/example_sbert_model:0.14; docker push jstrmec/example_sbert_model:0.14

​

Deploy the model in Fusion

1. In Fusion, navigate to Collections > Jobs.
2. Add a job by clicking the Add+ Button and selecting Create Seldon Core Model Deployment.
3. Fill in each of the text fields:

   Parameter	Description
   Job ID	A string used by the Fusion API to reference the job after its creation.
   Model name	A name for the deployed model. This is used to generate the deployment name in Seldon Core. It is also the name that you reference as a model-id when making predictions with the ML Service.
   Model replicas	The number of load-balanced replicas of the model to deploy; specify multiple replicas for a higher-volume intake.
   Docker Repository	The public or private repository where the Docker image is located. If you’re using Docker Hub, fill in the Docker Hub username here.
   Image name	The name of the image with an optional tag. If no tag is given, latest is used.
   Kubernetes secret	If you’re using a private repository, supply the name of the Kubernetes secret used for access.
   Output columns	A list of column names that the model’s predict method returns.

4. Click Save, then Run and Start. When the job finishes successfully, you can proceed to the next section.

​

Apply an API key to the deployment

1. Create and modify a <seldon_model_name>_sdep.yaml file.
   In the first line, kubectl get sdep gets the details for the currently running Seldon Deployment job and saves those details to a YAML file. kubectl apply -f open_sdep.yaml adds the key to the Seldon Deployment job the next time it launches.
   Copy

   kubectl get sdep <seldon_model_name> -o yaml > <seldon_model_name>_sdep.yaml
   # Modify <seldon_model_name>_sdep.yaml to add
          - env:
            - name: API_KEY
              value: "your-api-key-here"
   kubectl apply -f <seldon_model_name>_sdep.yaml
   

2. Delete sdep before redeploying the model. The currently running Seldon Deployment job does not have the key applied to it. Delete it before redeploying and the new job will have the key.
   Copy

   kubectl delete sdep <seldon_model_name>
   

3. Lastly, you can encode into Milvus.

​

Create a Milvus collection

1. In Fusion, navigate to Collections > Jobs.
2. Click the Add+ Button and select Create Collections in Milvus.
   This job creates a collection in Milvus for storing the vectors sent to it. The job is needed because a collection does not automatically spawn at indexing or query time if it does not already exist.
3. Name the job and the collection.
4. Click Add on the right side of the job panel.
   The key to creating the collection is the Dimension text field; this must exactly match the shape value your output prediction has.
   In our example the shape is (1,384), so 384 will be in the collections Dimension field: The Metric field should typically be left at the default of Inner Product, but this also depends on use case and model type.
5. Click Save, then Run and Start.

​

Configure the Fusion pipelines

1. Create a new index pipeline or load an existing one for editing.
2. Click Add a Stage and then Encode to Milvus.
3. In the new stage, fill in these fields:
   • The name of your model
   • The output name you have for your model job
   • The field you’d like to encode
   • The collection name
4. Save the stage in the pipeline and index your data with it.

1. Create a new query pipeline or load an existing one for editing.
2. Click Add a Stage and then Milvus Query.
3. Fill in the configuration fields, then save the stage.
4. Add a Milvus Ensemble Query stage.
   This stage is necessary to have the Milvus collection scores taken into account in ranking and to weight multiple collections. The Milvus Results Context Key from the Milvus Query Stage is used in this stage to preform math on the Milvus result scores. One (1) is a typical multiplier for the Milvus results but any number can be used.
5. Save the stage and then run a query by typing a search term.
6. To verify the Milvus results are correct, use the Compare+ button to see another pipeline without the model implementation and compare the number of results.

​

Prerequisites

• A Fusion instance with an app and indexed data
• An understanding of Python and the ability to write Python code
• Docker installed locally, plus a private or public Docker repository
• Seldon-core installed locally: pip install seldon-core
• Code editor; you can use any editor, but Visual Studio Code is used in the example
• Model: paraphrase-multilingual-MiniLM-L12-v2 from Hugging Face
• Docker image: example_sbert_model

​

Tips

• Always test your Python code locally before uploading to Docker and then Fusion. This simplifies troubleshooting significantly.
• Once you’ve created your Docker you can also test locally by doing docker run with a specified port, like 9000, which you can then curl to confirm functionality in Fusion. See the testing example below.

LucidAcademyLucidworks offers free training to help you get started.The Course for Intro to Machine Learning in Fusion focuses on using machine learning to infer the goals of customers and users in order to deliver a more sophisticated search experience:

Visit the LucidAcademy to see the full training catalog.

​

Local testing example

• multilingual miniLM model
• e5-model

1. Docker command:
   Copy

    docker run -p 127.0.0.1:9000:9000 <your-docker-image>
   

2. Curl to hit Docker:
   Copy

    curl -X POST -H 'Content-Type: application/json' -d '{"data": { "ndarray": ["Sentence to test"], "names":["text"]} }' https://localhost:9000/api/v1.0/predictions
   

3. Curl model in Fusion:
   Copy

    curl -u $FUSION_USER:$FUSION_PASSWORD -X POST -H 'Content-Type: application/json' -d '{"text": "i love fusion"}' https://<your-fusion>.lucidworks.com:6764/api/ai/ml-models/<your-model>/prediction
   

4. See all your deployed models:
   Copy

    curl -u USERNAME:PASSWORD http://FUSION_HOST:FUSION_PORT/api/ai/ml-models
   

​

Download the model

​

Format a Python class

class MyModel(object):
    """
    Model template. You can load your model parameters in __init__ from a
    location accessible at runtime
    """

    def __init__(self):
        """
        Add any initialization parameters. These will be passed at runtime
        from the graph definition parameters defined in your seldondeployment
        kubernetes resource manifest.
        """
        print("Initializing")

    def predict(self,X,features_names,**kwargs):
        """
        Return a prediction.

        Parameters
        ----------
        X : array-like
        feature_names : array of feature names (optional)
        """
        print("Predict called - will run identity function")
        return X

    def  class_names(self):
        return ["X_name"]

import logging
import os

from transformers import AutoTokenizer, AutoModel
from torch.nn import functional as F
from typing import Iterable
import numpy as np
import torch

log = logging.getLogger()

class mini():
    def __init__(self):
        self.tokenizer = AutoTokenizer.from_pretrained('sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2')
        self.model= AutoModel.from_pretrained('sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2')

    #Mean Pooling
    def mean_pooling(self, model_output, attention_mask):
        token_embeddings = model_output[0] #First element of model_output contains all token embeddings
        input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
        return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)

    def predict(self, X:np.ndarray, names=None, **kwargs):
        #   In Fusion there are several variables passed in the numpy array with the Milvus Query stage,
        #   Encode to Milvus index stage, and Vectorize Seldon index and query stage:
        #   [pipeline, bool, and text]. Text is what variable will be encoded, so that is what will be set to 'text'
        #   When using the Machine Learning stage, the input map keys should match what what is in this file.

        model_input = dict(zip(names, X))
        text = model_input["text"]

        with torch.inference_mode(): # Allows torch to run more quickly
          # Tokenize sentences
          encoded_input = self.tokenizer(text, padding=True, truncation=True, return_tensors='pt')
          log.debug('encoded input',str(encoded_input))
          model_output = self.model(**encoded_input)
          log.debug('model output',str(model_output))

          # Perform pooling. In this case, max pooling.
          sentence_embeddings = self.mean_pooling(model_output, encoded_input['attention_mask'])
          # Normalize embeddings, because Fusion likes it that way.
          sentence_embeddings = F.normalize(sentence_embeddings, p=2, dim=-1)
          # Fixing the shape of the emebbedings to match (1, 384).
          final = [sentence_embeddings.squeeze().cpu().detach().numpy().tolist()]
        return final

    def class_names(self) -> Iterable[str]:
        return ["vector"]

• init: The init function is where models, tokenizers, vectorizers, and the like should be set to self for invoking.
  It is recommended that you include your model’s trained parameters directly into the Docker container rather than reaching out to external storage inside init.
• predict: The predict function processes the field or query that Fusion passes to the model.
  The predict function must be able to handle any text processing needed for the model to accept input invoked in its model.evaluate(), model.predict(), or equivalent function to get the expected model result. If the output needs additional manipulation, that should be done before the result is returned.
  For embedding models the return value must have the shape of (1, DIM), where DIM (dimension) is a consistent integer, to enable Fusion to handle the vector encoding into Milvus or Solr.

​

Create a Dockerfile

#It is important that python version is 3.x-slim for seldon-core
FROM python:3.9-slim
# Whatever directory(folder)the python file for your python class, Dockerfile, and
# requirements.txt is in should be copied then denoted as the work directory.
COPY . /app
WORKDIR /app

# The requirements file for the Docker container
COPY requirements.txt requirements.txt
RUN pip install --no-cache-dir -r requirements.txt

# Copy source code
COPY . .

# GRPC - Allows Fusion to do a Remote Procedure Call
EXPOSE 5000

# Define environment variable for seldon-core
# !!!MODEL_NAME must be the EXACT same as the python file & python class name!!!
ENV MODEL_NAME mini
ENV SERVICE_TYPE MODEL
ENV PERSISTENCE 0

# Changing active directory folder (same one as above on lines 5 & 6) to default user, required for Fusion
RUN chown -R 8888 /app

# Command to wrap python class with seldon-core to allow it to be usable in Fusion
CMD ["sh", "-c", "seldon-core-microservice $MODEL_NAME --service-type $SERVICE_TYPE --persistence $PERSISTENCE"]

# You can use the following if You need shell features like environment variable expansion or
# You need to use shell constructs like pipes, redirects, etc.
# See https://docs.docker.com/reference/dockerfile/#cmd for more details.
# CMD exec seldon-core-microservice $MODEL_NAME --service-type $SERVICE_TYPE --persistence $PERSISTENCE

​

Create a requirements file

seldon-core
torch
transformers
numpy

pip freeze > requirements.txt

​

Build and push the Docker image

DOCKER_DEFAULT_PLATFORM=linux/amd64 docker build . -t [DOCKERHUB-USERNAME]/[REPOSITORY]:[VERSION-TAG]

docker push [DOCKERHUB USERNAME]/[REPOSITORY]:[VERSION-TAG]

DOCKER_DEFAULT_PLATFORM=linux/amd64 docker build . -t jstrmec/example_sbert_model:0.14; docker push jstrmec/example_sbert_model:0.14

​

Deploy the model in Fusion

1. In Fusion, navigate to Collections > Jobs.
2. Add a job by clicking the Add+ Button and selecting Create Seldon Core Model Deployment.
3. Fill in each of the text fields:

   Parameter	Description
   Job ID	A string used by the Fusion API to reference the job after its creation.
   Model name	A name for the deployed model. This is used to generate the deployment name in Seldon Core. It is also the name that you reference as a model-id when making predictions with the ML Service.
   Model replicas	The number of load-balanced replicas of the model to deploy; specify multiple replicas for a higher-volume intake.
   Docker Repository	The public or private repository where the Docker image is located. If you’re using Docker Hub, fill in the Docker Hub username here.
   Image name	The name of the image with an optional tag. If no tag is given, latest is used.
   Kubernetes secret	If you’re using a private repository, supply the name of the Kubernetes secret used for access.
   Output columns	A list of column names that the model’s predict method returns.

4. Click Save, then Run and Start. When the job finishes successfully, you can proceed to the next section.

​

Apply an API key to the deployment

1. Create and modify a <seldon_model_name>_sdep.yaml file.
   In the first line, kubectl get sdep gets the details for the currently running Seldon Deployment job and saves those details to a YAML file. kubectl apply -f open_sdep.yaml adds the key to the Seldon Deployment job the next time it launches.
   Copy

   kubectl get sdep <seldon_model_name> -o yaml > <seldon_model_name>_sdep.yaml
   # Modify <seldon_model_name>_sdep.yaml to add
          - env:
            - name: API_KEY
              value: "your-api-key-here"
   kubectl apply -f <seldon_model_name>_sdep.yaml
   

2. Delete sdep before redeploying the model. The currently running Seldon Deployment job does not have the key applied to it. Delete it before redeploying and the new job will have the key.
   Copy

   kubectl delete sdep <seldon_model_name>
   

3. Lastly, you can encode into Milvus.

​

Create a Milvus collection

1. In Fusion, navigate to Collections > Jobs.
2. Click the Add+ Button and select Create Collections in Milvus.
   This job creates a collection in Milvus for storing the vectors sent to it. The job is needed because a collection does not automatically spawn at indexing or query time if it does not already exist.
3. Name the job and the collection.
4. Click Add on the right side of the job panel.
   The key to creating the collection is the Dimension text field; this must exactly match the shape value your output prediction has.
   In our example the shape is (1,384), so 384 will be in the collections Dimension field: The Metric field should typically be left at the default of Inner Product, but this also depends on use case and model type.
5. Click Save, then Run and Start.

​

Configure the Fusion pipelines

1. Create a new index pipeline or load an existing one for editing.
2. Click Add a Stage and then Encode to Milvus.
3. In the new stage, fill in these fields:
   • The name of your model
   • The output name you have for your model job
   • The field you’d like to encode
   • The collection name
4. Save the stage in the pipeline and index your data with it.

1. Create a new query pipeline or load an existing one for editing.
2. Click Add a Stage and then Milvus Query.
3. Fill in the configuration fields, then save the stage.
4. Add a Milvus Ensemble Query stage.
   This stage is necessary to have the Milvus collection scores taken into account in ranking and to weight multiple collections. The Milvus Results Context Key from the Milvus Query Stage is used in this stage to preform math on the Milvus result scores. One (1) is a typical multiplier for the Milvus results but any number can be used.
5. Save the stage and then run a query by typing a search term.
6. To verify the Milvus results are correct, use the Compare+ button to see another pipeline without the model implementation and compare the number of results.

​

Prerequisites

• A Fusion instance with an app and indexed data
• An understanding of Python and the ability to write Python code
• Docker installed locally, plus a private or public Docker repository
• Seldon-core installed locally: pip install seldon-core
• Code editor; you can use any editor, but Visual Studio Code is used in the example
• Model: paraphrase-multilingual-MiniLM-L12-v2 from Hugging Face
• Docker image: example_sbert_model

​

Tips

• Always test your Python code locally before uploading to Docker and then Fusion. This simplifies troubleshooting significantly.
• Once you’ve created your Docker you can also test locally by doing docker run with a specified port, like 9000, which you can then curl to confirm functionality in Fusion. See the testing example below.

LucidAcademyLucidworks offers free training to help you get started.The Course for Intro to Machine Learning in Fusion focuses on using machine learning to infer the goals of customers and users in order to deliver a more sophisticated search experience:

Visit the LucidAcademy to see the full training catalog.

​

Local testing example

• multilingual miniLM model
• e5-model

1. Docker command:
   Copy

    docker run -p 127.0.0.1:9000:9000 <your-docker-image>
   

2. Curl to hit Docker:
   Copy

    curl -X POST -H 'Content-Type: application/json' -d '{"data": { "ndarray": ["Sentence to test"], "names":["text"]} }' https://localhost:9000/api/v1.0/predictions
   

3. Curl model in Fusion:
   Copy

    curl -u $FUSION_USER:$FUSION_PASSWORD -X POST -H 'Content-Type: application/json' -d '{"text": "i love fusion"}' https://<your-fusion>.lucidworks.com:6764/api/ai/ml-models/<your-model>/prediction
   

4. See all your deployed models:
   Copy

    curl -u USERNAME:PASSWORD http://FUSION_HOST:FUSION_PORT/api/ai/ml-models
   

​

Download the model

​

Format a Python class

class MyModel(object):
    """
    Model template. You can load your model parameters in __init__ from a
    location accessible at runtime
    """

    def __init__(self):
        """
        Add any initialization parameters. These will be passed at runtime
        from the graph definition parameters defined in your seldondeployment
        kubernetes resource manifest.
        """
        print("Initializing")

    def predict(self,X,features_names,**kwargs):
        """
        Return a prediction.

        Parameters
        ----------
        X : array-like
        feature_names : array of feature names (optional)
        """
        print("Predict called - will run identity function")
        return X

    def  class_names(self):
        return ["X_name"]

import logging
import os

from transformers import AutoTokenizer, AutoModel
from torch.nn import functional as F
from typing import Iterable
import numpy as np
import torch

log = logging.getLogger()

class mini():
    def __init__(self):
        self.tokenizer = AutoTokenizer.from_pretrained('sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2')
        self.model= AutoModel.from_pretrained('sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2')

    #Mean Pooling
    def mean_pooling(self, model_output, attention_mask):
        token_embeddings = model_output[0] #First element of model_output contains all token embeddings
        input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
        return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)

    def predict(self, X:np.ndarray, names=None, **kwargs):
        #   In Fusion there are several variables passed in the numpy array with the Milvus Query stage,
        #   Encode to Milvus index stage, and Vectorize Seldon index and query stage:
        #   [pipeline, bool, and text]. Text is what variable will be encoded, so that is what will be set to 'text'
        #   When using the Machine Learning stage, the input map keys should match what what is in this file.

        model_input = dict(zip(names, X))
        text = model_input["text"]

        with torch.inference_mode(): # Allows torch to run more quickly
          # Tokenize sentences
          encoded_input = self.tokenizer(text, padding=True, truncation=True, return_tensors='pt')
          log.debug('encoded input',str(encoded_input))
          model_output = self.model(**encoded_input)
          log.debug('model output',str(model_output))

          # Perform pooling. In this case, max pooling.
          sentence_embeddings = self.mean_pooling(model_output, encoded_input['attention_mask'])
          # Normalize embeddings, because Fusion likes it that way.
          sentence_embeddings = F.normalize(sentence_embeddings, p=2, dim=-1)
          # Fixing the shape of the emebbedings to match (1, 384).
          final = [sentence_embeddings.squeeze().cpu().detach().numpy().tolist()]
        return final

    def class_names(self) -> Iterable[str]:
        return ["vector"]

• init: The init function is where models, tokenizers, vectorizers, and the like should be set to self for invoking.
  It is recommended that you include your model’s trained parameters directly into the Docker container rather than reaching out to external storage inside init.
• predict: The predict function processes the field or query that Fusion passes to the model.
  The predict function must be able to handle any text processing needed for the model to accept input invoked in its model.evaluate(), model.predict(), or equivalent function to get the expected model result. If the output needs additional manipulation, that should be done before the result is returned.
  For embedding models the return value must have the shape of (1, DIM), where DIM (dimension) is a consistent integer, to enable Fusion to handle the vector encoding into Milvus or Solr.

​

Create a Dockerfile

#It is important that python version is 3.x-slim for seldon-core
FROM python:3.9-slim
# Whatever directory(folder)the python file for your python class, Dockerfile, and
# requirements.txt is in should be copied then denoted as the work directory.
COPY . /app
WORKDIR /app

# The requirements file for the Docker container
COPY requirements.txt requirements.txt
RUN pip install --no-cache-dir -r requirements.txt

# Copy source code
COPY . .

# GRPC - Allows Fusion to do a Remote Procedure Call
EXPOSE 5000

# Define environment variable for seldon-core
# !!!MODEL_NAME must be the EXACT same as the python file & python class name!!!
ENV MODEL_NAME mini
ENV SERVICE_TYPE MODEL
ENV PERSISTENCE 0

# Changing active directory folder (same one as above on lines 5 & 6) to default user, required for Fusion
RUN chown -R 8888 /app

# Command to wrap python class with seldon-core to allow it to be usable in Fusion
CMD ["sh", "-c", "seldon-core-microservice $MODEL_NAME --service-type $SERVICE_TYPE --persistence $PERSISTENCE"]

# You can use the following if You need shell features like environment variable expansion or
# You need to use shell constructs like pipes, redirects, etc.
# See https://docs.docker.com/reference/dockerfile/#cmd for more details.
# CMD exec seldon-core-microservice $MODEL_NAME --service-type $SERVICE_TYPE --persistence $PERSISTENCE

​

Create a requirements file

seldon-core
torch
transformers
numpy

pip freeze > requirements.txt

​

Build and push the Docker image

DOCKER_DEFAULT_PLATFORM=linux/amd64 docker build . -t [DOCKERHUB-USERNAME]/[REPOSITORY]:[VERSION-TAG]

docker push [DOCKERHUB USERNAME]/[REPOSITORY]:[VERSION-TAG]

DOCKER_DEFAULT_PLATFORM=linux/amd64 docker build . -t jstrmec/example_sbert_model:0.14; docker push jstrmec/example_sbert_model:0.14

​

Deploy the model in Fusion

1. In Fusion, navigate to Collections > Jobs.
2. Add a job by clicking the Add+ Button and selecting Create Seldon Core Model Deployment.
3. Fill in each of the text fields:

   Parameter	Description
   Job ID	A string used by the Fusion API to reference the job after its creation.
   Model name	A name for the deployed model. This is used to generate the deployment name in Seldon Core. It is also the name that you reference as a model-id when making predictions with the ML Service.
   Model replicas	The number of load-balanced replicas of the model to deploy; specify multiple replicas for a higher-volume intake.
   Docker Repository	The public or private repository where the Docker image is located. If you’re using Docker Hub, fill in the Docker Hub username here.
   Image name	The name of the image with an optional tag. If no tag is given, latest is used.
   Kubernetes secret	If you’re using a private repository, supply the name of the Kubernetes secret used for access.
   Output columns	A list of column names that the model’s predict method returns.

4. Click Save, then Run and Start. When the job finishes successfully, you can proceed to the next section.

​

Apply an API key to the deployment

1. Create and modify a <seldon_model_name>_sdep.yaml file.
   In the first line, kubectl get sdep gets the details for the currently running Seldon Deployment job and saves those details to a YAML file. kubectl apply -f open_sdep.yaml adds the key to the Seldon Deployment job the next time it launches.
   Copy

   kubectl get sdep <seldon_model_name> -o yaml > <seldon_model_name>_sdep.yaml
   # Modify <seldon_model_name>_sdep.yaml to add
          - env:
            - name: API_KEY
              value: "your-api-key-here"
   kubectl apply -f <seldon_model_name>_sdep.yaml
   

2. Delete sdep before redeploying the model. The currently running Seldon Deployment job does not have the key applied to it. Delete it before redeploying and the new job will have the key.
   Copy

   kubectl delete sdep <seldon_model_name>
   

3. Lastly, you can encode into Milvus.

​

Create a Milvus collection

1. In Fusion, navigate to Collections > Jobs.
2. Click the Add+ Button and select Create Collections in Milvus.
   This job creates a collection in Milvus for storing the vectors sent to it. The job is needed because a collection does not automatically spawn at indexing or query time if it does not already exist.
3. Name the job and the collection.
4. Click Add on the right side of the job panel.
   The key to creating the collection is the Dimension text field; this must exactly match the shape value your output prediction has.
   In our example the shape is (1,384), so 384 will be in the collections Dimension field: The Metric field should typically be left at the default of Inner Product, but this also depends on use case and model type.
5. Click Save, then Run and Start.

​

Configure the Fusion pipelines

1. Create a new index pipeline or load an existing one for editing.
2. Click Add a Stage and then Encode to Milvus.
3. In the new stage, fill in these fields:
   • The name of your model
   • The output name you have for your model job
   • The field you’d like to encode
   • The collection name
4. Save the stage in the pipeline and index your data with it.

1. Create a new query pipeline or load an existing one for editing.
2. Click Add a Stage and then Milvus Query.
3. Fill in the configuration fields, then save the stage.
4. Add a Milvus Ensemble Query stage.
   This stage is necessary to have the Milvus collection scores taken into account in ranking and to weight multiple collections. The Milvus Results Context Key from the Milvus Query Stage is used in this stage to preform math on the Milvus result scores. One (1) is a typical multiplier for the Milvus results but any number can be used.
5. Save the stage and then run a query by typing a search term.
6. To verify the Milvus results are correct, use the Compare+ button to see another pipeline without the model implementation and compare the number of results.

​

Prerequisites

• A Fusion instance with an app and indexed data
• An understanding of Python and the ability to write Python code
• Docker installed locally, plus a private or public Docker repository
• Seldon-core installed locally: pip install seldon-core
• Code editor; you can use any editor, but Visual Studio Code is used in the example
• Model: paraphrase-multilingual-MiniLM-L12-v2 from Hugging Face
• Docker image: example_sbert_model

​

Tips

• Always test your Python code locally before uploading to Docker and then Fusion. This simplifies troubleshooting significantly.
• Once you’ve created your Docker you can also test locally by doing docker run with a specified port, like 9000, which you can then curl to confirm functionality in Fusion. See the testing example below.

LucidAcademyLucidworks offers free training to help you get started.The Course for Intro to Machine Learning in Fusion focuses on using machine learning to infer the goals of customers and users in order to deliver a more sophisticated search experience:

Visit the LucidAcademy to see the full training catalog.

​

Local testing example

• multilingual miniLM model
• e5-model

1. Docker command:
   Copy

    docker run -p 127.0.0.1:9000:9000 <your-docker-image>
   

2. Curl to hit Docker:
   Copy

    curl -X POST -H 'Content-Type: application/json' -d '{"data": { "ndarray": ["Sentence to test"], "names":["text"]} }' https://localhost:9000/api/v1.0/predictions
   

3. Curl model in Fusion:
   Copy

    curl -u $FUSION_USER:$FUSION_PASSWORD -X POST -H 'Content-Type: application/json' -d '{"text": "i love fusion"}' https://<your-fusion>.lucidworks.com:6764/api/ai/ml-models/<your-model>/prediction
   

4. See all your deployed models:
   Copy

    curl -u USERNAME:PASSWORD http://FUSION_HOST:FUSION_PORT/api/ai/ml-models
   

​

Download the model

​

Format a Python class

class MyModel(object):
    """
    Model template. You can load your model parameters in __init__ from a
    location accessible at runtime
    """

    def __init__(self):
        """
        Add any initialization parameters. These will be passed at runtime
        from the graph definition parameters defined in your seldondeployment
        kubernetes resource manifest.
        """
        print("Initializing")

    def predict(self,X,features_names,**kwargs):
        """
        Return a prediction.

        Parameters
        ----------
        X : array-like
        feature_names : array of feature names (optional)
        """
        print("Predict called - will run identity function")
        return X

    def  class_names(self):
        return ["X_name"]

import logging
import os

from transformers import AutoTokenizer, AutoModel
from torch.nn import functional as F
from typing import Iterable
import numpy as np
import torch

log = logging.getLogger()

class mini():
    def __init__(self):
        self.tokenizer = AutoTokenizer.from_pretrained('sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2')
        self.model= AutoModel.from_pretrained('sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2')

    #Mean Pooling
    def mean_pooling(self, model_output, attention_mask):
        token_embeddings = model_output[0] #First element of model_output contains all token embeddings
        input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
        return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)

    def predict(self, X:np.ndarray, names=None, **kwargs):
        #   In Fusion there are several variables passed in the numpy array with the Milvus Query stage,
        #   Encode to Milvus index stage, and Vectorize Seldon index and query stage:
        #   [pipeline, bool, and text]. Text is what variable will be encoded, so that is what will be set to 'text'
        #   When using the Machine Learning stage, the input map keys should match what what is in this file.

        model_input = dict(zip(names, X))
        text = model_input["text"]

        with torch.inference_mode(): # Allows torch to run more quickly
          # Tokenize sentences
          encoded_input = self.tokenizer(text, padding=True, truncation=True, return_tensors='pt')
          log.debug('encoded input',str(encoded_input))
          model_output = self.model(**encoded_input)
          log.debug('model output',str(model_output))

          # Perform pooling. In this case, max pooling.
          sentence_embeddings = self.mean_pooling(model_output, encoded_input['attention_mask'])
          # Normalize embeddings, because Fusion likes it that way.
          sentence_embeddings = F.normalize(sentence_embeddings, p=2, dim=-1)
          # Fixing the shape of the emebbedings to match (1, 384).
          final = [sentence_embeddings.squeeze().cpu().detach().numpy().tolist()]
        return final

    def class_names(self) -> Iterable[str]:
        return ["vector"]

• init: The init function is where models, tokenizers, vectorizers, and the like should be set to self for invoking.
  It is recommended that you include your model’s trained parameters directly into the Docker container rather than reaching out to external storage inside init.
• predict: The predict function processes the field or query that Fusion passes to the model.
  The predict function must be able to handle any text processing needed for the model to accept input invoked in its model.evaluate(), model.predict(), or equivalent function to get the expected model result. If the output needs additional manipulation, that should be done before the result is returned.
  For embedding models the return value must have the shape of (1, DIM), where DIM (dimension) is a consistent integer, to enable Fusion to handle the vector encoding into Milvus or Solr.

​

Create a Dockerfile

#It is important that python version is 3.x-slim for seldon-core
FROM python:3.9-slim
# Whatever directory(folder)the python file for your python class, Dockerfile, and
# requirements.txt is in should be copied then denoted as the work directory.
COPY . /app
WORKDIR /app

# The requirements file for the Docker container
COPY requirements.txt requirements.txt
RUN pip install --no-cache-dir -r requirements.txt

# Copy source code
COPY . .

# GRPC - Allows Fusion to do a Remote Procedure Call
EXPOSE 5000

# Define environment variable for seldon-core
# !!!MODEL_NAME must be the EXACT same as the python file & python class name!!!
ENV MODEL_NAME mini
ENV SERVICE_TYPE MODEL
ENV PERSISTENCE 0

# Changing active directory folder (same one as above on lines 5 & 6) to default user, required for Fusion
RUN chown -R 8888 /app

# Command to wrap python class with seldon-core to allow it to be usable in Fusion
CMD ["sh", "-c", "seldon-core-microservice $MODEL_NAME --service-type $SERVICE_TYPE --persistence $PERSISTENCE"]

# You can use the following if You need shell features like environment variable expansion or
# You need to use shell constructs like pipes, redirects, etc.
# See https://docs.docker.com/reference/dockerfile/#cmd for more details.
# CMD exec seldon-core-microservice $MODEL_NAME --service-type $SERVICE_TYPE --persistence $PERSISTENCE

​

Create a requirements file

seldon-core
torch
transformers
numpy

pip freeze > requirements.txt

​

Build and push the Docker image

DOCKER_DEFAULT_PLATFORM=linux/amd64 docker build . -t [DOCKERHUB-USERNAME]/[REPOSITORY]:[VERSION-TAG]

docker push [DOCKERHUB USERNAME]/[REPOSITORY]:[VERSION-TAG]

DOCKER_DEFAULT_PLATFORM=linux/amd64 docker build . -t jstrmec/example_sbert_model:0.14; docker push jstrmec/example_sbert_model:0.14

​

Deploy the model in Fusion

1. In Fusion, navigate to Collections > Jobs.
2. Add a job by clicking the Add+ Button and selecting Create Seldon Core Model Deployment.
3. Fill in each of the text fields:

   Parameter	Description
   Job ID	A string used by the Fusion API to reference the job after its creation.
   Model name	A name for the deployed model. This is used to generate the deployment name in Seldon Core. It is also the name that you reference as a model-id when making predictions with the ML Service.
   Model replicas	The number of load-balanced replicas of the model to deploy; specify multiple replicas for a higher-volume intake.
   Docker Repository	The public or private repository where the Docker image is located. If you’re using Docker Hub, fill in the Docker Hub username here.
   Image name	The name of the image with an optional tag. If no tag is given, latest is used.
   Kubernetes secret	If you’re using a private repository, supply the name of the Kubernetes secret used for access.
   Output columns	A list of column names that the model’s predict method returns.

4. Click Save, then Run and Start. When the job finishes successfully, you can proceed to the next section.

​

Apply an API key to the deployment

1. Create and modify a <seldon_model_name>_sdep.yaml file.
   In the first line, kubectl get sdep gets the details for the currently running Seldon Deployment job and saves those details to a YAML file. kubectl apply -f open_sdep.yaml adds the key to the Seldon Deployment job the next time it launches.
   Copy

   kubectl get sdep <seldon_model_name> -o yaml > <seldon_model_name>_sdep.yaml
   # Modify <seldon_model_name>_sdep.yaml to add
          - env:
            - name: API_KEY
              value: "your-api-key-here"
   kubectl apply -f <seldon_model_name>_sdep.yaml
   

2. Delete sdep before redeploying the model. The currently running Seldon Deployment job does not have the key applied to it. Delete it before redeploying and the new job will have the key.
   Copy

   kubectl delete sdep <seldon_model_name>
   

3. Lastly, you can encode into Milvus.

​

Create a Milvus collection

1. In Fusion, navigate to Collections > Jobs.
2. Click the Add+ Button and select Create Collections in Milvus.
   This job creates a collection in Milvus for storing the vectors sent to it. The job is needed because a collection does not automatically spawn at indexing or query time if it does not already exist.
3. Name the job and the collection.
4. Click Add on the right side of the job panel.
   The key to creating the collection is the Dimension text field; this must exactly match the shape value your output prediction has.
   In our example the shape is (1,384), so 384 will be in the collections Dimension field: The Metric field should typically be left at the default of Inner Product, but this also depends on use case and model type.
5. Click Save, then Run and Start.

​

Configure the Fusion pipelines

1. Create a new index pipeline or load an existing one for editing.
2. Click Add a Stage and then Encode to Milvus.
3. In the new stage, fill in these fields:
   • The name of your model
   • The output name you have for your model job
   • The field you’d like to encode
   • The collection name
4. Save the stage in the pipeline and index your data with it.

1. Create a new query pipeline or load an existing one for editing.
2. Click Add a Stage and then Milvus Query.
3. Fill in the configuration fields, then save the stage.
4. Add a Milvus Ensemble Query stage.
   This stage is necessary to have the Milvus collection scores taken into account in ranking and to weight multiple collections. The Milvus Results Context Key from the Milvus Query Stage is used in this stage to preform math on the Milvus result scores. One (1) is a typical multiplier for the Milvus results but any number can be used.
5. Save the stage and then run a query by typing a search term.
6. To verify the Milvus results are correct, use the Compare+ button to see another pipeline without the model implementation and compare the number of results.

​

Prerequisites

• A Fusion instance with an app and indexed data
• An understanding of Python and the ability to write Python code
• Docker installed locally, plus a private or public Docker repository
• Seldon-core installed locally: pip install seldon-core
• Code editor; you can use any editor, but Visual Studio Code is used in the example
• Model: paraphrase-multilingual-MiniLM-L12-v2 from Hugging Face
• Docker image: example_sbert_model

​

Tips

• Always test your Python code locally before uploading to Docker and then Fusion. This simplifies troubleshooting significantly.
• Once you’ve created your Docker you can also test locally by doing docker run with a specified port, like 9000, which you can then curl to confirm functionality in Fusion. See the testing example below.

LucidAcademyLucidworks offers free training to help you get started.The Course for Intro to Machine Learning in Fusion focuses on using machine learning to infer the goals of customers and users in order to deliver a more sophisticated search experience:

Visit the LucidAcademy to see the full training catalog.

​

Local testing example

• multilingual miniLM model
• e5-model

1. Docker command:
   Copy

    docker run -p 127.0.0.1:9000:9000 <your-docker-image>
   

2. Curl to hit Docker:
   Copy

    curl -X POST -H 'Content-Type: application/json' -d '{"data": { "ndarray": ["Sentence to test"], "names":["text"]} }' https://localhost:9000/api/v1.0/predictions
   

3. Curl model in Fusion:
   Copy

    curl -u $FUSION_USER:$FUSION_PASSWORD -X POST -H 'Content-Type: application/json' -d '{"text": "i love fusion"}' https://<your-fusion>.lucidworks.com:6764/api/ai/ml-models/<your-model>/prediction
   

4. See all your deployed models:
   Copy

    curl -u USERNAME:PASSWORD http://FUSION_HOST:FUSION_PORT/api/ai/ml-models
   

​

Download the model

​

Format a Python class

class MyModel(object):
    """
    Model template. You can load your model parameters in __init__ from a
    location accessible at runtime
    """

    def __init__(self):
        """
        Add any initialization parameters. These will be passed at runtime
        from the graph definition parameters defined in your seldondeployment
        kubernetes resource manifest.
        """
        print("Initializing")

    def predict(self,X,features_names,**kwargs):
        """
        Return a prediction.

        Parameters
        ----------
        X : array-like
        feature_names : array of feature names (optional)
        """
        print("Predict called - will run identity function")
        return X

    def  class_names(self):
        return ["X_name"]

import logging
import os

from transformers import AutoTokenizer, AutoModel
from torch.nn import functional as F
from typing import Iterable
import numpy as np
import torch

log = logging.getLogger()

class mini():
    def __init__(self):
        self.tokenizer = AutoTokenizer.from_pretrained('sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2')
        self.model= AutoModel.from_pretrained('sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2')

    #Mean Pooling
    def mean_pooling(self, model_output, attention_mask):
        token_embeddings = model_output[0] #First element of model_output contains all token embeddings
        input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
        return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)

    def predict(self, X:np.ndarray, names=None, **kwargs):
        #   In Fusion there are several variables passed in the numpy array with the Milvus Query stage,
        #   Encode to Milvus index stage, and Vectorize Seldon index and query stage:
        #   [pipeline, bool, and text]. Text is what variable will be encoded, so that is what will be set to 'text'
        #   When using the Machine Learning stage, the input map keys should match what what is in this file.

        model_input = dict(zip(names, X))
        text = model_input["text"]

        with torch.inference_mode(): # Allows torch to run more quickly
          # Tokenize sentences
          encoded_input = self.tokenizer(text, padding=True, truncation=True, return_tensors='pt')
          log.debug('encoded input',str(encoded_input))
          model_output = self.model(**encoded_input)
          log.debug('model output',str(model_output))

          # Perform pooling. In this case, max pooling.
          sentence_embeddings = self.mean_pooling(model_output, encoded_input['attention_mask'])
          # Normalize embeddings, because Fusion likes it that way.
          sentence_embeddings = F.normalize(sentence_embeddings, p=2, dim=-1)
          # Fixing the shape of the emebbedings to match (1, 384).
          final = [sentence_embeddings.squeeze().cpu().detach().numpy().tolist()]
        return final

    def class_names(self) -> Iterable[str]:
        return ["vector"]

• init: The init function is where models, tokenizers, vectorizers, and the like should be set to self for invoking.
  It is recommended that you include your model’s trained parameters directly into the Docker container rather than reaching out to external storage inside init.
• predict: The predict function processes the field or query that Fusion passes to the model.
  The predict function must be able to handle any text processing needed for the model to accept input invoked in its model.evaluate(), model.predict(), or equivalent function to get the expected model result. If the output needs additional manipulation, that should be done before the result is returned.
  For embedding models the return value must have the shape of (1, DIM), where DIM (dimension) is a consistent integer, to enable Fusion to handle the vector encoding into Milvus or Solr.

​

Create a Dockerfile

#It is important that python version is 3.x-slim for seldon-core
FROM python:3.9-slim
# Whatever directory(folder)the python file for your python class, Dockerfile, and
# requirements.txt is in should be copied then denoted as the work directory.
COPY . /app
WORKDIR /app

# The requirements file for the Docker container
COPY requirements.txt requirements.txt
RUN pip install --no-cache-dir -r requirements.txt

# Copy source code
COPY . .

# GRPC - Allows Fusion to do a Remote Procedure Call
EXPOSE 5000

# Define environment variable for seldon-core
# !!!MODEL_NAME must be the EXACT same as the python file & python class name!!!
ENV MODEL_NAME mini
ENV SERVICE_TYPE MODEL
ENV PERSISTENCE 0

# Changing active directory folder (same one as above on lines 5 & 6) to default user, required for Fusion
RUN chown -R 8888 /app

# Command to wrap python class with seldon-core to allow it to be usable in Fusion
CMD ["sh", "-c", "seldon-core-microservice $MODEL_NAME --service-type $SERVICE_TYPE --persistence $PERSISTENCE"]

# You can use the following if You need shell features like environment variable expansion or
# You need to use shell constructs like pipes, redirects, etc.
# See https://docs.docker.com/reference/dockerfile/#cmd for more details.
# CMD exec seldon-core-microservice $MODEL_NAME --service-type $SERVICE_TYPE --persistence $PERSISTENCE

​

Create a requirements file

seldon-core
torch
transformers
numpy

pip freeze > requirements.txt

​

Build and push the Docker image

DOCKER_DEFAULT_PLATFORM=linux/amd64 docker build . -t [DOCKERHUB-USERNAME]/[REPOSITORY]:[VERSION-TAG]

docker push [DOCKERHUB USERNAME]/[REPOSITORY]:[VERSION-TAG]

DOCKER_DEFAULT_PLATFORM=linux/amd64 docker build . -t jstrmec/example_sbert_model:0.14; docker push jstrmec/example_sbert_model:0.14

​

Deploy the model in Fusion

1. In Fusion, navigate to Collections > Jobs.
2. Add a job by clicking the Add+ Button and selecting Create Seldon Core Model Deployment.
3. Fill in each of the text fields:

   Parameter	Description
   Job ID	A string used by the Fusion API to reference the job after its creation.
   Model name	A name for the deployed model. This is used to generate the deployment name in Seldon Core. It is also the name that you reference as a model-id when making predictions with the ML Service.
   Model replicas	The number of load-balanced replicas of the model to deploy; specify multiple replicas for a higher-volume intake.
   Docker Repository	The public or private repository where the Docker image is located. If you’re using Docker Hub, fill in the Docker Hub username here.
   Image name	The name of the image with an optional tag. If no tag is given, latest is used.
   Kubernetes secret	If you’re using a private repository, supply the name of the Kubernetes secret used for access.
   Output columns	A list of column names that the model’s predict method returns.

4. Click Save, then Run and Start. When the job finishes successfully, you can proceed to the next section.

​

Apply an API key to the deployment

1. Create and modify a <seldon_model_name>_sdep.yaml file.
   In the first line, kubectl get sdep gets the details for the currently running Seldon Deployment job and saves those details to a YAML file. kubectl apply -f open_sdep.yaml adds the key to the Seldon Deployment job the next time it launches.
   Copy

   kubectl get sdep <seldon_model_name> -o yaml > <seldon_model_name>_sdep.yaml
   # Modify <seldon_model_name>_sdep.yaml to add
          - env:
            - name: API_KEY
              value: "your-api-key-here"
   kubectl apply -f <seldon_model_name>_sdep.yaml
   

2. Delete sdep before redeploying the model. The currently running Seldon Deployment job does not have the key applied to it. Delete it before redeploying and the new job will have the key.
   Copy

   kubectl delete sdep <seldon_model_name>
   

3. Lastly, you can encode into Milvus.

​

Create a Milvus collection

1. In Fusion, navigate to Collections > Jobs.
2. Click the Add+ Button and select Create Collections in Milvus.
   This job creates a collection in Milvus for storing the vectors sent to it. The job is needed because a collection does not automatically spawn at indexing or query time if it does not already exist.
3. Name the job and the collection.
4. Click Add on the right side of the job panel.
   The key to creating the collection is the Dimension text field; this must exactly match the shape value your output prediction has.
   In our example the shape is (1,384), so 384 will be in the collections Dimension field: The Metric field should typically be left at the default of Inner Product, but this also depends on use case and model type.
5. Click Save, then Run and Start.

​

Configure the Fusion pipelines

1. Create a new index pipeline or load an existing one for editing.
2. Click Add a Stage and then Encode to Milvus.
3. In the new stage, fill in these fields:
   • The name of your model
   • The output name you have for your model job
   • The field you’d like to encode
   • The collection name
4. Save the stage in the pipeline and index your data with it.

1. Create a new query pipeline or load an existing one for editing.
2. Click Add a Stage and then Milvus Query.
3. Fill in the configuration fields, then save the stage.
4. Add a Milvus Ensemble Query stage.
   This stage is necessary to have the Milvus collection scores taken into account in ranking and to weight multiple collections. The Milvus Results Context Key from the Milvus Query Stage is used in this stage to preform math on the Milvus result scores. One (1) is a typical multiplier for the Milvus results but any number can be used.
5. Save the stage and then run a query by typing a search term.
6. To verify the Milvus results are correct, use the Compare+ button to see another pipeline without the model implementation and compare the number of results.