Seldon
Seldon is an open-source platform used to deploy, scale, and monitor machine learning (ML) models in Kubernetes environments. It is integrated into Fusion to enable model-serving capabilities for inference workflows. For more information, see Seldon Core documentation for more details.
Seldon models and endpoints
Fusion Seldon models are trained on different lengths of text such as short and long sentences or phrases, as well as varied collections for B2B, B2C, and knowledge management entities. Models are exposed via a unique endpoint, and are used in a pipeline stage or external application. Endpoints can be secured using Istio or other methods through the Fusion gateway. Supported models include general purpose sentiment prediction, biomedical text collections, and large corpuses in a multiple languages. Based on the model, Seldon supports input formats such asndarray, tensor, and json. Outputs are in a structured JSON format, typically wrapped in metadata, which supports monitoring and logging functions.
Seldon configuration and management
This section provides links to more detailed information to configure and manage Seldon deployments for Fusion users.- Create Seldon Core Model deployment in a Fusion cluster
- Update Seldon Model deployment
- Delete Seldon Core Model deployment
- Configure Ray/Seldon vector search in Neural Hybrid Search
- Ray/Seldon Vectorize Query stage to generate a vector based on a query string
- Ray/Seldon Vectorize Field stage to invoke a machine learning model that encodes a string field to a vector representation
Configure Ray/Seldon vector search
Configure Ray/Seldon vector search
You can use Seldon or Ray models to vectorize text for Neural Hybrid Search.To vectorize text with Ray, you first need to develop and deploy a machine learning model with Ray.
This feature is only available in Fusion 5.9.5 and later versions of Fusion 5.9.
Prerequisites
Seldon
To vectorize text with Seldon, you first need to develop and deploy a machine learning model with Seldon.Ray
This feature is only available in Fusion 5.9.x for versions 5.9.12 and later.
Configure index pipeline
- Sign into Fusion, go to Indexing > Index Pipelines, then select an existing pipeline or create a new one.
- Click Add a new pipeline stage, then select Ray/Seldon Vectorize Field. In Fusion 5.9.11 and earlier, this stage is called Seldon Vectorize Field.
- Fill in the required fields:
- Enter a Model ID. This is the name of the model you developed and deployed.
- Enter the Model Input Field. For example,
text. - Enter the Model Output Vector Field. For example,
vector. - Enter the Source Field. For example,
body_t. - Enter the Destination Field. For example,
body_512_v.
- Click Save.
- Make sure the Seldon Vectorize Field stage is ordered before the Solr Indexer stage.
Configure query pipeline
- Go to Querying > Query Pipelines, then select an existing pipeline.
- Click Add a new pipeline stage, then select Ray/Seldon Vectorize Query. In Fusion 5.9.11 and earlier, this stage is called Seldon Vectorize Field.
- Fill in the required fields, making sure to search against the field into which you indexed the vectors:
- Enter a Model ID. This is the name of the model you developed and deployed].
- Enter the Query Input.
- Enter the Model Input Field. For example,
text. - Enter the Model Output Vector Field. For example,
vector. - Enter the Vector Context Key. For example,
vector.
- Click Save.
- Make sure the Ray/Seldon Vectorize Query stage is ordered before the Solr Query stage.
Perform vector searches
After setting up the stages, you can perform vector searches via theknn query parser as you would with Solr. Specify the search vector and include it in the query. For example, change the q parameter to a knn query parser string.The Ray/Seldon Vectorize Query stage will encode user queries using the specified model and modify the q parameter to use the knn query parser, turning the query into a vector search.Update a Seldon model deployment
Seldon does not support in-place updates to certain deployment properties, for example, protocol or runtime class. To apply significant updates:- Deploy a new version of the model using a different deployment name.
- Execute test requests to validate the new endpoint.
- Update the Fusion pipeline with the new endpoint information.
- Delete the previous Seldon deployment when the migration is complete.