Fusion AI Release Notes

Subscribe

Latest Release: 5.3.0 (18 November 2020)

Fusion comes bundled with everything you need to get started.

Solr 8.6.3

ZooKeeper 3.5.7

Spark 2.4.5

New features

Milvus integration

Fusion 5.3 extends support for semantic search using vectors and embeddings by integrating with Milvus, a highly scalable embeddings engine that allows Fusion to streamline the methodologies that use deep learning for question/answer solutions like Smart Answers, recommendations based on similarity, and regular search.

A number of new components are introduced to manage and utilize Milvus:

New deep learning models

In Fusion 5.3, we are refreshing our deep learning models methodologies to be used in training and inference for semantic search-based Smart Answers. The following models are new in this release:

  • bpe_en_300d_10K

  • bpe_en_300d_200K

  • bpe_ja_300d_100K

  • bpe_ko_300d_100K

  • bpe_zh_300d_50K

  • pe_multi_300d_320K

The bpe_{language}_{dim_size}_{vocab_size} models are general pre-trained BPEmb embeddings that are available for different languages, including Chinese/Japanese/Korean (CJK) languages and multilingual. These are also useful in scenarios when vocabulary is very big or when the data might contain a lot of misspellings.

  • distilbert_en

  • distilbert_multi

These are distilled, performance-optimized versions of BERT models designed to be used on scale. Available for English language and multilingual applications.

  • biobert_v1.1

This is a BERT model that was pre-trained on large-scale biomedical corpora which makes it more suitable for biomedical domain applications.

Answer Extraction

To enhance how our Smart Answers customers interact with results sets that are composed of large documents, Fusion 5.3 adds Answer Extraction, allowing you to extract a paragraph, sentence, or phrase to answer questions.

When a large document is presented as a result to a query, Answer Extraction extracts the sentences out of the document that are most similar to the query content. To configure this feature, you train a model that gets deployed at the end of the Smart Answers query pipeline stage, after the resulting set of large documents is returned from Solr for final ranking. The model outputs the sentences from each document that are the most similar to the query.

Answer Extraction workflow

The Answer Extraction model is now available from the Lucidworks official Docker to be deployed as a Seldon model. See Extract Short Answers from Longer Documents for detailed configuration steps.

Improvements

  • You can now view, edit, publish, and delete unpublished rules created by other users in the Rules Editor.

  • AI jobs can now read and write from GCS directly. Meaning, the customer engagement data aka signals do not necessarily have to be stored in Solr for the jobs to work. This development is primarily to avoid situations where Solr is unable to keep up with the write requests from the jobs. GCS handles high speed, high scale writes efficiently.

Previous releases

5.2.1 (23 September 2020)

Component Versions

Solr 8.4.1

ZooKeeper 3.5.7

Spark 2.4.5

Important
A patch is released for this version that addresses an issue that can cause a serious performance issue for high-scale deployments. Please upgrade from Fusion 5.2.1 to 5.2.2 as soon as possible. Contact your Customer Excellence representative if you have questions about the potential impact to your deployment.

5.2.0 (24 August 2020)

Component Versions

Solr 8.4.1

ZooKeeper 3.5.7

Spark 2.4.5

  • The new Classification job analyzes how existing documents are categorized and produces a model that can be used to predict the categories of new documents at index time.

  • The new BPR Recommender job produces better signals-based recommendation data with a shorter running time than the ALS Recommender job (deprecated in this release).

  • The new Content-Based Recommender job analyzes similarities in your content to provide recommendations when you don’t have enough signals data for signals-based recommendation methods.

No new features or improvements are introduced in Fusion AI 5.1.4. See the Fusion 5.1.4 Release Notes for full details.

5.1.4 (25 June 2020)

Important
A patch is released for this version that addresses an issue that can cause a serious performance issue for high-scale deployments. Please upgrade from Fusion 5.1.4 to 5.1.5 as soon as possible. Contact your Customer Excellence representative if you have questions about the potential impact to your deployment.
Component Versions

Solr 8.4.1

ZooKeeper 3.5.6

Spark 2.4.5

No new features or improvements are introduced in Fusion AI 5.1.4. See the Fusion 5.1.4 Release Notes for full details.