Fusion Server 5.2.0 Release Notes

Subscribe

Release date: 24 August 2020

Component versions:

Solr 8.4.1

ZooKeeper 3.5.7

Spark 2.4.5

New Features

Fusion

Fusion DSL

Fusion Domain Specific Language (DSL) provides expected search results as a JSON response in a way that reduces search query complexity for the user. Previously, users needed to understand complex syntax to express certain search queries in Fusion (for example, the best way to express a facet filter). DSL gives Fusion control over how to execute a query by transforming a structured query input into a Solr request, where we can add intelligence around the index and the user’s intent.

Fusion Search DSL provides:

  1. Standardized feature implementation with predefined syntax, abstracting away underlying SOLR semantics

  2. Future optionality to serve features from other platforms

  3. Revamped critical DSL object-based Query Rewrites, rather than arbitrary and unpredictable string manipulation

See Fusion Domain Specific Language for additional information, including examples.

Pipeline Stage and Blob Editor

Fusion 5.2.0 introduces the ability to create and copy pipeline stages by pasting JSON objects in the Fusion UI. Additionally, it allows users to edit certain blobs in a similar fashion. Only JSON is supported, and JSON validation is included to prevent the user from saving an invalid object.

Pipelines stages

You can copy and paste pipeline stage JSON data directly from the Fusion UI by clicking the JSON View button in the stage’s configuration panel.

Existing stages are READ-ONLY and have the option to Copy only.

Copy pipeline stage JSON

New stages will have Copy and Paste buttons. Changes are applied to the stage with the Apply button.

Note
Users will still need to Save the stage for the changes made in the editor to be saved.

This allows you to quickly and easily duplicate and edit stages within your app or share your configurations with others.

Paste pipeline stage JSON

See Index Pipeline Stages and Query Pipeline Stages for more information. For instructions, see Use the Pipeline Stage JSON Editor.

Blobs

The blobs editor is accessed by clicking the Edit blob button in the blobs configuration panel:

Edit blob button

The blobs editor is similar to the pipeline stage editor with the following exceptions:

  • Multiple file types are supported, including:

    • CSV

    • Java

    • JavaScript

    • Python

    • SCALA

    • Typescript

    • Plain text

  • Existing blobs are not considered READ-ONLY, giving users the option to edit existing blobs.

  • When the user clicks the Save button, changes to the blob data take effect immediately.

See Blob Storage for more information. For instructions, see Use the Blob Editor.

Highlight Query Pipeline Stage

Fusion 5.2.0 introduces a new query pipeline stage, Highlight.

The Highlight stage allows you to highlight key parts of fields by generating snippets and appending match tags to the highlighted terms. The highlight match tags can then be styled in your web search application.

Highlight stage highlights

To learn more, including how this stage differs from other highlighting options, see the Highlight Stage reference article. To learn how to use the Highlight stage, see Configure the Highlight Query Pipeline Stage.

Config-sync Microservice

Config-sync is a stateless microservice used to integrate Fusion’s configuration management with GitHub.

Features:

  • Capture changes to Fusion config in centralized source control

  • Use familiar tools and workflows, including visual diff, Pull Requests, WebHooks, and email notifications

  • Enable a simple review process before promoting changes to another environment, such as stage to production

  • Prevent manual changes in production environments; all changes are applied using code and API calls are driven by changes to a git branch

  • Extensible: Integrate with other tools like Spinnaker, Argo CD, or Jenkins (apply config changes as part of a CI pipeline)

  • Ability to rollback to a previous version using Git revert

For more information on the microservice, see Config-sync Microservice. For instructions on how to use the microservice, see How to Set Up Config Sync.

Fusion AI

  • The new Classification job analyzes how existing documents are categorized and produces a model that can be used to predict the categories of new documents at index time.

  • The new BPR Recommender job produces better signals-based recommendation data with a shorter running time than the ALS Recommender job (deprecated in this release).

  • The new Content-Based Recommender job analyzes similarities in your content to provide recommendations when you don’t have enough signals data for signals-based recommendation methods.

Predictive Merchandiser

Templates

The Templates screen in the Rules Editor allows you to design, test, and implement a wide variety of search experiences. Results are conditionally delivered from multiple different query pipelines, enabling you to use Fusion’s search, browse, and AI-driven functionality throughout your site.

Templates with zones

See Predictive Merchandiser Templates for more information. For instructions on how to use Templates, see Use Predictive Merchandiser Templates.

Improvements

Fusion

  • Fusion SQL adds search and math expressions to the SQL experience, with new, intuitive, easy-to-learn functions that simplify the way power users query Fusion to explore unstructured text and patterns in data. In this release you can combine structured and unstructured data stored in Solr to understand numeric distributions and correlations, fit models, explore clusters, perform powerful sampling, use text analyzers for entity extraction, find significant terms, and perform time series analysis.

    Learn more about Fusion SQL or explore the new functions for these features:

  • You can now download connectors directly from the Fusion UI. Powered by the Connector Plugins Repository API, you can easily view which connectors are available to you and download new connectors with a single click. Navigate to Indexing > Datasources, click Add, and click the name of the connector you want to install:

  • The Field Facet query pipeline stage now allows users to specify Field facet threads:

    field facet threads

    Specifies how many threads the search engine will use while loading the underlying fields used in faceting. Recommended that this is kept at 0 or blank, which means the data is loaded in the thread already running the query.
  • Fusion microservices can now be configured to use Transport Layer Security (TLS).

    When enabled, Fusion generates a TLS certificate for each pod when the pod starts. This allows Fusion to use the Kubernetes endpoints API to reach each pod by its IP address and perform load balancing, circuit breaking, and retries in the Fusion microservices.

  • Various Spark jobs improvements:

    • Improved logging

    • All Spark writing operations now support min-output-partitions

    • SQL aggregation jobs are audited for natural key usage

    • Recommender aggregation

  • Previously, the Apache Pulsar client automatically blocked message objects if the queue was full. Now, this behavior is configurable as a boolean value. By default, this is set to false in the query service to ensure errors are successfully logged even if the Pulsar service becomes available. Outside of the query service, the default is true.

  • Pulsar is updated to version 2.6.0.

  • The Query Workbench no longer displays additional pages if there are not enough results to populate those pages.

  • The Query Workbench now displays error messages instead of writing error messages to the console only.

  • AsyncJoinQueryStage is updated to always produce an error message to improve debugging.

  • You can now configure your realm with additional headers to check for allowed IPs. This is useful when your gateway is behind a proxy which is replacing the incoming IP.

  • A new Index Stage SDK API method is added: {{Document.allFields()}}. This method returns all document fields, including internal fields. (Internal fields start with the {{lw}} prefix.)

Fusion AI

  • Solr’s zkhost is now included in the Jupyter environment. It no longer needs to be explicitly defined for each Jupyter notebook.

  • You can now use more than 30 parameters in a single business rule.

Predictive Merchandiser

  • Publishing performance is significantly improved.

Other changes

Fusion

  • Pipeline context can now properly serialize/deserialize arbitrary values and nested maps/lists when sent via Pulsar Message.

  • The Pulsar default namespace is now set when creating or updating an index pipeline or profile. Previously, this was set when the stage was first enacted.

  • All paths starting with /apollo are removed.

  • Miscellaneous UI improvements.

Fusion AI

  • The Rules Editor’s Dashboard is renamed to Analytics.

Predictive Merchandiser

  • Enabled the ability to use nonexistent fields and values when creating a rule.

  • Miscellaneous UI improvements.

Bug fixes

Fusion

  • Fixed a bug which sometimes prevented indexing of access control items at the end of a connector job.

  • The AD ACL V2 connector now uses the group.members field instead of user.memberOf. This addresses a change in Windows Server 2019 that caused incremental crawls to fail.

  • Fixed a bug that sometimes prevented job status icons from updating.

  • Fixed a bug with the job service that sometimes resulted in certain error conditions to create multiple, redundant method calls.

  • Fixed a bug that prevented Spark executors from receiving the Prometheus pushgateway address.

Fusion AI

Predictive Merchandiser

  • Fixed a bug that prevented facet names from appearing in Predictive Merchandiser if the facets were created in a query pipeline using the Field Facet stage.

  • Fixed a bug that caused errors in the Predictive Merchandiser UI when switching between query profiles with grouping in use and query profiles without.

  • Fixed a bug that prevented a set-params rule from saving successfully multiple parameters are defined.

Deprecations

Important
The following features and functionalities are deprecated in Fusion 5.2.0 and will be removed in a future release.

Known Issues

  • Training on the BPR recommendation job when metadata settings are specified will fail.

    Workaround Steps:

    1. Download the shell script from here.

    2. Run the script in a terminal ./rec-5.2.0-workflow-fix.sh <namespace>

      Note
      Replace <namespace> with the k8s namespace where Fusion is deployed
  • Running the recommendation or classification workflows may result in pods failing with a docker image pull error.

    Workaround Steps:

    1. Download the shell script.

    2. Run the scrupt in a terminal ./workflow-5.2.0-tag-fix.sh <namespace>

      Note
      Replace <namespace> with the k8s namespace where Fusion is deployed