Fusion Server 5.1.0 Release NotesSubscribe

Release date: 12 March 2020

Component versions:

Solr 8.4.1

ZooKeeper 3.5.6

Spark 2.4.3

Important
Looking to upgrade from Fusion 5.0.x to 5.1.0? See the upgrade procedure.

New features

Fusion

Index Stage SDK

Lucidworks now provides an Index Stage SDK in a public repository on GitHub with all the resources you need to develop custom index stages with Java.

Clone the repository to get started:

git clone https://github.com/lucidworks/index-stage-sdk

See Gradle quickstart documentation for more information on Java Projects.

New V2 connectors

Three new V2 connectors are released:

  • Couchbase V2 - The Couchbase V2 connector uses the Couchbase Java client to retrieve data stored in Couchbase.

  • Sharepoint V2 - The SharePoint connector retrieves content and metadata from an on-premises SharePoint repository. The new connector includes many benefits associated with V2 connectors.

  • Active Directory Connector for ACLs V2 - The Active Directory Connector for ACLs (V2) indexes Active Control List (ACL) information into a configured "sidecar" Solr collection, so that it can be used by other connectors.

Apache Pulsar

Fusion 5.1.0 adds Apache Pulsar, a message bus that combines queuing with publishing-subscribing features for server-to-server messaging. A producer publishes a message to a topic and a consumer subscribes to that topic using a subscription type to receive the topic messages.

Higlights include:

  • Supports native streaming transformations through Pulsar Functions

  • Uses Apache Zookeeper for configuration management

  • Uses Apache BookKeeper for durable storage

  • Each document processed has a result object persisted to another topic

Read more about Apache Pulsar in Fusion.

More new features

  • A new metrics report Java library improves the functionality of the DevOps center.

  • Fusion creates a log file for login attempts by producing an audit file with the API gateway.

Fusion AI

Sentiment analysis

One major part of understanding intent is understanding sentiment. When you assign sentiment values to incoming queries, or to documents during indexing, you can leverage that data to deliver more insight to your business applications.

This release includes two state-of-the-art, pre-trained sentiment analysis models that you can use to embed sentiment scoring into your query and indexing pipelines:

sentiment-general sentiment-reviews

A general-purpose model, trained on short sentences.

Suitable for short texts and for intent prediction.

See Deploy The sentiment-general Model for instructions.

A model trained on a variety of customer reviews.

Optimized for longer texts. It also supports highlighting the tokens that provide stronger sentiment.

See Deploy The sentiment-reviews Model for instructions.

See Sentiment Analysis and Prediction for more details and instructions.

Predictive Merchandiser

  • Clicking the tiles in the Analytics screen now updates the graph below to display detailed metrics.

    Analytics Chart New Feature
  • The ability to switch apps from within the Rules Editor is added:

    Switch apps in the Rules Editor.
  • Support is added for individually selecting rules to publish in the Fusion UI:

    Individually publish rules in the Fusion UI.

Improvements

Fusion

  • Index stage plugins can now be uploaded directly to the Blob manager. To upload to the API with a PUT request, use the api/index-stages/plugins endpoint. To delete the plugin, use a DELETE request with the api/index-stages/plugins/<pluginId> endpoint.

  • The default realms created when Fusion first starts can be overridden using the security.initial-realm-configs spring boot properties. For example:

    security:
     initial-realm-configs:
      {
        "realmType": "jwt",
        "name": "jwt_okta",
        "enabled": true,
        "roleNames": [
          "admin"
        ],
        "config": {
          "autoCreateUsers": true,
          "jwtIssuer": "https://dev-362383.okta.com/oauth2/default",
          "jwkSetUri": "https://dev-362383.okta.com/oauth2/default/v1/keys",
          "groups": {
            "groupKey": "fusion-groups",
            "roleMapping": [
              [
                "admin",
                "admin"
              ]
            ]
          }
        }
  • The Index Workbench now supports "save as" functionality, like the Query Workbench.

  • Users can now import Solr synonym definitions into Fusion through the Blob Manager.

  • A new category, Custom, is added to index and query pipelines. Custom index pipelines include External collection query, JavaScript, and Simple. Custom query pipelines include Javascript.

    Custom pipeline category

  • The externalTrafficPolicy for the api-gateway service is now set to Local by default. Existing users can configure this property by adding it to their custom_fusion_values.yml file:

    api-gateway:
      externalTrafficPolicy: "Local"
  • Trusted realm supports IPv6 normalization and netmask in CIDR notation.

  • With Solr 8.x and 7.7.y, the SOLR-OPTS environment variable, found in the Java properties, can modify the max.booleanClauses variable.

    solr:
        image: solr:8.3
        container_name: solr
        hostname: solr
        environment:
          ZK_HOST: zk:2181
          SOLR_OPTS: -Dsolr.max.booleanClauses=5000
  • The error message generated by sending click signals to a collection that has signals disabled is improved.

  • Users with the role of "developer" now have expanded UI access.

  • Various UI improvements are made.

Fusion AI

  • A new parameter, loadOnStartup, is added for machine learning models. Models with loadOnStartup=true will be automatically loaded when the machine learning service is started or the model is loaded into the machine learning service. Default behavior requires the models to be loaded on demand.

Predictive Merchandiser

Facets

  • Boost, bury, or block facets from your search results:

    Search Facet Rules

  • Boosted facets can now be reordered with drag-and-drop:

    Reorder Facets
  • Rule creation now includes selected facets as part of the rule.

  • Users can now block facet filters for facets created in the Rules Editor.

Rules

  • When creating a rule, active facets become part of the rule.

  • When creating a new rule for a term that is associated with a query rewrite rule, the new rule will apply to the rewritten term.

    For example, if a query rewrite rule exists to rewrite the query Mac as Macbook Pro, creating a new rule for Mac will automatically apply the rule to Macbook Pro instead.

  • You can now assign a new rule to each of your product pins. For example, you can pin products based on a sales promotion, holiday event, etc.

  • Sort rules by priority with drag-and-drop, just as you can with facets.

  • You can now publish approved rules individually, instead of waiting for all pending approvals to be processed.

  • Rules created for a query rewrite now apply to the rewrite instead of the original term.

  • Added an optional Frequency column to the Query Rewrites dashboard for Phrases and Synonyms only, which you can use to sort and focus on high-value rewrites. You can add this column by customizing the columns for those rewrite types.

  • Added Reason meta-data for head/tail rewrites, to explain the reason why the rewrite was suggested.

  • Added Type meta-data for spell corrections rewrites, to explain whether a token-level correction or phrase-level correction is recommended.

  • Added a new option for users to enable or disable query rewrite rule types.

    Enable or disable rule types

  • Rules with high confidence ratings can be published automatically to production from the AI Jobs. Fusion 5.1 does not auto-publish from AI Jobs by default.

Other improvements

  • Recommended products that have been generated by signals are included in the Product Details panel.

    Recommended products

  • You can now boost, bury, and block multiple products at once.

    Boost bury and block multiple products

  • Documents now feature an icon for drag-and-drop functionality.

  • A date range filter component is added to the Dashboard for quickly sorting analytics data:

    Quick Filter by Date

  • Various UI improvements.

Addressed security vulnerabilities

Fusion

  • Fixed an issue with a component in the admin-ui package that resulted in an security alert.

Bug fixes

Fusion

  • The Parallel Bulk Loader (PBL) will no longer fail when a document fails to be uploaded to the index pipeline using POST. Enable this functionality in the PBL configuration.

  • The Tika parser now honours MIME type parameters, for example: {{application/vnd.wordperfect; version=5.0}}`. Previously, content MIME types parameters weren’t always correctly matched to types supported by Tika. This resulted in the Tika parser not recognizing content as parseable.

  • Fixed a bug that prevented files with non-ascii filenames from being uploaded to the index workbench.

  • Fixed a bug that sometimes prevented renamed files from showing up in query results after being reindexed.

  • Fixed a bug that caused the query workbench to fail to load if there were over 10,000 Solr fields.

  • Fusion no longer blocks access to the Blob Store and Collections Manager when the collections service is down.

  • The Github connector no longer authenticates with a username and password, as this functionality is deprecated. Now, a personal access token is required.

  • Connectors can now join a running job.

  • Fixed a bug that sometimes resulted in an ArrayIndexOutOfBoundsException error when running a spell correction job.

  • Fixed a bug that prevented apps created in Fusion 4.2.x from being imported to 5.y.

  • All zkConnectionStrings values can be set with a single global variable, zkConnectionString. Previously, the values were set individually per service.

  • Jupyter sessions will no longer expire while the user is still active.

  • Fixed a bug with the de-duplicate feature that allowed spurious fields to be written to Solr.

  • Fixed a bug that required a non-empty, non-root context path for deploying an application into the webapps service.

Fusion AI

  • Switching between apps in the Fusion UI while viewing the query rewrite tab will no longer bring you to the rules tab.

  • When multiple rules are selected in bulk, the enable and disable buttons are now inactive.

  • Fixed a bug that prevented faceting query rewrites in Internet Explorer 11.

  • Fixed a bug that disabled the option to delete query rewrite synonyms if a synonym was edited after being added.

  • Fixed a bug that prevented the user from publishing rules by type.

  • Fixed a bug that prevented query rewrite from creating multi-word synonyms.

Predictive Merchandiser

  • Fixed a bug that prevented facet list items from reappearing when unselecting a facet category.

  • Fixed a bug with the typeahead functionality of the search box.

  • Fixed a bug that prevented items in the product details table from appearing.

Known issues

Fusion

  • When subscriptions are refreshed, only the original node receives the request to start or stop the subscription. This is fixed in Fusion 5.1.1.