Fusion Server 4.0.0 Release Notes

Release date: 21 February 2018

Component versions:

Solr 7.2.1

ZooKeeper 3.4.10

Spark 2.2.1

Jetty 3.8.10

Ignite 2.3.0

New features

  • Fusion AI

    Fusion’s artificial intelligence and machine learning features are now licensed separately, as the new Fusion AI product. These features are now enabled only with a Fusion AI license:

    • Signals

    • App Insights (new in 4.0)

    • Smart jobs

      • ALS Recommender Jobs

      • Bisecting KMeans Clustering Jobs

      • Cluster Labeling Jobs

      • Co-Occurrence Similarity Jobs

      • Collection Analysis Jobs

      • Document Clustering Jobs

      • Ground Truth Jobs

      • Head/Tail Analysis Jobs (new in 4.0)

      • Item Similarity Recommender Jobs

      • Levenshtein Spell Checking Jobs

      • Logistic Regression Classifier Training Jobs

      • Matrix Decomposition-Based Query-Query Similarity Jobs

      • Outlier Detection Jobs

      • Random Forest Classifier Training Jobs

      • Ranking Metrics Jobs

      • SQL-Based Experiment Metric Jobs

      • Statistically Interesting Phrases Jobs

      • Token and Phrase Spell Correction Jobs (new in 4.0)

    • REST APIs

      • Experiments API

      • Recommendations API

      • Signals Aggregator API

      • Signals API

    • Index pipeline stages

      • Machine Learning index stage

      • OpenNLP NER Extractor index stage

      • Signal Formatter index stage

      • Update Experiment index stage

    • Query pipeline stages

      • Advanced Boosting query stage

      • Analytics Catalog query stage

      • Experiment query stage

      • Machine Learning query stage

      • More Like This query stage

      • Recommend Items for Item query stage

      • Recommend Items for User query stage

      • Recommendation Boosting query stage

  • Apps

    Fusion apps provide tailored search functionality to specific groups of users.

    An app is a named set of linked objects, including collections, datasources, index and query pipelines, index and query profiles, parsers, and more. Using roles and security realms, you can define security on a per-app basis.

    • A new Apps API is available for managing apps.

    • The Launcher page now contains apps instead of workflow contexts. Workflows are the same for all apps. See Working with Apps.

    • Apps are objects that can be imported and exported using the Objects API, and their links can be managed using the Links API.

    • App objects and their links can be viewed using the Object Explorer.

  • License management

    Fusion now requires a license. A free 30-day trial license is included in the download. You can now manage your Fusion license using the Fusion UI or REST API. For information about licensing for upgrades from 3.x to 4.0, contact Lucidworks.

  • Confluence connector

    This new connector retrieves data from the Atlassian Confluence Wiki CMS. You can configure this datasource to crawl pages, spaces, blog posts, comments, and attachments. See the Confluence Connector reference topic.

  • Webapps API

    This API lays the groundwork for a future release in which Fusion App Studio can interface directly with Fusion Server to customize the front-end interfaces for your apps. Though this API is not usable in this release, you can preview it in the Webapps API reference topic.

  • New index pipeline stages

Improvements

  • Connector improvements

    • Sharepoint connector:

      • Performance improvements

      • Security trimming: support for groups, domain/username

      • XML parsing updates

      • Active Directory Federation Services (ADFS) support

    • Jive connector:

      • hierarchical crawl support from root Jive space

      • Security trimming: support for objects that inherit ACLs

      • Update permissions on re-crawl for Spaces

      • indexing of events, videos, and ideas

    • Jira connector:

      • Attachments and work logs now support issue reporter inheritance

    • Crawl stability for Box and Google Drive

    • Web connector: headless Firefox crawling for JS

  • New APIs for profiles

    Index profiles and query profiles are now created with definitions that include the collection, pipeline, and handler. As such, the endpoints for profiles are no longer located under the /collections endpoint; this change supports experiment management in Fusion AI. New top-level endpoints have been added in this release.

    • The endpoint at /collections/<collectionName>/query-profiles has been replaced by two new endpoints:

      • Query Profiles API (/query-profiles)

        Create, read, update, and delete query profiles.

      • Query API (/query)

        Run a query through a query profile by specifying the profile ID and appending the request with a Solr query string, as in /api/apollo/query/<id>?<solrQuery>.

    • Similarly, the endpoint at /collections/<collectionName>/index-profiles has been replaced by two new endpoints:

      • Index Profiles API (/index-profiles)

        Create, read, update, and delete index profiles.

      • Index API (/index)

        Index data through an index profile by specifying the profile ID.

  • UI redesign

    The Fusion UI has a new look and feel, and menus have been redesigned. See Fusion Server UI for a brief tour.

  • Objects API improvements

    The Objects API now outputs a .zip file for every export, and supports the new app object type. Another new object type, appkit-app, is for use in future releases.

  • Solr cluster management

  • SQL for aggregations

    By default, aggregation jobs are now defined using Spark SQL. To use the old aggregation configuration scheme in the Fusion UI, click Advanced in the Aggregation job configuration panel.

  • Multiple Hadoop connectors

  • Blob Store API changes

    • A new /blobs/convert endpoint migrates blob manifests from 3.x to 4.x format.

    • The /blobs endpoint has new parameters to support querying specific directories in the blob store: dir, recurse, and showdirs.

    • When uploading a blob via the /blobs/{id} endpoint, new parameters are supported: fusion-blob-type, commit, and update.

    • The /blobs/{id} endpoint now supports a HEAD method to retrieve blob metadata, equivalent to the /blobs/{id}/manifest endpoint.

    • The /blobs/{id}/manifest endpoint has been removed.

    • Fusion no longer supports uploading blobs via forms.

    See the Blob Store API reference topic for details.

  • New framework for connectors

    A new connectors framework allows connectors to be hosted or remote:

    • Hosted connectors are cluster-aware, so that cluster nodes are notified when connector services start up on other nodes. This allows easy scaling of the crawl process.

    • Remote connectors act as lightweight Fusion clients which can be located wherever your data is stored.

    See Connectors for more details.

    Note
    In the 4.0.0 release, the Lucidworks-provided connectors follow the classic framework. The new framework will be applied to Lucidworks-provided connectors in future releases. You can use the new Connectors SDK to develop custom connectors using the new framework.

Other changes

  • Solr 4.x is no longer supported. This release of Fusion Server requires Solr 6.x or higher.

  • The REST API endpoints for production deployments now use the path /api/ and not /api/apollo/. The old path remains available but deprecated.

  • A new webapps-role role and webapps-system-account realm were added to support apps.

  • The <collection>_logs and <collection>_signals collections are now combined into a single <collection>_signals collection.

    Important
    The _signals collection schema has changed in this release. If you have existing signals data, you must use the migrator script to convert it correctly. The migrator script will be provided following this 4.0 release. See Signals for details about the new schema.
  • The Dropbox connector is no longer available.

  • The Synonyms API has been removed.

  • Anonymous metrics submission

  • Aggregator API is deprecated

    The /aggregator endpoints are deprecated in this release. Aggregations are now managed through the /spark endpoints.

  • The Experiments API is deprecated.

    Experiments are now managed through the /spark endpoints.

Known issues

  • After SSL is enabled on Fusion Server 4.0.1, the server incorrectly redirects certain requests, including requests for https://fusion-server:port/ and https://fusion-server:port/admin, resulting in a failure to reach the login page. To log in to the Admin UI after enabling SSL, use the direct URL to the login page as a workaround: https://fusion-server:port/admin/login.

    Fusion Server 4.0.2 resolves this issue.

  • Some headers are missing from request.headers in the trusted-http security realm, including the following:

    • fusion-user-realm-name

    • fusion-user-realm-type

    • fusion-user-name

    • fusion-user-permissions lacks the role restriction assigned with the security realm.

  • The Tika parser and Tika stage Return parsed content as XML option works only if the input document is also HTML/XML.