Fusion Server Release Notes

Current Fusion Server Release: 4.1.1

Release date: 7 November 2018

Component versions:

  • Solr 7.4.0

  • ZooKeeper 3.4.13

  • Spark 2.3.1

  • Jetty 9.3.25.v20180904

  • Ignite 2.3.0

Past Fusion Server releases

Version Release Date Major Changes Component Versions Release Notes

4.1.0

17 July 2018

  • App Studio integration

    Fusion App Studio is now integrated into the Fusion Server UI for seamless search UI development. You can open App Studio from the Fusion workspace by navigating to App Studio > Build new UI.

    Integrated App Studio

  • Fusion Server UI search/autocomplete

    In the Fusion workspace, you can now search objects and documentation, with autocomplete to suggest relevant results.

    UI Search and Autocomplete

  • New OneDrive connector

  • New Download Blob job

  • New log shipper and log console

  • Connector improvements

    • The Confluence connector now indexes labels for spaces and pages.

    • The Web connector can be configured to deduplicate pages whose content is identical.

    • The Slack connector is no longer deprecated.

  • UI improvements

    • The Object Explorer now supports filtering by object type.

    • Objects in the UI now include an "Explore" button that opens the Object Explorer view for that object.

    • UI panels can now be resized horizontally.

    • Text input areas can now be opened as pop-out modals for easier editing.

  • Connectors SDK

  • Import/export improvements

    • Additional parameter substitutions are now supported on object import.

  • Better heuristics for the splits_per_shard field in the Catalog API.

  • Fusion’s SparkSQL engine is no longer a beta release.

  • Ports are now preconfigured for Spark multinode communication, using these parameters in the Configurations API:

    • spark.driver.port

    • spark.executor.port

    • spark.blockManager.port

    • spark.fileserver.port

  • A request ID can now be defined in order to add context to logs.

  • The Jobs API now provides better performance.

  • The system collections named audit_logs and logs have been replaced and extended with a new system collection, system_logs.

  • Solr 7.4.0

  • ZooKeeper 3.4.11

  • Spark 2.3.1

  • Jetty 9.3.8.v20160314

  • Ignite 2.3.0

4.0.2

22 May 2018

  • A new Windows Share connector provides support for SMB 2 and 3.

    Note
    The previous Windows Share Connector is deprecated.
  • Additional options for the Zendesk connector:

    • article_containers

    • enable_security_trimming

    • index_article_attachments

    • index_articles

    • index_tickets

    • page_size

    • zendesk_account_plan

  • Web connector improvements

    • Instead of JBrowserDriver and Mozilla Firefox, the Web connector now uses PhantomJS and Google Chromium for JavaScript evaluation. When f.crawlJS/"Evaluate JavaScript" is enabled, then Fusion uses PhantomJS to evaluate JavaScript content. Lucidworks recommends also enabling f.useHighPerfJsEval/"High Performance Mode", in which case Chromium is used, providing the fastest possible JavaScript evaluation.

      Note
      This feature requires downloading a separate setup script. See the Web connector topic for instructions.
    • Support for screenshots

      When f.crawlJS/"Evaluate JavaScript" and f.useHighPerfJsEval/"High Performance Mode" are enabled, then f.takeScreenshot/"Index a screenshot of rendered page" can also be enabled.

      When this feature is enabled, Fusion takes a screenshot of each fully-rendered page and indexes it in a field called screenshot_bin.

      Important
      Your schema must specify this field as a binary field or indexing will fail. See the Web connector topic for instructions.

    New options for the Web connector:

    • f.useHighPerfJsEval

    • f.headlessBrowser

    • f.takeScreenshot

    • f.screenshotFullscreen

    • f.viewportWidth

    • f.viewportHeight

    • f.deviceScreenFactor

    • f.simulateMobile

    • f.mobileScreenWidth

    • f.mobileScreenHeight

    • f.chromeBinaryPath

    • f.quitTimeoutMs

    • f.useRequestCounter

    • f.requestCounterMinWaitMs

    • f.requestCounterMaxWaitMs

    The following configuration keys are deprecated and will be removed in a future release:

    • f.useFirefox

    • f.firefoxHeadlessBrowser

    • f.firefoxBinaryPath

  • The Jira connector now supports security trimming when single sign-on (SSO) is used to authenticate to the Jira servers.

  • The Jobs API now performs faster.

  • This release fixes a bug in the authentication proxy which incorrectly redirected certain requests when SSL was enabled.

  • Spark job IDs are now transformed so that invalid characters are replaced with underscores (_). Valid characters are A-Z, a-z, 0-9, underscore (\_), backslash (\), and hyphen (-).

  • Solr 7.2.1

  • ZooKeeper 3.4.10

  • Spark 2.2.1

  • Jetty 3.8.10

  • Ignite 2.3

4.0.1

28 February 2018

  • Import and export for index profiles and query profiles

    App import now includes index profiles and query profiles. These can also be exported and imported separately with the Objects API, using the index-profile and query-profile object types.

  • Solr 7.2.1

  • ZooKeeper 3.4.10

  • Spark 2.2.1

  • Jetty 3.8.10

  • Ignite 2.3

4.0.0

21 February 2018

  • Fusion AI

    Fusion’s artificial intelligence and machine learning features are now licensed separately, as the new Fusion AI product. These features are now enabled only with a Fusion AI license:

    • Signals

    • App Insights (new in 4.0)

    • Smart jobs

      • ALS Recommender Jobs

      • Bisecting KMeans Clustering Jobs

      • Cluster Labeling Jobs

      • Co-Occurrence Similarity Jobs

      • Collection Analysis Jobs

      • Document Clustering Jobs

      • Ground Truth Jobs

      • Head/Tail Analysis Jobs (new in 4.0)

      • Item Similarity Recommender Jobs

      • Levenshtein Spell Checking Jobs

      • Logistic Regression Classifier Training Jobs

      • Matrix Decomposition-Based Query-Query Similarity Jobs

      • Outlier Detection Jobs

      • Random Forest Classifier Training Jobs

      • Ranking Metrics Jobs

      • SQL-Based Experiment Metric Jobs

      • Statistically Interesting Phrases Jobs

      • Token and Phrase Spell Correction Jobs (new in 4.0)

    • REST APIs

      • Experiments API

      • Recommendations API

      • Signals Aggregator API

      • Signals API

    • Index pipeline stages

      • Machine Learning index stage

      • OpenNLP NER Extractor index stage

      • Signal Formatter index stage

      • Update Experiment index stage

    • Query pipeline stages

      • Advanced Boosting query stage

      • Analytics Catalog query stage

      • Experiment query stage

      • Machine Learning query stage

      • More Like This query stage

      • Recommend Items for Item query stage

      • Recommend Items for User query stage

      • Recommendation Boosting query stage

  • Apps

    Fusion apps provide tailored search functionality to specific groups of users.

    An app is a named set of linked objects, including collections, datasources, index and query pipelines, index and query profiles, parsers, and more. Using roles and security realms, you can define security on a per-app basis.

    • A new Apps API is available for managing apps.

    • The Launcher page now contains apps instead of workflow contexts. Workflows are the same for all apps. See Working with Apps.

    • Apps are objects that can be imported and exported using the Objects API, and their links can be managed using the Links API.

    • App objects and their links can be viewed using the Object Explorer.

  • License management

    Fusion now requires a license. A free 30-day trial license is included in the download. You can now manage your Fusion license using the Fusion UI or REST API. For information about licensing for upgrades from 3.x to 4.0, contact Lucidworks.

  • Connectors SDK

    Fusion now comes with a Java kit for developing custom connectors. See Developing Custom Connectors.

  • Confluence connector

    This new connector retrieves data from the Atlassian Confluence Wiki CMS. You can configure this datasource to crawl pages, spaces, blog posts, comments, and attachments. See the Confluence Connector reference topic.

  • Webapps API

    This API lays the groundwork for a future release in which Fusion App Studio can interface directly with Fusion Server to customize the front-end interfaces for your apps. Though this API is not usable in this release, you can preview it in the Webapps API reference topic.

  • New index pipeline stages

  • Connector improvements

    • Sharepoint connector:

      • Performance improvements

      • Security trimming: support for groups, domain/username

      • XML parsing updates

      • Active Directory Federation Services (ADFS) support

    • Jive connector:

      • hierarchical crawl support from root Jive space

      • Security trimming: support for objects that inherit ACLs

      • Update permissions on re-crawl for Spaces

      • indexing of events, videos, and ideas

    • Jira connector:

      • Attachments and work logs now support issue reporter inheritance

    • Crawl stability for Box and Google Drive

    • Web connector: headless Firefox crawling for JS

  • New APIs for profiles

    Index profiles and query profiles are now created with definitions that include the collection, pipeline, and handler. As such, the endpoints for profiles are no longer located under the /collections endpoint; this change supports experiment management in Fusion AI. New top-level endpoints have been added in this release.

    • The endpoint at /collections/<collectionName>/query-profiles has been replaced by two new endpoints:

      • Query Profiles API (/query-profiles)

        Create, read, update, and delete query profiles.

      • Query API (/query)

        Run a query through a query profile by specifying the profile ID and appending the request with a Solr query string, as in /api/apollo/query/<id>?<solrQuery>.

    • Similarly, the endpoint at /collections/<collectionName>/index-profiles has been replaced by two new endpoints:

      • Index Profiles API (/index-profiles)

        Create, read, update, and delete index profiles.

      • Index API (/index)

        Index data through an index profile by specifying the profile ID.

  • UI redesign

    The Fusion UI has a new look and feel, and menus have been redesigned. See Fusion Server UI for a brief tour.

  • Objects API improvements

    The Objects API now outputs a .zip file for every export, and supports the new app object type. Another new object type, appkit-app, is for use in future releases.

  • Solr cluster management

  • SQL for aggregations

    By default, aggregation jobs are now defined using Spark SQL. To use the old aggregation configuration scheme in the Fusion UI, click Advanced in the Aggregation job configuration panel.

  • Multiple Hadoop connectors

  • Blob Store API changes

    • A new /blobs/convert endpoint migrates blob manifests from 3.x to 4.x format.

    • The /blobs endpoint has new parameters to support querying specific directories in the blob store: dir, recurse, and showdirs.

    • When uploading a blob via the /blobs/{id} endpoint, new parameters are supported: fusion-blob-type, commit, and update.

    • The /blobs/{id} endpoint now supports a HEAD method to retrieve blob metadata, equivalent to the /blobs/{id}/manifest endpoint.

    • The /blobs/{id}/manifest endpoint has been removed.

    • Fusion no longer supports uploading blobs via forms.

    See the Blob Store API reference topic for details.

  • New framework for connectors

    A new connectors framework allows connectors to be hosted or remote:

    • Hosted connectors are cluster-aware, so that cluster nodes are notified when connector services start up on other nodes. This allows easy scaling of the crawl process.

    • Remote connectors act as lightweight Fusion clients which can be located wherever your data is stored.

    See Connectors for more details.

    Note
    In the 4.0.0 release, the Lucidworks-provided connectors follow the classic framework. The new framework will be applied to Lucidworks-provided connectors in future releases. You can use the new Connectors SDK to develop custom connectors using the new framework.
  • Solr 7.2.1

  • ZooKeeper 3.4.10

  • Spark 2.2.1

  • Jetty 3.8.10

  • Ignite 2.3