Fusion Server Release Notes

Current Fusion Server Release: 4.2.0

Release date: 28 February 2019

Component versions:

  • Solr 7.5

  • ZooKeeper 3.4.13

  • Spark 2.3.1

  • Jetty 9.4.11.v20180605

  • Ignite 2.3.0

Past Fusion Server releases

Version Release Date Major Changes Component Versions Release Notes

4.1.2

17 December 2018

  • Performance during security trimming operations is improved in this release.

  • The Web connector now supports proxying in the default mode for JavaScript evaluation in addition to the high-performance mode.

  • JBDC datasources no longer need to be cleared in order to update the value of the convert_typeConvert type property.

  • Spark jobs now run correctly when Solr is configured to use SSL.

  • Single sign-on (SSO) now works correctly with Kerberos.

  • The Web connector now extracts <script> elements when they are selected with jsoup.

  • The Windows Share SMB 2/3 connector now uses Unix timestamp format for the lastModified_dt field instead of Windows format.

  • The Confluence connector now accepts null values for the f.includedSpaces property.

  • The Box connector now accepts a subfolder as startLink.

  • The Solr Partial Update Indexer pipeline stage now correctly deletes field values in the index when they have been deleted in SharePoint or SharePoint Online.

  • The Solr connector now respects the value of solr_request_handler/Solr request handler when its value has been modified from the default.

  • Fixes for the SharePoint and SharePoint Online connectors:

    • The connectors now respect the value of global_security_filter_cache/Enable global security filter cache when its value has been modified from the default.

    • Special characters are now correctly escaped in URLs.

  • Solr 7.4.0

  • ZooKeeper 3.4.13

  • Spark 2.3.1

  • Jetty 9.3.25.v20180904

  • Ignite 2.3.0

4.1.1

7 November 2018

  • You can now upload Javascript files to the blob store and use them in these new pipeline stages:

  • ZooKeeper has been upgraded from version 3.4.11 to 3.4.13.

  • Jetty has been upgraded from version 9.3.8.v20160314 to 9.3.25.v20180904.

  • Two new blob types were added, to support the Managed Javascript index stage and query stage:

    • file:js-index

    • file:js-query

  • Web connector improvements:

    • The connector can now send custom headers with a new addedHeaders property.

    • When the Web connector crawls a Web site and Server Name Indication (SNI) is enabled but the Web site doesn’t support it, you may receive an unrecognized name error. A new property, useIpAddressForSslConnections configures the connector to use the IP address instead of the hostname.

    • Kerberos is now supported with these new properties:

      • kerberosEnabled - Boolean; default false. If true, the connector attempts to perform Kerberos/Spnego authentication when a Web request returns a 401 WWW-Negotiate challenge. If false, the Web connector refuses to attempt Kerberos/Spnego authentication.

      • kerberosPrincipalName - Optional - Use this principal name as the logged-in Kerberos user instead of the environment’s default. If set, you must also specify a keytab with one either kerberosKeytabFilePath or kerberosKeytabBase64 (see below).

      • kerberosKeytabFile - Optional - The path to the Kerberos keytab file that contains the credentials.

      • kerberosKeytabBase64 - Optional - A base64-encoded Kerberos keytab file that contains the credentials.

      See the main topic for additional configuration details.

  • The ServiceNow connector now supports proxying.

  • The JDBC Connector has a new convert_type parameter for binary streaming data from a SQL Server

    By default, a JDBC column will automatically use the Solr field type that matches the underlying database’s type of the column. If checked, this will use the field name of the column to choose the Solr field type.

  • The Slack connector can now filter by channel names using a new channel_filters property.

  • When you import objects using the Objects API, a new context parameter specifies the name of an existing app which will be the new context for the imported objects. To support this, the Links API has two new linkType values:

    • inContextOf

    • hasContext

  • Performance improvements in proxy.

    The proxy service now reuses HTTP client instances for better throughput. Customers with high QPS expectations (~ 1000 QPS) are strongly encouraged to upgrade to 4.1.1.

  • The Fusion SQL Service (bin/sql) has been added to the default group in conf/fusion.properties. If you do not intend to run the SQL service for self-service analytics, you can remove it from the default group. See Fusion SQL service for more information on using the SQL service.

  • When exporting and importing objects with secret keys, the format is now secret.{object_type}.{object_id}.{number}.password. In the case of data source objects, the format is now secret.{object_type}.{object_id}.{datasource_type}.{number}.password.

  • The Index Profiles API no longer sets bufferDocsForSolr by default.

  • The Query Workbench now correctly displays all metadata fields.

  • JavaScript is now thread-safe in pipeline stages.

    This resolves an issue in previous releases where variables in Javascript stages that were not declared with var were shared between threads and other stages using variables with the same name.

  • The JWT security realm now works with multiple JWTs.

  • Headers missing from request.headers in the trusted-http security realm are available again. This includes:

    • fusion-user-realm-name

    • fusion-user-realm-type

    • fusion-user-name

    • fusion-user-permissions now has the role restriction assigned with the security realm.

  • Solr 7.4.0

  • ZooKeeper 3.4.13

  • Spark 2.3.1

  • Jetty 9.3.25.v20180904

  • Ignite 2.3.0

4.1.0

17 July 2018

  • App Studio integration

    Fusion App Studio is now integrated into the Fusion Server UI for seamless search UI development. You can open App Studio from the Fusion workspace by navigating to App Studio > Build new UI.

    Integrated App Studio

  • Fusion Server UI search/autocomplete

    In the Fusion workspace, you can now search objects and documentation, with autocomplete to suggest relevant results.

    UI Search and Autocomplete

  • New OneDrive connector

  • New Download Blob job

  • New log shipper and log console

  • Connector improvements

    • The Confluence connector now indexes labels for spaces and pages.

    • The Web connector can be configured to deduplicate pages whose content is identical.

    • The Slack connector is no longer deprecated.

  • UI improvements

    • The Object Explorer now supports filtering by object type.

    • Objects in the UI now include an "Explore" button that opens the Object Explorer view for that object.

    • UI panels can now be resized horizontally.

    • Text input areas can now be opened as pop-out modals for easier editing.

  • Connectors SDK

  • Import/export improvements

    • Additional parameter substitutions are now supported on object import.

  • Better heuristics for the splits_per_shard field in the Catalog API.

  • Fusion’s SparkSQL engine is no longer a beta release.

  • Ports are now preconfigured for Spark multinode communication, using these parameters in the Configurations API:

    • spark.driver.port

    • spark.executor.port

    • spark.blockManager.port

    • spark.fileserver.port

  • A request ID can now be defined in order to add context to logs.

  • The Jobs API now provides better performance.

  • The system collections named audit_logs and logs have been replaced and extended with a new system collection, system_logs.

  • Solr 7.4.0

  • ZooKeeper 3.4.11

  • Spark 2.3.1

  • Jetty 9.3.8.v20160314

  • Ignite 2.3.0

4.0.2

22 May 2018

  • A new Windows Share connector provides support for SMB 2 and 3.

    Note
    The previous Windows Share Connector is deprecated.
  • Additional options for the Zendesk connector:

    • article_containers

    • enable_security_trimming

    • index_article_attachments

    • index_articles

    • index_tickets

    • page_size

    • zendesk_account_plan

  • Web connector improvements

    • Instead of JBrowserDriver and Mozilla Firefox, the Web connector now uses PhantomJS and Google Chromium for JavaScript evaluation. When f.crawlJS/"Evaluate JavaScript" is enabled, then Fusion uses PhantomJS to evaluate JavaScript content. Lucidworks recommends also enabling f.useHighPerfJsEval/"High Performance Mode", in which case Chromium is used, providing the fastest possible JavaScript evaluation.

      Note
      This feature requires downloading a separate setup script. See the Web connector topic for instructions.
    • Support for screenshots

      When f.crawlJS/"Evaluate JavaScript" and f.useHighPerfJsEval/"High Performance Mode" are enabled, then f.takeScreenshot/"Index a screenshot of rendered page" can also be enabled.

      When this feature is enabled, Fusion takes a screenshot of each fully-rendered page and indexes it in a field called screenshot_bin.

      Important
      Your schema must specify this field as a binary field or indexing will fail. See the Web connector topic for instructions.

    New options for the Web connector:

    • f.useHighPerfJsEval

    • f.headlessBrowser

    • f.takeScreenshot

    • f.screenshotFullscreen

    • f.viewportWidth

    • f.viewportHeight

    • f.deviceScreenFactor

    • f.simulateMobile

    • f.mobileScreenWidth

    • f.mobileScreenHeight

    • f.chromeBinaryPath

    • f.quitTimeoutMs

    • f.useRequestCounter

    • f.requestCounterMinWaitMs

    • f.requestCounterMaxWaitMs

    The following configuration keys are deprecated and will be removed in a future release:

    • f.useFirefox

    • f.firefoxHeadlessBrowser

    • f.firefoxBinaryPath

  • The Jira connector now supports security trimming when single sign-on (SSO) is used to authenticate to the Jira servers.

  • The Jobs API now performs faster.

  • This release fixes a bug in the authentication proxy which incorrectly redirected certain requests when SSL was enabled.

  • Spark job IDs are now transformed so that invalid characters are replaced with underscores (_). Valid characters are A-Z, a-z, 0-9, underscore (\_), backslash (\), and hyphen (-).

  • Solr 7.2.1

  • ZooKeeper 3.4.10

  • Spark 2.2.1

  • Jetty 3.8.10

  • Ignite 2.3

4.0.1

28 February 2018

  • Import and export for index profiles and query profiles

    App import now includes index profiles and query profiles. These can also be exported and imported separately with the Objects API, using the index-profile and query-profile object types.

  • Solr 7.2.1

  • ZooKeeper 3.4.10

  • Spark 2.2.1

  • Jetty 3.8.10

  • Ignite 2.3

4.0.0

21 February 2018

  • Fusion AI

    Fusion’s artificial intelligence and machine learning features are now licensed separately, as the new Fusion AI product. These features are now enabled only with a Fusion AI license:

    • Signals

    • App Insights (new in 4.0)

    • Smart jobs

      • ALS Recommender Jobs

      • Bisecting KMeans Clustering Jobs

      • Cluster Labeling Jobs

      • Co-Occurrence Similarity Jobs

      • Collection Analysis Jobs

      • Document Clustering Jobs

      • Ground Truth Jobs

      • Head/Tail Analysis Jobs (new in 4.0)

      • Item Similarity Recommender Jobs

      • Levenshtein Spell Checking Jobs

      • Logistic Regression Classifier Training Jobs

      • Matrix Decomposition-Based Query-Query Similarity Jobs

      • Outlier Detection Jobs

      • Random Forest Classifier Training Jobs

      • Ranking Metrics Jobs

      • SQL-Based Experiment Metric Jobs

      • Statistically Interesting Phrases Jobs

      • Token and Phrase Spell Correction Jobs (new in 4.0)

    • REST APIs

      • Experiments API

      • Recommendations API

      • Signals Aggregator API

      • Signals API

    • Index pipeline stages

      • Machine Learning index stage

      • OpenNLP NER Extractor index stage

      • Signal Formatter index stage

      • Update Experiment index stage

    • Query pipeline stages

      • Advanced Boosting query stage

      • Analytics Catalog query stage

      • Experiment query stage

      • Machine Learning query stage

      • More Like This query stage

      • Recommend Items for Item query stage

      • Recommend Items for User query stage

      • Recommendation Boosting query stage

  • Apps

    Fusion apps provide tailored search functionality to specific groups of users.

    An app is a named set of linked objects, including collections, datasources, index and query pipelines, index and query profiles, parsers, and more. Using roles and security realms, you can define security on a per-app basis.

    • A new Apps API is available for managing apps.

    • The Launcher page now contains apps instead of workflow contexts. Workflows are the same for all apps. See Working with Apps.

    • Apps are objects that can be imported and exported using the Objects API, and their links can be managed using the Links API.

    • App objects and their links can be viewed using the Object Explorer.

  • License management

    Fusion now requires a license. A free 30-day trial license is included in the download. You can now manage your Fusion license using the Fusion UI or REST API. For information about licensing for upgrades from 3.x to 4.0, contact Lucidworks.

  • Connectors SDK

    Fusion now comes with a Java kit for developing custom connectors. See Developing Custom Connectors.

  • Confluence connector

    This new connector retrieves data from the Atlassian Confluence Wiki CMS. You can configure this datasource to crawl pages, spaces, blog posts, comments, and attachments. See the Confluence Connector reference topic.

  • Webapps API

    This API lays the groundwork for a future release in which Fusion App Studio can interface directly with Fusion Server to customize the front-end interfaces for your apps. Though this API is not usable in this release, you can preview it in the Webapps API reference topic.

  • New index pipeline stages

  • Connector improvements

    • Sharepoint connector:

      • Performance improvements

      • Security trimming: support for groups, domain/username

      • XML parsing updates

      • Active Directory Federation Services (ADFS) support

    • Jive connector:

      • hierarchical crawl support from root Jive space

      • Security trimming: support for objects that inherit ACLs

      • Update permissions on re-crawl for Spaces

      • indexing of events, videos, and ideas

    • Jira connector:

      • Attachments and work logs now support issue reporter inheritance

    • Crawl stability for Box and Google Drive

    • Web connector: headless Firefox crawling for JS

  • New APIs for profiles

    Index profiles and query profiles are now created with definitions that include the collection, pipeline, and handler. As such, the endpoints for profiles are no longer located under the /collections endpoint; this change supports experiment management in Fusion AI. New top-level endpoints have been added in this release.

    • The endpoint at /collections/<collectionName>/query-profiles has been replaced by two new endpoints:

      • Query Profiles API (/query-profiles)

        Create, read, update, and delete query profiles.

      • Query API (/query)

        Run a query through a query profile by specifying the profile ID and appending the request with a Solr query string, as in /api/apollo/query/<id>?<solrQuery>.

    • Similarly, the endpoint at /collections/<collectionName>/index-profiles has been replaced by two new endpoints:

      • Index Profiles API (/index-profiles)

        Create, read, update, and delete index profiles.

      • Index API (/index)

        Index data through an index profile by specifying the profile ID.

  • UI redesign

    The Fusion UI has a new look and feel, and menus have been redesigned. See Fusion Server UI for a brief tour.

  • Objects API improvements

    The Objects API now outputs a .zip file for every export, and supports the new app object type. Another new object type, appkit-app, is for use in future releases.

  • Solr cluster management

  • SQL for aggregations

    By default, aggregation jobs are now defined using Spark SQL. To use the old aggregation configuration scheme in the Fusion UI, click Advanced in the Aggregation job configuration panel.

  • Multiple Hadoop connectors

  • Blob Store API changes

    • A new /blobs/convert endpoint migrates blob manifests from 3.x to 4.x format.

    • The /blobs endpoint has new parameters to support querying specific directories in the blob store: dir, recurse, and showdirs.

    • When uploading a blob via the /blobs/{id} endpoint, new parameters are supported: fusion-blob-type, commit, and update.

    • The /blobs/{id} endpoint now supports a HEAD method to retrieve blob metadata, equivalent to the /blobs/{id}/manifest endpoint.

    • The /blobs/{id}/manifest endpoint has been removed.

    • Fusion no longer supports uploading blobs via forms.

    See the Blob Store API reference topic for details.

  • New framework for connectors

    A new connectors framework allows connectors to be hosted or remote:

    • Hosted connectors are cluster-aware, so that cluster nodes are notified when connector services start up on other nodes. This allows easy scaling of the crawl process.

    • Remote connectors act as lightweight Fusion clients which can be located wherever your data is stored.

    See Connectors for more details.

    Note
    In the 4.0.0 release, the Lucidworks-provided connectors follow the classic framework. The new framework will be applied to Lucidworks-provided connectors in future releases. You can use the new Connectors SDK to develop custom connectors using the new framework.
  • Solr 7.2.1

  • ZooKeeper 3.4.10

  • Spark 2.2.1

  • Jetty 3.8.10

  • Ignite 2.3