Fusion Server Release Notes

Current Fusion Server Release: 4.2.2

Release date: 17 May 2019

Component versions:

  • Solr 7.5

  • ZooKeeper 3.4.13

  • Spark 2.3.1

  • Jetty 9.4.11.v20180605

  • Ignite 2.3.0

Past Fusion Server releases

Version Release Date Major Changes Component Versions Release Notes

4.2.1

5 April 2019

  • The Users page (System > Access Control > Users) now displays the security realm to which each user is assigned.

  • SharePoint and SharePoint Online connector improvements:

    • The SharePoint and SharePoint Online connectors now delete indexed documents after they are deleted from the SharePoint source.

    • The SharePoint connector now retries document fetches after network errors.

    • Addressed an issue with fetching and indexing of Sharepoint list item attachments.

    • Addressed an issue with running the Sharepoint connector without an associated parser configuration.

    • Addressed an incremental crawl scenario for deleted sites.

  • Confluence connector improvements:

    • Fixed an issue with the handling of Confluence 5.6 archive API endpoint.

    • Added security trimming support for the "confluence-administrators" group.

    • Fixed an issue with incremental ACL updates for changes with _lw_confluence_anonymous permission setting

  • The HTML parser now parses documents whose Mime-Type value does not exactly match text/html, such as text/html; charset=utf-8 and so on. This allows it to parse .aspx pages from SharePoint datasources, for example.

  • The Windows Share connector now retrieves document owner metadata.

  • For connectors that use the Crawl database type/crawlDBType parameter, the default value is now on-disk instead of in-memory in order to support larger crawls by default.

  • Addressed an issue with persisting proxy settings during Javascript evaluation in the Web connector.

  • Addressed an incremental update issue due to unnecessary serialization of the ID field in the MongoDB connector.

  • Updated the Zendesk connector to use RequestEntityProcessing.BUFFERED instead of .CHUNKED to avoid a documented Jersey issue.

  • DevOps Center improvements:

    • In the Log Viewer, exported data is now constrained by all selected filters in addition to the time range selector.

    • The time range selector in the Log Viewer now behaves more consistently.

    • Fixed some issues with filtering in the Hosts tab and the Log Viewer.

    • In the Log Viewer, log messages whose level is DEBUG are now distinguished by an icon for consistency with other log levels.

  • The size of the system_logs collection was significantly reduced by changing the log level for marking user.sessions meter to DEBUG.

  • You can now use the query pipeline to set the status code for Fusion’s response.

  • SQL service improvements:

    • Fusion’s SQL service now supports non-phrase predicates.

    • Results can now be sorted by the score of the underlying Solr query.

    • Random sampling is now supported.

    • The service now provides more consistent performant results from unlimited selects.

    • Fixed an issue in 4.2.0 which cause unlimited "group by" aggregations to use a bucket size limit of 1.

  • Fusion now captures the same metrics about the connectors-rpc services as it does about the connectors-classic service.

  • A variety of minor UI issues were fixed.

  • FS (V2) connector saves Item metadata in Crawldb - recrawls work as expected (the new/delete/update items will be picked up) - On migrations, it is suggested to clear the datasource and start with a new crawl, but it is not required.

  • Now when you upload a .war file to the App Studio interface, the View Published UI button remains available, resolving a known issue in 4.2.0.

  • Logging out using Chrome version 73.0.3683.75 no longer causes an infinite loop; this resolves a known issue in 4.2.0.

  • You can now re-install a connector using the same ID but a different filename than the original.

  • Several bug fixes resolve issues with ACL metadata using the Confluence connector.

  • Better permissions checks when importing jobs and blobs.

  • When adding support for business rules to migrated apps, it is no longer necessary to specify the app.id; it is inferred automatically from the current app context.

  • Exporting an app with deep=false now exports only the app definition without the linked objects.

  • Fixed an issue which sometimes prevented "chained" jobs from firing, that is, jobs that are triggered by the success or failure of another job.

  • Fixed an issue that prevented users from disabling the Asynchronous Execution Config/asyncConfig option in some pipeline stages.

  • In the Solr Query pipeline stage, configuring one or more requestHandlers now blocks usage of the select request handler. To configure usage of multiple request handlers including select, add select to your requestHandlers.

  • Fixed an issue which prevented the parsing of paths containing backslashes in fusion.properties on Windows.

  • Fixed an issue when using the default init/systemd/install.sh systemd service which cause all Fusion services to fail when one service failed. To avoid this issue, use /opt/fusion/4.2.1/init/systemd/install-services.sh instead; see the instructions at fusion/4.2.1/packaging/init/systemd/README.

  • Fixed an issue which caused intermittent Zendesk connector failures with a ConnectionClosedException: Premature end of chunk coded message body: closing chunk expected message.

  • Solr 7.5

  • ZooKeeper 3.4.13

  • Spark 2.3.1

  • Jetty 9.4.11.v20180605

  • Ignite 2.3.0

4.1.3

12 April 2019

  • The Web connector and the SharePoint connector now support multiple start links using a new Bulk Start Links/f.bulkStartLinks property.

  • Additional Web connector improvements:

    • The connector now supports HTTPS when configured for proxying in high-performance mode.

    • The connector now supports inclusive/exclusive regular expressions after a ? in URL values.

    • New configuration properties:

      • Maintain Cookies for No-Auth Crawls/f.maintainCookies

        If you are not using authentication, then by default cookies are not stored in between web requests (stateless). If checked, cookies will be maintained between requests during the web crawl even when you are not using authentication. If you are using authentication, this checkbox has no effect on the crawl and can be ignored.

      • Obey link nofollow attributes/f.obeyLinkNofollow

        If true, rel='nofollow' on links are obeyed.

      • Obey robots meta tags and headers/f.obeyRobotsMeta

        If true, rules like noindex, nofollow, and others found in a robots meta tag on a page or in the headers of the HTTP response are obeyed.

  • New Box.com connector configuration properties:

    • API Connection Timeout (ms)/f.fs.connectTimeoutMs

      The Box API connection timeout in milliseconds.

    • API Read Timeout (ms)/f.fs.readTimeoutMs

      The Box API read timeout in milliseconds.

  • The Confluence connector has new security trimming configuration properties, User group cache limit/f.cacheUserGroupLimit. Only applicable when cacheUserGroups is enabled, this will limit the number of users who will have their groups cached. This is used for testing purposes only. The default of -1 will cause all users to be cached.

  • Diagnostic mode is now available for the MongoDB connector.

  • SharePoint and SharePoint Online connector improvements:

    • New configuration properties:

      • Retry requests/f.enable_retry

        Enable this to make the connector automatically try again if a request to sharepoint fails unexpectedly.

      • Number of retries/f.retry_attempts

        If Retry requests is enabled, this number determines how many times the connector should retry a request before giving up.

    • Addressed an issue with running the Sharepoint connector without an associated parser configuration.

    • The connectors now delete indexed documents after they are deleted from the SharePoint source.

  • The HTML parser now parses documents whose Mime-Type value does not exactly match text/html, such as text/html; charset=utf-8 and so on. This allows it to parse .aspx pages from SharePoint datasources, for example.

  • Fusion now redirects to the login page when you log out on Chrome 73.

  • Migrated objects are now correctly named as collectionName_objectName instead of objectName.

  • The Query Workbench now loads datasets that contain large numbers of fields.

  • A display bug was fixed in the Object Explorer.

  • The Confluence connector now updates document permissions when their space permissions have changed.

  • Fixed an issue which cause a Could not read triggers for job error for some scheduled jobs.

  • Fixed a problem where a job scheduled to run "After Another Job Completes" would fail to run when the job scheduler leadership changed.

  • Fixed an issue with incremental ACL updates for changes with _lw_confluence_anonymous permission setting in the Confluence connector.

  • Solr 7.4.0

  • ZooKeeper 3.4.13

  • Spark 2.3.1

  • Jetty 9.3.25.v20180904

  • Ignite 2.3.0

4.2.0

28 February 2019

Connector

Default value

Web

false

Sharepoint

true

Sharepoint Online

true

SMB2

false

Box

false

Google Drive

false

Dropbox

false

Local Filesystem

false

  • Several connectors have a new Enable Plugin Parsing/pluginParsing parameter. When it is enabled, the connector parses raw content before streaming it to the index pipeline. The following connectors support this parameter:

    • Local Filesystem

    • OneDrive

    • Sitecore

    • Windows Share (SMB 2/3)

      • The dashboards framework has been upgraded to Banana 1.6.23. See the Banana release notes.

      • For tighter security, CORS is now disallowed by default. You can enable it, if needed, by editing the proxy.corsAllowOrigin property in conf/fusion.properties.

      • The Synonyms UI is no longer available. See the new Synonym Detection feature, available with a Fusion AI license.

      • The synonyms collection has been replaced with the new query_rewrite collection.

      • The Recommendations API is deprecated and will be removed in a future release.

  • Solr 7.5

  • ZooKeeper 3.4.13

  • Spark 2.3.1

  • Jetty 9.4.11.v20180605

  • Ignite 2.3.0

4.1.2

17 December 2018

  • Performance during security trimming operations is improved in this release.

  • The Web connector now supports proxying in the default mode for JavaScript evaluation in addition to the high-performance mode.

  • JBDC datasources no longer need to be cleared in order to update the value of the convert_typeConvert type property.

  • Spark jobs now run correctly when Solr is configured to use SSL.

  • Single sign-on (SSO) now works correctly with Kerberos.

  • The Web connector now extracts <script> elements when they are selected with jsoup.

  • The Windows Share SMB 2/3 connector now uses Unix timestamp format for the lastModified_dt field instead of Windows format.

  • The Confluence connector now accepts null values for the f.includedSpaces property.

  • The Box connector now accepts a subfolder as startLink.

  • The Solr Partial Update Indexer pipeline stage now correctly deletes field values in the index when they have been deleted in SharePoint or SharePoint Online.

  • The Solr connector now respects the value of solr_request_handler/Solr request handler when its value has been modified from the default.

  • Fixes for the SharePoint and SharePoint Online connectors:

    • The connectors now respect the value of global_security_filter_cache/Enable global security filter cache when its value has been modified from the default.

    • Special characters are now correctly escaped in URLs.

  • Solr 7.4.0

  • ZooKeeper 3.4.13

  • Spark 2.3.1

  • Jetty 9.3.25.v20180904

  • Ignite 2.3.0

4.1.1

7 November 2018

  • You can now upload Javascript files to the blob store and use them in these new pipeline stages:

  • ZooKeeper has been upgraded from version 3.4.11 to 3.4.13.

  • Jetty has been upgraded from version 9.3.8.v20160314 to 9.3.25.v20180904.

  • Two new blob types were added, to support the Managed Javascript index stage and query stage:

    • file:js-index

    • file:js-query

  • Web connector improvements:

    • The connector can now send custom headers with a new addedHeaders property.

    • When the Web connector crawls a Web site and Server Name Indication (SNI) is enabled but the Web site doesn’t support it, you may receive an unrecognized name error. A new property, useIpAddressForSslConnections configures the connector to use the IP address instead of the hostname.

    • Kerberos is now supported with these new properties:

      • kerberosEnabled - Boolean; default false. If true, the connector attempts to perform Kerberos/Spnego authentication when a Web request returns a 401 WWW-Negotiate challenge. If false, the Web connector refuses to attempt Kerberos/Spnego authentication.

      • kerberosPrincipalName - Optional - Use this principal name as the logged-in Kerberos user instead of the environment’s default. If set, you must also specify a keytab with one either kerberosKeytabFilePath or kerberosKeytabBase64 (see below).

      • kerberosKeytabFile - Optional - The path to the Kerberos keytab file that contains the credentials.

      • kerberosKeytabBase64 - Optional - A base64-encoded Kerberos keytab file that contains the credentials.

      See the main topic for additional configuration details.

  • The ServiceNow connector now supports proxying.

  • The JDBC Connector has a new convert_type parameter for binary streaming data from a SQL Server

    By default, a JDBC column will automatically use the Solr field type that matches the underlying database’s type of the column. If checked, this will use the field name of the column to choose the Solr field type.

  • The Slack connector can now filter by channel names using a new channel_filters property.

  • When you import objects using the Objects API, a new context parameter specifies the name of an existing app which will be the new context for the imported objects. To support this, the Links API has two new linkType values:

    • inContextOf

    • hasContext

  • Performance improvements in proxy.

    The proxy service now reuses HTTP client instances for better throughput. Customers with high QPS expectations (~ 1000 QPS) are strongly encouraged to upgrade to 4.1.1.

  • The Fusion SQL Service (bin/sql) has been added to the default group in conf/fusion.properties. If you do not intend to run the SQL service for self-service analytics, you can remove it from the default group. See Fusion SQL service for more information on using the SQL service.

  • When exporting and importing objects with secret keys, the format is now secret.{object_type}.{object_id}.{number}.password. In the case of data source objects, the format is now secret.{object_type}.{object_id}.{datasource_type}.{number}.password.

  • The Index Profiles API no longer sets bufferDocsForSolr by default.

  • The Query Workbench now correctly displays all metadata fields.

  • JavaScript is now thread-safe in pipeline stages.

    This resolves an issue in previous releases where variables in Javascript stages that were not declared with var were shared between threads and other stages using variables with the same name.

  • The JWT security realm now works with multiple JWTs.

  • Headers missing from request.headers in the trusted-http security realm are available again. This includes:

    • fusion-user-realm-name

    • fusion-user-realm-type

    • fusion-user-name

    • fusion-user-permissions now has the role restriction assigned with the security realm.

  • Solr 7.4.0

  • ZooKeeper 3.4.13

  • Spark 2.3.1

  • Jetty 9.3.25.v20180904

  • Ignite 2.3.0

4.1.0

17 July 2018

  • App Studio integration

    Fusion App Studio is now integrated into the Fusion Server UI for seamless search UI development. You can open App Studio from the Fusion workspace by navigating to App Studio > Build new UI.

    Integrated App Studio

  • Fusion Server UI search/autocomplete

    In the Fusion workspace, you can now search objects and documentation, with autocomplete to suggest relevant results.

    UI Search and Autocomplete

  • New OneDrive connector

  • New Download Blob job

  • New log shipper and log console

  • Connector improvements

    • The Confluence connector now indexes labels for spaces and pages.

    • The Web connector can be configured to deduplicate pages whose content is identical.

    • The Slack connector is no longer deprecated.

  • UI improvements

    • The Object Explorer now supports filtering by object type.

    • Objects in the UI now include an "Explore" button that opens the Object Explorer view for that object.

    • UI panels can now be resized horizontally.

    • Text input areas can now be opened as pop-out modals for easier editing.

  • Connectors SDK

  • Import/export improvements

    • Additional parameter substitutions are now supported on object import.

  • Better heuristics for the splits_per_shard field in the Catalog API.

  • Fusion’s SparkSQL engine is no longer a beta release.

  • Ports are now preconfigured for Spark multinode communication, using these parameters in the Configurations API:

    • spark.driver.port

    • spark.executor.port

    • spark.blockManager.port

    • spark.fileserver.port

  • A request ID can now be defined in order to add context to logs.

  • The Jobs API now provides better performance.

  • The system collections named audit_logs and logs have been replaced and extended with a new system collection, system_logs.

  • Solr 7.4.0

  • ZooKeeper 3.4.11

  • Spark 2.3.1

  • Jetty 9.3.8.v20160314

  • Ignite 2.3.0

4.0.2

22 May 2018

  • A new Windows Share connector provides support for SMB 2 and 3.

    Note
    The previous Windows Share Connector is deprecated.
  • Additional options for the Zendesk connector:

    • article_containers

    • enable_security_trimming

    • index_article_attachments

    • index_articles

    • index_tickets

    • page_size

    • zendesk_account_plan

  • Web connector improvements

    • Instead of JBrowserDriver and Mozilla Firefox, the Web connector now uses PhantomJS and Google Chromium for JavaScript evaluation. When f.crawlJS/"Evaluate JavaScript" is enabled, then Fusion uses PhantomJS to evaluate JavaScript content. Lucidworks recommends also enabling f.useHighPerfJsEval/"High Performance Mode", in which case Chromium is used, providing the fastest possible JavaScript evaluation.

      Note
      This feature requires downloading a separate setup script. See the Web connector topic for instructions.
    • Support for screenshots

      When f.crawlJS/"Evaluate JavaScript" and f.useHighPerfJsEval/"High Performance Mode" are enabled, then f.takeScreenshot/"Index a screenshot of rendered page" can also be enabled.

      When this feature is enabled, Fusion takes a screenshot of each fully-rendered page and indexes it in a field called screenshot_bin.

      Important
      Your schema must specify this field as a binary field or indexing will fail. See the Web connector topic for instructions.

    New options for the Web connector:

    • f.useHighPerfJsEval

    • f.headlessBrowser

    • f.takeScreenshot

    • f.screenshotFullscreen

    • f.viewportWidth

    • f.viewportHeight

    • f.deviceScreenFactor

    • f.simulateMobile

    • f.mobileScreenWidth

    • f.mobileScreenHeight

    • f.chromeBinaryPath

    • f.quitTimeoutMs

    • f.useRequestCounter

    • f.requestCounterMinWaitMs

    • f.requestCounterMaxWaitMs

    The following configuration keys are deprecated and will be removed in a future release:

    • f.useFirefox

    • f.firefoxHeadlessBrowser

    • f.firefoxBinaryPath

  • The Jira connector now supports security trimming when single sign-on (SSO) is used to authenticate to the Jira servers.

  • The Jobs API now performs faster.

  • This release fixes a bug in the authentication proxy which incorrectly redirected certain requests when SSL was enabled.

  • Spark job IDs are now transformed so that invalid characters are replaced with underscores (_). Valid characters are A-Z, a-z, 0-9, underscore (\_), backslash (\), and hyphen (-).

  • Solr 7.2.1

  • ZooKeeper 3.4.10

  • Spark 2.2.1

  • Jetty 3.8.10

  • Ignite 2.3.0

4.0.1

28 February 2018

  • Import and export for index profiles and query profiles

    App import now includes index profiles and query profiles. These can also be exported and imported separately with the Objects API, using the index-profile and query-profile object types.

  • Solr 7.2.1

  • ZooKeeper 3.4.10

  • Spark 2.2.1

  • Jetty 3.8.10

  • Ignite 2.3.0

4.0.0

21 February 2018

  • Fusion AI

    Fusion’s artificial intelligence and machine learning features are now licensed separately, as the new Fusion AI product. These features are now enabled only with a Fusion AI license:

    • Signals

    • App Insights (new in 4.0)

    • Smart jobs

      • ALS Recommender Jobs

      • Bisecting KMeans Clustering Jobs

      • Cluster Labeling Jobs

      • Co-Occurrence Similarity Jobs

      • Collection Analysis Jobs

      • Document Clustering Jobs

      • Ground Truth Jobs

      • Head/Tail Analysis Jobs (new in 4.0)

      • Item Similarity Recommender Jobs

      • Levenshtein Spell Checking Jobs

      • Logistic Regression Classifier Training Jobs

      • Matrix Decomposition-Based Query-Query Similarity Jobs

      • Outlier Detection Jobs

      • Random Forest Classifier Training Jobs

      • Ranking Metrics Jobs

      • SQL-Based Experiment Metric Jobs

      • Statistically Interesting Phrases Jobs

      • Token and Phrase Spell Correction Jobs (new in 4.0)

    • REST APIs

      • Experiments API

      • Recommendations API

      • Signals Aggregator API

      • Signals API

    • Index pipeline stages

      • Machine Learning index stage

      • OpenNLP NER Extractor index stage

      • Signal Formatter index stage

      • Update Experiment index stage

    • Query pipeline stages

      • Advanced Boosting query stage

      • Analytics Catalog query stage

      • Experiment query stage

      • Machine Learning query stage

      • More Like This query stage

      • Recommend Items for Item query stage

      • Recommend Items for User query stage

      • Recommendation Boosting query stage

  • Apps

    Fusion apps provide tailored search functionality to specific groups of users.

    An app is a named set of linked objects, including collections, datasources, index and query pipelines, index and query profiles, parsers, and more. Using roles and security realms, you can define security on a per-app basis.

    • A new Apps API is available for managing apps.

    • The Launcher page now contains apps instead of workflow contexts. Workflows are the same for all apps. See Working with Apps.

    • Apps are objects that can be imported and exported using the Objects API, and their links can be managed using the Links API.

    • App objects and their links can be viewed using the Object Explorer.

  • License management

    Fusion now requires a license. A free 30-day trial license is included in the download. You can now manage your Fusion license using the Fusion UI or REST API. For information about licensing for upgrades from 3.x to 4.0, contact Lucidworks.

  • Connectors SDK

    Fusion now comes with a Java kit for developing custom connectors. See Developing Custom Connectors.

  • Confluence connector

    This new connector retrieves data from the Atlassian Confluence Wiki CMS. You can configure this datasource to crawl pages, spaces, blog posts, comments, and attachments. See the Confluence Connector reference topic.

  • Webapps API

    This API lays the groundwork for a future release in which Fusion App Studio can interface directly with Fusion Server to customize the front-end interfaces for your apps. Though this API is not usable in this release, you can preview it in the Webapps API reference topic.

  • New index pipeline stages

  • Connector improvements

    • Sharepoint connector:

      • Performance improvements

      • Security trimming: support for groups, domain/username

      • XML parsing updates

      • Active Directory Federation Services (ADFS) support

    • Jive connector:

      • hierarchical crawl support from root Jive space

      • Security trimming: support for objects that inherit ACLs

      • Update permissions on re-crawl for Spaces

      • indexing of events, videos, and ideas

    • Jira connector:

      • Attachments and work logs now support issue reporter inheritance

    • Crawl stability for Box and Google Drive

    • Web connector: headless Firefox crawling for JS

  • New APIs for profiles

    Index profiles and query profiles are now created with definitions that include the collection, pipeline, and handler. As such, the endpoints for profiles are no longer located under the /collections endpoint; this change supports experiment management in Fusion AI. New top-level endpoints have been added in this release.

    • The endpoint at /collections/<collectionName>/query-profiles has been replaced by two new endpoints:

      • Query Profiles API (/query-profiles)

        Create, read, update, and delete query profiles.

      • Query API (/query)

        Run a query through a query profile by specifying the profile ID and appending the request with a Solr query string, as in /api/apollo/query/<id>?<solrQuery>.

    • Similarly, the endpoint at /collections/<collectionName>/index-profiles has been replaced by two new endpoints:

      • Index Profiles API (/index-profiles)

        Create, read, update, and delete index profiles.

      • Index API (/index)

        Index data through an index profile by specifying the profile ID.

  • UI redesign

    The Fusion UI has a new look and feel, and menus have been redesigned. See Fusion Server UI for a brief tour.

  • Objects API improvements

    The Objects API now outputs a .zip file for every export, and supports the new app object type. Another new object type, appkit-app, is for use in future releases.

  • Solr cluster management

  • SQL for aggregations

    By default, aggregation jobs are now defined using Spark SQL. To use the old aggregation configuration scheme in the Fusion UI, click Advanced in the Aggregation job configuration panel.

  • Multiple Hadoop connectors

  • Blob Store API changes

    • A new /blobs/convert endpoint migrates blob manifests from 3.x to 4.x format.

    • The /blobs endpoint has new parameters to support querying specific directories in the blob store: dir, recurse, and showdirs.

    • When uploading a blob via the /blobs/{id} endpoint, new parameters are supported: fusion-blob-type, commit, and update.

    • The /blobs/{id} endpoint now supports a HEAD method to retrieve blob metadata, equivalent to the /blobs/{id}/manifest endpoint.

    • The /blobs/{id}/manifest endpoint has been removed.

    • Fusion no longer supports uploading blobs via forms.

    See the Blob Store API reference topic for details.

  • New framework for connectors

    A new connectors framework allows connectors to be hosted or remote:

    • Hosted connectors are cluster-aware, so that cluster nodes are notified when connector services start up on other nodes. This allows easy scaling of the crawl process.

    • Remote connectors act as lightweight Fusion clients which can be located wherever your data is stored.

    See Connectors for more details.

    Note
    In the 4.0.0 release, the Lucidworks-provided connectors follow the classic framework. The new framework will be applied to Lucidworks-provided connectors in future releases. You can use the new Connectors SDK to develop custom connectors using the new framework.
  • Solr 7.2.1

  • ZooKeeper 3.4.10

  • Spark 2.2.1

  • Jetty 3.8.10

  • Ignite 2.3.0