Fusion Server 4.2.0 Release Notes

​

New features

• Dynamic, real-time system visualizations with the new DevOps Center](/docs/4.2/fusion-server/concepts/system/devops-center) The DevOps Center generates real-time dashboards for visualizing metrics throughout your Fusion system, plus a log viewer where you can explore events and focus on specific event types. You can also export metrics and events for any timeframe, in CSV format, for external analysis.
• Faster query pipeline performance with asynchronous stage processing Query pipelines can now be forked for parallel processing so that faster stages can proceed while requests to external resources wait for their responses.
  In this release, you can enable asynchronous execution in the following query stages:
  • Active Directory Security Trimming
  • Apply Rules
  • Boost with Signals
  • JDBC Lookup
  • Security Trimming
  • Solr Subquery
  A new Merge Async Results stage joins the forked pipeline before the final Solr stage.
  For complete instructions, see Query Pipelines.
• Sitecore support The new Sitecore connector provides full crawl and incremental crawl support for versions 8.x and 9.x of the popular Sitecore CMS, indexing both document content and metadata.
• A new Dropbox connector supports the latest Dropbox API.
• New Solr Update XML parser This is a simple parser for Solr’s various update formats (XML, CSV, JSON, and javaBin).
• New query pipeline stages These new query pipeline stages support new Fusion AI 4.2.0 features as well as the asynchronous pipeline processing described above.
  • Query rewriting stages:
    • Text Tagger
    • Apply Rules
  • Response rewriting stages:
    • Modify Response with Rules
    • Response Document Exclusion Stage
    • Response Document Field Redaction Stage
    • Response Pairwise Swap
    • Response Shuffle Stage
  • Merge Async Results stage for asynchronous processing (see above).
    The default query pipeline now includes the new Text Tagger, Apply Rules, and Modify Response with Rules stages.
• New collections
  • query_rewrite_staging Rules and certain Spark job results are written to this collection temporarily. See Query Rewriting for details.
  • query_rewrite Query pipelines read rules and job results from this collection in order to perform query rewriting. Docs are migrated to this collection from the query_rewrite_staging collection.
  • job_reports Job histories are now written to this collection.
  • _user_prefs This collection stores App Studio social data, such as user tags, bookmarks, and so on.
  • system_monitor The new system metrics used for the DevOps Center are written to this collection. These new metrics replace the metrics previously written to the system_metrics collection.
• New REST APIs and endpoints
  • New Custom Rules API
  • New Query Rewrite API
  • New Webapps Admin API
  • New Webapps Appkit API
  The Webapps API is deprecated in favor of the new Webapps Appkit API.
  • /index-pipelines/{id}/collections/{collection}/indexMultiple submits a set of documents to an index pipeline.
  • /spark/reports/{job} gets the job results from a specific job.
  • /webapps/{id}/war/manifest gets the .war file manifest for specified Web app.

​

Improvements

• Broader access to the Object Explorer A new Explore button appears in configuration panels for Fusion objects that can be viewed in the Object Explorer. Click the button to see the object’s relationships to other Fusion objects.
• Use external management tools to control Fusion configurations Many of the values in conf/fusion.properties can now be set using environment variables, enabling you to set them using systemd, Docker, Kubernetes, and so on. Default values are also provided. For example, in api.port = ${API_PORT:-8765}, the default value is 8765 unless API_PORT is defined.
  Note that ZOOKEEPER_PORT cannot not be used, and value of zookeeper.port in fusion.properties must be the same as the value of clientPort in conf/zookeeper/zoo.cfg.
• Improved connectors functionality
  • Improved incremental crawl performance across all connectors with MapDB upgrade.
  • Kerberos support in the Jive connector.
  • javascript evaluation.
  • Web connector now supports website authentication credentials files in container path.
  • javascript evaluation. for crawling websites.
  • SharePoint V1 connector now have bulk start link URL list import.
  • Confluence connector now supports API token based authentication.
  • The SMB2/3 Connector has improved support for crawling distributed file systems.
  • The SMB2/3 Connector now saves the original file path and redirected file paths when crawling distributed file systems.
  • JDBC V1 connector has improved settings for managing index commits during crawls.
  • The OneDrive connector can now crawl user-specific drives.
  • Some connectors now give you the option to index metadata about documents that were discarded because they were too large or too small using the new f.index_items_discarded/Index discarded document metadata parameter. A new field, _lw_skipped_reason_s, indicates the reason that the document was skipped during indexing. The new key is available for these connectors:
    • Web: default = false
    • SharePoint: default = true
    • SharePoint Online: default = true
    • SMB2: default = false
    • Box: default = false
    • Google Drive: default = false
    • Dropbox: default = false
    • Local Filesystem: default = false
  • Several connectors have a new Enable Plugin Parsing/pluginParsing parameter. When it is enabled, the connector parses raw content before streaming it to the index pipeline. The following connectors support this parameter:
    • Local Filesystem
    • OneDrive
    • Sitecore
    • Windows Share (SMB 2/3)
• The dashboards framework has been upgraded to Banana 1.6.23. See the Banana release notes.
• For tighter security, CORS is now disallowed by default. You can enable it, if needed, by editing the proxy.corsAllowOrigin property in conf/fusion.properties.

​

Other changes

• The Synonyms UI is no longer available. See the new Synonym Detection feature, available with a Fusion AI license.
• The synonyms collection has been replaced with the new query_rewrite collection.
• The Recommendations API is deprecated and will be removed in a future release.

​

Known issues

• When under load, the Fusion proxy service can occasionally become stuck, causing user authentication to fail. This is the result of the proxy InputStream failing to close properly.
  An upgrade to Fusion 4.2.4 is required to fix this issue. See Upgrade Fusion Server 4.2.x to 4.2.y.
  Upgrade Fusion Server 4.2.x to 4.2.y

  ​

  Introduction

  This article describes how to perform the following upgrade:

  • From version: Fusion 4.2.x
  • To version: Fusion 4.2.y

  Only specific version-to-version upgrade sequences are supported. Some upgrades require multiple steps.

  For more information, see Fusion 4.x.
  For Fusion 3.1 and later releases, a migrator is available for upgrading Fusion.
  During the upgrade process, the migrator uses a properties file. After downloading and installing the migrator, the properties files is in the /opt/lucidworks/fusion/4.2.x/var/upgrade directory (on Unix or MacOS) or the C:\lucidworks\fusion\4.2.*x\var\upgrade\ directory (on Windows). The file names reference the versions you are upgrading from and to. For example:

  • To upgrade 4.2.3 to 4.2.5, the migrator uses the 4.2.x-4.2.x.properties file.

  Migration entails down time and multiple starts and stops of Fusion services. Plan accordingly, especially in terms of disabling external load balancers or monitors that might react adversely to the starts and stops.
  Download the latest migrator immediately before upgrading. This helps ensure that the upgrade goes smoothly.

  The newer Fusion instance must be newly untarred and never started.

  ​

  About the upgrade

  This section describes how connectors, object migrations, and signals are migrated during an upgrade.

  ​

  Connectors

  In Fusion 3.1.0 and later, only a subset of connectors are included by default.The migrator detects which connectors were used in the older version of Fusion, and installs them automatically in Fusion 4.0.y. This step requires an Internet connection. If no connection is available, then download the connectors from Fusion 4.x Connector Downloads and install them as bootstrap plugins.If a connector to be upgraded was not available during the upgrade, then a message in /opt/lucidworks/fusion/3.1.x/var/upgrade/tmp/migrator.log (on Unix) or C:\lucidworks\fusion\3.1.*x\var\upgrade\tmp\migrator.log (on Windows) indicates this.Only datasources for connectors that are supported in the new Fusion version are upgraded. Datasources for custom connectors are not upgraded.

  ​

  If no Internet connection is available

  If no Internet connection is available during an upgrade, the migrator cannot automatically download the connectors it needs. Using the Fusion UI or API later to install the connectors also might not be an option.
  In this case, download the connectors from Fusion 4.x Connector Downloads for all existing connectors and place them in apps/connectors/bootstrap-plugins for the new deployment (on all Fusion nodes). Do so at the time indicated in the procedures that follow.

  ​

  Adding connectors during an upgrade

  You can add connectors during an upgrade (that is, add connectors that are not in the old deployment).
  Download the connectors from Fusion 4.x Connector Downloads and place them in apps/connectors/bootstrap-plugins for the new version (on all Fusion nodes).

  ​

  Object migrations and transformations

  The migrator automatically migrates these Fusion 4.2 object types, transforming them as needed:

  • Collections
  • Index pipelines
  • Query pipelines
  • Search cluster configurations
  • Datasources
  • Parsing configurations
  • Object groups
  • Links
  • Tasks
  • Jobs
  • Spark objects
  • Apps
  • Appkit apps
  • Index profiles
  • Query profiles
  • Blobs

  In Fusion Server 4.0 and later, most objects exist in the context of apps. When you upgrade from Fusion Server 4.2.x to 4.2.y, the migrator upgrades app objects, all objects in or linked to objects in apps, and objects that are not linked to apps. You can explore the objects in Object Explorer.

  ​

  Access control migration

  The migrator upgrades all access control configurations:

  • Security realms
  • Roles
  • Users

  None of these should require adjustments after migration.

  ​

  Review known issues

  Before upgrading, review the known issues to see whether any of them apply to the circumstances of your upgrade. Some known issues might require actions before upgrading.
  That article also contains instructions regarding what to do if an upgrade step fails.

  ​

  Upgrade on Unix

  Use this procedure to upgrade Fusion on a single Unix node or on multiple Unix nodes.
  Perform the steps in this procedure on the indicated nodes on which Fusion is running (“Fusion nodes”). To perform an upgrade, Fusion nodes must have at least these services running:

  • API service (api)
  • Proxy service (proxy)

  For every step on multiple nodes, ensure that the step completes on all Fusion nodes before going to the next step. There is the notion of a “main node” during the migration process. This node will be used for certain centralized migration activities that do not need to be done on every node, such as downloading connectors that are then uploaded to blob storage that is shared by all, etc. Just pick one of your Fusion nodes to be the “main node”; there is no special requirement as to which one you pick.

  ​

  Ensure that your current version of Fusion has a valid license

  Ensure that your current version of Fusion has a valid permanent Fusion license before proceeding with the upgrade. Place a valid license.properties file in the /opt/lucidworks/fusion/4.2.x/conf directory.

  ​

  Download and install the newer version of Fusion

  Perform these tasks on all Fusion nodes:

  1. Select the Fusion release to which you are upgrading from Fusion Server 4.x File Download Links.
  2. Extract the newer version of Fusion:
     Copy

     cd /opt/lucidworks
     mv ~/Downloads/fusion-4.2.y.tar.gz ./
     tar -xf fusion-4.2.y.tar.gz
     

     For example, if Fusion is currently installed in /opt/lucidworks/fusion/4.2.x, then change your working directory to /opt/lucidworks/ and extract the file there. do not run the new version of Fusion yet.
  3. Ensure that the new version of Fusion has a valid permanent Fusion license before proceeding with the upgrade. Place a valid license.properties file in the /opt/lucidworks/fusion/4.2.y/conf directory.
  4. (If there are custom jar files) If your deployment has custom jar files, add them to the new Fusion deployment.
  5. (If you are performing an upgrade without Internet access) Without Internet access, the migrator cannot download new versions of connectors automatically. Download the new versions of connector zip files for your current connectors from Fusion 4.x Connector Downloads and place them in apps/connectors/bootstrap-plugins for the new deployment.
  6. (If you are adding new connectors) If you want your new deployment to use connectors that are not in the current deployment, you can add them now. Download the connector zip files from Fusion 4.x Connector Downloads and place them in apps/connectors/bootstrap-plugins for the new deployment.
  7. Verify that there is sufficient disk space for a second copy of the Solr index directory, fusion/4.2.x/data/solr. If there is not sufficient disk space, free up space before proceeding.

  ​

  Download and install the Fusion migrator

  Perform these tasks on all Fusion nodes:

  1. Download the latest migrator zip file for Unix. (Do this now, even if you have downloaded the migrator before, to ensure that you have the latest version.)
  2. Create FUSION_OLD and FUSION_NEW environment variables that point to the old and new Fusion installation directories respectively (using the full path).
     Copy

     export FUSION_OLD="/opt/lucidworks/fusion/4.2.x"
     export FUSION_NEW="/opt/lucidworks/fusion/4.2.y"
     

     For example, when upgrading from Fusion 4.2.0 to 4.2.6:
     Copy

     export FUSION_OLD="/opt/lucidworks/fusion/4.2.0"
     export FUSION_NEW="/opt/lucidworks/fusion/4.2.6"
     

  3. Create this directory:
     Copy

     mkdir $FUSION_OLD/var/upgrade
     

  4. Install the migrator:
     Copy

     tar -zxv -C $FUSION_OLD/var/upgrade --strip-components=1 -f fusion-migrator.tar.gz
     

  ​

  Run the migrator

  Perform these tasks on the indicated nodes:

  1. (On all Fusion nodes) Start all Fusion services for the old version of Fusion:
     Copy

     $FUSION_OLD/bin/fusion start
     

  2. (Only on the main Fusion node) Run the migrator to export the configuration data from the old version of Fusion:
     Copy

     java -jar $FUSION_OLD/var/upgrade/migrator.jar --export
     

     This message indicates that the step finished successfully:
     Copy

     Old Fusion configuration export (--export) finished successfully.
     

  3. (On all Fusion nodes) Stop the old versions of Fusion services and Solr; but not ZooKeeper:
     Copy

     $FUSION_OLD/bin/log-shipper stop
     $FUSION_OLD/bin/sql stop
     $FUSION_OLD/bin/admin-ui stop
     $FUSION_OLD/bin/webapps stop
     $FUSION_OLD/bin/proxy stop
     $FUSION_OLD/bin/connectors-rpc stop
     $FUSION_OLD/bin/connectors-classic stop
     $FUSION_OLD/bin/api stop
     $FUSION_OLD/bin/solr stop
     

     If Spark services are running, also stop those:
     Copy

     $FUSION_OLD/bin/spark-master stop
     $FUSION_OLD/bin/spark-worker stop
     

     You can see what is running with $FUSION_OLD/bin/fusion status.
  4. (Only on secondary Fusion nodes) Prepare secondary nodes:
     Copy

     java -jar $FUSION_OLD/var/upgrade/migrator.jar --prepare-secondary
     

     This message indicates that the step finished successfully:
     Copy

     Prepare secondary nodes (--prepare-secondary) finished successfully.
     

  5. (On all Fusion nodes) Stop ZooKeeper for the old version of Fusion (unless you are using an external ZooKeeper instance, in which case you can ignore this step):
     Copy

     $FUSION_OLD/bin/zookeeper stop
     

  6. (Only on the main Fusion node) Transform configuration data on the main Fusion node:
     Copy

     java -jar $FUSION_OLD/var/upgrade/migrator.jar --main-transform
     

     Depending on the size of your Solr index, this step can take a long time (for example, multiple tens of minutes).
     This message indicates that the step finished successfully:
     Copy

     Fusion data transformations on main node (--main-transform) finished successfully.
     

  7. (On all Fusion nodes) Start ZooKeeper for the new version of Fusion (unless you are using an external ZooKeeper instance, in which case you can ignore this step):
     Copy

     $FUSION_NEW/bin/zookeeper start
     

  8. (Only on the main Fusion node) Import the first part of configuration data into the new version of Fusion:
     Copy

     java -jar $FUSION_OLD/var/upgrade/migrator.jar --zookeeper-import
     

     This message indicates that the step finished successfully:
     Copy

     New Fusion Zookeeper import (--zookeeper-import) finished successfully.
     

  9. (On all Fusion nodes) Start all Fusion services for the new version of Fusion:
     Copy

     $FUSION_NEW/bin/fusion start
     

  10. (Only on the main Fusion node) Import the second part of configuration data into the new version of Fusion:
      Copy

      java -jar $FUSION_OLD/var/upgrade/migrator.jar --fusion-import
      

      This message indicates that the step finished successfully:
      Copy

      New Fusion object import (--fusion-import) finished successfully.
      

      After migration, you can find details about the results in the fusion/4.2.x/var/upgrade/tmp directory. If the migration produces unexpected results, the files in this directory are helpful for troubleshooting.

  ​

  Validate the new version of Fusion

  How to validate the new version of Fusion

  1. (Only on the main Fusion node) Restart the new version of Fusion (all services defined in fusion.properties):
     Copy

     $FUSION_NEW/bin/fusion restart
     

  2. Log into the Fusion UI (your admin password is the same as for the old installation), and confirm the release number of the new version of Fusion:
     1. Clear your browser’s cache. Otherwise, you might inadvertently access a cached version of the old Fusion UI and see inconsistent behavior.
     2. In a browser, open the Fusion UI at http://localhost:8764/ (replace localhost with your server name or IP address if necessary).
     3. Log in.
     4. Navigate to Admin > About Fusion. The About Fusion panel should display the newer Fusion release number.
  3. Ensure that all connectors were installed automatically during the upgrade.
     • For Fusion 4.x from the Fusion launcher, click the tile for a migrated app. Click System > Blobs. If any connectors are missing from the list, click Add > Connector Plugin and install them manually.
     • For Fusion 3.x from the Fusion launcher, click Devops > Home > Blobs. If any connectors are missing from the list, click Add > Connector Plugin and install them manually.
  4. Ensure that all customizations you made in the former version of Fusion are present in the new one.
  5. When you are satisfied with the migration and you have backed up the fusion/4.2.x/ directory, you can rm -fr fusion/4.2.*x/ to remove the older version of Fusion (on all Fusion nodes).

  ​

  Add support for business rules to existing apps

  Fusion AI 4.2 introduces functionality for using business rules, that is, manually created formulas for rewriting queries and responses.
  You can add support for business rules to apps that were created in versions of Fusion AI prior to version 4.2. To do so, perform the steps in this section.
  The add-rule-objects-xyz.zip file (where xyz is a version number) specifies the objects to add to an app. It is supplied in the Fusion migrator zip file at the top level. After installing the migrator, the location is $FUSION_OLD/var/upgrade/import-files.

  Adding support for business rules has a costs. Additional collections and objects are created. Only add support for business rules to apps in which you plan to use them.

  You have a choice. You can update each app using the Fusion UI or the Fusion API.

  ​

  Fusion UI

  For each app in which you plan to use business rules, import the objects in the add-rule-objects-xyz.zip file into the app.How to import business rule objects

  1. In the Fusion launcher, click the app into which you want to import objects. The Fusion workspace appears.
  2. In the upper left, click System > Import Fusion Objects. The Import Fusion Objects window opens.
  3. For the data file, select add-rule-objects-xyz.zip from your local filesystem. The location in the extracted migrator files is $FUSION_OLD/var/upgrade/import-files.
  4. Click Import.
  5. Edit the Application ID parameter value to use the app name. If the app name contains spaces, replace those with underscore characters. For example, Lucene Revolution would become Lucene_Revolution.
  6. Click Import.
  7. If there are conflicts, Fusion prompts you to specify an import policy. Click Merge to skip all conflicting objects and import only the non-conflicting objects. Fusion confirms that the import was successful.
  8. Click Close to close the Import Fusion Objects window.

  ​

  Fusion API

  For each app in which you plan to use business rules, import the objects in the add-rule-objects-xyz.zip file into the app.How to import business rule objects

  1. Create an app-name.txt file with the following content:
     Copy

     {"app.id" : "_name-of-migrated-app_"}
     

     For example, for the app Lucene Revolution:
     Copy

     {"app.id" : "Lucene_Revolution"}
     

     Here, we assume that you create the files in your home directory, for which the $HOME environment variable is defined.
  2. Import the business rule objects:
     Copy

     curl -u USERNAME:PASSWORD -H "Content-Type:multipart/form-data" -X POST -F 'variableValues=path-to-txt-file/app-name.txt' -F "importData=path-to-zip-file/add-rule-objects-xyz.zip" "http://<FUSION_HOST>/api/objects/import?importPolicy=merge"
     

     For example:
     Copy

     curl -u admin:Very-SecurePW8 -H "Content-Type:multipart/form-data" -X POST -F 'variableValues=$HOME/Lucene_Revolution.txt' -F "importData=$FUSION_OLD/var/upgrade/import-files/add-rule-objects-001.zip" "http://<FUSION_HOST>/api/objects/import?importPolicy=merge"
     

  ​

  Upgrade on Windows

  Use this procedure to upgrade Fusion on a single Windows node or multiple Windows nodes.
  Perform the steps in this procedure on the indicated nodes on which Fusion is running (“Fusion nodes”). To perform an upgrade, Fusion nodes must have at least these services running:

  • API service (api)
  • Proxy service (proxy)

  If you are upgrading Fusion on multiple nodes, then, for every step on multiple nodes, ensure that the step completes on all Fusion nodes before going to the next step. There is the notion of a “main node” during the migration process. This node will be used for certain centralized migration activities that do not need to be done on every node, such as downloading connectors that are then uploaded to blob storage that is shared by all, etc. Just pick one of your Fusion nodes to be the “main node”; there is no special requirement as to which one you pick.

  ​

  Ensure that your current version of Fusion has a valid license

  Ensure that your current version of Fusion has a valid permanent Fusion license before proceeding with the upgrade. Place a valid license.properties file in the C:\lucidworks\fusion\4.2.x\conf directory.

  ​

  Download and install the newer version of Fusion

  Perform these tasks on all Fusion nodes:

  1. Select the Fusion release to which you are upgrading from Fusion Server 4.x File Download Links.
  2. Move the fusion-4.2.y.zip file to the directory that contains the fusion\ directory. For example, if Fusion is installed in C:\lucidworks\fusion\4.2.x, then move the file to C:\lucidworks.
  3. Unzip the fusion-4.2.y.zip file. do not run the new version of Fusion yet.
  4. For Fusion 4.x, ensure that the new version of Fusion has a valid permanent Fusion license before proceeding with the upgrade. Place a valid license.properties file in the C:\lucidworks\fusion\4.2.y\conf directory.
  5. (If there are custom jar files) If your deployment has custom jar files, add them to the new Fusion deployment.
  6. (If you are performing an upgrade without Internet access) Without Internet access, the migrator cannot download new versions of connectors automatically. Download the new versions of connector zip files for your current connectors from Fusion 4.x Connector Downloads and place them in apps\connectors\bootstrap-plugins for the new deployment.
  7. (If you are adding new connectors) If you want your new deployment to use connectors that are not in the current deployment, you can add them now. Download the connector zip files from Fusion 4.x Connector Downloads and place them in apps\connectors\bootstrap-plugins for the new deployment.
  8. Verify that there is sufficient disk space for a second copy of the Solr index directory, fusion\4.2.x\data\solr. If there is not sufficient disk space, free up space before proceeding.

  ​

  Download and install the Fusion migrator

  Perform these tasks on all Fusion nodes:

  1. Download the latest migrator zip file for Windows. (Do this now, even if you have downloaded the migrator before, to ensure that you have the latest version.)
  2. Open a Command Prompt window and create FUSION_OLD and FUSION_NEW environment variables that point to the old and new Fusion installation directories respectively. For example:
     Copy

     set FUSION_OLD=C:\lucidworks\fusion\4.2.0
     set FUSION_NEW=C:\lucidworks\fusion\4.2.6
     

  3. Create a fusion\4.2.x\var\upgrade directory.
  4. Unzip the migrator zip file, and move the contents of the extracted folder to +fusion\+4.2.x\var\upgrade.

  ​

  Run the migrator

  Perform these tasks on the indicated nodes:

  1. (On all Fusion nodes) Start all Fusion services for the old version of Fusion:
     Copy

     %FUSION_OLD%\bin\fusion.cmd start
     

  2. (Only on the main Fusion node) Run the migrator to export the configuration data from the old version of Fusion:
     Copy

     java -jar "%FUSION_OLD%\var\upgrade\migrator.jar" --export
     

     This message indicates that the step finished successfully:
     Copy

     Old Fusion configuration export (--export) finished successfully.
     

  3. (On all Fusion nodes) Stop the old versions of Fusion services and Solr; but not ZooKeeper:
     Copy

     %FUSION_OLD%\bin\log-shipper.cmd stop
     %FUSION_OLD%\bin\sql.cmd stop
     %FUSION_OLD%\bin\admin-ui.cmd stop
     %FUSION_OLD%\bin\webapps.cmd stop
     %FUSION_OLD%\bin\proxy.cmd stop
     %FUSION_OLD%\bin\connectors-rpc.cmd stop
     %FUSION_OLD%\bin\connectors-classic.cmd stop
     %FUSION_OLD%\bin\api.cmd stop
     %FUSION_OLD%\bin\solr.cmd stop
     

     If Spark and SQL services are running, also stop those:
     Copy

     %FUSION_OLD%\bin\spark-master.cmd stop
     %FUSION_OLD%\bin\spark-worker.cmd stop
     

     You can see what is running with %FUSION_OLD%\bin\fusion status.
  4. (Only on secondary Fusion nodes) Prepare secondary nodes:
     Copy

     java -jar "%FUSION_OLD%\var\upgrade\migrator.jar" --prepare-secondary
     

     This message indicates that the step finished successfully:
     Copy

     Prepare secondary nodes (--prepare-secondary) finished successfully.
     

  5. (On all Fusion nodes) Stop ZooKeeper for the old version of Fusion (unless you are using an external ZooKeeper instance, in which case you can ignore this step):
     Copy

     %FUSION_OLD%\bin\zookeeper.cmd stop
     

  6. (Only on the main Fusion node) Transform configuration data on the main Fusion node:
     Copy

     java -jar "%FUSION_OLD%\var\upgrade\migrator.jar" --main-transform
     

     Depending on the size of your Solr index, this step can take a long time (for example, multiple tens of minutes).
     This message indicates that the step finished successfully:
     Copy

     Fusion data transformations on main node (--main-transform) finished successfully.
     

  7. (On all Fusion nodes) Start ZooKeeper for the new version of Fusion (unless you are only using an external ZooKeeper instance, in which case you can ignore this step):
     Copy

     %FUSION_NEW%\bin\zookeeper.cmd start
     

  8. (Only on the main Fusion node) Import the first part of configuration data into the new version of Fusion:
     Copy

     java -jar "%FUSION_OLD%\var\upgrade\migrator.jar" --zookeeper-import
     

     This message indicates that the step finished successfully:
     Copy

     New Fusion Zookeeper import (--zookeeper-import) finished successfully.
     

  9. (On all Fusion nodes) Start Solr for the new Fusion version:
     Copy

     %FUSION_NEW%\bin\solr.cmd start
     

  10. (Only on the main Fusion node) Run a script to remove all old plugins from the blob store. Replace solr-address and solr-port as appropriate (as shown in the example):
      Copy

      java -cp "%FUSION_OLD%\var\upgrade\jython-standalone-2.7.1.jar;%FUSION_OLD%\var\upgrade\migrator.jar" org.python.util.jython "%FUSION_OLD%\var\upgrade\transformations\manual_delete_old_plugin_blobs.py" --solr-address solr-address --solr-port solr-port
      

      For example:
      Copy

      java -cp "%FUSION_OLD%\var\upgrade\jython-standalone-2.7.1.jar;%FUSION_OLD%\var\upgrade\migrator.jar" org.python.util.jython "%FUSION_OLD%\var\upgrade\transformations\manual_delete_old_plugin_blobs.py" --solr-address localhost --solr-port 8983
      

      This message indicates that plugins were deleted successfully:
      Copy

      Deleted old plugin blobs from solr <?xml version="1.0" encoding="UTF-8"?>
      <response>
      
      <lst name="responseHeader">
      <int name="status">0</int>
      <int name="QTime">246</int>
      </lst>
      </response>
      Old connector plugin blobs were deleted successfully.
      

  11. (On all Fusion nodes) Start all Fusion services for the new version of Fusion:
      Copy

      %FUSION_NEW%\bin\fusion.cmd start
      

  12. (Only on the main Fusion node) Import the second part of configuration data into the new version of Fusion:
      Copy

      java -jar "%FUSION_OLD%\var\upgrade\migrator.jar" --fusion-import
      

      This message indicates that the step finished successfully:
      Copy

      New Fusion object import (--fusion-import) finished successfully.
      

  After migration, you can find details about the results in the +fusion\+4.2.x\var\upgrade\tmp directory. If the migration produces unexpected results, the files in this directory are helpful for troubleshooting.

  ​

  Validate the new version of Fusion

  How to validate the new version of Fusion

  1. (On all Fusion nodes) Restart all Fusion services for the new version of Fusion:
     Copy

     %FUSION_NEW%\bin\fusion.cmd restart
     

  2. Log into the Fusion UI (your admin password is the same as for the old installation), and confirm the release number of the new version of Fusion:
     1. Clear your browser’s cache. Otherwise, you might inadvertently access a cached version of the old Fusion UI and see inconsistent behavior.
     2. In a browser, open the Fusion UI at http://localhost:8764/ (replace localhost with your server name or IP address if necessary).
     3. Log in.
     4. Navigate to Admin > About Fusion. The About Fusion panel should display the newer Fusion release number.
  3. Ensure that all connectors were installed automatically during the upgrade.
     1. For Fusion 4.x from the Fusion launcher, click the tile for a migrated app. Click System > Blobs. If any connectors are missing from the list, click Add > Connector Plugin and install them manually.
     2. For Fusion 3.x from the Fusion launcher, click Devops > Home > Blobs. If any connectors are missing from the list, click Add > Connector Plugin and install them manually.
  4. Ensure that all customizations you made in the former version of Fusion are present in the new one.
  5. When you are satisfied with the migration and you have backed up the +fusion\+4.2.x directory, you can remove the older version of Fusion by removing that directory (on all Fusion nodes).

  ​

  Add support for business rules to existing apps

  Fusion AI 4.2 introduces functionality for using business rules, that is, manually created formulas for rewriting queries and responses.
  You can add support for business rules to apps that were created in versions of Fusion AI prior to version 4.2. To do so, perform the steps in this section.
  The add-rule-objects-xyz.zip file (where xyz is a version number) specifies the objects to add to an app. It is supplied in the Fusion migrator zip file at the top level. After installing the migrator, the location is %FUSION_OLD%\var\upgrade\import-files\.
  You have a choice. You can update each app using the Fusion UI or the Fusion API.

  ​

  Fusion UI

  For each app in which you plan to use business rules, import the objects in the add-rule-objects-xyz.zip file into the app.How to import business rule objects

  1. In the Fusion launcher, click the app into which you want to import objects. The Fusion workspace appears.
  2. In the upper left, click System > Import Fusion Objects. The Import Fusion Objects window opens.
  3. For the data file, select add-rule-objects-xyz.zip from your local filesystem. The location in the extracted migrator files is %FUSION_OLD%\var\upgrade\import-files\.
  4. Click Import.
  5. Edit the Application ID parameter value to use the app name. If the app name contains spaces, replace those with underscore characters. For example, Lucene Revolution would become Lucene_Revolution.
  6. Click Import.
  7. If there are conflicts, Fusion prompts you to specify an import policy. Click Merge to skip all conflicting objects and import only the non-conflicting objects. Fusion confirms that the import was successful.
  8. Click Close to close the Import Fusion Objects window.

  ​

  Fusion API

  For each app in which you plan to use business rules, import the objects in the add-rule-objects-xyz.zip file into the app.How to import business rule objects

  1. Create an app-name.txt file with the following content:
     Copy

     {"app.id" : "_name-of-migrated-app_"}
     

     For example, for the app Lucene Revolution:
     Copy

     {"app.id" : "Lucene_Revolution"}
     

     Here, we assume that you create the files in your home directory, for which the %HOMEPATH% environment variable is defined.
  2. Import the business rule objects:
     Copy

     curl -u USERNAME:PASSWORD -H "Content-Type:multipart/form-data" -X POST -F 'variableValues=path-to-txt-file\app-name.txt' -F "importData=path-to-zip-file\add-rule-objects-xyz.zip" "http://<FUSION_HOST>/api/objects/import?importPolicy=merge"
     

     For example:
     Copy

     curl -u admin:Very-SecurePW8 -H "Content-Type:multipart/form-data" -X POST -F 'variableValues=%HOMEPATH%\Lucene_Revolution.txt' -F "importData=%FUSION_OLD%\import-files\add-rule-objects-001.zip" "http://<FUSION_HOST>/api/objects/import?importPolicy=merge"
     

• Connectors
  • Repeatedly stopping a V2 datasource job and clearing the datasource may result in an out-of-memory condition. To recover from this state, restart the connectors-rpc process:
  Copy

  fusion/4.2.0/bin/connectors-rpc restart
  

  • In a cluster environment, after installing a V2 connector (OneDrive connectors) where multiple connectors-rpc nodes are running, the connector may only install and run on one node instead of propagating to all nodes. If this happens, restart all nodes and then re-install the connector.
  • Although the Web V1 connector’s default value for crawlDBType/Crawl database type is “in-memory”, this can cause an out-of-memory condition when crawling large sites. Change the value to “on-disk”.
  • If new items are not picked up when recrawling a Box.com folder, delete the records in the system_box_distributed_crawl collection, like this:
  Copy

  curl “http://localhost:8983/solr/system_box_distributed_crawl/update?commit=true” -H “Content-Type: text/xml” --data-binary ‘<delete><query>*:*</query></delete>’
  

  Then run the datasource job again.
  • The Local Filesystem connector may slow down while crawling an empty folder.
  • With the SMB2/3 connector, if multiple start links point to some of the same data, then the data is indexed multiple times. Remove redundant start links and use only the “parent” link.
  • FS (V2) connector does not save Item metadata in Crawldb. recrawls do not work as expected (some items will be missing or not be evaluated on the next crawl).
  • When re-installing connector using same plugin id but different file name a deadlock condition may occur resulting in a timeout error.
• Fusion UI
  • In the Query Workbench, some document fields may not appear in the dropdown list of fields for faceting. To work around this, enter the name of the field in the text box and press Return. If the field exists in the dataset, it will be added as a facet even though it does not appear in the list.
  • When using Compare mode in the Query Workbench, configuring the list of display fields may change the display in both panels instead of only the working panel.
  • After logging out using Chrome version 73.0.3683.75, the login page may not automatically appear. To work around this, do a hard refresh (by holding down the CTRL key while clicking the Reload button).
  • In the Query Rewriting UI, creating a query rewrite with the same name as an existing query rewrite deletes the existing one. Be sure to create new query rewrites using unique names.
  • Jobs may display incorrect information about their current status or the time at which they last ran. To work around this, use the Jobs API to verify a job’s status and history.
  • After you upload a .war file to the App Studio interface, the View Published UI button disappears from the App Studio configuration panel. To restore this button, click Edit, then click Return to Fusion.
  • New index pipelines may not appear in the Index Workbench until you do a hard refresh (by holding down the CTRL key while clicking the Reload button).
  • In the Query Rewriting UI, after selecting multiple business rules where some rules have tags, adding more tags to the selected rules deletes their existing tags. To work around this, add tags to individual rules instead of adding them in bulk.