Solr 7.5 | ZooKeeper 3.4.13 | Spark 2.3.2 | Jetty 9.4.12.v20180830 | Ignite 2.6.0 |
New features
No new features were introduced in this release.Improvements
- The Users page (System > Access Control > Users) now displays the security realm to which each user is assigned.
-
SharePoint and SharePoint Online connector improvements:
- The SharePoint and SharePoint Online connectors now delete indexed documents after they are deleted from the SharePoint source.
- The SharePoint connector now retries document fetches after network errors.
- Addressed an issue with fetching and indexing of SharePoint list item attachments.
- Addressed an issue with running the SharePoint connector without an associated parser configuration.
-
Addressed an incremental crawl scenario for deleted sites.
- Confluence connector improvements:
- Fixed an issue with the handling of Confluence 5.6 archive API endpoint.
- Added security trimming support for the Confluence connector “confluence-administrators” group.
-
Fixed an issue with incremental ACL updates for changes with
_lw_confluence_anonymous
permission setting. -
The HTML parser now parses documents whose
Mime-Type
value does not exactly matchtext/html
, such astext/html; charset=utf-8
and so on. This allows it to parse.aspx
pages from SharePoint datasources, for example. - The Windows Share connector now retrieves document owner metadata.
-
For connectors that use the Crawl database type/
crawlDBType
parameter, the default value is nowon-disk
instead ofin-memory
in order to support larger crawls by default. - Addressed an issue with persisting proxy settings during Javascript evaluation in the Web connector.
- Addressed an incremental update issue due to unnecessary serialization of the ID field in the MongoDB connector.
-
Updated the Zendesk connector to use
RequestEntityProcessing.BUFFERED
instead of.CHUNKED
to avoid a documented Jersey issue. -
DevOps Center improvements:
- In the Log Viewer, exported data is now constrained by all selected filters in addition to the time range selector.
- The time range selector in the Log Viewer now behaves more consistently.
- Fixed some issues with filtering in the Hosts tab and the Log Viewer.
- In the Log Viewer, log messages whose level is DEBUG are now distinguished by an icon for consistency with other log levels.
-
The size of the
system_logs
collection was significantly reduced by changing the log level formarking user.sessions meter
toDEBUG
. - You can now use the query pipeline to set the status code for Fusion’s response.
-
SQL service improvements:
- Fusion’s SQL service now supports non-phrase predicates.
- Results can now be sorted by the score of the underlying Solr query.
- Random sampling is now supported.
- The service now provides more consistent performant results from unlimited selects.
- Fixed an issue in 4.2.0 which cause unlimited “group by” aggregations to use a bucket size limit of 1.
-
Fusion now captures the same metrics about the
connectors-rpc
services as it does about theconnectors-classic
service. - A variety of minor UI issues were fixed.
Other changes
- FS (V2) connector saves Item metadata in Crawldb. recrawls work as expected (the new/delete/update items will be picked up) - On migrations, it is suggested to clear the datasource and start with a new crawl, but it is not required.
-
Now when you upload a
.war
file to the App Studio interface, the View Published UI button remains available, resolving a known issue in 4.2.0. - Logging out using Chrome version 73.0.3683.75 no longer causes an infinite loop; this resolves a known issue in 4.2.0.
- You can now re-install a connector using the same ID but a different filename than the original.
- Several bug fixes resolve issues with ACL metadata using the Confluence connector.
- Better permissions checks when importing jobs and blobs.
-
When Upgrade Fusion Server 3.1.x to 4.2.y, it is no longer necessary to specify the
app.id
; it is inferred automatically from the current app context.
Upgrade Fusion Server 3.1.x to 4.2.y
Upgrade Fusion Server 3.1.x to 4.2.y
Introduction
This article describes how to perform the following upgrade:- From version: Fusion
3.1.x
- To version: Fusion
4.2.y
Only specific version-to-version upgrade sequences are supported. Some upgrades require multiple steps.
/opt/lucidworks/fusion/3.1.x/var/upgrade
directory (on Unix or MacOS) or the C:\lucidworks\fusion\3.1.x\var\upgrade\
directory (on Windows). The file names reference the versions you are upgrading from and to. For example:- To upgrade
3.1.5
to4.2.5
, the migrator uses the3.1.x-4.2.x.properties
file.
The newer Fusion instance must be newly untarred and never started.
About the upgrade
This section describes how connectors, object migrations, and signals are migrated during an upgrade.Connectors
In Fusion 3.1.0 and later, only a subset of connectors are included by default.The migrator detects which connectors were used in the older version of Fusion, and installs them automatically in Fusion4.0.y
. This step requires an Internet connection. If no connection is available, then download the connectors from Fusion 4.x Connector Downloads and [install them as bootstrap plugins.If a connector to be upgraded was not available during the upgrade, then a message in /opt/lucidworks/fusion/3.1.x/var/upgrade/tmp/migrator.log
(on Unix) or C:\lucidworks\fusion\3.1.*x\var\upgrade\tmp\migrator.log
(on Windows) indicates this.Only datasources for connectors that are supported in the new Fusion version are upgraded. Datasources for custom connectors are not upgraded.If no Internet connection is available
If no Internet connection is available during an upgrade, the migrator cannot automatically download the connectors it needs. Using the Fusion UI or API later to install the connectors also might not be an option.In this case, download the connectors from Fusion 4.x Connector Downloads for all existing connectors and place them inapps/connectors/bootstrap-plugins
for the new deployment (on all Fusion nodes). Do so at the time indicated in the procedures that follow.Adding connectors during an upgrade
You can add connectors during an upgrade (that is, add connectors that are not in the old deployment).Download the connectors from Fusion 4.x Connector Downloads and place them inapps/connectors/bootstrap-plugins
for the new version (on all Fusion nodes).Object migrations and transformations
The migrator upgrades all Fusion 3.1 object types:- Collections
- Index pipelines
- Query pipelines
- Search cluster configurations
- Schedules
- Aggregations
- Datasources
- Dashboards
- Parsing configurations
- Object groups
- Links
- Tasks
- Jobs
- Spark objects
- Apps
- Appkit apps
- Index profiles
- Query profiles
- Blobs
In Fusion Server 4.0 and later, most objects exist in the context of apps. When you upgrade from Fusion 3.1.x to Fusion Server 4.2.y, the migrator creates the app
default
and places object in that app or links them to it, as needed. Some objects are not linked to apps. You can explore objects in Object Explorer.Noteworthy transformations
While migrating objects, the migrator makes these transformations:- It adds the new
log-shipper
service tofusion.properties
. - It removes obsolete log-cleanup tasks.
- It converts the format of job history records.
Migration of index profiles and query profiles
In Fusion Server 4.0, index profiles and query profiles are objects. They have capabilities that exceed those of index profiles and query profiles in prior releases.- Prior to Fusion 4.0. A single index profile could reference multiple index pipelines. A single query profile could reference multiple query pipelines.
- In Fusion 4.0 and later. A single index profile can reference a single index pipeline. A single query profile can reference a single query pipeline.
- Index profiles.
- If a 3.1.x index profile contains a reference to the index pipeline
default
, then the migrated profile retains that single reference. - If a 3.1.x index profile does not contain a reference to the index pipeline
default
, then the migrated profile references an index pipeline that has the same name as the index profile.
- If a 3.1.x index profile contains a reference to the index pipeline
- Query profiles.
- If a 3.1.x query profile contains a reference to the query pipeline
default
, then the migrated profile retains that single reference. - If a 3.1.x query profile does not contain a reference to the query pipeline
default
, then the migrated profile references a query pipeline that has the same name as the query profile.
- If a 3.1.x query profile contains a reference to the query pipeline
Access control migration
The migrator upgrades all access control configurations:- Security realms. Security realms do not require adjustments after migration.
- Roles. System-created roles do not require adjustments after migration. User-created roles probably require adjustments after migration.
- Users. System-created users do not require adjustments after migration. User-created users probably require adjustments after migration.
Migrations that should not require adjustments
This section describes fully automatic parts of migration. You should not have to adjust these items after migration.The migrator performs these migration tasks:- It migrates encrypted passwords, so users can log in with the same credentials they used on the older version of Fusion or Fusion Server.
- It adds the user
webapps-system-account
. Fusion Server uses this account. You do not need to do anything with it. - It adds these permissions to the role
developer
:
- Updates these permissions for the role
search
:
- Adds the role
webapps-role
with these permissions:
Migrations that might require adjustments
This section describes parts of migration for which you might need to make adjustments after migration.Following migration, we recommend that you review the API and UI permissions for roles, and the roles and API permissions for users.
- If you have created your own developer roles, add these permissions to the roles:
- If you have created your own search roles, add these permissions to the roles:
Signals and Spark jobs
Fusion Server 4.2.y indexes search log events into theCOLLECTION_NAME_signals
collections as response
signals, instead of indexing them into the COLLECTION_NAME_logs
collections.Roughly 90 new fields used by App Insights are added to existing COLLECTION_NAME_signals
collections.You are encouraged to adopt the new signal fields, but you can continue using the old dynamic field names, such as user_id_s
, until Fusion 5. If you adopt the new signals schema, then you must update any Spark jobs that rely on the old field names.Fusion Server 4.2.y replaces the _signals_ingest
index pipeline with a new version that works with the new signals schema.If you made changes to the _signals_ingest
pipeline, then you will need to manually add those changes to the new configuration after migration.The migrator preserves legacy-style aggregation jobs. Optionally, you can manually convert these to SQL-based aggregation jobs.Fusion Server 4.2.y adds a new session rollup job for each collection with signals enabled. The session rollup job creates session signals that contain aggregated information about user activity in a session.Fusion Server 4.2.y adds a new head/tail analysis job for each collection with signals enabled. The head/tail analysis job uses signals to compute interesting metrics for head-and-tail queries.Review known issues
Before upgrading, review the known issues to see whether any of them apply to the circumstances of your upgrade. Some known issues might require actions before upgrading.That article also contains instructions regarding what to do if an upgrade step fails.Upgrade on Unix
Use this procedure to upgrade Fusion on a single Unix node or on multiple Unix nodes.Perform the steps in this procedure on the indicated nodes on which Fusion is running (“Fusion nodes”). To perform an upgrade, Fusion nodes must have at least these services running:- API service (
api
) - Fusion UI service (
ui
)
For every step on multiple nodes, ensure that the step completes on all Fusion nodes before going to the next step. There is the notion of a “main node” during the migration process. This node will be used for certain centralized migration activities that do not need to be done on every node, such as downloading connectors that are then uploaded to blob storage that is shared by all, etc. Just pick one of your Fusion nodes to be the “main node”; there is no special requirement as to which one you pick.
Install the Java JDK
Before upgrading to Fusion Server 4.2 or later, you must install the Java JDK on all nodes that will run Fusion Server. Prior versions of Fusion Server could use either the JRE or the JDK.For more information, see Java requirements.Download and install the newer version of Fusion
Perform these tasks on all Fusion nodes:- Select the Fusion release to which you are upgrading from Fusion Server 4.x File Download Links.
- Extract the newer version of Fusion:
For example, if Fusion is currently installed in
/opt/lucidworks/fusion/3.1.x
, then change your working directory to/opt/lucidworks/
and extract the file there. do not run the new version of Fusion yet. - Ensure that the new version of Fusion has a valid permanent Fusion license before proceeding with the upgrade. Place a valid
license.properties
file in the/opt/lucidworks/fusion/4.2.y/conf
directory. - (If there are custom
jar
files) If your deployment has customjar
files, add them to the new Fusion deployment. - (If you are performing an upgrade without Internet access) Without Internet access, the migrator cannot download new versions of connectors automatically. Download the new versions of connector zip files for your current connectors from Fusion 4.x Connector Downloads and place them in
apps/connectors/bootstrap-plugins
for the new deployment. - (If you are adding new connectors) If you want your new deployment to use connectors that are not in the current deployment, you can add them now. Download the connector zip files from Fusion 4.x Connector Downloads and place them in
apps/connectors/bootstrap-plugins
for the new deployment. - Verify that there is sufficient disk space for a second copy of the Solr index directory,
fusion/3.1.x/data/solr
. If there is not sufficient disc space, free up space before proceeding.
Download and install the Fusion migrator
Perform these tasks on all Fusion nodes:- Download the latest migrator zip file for Unix. (Do this now, even if you have downloaded the migrator before, to ensure that you have the latest version.)
- Create
FUSION_OLD
andFUSION_NEW
environment variables that point to the old and new Fusion installation directories respectively (using the full path).For example, when upgrading from Fusion 3.1.5 to 4.2.6: - Create this directory:
- Install the migrator:
Run the migrator
Perform these tasks on the indicated nodes:- (On all Fusion nodes) Start all Fusion services for the old version of Fusion:
- (Only on the main Fusion node) Run the migrator to export the configuration data from the old version of Fusion:
This message indicates that the step finished successfully:
- (On all Fusion nodes) Stop the old versions of Fusion services and Solr; but not ZooKeeper:
You can see what is running with
$FUSION_OLD/bin/fusion status
.- (On all Fusion nodes) Stop ZooKeeper for the old version of Fusion (unless you are using an external ZooKeeper instance, in which case you can ignore this step):
- (Only on the main Fusion node) Transform configuration data on the main Fusion node:
This message indicates that the step finished successfully:Depending on the size of your Solr index, this step can take a long time (for example, multiple tens of minutes).
- (On all Fusion nodes) Start ZooKeeper for the new version of Fusion (unless you are using an external ZooKeeper instance, in which case you can ignore this step):
- (Only on the main Fusion node) Import the first part of configuration data into the new version of Fusion:
This message indicates that the step finished successfully:
- (If upgrading from Fusion 3.1.0 through 3.1.3; On all Fusion nodes) Start Solr for the new Fusion version:
- (If upgrading from Fusion 3.1.0 through 3.1.3; Only on the main Fusion node) Run a script to remove all old plugins from the blob store. Replace
solr-address
andsolr-port
as appropriate (as shown in the example):For example:This message indicates that plugins were deleted successfully: - (On all Fusion nodes) Start all Fusion services for the new version of Fusion:
- (Only on the main Fusion node) Import the second part of configuration data into the new version of Fusion:
This message indicates that the step finished successfully:
After migration, you can find details about the results in the
fusion/3.1.x/var/upgrade/tmp
directory. If the migration produces unexpected results, the files in this directory are helpful for troubleshooting.Validate the new version of Fusion
How to validate the new version of Fusion- (Only on the main Fusion node) Restart the new version of Fusion (all services defined in
fusion.properties
): - Log into the Fusion UI (your
admin
password is the same as for the old installation), and confirm the release number of the new version of Fusion:- Clear your browser’s cache.
Otherwise, you might inadvertently access a cached version of the old Fusion UI and see inconsistent behavior. - In a browser, open the Fusion UI at
http://localhost:8764/
(replacelocalhost
with your server name or IP address if necessary). - Log in.
- Navigate to Admin > About Fusion.
The About Fusion panel should display the newer Fusion release number.
- Clear your browser’s cache.
- Ensure that all connectors were installed automatically during the upgrade.
- For Fusion 4.x from the Fusion launcher, click the tile for a migrated app. Click System > Blobs. If any connectors are missing from the list, click Add > Connector Plugin and install them manually.
- For Fusion 3.x from the Fusion launcher, click Devops > Home
> Blobs. If any connectors are missing from the list, click Add > Connector Plugin and install them manually.
- Ensure that all customizations you made in the former version of Fusion are present in the new one.
- If you migrated from version 3.1.x to version 4.2.0 or later, confirm associations between blobs and apps and make adjustments if necessary.
- When you are satisfied with the migration and you have backed up the
fusion/3.1.x/
directory, you canrm -fr fusion/3.1.*x/
to remove the older version of Fusion (on all Fusion nodes).
Add support for business rules to existing apps
Fusion AI 4.2 introduces functionality for using business rules, that is, manually created formulas for rewriting queries and responses.You can add support for business rules to apps that were created in versions of Fusion AI prior to version 4.2. To do so, perform the steps in this section.Theadd-rule-objects-xyz.zip
file (where xyz
is a version number) specifies the objects to add to an app. It is supplied in the Fusion migrator zip file at the top level. After installing the migrator, the location is $FUSION_OLD/var/upgrade/import-files
.Adding support for business rules has a costs. Additional collections and objects are created. Only add support for business rules to apps in which you plan to use them.
Fusion UI
For each app in which you plan to use business rules, import the objects in theadd-rule-objects-xyz.zip
file into the app.How to import business rule objects- In the Fusion launcher, click the app into which you want to import objects. The Fusion workspace appears.
- In the upper left, click System
> Import Fusion Objects. The Import Fusion Objects window opens.
- For the data file, select
add-rule-objects-xyz.zip
from your local filesystem. The location in the extracted migrator files is$FUSION_OLD/var/upgrade/import-files
. - Click Import.
- Edit the
Application ID
parameter value to use the app name. If the app name contains spaces, replace those with underscore characters. For example,Lucene Revolution
would becomeLucene_Revolution
. - Click Import.
- If there are conflicts, Fusion prompts you to specify an import policy. Click Merge to skip all conflicting objects and import only the non-conflicting objects. Fusion confirms that the import was successful.
- Click Close to close the Import Fusion Objects window.
Fusion API
For each app in which you plan to use business rules, import the objects in theadd-rule-objects-xyz.zip
file into the app.How to import business rule objects- Create an
app-name.txt
file with the following content:For example, for the appLucene Revolution
:Here, we assume that you create the files in your home directory, for which the$HOME
environment variable is defined. - Import the business rule objects:
For example:
Upgrade on Windows
Use this procedure to upgrade Fusion on a single Windows node or multiple Windows nodes.Perform the steps in this procedure on the indicated nodes on which Fusion is running (“Fusion nodes”). To perform an upgrade, Fusion nodes must have at least these services running:- API service (
api
) - Fusion UI service (
ui
)
If you are upgrading Fusion on multiple nodes, then, for every step on multiple nodes, ensure that the step completes on all Fusion nodes before going to the next step. There is the notion of a “main node” during the migration process. This node will be used for certain centralized migration activities that do not need to be done on every node, such as downloading connectors that are then uploaded to blob storage that is shared by all, etc. Just pick one of your Fusion nodes to be the “main node”; there is no special requirement as to which one you pick.
Install the Java JDK
Before upgrading to Fusion Server 4.2 or later, you must install the Java JDK on all nodes that will run Fusion Server. Prior versions of Fusion Server could use either the JRE or the JDK.For more information, see Java requirements.Download and install the newer version of Fusion
Perform these tasks on all Fusion nodes:- Select the Fusion release to which you are upgrading from Fusion Server 4.x File Download Links.
- Move the
fusion-4.2.y.zip
file to the directory that contains thefusion\
directory. For example, if Fusion is installed inC:\lucidworks\fusion\3.1.x
, then move the file toC:\lucidworks
. - Unzip the
fusion-4.2.y.zip
file. do not run the new version of Fusion yet. - For Fusion 4.x, ensure that the new version of Fusion has a valid permanent Fusion license before proceeding with the upgrade. Place a valid
license.properties
file in theC:\lucidworks\fusion\4.2.y\conf
directory. - (If there are custom
jar
files) If your deployment has customjar
files, add them to the new Fusion deployment. - (If you are performing an upgrade without Internet access) Without Internet access, the migrator cannot download new versions of connectors automatically. Download the new versions of connector zip files for your current connectors from Fusion 4.x Connector Downloads and place them in
apps\connectors\bootstrap-plugins
for the new deployment. - (If you are adding new connectors) If you want your new deployment to use connectors that are not in the current deployment, you can add them now. Download the connector zip files from Fusion 4.x Connector Downloads and place them in
apps\connectors\bootstrap-plugins
for the new deployment. - Verify that there is sufficient disk space for a second copy of the Solr index directory,
fusion\3.1.x\data\solr
. If there is not sufficient disc space, free up space before proceeding.
Download and install the Fusion migrator
Perform these tasks on all Fusion nodes:- Download the latest migrator zip file for Windows. (Do this now, even if you have downloaded the migrator before, to ensure that you have the latest version.)
- Open a Command Prompt window and create
FUSION_OLD
andFUSION_NEW
environment variables that point to the old and new Fusion installation directories respectively. For example: - Create a
fusion\3.1.x\var\upgrade
directory. - Unzip the migrator zip file, and move the contents of the extracted folder to
+fusion\+3.1.x\var\upgrade
.
Run the migrator
Perform these tasks on the indicated nodes:- (On all Fusion nodes) Start all Fusion services for the old version of Fusion:
- (Only on the main Fusion node) Run the migrator to export the configuration data from the old version of Fusion:
This message indicates that the step finished successfully:
- (On all Fusion nodes) Stop the old versions of Fusion services and Solr; but not ZooKeeper:
You can see what is running with
%FUSION_OLD%\bin\fusion status
. - (Only on secondary Fusion nodes) Prepare secondary nodes:
This message indicates that the step finished successfully:
- (On all Fusion nodes) Stop ZooKeeper for the old version of Fusion (unless you are using an external ZooKeeper instance, in which case you can ignore this step):
- (Only on the main Fusion node) Transform configuration data on the main Fusion node:
This message indicates that the step finished successfully:Depending on the size of your Solr index, this step can take a long time (for example, multiple tens of minutes).
- (On all Fusion nodes) Start ZooKeeper for the new version of Fusion (unless you are only using an external ZooKeeper instance, in which case you can ignore this step):
- (Only on the main Fusion node) Import the first part of configuration data into the new version of Fusion:
This message indicates that the step finished successfully:
- (If upgrading from Fusion 3.1.0 through 3.1.3; On all Fusion nodes) Start Solr for the new Fusion version:
- (If upgrading from Fusion 3.1.0 through 3.1.3; Only on the main Fusion node) Run a script to remove all old plugins from the blob store. Replace
solr-address
andsolr-port
as appropriate (as shown in the example):For example:This message indicates that plugins were deleted successfully: - (On all Fusion nodes) Start all Fusion services for the new version of Fusion:
- (Only on the main Fusion node) Import the second part of configuration data into the new version of Fusion:
This message indicates that the step finished successfully:
After migration, you can find details about the results in the
+fusion\+3.1.x\var\upgrade\tmp
directory. If the migration produces unexpected results, the files in this directory are helpful for troubleshooting.Validate the new version of Fusion
How to validate the new version of Fusion- (On all Fusion nodes) Restart all Fusion services for the new version of Fusion:
- Log into the Fusion UI (your
admin
password is the same as for the old installation), and confirm the release number of the new version of Fusion:- Clear your browser’s cache. Otherwise, you might inadvertently access a cached version of the old Fusion UI and see inconsistent behavior.
- In a browser, open the Fusion UI at
http://localhost:8764/
(replacelocalhost
with your server name or IP address if necessary). - Log in.
- Navigate to Admin > About Fusion.
The About Fusion panel should display the newer Fusion release number.
- Ensure that all connectors were installed automatically during the upgrade.
- For Fusion 4.x from the Fusion launcher, click the tile for a migrated app. Click System > Blobs. If any connectors are missing from the list, click Add > Connector Plugin and install them manually.
- For Fusion 3.x from the Fusion launcher, click Devops > Home
> Blobs. If any connectors are missing from the list, click Add > Connector Plugin and install them manually.
- Ensure that all customizations you made in the former version of Fusion are present in the new one.
- If you migrated from version 3.1.x to version 4.2.0 or later, confirm associations between blobs and apps and make adjustments if necessary.
- When you are satisfied with the migration and you have backed up the
+fusion\+3.1.x
directory, you can remove the older version of Fusion by removing that directory (on all Fusion nodes).
Add support for business rules to existing apps
Fusion AI 4.2 introduces functionality for using business rules, that is, manually created formulas for rewriting queries and responses.You can add support for business rules to apps that were created in versions of Fusion AI prior to version 4.2. To do so, perform the steps in this section.Theadd-rule-objects-xyz.zip
file (where xyz
is a version number) specifies the objects to add to an app. It is supplied in the Fusion migrator zip file at the top level. After installing the migrator, the location is %FUSION_OLD%\var\upgrade\import-files\
.You have a choice. You can update each app using the Fusion UI or the Fusion API.Fusion UI
For each app in which you plan to use business rules, import the objects in theadd-rule-objects-xyz.zip
file into the app.How to import business rule objects- In the Fusion launcher, click the app into which you want to import objects. The Fusion workspace appears.
- In the upper left, click System
> Import Fusion Objects. The Import Fusion Objects window opens.
- For the data file, select
add-rule-objects-xyz.zip
from your local filesystem. The location in the extracted migrator files is%FUSION_OLD%\var\upgrade\import-files\
. - Click Import.
- Edit the
Application ID
parameter value to use the app name. If the app name contains spaces, replace those with underscore characters. For example,Lucene Revolution
would becomeLucene_Revolution
. - Click Import.
- If there are conflicts, Fusion prompts you to specify an import policy. Click Merge to skip all conflicting objects and import only the non-conflicting objects. Fusion confirms that the import was successful.
- Click Close to close the Import Fusion Objects window.
Fusion API
For each app in which you plan to use business rules, import the objects in theadd-rule-objects-xyz.zip
file into the app.How to import business rule objects- Create an
app-name.txt
file with the following content:For example, for the appLucene Revolution
:Here, we assume that you create the files in your home directory, for which the%HOMEPATH%
environment variable is defined. - Import the business rule objects:
For example:
-
Exporting an app with
deep=false
now exports only the app definition without the linked objects. - Fixed an issue which sometimes prevented “chained” jobs from firing, that is, jobs that are triggered by the success or failure of another job.
-
Fixed an issue that prevented users from disabling the Asynchronous Execution Config/
asyncConfig
option in some pipeline stages. -
In the Solr Query pipeline stage, configuring one or more
requestHandlers
now blocks usage of theselect
request handler. To configure usage of multiple request handlers includingselect
, addselect
to yourrequestHandlers
. -
Fixed an issue which prevented the parsing of paths containing backslashes in
fusion.properties
on Windows. -
Fixed an issue when using the default
init/systemd/install.sh
systemd service which cause all Fusion services to fail when one service failed. To avoid this issue, use/opt/fusion/4.2.1/init/systemd/install-services.sh
instead; see the instructions atfusion/4.2.1/packaging/init/systemd/README
. -
Fixed an issue which caused intermittent Zendesk connector failures with a
ConnectionClosedException: Premature end of chunk coded message body: closing chunk expected
message.
Known issues
-
When under load, the Fusion proxy service can occasionally become stuck, causing user authentication to fail. This is the result of the proxy
InputStream
failing to close properly. An upgrade to Fusion 4.2.4 is required to fix this issue. See Upgrade Fusion Server 4.2.x to 4.2.y. - The Zendesk connector will fail occasionally when fetching content, due to a known HTTP client library issue.
Upgrade Fusion Server 4.2.x to 4.2.y
Upgrade Fusion Server 4.2.x to 4.2.y
Introduction
This article describes how to perform the following upgrade:- From version: Fusion
4.2.x
- To version: Fusion
4.2.y
Only specific version-to-version upgrade sequences are supported. Some upgrades require multiple steps.
/opt/lucidworks/fusion/4.2.x/var/upgrade
directory (on Unix or MacOS) or the C:\lucidworks\fusion\4.2.*x\var\upgrade\
directory (on Windows). The file names reference the versions you are upgrading from and to. For example:- To upgrade
4.2.3
to4.2.5
, the migrator uses the4.2.x-4.2.x.properties
file.
The newer Fusion instance must be newly untarred and never started.
About the upgrade
This section describes how connectors, object migrations, and signals are migrated during an upgrade.Connectors
In Fusion 3.1.0 and later, only a subset of connectors are included by default.The migrator detects which connectors were used in the older version of Fusion, and installs them automatically in Fusion4.0.y
. This step requires an Internet connection. If no connection is available, then download the connectors from Fusion 4.x Connector Downloads and [install them as bootstrap plugins.If a connector to be upgraded was not available during the upgrade, then a message in /opt/lucidworks/fusion/3.1.x/var/upgrade/tmp/migrator.log
(on Unix) or C:\lucidworks\fusion\3.1.*x\var\upgrade\tmp\migrator.log
(on Windows) indicates this.Only datasources for connectors that are supported in the new Fusion version are upgraded. Datasources for custom connectors are not upgraded.If no Internet connection is available
If no Internet connection is available during an upgrade, the migrator cannot automatically download the connectors it needs. Using the Fusion UI or API later to install the connectors also might not be an option.In this case, download the connectors from Fusion 4.x Connector Downloads for all existing connectors and place them inapps/connectors/bootstrap-plugins
for the new deployment (on all Fusion nodes). Do so at the time indicated in the procedures that follow.Adding connectors during an upgrade
You can add connectors during an upgrade (that is, add connectors that are not in the old deployment).Download the connectors from Fusion 4.x Connector Downloads and place them inapps/connectors/bootstrap-plugins
for the new version (on all Fusion nodes).Object migrations and transformations
The migrator automatically migrates these Fusion 4.2 object types, transforming them as needed:- Collections
- Index pipelines
- Query pipelines
- Search cluster configurations
- Datasources
- Parsing configurations
- Object groups
- Links
- Tasks
- Jobs
- Spark objects
- Apps
- Appkit apps
- Index profiles
- Query profiles
- Blobs
In Fusion Server 4.0 and later, most objects exist in the context of apps. When you upgrade from Fusion Server 4.2.x to 4.2.y, the migrator upgrades app objects, all objects in or linked to objects in apps, and objects that are not linked to apps. You can explore the objects in Object Explorer.
Access control migration
The migrator upgrades all access control configurations:- Security realms
- Roles
- Users
Review known issues
Before upgrading, review the known issues to see whether any of them apply to the circumstances of your upgrade. Some known issues might require actions before upgrading.That article also contains instructions regarding what to do if an upgrade step fails.Upgrade on Unix
Use this procedure to upgrade Fusion on a single Unix node or on multiple Unix nodes.Perform the steps in this procedure on the indicated nodes on which Fusion is running (“Fusion nodes”). To perform an upgrade, Fusion nodes must have at least these services running:- API service (
api
) - Proxy service (
proxy
)
For every step on multiple nodes, ensure that the step completes on all Fusion nodes before going to the next step. There is the notion of a “main node” during the migration process. This node will be used for certain centralized migration activities that do not need to be done on every node, such as downloading connectors that are then uploaded to blob storage that is shared by all, etc. Just pick one of your Fusion nodes to be the “main node”; there is no special requirement as to which one you pick.
Ensure that your current version of Fusion has a valid license
Ensure that your current version of Fusion has a valid permanent Fusion license before proceeding with the upgrade. Place a validlicense.properties
file in the /opt/lucidworks/fusion/4.2.x/conf
directory.Download and install the newer version of Fusion
Perform these tasks on all Fusion nodes:- Select the Fusion release to which you are upgrading from Fusion Server 4.x File Download Links.
- Extract the newer version of Fusion:
For example, if Fusion is currently installed in
/opt/lucidworks/fusion/4.2.x
, then change your working directory to/opt/lucidworks/
and extract the file there. do not run the new version of Fusion yet. - Ensure that the new version of Fusion has a valid permanent Fusion license before proceeding with the upgrade. Place a valid
license.properties
file in the/opt/lucidworks/fusion/4.2.y/conf
directory. - (If there are custom
jar
files) If your deployment has customjar
files, add them to the new Fusion deployment. - (If you are performing an upgrade without Internet access) Without Internet access, the migrator cannot download new versions of connectors automatically. Download the new versions of connector zip files for your current connectors from Fusion 4.x Connector Downloads and place them in
apps/connectors/bootstrap-plugins
for the new deployment. - (If you are adding new connectors) If you want your new deployment to use connectors that are not in the current deployment, you can add them now. Download the connector zip files from Fusion 4.x Connector Downloads and place them in
apps/connectors/bootstrap-plugins
for the new deployment. - Verify that there is sufficient disk space for a second copy of the Solr index directory,
fusion/4.2.x/data/solr
. If there is not sufficient disk space, free up space before proceeding.
Download and install the Fusion migrator
Perform these tasks on all Fusion nodes:- Download the latest migrator zip file for Unix. (Do this now, even if you have downloaded the migrator before, to ensure that you have the latest version.)
- Create
FUSION_OLD
andFUSION_NEW
environment variables that point to the old and new Fusion installation directories respectively (using the full path).For example, when upgrading from Fusion 4.2.0 to 4.2.6: - Create this directory:
- Install the migrator:
Run the migrator
Perform these tasks on the indicated nodes:- (On all Fusion nodes) Start all Fusion services for the old version of Fusion:
- (Only on the main Fusion node) Run the migrator to export the configuration data from the old version of Fusion:
This message indicates that the step finished successfully:
- (On all Fusion nodes) Stop the old versions of Fusion services and Solr; but not ZooKeeper:
If Spark services are running, also stop those:You can see what is running with
$FUSION_OLD/bin/fusion status
. - (Only on secondary Fusion nodes) Prepare secondary nodes:
This message indicates that the step finished successfully:
- (On all Fusion nodes) Stop ZooKeeper for the old version of Fusion (unless you are using an external ZooKeeper instance, in which case you can ignore this step):
- (Only on the main Fusion node) Transform configuration data on the main Fusion node:
This message indicates that the step finished successfully:Depending on the size of your Solr index, this step can take a long time (for example, multiple tens of minutes).
- (On all Fusion nodes) Start ZooKeeper for the new version of Fusion (unless you are using an external ZooKeeper instance, in which case you can ignore this step):
- (Only on the main Fusion node) Import the first part of configuration data into the new version of Fusion:
This message indicates that the step finished successfully:
- (On all Fusion nodes) Start all Fusion services for the new version of Fusion:
- (Only on the main Fusion node) Import the second part of configuration data into the new version of Fusion:
This message indicates that the step finished successfully:After migration, you can find details about the results in the
fusion/4.2.x/var/upgrade/tmp
directory. If the migration produces unexpected results, the files in this directory are helpful for troubleshooting.
Validate the new version of Fusion
How to validate the new version of Fusion- (Only on the main Fusion node) Restart the new version of Fusion (all services defined in
fusion.properties
): - Log into the Fusion UI (your
admin
password is the same as for the old installation), and confirm the release number of the new version of Fusion:- Clear your browser’s cache. Otherwise, you might inadvertently access a cached version of the old Fusion UI and see inconsistent behavior.
- In a browser, open the Fusion UI at
http://localhost:8764/
(replacelocalhost
with your server name or IP address if necessary). - Log in.
- Navigate to Admin > About Fusion. The About Fusion panel should display the newer Fusion release number.
- Ensure that all connectors were installed automatically during the upgrade.
- For Fusion 4.x from the Fusion launcher, click the tile for a migrated app. Click System > Blobs. If any connectors are missing from the list, click Add > Connector Plugin and install them manually.
- For Fusion 3.x from the Fusion launcher, click Devops > Home
> Blobs. If any connectors are missing from the list, click Add > Connector Plugin and install them manually.
- Ensure that all customizations you made in the former version of Fusion are present in the new one.
- When you are satisfied with the migration and you have backed up the
fusion/4.2.x/
directory, you canrm -fr fusion/4.2.*x/
to remove the older version of Fusion (on all Fusion nodes).
Add support for business rules to existing apps
Fusion AI 4.2 introduces functionality for using business rules, that is, manually created formulas for rewriting queries and responses.You can add support for business rules to apps that were created in versions of Fusion AI prior to version 4.2. To do so, perform the steps in this section.Theadd-rule-objects-xyz.zip
file (where xyz
is a version number) specifies the objects to add to an app. It is supplied in the Fusion migrator zip file at the top level. After installing the migrator, the location is $FUSION_OLD/var/upgrade/import-files
.Adding support for business rules has a costs. Additional collections and objects are created. Only add support for business rules to apps in which you plan to use them.
Fusion UI
For each app in which you plan to use business rules, import the objects in theadd-rule-objects-xyz.zip
file into the app.How to import business rule objects- In the Fusion launcher, click the app into which you want to import objects. The Fusion workspace appears.
-
In the upper left, click System
> Import Fusion Objects. The Import Fusion Objects window opens.
-
For the data file, select
add-rule-objects-xyz.zip
from your local filesystem. The location in the extracted migrator files is$FUSION_OLD/var/upgrade/import-files
. - Click Import.
-
Edit the
Application ID
parameter value to use the app name. If the app name contains spaces, replace those with underscore characters. For example,Lucene Revolution
would becomeLucene_Revolution
. - Click Import.
- If there are conflicts, Fusion prompts you to specify an import policy. Click Merge to skip all conflicting objects and import only the non-conflicting objects. Fusion confirms that the import was successful.
- Click Close to close the Import Fusion Objects window.
Fusion API
For each app in which you plan to use business rules, import the objects in theadd-rule-objects-xyz.zip
file into the app.How to import business rule objects- Create an
app-name.txt
file with the following content:For example, for the appLucene Revolution
:Here, we assume that you create the files in your home directory, for which the$HOME
environment variable is defined. - Import the business rule objects:
For example:
Upgrade on Windows
Use this procedure to upgrade Fusion on a single Windows node or multiple Windows nodes.Perform the steps in this procedure on the indicated nodes on which Fusion is running (“Fusion nodes”). To perform an upgrade, Fusion nodes must have at least these services running:- API service (
api
) - Proxy service (
proxy
)
If you are upgrading Fusion on multiple nodes, then, for every step on multiple nodes, ensure that the step completes on all Fusion nodes before going to the next step. There is the notion of a “main node” during the migration process. This node will be used for certain centralized migration activities that do not need to be done on every node, such as downloading connectors that are then uploaded to blob storage that is shared by all, etc. Just pick one of your Fusion nodes to be the “main node”; there is no special requirement as to which one you pick.
Ensure that your current version of Fusion has a valid license
Ensure that your current version of Fusion has a valid permanent Fusion license before proceeding with the upgrade. Place a validlicense.properties
file in the C:\lucidworks\fusion\4.2.x\conf
directory.Download and install the newer version of Fusion
Perform these tasks on all Fusion nodes:- Select the Fusion release to which you are upgrading from Fusion Server 4.x File Download Links.
-
Move the
fusion-4.2.y.zip
file to the directory that contains thefusion\
directory. For example, if Fusion is installed inC:\lucidworks\fusion\4.2.x
, then move the file toC:\lucidworks
. -
Unzip the
fusion-4.2.y.zip
file. do not run the new version of Fusion yet. -
For Fusion 4.x, ensure that the new version of Fusion has a valid permanent Fusion license before proceeding with the upgrade. Place a valid
license.properties
file in theC:\lucidworks\fusion\4.2.y\conf
directory. -
(If there are custom
jar
files) If your deployment has customjar
files, add them to the new Fusion deployment. -
(If you are performing an upgrade without Internet access) Without Internet access, the migrator cannot download new versions of connectors automatically. Download the new versions of connector zip files for your current connectors from Fusion 4.x Connector Downloads and place them in
apps\connectors\bootstrap-plugins
for the new deployment. -
(If you are adding new connectors) If you want your new deployment to use connectors that are not in the current deployment, you can add them now. Download the connector zip files from Fusion 4.x Connector Downloads and place them in
apps\connectors\bootstrap-plugins
for the new deployment. -
Verify that there is sufficient disk space for a second copy of the Solr index directory,
fusion\4.2.x\data\solr
. If there is not sufficient disk space, free up space before proceeding.
Download and install the Fusion migrator
Perform these tasks on all Fusion nodes:- Download the latest migrator zip file for Windows. (Do this now, even if you have downloaded the migrator before, to ensure that you have the latest version.)
- Open a Command Prompt window and create
FUSION_OLD
andFUSION_NEW
environment variables that point to the old and new Fusion installation directories respectively. For example: - Create a
fusion\4.2.x\var\upgrade
directory. - Unzip the migrator zip file, and move the contents of the extracted folder to
+fusion\+4.2.x\var\upgrade
.
Run the migrator
Perform these tasks on the indicated nodes:-
(On all Fusion nodes) Start all Fusion services for the old version of Fusion:
-
(Only on the main Fusion node) Run the migrator to export the configuration data from the old version of Fusion:
This message indicates that the step finished successfully:
-
(On all Fusion nodes) Stop the old versions of Fusion services and Solr; but not ZooKeeper:
If Spark and SQL services are running, also stop those:You can see what is running with
%FUSION_OLD%\bin\fusion status
. -
(Only on secondary Fusion nodes) Prepare secondary nodes:
This message indicates that the step finished successfully:
-
(On all Fusion nodes) Stop ZooKeeper for the old version of Fusion (unless you are using an external ZooKeeper instance, in which case you can ignore this step):
-
(Only on the main Fusion node) Transform configuration data on the main Fusion node:
This message indicates that the step finished successfully:Depending on the size of your Solr index, this step can take a long time (for example, multiple tens of minutes).
-
(On all Fusion nodes) Start ZooKeeper for the new version of Fusion (unless you are only using an external ZooKeeper instance, in which case you can ignore this step):
-
(Only on the main Fusion node) Import the first part of configuration data into the new version of Fusion:
This message indicates that the step finished successfully:
-
(On all Fusion nodes) Start Solr for the new Fusion version:
-
(Only on the main Fusion node) Run a script to remove all old plugins from the blob store. Replace
solr-address
andsolr-port
as appropriate (as shown in the example):For example:This message indicates that plugins were deleted successfully: -
(On all Fusion nodes) Start all Fusion services for the new version of Fusion:
-
(Only on the main Fusion node) Import the second part of configuration data into the new version of Fusion:
This message indicates that the step finished successfully:
After migration, you can find details about the results in the
+fusion\+4.2.x\var\upgrade\tmp
directory. If the migration produces unexpected results, the files in this directory are helpful for troubleshooting.Validate the new version of Fusion
How to validate the new version of Fusion- (On all Fusion nodes) Restart all Fusion services for the new version of Fusion:
- Log into the Fusion UI (your
admin
password is the same as for the old installation), and confirm the release number of the new version of Fusion:- Clear your browser’s cache. Otherwise, you might inadvertently access a cached version of the old Fusion UI and see inconsistent behavior.
- In a browser, open the Fusion UI at
http://localhost:8764/
(replacelocalhost
with your server name or IP address if necessary). - Log in.
- Navigate to Admin > About Fusion. The About Fusion panel should display the newer Fusion release number.
- Ensure that all connectors were installed automatically during the upgrade.
- For Fusion 4.x from the Fusion launcher, click the tile for a migrated app. Click System > Blobs. If any connectors are missing from the list, click Add > Connector Plugin and install them manually.
- For Fusion 3.x from the Fusion launcher, click Devops > Home
> Blobs. If any connectors are missing from the list, click Add > Connector Plugin and install them manually.
- Ensure that all customizations you made in the former version of Fusion are present in the new one.
- When you are satisfied with the migration and you have backed up the
+fusion\+4.2.x
directory, you can remove the older version of Fusion by removing that directory (on all Fusion nodes).
Add support for business rules to existing apps
Fusion AI 4.2 introduces functionality for using business rules, that is, manually created formulas for rewriting queries and responses.You can add support for business rules to apps that were created in versions of Fusion AI prior to version 4.2. To do so, perform the steps in this section.Theadd-rule-objects-xyz.zip
file (where xyz
is a version number) specifies the objects to add to an app. It is supplied in the Fusion migrator zip file at the top level. After installing the migrator, the location is %FUSION_OLD%\var\upgrade\import-files\
.You have a choice. You can update each app using the Fusion UI or the Fusion API.Fusion UI
For each app in which you plan to use business rules, import the objects in theadd-rule-objects-xyz.zip
file into the app.How to import business rule objects- In the Fusion launcher, click the app into which you want to import objects. The Fusion workspace appears.
- In the upper left, click System
> Import Fusion Objects. The Import Fusion Objects window opens.
- For the data file, select
add-rule-objects-xyz.zip
from your local filesystem. The location in the extracted migrator files is%FUSION_OLD%\var\upgrade\import-files\
. - Click Import.
- Edit the
Application ID
parameter value to use the app name. If the app name contains spaces, replace those with underscore characters. For example,Lucene Revolution
would becomeLucene_Revolution
. - Click Import.
- If there are conflicts, Fusion prompts you to specify an import policy. Click Merge to skip all conflicting objects and import only the non-conflicting objects. Fusion confirms that the import was successful.
- Click Close to close the Import Fusion Objects window.
Fusion API
For each app in which you plan to use business rules, import the objects in theadd-rule-objects-xyz.zip
file into the app.How to import business rule objects- Create an
app-name.txt
file with the following content:For example, for the appLucene Revolution
:Here, we assume that you create the files in your home directory, for which the%HOMEPATH%
environment variable is defined. - Import the business rule objects:
For example:
-
While indexing a SharePoint load test environment, some documents may be skipped due to an immense term error caused by a lengthy value in the
signature_s
field. -
The Spark Driver may run on restricted API nodes despite its access being forbidden by setting the
DrunSparkDriver
field toFalse
. - Jobs in the Query Workbench’s Recommender do not include associated Recommendation collections.
- The Query Workbench’s Text Tagger is not working with the spell correction collection as expected.
- Attempting to import the SharePoint connector object results in a validation error.
- When using the Tika parser with “Add original document content (raw bytes)” enabled, the resulting Base64 string contains too many bytes. This causes distortion of the data.
- The v2 connector reduces a multivalued field to a single value.
- The Google Drive Connector is not permitting users with the appropriate permissions to access data.
- While using the Google Drive Connector, duplicate document records are being produced for each user that has access to the folder and its items.
-
User access to group rules are not being appropriately restricted by the values in the
rules-<group name>
field. - When searching the DevOps center for messages by a specific logger class, a partial class name does not return results. The full class name must be used. For example, searching for “JmxServiceInstanceMetricReporter” will not return results. Users must search the full class name, “apollo.metrics.JmxServiceInstanceMetricReporter”.
-
Basic select SQL queries on
{collection}_signals
,{collection}_query_rewrite
,{collection}_query_rewrite_staging
fail unless an ORDER BY clause is included. - Spark jobs are not appearing in Job History until after the job is completed.
-
The
error.log
andoutput.log
files are being written to thevar/api/work
path instead ofvar/log/api
. -
When building a new UI in Fusion App Studio, the
Last Published Date
field contains a date and time value, despite the UI being unpublished. This value does not change when the new UI is published. This is only known to affect users of Windows 10. -
If a connector is deleted while the
connectors-rpc
service is stopped, theconnectors-rpc
service will fail to start. -
Setting the
Maximum Output Count
value does not stop the crawl process from continuing. - The Job History tab in the Scheduler will occasionally list the start and end time as identical values despite having a duration greater than 0 seconds.
- Recently created jobs are not immediately available in the Scheduler.
-
When Minimum Matching (
mm
) is applied to a synonym expansion, both parts of the synonym pair are required to produce query results. - The status indicator for services listed in the Cluster dashboard may not update properly. This is only known to affect users of Windows 10 and Internet Explorer 11.
- The number of collections displayed in the Cluster dashboard may not reflect the actual number of collections. This is only known to affect users of Windows 10 and Internet Explorer 11.
- In some cases, recently updated items in the Service Logs table will not appear.
-
For Windows 10 users, the Access Logs section of the DevOps Center does not display all request types, including
GET
,PUT
, etc. Additionally, values may be missing from some columns in the log table. - Non-administrator users are being granted the ability to delete index and query pipelines, despite not having the appropriate permissions.
- While creating a new rule in Fusion UI, some options may no appear due to the width restrictions of the page.