Solr 7.7.2 | ZooKeeper 3.4.13 | Spark 2.3.2 | Jetty 9.4.12.v20180830 | Ignite 2.6.0 |
New features
- The Confluence connector now supports New Technology LAN Manager (NTLM) authentication.
-
SQL aggregations aggregations now support the use of the
NOT
operator insideAND
orOR
clauses. For example:
Improvements
- V2 connectors now support multivalued fields.
- Wait times at the end of V2 connector jobs have been reduced.
-
Users can now enable or disable spell correction and tail rewrites separately in the Text Tagger stage.
- Recently created jobs are now available in the Scheduler immediately.
- Authentication is no longer required to send SMTP emails.
- Changed how the Fusion UI handles proxied requests to Solr in order to reduce errors.
-
Images sent to the Apache Tika parser with the
Include images
option unchecked are no longer discarded. -
Added a new field,
unsafe
, to the index pipeline schema.
Other changes
- Fixed an issue with the SharePoint connector, which was skipping some documents in the index pipeline.
- Fixed an error with the SharePoint Online connector’s security trimming feature, which failed to filter by the “Everyone except external users” group.
-
Fixed an issue which caused new information added to
fusion.properties
to be appended to an existing value instead of creating a new line. - Fixed an issue which caused the misspelling, phrase, and synonym detection features to publish corrections which had low confidence scores.
- Fixed an issue which caused file-extension filtering in the SharePoint V1 connector to be case-sensitive.
- Fixed an issue which produced a Null Pointer Exception (NPE) error while using SharePoint security trimming on a user with no user groups.
- Fixed how Cross-Origin Resource Sharing (CORS) is handled when the port is missing from the host/origin headers.
- Fixed an issue that caused the SmartForm login process in the Web V1 connector to fail.
-
Fixed an issue which caused multivalued fields that were being sent to the signals collection to be assigned the single value string field type
_s
instead of the multivalued string field type_ss
. - Fixed minor issues with the drag-to-reorder feature in the Fusion UI.
- Fixed an issue that caused the Confluence connector to not create the title field or update the last modified date value.
- Improved how bi-directional synonyms are handled.
- Improved processes for using multiple Chrome/PhantomJS workers in parallel to crawl JavaScript-enabled websites.
- Fixed an error that could result from the Connectors Classic Service failing to start while attempting to migrate from 3.1.5 to 4.2.1.
- Fixed an error that caused the SharePoint Online datasources to disappear while upgrading Fusion.
- Fixed an issue which caused a target collection to not be found while attempting to transfer documents from a Solr collection in Fusion 4.1.x to a Solr collection in Fusion 3.1.x.
Addressed Security Vulnerabilities
- Fusion 4.2.3 upgrades Solr to version 7.7.2, which uses Jackson Release 2.9.8. This resolves multiple Common Vulnerabilities and Exposures (CVE).
- Fusion 4.2.3 upgrades Hibernate Validator to version 4.1.0, which resolves multiple CVEs.
Known issues
- When under load, the Fusion proxy service can occasionally become stuck, causing user authentication to fail. This is the result of the proxy
InputStream
failing to close properly.
An upgrade to Fusion 4.2.4 is required to fix this issue. See Upgrade Fusion Server 4.2.x to 4.2.y.
Upgrade Fusion Server 4.2.x to 4.2.y
Upgrade Fusion Server 4.2.x to 4.2.y
Introduction
This article describes how to perform the following upgrade:- From version: Fusion
4.2.x
- To version: Fusion
4.2.y
Only specific version-to-version upgrade sequences are supported. Some upgrades require multiple steps.
/opt/lucidworks/fusion/4.2.x/var/upgrade
directory (on Unix or MacOS) or the C:\lucidworks\fusion\4.2.*x\var\upgrade\
directory (on Windows). The file names reference the versions you are upgrading from and to. For example:- To upgrade
4.2.3
to4.2.5
, the migrator uses the4.2.x-4.2.x.properties
file.
The newer Fusion instance must be newly untarred and never started.
About the upgrade
This section describes how connectors, object migrations, and signals are migrated during an upgrade.Connectors
In Fusion 3.1.0 and later, only a subset of connectors are included by default.The migrator detects which connectors were used in the older version of Fusion, and installs them automatically in Fusion4.0.y
. This step requires an Internet connection. If no connection is available, then download the connectors from Fusion 4.x Connector Downloads and [install them as bootstrap plugins.If a connector to be upgraded was not available during the upgrade, then a message in /opt/lucidworks/fusion/3.1.x/var/upgrade/tmp/migrator.log
(on Unix) or C:\lucidworks\fusion\3.1.*x\var\upgrade\tmp\migrator.log
(on Windows) indicates this.Only datasources for connectors that are supported in the new Fusion version are upgraded. Datasources for custom connectors are not upgraded.If no Internet connection is available
If no Internet connection is available during an upgrade, the migrator cannot automatically download the connectors it needs. Using the Fusion UI or API later to install the connectors also might not be an option.In this case, download the connectors from Fusion 4.x Connector Downloads for all existing connectors and place them inapps/connectors/bootstrap-plugins
for the new deployment (on all Fusion nodes). Do so at the time indicated in the procedures that follow.Adding connectors during an upgrade
You can add connectors during an upgrade (that is, add connectors that are not in the old deployment).Download the connectors from Fusion 4.x Connector Downloads and place them inapps/connectors/bootstrap-plugins
for the new version (on all Fusion nodes).Object migrations and transformations
The migrator automatically migrates these Fusion 4.2 object types, transforming them as needed:- Collections
- Index pipelines
- Query pipelines
- Search cluster configurations
- Datasources
- Parsing configurations
- Object groups
- Links
- Tasks
- Jobs
- Spark objects
- Apps
- Appkit apps
- Index profiles
- Query profiles
- Blobs
In Fusion Server 4.0 and later, most objects exist in the context of apps. When you upgrade from Fusion Server 4.2.x to 4.2.y, the migrator upgrades app objects, all objects in or linked to objects in apps, and objects that are not linked to apps. You can explore the objects in Object Explorer.
Access control migration
The migrator upgrades all access control configurations:- Security realms
- Roles
- Users
Review known issues
Before upgrading, review the known issues to see whether any of them apply to the circumstances of your upgrade. Some known issues might require actions before upgrading.That article also contains instructions regarding what to do if an upgrade step fails.Upgrade on Unix
Use this procedure to upgrade Fusion on a single Unix node or on multiple Unix nodes.Perform the steps in this procedure on the indicated nodes on which Fusion is running (“Fusion nodes”). To perform an upgrade, Fusion nodes must have at least these services running:- API service (
api
) - Proxy service (
proxy
)
For every step on multiple nodes, ensure that the step completes on all Fusion nodes before going to the next step. There is the notion of a “main node” during the migration process. This node will be used for certain centralized migration activities that do not need to be done on every node, such as downloading connectors that are then uploaded to blob storage that is shared by all, etc. Just pick one of your Fusion nodes to be the “main node”; there is no special requirement as to which one you pick.
Ensure that your current version of Fusion has a valid license
Ensure that your current version of Fusion has a valid permanent Fusion license before proceeding with the upgrade. Place a validlicense.properties
file in the /opt/lucidworks/fusion/4.2.x/conf
directory.Download and install the newer version of Fusion
Perform these tasks on all Fusion nodes:- Select the Fusion release to which you are upgrading from Fusion Server 4.x File Download Links.
- Extract the newer version of Fusion:
For example, if Fusion is currently installed in
/opt/lucidworks/fusion/4.2.x
, then change your working directory to/opt/lucidworks/
and extract the file there. do not run the new version of Fusion yet. - Ensure that the new version of Fusion has a valid permanent Fusion license before proceeding with the upgrade. Place a valid
license.properties
file in the/opt/lucidworks/fusion/4.2.y/conf
directory. - (If there are custom
jar
files) If your deployment has customjar
files, add them to the new Fusion deployment. - (If you are performing an upgrade without Internet access) Without Internet access, the migrator cannot download new versions of connectors automatically. Download the new versions of connector zip files for your current connectors from Fusion 4.x Connector Downloads and place them in
apps/connectors/bootstrap-plugins
for the new deployment. - (If you are adding new connectors) If you want your new deployment to use connectors that are not in the current deployment, you can add them now. Download the connector zip files from Fusion 4.x Connector Downloads and place them in
apps/connectors/bootstrap-plugins
for the new deployment. - Verify that there is sufficient disk space for a second copy of the Solr index directory,
fusion/4.2.x/data/solr
. If there is not sufficient disk space, free up space before proceeding.
Download and install the Fusion migrator
Perform these tasks on all Fusion nodes:- Download the latest migrator zip file for Unix. (Do this now, even if you have downloaded the migrator before, to ensure that you have the latest version.)
- Create
FUSION_OLD
andFUSION_NEW
environment variables that point to the old and new Fusion installation directories respectively (using the full path).For example, when upgrading from Fusion 4.2.0 to 4.2.6: - Create this directory:
- Install the migrator:
Run the migrator
Perform these tasks on the indicated nodes:- (On all Fusion nodes) Start all Fusion services for the old version of Fusion:
- (Only on the main Fusion node) Run the migrator to export the configuration data from the old version of Fusion:
This message indicates that the step finished successfully:
- (On all Fusion nodes) Stop the old versions of Fusion services and Solr; but not ZooKeeper:
If Spark services are running, also stop those:You can see what is running with
$FUSION_OLD/bin/fusion status
. - (Only on secondary Fusion nodes) Prepare secondary nodes:
This message indicates that the step finished successfully:
- (On all Fusion nodes) Stop ZooKeeper for the old version of Fusion (unless you are using an external ZooKeeper instance, in which case you can ignore this step):
- (Only on the main Fusion node) Transform configuration data on the main Fusion node:
This message indicates that the step finished successfully:Depending on the size of your Solr index, this step can take a long time (for example, multiple tens of minutes).
- (On all Fusion nodes) Start ZooKeeper for the new version of Fusion (unless you are using an external ZooKeeper instance, in which case you can ignore this step):
- (Only on the main Fusion node) Import the first part of configuration data into the new version of Fusion:
This message indicates that the step finished successfully:
- (On all Fusion nodes) Start all Fusion services for the new version of Fusion:
- (Only on the main Fusion node) Import the second part of configuration data into the new version of Fusion:
This message indicates that the step finished successfully:After migration, you can find details about the results in the
fusion/4.2.x/var/upgrade/tmp
directory. If the migration produces unexpected results, the files in this directory are helpful for troubleshooting.
Validate the new version of Fusion
How to validate the new version of Fusion- (Only on the main Fusion node) Restart the new version of Fusion (all services defined in
fusion.properties
): - Log into the Fusion UI (your
admin
password is the same as for the old installation), and confirm the release number of the new version of Fusion:- Clear your browser’s cache. Otherwise, you might inadvertently access a cached version of the old Fusion UI and see inconsistent behavior.
- In a browser, open the Fusion UI at
http://localhost:8764/
(replacelocalhost
with your server name or IP address if necessary). - Log in.
- Navigate to Admin > About Fusion. The About Fusion panel should display the newer Fusion release number.
- Ensure that all connectors were installed automatically during the upgrade.
- For Fusion 4.x from the Fusion launcher, click the tile for a migrated app. Click System > Blobs. If any connectors are missing from the list, click Add > Connector Plugin and install them manually.
- For Fusion 3.x from the Fusion launcher, click Devops > Home
> Blobs. If any connectors are missing from the list, click Add > Connector Plugin and install them manually.
- Ensure that all customizations you made in the former version of Fusion are present in the new one.
- When you are satisfied with the migration and you have backed up the
fusion/4.2.x/
directory, you canrm -fr fusion/4.2.*x/
to remove the older version of Fusion (on all Fusion nodes).
Add support for business rules to existing apps
Fusion AI 4.2 introduces functionality for using business rules, that is, manually created formulas for rewriting queries and responses.You can add support for business rules to apps that were created in versions of Fusion AI prior to version 4.2. To do so, perform the steps in this section.Theadd-rule-objects-xyz.zip
file (where xyz
is a version number) specifies the objects to add to an app. It is supplied in the Fusion migrator zip file at the top level. After installing the migrator, the location is $FUSION_OLD/var/upgrade/import-files
.Adding support for business rules has a costs. Additional collections and objects are created. Only add support for business rules to apps in which you plan to use them.
Fusion UI
For each app in which you plan to use business rules, import the objects in theadd-rule-objects-xyz.zip
file into the app.How to import business rule objects- In the Fusion launcher, click the app into which you want to import objects. The Fusion workspace appears.
-
In the upper left, click System
> Import Fusion Objects. The Import Fusion Objects window opens.
-
For the data file, select
add-rule-objects-xyz.zip
from your local filesystem. The location in the extracted migrator files is$FUSION_OLD/var/upgrade/import-files
. - Click Import.
-
Edit the
Application ID
parameter value to use the app name. If the app name contains spaces, replace those with underscore characters. For example,Lucene Revolution
would becomeLucene_Revolution
. - Click Import.
- If there are conflicts, Fusion prompts you to specify an import policy. Click Merge to skip all conflicting objects and import only the non-conflicting objects. Fusion confirms that the import was successful.
- Click Close to close the Import Fusion Objects window.
Fusion API
For each app in which you plan to use business rules, import the objects in theadd-rule-objects-xyz.zip
file into the app.How to import business rule objects- Create an
app-name.txt
file with the following content:For example, for the appLucene Revolution
:Here, we assume that you create the files in your home directory, for which the$HOME
environment variable is defined. - Import the business rule objects:
For example:
Upgrade on Windows
Use this procedure to upgrade Fusion on a single Windows node or multiple Windows nodes.Perform the steps in this procedure on the indicated nodes on which Fusion is running (“Fusion nodes”). To perform an upgrade, Fusion nodes must have at least these services running:- API service (
api
) - Proxy service (
proxy
)
If you are upgrading Fusion on multiple nodes, then, for every step on multiple nodes, ensure that the step completes on all Fusion nodes before going to the next step. There is the notion of a “main node” during the migration process. This node will be used for certain centralized migration activities that do not need to be done on every node, such as downloading connectors that are then uploaded to blob storage that is shared by all, etc. Just pick one of your Fusion nodes to be the “main node”; there is no special requirement as to which one you pick.
Ensure that your current version of Fusion has a valid license
Ensure that your current version of Fusion has a valid permanent Fusion license before proceeding with the upgrade. Place a validlicense.properties
file in the C:\lucidworks\fusion\4.2.x\conf
directory.Download and install the newer version of Fusion
Perform these tasks on all Fusion nodes:- Select the Fusion release to which you are upgrading from Fusion Server 4.x File Download Links.
-
Move the
fusion-4.2.y.zip
file to the directory that contains thefusion\
directory. For example, if Fusion is installed inC:\lucidworks\fusion\4.2.x
, then move the file toC:\lucidworks
. -
Unzip the
fusion-4.2.y.zip
file. do not run the new version of Fusion yet. -
For Fusion 4.x, ensure that the new version of Fusion has a valid permanent Fusion license before proceeding with the upgrade. Place a valid
license.properties
file in theC:\lucidworks\fusion\4.2.y\conf
directory. -
(If there are custom
jar
files) If your deployment has customjar
files, add them to the new Fusion deployment. -
(If you are performing an upgrade without Internet access) Without Internet access, the migrator cannot download new versions of connectors automatically. Download the new versions of connector zip files for your current connectors from Fusion 4.x Connector Downloads and place them in
apps\connectors\bootstrap-plugins
for the new deployment. -
(If you are adding new connectors) If you want your new deployment to use connectors that are not in the current deployment, you can add them now. Download the connector zip files from Fusion 4.x Connector Downloads and place them in
apps\connectors\bootstrap-plugins
for the new deployment. -
Verify that there is sufficient disk space for a second copy of the Solr index directory,
fusion\4.2.x\data\solr
. If there is not sufficient disk space, free up space before proceeding.
Download and install the Fusion migrator
Perform these tasks on all Fusion nodes:- Download the latest migrator zip file for Windows. (Do this now, even if you have downloaded the migrator before, to ensure that you have the latest version.)
- Open a Command Prompt window and create
FUSION_OLD
andFUSION_NEW
environment variables that point to the old and new Fusion installation directories respectively. For example: - Create a
fusion\4.2.x\var\upgrade
directory. - Unzip the migrator zip file, and move the contents of the extracted folder to
+fusion\+4.2.x\var\upgrade
.
Run the migrator
Perform these tasks on the indicated nodes:-
(On all Fusion nodes) Start all Fusion services for the old version of Fusion:
-
(Only on the main Fusion node) Run the migrator to export the configuration data from the old version of Fusion:
This message indicates that the step finished successfully:
-
(On all Fusion nodes) Stop the old versions of Fusion services and Solr; but not ZooKeeper:
If Spark and SQL services are running, also stop those:You can see what is running with
%FUSION_OLD%\bin\fusion status
. -
(Only on secondary Fusion nodes) Prepare secondary nodes:
This message indicates that the step finished successfully:
-
(On all Fusion nodes) Stop ZooKeeper for the old version of Fusion (unless you are using an external ZooKeeper instance, in which case you can ignore this step):
-
(Only on the main Fusion node) Transform configuration data on the main Fusion node:
This message indicates that the step finished successfully:Depending on the size of your Solr index, this step can take a long time (for example, multiple tens of minutes).
-
(On all Fusion nodes) Start ZooKeeper for the new version of Fusion (unless you are only using an external ZooKeeper instance, in which case you can ignore this step):
-
(Only on the main Fusion node) Import the first part of configuration data into the new version of Fusion:
This message indicates that the step finished successfully:
-
(On all Fusion nodes) Start Solr for the new Fusion version:
-
(Only on the main Fusion node) Run a script to remove all old plugins from the blob store. Replace
solr-address
andsolr-port
as appropriate (as shown in the example):For example:This message indicates that plugins were deleted successfully: -
(On all Fusion nodes) Start all Fusion services for the new version of Fusion:
-
(Only on the main Fusion node) Import the second part of configuration data into the new version of Fusion:
This message indicates that the step finished successfully:
After migration, you can find details about the results in the
+fusion\+4.2.x\var\upgrade\tmp
directory. If the migration produces unexpected results, the files in this directory are helpful for troubleshooting.Validate the new version of Fusion
How to validate the new version of Fusion- (On all Fusion nodes) Restart all Fusion services for the new version of Fusion:
- Log into the Fusion UI (your
admin
password is the same as for the old installation), and confirm the release number of the new version of Fusion:- Clear your browser’s cache. Otherwise, you might inadvertently access a cached version of the old Fusion UI and see inconsistent behavior.
- In a browser, open the Fusion UI at
http://localhost:8764/
(replacelocalhost
with your server name or IP address if necessary). - Log in.
- Navigate to Admin > About Fusion. The About Fusion panel should display the newer Fusion release number.
- Ensure that all connectors were installed automatically during the upgrade.
- For Fusion 4.x from the Fusion launcher, click the tile for a migrated app. Click System > Blobs. If any connectors are missing from the list, click Add > Connector Plugin and install them manually.
- For Fusion 3.x from the Fusion launcher, click Devops > Home
> Blobs. If any connectors are missing from the list, click Add > Connector Plugin and install them manually.
- Ensure that all customizations you made in the former version of Fusion are present in the new one.
- When you are satisfied with the migration and you have backed up the
+fusion\+4.2.x
directory, you can remove the older version of Fusion by removing that directory (on all Fusion nodes).
Add support for business rules to existing apps
Fusion AI 4.2 introduces functionality for using business rules, that is, manually created formulas for rewriting queries and responses.You can add support for business rules to apps that were created in versions of Fusion AI prior to version 4.2. To do so, perform the steps in this section.Theadd-rule-objects-xyz.zip
file (where xyz
is a version number) specifies the objects to add to an app. It is supplied in the Fusion migrator zip file at the top level. After installing the migrator, the location is %FUSION_OLD%\var\upgrade\import-files\
.You have a choice. You can update each app using the Fusion UI or the Fusion API.Fusion UI
For each app in which you plan to use business rules, import the objects in theadd-rule-objects-xyz.zip
file into the app.How to import business rule objects- In the Fusion launcher, click the app into which you want to import objects. The Fusion workspace appears.
- In the upper left, click System
> Import Fusion Objects. The Import Fusion Objects window opens.
- For the data file, select
add-rule-objects-xyz.zip
from your local filesystem. The location in the extracted migrator files is%FUSION_OLD%\var\upgrade\import-files\
. - Click Import.
- Edit the
Application ID
parameter value to use the app name. If the app name contains spaces, replace those with underscore characters. For example,Lucene Revolution
would becomeLucene_Revolution
. - Click Import.
- If there are conflicts, Fusion prompts you to specify an import policy. Click Merge to skip all conflicting objects and import only the non-conflicting objects. Fusion confirms that the import was successful.
- Click Close to close the Import Fusion Objects window.
Fusion API
For each app in which you plan to use business rules, import the objects in theadd-rule-objects-xyz.zip
file into the app.How to import business rule objects- Create an
app-name.txt
file with the following content:For example, for the appLucene Revolution
:Here, we assume that you create the files in your home directory, for which the%HOMEPATH%
environment variable is defined. - Import the business rule objects:
For example:
-
The JDBC V1 connector ignores the
max_docs
field value when crawling a datasource. - It is not possible to use a future date as a business rule condition.
- Fusion overwrites existing Underperforming Query rules when the head/tail analysis job runs. This issue is fixed in Fusion 4.2.6.
-
The Scala script job sometimes fails when indentation is used.
This is fixed in Fusion 4.2.6.
- Jira no longer supports user/password authentication. As a result, the Jira connector must use a different authentication method.