Upgrade Fusion 3.0.x to 3.1.x

A migrator is available for upgrading Fusion 3.0.x to 3.1.x. The migrator migrates all 3.0 Fusion objects to Fusion 3.1 for a seamless upgrade. Migration will involve down time, and the process will involve multiple starts and stops of Fusion services. Please plan accordingly, especially in terms of disabling external load balancers or monitors that might react adversely to the starts and stops.

Note: The Fusion 3.1 instance must be newly untarred and never started.

All Fusion 3.0 object types are migrated:

  • Collections

  • Index pipelines

  • Query pipelines

  • Search cluster configurations

  • Schedules

  • Aggregations

  • Datasources

  • Dashboards

  • Parsers

Some objects are transformed for compatibility with the 3.1 set of object types:

  • New 3.1 object types are group, link, task, job, and spark.

  • The aggregation object type has been removed. Aggregation objects are migrated to the spark object type.

  • The schedule object type has been removed. Schedule objects are migrated to the job object type.

  • The new HTML and XML parser stages are added to all default parsers.

  • Any installed JDBC drivers from Fusion 3.0 are uploaded to the 3.1 blob store.

In Fusion 3.1, only four connectors are included by default. The migrator detects which connectors were used in Fusion 3.0, and installs them automatically in Fusion 3.1. This step requires an internet connection. If no connection is available, then connectors must be downloaded at http://lucidworks.com/connectors/ and installed separately. If this is necessary, then a message in fusion/3.0.x/var/upgrade/tmp/migrator.log will indicate so.

Important
Be sure to download the correct migrator package for your platform (Unix or Windows).

Upgrade on Unix (Single Node)

How to upgrade Fusion 3.0.x to 3.1.x on a single Unix node
  1. Download the latest version of Fusion and the migrator.

  2. Install Fusion 3.1.x:

    $ cd /fusion-parent/
    $ mv ~/Downloads/fusion-3.1.x.tar.gz ./
    $ tar -xf fusion-3.1.x.tar.gz

    For example, if Fusion 3.0.x is installed in /opt/fusion/3.0.x, then extract the file in the /opt/ directory. Don’t run Fusion 3.1.x.

  3. Create this folder:

    $ mkdir fusion/3.0.x/var/upgrade
  4. Install the migrator:

    $ tar -zxv -C ./fusion/3.0.x/var/upgrade --strip-components=1 -f ~/Downloads/fusion-migrator.tar.gz
  5. Stop Fusion 3.0.x if it is running:

    $ fusion/3.0.x/bin/fusion stop
  6. Run migrator.jar, using absolute paths:

    $ java -jar /path/to/fusion/3.0.x/var/upgrade/migrator.jar -f /path/to/fusion/3.0.x -t /path/to/fusion/3.1.x -p 3.0.x-3.1.x.properties
    Important
    In the above command, each x in the name of the properties file is a literal x. It isn’t a placeholder for a third-level version number.

    The end of the output from a successful migration looks like this:

    2017-07-12T13:35:38,654 - INFO  [main:Main@220] - Stopping new Fusion services
    2017-07-12T13:35:38,654 - INFO  [main:ServicesManager@194] - Starting process: java -jar /Users/username/Fusion/fusion/3.1.0/apps/lucidworks-agent.jar stop --no-agent zookeeper solr api connectors
    2017-07-12T13:35:43,430 - INFO  [main:Iterator@116] - Successfully stopped connectors (process ID 7748)
    2017-07-12T13:35:45,948 - INFO  [main:Iterator@116] - Successfully stopped api (process ID 7598)
    2017-07-12T13:35:50,237 - INFO  [main:Iterator@116] - Successfully stopped solr (process ID 7595)
    2017-07-12T13:35:51,375 - INFO  [main:Iterator@116] - Successfully stopped zookeeper (process ID 7575)
    Tip
    After migration, you can find details about the results in the fusion/3.0.x/var/upgrade/tmp directory. If the migration produces unexpected results, the files in this directory are helpful for troubleshooting.
  7. Set the FUSION_HOME environment variable to be the full path of the new Fusion 3.1.x folder:

    $ export FUSION_HOME=fusion/3.1.x
  8. Run Fusion 3.1.x (all services defined in fusion.properties):

    $ cd $FUSION_HOME
    $ ./bin/fusion start
  9. Validate the installation:

    1. Clear your browser’s cache.

      Otherwise, you might inadvertently access a cached version of the old Fusion UI and see inconsistent behavior.

    2. Access the Fusion UI at http://localhost:8764/.

    3. Log in.

    4. Navigate to admin > About Fusion.

      The About Fusion panel should display "Version: 3.1.x".

Tip
When you are satisfied with the migration and you have backed up the fusion/3.0.x/ directory, you can rm -fr fusion/3.0.x/ to remove Fusion 3.0.

Upgrade on Unix (Multiple Nodes)

How to upgrade Fusion 3.0.x to 3.1.x on multiple Unix nodes

Important: For every step on multiple nodes, ensure that the step completes on all nodes before going to the next step. There is also the notion of a "main node" during the migration process. This node will be used for certain centralized migration activities that do not need to be done on every node, such as downloading connectors that are then uploaded to blob storage that is shared by all, etc. Just pick one of your Fusion nodes to be the "main node"; there’s no special requirement as to which one you pick.

  1. Download the latest version of Fusion and the migrator.

  2. On all nodes, install Fusion 3.1.x:

    $ cd /fusion-parent/
    $ mv ~/Downloads/fusion-3.1.0.tar.gz ./
    $ tar -xf fusion-3.1.0.tar.gz

    For example, if Fusion 3.0.0 is installed in /opt/fusion/3.0.0, then extract the file in the /opt/ directory. Don’t run Fusion 3.1.x.

  3. Create this folder:

    $ mkdir fusion/3.0.x/var/upgrade
  4. On all nodes, install the migrator:

    $ tar -zxv -C ./fusion/3.0.x/var/upgrade --strip-components=1 -f ~/Downloads/fusion-migrator.tar.gz
  5. On the main node, run the migrator with the argument -1:

    $ java -jar $FUSION_HOME/var/upgrade/migrator.jar -f /var/lib/fusion/binary/fusion/3.0.x -t /var/lib/fusion/binary/fusion/3.1.x -p 3.0.x-3.1.x.properties -1

    Note: In the above command and in what follows, each x in the name of the properties file is just an x. It isn’t a placeholder for a third-level version number.

    This message indicates that the step finished successfully:

    Step 1 finished successfully.
  6. On all nodes, stop the Fusion UI, Fusion connectors, Fusion API, and Solr; but not ZooKeeper:

    $ ./fusion/bin/ui stop && ./fusion/bin/connectors stop && ./fusion/bin/api stop && ./fusion/bin/solr stop
  7. On all nodes, run the migrator with the argument -2:

    $ java -jar $FUSION_HOME/var/upgrade/migrator.jar -f /var/lib/fusion/binary/fusion/3.0.x -t /var/lib/fusion/binary/fusion/3.1.x -p 3.0.x-3.1.x.properties -2

    This message indicates that the step finished successfully:

    Step 2 finished successfully.
  8. On all nodes, stop ZooKeeper:

    $ ./fusion/bin/zookeeper stop
  9. On the main node, run the migrator with the argument -3:

    $ java -jar $FUSION_HOME/var/upgrade/migrator.jar -f /var/lib/fusion/binary/fusion/3.0.x -t /var/lib/fusion/binary/fusion/3.1.x -p 3.0.x-3.1.x.properties -3

    This message indicates that the step finished successfully:

    Step 3 finished successfully.
  10. On all nodes except the main node, run the migrator with the argument -7:

    java -jar $FUSION_HOME/var/upgrade/migrator.jar -f /var/lib/fusion/binary/fusion/3.0.x -t /var/lib/fusion/binary/fusion/3.1.x -p 3.0.x-3.1.x.properties -7

    This message indicates that the step finished successfully:

    Step 7 finished successfully.
  11. On all nodes, start ZooKeeper:

    ./fusion/3.1.0/bin/zookeeper start
  12. On the main node, run the migrator with the argument -4:

    $ java -jar /var/lib/fusion/binary/fusion/3.0.x/var/upgrade/migrator.jar -f /var/lib/fusion/binary/fusion/3.0.x -t /var/lib/fusion/binary/fusion/3.1.x -p 3.0.x-3.1.x.properties -4

    This message indicates that the step finished successfully:

    Step 4 finished successfully.
  13. On all nodes, start the Fusion UI, Fusion connectors, Fusion API, and Solr:

    $ ./fusion/3.1.0/bin/fusion start
  14. On the main node, run the migrator with the argument -5:

    java -jar /var/lib/fusion/binary/fusion/3.0.x/var/upgrade/migrator.jar -f /var/lib/fusion/binary/fusion/3.0.x -t /var/lib/fusion/binary/fusion/3.1.x -p 3.0.x-3.1.x.properties -5

    This message indicates that the step finished successfully:

    Step 5 finished successfully.
  15. On the main node, run the migrator with the argument -6:

    java -jar /var/lib/fusion/binary/fusion/3.0.x/var/upgrade/migrator.jar -f /var/lib/fusion/binary/fusion/3.0.x -t /var/lib/fusion/binary/fusion/3.1.x -p 3.0.x-3.1.x.properties -6

    This message indicates that the step finished successfully:

    Step 6 finished successfully.
    Tip
    After migration, you can find details about the results in the fusion/3.0.x/var/upgrade/tmp directory. If the migration produces unexpected results, the files in this directory are helpful for troubleshooting.
  16. Set the FUSION_HOME environment variable to be the full path of the new Fusion 3.1.x folder:

    # export FUSION_HOME=fusion/3.1.x
  17. Run Fusion 3.1.x (all services defined in fusion.properties):

    # cd $FUSION_HOME
    # ./bin/fusion start
  18. Validate the installation:

    1. Clear your browser’s cache.

      Otherwise, you might inadvertently access a cached version of the old Fusion UI and see inconsistent behavior.

    2. Access the Fusion UI at http://localhost:8764/.

    3. Log in.

    4. Navigate to admin > About Fusion.

      The About Fusion panel should display "Version: 3.1.x".

Tip
When you are satisfied with the migration and you have backed up the fusion/3.0.x/ directory, you can rm -fr fusion/3.0.x/ to remove Fusion 3.0.

Upgrade on Windows (Single Node)

How to upgrade Fusion 3.0.x to 3.1.x on a single Windows node
  1. Download Fusion 3.1 and the migrator zip file for Windows.

  2. Copy the fusion-3.1.0.zip file to the directory that contains the fusion\ directory.

    For example, if Fusion 3.0 is installed at C:\Users\admin\fusion\3.0.1, then copy the file to C:\Users\admin\fusion-3.1.x.zip.

  3. Unzip the fusion-3.1.x.zip file, for example with 7zip. Don’t run Fusion 3.1.x.

  4. Create a fusion\3.0.x\var\upgrade directory.

  5. Unzip the migrator zip file, and move the contents of the extracted folder to fusion\3.0.x\var\upgrade.

  6. Make sure Fusion is stopped.

  7. Run migrator.jar.

    Tip
    After migration, you can find details about the results in the fusion\3.0.x\var\upgrade\tmp directory. If the migration produces unexpected results, the files in this directory are helpful for troubleshooting.
  8. Run Fusion 3.1.x.

  9. Validate the installation:

    1. Clear your browser’s cache.

      Otherwise, you might inadvertently access a cached version of the old Fusion UI and see inconsistent behavior.

    2. Access the Fusion UI at http://localhost:8764/.

    3. Log in.

    4. Navigate to admin > About Fusion.

      The About Fusion panel should display "Version: 3.1.x".

Tip
When you are satisfied with the migration and you have backed up the fusion\3.0.x directory, you can remove that directory.