Upgrade Fusion 2.1.4 or 2.4.x to Fusion 3.0.0

This article describes how to upgrade from Fusion 2.1.4 or 2.4.x to Fusion 3.0.0 for Linux and Windows.

Note
 To upgrade to Fusion 3.0.0 from an earlier version of Fusion, first follow the steps to upgrade from 1.2 to 2.4; then follow the steps below.

Linux

The steps in this section describe how to upgrade from Fusion 2.1.4 or 2.4.x to Fusion 3.0 on Linux.

Prerequisites

Ensure that these prerequisites are met before upgrading Fusion.

Note
 If you have multi-node Fusion, this software is required on all nodes.

The upgrade scripts require Python 2.7, and that the following Python libraries are already installed:

Steps to upgrade from Fusion 2.4.x to 3.0.0 on Linux

These steps describe how to upgrade from Fusion 2.1.4 or 2.4.x to Fusion 3.0.0 on Linux.

Note
 If you have a multi-node installation and unless otherwise stated, you must perform each step on each node as you go. For example, if you have 3 nodes, perform step 1 on node 1, node 2, and then node 3. Next, perform step 2 on node 1, node 2, and then node 3. Run steps that are only needed on a single node on your first node; not on the other nodes.

  1. (On all nodes) Download Fusion 3.0.0.

  2. (On all nodes) Extract Fusion to a folder. Here we’ll call it fusion-new, as an example:

    mkdir fusion-new
tar -C fusion-new --strip-components=1 -xf fusion-3.0.0.tar.gz

  3. (On all nodes) Set the FUSION_OLD_HOME environment variable to the location of the old version of Fusion (2.x.y):

    export FUSION_OLD_HOME=/path/to/fusion/2.x.y

  4. (On all nodes) Set the FUSION_HOME environment variable to the location of the new version of Fusion (3.0):

    export FUSION_HOME=fusion-new/3.0.0

  5. (On all nodes) Copy data from the older fusion instance to the new fusion instance:

    cp -R $FUSION_OLD_HOME/data/*  $FUSION_HOME/data/

  6. (On all nodes) Clone the fusion-upgrade-scripts repository, if you haven’t already:

    git clone https://github.com/lucidworks/fusion-upgrade-scripts

  7. (On all nodes) Upgrade the configuration. This upgrades the customized properties in $FUSION_OLD_HOME/conf/config.sh to the new properties file in 3.0.0 ($FUSION_HOME/conf/fusion.properties).

    cd fusion-upgrade-scripts/src
python upgrade-to-3.0.py --upgrade config

    Alternatively, if all of the Fusion nodes use the same configuration, then you can just run this on one node, and then copy the new fusion.properties file to all of the other nodes.

  8. (On all nodes) Check the upgrade configuration in $FUSION_HOME/conf/config.sh and ensure that all of the modified properties in config.sh are reflected. Pay particular attention to the ZK connection strings if your existing Fusion installation is connected to an external ZooKeeper cluster.

  9. (On all nodes) If you are running an external Solr or Zookeeper, then modify the group.default property in fusion.properties to reflect the services that should be started when the bin/fusion script is executed.

  10. (On all nodes) Start the ZooKeeper service that the new Fusion installation will use. If you are using the ZooKeeper bundled within Fusion, that would be:

    cd $FUSION_HOME
./bin/zookeeper start

  11. (On all nodes) Copy your ZooKeeper configuration file over from the old installation to the new one:

    cp $FUSION_OLD_HOME\conf\zookeeper\zoo.cfg $FUSION_NEW_HOME\conf\zookeeper\zoo.cfg

  12. (For multi-node Fusion, only run this step on the first node) Upgrade ZooKeeper:

    cd fusion-upgrade-scripts-internal/src
python upgrade-to-3.0.py --upgrade zk

  13. (On all nodes) Run Fusion 3.0.0 (start all services defined in fusion.properties):

    cd $FUSION_HOME
./bin/fusion start

    Ensure that all services have started on all nodes before proceeding.

  14. (For multi-node Fusion, only run this step on the first node) Upgrade your custom dashboards, which are stored in the Solr collection system_banana (after the upgrade, they are stored in the Solr collection system_blobs):

    cd fusion-upgrade-scripts-internal/src
python upgrade-to-3.0.py --upgrade banana

  15. (On all nodes) Use the Fusion UI to validate the deployment.

    Tip
    		 Before opening the Fusion UI, clear your browser’s cache. Otherwise, you might inadvertently access a cached version of the old Fusion UI and see inconsistent behavior.

Windows

The steps in this section describe how to upgrade from Fusion 2.1.4 or 2.4.x to Fusion 3.0 on Windows.

Prerequisites

Ensure that these prerequisites are met before upgrading Fusion.

Note
 If you have multi-node Fusion, perform these steps on all nodes.

  1. Install python 2.7.13 from here: https://www.python.org/ftp/python/2.7.13/python-2.7.13.amd64.msi

  2. Open a normal Command Prompt window and set a PYTHON_HOME environment variable to the Python installation directory you just created:

    set PYTHON_HOME=C:\path\to\python\directory

    Example:

    set PYTHON_HOME=C:\Python27

    Keep this command prompt window open.

  3. Install the kazoo and requests python packages

    "%PYTHON_HOME%\Scripts\pip" install kazoo
"%PYTHON_HOME%\Scripts\pip" install requests
Steps to upgrade from Fusion 2.4.x to 3.0.0 on Windows

These steps describe how to upgrade from Fusion 2.1.4 or 2.4.x to Fusion 3.0.0 on Windows.

Note
 If you have a multi-node installation and unless otherwise stated, you must perform each step on each node as you go. For example, if you have 3 nodes, perform step 1 on node 1, node 2, and then node 3. Next, perform step 2 on node 1, node 2, and then node 3. Run steps that are only needed on a single node on your first node; not on the other nodes.

  1. (On all nodes) Create a fusion-new folder somewhere on your computer, and download and extract Fusion 3.0.0 to it.

  2. (On all nodes) Set the FUSION_HOME environment variable to the full path of the fusion\3.0.0 directory:

    Example:

    SET FUSION_HOME=C:\lucidworks\fusion-3.0.0\fusion\3.0.0

    At this point, %FUSION_HOME%\bin\fusion.cmd should point to the Fusion 3.0.0 fusion.cmd file.

  3. (On all nodes) Set the FUSION_OLD_HOME environment variable to the full path of the old Fusion home directory; for example:

    SET FUSION_OLD_HOME=C:\lucidworks\fusion

    At this point, %FUSION_OLD_HOME%\bin\fusion.cmd should point to the Fusion 2.x.y fusion.cmd file.

  4. (On all nodes) Stop the old Fusion services:

    cd "%FUSION_OLD_HOME%"
"%FUSION_OLD_HOME%\bin\fusion" stop

  5. (On all nodes) In Windows Explorer, copy the %FUSION_OLD_HOME%\data directory on top of the %FUSION_HOME%\data directory; respond with Overwrite All when prompted.

  6. (On all nodes) Edit %FUSION_HOME%\conf\fusion.properties and apply all custom JVM settings that you had applied to your %FUSION_OLD_HOME%\conf\config.cmd and %FUSION_OLD_HOME%\bin\*.cmd files into the new jvmOptions properties. If you have multi-node Fusion, make sure you uncomment default.zk.connect and set it to what you have FUSION_ZK set to.

    For example, many customers customize the max heap size for %FUSION_OLD_HOME%\bin\connectors.cmd. Here is an example for someone who changed it to use a max heap size of 4 GB:

    set "JAVA_OPTIONS=-Djava.net.preferIPv4Stack=true -Xmx4g -Dapple.awt.UIElement=true"

    If this were the case, you would need to use this entry in %FUSION_HOME%\conf\fusion.properties for the connectors.jvmOptions variable:

    connectors.jvmOptions=-Xmx4g -Xss256k  -Dcom.lucidworks.connectors.pipelines.embedded=false

  7. (On all nodes) Copy your ZooKeeper configuration file over from the old installation to the new one, that is, copy %FUSION_OLD_HOME%\conf\zookeeper\zoo.cfg to %FUSION_NEW_HOME%\conf\zookeeper\zoo.cfg.

  8. (On all nodes) Start the new ZooKeeper service:

    "%FUSION_HOME%\bin\zookeeper" start

    Ensure that ZooKeeper services have started on all nodes before proceeding.

  9. (For multi-node Fusion, only run this step on the first node) Download the fusion-upgrade-scripts project from https://github.com/lucidworks/fusion-upgrade-scripts. Click Download > Download Zip.

    Extract this zip file somewhere on your local computer, for example, to C:\lucidworks. When the zip file is extracted, you will have the directory C:\lucidworks\fusion-upgrade-scripts-master.

  10. (For multi-node Fusion, only run this step on the first node) Run the ZooKeeper upgrade script:

    cd "C:\lucidworks\fusion-upgrade-scripts-master\src"
"%PYTHON_HOME%\python" upgrade-to-3.0.py --upgrade zk

    Example output:

    C:\lucidworks\fusion-upgrade-scripts-master\src>"%PYTHON_HOME%\python" upgrade-to-3.0.py --upgrade zk
2017-10-04 08:45:25,517 - root - load_fusion_3x_config.py:36 - INFO - Creating config file using agent
2017-10-04 08:45:27,563 - root - load_fusion_3x_config.py:45 - INFO - Generated config file at path 'ui.config.json'
2017-10-04 08:45:27,563 - root - zookeeper_client.py:12 - INFO - Starting zookeeper client
2017-10-04 08:45:27,565 - kazoo.client - connection.py:599 - INFO - Connecting to 127.0.0.1:9983
2017-10-04 08:45:27,596 - kazoo.client - client.py:465 - INFO - Zookeeper connection established, state: CONNECTED
2017-10-04 08:45:27,612 - root - upgrade-to-3.0.py:64 - INFO - Migrating from fusion version '2.4.5' to '3.0.0'
2017-10-04 08:45:27,612 - root - upgrade-to-3.0.py:67 - INFO - Copying znodes from old fusion paths to new paths
2017-10-04 08:45:27,612 - root - znodes_migration.py:15 - INFO - Migrating Solr data to new ZK namespace /lwfusion/3.0.0/solr
2017-10-04 08:45:47,562 - root - znodes_migration.py:21 - INFO - Migrating api data to new ZK namespace lwfusion/3.0.0/core
2017-10-04 08:45:58,421 - root - znodes_migration.py:28 - INFO - Migrating proxy data to new ZK namespace lwfusion/3.0.0/proxy
2017-10-04 08:45:59,338 - root - upgrade-to-3.0.py:69 - INFO - Migration from old znode paths to new paths complete
2017-10-04 08:45:59,352 - root - api_pojo_migrator.py:16 - INFO - Updating search-cluster payload at path 'lwfusion/3.0.0/core/search-clusters/default'
2017-10-04 08:45:59,405 - root - proxy_pojo_migrator.py:15 - INFO - Updating init-meta payload at path 'lwfusion/3.0.0/proxy/sys/init-meta'
2017-10-04 08:45:59,490 - root - resource_manager.py:17 - INFO - Loading file from path C:\lucidworks\fusion-upgrade-scripts-master\src\utils\../resources/migrators.json
2017-10-04 08:45:59,490 - root - connectors_migrator.py:126 - INFO - Trying to migrate datasource: test2352352, type: jdbc
2017-10-04 08:45:59,490 - root - upgrade-to-3.0.py:78 - INFO - Performing splitter migrator
2017-10-04 08:45:59,523 - kazoo.client - connection.py:566 - INFO - Closing connection to 127.0.0.1:9983
2017-10-04 08:45:59,523 - kazoo.client - client.py:469 - INFO - Zookeeper session lost, state: CLOSED

  11. (On all nodes) Run Fusion 3.0.0 (start all services defined in fusion.properties), and wait for all services to start:

    "%FUSION_HOME%\bin\fusion" start

  12. (For multi-node Fusion, only run this step on the first node) Upgrade your custom dashboards, which are stored in the Solr collection system_banana (after the upgrade, they are stored in the Solr collection system_blobs):

    cd "C:\lucidworks\fusion-upgrade-scripts-master\src"
"%PYTHON_HOME%\python" upgrade-to-3.0.py --upgrade banana

    If you get an error containing Unknown collection system_banana, you can safely ignore it. That just means that you had no dashboards to migrate in the first place.

  13. (On all nodes) Use the Fusion UI to validate the deployment.

    Tip
    		 Before opening the Fusion UI, clear your browser’s cache. Otherwise, you might inadvertently access a cached version of the old Fusion UI and see inconsistent behavior.