Install a Fusion Cluster (Unix)

This article describes how to install a Fusion cluster on multiple Unix nodes.

Three cluster arrangements are suitable for production deployments. Two additional cluster arrangements are recommended solely for testing deployments (if they are used at all). At some point, it is important to test what you plan to deploy in production. For more information, see the deployment types.

Production deployments

Here, we give installation instructions for three Fusion cluster arrangements suitable for production use.

In these Fusion cluster arrangements, the ZooKeeper cluster is on separate (non-Fusion) nodes.

Preliminary steps

Before proceeding to one of the sections that follow, perform these steps:

To prepare for setting up a Fusion cluster
  1. Prepare your firewall so that the Fusion nodes can communicate with each other. The default ports list contains a list of all ports used by Fusion. From this list, it is important that the ZooKeeper ports, Apache Ignite ports, and the Spark ports (if you are using Spark) are open between the different nodes for cross-cluster communication.

    If you plan to use an external SolrCloud cluster and/or an external ZooKeeper cluster, then also prepare your firewall so that Fusion nodes can communicate with the SolrCloud and ZooKeeper nodes.

  2. Download the Fusion tar/zip file to each node that will run Fusion.

    Note
    To leverage the copies of Solr and/or ZooKeeper that are distributed with Fusion on nodes that will not run Fusion (as a simple means of obtaining compatible versions of the other software), also download the Fusion tar/zip file to each of those nodes. Below, you will edit configuration files so that Fusion doesn’t run on those nodes.
  3. On each node, change your working directory to the directory in which you placed the Fusion tar/zip file and unpack the archive, for example:

    $ cd /opt/lucidworks
    $ tar -xf fusion-version.x.tar.gz

    The resulting directory is named fusion/3.1.x. You can rename this if you wish. This directory is considered your Fusion home directory. See Directories and Logs for the contents of the fusion/3.1.x directory.

Important: In the sections that follow, for every step on multiple nodes, complete the step on all nodes before going to the next step. It is especially important that you don’t start Fusion on any node until the instructions say to do so.

In the steps below, the port numbers reflect default port numbers and one common choice (port 2181 for nodes in an external ZooKeeper cluster). Port numbers for your nodes might differ.

Cluster arrangement 1: ZooKeeper cluster on separate nodes & same-node Solr instances

Each node in the Fusion cluster has Fusion and Solr installed.

Fusion cluster

To set up a Fusion cluster

Perform the steps in the section Preliminary steps, and then perform these steps:

  1. Edit conf/fusion.properties and remove zookeeper from the group.default list. This will make it so that ZooKeeper doesn’t start when you start Fusion.

  2. On each Fusion node, edit default.zk.connect in fusion/3.1.x/conf/fusion.properties to point to the ZooKeeper hosts:

    default.zk.connect=[ZK host 1]:2181,[ZK host 2]:2181,[ZK host 3]:2181
  3. On each node, start ZooKeeper with bin/zookeeper start. Zookeeper should start without errors. If a ZooKeeper instance fails to start, check the log at fusion/3.1.x/var/log/zookeeper/zookeeper.log.

  4. On each node, start Solr with bin/solr start. Solr should start without errors. If a Solr instance fails to start, check the log at fusion/3.1.x/var/log/solr/solr.log.

  5. On each node, start the rest of Fusion using bin/fusion start.

  6. Create an admin password and log in to Fusion at http://FIRST_NODE_IP:8764, where FIRST_NODE_IP is the IP address of your first Fusion node.

  7. Verify the Solr cluster is healthy by looking at http://ANY_NODE_IP:8983/solr/#/~cloud, where ANY_NODE_IP is the IP address of a Solr node. All of the nodes should appear green.

  8. If necessary, prepare high availability by setting up a load balancer in front of Fusion so that it load balances between the Fusion UI URL’s at http://NODE_IP:8764.

Cluster arrangement 2A or 2B: ZooKeeper cluster on separate nodes & external SolrCloud cluster

Each node in the Fusion cluster has Fusion installed. Fusion uses an external SolrCloud cluster for both data (for example, signals, aggregated signals, and indexes) and logs.

The ZooKeeper cluster is on non-Fusion, non-Solrcloud nodes; or it is on SolrCloud nodes.

2A

2B

Fusion cluster

External SolrCloud cluster

To set up a Fusion cluster

Perform the steps in the section Preliminary steps, and then perform these steps:

  1. Edit conf/fusion.properties and remove zookeeper from the group.default list. This will make it so that ZooKeeper doesn’t start when you start Fusion.

  2. On each Fusion node, edit default.zk.connect in fusion/3.1.x/conf/fusion.properties to point to the ZooKeeper hosts:

    default.zk.connect=[ZK host 1]:2181,[ZK host 2]:2181,[ZK host 3]:2181
  3. Edit conf/fusion.properties and remove solr from the group.default list. This will make it so that Solr doesn’t start when you start Fusion.

  4. On each Fusion node, edit default.solrZk.connect in fusion/3.1.x/conf/fusion.properties to point to the ZooKeeper of the external SolrCloud:

    default.solrZk.connect=[ZK host 1]:2181,[ZK host 2]:2181,[ZK host 3]:2181/solr-zk-namespace

    Where solr-zk-namespace is the Solr ZooKeeper namespace (chroot) of your external SolrCloud cluster.

  5. On each node, start ZooKeeper with bin/zookeeper start. Zookeeper should start without errors. If a ZooKeeper instance fails to start, check the log at fusion/3.1.x/var/log/zookeeper/zookeeper.log.

  6. On each node, start Solr with bin/solr start. Solr should start without errors. If a Solr instance fails to start, check the log at fusion/3.1.x/var/log/solr/solr.log.

  7. On each node, start the rest of Fusion using bin/fusion start.

  8. Create an admin password and log in to Fusion at http://FIRST_NODE_IP:8764, where FIRST_NODE_IP is the IP address of your first Fusion node.

  9. Verify the Solr cluster is healthy by looking at http://ANY_NODE_IP:8983/solr/#/~cloud, where ANY_NODE_IP is the IP address of a Solr node. All of the nodes should appear green.

  10. If necessary, prepare high availability by setting up a load balancer in front of Fusion so that it load balances between the Fusion UI URL’s at http://NODE_IP:8764.

Cluster arrangement 3A or 3B: ZooKeeper cluster on separate nodes & both same-node Solr instances and an external SolrCloud cluster

Each node in the Fusion cluster has Fusion and Solr installed, and there is a SolrCloud cluster on other nodes. Fusion uses the same-node Solr instances for logs and for raw signal data. Fusion uses the external Solr instances for aggregated signal data and indexes.

The ZooKeeper cluster is on non-Fusion, non-Solrcloud nodes; or it is on SolrCloud nodes.

3A

3B

Fusion cluster

Same-node Solr plus SolrCloud

To set up a Fusion cluster

Perform the steps in the section Preliminary steps, and then perform these steps:

  1. Edit conf/fusion.properties and remove zookeeper from the group.default list. This will make it so that ZooKeeper doesn’t start when you start Fusion.

  2. On each Fusion node, edit default.zk.connect in fusion/3.1.x/conf/fusion.properties to point to the ZooKeeper hosts:

    default.zk.connect=[ZK host 1]:2181,[ZK host 2]:2181,[ZK host 3]:2181
  3. Edit conf/fusion.properties and remove solr from the group.default list. This will make it so that Solr doesn’t start when you start Fusion.

  4. On each Fusion node, edit default.solrZk.connect in fusion/3.1.x/conf/fusion.properties to point to the ZooKeeper of the external SolrCloud:

    default.solrZk.connect=[ZK host 1]:2181,[ZK host 2]:2181,[ZK host 3]:2181/solr-zk-namespace

    Where solr-zk-namespace is the Solr ZooKeeper namespace (chroot) of your external SolrCloud cluster.

  5. Assign a number to each Fusion node, starting at 1. We refer to the number we assign to each node as the ZooKeeper myid.

  6. On each Fusion node, create a fusion/3.1.x/data/zookeeper directory, and a file called myid in that directory. Edit the file and save the ZooKeeper myid assigned for this node as the only contents.

  7. On each Fusion node, open the fusion/3.1.x/conf/zookeeper/zoo.cfg file in a text editor and add the following after the clientPort line (change the hostnames or IP addresses to the correct ones for your servers):

    server.1=[Hostname/IP for ZooKeeper with myid 1]:2888:3888
    server.2=[Hostname/IP for ZooKeeper with myid 2]:2888:3888
    server.3=[Hostname/IP for ZooKeeper with myid 3]:2888:3888
    Note
    Don’t use localhost or 127.0.0.1 as the hostname/IP. Specify the hostname/IP that other nodes will use when communicating with the current node.
  8. On each Fusion node, edit default.zk.connect in fusion/3.1.x/conf/fusion.properties to point to the ZooKeeper hosts:

    default.zk.connect=[ZK host 1]:9983,[ZK host 2]:9983,[ZK host 3]:9983
  9. On each node, start ZooKeeper with bin/zookeeper start. Zookeeper should start without errors. If a ZooKeeper instance fails to start, check the log at fusion/3.1.x/var/log/zookeeper/zookeeper.log.

  10. On each node, start Solr with bin/solr start. Solr should start without errors. If a Solr instance fails to start, check the log at fusion/3.1.x/var/log/solr/solr.log.

  11. On each node, start the rest of Fusion using bin/fusion start.

  12. Create an admin password and log in to Fusion at http://FIRST_NODE_IP:8764, where FIRST_NODE_IP is the IP address of your first Fusion node.

  13. Verify the Solr cluster is healthy by looking at http://ANY_NODE_IP:8983/solr/#/~cloud, where ANY_NODE_IP is the IP address of a Solr node. All of the nodes should appear green.

  14. If necessary, prepare high availability by setting up a load balancer in front of Fusion so that it load balances between the Fusion UI URL’s at http://NODE_IP:8764.

Test deployments

You can use these cluster arrangements for testing, but we don’t recommended them for production deployments.

You can also use the cluster arrangements in Production deployments for testing. In fact, prior to a production roll-out, you should test Fusion in a test environment that matches your planned production environment.

In these Fusion cluster arrangements, the ZooKeeper cluster is on Fusion nodes.

Note
We don’t recommend these cluster arrangements for production Fusion clusters. Place ZooKeeper on separate nodes.

Cluster arrangement 4: ZooKeeper cluster on Fusion nodes & same-node Solr instances

Deploy Fusion on multiple nodes to create a Fusion cluster. Each node in the Fusion cluster has Fusion, Solr, and ZooKeeper installed.

Same-node Solr plus SolrCloud

To set up a Fusion cluster

Perform the steps in the section Preliminary steps, and then perform these steps:

  1. Assign a number to each Fusion node, starting at 1. We refer to the number we assign to each node as the ZooKeeper myid.

  2. On each Fusion node, create a fusion/3.1.x/data/zookeeper directory, and a file called myid in that directory. Edit the file and save the ZooKeeper myid assigned for this node as the only contents.

  3. On each Fusion node, open the fusion/3.1.x/conf/zookeeper/zoo.cfg file in a text editor and add the following after the clientPort line (change the hostnames or IP addresses to the correct ones for your servers):

    server.1=[Hostname/IP for ZooKeeper with myid 1]:2888:3888
    server.2=[Hostname/IP for ZooKeeper with myid 2]:2888:3888
    server.3=[Hostname/IP for ZooKeeper with myid 3]:2888:3888
    Note
    Don’t use localhost or 127.0.0.1 as the hostname/IP. Specify the hostname/IP that other nodes will use when communicating with the current node.
  4. On each Fusion node, edit default.zk.connect in fusion/3.1.x/conf/fusion.properties to point to the ZooKeeper hosts:

    default.zk.connect=[ZK host 1]:9983,[ZK host 2]:9983,[ZK host 3]:9983
  5. On each node, start ZooKeeper with bin/zookeeper start. Zookeeper should start without errors. If a ZooKeeper instance fails to start, check the log at fusion/3.1.x/var/log/zookeeper/zookeeper.log.

  6. On each node, start Solr with bin/solr start. Solr should start without errors. If a Solr instance fails to start, check the log at fusion/3.1.x/var/log/solr/solr.log.

  7. On each node, start the rest of Fusion using bin/fusion start.

  8. Create an admin password and log in to Fusion at http://FIRST_NODE_IP:8764, where FIRST_NODE_IP is the IP address of your first Fusion node.

  9. Verify the Solr cluster is healthy by looking at http://ANY_NODE_IP:8983/solr/#/~cloud, where ANY_NODE_IP is the IP address of a Solr node. All of the nodes should appear green.

  10. If necessary, prepare high availability by setting up a load balancer in front of Fusion so that it load balances between the Fusion UI URL’s at http://NODE_IP:8764.

Cluster arrangement 5: ZooKeeper cluster on Fusion nodes & external SolrCloud cluster

Deploy Fusion on multiple nodes to create a Fusion cluster. Fusion and ZooKeeper are installed on the first three nodes in the Fusion cluster. If you add more Fusion nodes, they don’t need ZooKeeper. Fusion uses an external SolrCloud cluster for both data (for example, signals, aggregated signals, and indexes) and logs.

Same-node ZooKeeper plus external SolrCloud

To set up a Fusion cluster

Perform the steps in the section Preliminary steps, and then perform these steps:

  1. Edit conf/fusion.properties and remove solr from the group.default list. This will make it so that Solr doesn’t start when you start Fusion.

  2. On each Fusion node, edit default.solrZk.connect in fusion/3.1.x/conf/fusion.properties to point to the ZooKeeper of the external SolrCloud:

    default.solrZk.connect=[ZK host 1]:2181,[ZK host 2]:2181,[ZK host 3]:2181/solr-zk-namespace

    Where solr-zk-namespace is the Solr ZooKeeper namespace (chroot) of your external SolrCloud cluster.

  3. Assign a number to each Fusion node, starting at 1. We refer to the number we assign to each node as the ZooKeeper myid.

  4. On each Fusion node, create a fusion/3.1.x/data/zookeeper directory, and a file called myid in that directory. Edit the file and save the ZooKeeper myid assigned for this node as the only contents.

  5. On each Fusion node, open the fusion/3.1.x/conf/zookeeper/zoo.cfg file in a text editor and add the following after the clientPort line (change the hostnames or IP addresses to the correct ones for your servers):

    server.1=[Hostname/IP for ZooKeeper with myid 1]:2888:3888
    server.2=[Hostname/IP for ZooKeeper with myid 2]:2888:3888
    server.3=[Hostname/IP for ZooKeeper with myid 3]:2888:3888
    Note
    Don’t use localhost or 127.0.0.1 as the hostname/IP. Specify the hostname/IP that other nodes will use when communicating with the current node.
  6. On each Fusion node, edit default.zk.connect in fusion/3.1.x/conf/fusion.properties to point to the ZooKeeper hosts:

    default.zk.connect=[ZK host 1]:9983,[ZK host 2]:9983,[ZK host 3]:9983
  7. On each node, start ZooKeeper with bin/zookeeper start. Zookeeper should start without errors. If a ZooKeeper instance fails to start, check the log at fusion/3.1.x/var/log/zookeeper/zookeeper.log.

  8. On each node, start Solr with bin/solr start. Solr should start without errors. If a Solr instance fails to start, check the log at fusion/3.1.x/var/log/solr/solr.log.

  9. On each node, start the rest of Fusion using bin/fusion start.

  10. Create an admin password and log in to Fusion at http://FIRST_NODE_IP:8764, where FIRST_NODE_IP is the IP address of your first Fusion node.

  11. Verify the Solr cluster is healthy by looking at http://ANY_NODE_IP:8983/solr/#/~cloud, where ANY_NODE_IP is the IP address of a Solr node. All of the nodes should appear green.

  12. If necessary, prepare high availability by setting up a load balancer in front of Fusion so that it load balances between the Fusion UI URL’s at http://NODE_IP:8764.