ZooKeeper Export API

The ZooKeeper Export API provides methods to upload or download information from Fusion’s ZooKeeper service. This service provides an alternative to the ZooKeeper clients zkCli.sh and zk-shell which are part of the Apache Zookeeper distribution included as part of the Fusion distribution. The service may be used to export ZooKeeper data for any Fusion release.

For details on using this script during the Fusion upgrade procedure, see Upgrade to Fusion 4.x.
For details on using this script to migrate Fusion configurations from one deployment to another, see Migrate Fusion Objects.

Upgrade to Fusion 4.x

When you have a Fusion-based search application running, at some point it might be necessary to upgrade to a later version of Fusion. We provide a migrator tool to simplify the upgrade process.

See the release history to find out what is new, including which versions of Solr, Spark, and ZooKeeper are bundled with each Fusion release.

The migrator transfers over most of the objects that make up your search application, all configurations and customizations for your application, and all data in collections in the application.

In some cases, manual steps are required for objects that the migrator cannot handle automatically. We give you instructions and guidance about what might be required. You should also review the log of the upgrade in /opt/fusion/x.y.z/var/upgrade/tmp/migrator.log (on Unix) or C:\lucidworks\var\fusion\x.y.z\upgrade\tmp\migrator.log (on Windows). The x.y.z directory is for the Fusion version that you are migrating from.

Key points

Following are some key points about upgrading Fusion:

Migration involves down time. The upgrade process involves multiple starts and stops of Fusion services. Please plan accordingly, especially in terms of disabling external load balancers or monitors that might react adversely to the starts and stops.
Current deployment is preserved. Upgrades preserve the current Fusion deployment, copying information over from the current deployment to the new one. This provides a rapid roll-back option if you encounter problems during the upgrade process.
If the upgrade fails. If an upgrade fails, there is a procedure for dealing with that.

Supported upgrade sequences

Only specific version-to-version upgrade sequences are supported. Some upgrades require multiple steps.

These upgrade sequences are supported.

Upgrades to the current version

3.1.x to 4.2.y. From any 3.1.x version to 4.2.6 SP1 (one step, using the migrator)
4.0.x to 4.2.y. From any 4.0.x version to 4.2.6 SP1 (one step, using the migrator)
4.1.x to 4.2.y. From any 4.1.x version to 4.2.6 SP1 (one step, using the migrator)

For links to these procedures, see Per-version instruction sets.

Upgrades to prior versions

Using the migrator:

3.1.x to 4.0.y. From 3.1.5 directly to 4.0.2 (one step) For more information, see Upgrade Fusion 3.1.x to 4.0.y.
4.0.x to 4.0.y. From 4.0.0 or 4.0.1 to 4.0.2 (one step) For more information, see Upgrade Fusion Server 4.0.x to 4.0.y.
3.1.x to 4.1.y. From any 3.1.x version to 4.1.3 (one step, using the migrator) For more information, see Upgrade Fusion Server 3.1.x to 4.1.y.
4.0.x to 4.1.y. From 4.0.2 to 4.1.3 (one step, using the migrator) For more information, see Upgrade Fusion Server 4.0.x to 4.1.y.
4.1.x to 4.1.y. From 4.1.0 to 4.1.3 (one step, using the migrator) For more information, see Upgrade Fusion Server 4.1.x to 4.1.y.

Example

For example, to upgrade from Fusion 3.0.1 to Fusion Server 4.2.5, you would perform the following upgrades (both of them using the migrator):

Upgrade from Fusion 3.0.1 to Fusion 3.1.5
Upgrade from Fusion 3.1.5 to Fusion Server 4.2.5

Per-version instruction sets

To upgrade to a later version of Fusion from an existing installation requires transferring over all configurations and data from your existing Fusion installation to the new version.How to upgrade from Fusion 3.1.x to Fusion Server 4.2.yPerform the steps in this article:Upgrade from Fusion Server 3.1.x to 4.2.y - Run a migrator to upgrade from Fusion Server 3.1.x to 4.2.y.How to upgrade from Fusion 4.0.x to Fusion Server 4.2.yPerform the steps in this article:Upgrade from Fusion Server 4.0.x to 4.2.y - Run a migrator to upgrade from Fusion Server 4.0.x to 4.2.y.How to upgrade from Fusion 4.1.x to Fusion Server 4.2.yPerform the steps in this article:Upgrade from Fusion Server 4.1.x to 4.2.y - Run a migrator to upgrade from Fusion Server 4.1.x to 4.2.y.How to upgrade from Fusion 4.2.x to Fusion Server 4.2.yPerform the steps in this article:Upgrade from Fusion Server 4.2.x to 4.2.y - Run a migrator to upgrade from Fusion Server 4.2.x to 4.2.y.

Migrate Fusion Objects

You can use the Fusion UI and the Objects API to migrate collections and related objects, including your searchable data, configuration data, pipelines, aggregations, and other objects on which your collections depend. You can also migrate entire apps.

When upgrading from one Fusion version to a later one, you must use the migrator to migrate objects. The migrator automates the process of translating relevant objects to the new version. Exporting an object from one Fusion version and importing it into a different Fusion version isn’t supported.

You might need to migrate objects in the following circumstances:

When migrating data from development environments into testing and production environments
To back up data, so you can restore it after an incident of data loss
When the migrator script was not able to migrate an object automatically

When you export and import objects to migrate them, make sure you that any data gets to where it is going before deleting the sources.

Migration approaches

Several approaches are available for migrating Fusion objects. This table summarizes the approaches.

	Export an app	Import an app	Export an object	Import an object	Add an object to an app
Fusion UI	App configuration	Launcher (entire app) App configuration (combine apps)	-	-	Object Explorer
Objects API (endpoints)	GET from `export` endpoint	POST to `import` endpoint	GET from `export` endpoint	POST to `import` endpoint	-

For more information about using the Objects API to export and import objects, see Objects API.The remainder of this topic describes approaches in the Fusion UI.Use the parts of the Fusion UI indicated in the table to export and import apps and specific objects. Exporting creates a zip file. To import, you select a data file and possibly a variable file.The approach with Object Explorer differs. With Object Explorer, you can add objects from other apps (or that are not linked to any apps) to the currently open app.

Export an app with the Fusion UI

How to export an app with the Fusion UI

Navigate to the launcher.
Hover over the app you want to export and click the Configure icon:
In the app config window, click Export app to zip: See import for information to import the downloaded zip file into other instances of Fusion 4.x Server.

Import an app with the Fusion UI

How to import an app with the Fusion UI

Navigate to the launcher.
Click Import app.
Under Data File, click Choose File and select the zip file containing the app you want to import.
If your app has usernames and passwords in a separate file, select it under Variables File.
If the Variables File is needed, it must be a separate file that is not in a .zip file. It is a .json map of variables to values. The following is an example:

{
 "secret.dataSources.Inventory_BR_S3_DS.password":"SOMETHING",
 "secret.dataSources.LLM_A_BR_S3_DS.password":"FmJSaDE9Tj5REDACTED",
 "secret.dataSources.LLM_BR_Inventory_S3_DS.password":"FmJSaDE9Tj5GzIVvethAC4Huh",
 "secret.dataSources.LLM_BR_Load_S3_DS.password":"FmJSaDE9Tj5GzIVvethAC4"
}

In Fusion 4.1 and later, you can sometimes edit parameter values to use the new values in the imported app. If this is the case, Fusion displays a dialog box that lets you edit the parameter values. Make desired changes, and then click Import.

Copy an app

To copy an app from one deployment to a different one, export the app on the source deployment, and then import the app on the target deployment.

Import objects into an app

You can import objects into the currently open app.How to import objects into an open app

In the Fusion launcher, click the app into which you want to import objects.
The Fusion workspace appears.
In the upper left, click System > Import Fusion Objects. The Import Fusion Objects window opens.
Select the data file from your local filesystem.
If you are importing usernames and passwords in a separate file, select it under Variables File.
Click Import.
If there are conflicts, Fusion prompts you to specify an import policy:
- Click Overwrite to overwrite the objects on the target system with the ones in the import file.
- Click Merge to skip all conflicting objects and import only the non-conflicting objects.
- Click Start Over to abort the import.
  Fusion confirms that the import was successful:
Click Close to close the Import Fusion Objects window.

Add an object to an app

You can add objects present in other apps (or in no apps) to the open app. Some objects are linked to other apps. You can also add those directly to an app.

Add an object to an app – While in the Fusion workspace for the app to which you want to add an object, open Object Explorer and click In Any App. Search for or browse to the object you want to add. Hover over the object, click the App icon, and then click Add to this app.
Add an object to an app directly – In cases when an object is linked to an app, but is not linked directly to the app (it is linked via some dependency), you can add the object to an app directly. While in the Fusion workspace for the app to which you want to add an object directly, open Object Explorer and click In Any App. Search for or browse to the object you want to add. Hover over the object, click the App icon, and then click Add to this app directly.

The REST API only supports requests to export ZooKeeper configurations. The Fusion distribution includes a utility script zkImportExport.sh which can be used to export ZooKeeper configuration from arbitrary Fusion instances.

ZooKeeper

Apache ZooKeeper is a distributed configuration service, synchronization service, and naming registry. Fusion uses ZooKeeper to configure and manage all Fusion components in a single Fusion deployment.

znode. ZooKeeper data is organized into a hierarchal name space of data nodes called znodes.
A znode can have data associated with it as well as child znodes.
The data in a znode is stored in a binary format, but it is possible to import, export, and view this information as JSON data.
Paths to znodes are always expressed as canonical, absolute, slash-separated paths; there are no relative reference.
ephemeral nodes. An ephemeral node is a znode which exists only for the duration of an active session.
When the session ends the znode is deleted. An ephemeral znode cannot have children.
server. A ZooKeeper service consists of one or more machines; each machine is a server which runs in its own JVM and listens on its own set of ports.
For testing, you can run several ZooKeeper servers at once on a single workstation by configuring the ports for each server.
quorum. A quorum is a set of ZooKeeper servers. It must be an odd number. For most deployments, only 3 servers are required.
client. A client is any host or process which uses a ZooKeeper service.

See the official ZooKeeper documentation for details about using and managing a ZooKeeper service.

Utility script `zkImportExport.sh`

This script is located in the top-level Fusion scripts directory. The script takes the following command-line arguments:

-c,--cmd <arg>        Command, one of: 'export', 'import', 'update', 'delete'.

-e,--encode <arg>     Type of encoding for znodes. Valid options:
                       'none', 'utf-8', 'base64', default is 'base64'.
                       Option 'none' will not return any data from the znodes.

-ep,--exclude <arg>   Exclude znode paths, followed by list of paths.
                      Can only be used to exclude nodes one level below the root node.

-eph,--ephemeral      Include ephemeral nodes while exporting znodes, boolean, default false.

-f,--filename <arg>   Name of file containing import/export data.

-h,--help             Display help page.

-ip,--include <arg>   Include znode paths to include, followed by a list of paths.
                      Can only be used to include nodes one level below the root node.

-nr,--non-recursive   Do not perform recursive operations on znodes.

-o,--overwrite        Overwrite data for existing znodes. Valid only with 'update' command.

-p,--path <arg>       Path from ZooKeeper root node, e.g. '/lucid/query-pipelines'.

-r,--recursive        Perform recursive operations on znodes.

-z,--zkhost <arg>     ZooKeeper Connect string, required.

Required arguments are:

-c, —cmd : operation to perform.
-z, —zkhost : the ZooKeeper Connect string.

Examples

Export all data from a local single-node ZooKeeper service, save data to a file:

zkImportExport.sh -zkhost fusion-host:9983 -cmd export -path / -filename znode_dump.json

Export all Fusion configurations from a local single-node ZooKeeper service, save data to a file:

zkImportExport.sh -zkhost fusion-host:9983 -cmd export -path /lucid -filename znode_lucid_dump.json

Export Fusion user databases, groups, roles, and realms configurations from a local single-node ZooKeeper service, save data to a file:

zkImportExport.sh -zkhost fusion-host:9983 -cmd export -path /lucid-apollo-admin -filename znode_lucid_admin_dump.json

Initial import of saved Fusion configuration into a new ZooKeeper:

zkImportExport.sh -zkhost fusion-host:9983 -cmd import -filename znode_lucid_dump.json

Note that the above command will fail if there is conflict between existing znode structures or contents between the ZooKeeper service and the dump file. Update information for Fusion’s ZooKeeper service:

zkImportExport.sh -zkhost fusion-host:9983 -cmd update -filename znode_lucid_dump.json

Remove a znode from Fusion’s ZooKeeper service:

zkImportExport.sh -zkhost fusion-host:9983 -cmd delete -path /lucid/test

Fusion REST API service ZKImportExport

The Fusion REST API can only be used to download information from ZooKeeper, via the ‘GET’ method with the following configuration:

  "zk-import-export::v1" : {
    "name" : "com.lucidworks.apollo.resources.ZKImportExportResource",
    "uri" : "/zk/export",
    "methods" : [ {
      "uri" : "/zk/export/{path:.*}",
      "name" : "getNodeInfo",
      "verb" : "GET",
      "pathParams" : [ {
        "name" : "path",
        "type" : "String"
      } ],
      "queryParams" : [ {
        "name" : "recursive",
        "type" : "Boolean"
      }, {
        "name" : "excludePaths",
        "type" : "List"
      }, {
        "name" : "includePaths",
        "type" : "List"
      }, {
        "name" : "encodeValues",
        "type" : "String"
      } ],
      "hasEntity" : false
    } ]
  }

GET data from path /lucid/query-pipelines

curl -u USERNAME:PASSWORD -X GET https://FUSION_HOST:8764/api/zk/export/lucid/query-pipelines

{
  "path" : "/lucid/query-pipelines",
  "children" : [ {
    "path" : "/lucid/query-pipelines/default",
    "data" : "ewogICJpZCIgOiAiZGVmYXVsdCIsCiAgInN0YWdlcyIgOiBbIHsKICAgICJ0eXBlIiA6ICJzZWFyY2gtZmllbGRzIiwKICAgICJpZCIgOiAiMzc1NmI1ZDctY2MwMC00MDAyLWJiOWQtNTQzNjQ4NzVjMjgyIiwKICAgICJyb3dzIiA6IDEwLAogICAgInN0YXJ0IiA6IDAsCiAgICAic2tpcCIgOiBmYWxzZSwKICAgICJsYWJlbCIgOiAic2VhcmNoLWZpZWxkcyIsCiAgICAidHlwZSIgOiAic2VhcmNoLWZpZWxkcyIKICB9LCB7CiAgICAidHlwZSIgOiAiZmFjZXQiLAogICAgImlkIiA6ICI3MTFmZWM1Ni03MzRmLTRiYTAtOWY1NS0wZTQ4YmU2NTllM2UiLAogICAgInNraXAiIDogZmFsc2UsCiAgICAibGFiZWwiIDogImZhY2V0IiwKICAgICJ0eXBlIiA6ICJmYWNldCIKICB9LCB7CiAgICAidHlwZSIgOiAic29sci1xdWVyeSIsCiAgICAiaWQiIDogIjYwNDU1YjU2LTdkN2MtNDZjYi1iN2QzLTIxOWU1N2U3MWNjMyIsCiAgICAiaHR0cE1ldGhvZCIgOiAiUE9TVCIsCiAgICAic2tpcCIgOiBmYWxzZSwKICAgICJsYWJlbCIgOiAic29sci1xdWVyeSIsCiAgICAidHlwZSIgOiAic29sci1xdWVyeSIKICB9IF0KfQ=="
  } ],
  "data" : ""
}

Get info for node /lucid/query-pipelines. Do not expand the znodes

curl -u USERNAME:PASSWORD -X GET https://FUSION_HOST:8764/api/zk/export/lucid/query-pipelines?recursive=true&encodeValues=none
{
  "path" : "/lucid/query-pipelines",
  "children" : [ {
    "path" : "/lucid/query-pipelines/default"
  } ]
}

Get info for node /lucid/query-pipelines. encode in utf-8

curl -u USERNAME:PASSWORD -X GET https://FUSION_HOST:8764/api/zk/export/lucid/query-pipelines?recursive=true&encodeValues=utf-8
{
  "path" : "/lucid/query-pipelines",
  "children" : [ {
    "path" : "/lucid/query-pipelines/default",
    "data" : "{\n  \"id\" : \"default\",\n  \"stages\" : [ {\n    \"type\" : \"search-fields\",\n    \"id\" : \"3756b5d7-cc00-4002-bb9d-54364875c282\",\n    \"rows\" : 10,\n    \"start\" : 0,\n    \"skip\" : false,\n    \"label\" : \"search-fields\",\n    \"type\" : \"search-fields\"\n  }, {\n    \"type\" : \"facet\",\n    \"id\" : \"711fec56-734f-4ba0-9f55-0e48be659e3e\",\n    \"skip\" : false,\n    \"label\" : \"facet\",\n    \"type\" : \"facet\"\n  }, {\n    \"type\" : \"solr-query\",\n    \"id\" : \"60455b56-7d7c-46cb-b7d3-219e57e71cc3\",\n    \"httpMethod\" : \"POST\",\n    \"skip\" : false,\n    \"label\" : \"solr-query\",\n    \"type\" : \"solr-query\"\n  } ]\n}"
  } ],
  "data" : ""
}

Get info for node /lucid/query-pipelines. Exclude path /lucid/query-pipelines/default

curl -u USERNAME:PASSWORD -X GET https://FUSION_HOST:8764/api/zk/export/lucid/query-pipelines?recursive=true&encodeValues=utf-8&excludePaths=/lucid/query-piplines/default
{
  "path" : "/lucid/query-pipelines",
  "data" : ""
}

Fusion Server

Fusion AI

App Studio

Key points

Supported upgrade sequences

Upgrades to the current version

Upgrades to prior versions

Example

Per-version instruction sets

Migration approaches

Export an app with the Fusion UI

Import an app with the Fusion UI

Copy an app

Import objects into an app

Add an object to an app

ZooKeeper

Utility script `zkImportExport.sh`

Examples

Fusion REST API service ZKImportExport

Fusion Server

Fusion AI

App Studio

​Key points

​Supported upgrade sequences

​Upgrades to the current version

​Upgrades to prior versions

​Example

​Per-version instruction sets

​Migration approaches

​Export an app with the Fusion UI

​Import an app with the Fusion UI

​Copy an app

​Import objects into an app

​Add an object to an app

​ZooKeeper

​Utility script zkImportExport.sh

​Examples

​Fusion REST API service ZKImportExport

Key points

Supported upgrade sequences

Upgrades to the current version

Upgrades to prior versions

Example

Per-version instruction sets

Migration approaches

Export an app with the Fusion UI

Import an app with the Fusion UI

Copy an app

Import objects into an app

Add an object to an app

ZooKeeper

Utility script `zkImportExport.sh`

Examples

Fusion REST API service ZKImportExport