ZooKeeper Import/Export API

Apache ZooKeeper is a distributed configuration service, synchronization service, and naming registry. Managed Fusion uses ZooKeeper to configure and manage all Managed Fusion components in a single Managed Fusion deployment. The ZooKeeper Import/Export API provides methods to upload or download information from Managed Fusion’s ZooKeeper service. This service provides an alternative to the ZooKeeper clients zkCli.sh and zk-shell which are part of the Apache Zookeeper distribution included as part of the Managed Fusion distribution. The ZKImportExport service may be used to export ZooKeeper data for any Managed Fusion release. It can be used to import configuration data into the ZooKeeper service for a new or existing Managed Fusion deployment. For details on using this script to migrate Managed Fusion configurations from one deployment to another, see Migrate Fusion Objects.

Migrate Fusion Objects

You can use the Fusion UI and the Objects API to migrate collections and related objects, including your searchable data, configuration data, pipelines, aggregations, and other objects on which your collections depend. You can also migrate entire apps.

To upgrade from one Fusion version to a 5.x version, see Fusion 5 Upgrades to migrate objects.

You might need to migrate objects in the following circumstances:

When migrating data from development environments into testing and production environments
To back up data, so you can restore it after an incident of data loss
When the migrator script was not able to migrate an object automatically

When you export and import objects to migrate them, make sure you that any data gets to where it is going before deleting the sources.

Migration approaches

Several approaches are available for migrating Fusion objects. This table summarizes the approaches.

	Export an app	Import an app	Export an object	Import an object	Add an object to an app
Fusion UI	App configuration	Launcher (entire app) App configuration (combine apps)	-	-	Object Explorer
Objects API (endpoints)	GET from `export` endpoint	POST to `import` endpoint	GET from `export` endpoint	POST to `import` endpoint	-

For more information about using the Objects API to export and import objects, see Objects API.The remainder of this topic describes approaches in the Fusion UI.Use the parts of the Fusion UI indicated in the table to export and import apps and specific objects. Exporting creates a zip file. To import, you select a data file and possibly a variable file.The approach with Object Explorer differs. With Object Explorer, you can add objects from other apps (or that are not linked to any apps) to the currently open app.

Export an app with the Fusion UI

How to export an app with the Fusion UI

Navigate to the launcher.
Hover over the app you want to export and click the Configure icon:
In the app config window, click Export app to zip:

See import for information to import the downloaded zip file into other instances of Fusion 5.x.

Import an app with the Fusion UI

How to import an app with the Fusion UI

Navigate to the launcher.
Click Import app.
Under Data File, click Choose File and select the zip file containing the app you want to import.

If your app has usernames and passwords in a separate file, select it under Variables File. If the Variables File is needed, it must be a separate file that is not in a .zip file. It is a .json map of variables to values. The following is an example:

{
"secret.dataSources.Inventory_BR_S3_DS.password":"SOMETHING",
"secret.dataSources.LLM_A_BR_S3_DS.password":"FmJSaDE9Tj5REDACTED",
"secret.dataSources.LLM_BR_Inventory_S3_DS.password":"FmJSaDE9Tj5GzIVvethAC4Huh",
"secret.dataSources.LLM_BR_Load_S3_DS.password":"FmJSaDE9Tj5GzIVvethAC4"
}

You can sometimes edit parameter values to use the new values in the imported app. If this is the case, Fusion displays a dialog box that lets you edit the parameter values. Make desired changes, and then click Import.

Copy an app

To copy an app from one deployment to a different one, export the app on the source deployment, and then import the app on the target deployment.

Import objects into an app

You can import objects into the currently open app.How to import objects into an open app

In the Fusion launcher, click the app into which you want to import objects. The Fusion workspace appears.
Click System > Import Fusion Objects. The Import Fusion Objects window opens.
Select the data file from your local filesystem. If you are importing usernames and passwords in a separate file, select it under Variables File.
Click Import. If there are conflicts, Fusion prompts you to specify an import policy:
- Click Overwrite to overwrite the objects on the target system with the ones in the import file.
- Click Merge to skip all conflicting objects and import only the non-conflicting objects.
- Click Start Over to abort the import. Fusion confirms that the import was successful:
Click Close to close the Import Fusion Objects window.

Add an object to an app

You can add objects present in other apps (or in no apps) to the open app. Some objects are linked to other apps. You can also add those directly to an app.

Add an object to an app – While in the Fusion workspace for the app to which you want to add an object, open Object Explorer and click In Any App. Search for or browse to the object you want to add. Hover over the object, click the App icon, and then click Add to this app.
Add an object to an app directly – In cases when an object is linked to an app, but is not linked directly to the app (it is linked via some dependency), you can add the object to an app directly. While in the Fusion workspace for the app to which you want to add an object directly, open Object Explorer and click In Any App. Search for or browse to the object you want to add. Hover over the object, click the App icon, and then click Add to this app directly.

The REST API only supports requests to export ZooKeeper configurations. The Managed Fusion distribution includes a utility script zkImportExport.sh which can be used to import ZooKeeper configuration as well as to export it from arbitrary Managed Fusion instances. See the official ZooKeeper documentation for details about using and managing a ZooKeeper service. For more information, view the API specification.

Utility script `zkImportExport.sh`

This script is located in the top-level Managed Fusion scripts directory. The script takes the following command-line arguments:

-c,--cmd <arg>        Command, one of: 'export', 'import', 'update', 'delete'.

-e,--encode <arg>     Type of encoding for znodes. Valid options:
                      'none', 'utf-8', 'base64', default is 'base64'.
                      Option 'none' will not return any data from the znodes.

-ep,--exclude <arg>   Exclude znode paths, followed by list of paths.
                      Can only be used to exclude nodes one level below the root node.

-eph,--ephemeral      Include ephemeral nodes while exporting znodes, boolean, default false.

-f,--filename <arg>   Name of file containing import/export data.

-h,--help             Display help page.

-ip,--include <arg>   Include znode paths to include, followed by a list of paths.
                      Can only be used to include nodes one level below the root node.

-nr,--non-recursive   Do not perform recursive operations on znodes.

-o,--overwrite        Overwrite data for existing znodes. Valid only with 'update' command.

-p,--path <arg>       Path from ZooKeeper root node, e.g. '/lucid/query-pipelines'.

-r,--recursive        Perform recursive operations on znodes.

-z,--zkhost <arg>     ZooKeeper Connect string, required.

Required arguments are:

-c, —cmd : operation to perform.
-z, —zkhost : the ZooKeeper Connect string.

Examples

Export all data from a local single-node ZooKeeper service, save data to a file:

zkImportExport.sh -zkhost EXAMPLE_COMPANY.b.lucidworks.cloud:9983 -cmd export -path / -filename znode_dump.json

Export all Fusion configurations from a local single-node ZooKeeper service, save data to a file:

zkImportExport.sh -zkhost EXAMPLE_COMPANY.b.lucidworks.cloud:9983 -cmd export -path /lucid -filename znode_lucid_dump.json

Export Fusion user databases, groups, roles, and realms configurations from a local single-node ZooKeeper service, save data to a file:

zkImportExport.sh -zkhost EXAMPLE_COMPANY.b.lucidworks.cloud:9983 -cmd export -path /lucid-apollo-admin -filename znode_lucid_admin_dump.json

Initial import of saved Fusion configuration into a new ZooKeeper:

zkImportExport.sh -zkhost EXAMPLE_COMPANY.b.lucidworks.cloud:9983 -cmd import -filename znode_lucid_dump.json

Note that the above command will fail if there is conflict between existing znode structures or contents between the ZooKeeper service and the dump file. Update information for Fusion’s ZooKeeper service:

zkImportExport.sh -zkhost EXAMPLE_COMPANY.b.lucidworks.cloud:9983 -cmd update -filename znode_lucid_dump.json

Remove a znode from Fusion’s ZooKeeper service:

zkImportExport.sh -zkhost EXAMPLE_COMPANY.b.lucidworks.cloud:9983 -cmd delete -path /lucid/test

Fusion REST API service ZKImportExport

The Fusion REST API can only be used to download information from ZooKeeper, via the ‘GET’ method with the following configuration:

  "zk-import-export::v1" : {
    "name" : "com.lucidworks.apollo.resources.ZKImportExportResource",
    "uri" : "/zk/export",
    "methods" : [ {
      "uri" : "/zk/export/{path:.*}",
      "name" : "getNodeInfo",
      "verb" : "GET",
      "pathParams" : [ {
        "name" : "path",
        "type" : "String"
      } ],
      "queryParams" : [ {
        "name" : "recursive",
        "type" : "Boolean"
      }, {
        "name" : "excludePaths",
        "type" : "List"
      }, {
        "name" : "includePaths",
        "type" : "List"
      }, {
        "name" : "encodeValues",
        "type" : "String"
      } ],
      "hasEntity" : false
    } ]
  }

GET data from path /lucid/query-pipelines

curl -u USERNAME:PASSWORD -X GET https://EXAMPLE_COMPANY.b.lucidworks.cloud/api/zk/export/lucid/query-pipelines

{
  "path" : "/lucid/query-pipelines",
  "children" : [ {
    "path" : "/lucid/query-pipelines/default",
    "data" : "ewogICJpZCIgOiAiZGVmYXVsdCIsCiAgInN0YWdlcyIgOiBbIHsKICAgICJ0eXBlIiA6ICJzZWFyY2gtZmllbGRzIiwKICAgICJpZCIgOiAiMzc1NmI1ZDctY2MwMC00MDAyLWJiOWQtNTQzNjQ4NzVjMjgyIiwKICAgICJyb3dzIiA6IDEwLAogICAgInN0YXJ0IiA6IDAsCiAgICAic2tpcCIgOiBmYWxzZSwKICAgICJsYWJlbCIgOiAic2VhcmNoLWZpZWxkcyIsCiAgICAidHlwZSIgOiAic2VhcmNoLWZpZWxkcyIKICB9LCB7CiAgICAidHlwZSIgOiAiZmFjZXQiLAogICAgImlkIiA6ICI3MTFmZWM1Ni03MzRmLTRiYTAtOWY1NS0wZTQ4YmU2NTllM2UiLAogICAgInNraXAiIDogZmFsc2UsCiAgICAibGFiZWwiIDogImZhY2V0IiwKICAgICJ0eXBlIiA6ICJmYWNldCIKICB9LCB7CiAgICAidHlwZSIgOiAic29sci1xdWVyeSIsCiAgICAiaWQiIDogIjYwNDU1YjU2LTdkN2MtNDZjYi1iN2QzLTIxOWU1N2U3MWNjMyIsCiAgICAiaHR0cE1ldGhvZCIgOiAiUE9TVCIsCiAgICAic2tpcCIgOiBmYWxzZSwKICAgICJsYWJlbCIgOiAic29sci1xdWVyeSIsCiAgICAidHlwZSIgOiAic29sci1xdWVyeSIKICB9IF0KfQ=="
  } ],
  "data" : ""
}

Get info for node /lucid/query-pipelines. Do not expand the znodes

curl -u USERNAME:PASSWORD -X GET https://EXAMPLE_COMPANY.b.lucidworks.cloud/api/zk/export/lucid/query-pipelines?recursive=true&encodeValues=none
{
  "path" : "/lucid/query-pipelines",
  "children" : [ {
    "path" : "/lucid/query-pipelines/default"
  } ]
}

Get info for node /lucid/query-pipelines. encode in utf-8

curl -u USERNAME:PASSWORD -X GET https://EXAMPLE_COMPANY.b.lucidworks.cloud/api/zk/export/lucid/query-pipelines?recursive=true&encodeValues=utf-8
{
  "path" : "/lucid/query-pipelines",
  "children" : [ {
    "path" : "/lucid/query-pipelines/default",
    "data" : "{\n  \"id\" : \"default\",\n  \"stages\" : [ {\n    \"type\" : \"search-fields\",\n    \"id\" : \"3756b5d7-cc00-4002-bb9d-54364875c282\",\n    \"rows\" : 10,\n    \"start\" : 0,\n    \"skip\" : false,\n    \"label\" : \"search-fields\",\n    \"type\" : \"search-fields\"\n  }, {\n    \"type\" : \"facet\",\n    \"id\" : \"711fec56-734f-4ba0-9f55-0e48be659e3e\",\n    \"skip\" : false,\n    \"label\" : \"facet\",\n    \"type\" : \"facet\"\n  }, {\n    \"type\" : \"solr-query\",\n    \"id\" : \"60455b56-7d7c-46cb-b7d3-219e57e71cc3\",\n    \"httpMethod\" : \"POST\",\n    \"skip\" : false,\n    \"label\" : \"solr-query\",\n    \"type\" : \"solr-query\"\n  } ]\n}"
  } ],
  "data" : ""
}

Get info for node /lucid/query-pipelines. Exclude path /lucid/query-pipelines/default

curl -u USERNAME:PASSWORD -X GET https://EXAMPLE_COMPANY.b.lucidworks.cloud/api/zk/export/lucid/query-pipelines?recursive=true&encodeValues=utf-8&excludePaths=/lucid/query-piplines/default
{
  "path" : "/lucid/query-pipelines",
  "data" : ""
}

UI tour

Index data

Query data

Metrics and analytics

Improve your queries

Administration

Developer documentation

Machine learning

Neural Hybrid Search

Release notes

FAQs

Migration approaches

Export an app with the Fusion UI

Import an app with the Fusion UI

Copy an app

Import objects into an app

Add an object to an app

Utility script `zkImportExport.sh`

Examples

Fusion REST API service ZKImportExport

UI tour

Index data

Query data

Metrics and analytics

Improve your queries

Administration

Developer documentation

Machine learning

Neural Hybrid Search

Release notes

FAQs

​Utility script zkImportExport.sh

​Examples

​Fusion REST API service ZKImportExport

Utility script `zkImportExport.sh`

Examples

Fusion REST API service ZKImportExport