Objects API

The Objects API lets you import and export objects between Managed Fusion instances. Managed Fusion objects include all your searchable data, plus pipelines, aggregations, and other configurations on which your collections depend. As an alternative in some cases, you can Migrate Fusion Objects.

Migrate Fusion Objects

You can use the Fusion UI and the Objects API to migrate collections and related objects, including your searchable data, configuration data, pipelines, aggregations, and other objects on which your collections depend. You can also migrate entire apps.

To upgrade from one Fusion version to a 5.x version, see Fusion 5 Upgrades to migrate objects.

You might need to migrate objects in the following circumstances:

When migrating data from development environments into testing and production environments
To back up data, so you can restore it after an incident of data loss
When the migrator script was not able to migrate an object automatically

When you export and import objects to migrate them, make sure you that any data gets to where it is going before deleting the sources.

Migration approaches

Several approaches are available for migrating Fusion objects. This table summarizes the approaches.


	Export an app	Import an app	Export an object	Import an object	Add an object to an app
Fusion UI	App configuration	Launcher (entire app) App configuration (combine apps)	-	-	Object Explorer
Objects API (endpoints)	GET from `export` endpoint	POST to `import` endpoint	GET from `export` endpoint	POST to `import` endpoint	-

For more information about using the Objects API to export and import objects, see Objects API.The remainder of this topic describes approaches in the Fusion UI.Use the parts of the Fusion UI indicated in the table to export and import apps and specific objects. Exporting creates a zip file. To import, you select a data file and possibly a variable file.The approach with Object Explorer differs. With Object Explorer, you can add objects from other apps (or that are not linked to any apps) to the currently open app.

Export an app with the Fusion UI

How to export an app with the Fusion UI

Navigate to the launcher.
Hover over the app you want to export and click the Configure icon:
In the app config window, click Export app to zip:

See import for information to import the downloaded zip file into other instances of Fusion 5.x.

Import an app with the Fusion UI

How to import an app with the Fusion UI

Navigate to the launcher.
Click Import app.
Under Data File, click Choose File and select the zip file containing the app you want to import.

If your app has usernames and passwords in a separate file, select it under Variables File. If the Variables File is needed, it must be a separate file that is not in a .zip file. It is a .json map of variables to values. The following is an example:

{
"secret.dataSources.Inventory_BR_S3_DS.password":"SOMETHING",
"secret.dataSources.LLM_A_BR_S3_DS.password":"FmJSaDE9Tj5REDACTED",
"secret.dataSources.LLM_BR_Inventory_S3_DS.password":"FmJSaDE9Tj5GzIVvethAC4Huh",
"secret.dataSources.LLM_BR_Load_S3_DS.password":"FmJSaDE9Tj5GzIVvethAC4"
}

You can sometimes edit parameter values to use the new values in the imported app. If this is the case, Fusion displays a dialog box that lets you edit the parameter values. Make desired changes, and then click Import.

Copy an app

To copy an app from one deployment to a different one, export the app on the source deployment, and then import the app on the target deployment.

Import objects into an app

You can import objects into the currently open app.How to import objects into an open app

In the Fusion launcher, click the app into which you want to import objects. The Fusion workspace appears.
Click System > Import Fusion Objects. The Import Fusion Objects window opens.
Select the data file from your local filesystem. If you are importing usernames and passwords in a separate file, select it under Variables File.
Click Import. If there are conflicts, Fusion prompts you to specify an import policy:
- Click Overwrite to overwrite the objects on the target system with the ones in the import file.
- Click Merge to skip all conflicting objects and import only the non-conflicting objects.
- Click Start Over to abort the import. Fusion confirms that the import was successful:
Click Close to close the Import Fusion Objects window.

Add an object to an app

You can add objects present in other apps (or in no apps) to the open app. Some objects are linked to other apps. You can also add those directly to an app.

Add an object to an app – While in the Fusion workspace for the app to which you want to add an object, open Object Explorer and click In Any App. Search for or browse to the object you want to add. Hover over the object, click the App icon, and then click Add to this app.
Add an object to an app directly – In cases when an object is linked to an app, but is not linked directly to the app (it is linked via some dependency), you can add the object to an app directly. While in the Fusion workspace for the app to which you want to add an object directly, open Object Explorer and click In Any App. Search for or browse to the object you want to add. Hover over the object, click the App icon, and then click Add to this app directly.

ImportantFor both the API and UI, migrating objects is intended for use within a specific Managed Fusion version. See Fusion 5 Upgrades to migrate objects from one Managed Fusion version to a specific Managed Fusion 5.x release.

Fusion 5 Upgrades

This guide describes how to perform Fusion 5 upgrades.

Before upgrading, be aware of changes by checking for Deprecations and Removals between versions.

Lucidworks recommends upgrading to the next minor version only. For example, you should upgrade from Fusion 5.6.1 to Fusion 5.7.1 before upgrading to Fusion 5.8.0.The general upgrade process is described in this article. However, the specific upgrade procedures may vary depending on your upgrade path. For the most accurate instructions, please refer to the upgrade article specific to your upgrade.

General upgrade process

Fusion natively supports deployments on supported Kubernetes platforms, including AKS, EKS, and GKE.Fusion includes an upgrade script for AKS, EKS, and GKE. This script is not generated for other Kubernetes deployments.Upgrades differ from platform to platform. See below for more information about upgrading on your platform of choice.Whenever you upgrade Fusion, you must also update your remote connectors, if you are running any. You can download the latest files at V2 Connectors Downloads.

Natively supported deployment upgrades

Deployment type	Platform
Azure Kubernetes Service (AKS)	`aks`
Amazon Elastic Kubernetes Service (EKS)	`eks`
Google Kubernetes Engine (GKE)	`gke`

Fusion includes upgrade scripts for natively supported deployment types. To upgrade:

Open the <platform>_<cluster>_<release>_upgrade_fusion.sh upgrade script file for editing.
Update the CHART_VERSION to your target Fusion version, and save your changes.
Run the <platform>_<cluster>_<release>_upgrade_fusion.sh script. The <release> value is the same as your namespace, unless you overrode the default value using the -r option.

After running the upgrade, use kubectl get pods to see the changes applied to your cluster. It may take several minutes to perform the upgrade, as new Docker images are pulled from DockerHub. To see the versions of running pods, do:

kubectl get po -o jsonpath='{..image}'  | tr -s '[[:space:]]' '\n' | sort | uniq

Other Kubernetes deployment upgrades

To update an existing installation, do:

RELEASE=f5
NAMESPACE=default
helm repo update
helm upgrade ${RELEASE} "lucidworks/fusion" --namespace "${NAMESPACE}" --values "${MY_VALUES}"

Except for ZooKeeper, all K8s deployments and statefulsets use a RollingUpdate update policy:

  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate

ZooKeeper instances use OnDelete to avoid changing critical stateful pods in the Fusion deployment. To apply changes to Zookeeper after performing the upgrade (uncommon), you need to manually delete the pods. For example:

kubectl delete pod f5-zookeeper-0

Delete one pod at a time. Verify the new pod is healthy and serving traffic, before deleting the next healthy pod.

You can also set the updateStrategy under the zookeeper section in your "${MY_VALUES}" file:

solr:
  ...
  zookeeper:
    updateStrategy:
      type: "RollingUpdate"

Upgrades with Helm v3

One of the most powerful features provided by Kubernetes and a cloud-native microservices architecture is the ability to do a rolling update on a live cluster. For example, Fusion 5 allows customers to upgrade from Fusion 5.1.0 to a later 5.x.y version on a live cluster with zero downtime or disruption of service.When Kubernetes performs a rolling update to an individual microservice, there is a mix of old and new services in the cluster. Requests from other services route to both versions.

Lucidworks ensures all changes we make to our service do not break the API interface exposed to other services in the same minor release version (5.x). We also ensure that the stored configuration remains compatible in the same minor release version.

Lucidworks releases minor updates to individual services frequently. Pull in those upgrades using Helm at your discretion.How to upgrade Fusion

Clone the fusion-cloud-native repo, if you haven’t already.
Locate the setup_f5_<platform>.sh script that matches your Kubernetes platform.
Run the script with the --upgrade option.
To see what would be upgraded, pass the --dry-run option to the script.

The scripts in the fusion-cloud-native repo automatically pull in the latest chart updates from our Helm repository and deploy any updates needed by doing a diff of your current installation and the latest release from Lucidworks.

Helm upgrade script

Once you deploy a working cluster, use the upgrade script created by the customize_fusion_values.sh script. The upgrade script hard-codes the parameters and eases the need to remember which parameters to pass to the script. This is helpful when working with multiple K8s clusters. Make sure you check the script into version control alongside your custom values YAML files.Whenever you change the custom values YAML files for your cluster, you need to run the upgrade script to apply the changes. The script calls helm upgrade with the correct parameters and --values options.

If you run helm upgrade without passing the custom values YAML files, the deployment will revert to using chart defaults, which you never want to do.

The script assumes your kubeconfig is pointing to the correct cluster and you’re using Heml v3. If not, the upgrade fails. Select the correct kubeconfig before running the script.

You can select all objects, or limit the operation to specific object types or IDs. In addition to export/import endpoints, a validation endpoint is provided for troubleshooting.

By default, system-created collections are not exported. See below for details.

For more information, view the API specification.

Object export and import

Collections and encrypted values are treated specially; details are provided below. During import, conflicts are resolved according to the specified import policy. For objects other than collections, no implicit filtering is performed; all objects are included by default. However, on export you can filter by type and ID.

Supported objects

Managed Fusion lets you export and import object types. ZIP files are the output for all file types listed. To filter by type, use the values below with the type query string parameter; see the examples below.

Object type	Value
apps	`app` The `deep` parameter is not needed for this object type. In the case of apps, all linked objects are included by default.
Appkit apps	`appkit`
blob	`blob`
collections (with their dependent objects) See below for collection-specific details.	`collection`
data-model	`data-model`
datasources	`datasource`
experiment	`experiment`
index pipelines	`index-pipeline`
index profiles	`index-profile`
job configurations	`spark`, `type`, or `data-source`
link	`link`
object group	`group`
parsing configurations	`parser`
query pipelines	`query-pipeline`
query profiles	`query-profile`
schedules for schedulable jobs (spark, type, or datasource)	`job`
search cluster configurations	`search-cluster`
subscription	`subscription`
template	`template`
zone	`zone`

Exporting and importing collections

Collections are processed with these dependent objects:

features
index profiles
query profiles

Datasources, parser configurations, and pipeline configurations are not included when collections are exported or imported. These must be exported and imported explicitly. Only user-created collections are included by default. Certain types of collections are excluded:

the “default” collection
collections whose type is not DATA
collections whose names start with “system_”
“Secondary” collections, that is, collections created by features
Instead, create the same features on the target system; this automatically creates the corresponding secondary collections.

You can override these exclusions by specifying a collection, like this:

curl -u USERNAME:PASSWORD https://EXAMPLE_COMPANY.b.lucidworks.cloud/api/objects/export?collection.ids=default > collection.zip

Encrypted passwords

Some objects, such as datasources and query pipelines, include encrypted passwords for accessing secure data.

On export, these encrypted values are replaced with one of the following:
- secret.dataSources.datasourceID.password
- secret.queryPipelines.pipelineID.stages.active-directory-security-trimming.stageID.bindPassword
On import, the original, plaintext passwords must be provided in a separate JSON map as multipart form data:
```
{
  "secret.dataSources.datasourceID.password":"MyDatasourcePassword",
  "secret.queryPipelines.pipelineID.stages.active-directory-security-trimming.stageID.bindPassword":"MyADPassword"
}
```
Passwords can also be Migrate Fusion Objects.
Variables that do not start with secret. are ignored.

Migrate Fusion Objects

To upgrade from one Fusion version to a 5.x version, see Fusion 5 Upgrades to migrate objects.

You might need to migrate objects in the following circumstances:

When migrating data from development environments into testing and production environments
To back up data, so you can restore it after an incident of data loss
When the migrator script was not able to migrate an object automatically

When you export and import objects to migrate them, make sure you that any data gets to where it is going before deleting the sources.

Migration approaches

Several approaches are available for migrating Fusion objects. This table summarizes the approaches.


	Export an app	Import an app	Export an object	Import an object	Add an object to an app
Fusion UI	App configuration	Launcher (entire app) App configuration (combine apps)	-	-	Object Explorer
Objects API (endpoints)	GET from `export` endpoint	POST to `import` endpoint	GET from `export` endpoint	POST to `import` endpoint	-

Export an app with the Fusion UI

How to export an app with the Fusion UI

Navigate to the launcher.
Hover over the app you want to export and click the Configure icon:
In the app config window, click Export app to zip:

See import for information to import the downloaded zip file into other instances of Fusion 5.x.

Import an app with the Fusion UI

How to import an app with the Fusion UI

Navigate to the launcher.
Click Import app.
Under Data File, click Choose File and select the zip file containing the app you want to import.

{
"secret.dataSources.Inventory_BR_S3_DS.password":"SOMETHING",
"secret.dataSources.LLM_A_BR_S3_DS.password":"FmJSaDE9Tj5REDACTED",
"secret.dataSources.LLM_BR_Inventory_S3_DS.password":"FmJSaDE9Tj5GzIVvethAC4Huh",
"secret.dataSources.LLM_BR_Load_S3_DS.password":"FmJSaDE9Tj5GzIVvethAC4"
}

You can sometimes edit parameter values to use the new values in the imported app. If this is the case, Fusion displays a dialog box that lets you edit the parameter values. Make desired changes, and then click Import.

Copy an app

To copy an app from one deployment to a different one, export the app on the source deployment, and then import the app on the target deployment.

Import objects into an app

You can import objects into the currently open app.How to import objects into an open app

In the Fusion launcher, click the app into which you want to import objects. The Fusion workspace appears.
Click System > Import Fusion Objects. The Import Fusion Objects window opens.
Select the data file from your local filesystem. If you are importing usernames and passwords in a separate file, select it under Variables File.
Click Import. If there are conflicts, Fusion prompts you to specify an import policy:
- Click Overwrite to overwrite the objects on the target system with the ones in the import file.
- Click Merge to skip all conflicting objects and import only the non-conflicting objects.
- Click Start Over to abort the import. Fusion confirms that the import was successful:
Click Close to close the Import Fusion Objects window.

Add an object to an app

You can add objects present in other apps (or in no apps) to the open app. Some objects are linked to other apps. You can also add those directly to an app.

Add an object to an app – While in the Fusion workspace for the app to which you want to add an object, open Object Explorer and click In Any App. Search for or browse to the object you want to add. Hover over the object, click the App icon, and then click Add to this app.
Add an object to an app directly – In cases when an object is linked to an app, but is not linked directly to the app (it is linked via some dependency), you can add the object to an app directly. While in the Fusion workspace for the app to which you want to add an object directly, open Object Explorer and click In Any App. Search for or browse to the object you want to add. Hover over the object, click the App icon, and then click Add to this app directly.

Import policies

On import, the importPolicy parameter is required. It specifies what to do if any object in the import list already exists on the target system:


`abort`	If there are conflicts, then import nothing.
`merge`	If there are conflicts, then skip the conflicting objects.
`overwrite`	If there are conflicts, then overwrite or delete/create the conflicting objects on the target system.

Filtering on export

On export, there are two ways to specify the objects to include:

by type
To export only specific types of objects, you can specify one or more object types using the type query string parameter. See Supported objects for a list of valid values.
by type and ID The type.ids parameter lets you list the IDs to match for the specified object type.

The type and type.ids parameters can be combined as needed.

Exporting linked objects

Related Managed Fusion objects are linked. You can view linked objects using the Links API or the Object Explorer. When exporting a specific Managed Fusion object, you can also export its linked objects without specifying each one individually. To export all objects linked to the specified object, include the deep="true" query parameter in your request. See the example below.

Validation

Objects are validated before import. If any objects fail validation, the whole import request is rejected. A separate endpoint is available for validating objects without importing them. Validation includes checking whether an object already exists on the target system and whether the user is authorized to create or modify the object. For collection objects, the following special validation is performed:

We check the searchClusterId of each collection and verify that a cluster with this ID exists on the target system or in the import file (error).
We check that features, index profiles, and query profiles belong only to the collections specified in the import file (error).
We check that a feature exists on the target system for each feature in the import file (error).
We check for index profiles or query profiles that do not exist on the target system or in the import file (warning).

Status messages


`Validation completed with no errors`	The validation method was called and no errors found, though there may be warnings.
`Validation found errors`	The validation was called and errors found. Validation does not stop on the first error, so the complete list of errors is reported.
`Validation was not completed because of system error`	The validation was interrupted by system error.
`Import was not performed because validation errors exist`	The import method was called, but import didn’t start because of validation errors.
`Import was not performed because of input data error`	The import method was called, but import didn’t start, because Managed Fusion could not find a substitution for one of the secret values in objects in import.
`Import was not completed because of system error`	The validation found no errors and import started, but it was interrupted by system error.
`Import was completed`	Validation found no errors and import finished successfully.

Examples

Export all objects

curl -u USERNAME:PASSWORD https://EXAMPLE_COMPANY.b.lucidworks.cloud/api/objects/export > all-objects.zip

Export all datasources

curl -u USERNAME:PASSWORD https://EXAMPLE_COMPANY.b.lucidworks.cloud/api/objects/export?type=datasource > datasources.zip

Export a specific datasource and all its linked objects

curl -u USERNAME:PASSWORD https://EXAMPLE_COMPANY.b.lucidworks.cloud/api/objects/export?datasource.ids=movies_csv-movies&deep=true > movies.zip

Export all apps

curl -u USERNAME:PASSWORD https://EXAMPLE_COMPANY.b.lucidworks.cloud/api/objects/export?type=app > all-apps.zip

Get all app IDs, then export one app by ID

curl -u USERNAME:PASSWORD https://EXAMPLE_COMPANY.b.lucidworks.cloud/api/apps

[ {
  "id" : "movies",
  "name" : "Movies",
  "description" : "Search the movielens database.",
  "dataUri" : "/App-Tile-01-460x160.png",
  "properties" : {
    "headerImageName" : "headerImage1",
    "tileColor" : "apps-darkblue",
    "previousCollectionId" : "movies"
  }
}, {
  "id" : "tech-pubs",
  "name" : "TechPubs",
  "description" : "Search the documentation.",
  "dataUri" : "/App-Tile-02-460x160.png",
  "properties" : {
    "headerImageName" : "headerImage2",
    "tileColor" : "apps-darkblue",
    "previousCollectionId" : "tech-pubs"
  }
} ]

curl -u USERNAME:PASSWORD https://EXAMPLE_COMPANY.b.lucidworks.cloud/api/objects/export?app.ids=tech-pubs > techpubs.zip

When you export an app that includes objects that are shared with other apps, then all apps linked to that object are also exported.

Export two apps by ID

curl -u USERNAME:PASSWORD https://EXAMPLE_COMPANY.b.lucidworks.cloud/api/objects/export?app.ids=tech-pubs,movies > two-apps.zip

Export all datasources and pipelines, plus two specific parsing configurations

curl -u USERNAME:PASSWORD 'https://EXAMPLE_COMPANY.b.lucidworks.cloud/api/objects/export?type=datasource,index-pipeline,query-pipeline&parser.ids=movies,tech-pubs' > export.zip

Import objects from a file and stop if there are conflicts

curl -u USERNAME:PASSWORD -H "Content-Type:multipart/form-data" -X POST -F 'importData=@/Users/admin/Fusion/export.zip' https://EXAMPLE_COMPANY.b.lucidworks.cloud/api/objects/import?importPolicy=abort

Import objects, substitute the password variables, and merge any conflicts

curl -u USERNAME:PASSWORD -H "Content-Type:multipart/form-data" -X POST -F 'importData=@/Users/admin/Fusion/export.zip' -F 'variableValues=@password_file.json' https://EXAMPLE_COMPANY.b.lucidworks.cloud/api/objects/import?importPolicy=merge

password_file.json must contain plaintext passwords.

Import a zip file of Managed Fusion objects and merge any conflicts

curl -u USERNAME:PASSWORD -H "Content-Type:multipart/form-data" -X POST -F 'importData=@/Users/admin/Fusion/techpubs.zip' https://EXAMPLE_COMPANY.b.lucidworks.cloud/api/objects/import?importPolicy=merge

Import objects into the context of an app

curl -v -H 'Content-Type:multipart/form-data' -X POST -F 'importData=@app_part.zip' 'https://EXAMPLE_COMPANY.b.lucidworks.cloud:8765/api/v1/objects/import?importPolicy=overwrite&context=APP_NAME'

Validate objects

curl -v -H 'Content-Type:multipart/form-data' -X POST -F 'https://EXAMPLE_COMPANY.b.lucidworks.cloud/api/v1/objects/validate'

UI tour

Index data

Query data

Metrics and analytics

Improve your queries

Administration

Developer documentation

Machine learning

Neural Hybrid Search

Release notes

FAQs

Migration approaches

Export an app with the Fusion UI

Import an app with the Fusion UI

Copy an app

Import objects into an app

Add an object to an app

General upgrade process

Natively supported deployment upgrades

Other Kubernetes deployment upgrades

Upgrades with Helm v3

Helm upgrade script

Object export and import

Supported objects

Exporting and importing collections

Encrypted passwords

Migration approaches

Export an app with the Fusion UI

Import an app with the Fusion UI

Copy an app

Import objects into an app

Add an object to an app

Import policies

Filtering on export

Exporting linked objects

Validation

Status messages

Examples

UI tour

Index data

Query data

Metrics and analytics

Improve your queries

Administration

Developer documentation

Machine learning

Neural Hybrid Search

Release notes

FAQs

​Object export and import

​Supported objects

​Exporting and importing collections

​Encrypted passwords

​Import policies

​Filtering on export

​Exporting linked objects

​Validation

​Status messages

​Examples

Object export and import

Supported objects

Exporting and importing collections

Encrypted passwords

Import policies

Filtering on export

Exporting linked objects

Validation

Status messages

Examples