Skip to main content
This guide describes how to perform Fusion 5 upgrades. The general upgrade process is described in this article. However, the specific upgrade procedures may vary depending on your upgrade path. For the most accurate instructions, please refer to the upgrade article specific to your upgrade.
Fusion 5.9.x must be upgraded one maintenance release at a time. If you are on version 5.9.12 and want to reach 5.9.14, you are required to upgrade to 5.9.13 first, then upgrade to 5.9.14.
Fusion 5.9.5 introduced changes that affect Spark jobs. If you are upgrading to Fusion 5.9.5, follow the Fusion 5.9.5 upgrade instructions.
Before upgrading, be aware of changes by checking for Deprecations and Removals between versions.

​
General upgrade process

Fusion natively supports deployments on supported Kubernetes platforms, including AKS, EKS, and GKE. Fusion includes an upgrade script for AKS, EKS, and GKE. This script is not generated for other Kubernetes deployments. Upgrades differ from platform to platform. See below for more information about upgrading on your platform of choice. Whenever you upgrade Fusion, you must also update your remote connectors, if you are running any. You can download the latest files at V2 Connectors Downloads.
Check for any running jobs prior to upgrading to maintain data integrity and prevent job state corruption.
  1. Verify job status with the Connectors API: 
    curl GET https://FUSION_HOST/api/connectors/jobs?state=RUNNING
  2. Complete document processing.
    1. Wait for all pending documents to be indexed.
    2. Verify no retry operations are pending.
    3. Confirm index pipeline statistics show completion.

​
Natively supported deployment upgrades

Deployment typePlatform
Azure Kubernetes Service (AKS)aks
Amazon Elastic Kubernetes Service (EKS)eks
Google Kubernetes Engine (GKE)gke
Fusion includes upgrade scripts for natively supported deployment types. To upgrade:
  1. Open the <platform>_<cluster>_<release>_upgrade_fusion.sh upgrade script file for editing.
  2. Update the CHART_VERSION to your target Fusion version, and save your changes.
  3. Run the <platform>_<cluster>_<release>_upgrade_fusion.sh script. The <release> value is the same as your namespace, unless you overrode the default value using the -r option.
After running the upgrade, use kubectl get pods to see the changes applied to your cluster. It may take several minutes to perform the upgrade, as new Docker images are pulled from DockerHub. To see the versions of running pods, do: 
kubectl get po -o jsonpath='{..image}'  | tr -s '[[:space:]]' '\n' | sort | uniq

​
Other Kubernetes deployment upgrades

To update an existing installation, do: 
RELEASE=f5
NAMESPACE=default
helm repo update
helm upgrade ${RELEASE} "lucidworks/fusion" --namespace "${NAMESPACE}" --values "${MY_VALUES}"
Except for ZooKeeper, all K8s deployments and statefulsets use a RollingUpdate update policy: 
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
ZooKeeper instances use OnDelete to avoid changing critical stateful pods in the Fusion deployment. To apply changes to Zookeeper after performing the upgrade (uncommon), you need to manually delete the pods. For example: 
kubectl delete pod f5-zookeeper-0
Delete one pod at a time. Verify the new pod is healthy and serving traffic, before deleting the next healthy pod.
You can also set the updateStrategy under the zookeeper section in your "${MY_VALUES}" file: 
solr:
  ...  
    zookeeper:
    updateStrategy:
      type: "RollingUpdate"

​
Upgrades with Helm v3

One of the most powerful features provided by Kubernetes and a cloud-native microservices architecture is the ability to do a rolling update on a live cluster. For example, Fusion 5 allows customers to upgrade from Fusion 5.1.0 to a later 5.x.y version on a live cluster with zero downtime or disruption of service. When Kubernetes performs a rolling update to an individual microservice, there is a mix of old and new services in the cluster. Requests from other services route to both versions.
Lucidworks ensures all changes we make to our service do not break the API interface exposed to other services in the same minor release version (5.x). We also ensure that the stored configuration remains compatible in the same minor release version.
Lucidworks releases minor updates to individual services frequently. Pull in those upgrades using Helm at your discretion. How to upgrade Fusion
  1. Clone the fusion-cloud-native repo, if you haven’t already.
  2. Locate the setup_f5_<platform>.sh script that matches your Kubernetes platform.
  3. Run the script with the --upgrade option.
    To see what would be upgraded, pass the --dry-run option to the script.
The scripts in the fusion-cloud-native repo automatically pull in the latest chart updates from our Helm repository and deploy any updates needed by doing a diff of your current installation and the latest release from Lucidworks.

​
Helm upgrade script

Once you deploy a working cluster, use the upgrade script created by the customize_fusion_values.sh script. The upgrade script hard-codes the parameters and eases the need to remember which parameters to pass to the script. This is helpful when working with multiple K8s clusters. Make sure you check the script into version control alongside your custom values YAML files. Whenever you change the custom values YAML files for your cluster, you need to run the upgrade script to apply the changes. The script calls helm upgrade with the correct parameters and --values options.
If you run helm upgrade without passing the custom values YAML files, the deployment will revert to using chart defaults, which you never want to do.
The script assumes your kubeconfig is pointing to the correct cluster and you’re using Heml v3. If not, the upgrade fails. Select the correct kubeconfig before running the script.

​
Learn more

This article includes instructions for upgrading Fusion from one version to another. In some cases, the instructions do not vary. Other upgrades require special instructions. Start by checking upgrade details for your target version before continuing to the General upgrade process.Remember, upgrade instructions may vary between deployment types too.Whenever you upgrade Fusion, you must also update your remote connectors, if you are running any. You can download the latest files at V2 Connectors Downloads.
Fusion values change between releases. Check the example values and update values as needed.

​
Upgrades from 5.8.x

​
to 5.9.y

Upgrading from 5.8.x to 5.9.y involves using the General upgrade process.

​
General upgrade process

Fusion natively supports deployments on supported Kubernetes platforms, including AKS, EKS, and GKE.Fusion includes an upgrade script for AKS, EKS, and GKE. This script is not generated for other Kubernetes deployments.Upgrades differ from platform to platform. See below for more information about upgrading on your platform of choice.Whenever you upgrade Fusion, you must also update your remote connectors. You can download the latest files at V2 Connectors Downloads.

​
Natively supported deployment upgrades

Deployment typePlatform
Azure Kubernetes Service (AKS)aks
Amazon Elastic Kubernetes Service (EKS)eks
Google Kubernetes Engine (GKE)gke
Fusion includes upgrade scripts for natively supported deployment types. To upgrade:
  1. Open the <platform>_<cluster>_<release>_upgrade_fusion.sh upgrade script file for editing.
  2. Update the CHART_VERSION to your target Fusion version, and save your changes.
  3. Run the <platform>_<cluster>_<release>_upgrade_fusion.sh script. The <release> value is the same as your namespace, unless you overrode the default value using the -r option.
After running the upgrade, use kubectl get pods to see the changes applied to your cluster. It may take several minutes to perform the upgrade, as new Docker images are pulled from DockerHub. To see the versions of running pods, do:
kubectl get po -o jsonpath='{..image}'  | tr -s '[[:space:]]' '\n' | sort | uniq
This article includes instructions for upgrading Fusion from one version to another. Some upgrades require special instructions. Start by checking upgrade details for your target version before continuing to the General upgrade process.
Fusion 5.9.x must be upgraded one maintenance release at a time. If you are on version 5.9.12 and want to reach 5.9.14, you are required to upgrade to 5.9.13 first, then upgrade to 5.9.14.
When you upgrade Fusion, you must also update your remote connectors, if you are running any. You can download the latest files at V2 Connectors Downloads.
Fusion values change between releases. Check the example values and update values as needed.

​
Upgrades from 5.9.x

​
to 5.9.y

Upgrading from 5.9.x to 5.9.y involves using the General upgrade process.
If you are using horizontal pod autoscaling, follow the steps in the Fusion 5.8.1 release notes. If you have already done this as part of a previous upgrade, you can skip this process.

​
to 5.9.12

When upgrading to Fusion 5.9.12, add the following to your values.yaml file to avoid a known issue that prevents the kuberay-operator pod from launching successfully:
kuberay-operator:
  crd:
    create: true

​
to 5.9.5

Upgrading to Fusion 5.9.5 has special considerations, due to changes introduced with Solr 9.6.1 and Lucene 9.10.0. All upgrades are susceptible to issues from these changes. Follow the upgrade procedures closely to avoid issues with the upgrade.See the following sections for an overview of the issues, or skip to the upgrade process.

​
Solr 9.6.1 changes

Prior to Fusion 5.9.5, Fusion utilized Solr 9.1.1 or earlier. Due to changes in Solr 9.3, some Solr configuration and collection configurations are no longer compatible. As Fusion 5.9.5 leverages Solr 9.6.1, it’s imperative to address these compatibility issues during the upgrade process.To address Solr and collection configuration issues, a Docker utility called fm-upgrade-apps-to-solr-9, also known as the Fusion migration script, is included in the Fusion 5.9.5 release. This utility performs the following tasks:
  • Removes the unused configuration, <circuitBreaker>, from solrconfig.xml. Solr no longer supports this configuration.
  • Removes the query response writer of class solr.XSLTResponseWriter.
  • Comments out processors of type solr.StatelessScriptUpdateProcessorFactory.
  • Removes <bool name="preferLocalShards"/> element from request handler.
  • Changes cache class attribute of elements "filterCache", "cache", "documentCache", "queryResultCache" to solr.search.CaffeineCache.
  • Removes keepShortTerm attribute from filter of class solr.NGramFilterFactory.
  • Updates collection configurations, as needed.

​
Lucene 9.10.0 changes

A Lucene update to 9.10.0 in Fusion 5.9.5 may cause issues with certain collections in Solr. The change to the FST posting format codec (from Lucene90PostingsWriterDoc to Lucene99PostingsWriterDoc) in Lucene is incompatible with Solr in Fusion. As a result, Solr will not open a new searcher for collections using the FST50 postings format.To identify collections potentially affected by the Lucene codec change, examine the field definitions within your Solr schema. Look for fields that specify the postingsFormat attribute with a value of FST50. Collections containing such fields may experience compatibility issues. For example:
<fieldType name="tagger" class="solr.TextField" omitNorms="true" omitTermFreqAndPositions="true" postingsFormat="FST50">
The following log excerpt demonstrates a typical error message encountered when an upgrade is impacted by the codec change:
Caused by: org.apache.lucene.index.CorruptIndexException: codec mismatch: actual codec=Lucene90PostingsWriterDoc vs expected codec=Lucene99PostingsWriterDoc (resource=ByteBufferIndexInput(path="/var/solr/data/acme_query_rewrite_staging_shard1_replica_t9/data/index/_cn_FST50_0.doc"))
	at org.apache.lucene.codecs.CodecUtil.checkHeaderNoMagic(CodecUtil.java:205) ~[?:?]
	at org.apache.lucene.codecs.CodecUtil.checkHeader(CodecUtil.java:194) ~[?:?]
	at org.apache.lucene.codecs.CodecUtil.checkIndexHeader(CodecUtil.java:254) ~[?:?]
	at org.apache.lucene.codecs.lucene99.Lucene99PostingsReader.<init>(Lucene99PostingsReader.java:80) ~[?:?]
	at org.apache.lucene.codecs.memory.FSTPostingsFormat.fieldsProducer(FSTPostingsFormat.java:60) ~[?:?]
	at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.<init>(PerFieldPostingsFormat.java:330) ~[?:?]
	at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat.fieldsProducer(PerFieldPostingsFormat.java:392) ~[?:?]
	at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:119) ~[?:?]
	at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:96) ~[?:?]
	at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:178) ~[?:?]
	at org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:220) ~[?:?]
	at org.apache.lucene.index.IndexWriter.lambda$getReader$0(IndexWriter.java:542) ~[?:?]
	at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:138) ~[?:?]
	at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:604) ~[?:?]
	at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:112) ~[?:?]
	at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:91) ~[?:?]
	at org.apache.solr.core.StandardIndexReaderFactory.newReader(StandardIndexReaderFactory.java:38) ~[?:?]
	at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:2399) ~[?:?]
To account for the postingsFormat="FST50" codec issue, a Docker utility called run-the-fm-upgrade-query-rewrite-docker-utility is provided alongside the Fusion 5.9.5 release. You can pull this image from Docker using docker pull lucidworks/run-the-fm-upgrade-query-rewrite-docker-utility:2.x.This utility performs two actions: prepare and restore. Use the prepare action before the Fusion 5.9.5 upgrade begins. At a high level, the prepare action performs the following actions:
  1. Removes postingsFormat="FST50" from all collections in the environment.
  2. Re-indexes documents to new, temporary collections.
  3. Compares the original collections to the new, temporary collections to ensure data integrity.
Use the restore action after the Fusion 5.9.5 upgrade finishes, which must include the Solr 9.6.1 upgrade. The restore action performs the following actions:
  1. Restores postingsFormat="FST50" to all collections in the environment that were changed with the prepare action.
  2. Re-indexes documents to new, permanent collections. These collections match the original collections that were in place prior to the prepare action.
  3. Compares the restored collections to the temporary collections to ensure data integrity.

​
Upgrade process

This section provides a high-level overview of the steps involved in upgrading to Fusion 5.9.5. Follow each step in the order given:
  1. Create a full backup of all Fusion collections. These backups are intended as an emergency failsafe only.
  2. Run the fm-upgrade-apps-to-solr-9 Docker utility. This updates the Solr configuration and collections for compatibility with Solr 9.6.1.
  3. Run the run-the-fm-upgrade-query-rewrite-docker-utility Docker utility. Use the prepare action to address potential collection compatibility issues with Lucene 9.10.0 codecs.
  4. Upgrade your Fusion environment to version 5.9.5. Use the upgrade scripts or your own upgrade process.
  5. Re-run the run-the-fm-upgrade-query-rewrite-docker-utility Docker utility. Use the restore action to restore collections to their original state.
  6. Validate the upgrade was successful. In addition to your usual validations, there are some extra things to check.
To mitigate potential upgrade issues, adhere to the following procedures.

​
Back up your Solr collections

Back up all Solr collections in each environment before continuing with the upgrade. For this upgrade, backups are intended as an emergency failsafe.
Performing a rollback after encountering the issue described is a difficult, time-consuming process and is not recommended.

​
Run the fm-upgrade-apps-to-solr-9 Docker utility

Use the fm-upgrade-apps-to-solr-9 Docker utility to mitigate issues related to the change from Solr 9.1.1 and earlier to Solr 9.6.1. To begin, run the fm-upgrade-apps-to-solr-9 Docker utility using the DRY_RUN environmental variable:
docker run --rm -v $(pwd):/upgrade-work -e ZK_HOST=zk:2181 -e DRY_RUN=1 lucidworks/fm-upgrade-apps-to-solr-9:1.2.0
The DRY_RUN variable prints the changes that would occur to the console without performing those actions. Review the changes thoroughly.If the changes look correct, run the fm-upgrade-apps-to-solr-9 Docker utility again without using the DRY_RUN environmental variable. The updated config files are saved to the /upgrade-work/updated-configs directory. The utility also creates backups for all configs in the /upgrade-work/backup-configs.The fm-upgrade-apps-to-solr-9 Docker utility has another environmental variable, REVERT, that allows you to revert any changes you made. To revert your changes, run:
docker run --rm -v $(pwd):/upgrade-work -e ZK_HOST=zk:2181 -e REVERT=1 lucidworks/fm-upgrade-apps-to-solr-9:1.2.0

​
Run the fm-upgrade-query-rewrite Docker utility

Next, mitigate codec issues related to the Lucene 9.10.0 update. Run the run-the-fm-upgrade-query-rewrite-docker-utility Docker utility prepare action:
kubectl run \
--image="lucidworks/fm-upgrade-query-rewrite:2.x" \
--restart=Never \
--env="HELM_RELEASE=FUSION_NAMESPACE" \
--env="TARGET_SOLR_VERSION=9.6.1" \
--env="ACTION=prepare" prepare-595-upgrade
--namespace=FUSION_NAMESPACE
Change FUSION_NAMESPACE to the name of your application namespace for the Fusion installation. You can find this value using helm list against your Fusion namespace. Locate the release using the fusion chart, and find the value in the name column. Typically, the release name is the same as your namespace name.Including --namespace=FUSION_NAMESPACE lets the update pod runs in the correct application namespace.The prepare action removes postingsFormat="FST50" from all collections in the environment before re-indexing data to temporary collections. When the prepare-595-upgrade pod shows the status Completed, the process is finished.

​
Upgrade your Fusion environment

Upgrade Fusion to version 5.9.5. Before beginning, ensure the Fusion admin is running and all collections are healthy. Then, complete the General upgrade process before returning to the next step in the process.Alternatively, your organization may use a custom upgrade process. In either case, ensure you have successfully upgraded to Solr 9.6.1 as part of the Fusion upgrade.
Do not make changes to the signals collection with the Rules Editor during the upgrade process. For production clusters, upgrade during a maintenance window.

​
Re-run the fm-upgrade-query-rewrite Docker utility

Use the run-the-fm-upgrade-query-rewrite-docker-utility utility’s restore action to restore the data from the temporary collections created by the prepare action. Before you begin, verify all collections appended with _temp_fix are online and healthy.
kubectl run \
--image="lucidworks/fm-upgrade-query-rewrite:2.x" \
--restart=Never \
--env="HELM_RELEASE=FUSION_NAMESPACE" \
--env="TARGET_SOLR_VERSION=9.6.1" \
--env="ACTION=restore" restore-595-upgrade
--namespace=FUSION_NAMESPACE
Change FUSION_NAMESPACE to the name of your application namespace for the Fusion installation. You can find this value using helm list against your Fusion namespace. Locate the release using the fusion chart, and find the value in the name column. Typically, the release name is the same as your namespace name.Including --namespace=FUSION_NAMESPACE lets the update pod runs in the correct application namespace.When the restore-595-upgrade pod shows the status Completed, the process is finished.For a complete summary of what this action does, refer to Upgrade Utility.

​
Validate the upgrade

In addition to your typical validation process, ensure Solr collections are healthy:
  1. Log into Fusion as the admin.
  2. Access the Solr Admin UI at https://FUSION_HOST:FUSION_PORT/api/solrAdmin/default/#/.
  3. Watch for error messages. For example, the following message reports errors for the query rewrite staging collections Acme1, Acme2, Acme3, and Acme4: Codec error
  4. Navigate to the Cloud graph screen at https://FUSION_HOST:FUSION_PORT/api/solrAdmin/default/#/~cloud?view=graph.
  5. Review each collection for errors.
After you have completed validations, delete the temporary prepare and restore pods that were created by the upgrade utility:
kubectl delete po prepare-595-upgrade
kubectl delete po restore-595-upgrade

​
Resolving post-upgrade issues

If you followed the previous instructions, and you are still experiencing issues with the codec, you need to re-feed the affected data. Contact Lucidworks Support for further support.