Product Selector

Fusion 5.9
    Fusion 5.9

    Fusion 5 Upgrade from 5.9.x

    This article includes instructions for upgrading Fusion from one version to another. In some cases, the instructions do not vary. Other upgrades require special instructions. Start by checking upgrade details for your target version before continuing to the General upgrade process.

    Remember, upgrade instructions may vary between deployment types too. Review Upgrades by deployment type for details.

    Whenever you upgrade Fusion, you must also update your remote connectors, if you are running any. Configure Remote V2 Connectors provides complete instructions for remote connector setup. You can download the latest files at V2 Connectors Downloads.

    Fusion values change between releases. Check the example values and update values as needed.

    Upgrades from 5.9.x

    to 5.9.y

    Upgrading from 5.9.x to 5.9.y involves using the General upgrade process.

    If you are using horizontal pod autoscaling, follow the steps in the Fusion 5.8.1 release notes. If you have already done this as part of a previous upgrade, you can skip this process.

    to 5.9.5

    Upgrading to Fusion 5.9.5 has special considerations, due to changes introduced with Solr 9.6.1 and Lucene 9.10.0. All upgrades are susceptible to issues from these changes. Follow the upgrade procedures closely to avoid issues with the upgrade.

    See the following sections for an overview of the issues, or skip to the upgrade process.

    Solr 9.6.1 changes

    Prior to Fusion 5.9.5, Fusion utilized Solr 9.1.1 or earlier. Due to changes in Solr 9.3, some Solr configuration and collection configurations are no longer compatible. As Fusion 5.9.5 leverages Solr 9.6.1, it’s imperative to address these compatibility issues during the upgrade process.

    To address Solr and collection configuration issues, a Docker utility called fm-upgrade-apps-to-solr-9, also known as the Fusion migration script, is included in the Fusion 5.9.5 release. This utility performs the following tasks:

    • Removes the unused configuration, <circuitBreaker>, from solrconfig.xml. Solr no longer supports this configuration.

    • Removes the query response writer of class solr.XSLTResponseWriter.

    • Comments out processors of type solr.StatelessScriptUpdateProcessorFactory.

    • Removes <bool name="preferLocalShards"/> element from request handler.

    • Changes cache class attribute of elements "filterCache", "cache", "documentCache", "queryResultCache" to solr.search.CaffeineCache.

    • Removes keepShortTerm attribute from filter of class solr.NGramFilterFactory.

    • Updates collection configurations, as needed.

    Lucene 9.10.0 changes

    A Lucene update to 9.10.0 in Fusion 5.9.5 may cause issues with certain collections in Solr. The change to the FST posting format codec (from Lucene90PostingsWriterDoc to Lucene99PostingsWriterDoc) in Lucene is incompatible with Solr in Fusion. As a result, Solr will not open a new searcher for collections using the FST50 postings format.

    To identify collections potentially affected by the Lucene codec change, examine the field definitions within your Solr schema. Look for fields that specify the postingsFormat attribute with a value of FST50. Collections containing such fields may experience compatibility issues. For example:

    <fieldType name="tagger" class="solr.TextField" omitNorms="true" omitTermFreqAndPositions="true" postingsFormat="FST50">

    The following log excerpt demonstrates a typical error message encountered when an upgrade is impacted by the codec change:

    Caused by: org.apache.lucene.index.CorruptIndexException: codec mismatch: actual codec=Lucene90PostingsWriterDoc vs expected codec=Lucene99PostingsWriterDoc (resource=ByteBufferIndexInput(path="/var/solr/data/acme_query_rewrite_staging_shard1_replica_t9/data/index/_cn_FST50_0.doc"))
	at org.apache.lucene.codecs.CodecUtil.checkHeaderNoMagic(CodecUtil.java:205) ~[?:?]
	at org.apache.lucene.codecs.CodecUtil.checkHeader(CodecUtil.java:194) ~[?:?]
	at org.apache.lucene.codecs.CodecUtil.checkIndexHeader(CodecUtil.java:254) ~[?:?]
	at org.apache.lucene.codecs.lucene99.Lucene99PostingsReader.<init>(Lucene99PostingsReader.java:80) ~[?:?]
	at org.apache.lucene.codecs.memory.FSTPostingsFormat.fieldsProducer(FSTPostingsFormat.java:60) ~[?:?]
	at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.<init>(PerFieldPostingsFormat.java:330) ~[?:?]
	at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat.fieldsProducer(PerFieldPostingsFormat.java:392) ~[?:?]
	at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:119) ~[?:?]
	at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:96) ~[?:?]
	at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:178) ~[?:?]
	at org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:220) ~[?:?]
	at org.apache.lucene.index.IndexWriter.lambda$getReader$0(IndexWriter.java:542) ~[?:?]
	at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:138) ~[?:?]
	at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:604) ~[?:?]
	at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:112) ~[?:?]
	at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:91) ~[?:?]
	at org.apache.solr.core.StandardIndexReaderFactory.newReader(StandardIndexReaderFactory.java:38) ~[?:?]
	at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:2399) ~[?:?]

    To account for the postingsFormat="FST50" codec issue, a Docker utility called fm-upgrade-query-rewrite is provided alongside the Fusion 5.9.5 release. You can pull this image from Docker using docker pull lucidworks/fm-upgrade-query-rewrite:2.x.

    This utility performs two actions: prepare and restore. Use the prepare action before the Fusion 5.9.5 upgrade begins. At a high level, the prepare action performs the following actions:

    1. Removes postingsFormat="FST50" from all collections in the environment.

    2. Re-indexes documents to new, temporary collections.

    3. Compares the original collections to the new, temporary collections to ensure data integrity.

    Use the restore action after the Fusion 5.9.5 upgrade finishes, which must include the Solr 9.6.1 upgrade. The restore action performs the following actions:

    1. Restores postingsFormat="FST50" to all collections in the environment that were changed with the prepare action.

    2. Re-indexes documents to new, permanent collections. These collections match the original collections that were in place prior to the prepare action.

    3. Compares the restored collections to the temporary collections to ensure data integrity.

    Upgrade process

    This section provides a high-level overview of the steps involved in upgrading to Fusion 5.9.5. Follow each step in the order given:

    1. Create a full backup of all Fusion collections. These backups are intended as an emergency failsafe only.

    2. Run the fm-upgrade-apps-to-solr-9 Docker utility. This updates the Solr configuration and collections for compatibility with Solr 9.6.1.

    3. Run the fm-upgrade-query-rewrite Docker utility. Use the prepare action to address potential collection compatibility issues with Lucene 9.10.0 codecs.

    4. Upgrade your Fusion environment to version 5.9.5. Use the upgrade scripts or your own upgrade process.

    5. Re-run the fm-upgrade-query-rewrite Docker utility. Use the restore action to restore collections to their original state.

    6. Validate the upgrade was successful. In addition to your usual validations, there are some extra things to check.

    To mitigate potential upgrade issues, adhere to the following procedures.

    Back up your Solr collections

    Back up all Solr collections in each environment before continuing with the upgrade. For this upgrade, backups are intended as an emergency failsafe.

    Performing a rollback after encountering the issue described is a difficult, time-consuming process and is not recommended.

    Run the fm-upgrade-apps-to-solr-9 Docker utility

    Use the fm-upgrade-apps-to-solr-9 Docker utility to mitigate issues related to the change from Solr 9.1.1 and earlier to Solr 9.6.1. To begin, run the fm-upgrade-apps-to-solr-9 Docker utility using the DRY_RUN environmental variable:

    docker run --rm -v $(pwd):/upgrade-work -e ZK_HOST=zk:2181 -e DRY_RUN=1 lucidworks/fm-upgrade-apps-to-solr-9:1.2.0

    The DRY_RUN variable prints the changes that would occur to the console without performing those actions. Review the changes thoroughly.

    If the changes look correct, run the fm-upgrade-apps-to-solr-9 Docker utility again without using the DRY_RUN environmental variable. The updated config files are saved to the /upgrade-work/updated-configs directory. The utility also creates backups for all configs in the /upgrade-work/backup-configs.

    The fm-upgrade-apps-to-solr-9 Docker utility has another environmental variable, REVERT, that allows you to revert any changes you made. To revert your changes, run:

    docker run --rm -v $(pwd):/upgrade-work -e ZK_HOST=zk:2181 -e REVERT=1 lucidworks/fm-upgrade-apps-to-solr-9:1.2.0

    Run the fm-upgrade-query-rewrite Docker utility

    Next, mitigate codec issues related to the Lucene 9.10.0 update. Run the fm-upgrade-query-rewrite Docker utility prepare action:

    kubectl run --generator=run-pod/v1 \
  --image="lucidworks/fm-upgrade-query-rewrite:2.x" \
  --restart=Never \
  --env="HELM_RELEASE=RELEASE_NAME" \
  --env="TARGET_SOLR_VERSION=9.6.1" \
  --env="ACTION=prepare" prepare-595-upgrade

    Change RELEASE_NAME to the release name of your Fusion installation, not the version. You can find this value using helm list against your Fusion namespace. Locate the release using the fusion chart, and find the value in the name column. Typically, the release name is the same as your namespace name.

    The prepare action removes postingsFormat="FST50" from all collections in the environment before re-indexing data to temporary collections. When the prepare-595-upgrade pod shows the status Completed, the process is finished.

    For a complete summary of what this action does, refer to Upgrade Utility.

    Upgrade your Fusion environment

    Upgrade Fusion to version 5.9.5. Before beginning, ensure the Fusion admin is running and all collections are healthy. Then, complete the General upgrade process before returning to the next step in the process.

    Alternatively, your organization may use a custom upgrade process. In either case, ensure you have successfully upgraded to Solr 9.6.1 as part of the Fusion upgrade.

    Do not make changes to the signals collection with the Rules Editor during the upgrade process. For production clusters, upgrade during a maintenance window.

    Re-run the fm-upgrade-query-rewrite Docker utility

    Use the fm-upgrade-query-rewrite utility’s restore action to restore the data from the temporary collections created by the prepare action. Before you begin, verify all collections appended with _temp_fix are online and healthy.

    Then, run the fm-upgrade-query-rewrite Docker utility restore action:

    kubectl run --generator=run-pod/v1 \
  --image="lucidworks/fm-upgrade-query-rewrite:2.x" \
  --restart=Never \
  --env="HELM_RELEASE=RELEASE_NAME" \
  --env="TARGET_SOLR_VERSION=9.6.1" \
  --env="ACTION=restore" restore-595-upgrade

    Change RELEASE_NAME value to the release name of your Fusion installation.

    When the restore-595-upgrade pod shows the status Completed, the process is finished.

    For a complete summary of what this action does, refer to Upgrade Utility.

    Validate the upgrade

    In addition to your typical validation process, ensure Solr collections are healthy:

    1. Log into Fusion as the admin.

    2. Access the Solr Admin UI at https://FUSION_HOST:FUSION_PORT/api/solrAdmin/default/#/.

    3. Watch for error messages. For example, the following message reports errors for the query rewrite staging collections Acme1, Acme2, Acme3, and Acme4:

      Codec error

    4. Navigate to the Cloud graph screen at https://FUSION_HOST:FUSION_PORT/api/solrAdmin/default/#/~cloud?view=graph.

    5. Review each collection for errors.

    After you have completed validations, delete the temporary prepare and restore pods that were created by the upgrade utility:

    kubectl delete po prepare-595-upgrade
kubectl delete po restore-595-upgrade

    Resolving post-upgrade issues

    If you followed the previous instructions, and you are still experiencing issues with the codec, you need to re-feed the affected data. Contact Lucidworks Support for further support.

    to 5.10.y

    Upgrading from 5.9.x to 5.10.y involves using the General upgrade process.

    General upgrade process

    Fusion natively supports deployments on various Kubernetes platforms, including AKS, EKS, and GKE. You can also deploy on a different Kubernetes platform of your choice.

    Fusion includes an upgrade script for AKS, EKS, and GKE. This script is not generated for other Kubernetes deployments. In this case, see Other Kubernetes deployment upgrades.

    Upgrades differ from platform to platform. See below for more information about upgrading on your platform of choice.

    Whenever you upgrade Fusion, you must also update your remote connectors, if you are running any. Configure Remote V2 Connectors provides complete instructions for remote connector setup. You can download the latest files at V2 Connectors Downloads.

    Natively supported deployment upgrades

    Deployment type Platform

    Azure Kubernetes Service (AKS)

    aks

    Amazon Elastic Kubernetes Service (EKS)

    eks

    Google Kubernetes Engine (GKE)

    gke

    Fusion includes upgrade scripts for natively supported deployment types. To upgrade:

    1. Open the <platform>_<cluster>_<release>_upgrade_fusion.sh upgrade script file for editing.

    2. Update the CHART_VERSION to your target Fusion version, and save your changes.

    3. Run the <platform>_<cluster>_<release>_upgrade_fusion.sh script. The <release> value is the same as your namespace, unless you overrode the default value using the -r option.

    After running the upgrade, use kubectl get pods to see the changes applied to your cluster. It may take several minutes to perform the upgrade, as new Docker images are pulled from DockerHub. To see the versions of running pods, do:

    kubectl get po -o jsonpath='{..image}'  | tr -s '[[:space:]]' '\n' | sort | uniq