Fusion 5.9.11 - Lucidworks documentation

Released on March 20, 2025, this maintenance release provides KNN-based performance enhancements for Neural Hybrid Search, Java 17 support for connectors, a new asynchronous parsing service, the Apps Manager API, and critical security updates. Upgrading to the latest version of Fusion 5.9 offers several key benefits:

Access to latest features: Stay current with the latest features and functionality to ensure compatibility and optimal performance.
Simplified process: Fusion 5.9.5 and later use an in-place upgrade strategy, making upgrades easier than ever.
Extended support: Upgrading keeps you up-to-date with the latest supported Kubernetes versions, as outlined in the Lucidworks Fusion Product Lifecycle policy.

For supported Kubernetes versions and key component versions, see Platform support and component versions.

Key highlights

KNN performance enhancements

The Neural Hybrid Query stage now uses K-Nearest Neighbors (KNN) instead of vector similarity (vecSim). KNN is a more efficient method for finding the most relevant results, leading to faster and more accurate searches that help users find the most relevant information faster. New configuration options give you finer control over the speed and accuracy of neural hybrid queries:

vectorDepth (Number of Vector Results) sets the number of vector results to return from the vector portion of the hybrid query. Increasing vectorDepth retrieves more vector results but may increase query time. Lowering it speeds up search but may reduce result diversity.
vecPreFilterBoolean (Block pre-filtering) indicates whether to prevent pre-filtering. Pre-filtering can improve performance, while blocking it can yield more accurate facet counts and search results.

Java 17 for connectors

Connectors have been updated to use Java 17. If you use remote connectors, you must upgrade to JVM 17. See Configure Remote V2 Connectors for more information.

Configure Remote V2 Connectors

If you need to index data from behind a firewall, you can configure a V2 connector to run remotely on-premises using TLS-enabled gRPC.

Prerequisites

Before you can set up an on-prem V2 connector, you must configure the egress from your network to allow HTTP/2 communication into the Fusion cloud. You can use a forward proxy server to act as an intermediary between the connector and Fusion.The following is required to run V2 connectors remotely:

The plugin zip file and the connector-plugin-standalone JAR.
A configured connector backend gRPC endpoint.
Username and password of a user with a remote-connectors or admin role.
If the host where the remote connector is running is not configured to trust the server’s TLS certificate, you must configure the file path of the trust certificate collection.

If your version of Fusion doesn’t have the remote-connectors role by default, you can create one. No API or UI permissions are required for the role.

Connector compatibility

Only V2 connectors are able to run remotely on-premises. You also need the remote connector client JAR file that matches your Fusion version. You can download the latest files at V2 Connectors Downloads.

Whenever you upgrade Fusion, you must also update your remote connectors to match the new version of Fusion.

The gRPC connector backend is not supported in Fusion environments deployed on AWS.

System requirements

The following is required for the on-prem host of the remote connector:

(Fusion 5.9.0-5.9.10) JVM version 11
(Fusion 5.9.11) JVM version 17
Minimum of 2 CPUs
4GB Memory

Note that memory requirements depend on the number and size of ingested documents.

Enable backend ingress

In your values.yaml file, configure this section as needed:

ingress:
  enabled: false
  pathtype: "Prefix"
  path: "/"
  #host: "ingress.example.com"
  ingressClassName: "nginx"   # Fusion 5.9.6 only
  tls:
    enabled: false
    certificateArn: ""
    # Enable the annotations field to override the default annotations
    #annotations: ""

Set enabled to true to enable the backend ingress.
Set pathtype to Prefix or Exact.
Set path to the path where the backend will be available.
Set host to the host where the backend will be available.
In Fusion 5.9.6 only, you can set ingressClassName to one of the following:
- nginx for Nginx Ingress Controller
- alb for AWS Application Load Balancer (ALB)
Configure TLS and certificates according to your CA’s procedures and policies.
TLS must be enabled in order to use AWS ALB for ingress.

Connector configuration example

kafka-bridge:
  target: mynamespace-connectors-backend.lucidworkstest.com:443 # mandatory
  plain-text: false # optional, false by default.
  proxy-server: # optional - needed when a forward proxy server is used to provide outbound access to the standalone connector
    host: host
    port: some-port
    user: user # optional
    password: password # optional
  trust: # optional - needed when the client's system doesn't trust the server's certificate
    cert-collection-filepath: path1

proxy: # mandatory fusion-proxy
  user: admin
  password: password123
  url: https://fusiontest.com/ # needed only when the connector plugin requires blob store access

plugin: # mandatory
  path: ./fs.zip
  type: #optional - the suffix is added to the connector id
    suffix: remote

Minimal example

kafka-bridge:
  target: mynamespace-connectors-backend.lucidworkstest.com:443

proxy:
  user: admin
  password: "password123"

plugin:
  path: ./testplugin.zip

Logback XML configuration file example

<configuration>
    <appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
        <encoder class="com.lucidworks.logging.logback.classic.LucidworksPatternLayoutEncoder">
            <pattern>%d - %-5p [%t:%C{3.}@%L] - %m{nolookups}%n</pattern>
            <charset>utf8</charset>
        </encoder>
    </appender>

    <appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <file>${LOGDIR:-.}/connector.log</file>
        <rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
            <!-- rollover daily -->
            <fileNamePattern>${LOGDIR:-.}/connector-%d{yyyy-MM-dd}.%i.log.gz</fileNamePattern>
            <maxFileSize>50MB</maxFileSize>
            <totalSizeCap>10GB</totalSizeCap>
        </rollingPolicy>
        <encoder class="com.lucidworks.logging.logback.classic.LucidworksPatternLayoutEncoder">
            <pattern>%d - %-5p [%t:%C{3.}@%L] - %m{nolookups}%n</pattern>
            <charset>utf8</charset>
        </encoder>
    </appender>

    <root level="INFO">
        <appender-ref ref="CONSOLE"/>
        <appender-ref ref="FILE"/>
    </root>
</configuration>

Run the remote connector

java [-Dlogging.config=[LOGBACK_XML_FILE]] \
  -jar connector-plugin-client-standalone.jar [YAML_CONFIG_FILE]

The logging.config property is optional. If not set, logging messages are sent to the console.

Test communication

You can run the connector in communication testing mode. This mode tests the communication with the backend without running the plugin, reports the result, and exits.

java -Dstandalone.connector.connectivity.test=true -jar connector-plugin-client-standalone.jar [YAML_CONFIG_FILE]

Encryption

In a deployment, communication to the connector’s backend server is encrypted using TLS. You should only run this configuration without TLS in a testing scenario. To disable TLS, set plain-text to true.

Egress and proxy server configuration

One of the methods you can use to allow outbound communication from behind a firewall is a proxy server. You can configure a proxy server to allow certain communication traffic while blocking unauthorized communication. If you use a proxy server at the site where the connector is running, you must configure the following properties:

Host. The hosts where the proxy server is running.
Port. The port the proxy server is listening to for communication requests.
Credentials. Optional proxy server user and password.

When you configure egress, it is important to disable any connection or activity timeouts because the connector uses long running gRPC calls.

Password encryption

If you use a login name and password in your configuration, run the following utility to encrypt the password:

Enter a user name and password in the connector configuration YAML.

Run the standalone JAR with this property:

-Dstandalone.connector.encrypt.password=true

Retrieve the encrypted passwords from the log that is created.
Replace the clear password in the configuration YAML with the encrypted password.

Connector restart (5.7 and earlier)

The connector will shut down automatically whenever the connection to the server is disrupted, to prevent it from getting into a bad state. Communication disruption can happen, for example, when the server running in the connectors-backend pod shuts down and is replaced by a new pod. Once the connector shuts down, connector configuration and job execution are disabled. To prevent that from happening, you should restart the connector as soon as possible.You can use Linux scripts and utilities to restart the connector automatically, such as Monit.

Recoverable bridge (5.8 and later)

If communication to the remote connector is disrupted, the connector will try to recover communication and gRPC calls. By default, six attempts will be made to recover each gRPC call. The number of attempts can be configured with the max-grpc-retries bridge parameters.

Job expiration duration (5.9.5 only)

The timeout value for irresponsive backend jobs can be configured with the job-expiration-duration-seconds parameter. The default value is 120 seconds.

Use the remote connector

Once the connector is running, it is available in the Datasources dropdown. If the standalone connector terminates, it disappears from the list of available connectors. Once it is re-run, it is available again and configured connector instances will not get lost.

Enable asynchronous parsing (5.9 and later)

To separate document crawling from document parsing, enable Tika Asynchronous Parsing on remote V2 connectors.

Asynchronous parsing service

An asynchronous parsing service for connectors has been added. While traditional synchronous parsing can create delays in indexing when handling large documents or high data volumes, asynchronous parsing processes files in the background, allowing indexing to continue without waiting for each document to be fully parsed. This new service brings more efficient data processing, improved search freshness, scalability without added complexity, and better support for diverse data types, supporting HTML, JSON, and other formats. A new parsing stage, Apache Tika Container, has been added to route asynchronous parsing through the new service. This stage is now required when using asynchronous parsing for connectors. To use asynchronous parsing for connectors, be sure that Async Parsing is checked in the datasource and the Apache Tika Container parser stage is enabled in the index pipeline. See Use Tika Asynchronous Parsing for detailed steps to set up asynchronous parsing.

Use Tika Asynchronous Parsing

Other parsers, such as HTML and JSON, are now supported by the asynchronous parsing service. By enabling asynchronous parsing, the parser configuration linked to your datasource is used.

To learn how to start using the new asynchronous parsing service, see the following demonstration:

This update also includes a new Async Parsing API.

Lucidworks offers free training to help you get started. The Course for Asynchronous Parsing Service focuses on how to use the asynchronous parsing service for connectors to improve efficiency and responsiveness:

Visit the LucidAcademy to see the full training catalog.

Job Config service

The new Job Config service provides more accurate and reliable job status reporting. This service uses asynchronous communication through Kafka for improved efficiency over the previous synchronous calls, with these benefits:

More accurate job tracking. You get real-time, reliable job status updates, reducing uncertainty about whether a job is running, completed, or failed.
Faster troubleshooting. With detailed job histories and improved reporting, teams can quickly diagnose issues instead of chasing down incomplete or outdated job statuses.
Seamless transition. For most users, no action is required—API calls through api-gateway automatically reroute. Internal API users need only a simple update to continue tracking jobs accurately.

The following API endpoints have been migrated from the admin service to the new job-config service:

/api/tasks/{id}
/api/jobs/{resource}/schedule
/api/tasks
/api/jobs/{resource}/actions
/api/tasks/_schema
/api/jobs
/api/jobs/{resource}
/api/jobs/{resource}/history
/api/jobs/_schema

For API calls made to the api-gateway service, you do not need to make any changes; the endpoints above are automatically rerouted to the new job-config service. See Job Config API for reference information about these endpoints.

If you are making internal API calls to the admin service using any of the endpoints above, you must update your API calls to point to the new job-config service.

Security updates

Lucidworks remains committed to providing a secure and resilient platform. Fusion 5.9.11 includes critical security updates across a number of Fusion services, including the admin, connectors, distributed compute, indexing, job configuration, and query services, ensuring continued protection and reliability for your deployments.

API path changes

Trailing slashes are no longer supported when making API calls. For example, this API call works:

http://HOSTNAME/api/apps/APP_NAME/query/QUERY_PROFILE?q=hello

But this API call does not work and results in an error:

http://HOSTNAME/api/apps/APP_NAME/query/QUERY_PROFILE/?q=hello

Bug fixes

Critical fix for Azure Kubernetes Service (AKS) and Amazon Elastic Kubernetes Service (EKS) compatibility on Kubernetes 1.30+
By upgrading the Kubernetes Client library to version 6.2.0, this update prevents token refresh failures that previously caused service disruptions. Affected services—including connectors backend, indexing, job rest server, and job launcher services—now operate reliably in OIDC-enabled AKS and EKS environments, strengthening Fusion’s stability on modern Kubernetes deployments.

Known issues

Saving large query pipelines may cause OOM failures under high load.
In Fusion 5.9.x versions through 5.9.13, saving a large query pipeline during high query volume can result in thread lock, excessive memory use, and eventual OOM errors in the Query service.

This issue is fixed in Fusion 5.9.14.
Scheduled jobs that perform Solr operations may fail to run or save due to permission-related issues in the job-config service.
In some cases, users without full permissions can create or modify scheduled tasks that execute actions they aren’t authorized to run directly, specifically tasks that invoked Solr using solr:// URIs.

These jobs may fail silently or prevent schedule changes from being saved. This issue can also affect the execution of jobs after upgrade. It is fixed in Fusion 5.9.13.
Deploying Fusion with TLS flags enabled fails in ArgoCD due to Helm chart rendering limitations.
When using ArgoCD to deploy Fusion 5.9.10 or 5.9.11 with TLS options enabled, Helm chart rendering fails due to the use of the lookup function, which is unsupported by ArgoCD. This prevents ArgoCD from generating manifests, blocking deployment workflows that rely on TLS configuration.

This issue is fixed in Fusion 5.9.12.

As a workaround, deploy Fusion without enabling TLS in ArgoCD-managed environments, or perform the deployment using Helm directly.
Incorrect Solr image repository in Helm charts.
Fusion 5.9.11’s Helm chart incorrectly references an internal Lucidworks Artifactory repository for the Solr image. This can prevent successful deployment in environments without access to internal infrastructure.

This issue is fixed in Fusion 5.9.12.

As a workaround, override the Solr image repository in your custom values.yaml file.
Jobs API returns incorrect timestamps for jobs scheduled after 12:00 UTC.
In Fusion 5.9.11, job timestamps returned by the API may be off by 12 hours if the time is after noon UTC. This affects scheduling accuracy and may cause the UI to misinterpret whether a change has been made.

This issue is fixed in Fusion 5.9.12.
Cannot update existing job triggers in the Schedulers view.
In Fusion 5.9.11, scheduled job triggers cannot be modified due to incorrect timestamp comparison logic in the Admin UI. You must delete and recreate the trigger instead.

This issue is fixed in Fusion 5.9.12.
Scheduler may stop working if ZooKeeper becomes briefly unavailable.
If all job-config pods lose connection to ZooKeeper in Fusion 5.9.11, they may fail to re-elect a leader, halting all scheduling until one of the pods is manually restarted.

This issue is fixed in Fusion 5.9.12.
Datasource jobs show incorrect “started by” user.
In Fusion 5.9.11, datasource jobs started in the UI incorrectly show default-subject as the initiating user in job history instead of the actual user.

This issue is fixed in Fusion 5.9.12.
Schema API incompatibility with nonstandard schema file names.
In Fusion 5.9.11, new apps save Solr schemas with the name managed-schema.xml instead of managed-schema, while older apps may include only a managed-schema file. This mismatch can cause errors when editing or previewing schemas using the Schema API, particularly when the expected file extension is missing. For existing config sets, it’s best to copy the contents from managed-schema to a new managed-schema.xml, then delete managed-schema.

Fusion 5.9.12 resolves this by supporting both file names, ensuring backward compatibility with older apps and consistent behavior for new ones.
Job scheduling may stop working if all job-config pods lose connection to ZooKeeper.
In rare cases, when a ZooKeeper connection is lost, the leader latch mechanism used by the job-config service might fail to recover. If no node can re-establish leadership, the job scheduler stops until at least one job-config pod is restarted.

This issue is fixed in Fusion 5.9.13.
Scheduled job triggers can’t be updated in the Schedulers view.
While the Admin UI displays the current trigger time correctly, changes to the time are not saved. This is due to a mismatch in how the API formats UTC timestamps, especially for times after 12:00 UTC, which prevents the UI from detecting that a change has been made.

To work around this, delete the existing scheduler entry and create a new one with the desired time.

This issue is resolved in Fusion 5.9.13.
Dependent job scheduling may fail
Jobs configured to run based on the success or failure of another job may not trigger as expected. This includes configurations using “on_success_or_failure” and “Start + Interval” options. This issue is resolved in Fusion 5.9.13.
Aborted jobs in Fusion are listed twice in the Job History.
This issue is fixed in Fusion 5.9.12.
When a new datasource is configured in Indexing > Index Workbench, the simulated results do not display, and the following error is generated: “Failed to simulate results from a working pipeline.” As a result, the Index Workbench cannot simulate results for new datasources, preventing users from configuring the indexing process within the workbench.
This issue is fixed in Fusion 5.9.12.

To work around this issue, configure each part of the indexing process separately instead of using the Index Workbench:
- Configure datasources in Indexing > Datasources.
- Configure parsers in Indexing > Parsers.
- Configure index pipelines in Indexing > Index Pipelines.
In Fusion 5.9.10, the Apache Spark 3.4.1 upgrade impacted jobs that use Python 3.7 behavior or compatibility, which may have automatically updated to Python 3.10.x and no longer function correctly. If you have not yet updated the code for Python 3.10.x, update your code to ensure compatibility with Python 3.10.x and then test your Spark jobs in a staging environment before deploying to production.
An issue prevents segment-based rule filtering from working correctly in Commerce Studio. This issue is fixed in Fusion 5.9.12.

Deprecations

For full details on deprecations, see Deprecations and Removals.

The Parsers Indexing CRUD API, which provides CRUD operations for parsers, allowing users to create, read, update, and delete parsers, is deprecated in Fusion 5.9.11. This feature was originally deprecated in Fusion 5.12.0. It will be removed in a future release no later than September 4, 2025.
The Async Parsing API replaces the Parsers Indexing CRUD API and is available in Fusion 5.9.11. This API provides improved functionality and aligns with Fusion’s updated architecture, ensuring consistency across versions.
The Word2Vec Model Training Job, which trains a shallow neural model to generate vector embeddings for text data, is deprecated in Fusion 5.9.11. It will be removed in a future release no later than September 4, 2025.
Lucidworks offers a wide array of AI solutions, including Lucidworks AI. Lucidworks AI provides easy-to-use, AI-powered data discovery and search capabilities including:
- Pre-trained embedding models
- Custom embedding model training
- Seamless integration with Fusion

Removals

Bitnami removal

Fusion 5.9.11 will be re-released with the same functionality but updated image references. In the meantime, Lucidworks will self-host the required images while we work to replace Bitnami images with internally built open-source alternatives. If you are a self-hosted Fusion customer, you must upgrade before August 28 to ensure continued access to container images and prevent deployment issues. You can reinstall your current version of Fusion or upgrade to Fusion 5.9.14, which includes the updated Helm chart and prepares your environment for long-term compatibility. See Prevent image pull failures due to Bitnami deprecation in Fusion 5.9.5 to 5.9.13 for more information on how to prevent image pull failures.

Platform support and component versions

Kubernetes platform support

Lucidworks has tested and validated support for the following Kubernetes platforms and versions:

Google Kubernetes Engine (GKE): 1.28, 1.29, 1.30, 1.31
Microsoft Azure Kubernetes Service (AKS): 1.28, 1.29, 1.30, 1.31
Amazon Elastic Kubernetes Service (EKS): 1.28, 1.29, 1.30, 1.31

Support is also offered for Rancher Kubernetes Engine (RKE and RKE2) and OpenShift 4 versions that are based on Kubernetes 1.28, 1.29, 1.30, 1.31; note that RKE2 may require some Helm chart modification. OpenStack and customized Kubernetes installations are not supported. For more information on Kubernetes version support, see the Kubernetes support policy.

Component versions

The following table details the versions of key components that may be critical to deployments and upgrades.

Component	Version
Solr	fusion-solr 5.9.11 (based on Solr 9.6.1)
ZooKeeper	3.9.1
Spark	3.4.1
Ingress Controllers	Nginx, Ambassador (Envoy), GKE Ingress Controller Istio not supported.

More information about support dates can be found at Lucidworks Fusion Product Lifecycle.

Introduction to Fusion

Getting Data In

Getting Data Out

Operations

Reference

Developer Docs

Neural Hybrid Search

Release Notes

​Key highlights

​KNN performance enhancements

​Java 17 for connectors

​Asynchronous parsing service

​Job Config service

​Security updates

​API path changes

​Bug fixes

​Known issues

​Deprecations

​Removals

​Bitnami removal

​Platform support and component versions

​Kubernetes platform support

​Component versions

Key highlights

KNN performance enhancements

Java 17 for connectors

Asynchronous parsing service

Job Config service

Security updates

API path changes

Bug fixes

Known issues

Deprecations

Removals

Bitnami removal

Platform support and component versions

Kubernetes platform support

Component versions