Fusion Connectors are continuously improving with new features, updated functionality, performance upgrades, and bug fixes. This article summarizes these changes.
This release removes the connector’s dependency on Microsoft’s Access Control Services (ACS) authentication by upgrading the authentication library from ADAL4J to MSAL4J. This upgrade is required for continued SharePoint Online operation after Microsoft’s ACS deprecation.
Microsoft deprecated Access Control Services (ACS) for SharePoint Online in April 2026.
Customers using ACS-based authentication must migrate to App-Only (certificate-based) authentication before their SharePoint Online connection stops working.
Existing OAuth credentials continue to work without changes.
The connector now uses Microsoft’s current MSAL4J library (v1.14.3) instead of the deprecated ADAL4J library (v1.6.7). This upgrade adds full support for certificate-based App-Only authentication for SharePoint Online while maintaining existing OAuth credential configurations without changes.New SharePoint Online deployments should use App-Only (certificate-based) authentication. See the SharePoint Optimized V2 connector documentation for setup instructions.
Crawls no longer fail with a StringIndexOutOfBoundsException when processing malformed SharePoint URLs, empty strings, or edge-case inputs. Bounds checking has been added to URL substring operations across SharePoint utility classes.
Resolved potential null pointer exception risks in ACL handling, corrected URL generation in CSOM calls, improved authentication header handling, and fixed login-name formatting issues. No configuration changes are required.
This release enables multiple JavaScript-evaluation datasources to run concurrently on shared Selenium Grid infrastructure and overhauls HTTP retry behavior with exponential backoff and more resilient defaults. The update also includes crawl-reliability fixes for cookies and SSL handling, along with a streamlined configuration UI.
Concurrent JavaScript evaluation datasources
Multiple Web V2 datasources with JavaScript evaluation enabled can now run simultaneously and share a Selenium Grid instance. Each datasource receives a fair share of grid capacity, recomputed dynamically as datasources register and deregister. Datasources that do not use JavaScript evaluation do not consume these resources.
Running multiple JavaScript-enabled datasources concurrently increases the load on Selenium Grid. If you plan to run more than one JavaScript-evaluation datasource at a time, scale your Selenium Grid instance to provide sufficient node capacity. See Scale Selenium Grid for concurrent JavaScript-evaluation datasources for guidance.
Updated retry and error tolerance default values
The default retry and fetching behaviors have been updated to provide more resilient out-of-the-box configuration for new datasources.
Changed from -1 (never) to 2. A document is removed from the index after 3 consecutive fetch errors instead of being retained indefinitely.
Existing datasources retain their currently configured values. Only newly created datasources use these updated defaults. Property descriptions in the configuration UI have been clarified to specify exactly what each value means.
Smarter HTTP retry handling
Retries now use exponential backoff with jitter and cover a broader set of recoverable HTTP status codes (429, 500, 502, 503, 504) in addition to transient network errors. This allows transient site errors to recover more gracefully without manual tuning. The Request Retry Count and Retry Max Delay fields control retry behavior.The Retry Max Delay field now stops retries when backoff exceeds the configured limit. Retries stop as soon as the next backoff wait would exceed the Retry Max Delay value, rather than continuing until Request Retry Count is exhausted. The limit that is reached first (count or delay) ends the retries. Previously, Retry Max Delay only capped the wait interval, but now it also acts as a stop condition.
Selenium Grid readiness check
A new validation check verifies that your Selenium Grid environment is reachable before the crawl starts. If Selenium Grid cannot be reached, the connector now reports a clear validation error, instead of failing during the crawl with an opaque error. Transient slot exhaustion is intentionally not treated as a validation failure and is handled at fetch time by the WebDriver pool’s 60-second create-timeout.
Before v2.2.3
v2.2.3 and later
Configuration UI improvements
Deprecated Kerberos and credential file properties are now hidden from the configuration UI, reducing clutter when configuring new datasources. Existing datasources that have values set for these properties continue to work.Tooltips for Request Retry Count, Retry Max Delay, Timeout, and Infinite Timeout have been rewritten to accurately describe the connector’s behavior, including cross-field interactions.
This release resolves two issues affecting cookie handling and SSL certificate validation.
The Web V2 connector now correctly copies cookies across requests when no authentication is configured. Previously, the Web connector dropped cookies set by the target site on follow-up requests when no authentication method was configured. Public sites that rely on cookies for consent banners, A/B routing, or language preferences now crawl correctly.
The Allow All Certificates setting now applies to both the HTTP client and the Selenium-based JavaScript evaluator. Sites with self-signed or internal-CA certificates can now be crawled end-to-end with JavaScript evaluation enabled. Previously, the HTTP client honored this setting but the Selenium-based JavaScript evaluator ignored it.
The Amazon AWS S3 Pro connector has been updated to version 2.0.0 and has earned the Pro designation, showing that this connector meets higher internal standards for stability, reliability, and production readiness.
Pro connectors are built on the same framework as V2 connectors but meet higher internal standards for stability, reliability, and production readiness. If you’re currently using other V2 connectors, the process for installing and upgrading a Pro connector remains the same.More V2 connectors will be upgraded to Pro connectors in the future.
The Amazon AWS S3 Pro connector brings simplified stray content deletion, improved timeout and shutdown handling, and S3-compatible endpoint support.
S3-compatible endpoint support
The connector now supports S3-compatible storage services through the new Endpoint Override configuration field. You can point the connector at VPC endpoints, LocalStack, MinIO, or S3-compatible services like Google Cloud Storage by setting the endpoint URL.
Endpoint override setting for the S3 Pro connector
Custom endpoint URL for S3-compatible services (for example, https://storage.googleapis.com for Google Cloud Storage, https://vpce-xxx.s3.us-east-1.vpce.amazonaws.com for VPC endpoints).
Improved timeout and shutdown handling
Stalled S3 downloads now fail quickly with clear error messages instead of hanging indefinitely. The connector includes configurable API call and socket timeouts for better control over long-running operations, and connector shutdown no longer produces spurious error log messages.
Maximum time in seconds to wait for data transfer over an open connection. Range: 1 to 600 seconds.
Enhanced change detection
Object change detection now uses ETags with timestamp fallback. In-place overwrites that preserve timestamps are now correctly detected and re-indexed. Zero-byte S3 folder placeholders whose timestamps refresh on writes elsewhere in the prefix are no longer re-indexed on every crawl.
Simplified stray content deletion
The Enable Stray Content Deletion toggle is now hidden in the UI and is enabled by default. New datasources automatically enable the stray content deletion and circuit breaker feature built into all V2 and Pro connectors starting in Fusion 5.18.0. Existing datasources retain their current settings.
S3 Pro connector stray content deletion and circuit breaker settings
The stray content deletion property remains accessible via the REST API for users who need explicit control for Fusion versions earlier than 5.18.0.
Retry logging
S3 client retry events now appear in connector logs with attempt count and request context, making it easier to identify transient AWS issues during troubleshooting.
Existing datasources continue to work without modification. The connector now displays as S3 Pro in the Fusion connector picker. The type ID (lucidworks.s3) remains unchanged, so existing datasources do not need to be created again.Datasources with out-of-range values for pageSize, retryCount, apiCallAttemptTimeoutSeconds, or socketTimeoutSeconds will fail validation the next time you save your datasource. Review and adjust any values exceeding the documented limits before updating your connector.
Timestamp columns from DB2, MS SQL Server, and Oracle databases are now indexed correctly. Previously, vendor-specific timestamp types were silently dropped with warning messages. The connector now normalizes these database-specific timestamp formats before indexing.This fix applies to:
IBM DB2 timestamp columns (com.ibm.db2.jcc.am.io)
Microsoft SQL Server datetime columns (SQLServerDateTime)
Oracle timestamp columns (oracle.sql.TIMESTAMP)
Simplified stray content deletion
The Enable Stray Content Deletion toggle is now hidden in the UI and is enabled by default. New datasources automatically enable the stray content deletion and circuit breaker feature built into all V2 and Pro connectors starting in Fusion 5.18.0. Existing datasources retain their current settings.
JDBC V2 connector stray content deletion and circuit breaker settings
If you are using Fusion 5.18.0 or later, stray content deletion behavior is determined by the Enable Stray Content Deletion settings built into Fusion’s core datasource settings, available on every V2 and Pro connector.If system-level stray content deletion is enabled and you have a delta query configured, saving the datasource fails with a validation error. Resolve the conflict by adjusting the system setting or removing the delta query.If you are using the JDBC V2 2.8.0 connector with a Fusion version earlier than 5.18.0, then the stray content deletion behavior is automatically determined by your delta query configuration.
When a delta query is configured, stray content deletion is disabled because delta queries only track changes, not deletions.
When no delta query is configured, stray content deletion is enabled by default.
The stray content deletion property remains accessible via the REST API for users who need explicit control for Fusion versions earlier than 5.18.0.
Existing datasources continue to work without modification. The stray content deletion behavior remains the same as before the upgrade.For new datasources with a delta query using Fusion 5.18.0 or later with stray content deletion enabled, saving your changes produces a validation error. On older Fusion versions, the datasource saves successfully with stray deletion automatically disabled.
Two recipes are now available for the REST V2 connector: a new GitHub recipe and an updated Confluence recipe that migrates to the Confluence v2 API.
These recipes require REST V2 connector version 1.1.0 or later and Fusion 5.9.x or later.
Both recipes use the REST V2 connector capabilities introduced in version 1.1.0, including hierarchical request structures for multi-level crawling, recursive requests for automatic depth traversal, skip indexation for cleaner indexes, cursor-based pagination, and content filtering by regular expressions and file size.You can access both recipes in the public GitHub repository. Recipes are treated as an open source resource where contributions are welcome.
The GitHub REST datasource configuration makes code, issues, pull requests, and commit history searchable. This connector helps teams find existing solutions before rebuilding them, accelerates developer onboarding, and surfaces decisions documented in pull requests and issues.The connector indexes 15 entity types per repository:
Repositories, issues, and pull requests
Branches, commits, and commit diffs
Tags, milestones, and releases
Collaborators
Comments for issues, pull requests, and commits
File contents: Folders and Blobs
Key features include per-branch commit crawling using the sha query parameter and recursive directory traversal through a three-stage pattern, as illustrated:The GitHub connector also supports binary content download with Tika parsing and pagination on 10 of 15 of its entity types.
Known limitations
Commits that are reachable from multiple branches are indexed once per branch. There is no cross-branch deduplication.
This connector does not support incremental crawls. Every crawl is a full crawl.
To index multiple repositories, create a new datasource and recipe for each repository.
Atlassian removed the Confluence v1 REST API from service on March 31, 2025. The previous recipe named confluence-v1.json uses the older API and does not work. All users must migrate to confluence.json.
The updated Confluence REST datasource configuration migrates to the Confluence v2 REST API, ensuring continued access to your knowledge base following Atlassian’s v1 REST API deprecation. The connector indexes comments and threaded replies alongside pages, capturing the discussions where context and decisions often live.The connector indexes 5 content types with full hierarchy preservation:
Spaces
Pages and blog posts
Page comments (footer and inline)
Blog comments (footer and inline)
Comment replies with threading
The updated recipe features 11 request configurations covering the full content hierarchy, hierarchical crawling up to four levels deep, as illustrated:The updated recipe also features granular comment crawling with separate endpoints for footer and inline comments, recursive reply threading on footer comments, and cursor-based pagination via Confluence’s _links.next.The updated recipe uses the current /wiki/api/v2 Confluence endpoints instead of the deprecated /wiki/rest/api/content endpoints. It crawls spaces as root objects, distinguishes between footer and inline comments with dedicated endpoints, supports comment reply crawling with recursive requests, and uses cursor-based pagination instead of offset-based pagination.Known limitations:
This connector does not support incremental crawls. Every crawl is a full crawl.
8 of 11 request configurations are dedicated to comment retrieval due to API structure
Only footer comment replies use recursive crawling. Inline replies retrieve direct replies only.
The FTP Pro connector is introduced as a powerful, modern replacement to the FTP V1 connector. Built on the V2 connector framework, it offers full feature parity with the FTP V1 connector while delivering enhanced security, improved configuration validation, and better resource management.This connector enables Fusion to ingest content from FTP, FTPS, and SFTP servers into Fusion.
This connector is compatible with Fusion 5.9.0 or later.
Pro connectors are built on the same framework as V2 connectors but meet higher internal standards for stability, reliability, and production readiness. If you’re currently using other V2 connectors, the process for installing and upgrading a Pro connector remains the same.More V2 connectors will be upgraded to Pro connectors in the future.
The FTP Pro connector brings enhanced security options, advanced filtering, improved connection management, and explicit protocol selection in addition to maintaining parity with the FTP V1 connector.
Enhanced authentication for SFTP
The FTP Pro connector introduces improved authentication methods, particularly for SFTP connections. You can now add an SSH private key, passphrase, and SSH host key fingerprint to verify the server identity for SFTP connections.
FTP Pro SSH private key and passphrase
FTP Pro SSH host key fingerprint
Advanced document filtering
The Pro connector adds new filtering capabilities to provide both allow-list and deny-list capabilities. You can now exclude files by file type with the Excluded file extensions field, or you can filter by file size with the new Minimum file size field.
Connection and retry management
New connection management fields provide better control over timeouts and error handling. The FTP Pro connector automatically detects and retries the connection when transient errors happen (for example, 4xx connection errors or timeouts) and does not attempt a retry in the case of permanent errors (for example, 5xx connection errors or permission failures).
FTP Pro connection properties
FTP Pro retry properties
Click the Advanced toggle to expose the retry properties.
The maximum wait time in milliseconds between retries. Range: 1000 to 300000 (5 minutes).
Restructured server configuration
The FTP Pro connector has restructured the server configuration settings. In the V1 connector, you provided a single Start link URL, and the connector used the URL scheme to detect the connection type. The Pro connector separates the connection settings into discrete fields, and you now explicitly select the connection type.
FTP Pro server property settings
Support for multiple paths
The FTP Pro connector adds support for crawling multiple paths as start points. In the V1 connector, you could only specify one starting path in the URL for the connector to crawl. The Pro connector allows multiple starting paths, so one connector can crawl all the documents you need.
More exposed fetch settings
Some fetch settings that were previously hardcoded into the V1 connector are now exposed and configurable in the Pro connector. These fetch settings provide configuration options at the system level to control fetch behavior and performance. Click the Advanced toggle to expose these settings. Then select the checkbox next to Core Properties, and select the checkbox next to Fetch Settings.
The default number of fetch threads has changed from 1 in the V1 connector to 5 in the Pro connector. This means the Pro connector creates 5 concurrent connections to your FTP server by default. Review your FTP server’s connection limits and adjust the Fetch Threads setting if necessary to avoid connection errors or rate limiting.
If you are currently using the FTP V1 connector, we encourage you to migrate to the FTP Pro connector. See the general migration guide for full instructions and the FTP-specific guide for considerations specific for the FTP connector.
The File Upload Pro has been updated to version 1.1.0 and has earned the Pro designation, showing that this connector meets higher internal standards for stability, reliability, and production readiness.
Pro connectors are built on the same framework as V2 connectors but meet higher internal standards for stability, reliability, and production readiness. If you’re currently using other V2 connectors, the process for installing and upgrading a Pro connector remains the same.More V2 connectors will be upgraded to Pro connectors in the future.
In all Fusion versions, the connector now installs as File Upload Pro instead of File Upload V2. If you are using Fusion versions earlier than 5.17.0, the Fusion UI lists the installed connector version as File Upload V2, but the File Upload Pro connector is still installed.
File Upload Pro installation comparison
Simplified connector configuration
This release removes unused core property settings from the Fetch Setting section of the File Upload Pro connector’s configuration screen. Only the Async Parsing setting remains visible in the Fetch Settings section, making the configuration process more straightforward by no longer displaying settings that this connector does not use.
In all Fusion versions, the connector now installs as Solr Pro instead of Solr V2. If you are using Fusion versions earlier than 5.17.0, the Fusion UI lists the installed connector version as Solr V2, but the Solr Pro connector is still installed.
This release resolves a resource leak issue where the SolrClient instance did not close after indexing jobs completed. The connector now properly closes the SolrClient after each job, preventing resource leaks in long-running environments. This improves stability and resource management for production environments using the Solr Pro connector.
The Solr Pro connector is introduced as a powerful, modern replacement to the Solr V1 connector. Built on the V2 connector framework, it offers full feature parity with the Solr V1 connector while delivering improved configuration validation, resource management, and user experience.This connector enables Fusion to ingest content from an external Solr cloud or standalone instance into Fusion.The Solr Pro connector maintains all parity with the V1 connector while adding new features and an improved user experience.
This connector is compatible with Solr 8 or 9 and Fusion 5.9.0 or later.
The Solr Pro connector indexes content from external Solr collections into Fusion. It does not support indexing content from Fusion.
Pro connectors are built on the same framework as V2 connectors but meet higher internal standards for stability, reliability, and production readiness. If you’re currently using other V2 connectors, the process for installing and upgrading a Pro connector remains the same.More V2 connectors will be upgraded to Pro connectors in the future.
The Solr Pro connector brings field and pre-query validation, multi-field sorting, faster pagination, and reliability improvements in addition to maintaining existing Solr V1 connector features.
Comprehensive field validation
The Solr Pro connector now includes comprehensive field validation to prevent common configuration errors.The Query and Request Handler fields are now required and are populated with default values to prevent empty crawls.The unique key field, typically id, is now automatically included in the field list.The Solr Pro connector now validates the relationship between sort fields, field lists, and filter queries before beginning a crawl.
Pre-query validation system
The Solr Pro v1.0.0 connector introduces a pre-query validation system, allowing you to identify query syntax errors and connectivity issues before a crawl begins.The Solr Pro connector now executes your actual configured query against your Solr environment during validation and provides immediate feedback on configuration issues before starting a crawl. If the query generates an error from the server in the validation process, the connector includes more detailed and actionable error messages than the Solr V1 connector so you can troubleshoot and resolve issues quickly.The pre-query validation feature uses a strict 10-second timeout, which prevents the Fusion UI from hanging on unreachable endpoints.
Multi-field sorting
The Solr V1 connector supported sorting by a single field. The Pro connector now supports sorting by multiple fields in the Filter Queries section, enabling robust and specific filtering of your Solr environment. Select Add in the Filter Queries section to add each new filter query.
Solr Pro filter queries
Simplified and faster pagination
This release simplifies the logic used to traverse large Solr collections by introducing a new cursor mark option. By moving to a consistent, single-strategy pagination approach, the connector behaves more predictably and reduces the likelihood of missing documents when processing large Solr collections.To enable cursor mark pagination, select Use cursor mark in the Solr Pro connector’s advanced settings.
Enable or disable the cursor mark, which is used for pagination. This field is enabled by default and results in faster deep paging. If this field is not enabled, then offset-based pagination is used, which was built into the V1 connector.
Note: Cursor mark pagination and offset pagination are distinct. An indexing job cannot be resumed if the pagination type is changed during a job.
More exposed connection and fetch settings
Some advanced connection and fetch settings that were hard-coded into the V1 connector are now exposed and configurable in the Solr Pro connector so you can configure the connector to meet your organization’s needs.You can access these settings by clicking the Advanced toggle.
Controls the connection pool size for high-concurrency environments.
Additionally, some system-level fetch settings that control behavior and performance are now configurable. These fetch settings are accessible by selecting the Advanced toggle, then selecting the checkboxes next to Core Properties and Fetch Settings.
Solr Pro fetch settings
None of the fetch settings are required fields. If you do not configure these fields, the Solr Pro connector uses its default values.
Reliability improvements
The Solr Pro connector introduces built-in fault tolerance to ensure that your production environment is stable and secure.Automatic retry patterns now detect when brief, temporary network disruptions occur between your Solr and Fusion environments. In these situations, the Solr connector can retry a job after a brief network interruption. If your Solr environment reports consistent connection failures, the connector can now temporarily stop requests, which prevents cascade failures in your Fusion environment.A new field, Query timeout, is available to configure how long an individual query request retries before timing out.
The individual query timeout, set in milliseconds. If you’re downloading large files, or if your network or firewall rules may cause slower connections to your Solr environment, consider setting this value higher to prevent query timeouts.
If you are currently using the Solr V1 connector, we encourage you to migrate to the Solr Pro connector. See the general migration guide for full instructions and the Solr-specific guide for considerations specific for the Solr connector.
This release simplifies infinite timeout configuration to a single checkbox, so you can test crawl performance without modifying a production-relevant setting.This release also fixes bugs related to duplicate content from canonical URLs and respecting robots.txt instructions.
Previously, the Web connector improperly handled the request-response cycle, which led to connection pool exhaustion over time when large file downloads exceeded the time set in the Connection Timeout field. Users who wanted to use infinite timeout to download large files had to edit the Connection Timeout value to -1, which overwrote the existing value and did not provide an intuitive experience.Version 2.2.2 adds a new field called Enable infinite timeout settings. When this checkbox is selected, the existing Connection Timeout value is ignored, and infinite timeout settings are active. To turn off infinite timeout settings, deselect the checkbox, and the Web connector uses the Connection Timeout value.
This improvement requires Fusion 5.9.16 or later in addition to version 2.2.2 of the Web V2 connector.
Previous versions of the Web V2 connector created duplicate documents in Fusion collections when websites use canonical URL tags. This releases fixes the duplication issue, resulting in more reliable document counts and accurate search results.
Previous versions of the Web V2 connector ignored the Obey robots meta tags and headers field, and the connector respected robots.txt directives even when the setting wasn’t active.This bug has been fixed, and now the Obey robots meta tags and headers field disregards robots meta tags and headers when the setting is inactive.
When processing XML files, previous versions of the Web V2 connector parsed non-breaking spaces to the HTML entity , which was then processed as XML. This caused the connector to throw an error because is not a valid XML entity.This has been fixed, and the connector now processes non-breaking spaces in XML files correctly.
In version 2.2.1 of the web connector, the Fetch Threads property in the Crawl Performance Properties section was removed because it reproduced the Fetch Threads property in the Fetch Settings section. This change caused datasource errors after upgrading from a previous version of the connector. Version 2.2.2 resolves this issue.
The JDBC V2 connector has been updated to version 2.7.0.This release includes binary content support, enabling full-text search of documents, images, and other binary data stored in database BINARY, VARBINARY, IMAGE, or BLOB columns. This release also fixes StackOverflowError messages related to crawls not starting and eliminates the need to manually restart your pods as a workaround.
Updating your connector to the latest version is fast and simple. Before you begin, download the connector update file Then, use the following instructions:
Open the Fusion UI and navigate to System > Blobs.
Expand the Connector Plugin section, then select lucidworks.jdbc.
Locate Replace Blob, click Choose File, and upload the downloaded zip file. Wait for the upload to finish.
Click Replace.
Confirm the version shown for the JDBC connector now reflects the updated file.
Many organizations rely on binary content, which includes PDF files, Word documents, spreadsheets, and images. To address the challenges of indexing text and non-text content, the JDBC V2 connector now supports indexing of binary content. Binary content is now streamed directly to Fusion’s indexing pipeline, enabling full-text extraction and search across multiple file formats.In previous versions of the JDBC connector, attempting to index files containing binary content results in an error similar to Metadata with key=Content, type=class [B not added. This error happened because the connector treated binary content as an unsupported metadata type and excluded it from indexing.To support binary content indexing, two new fields are available. Check the box next to Binary Content Settings in the connector’s settings to display and configure these fields.
The name of the database column containing binary content (VARBINARY, BINARY, BLOB, IMAGE, etc.). A value is required in order to use binary content indexing.
The name of the database column containing the file name or resource name for the binary content. If a value is not specified, the connector attempts to automatically detect the column name from common naming practices.
Existing content configurations will continue to work without modifications.
Binary content indexing is currently available for MSSQL and Oracle databases using the JDBC V2 connector.
Previously, the JDBC V2 connector experienced periodic failures where crawls failed to start with the error message “Start response was not received”. This issue has been resolved, and users no longer have to manually restart connector pods to prevent this issue.
The Web V2 v2.2.1 release delivers exciting improvements to JavaScript evaluation. These updates benefit users indexing sites with heavy JavaScript usage, long load times, or inconsistent indexing behavior such as missing fields or messages indicating that JavaScript is required.The release also adds automated Selenium service management. This removes the need for manual installation.
Updating your connector to the latest version is fast and simple. Before you begin, download the connector update file. Then, use the following instructions:
Open the Fusion UI and navigate to System > Blobs.
Expand the Connector Plugin section, then select lucidworks.web-v2.
Locate Replace Blob, click Choose File, and upload the downloaded zip file. Wait for the upload to finish.
Click Replace.
Confirm the version shown for the web connector now reflects the updated file.
Some sites take substantial time to load resources, execute JavaScript, and render elements. If the connector begins indexing before the page is fully ready, it may fail to index the page or certain elements. This behavior can vary from one page to another, which makes it difficult to determine the ideal configuration.To address these challenges, the Web V2 connector now includes enhanced controls for JavaScript evaluation.These changes also address common indexing issues with missing fields or error messages, such as You need to enable JavaScript to run this app.
AJAX request completion
A new option allows AJAX requests to finish before indexing proceeds. This gives content additional time to render and reduces the likelihood of missing data.
AJAX timing controls
A new property, AJAX timeout, defines how long an individual AJAX request is considered when determining whether AJAX activity has completed. Requests that exceed this limit are ignored.Request counter min wait and Request counter max wait define how long the connector waits when checking for pending AJAX requests. For example, a minimum of 500 milliseconds causes the connector to wait one half second before checking for pending AJAX requests. It continues checking until the defined maximum wait time is reached.
Updated defaults
Timeout, Script Timeout, and Implicit Wait Timeout default values are updated to reflect the latest field data across many real world use cases. If you have customized configurations for these fields, your configurations remain unchanged.
Specifies how long the connector waits for a page to finish loading, measured in milliseconds. A value of -1 allows the load process to continue without limit.
Specifies how long the connector waits for JavaScript execution to complete, measured in milliseconds. This defines the limit for script processing during page interactions.
Specifies how long the connector waits for elements to appear on the page, measured in milliseconds. This value applies when locating elements during automation tasks. Increasing this timeout may improve reliability on pages with slow or deferred JavaScript content.
These enhancements improve JavaScript execution reliability for dynamic content and resolve inconsistent crawling behavior on JavaScript heavy sites. They offer more precise control over how the Web V2 connector processes and indexes complex pages.
The Lucidworks Fusion Web V2 connector uses Selenium Grid to evaluate JavaScript and render client-side content during crawling and document extraction. This allows Fusion to accurately index web pages that rely on JavaScript for content.Previously, Selenium had to be installed manually to use with the Web V2 connector. In Web V2 v2.2.1 in Fusion 5.9.15 and later, the Selenium service is now automatically installed with the Web V2 connector and is automatically uninstalled when the connector is removed.
Manual installation of Selenium is still required in Fusion 5.9.14 or earlier, even if you use Web V2 v2.2.1. For details, see Selenium Grid setup.
A verification check is added to ensure the Selenium service is running whenever JavaScript evaluation is enabled, improving stability and preventing runtime errors. If the verification check fails, you will see a SELENIUM_CHROME_NODES_UNAVAILABLE error with a message that states:
JavaScript evaluation is enabled but no Chrome nodes are registered with Selenium hub. Ensure Chrome node pods are running and have registered with the hub.
Version 2.2.1 removes the Fetch Threads property in the Crawl Performance Properties section, a property that is reproduced in the Core Properties section. This change causes datasource errors after upgrading from a previous version of the connector. Upgrade to Web V2 v2.2.2 to resolve the datasource errors.
The Kaltura V2 connector has been updated to version 1.4.1.This release resolves a critical JDK 17 compatibility issue affecting the Kaltura V2 connector that was causing Gson JSON serialization failures and preventing successful crawls in production environments.
The connector would timeout with plugin activity errors due to Gson JSON serialization issues when processing API error responses.Version 1.4.1 implements JDK 17 compatibility fixes for Gson serialization:
Enhanced exception handling with updated error response parsing to work within JDK 17’s module system constraints.
Gson configuration updates to handle restricted field access appropriately.
Maintain compatibility with earlier JDK versions for backward compatiblilty.
The JDBC V2 connector has been updated to version 2.6.3.This release addresses a critical stack overflow issue affecting the JDBC V2 connector that was causing intermittent crawl failures and requiring periodic pod restarts.
Version 2.6.3 resolves an infinite recursion issue in the class loading subsystem, eliminating the stack overflow errors and ensuring stable connector operation.
The Web V2 connector has been updated to version 2.1.1.This release delivers feature enhancements and stability improvements to strengthen functionality, streamline configuration, and improve the user experience.Fixes and improvements:
Fixed an authentication regression introduced in v2.0.0 that caused certain authenticated datasources to fail with a “Username may not be null” error, even when credentials were configured. Authentication now works as expected for these datasources.
Restored support for viewport-related configuration properties used during JavaScript evaluation when indexing pages. Viewport width, viewport height, and device screen factor settings are now applied correctly by passing the configured values as command-line arguments to the browser.
Updated the configuration UI to hide unused properties. Only relevant options are now visible, reducing clutter.
These updates ensure smoother authentication, simplify configuration workflows, and provide greater flexibility for tailored crawling.
The Amazon Web Services (AWS) S3 V2 connector has been updated to version 1.5.0.This release adds support for the AWS S3 Object Lambda Access Point that lets you customize how data is processed when it is retrieved from AWS S3 buckets.The Lambda function associated with the access point can perform functions such as decrypting files and redacting personal information from files. For example, if you invoke the AWS S3 V2 connector using the access point and fetch encrypted files, the encrypted files are decrypted and returned to the connector.In the Fusion UI, the datasource includes the Object Lambda Access Point ARN - GET OBJECT field where you enter its Amazon Resource Name (ARN). This access point needs to be associated with the Lambda function on AWS S3 that supports the get-object API call. The connector fetches objects using the access point and triggers the corresponding Lambda function on AWS S3. If the Object Lambda Access Point ARN - GET OBJECT field does not contain a value, the AWS S3 V2 connector retrieves objects normally and the Lambda function is not invoked.
The Web V2 connector has been updated to version 2.1.0.This release adds JavaScript evaluation for the Web V2 connector, which allows the connector to extract content from a website that is only available after JavaScript has been rendered. The release also rebuilds some of the connector’s capabilities in Selenium Grid, which offers non-headless browser mode and concurrent processing. Additionally, this release restores features from the Web V1 connector.
ImportantIf you are using Web V2 2.1.0 or later, you must use Selenium Grid as part of your Web V2 connector setup.
JavaScript evaluation is available for remote and hosted connectors and supports authentication and headless browsing. The capabilities of a browser are essential, and this release introduces Selenium Grid to implement browser-based rendering.For hosted connectors, Selenium Grid support is available through Kubernetes. For remote connectors, Selenium Grid support is available through Docker Compose. See the Web V2 remote support repository for setup instructions and YAML files.The Selenium services require an x86 architecture to run properly. Running the Selenium services on an ARM-based system such as Apple Silicon is not supported.
ImportantUp to three Web V2 connectors can run simultaneously in a single cluster. This prevents reaching a max concurrency limit per Web V2 connector, which affects how much data can be sent to Selenium Grid at one time.
If you are authenticating to your website when crawling it, you can evaluate JavaScript while crawling websites. Select Evaluate JavaScript during SmartForms/SAML Login.The headless browsing setting in the Web V2 connector lets you runs browsers without actually seeing the browser. For websites that render pages on the server, the Headless browser field must be unchecked for the crawl to work correctly and retrieve links. For websites that render pages on the client side, the Headless browser field should be checked.
The depth property has been restored, allowing you to control the scope of your web crawl. The default value is -1, which does not limit the scope of the crawl. Configure this value in the Limit Document Properties section of the Web V2 connector.If a crawl fails because the start link is invalid, the Web V2 connector now marks the crawl as failed and Fusion logs an exception. This restores functionality from the Web V1 connector.
Previously, the Port field for Basic Authentication did not accept -1 as a value to accept any port. This is now resolved, and -1 is an accepted value.
Fixed a bug where Web V2 v2.0.0 failed to handle non-HTML responses such as JSON. When a JSON response was returned, the connector would complete with a success response but without indexing any data due to a premature stream closure error. This issue occurred only with Web V2 v2.0.0 on Fusion 5.9.11 and did not affect Web V2 v1.4.0.
Compatibility for this connector is extended to include all versions of Fusion 5.9.x.
The AWS S3 V2 connector is updated to v1.4.2.This release resolves a bug where recrawls failed to fetch data from folders. This occurred when a previous crawl saved errors in Crawldb and the datasourceConfig had:
The Web V2 connector has been updated to version 2.0.0. This release resolves a critical issue where the Restrict crawl to start-link checkbox appeared unchecked in the UI, while the setting was enabled in the backend. This discrepancy caused the option to be enforced even when the UI indicated it was disabled, potentially resulting in the indexing job failing silently.This release also includes general performance and compatibility improvements.Known issues in version 2.0.0:
The Link rewrite script option, which allows JavaScript to modify document links before fetching, is currently non-functional. A fix is planned for a future release.
The Max items setting enforces a limit that is one less than the configured value. For example, if set to 10,000, only 9,999 documents are fetched. A fix is planned for a future release.
ImportantStarting with Fusion 5.9.11, you must upgrade to the Web V2 connector v2.0.0 or later. Previous versions (e.g. 1.4.0) are incompatible with Fusion 5.9.11 and later versions of Fusion 5.9 due to changes introduced by the upgraded JDK in Fusion. If you are using an earlier version of Fusion, use Web V2 v1.4.0 instead.
The JDBC V2 Connector is updated to v2.6.2.This release resolves a bug that caused some updated or newly added records to be missed during delta indexing.Previously, the connector only looked at the end time of the last indexing job to find new data. This behavior meant any records added or changed during that job could be skipped during delta indexing.With this fix, the connector now looks at both the start and end times of the last job, ensuring that no records are missed when running a delta indexing job.
A recipe is added for Alfresco to enhance the search experience by implementing a hierarchical request feature that traverses multiple storage levels to locate and index file content. Several new features are added, including introducing a retry mechanism to reattempt requests failing due to server-side errors based on configurable retry counts and delay times. Additionally, this release features a skip indexation option that prevents indexing parent objects when they are only used to discover child objects. The connector now supports recursive requests to automatically retrieve nested objects regardless of depth. This release also allows for limiting documents through exclusion by regular expressions and file size constraints, and includes improvements like enhanced logging with query parameters and the addition of an index field to track the number of documents indexed per request configuration.
Hierarchical object discovery allows requests to traverse multiple levels by following the natural structure of the data under the source.
Retry feature is added to enable retrying requests if they fail due to a server-side error:
Retry Count: The number of attempts a request will be retried.
Max Delay Time: The maximum wait time in milliseconds between retries.
Skip indexation of objects feature
When enabled, the response is not indexed. This is useful when objects are requested only to discover their child objects without indexing the parent object.
Example indexing a list of files with their binary content:
Given a parent request (objectType=FILE), retrieve a list of file metadata. This request helps discover the IDs of files to be downloaded in a follow-up request.
Given a child request (objectType=FILE-DOWNLOAD with parentObjectType=FILE), download the binary content from previously discovered file metadata.
The indexed documents will include file metadata (from the FILE request) joined with binary content (from the FILE-DOWNLOAD request).
Enable Skip Indexation in FILE request to prevent indexing file metadata objects.
Recursive requests
Enables recursive retrieval of nested objects of the same ObjectType using the same request configuration. This is useful when the depth of nesting is unknown, automating the retrieval of all nested objects.
Example: Detect all nested folders from a system path where the depth of nested folders is unknown. Enable Recursive Requests to retrieve all levels automatically.
Limit documents
Exclude by RegEx allows specifying a list of key-value pairs to exclude objects from indexing.
Key references the field name of the object to exclude and supports JsonPath expressions for navigating nested objects (for example, objects.nested.path).
Value is a regular expression matched against the field value in the object. If the match succeeds, then the entire object is excluded.
Exclude by File Size allows setting minimum and maximum file sizes (in bytes) to exclude files outside the specified range.
Key references the field name of the object containing the file size and supports JsonPath expressions (for example, objects.nested.path).
Minimum File Size: Files smaller than this value will be excluded.
Maximum File Size: Files larger than this value will be excluded. (Set to -1 for no limit.)
Improved logging to include query parameters in requests.
Added index field _lw_rest_object_type_s to store the value of the ObjectType configuration property, which represents the name of the request. This helps track the number of documents indexed per request configuration.
Property Service Endpoints used for object discovery through two-level requests is deprecated. Instead, use List of Requests Configuration for configuring multiple request levels.
The REST V2 connector is introduced as a way to enable users to crawl content by exposing its data through a REST API. It can be configured to communicate with a wide selection of external datasources by making API calls and indexing the responses. As an out-of-the-box V2 connector, it provides a low-code user experience for indexing data from REST API-compatible sources.The REST V2 connector expedites setting up and indexing various datasources using premade configurations, called recipes. These recipes are templates that include all required parameters for datasource integration and can be easily adjusted to suit your specific needs. Unlike single-purpose connectors, the REST V2 connector can index data from any datasource supported by a corresponding recipe.
The REST V2 Connector relies on a public GitHub repository to store and manage recipes. Recipes are open-source and accessible to the community for use and contribution.
Two recipes are included in the initial release: Jira and Confluence. Additional recipes are being developed and will be released as they are finalized.
This release also includes two forms of authentication: OAuth and Basic Auth.
The REST connector supports crawl and recrawl:
Full crawl retrieves all objects from the datasource.
Recrawl relies on the strayContentDeletion feature from the connectors-service to ensure deleted objects from the source are also removed from the index.
Two-level object discovery:
Allows defining a root request to retrieve first-level objects.
Supports a list of child requests (children of the main request) to retrieve second-level objects.
By default, objects retrieved with the root request and child requests are indexed as individual Solr documents.
The REST connector supports two types of pagination:
Next page URL uses a URL to fetch the next page of results.
Batch size and index start use a batch size and starting index to paginate through results.
Parsing can be done in one of two ways:
When parsing within the plugin, the response is parsed as a JSON object structure using JSONPath. This is the default behavior.
When parsing with Fusion, the response is emitted directly to Fusion, where Fusion parsers handle the binary data. Enable this feature by setting the property Send as Binary Data.
By default, parent and child objects are indexed as individual Solr documents. To merge child objects into the parent object instead of indexing them separately, configure the property Child Response Mapping → Custom Solr Field. This feature works only when both parent and child objects are parsed as JSON objects.The following variables are used when configuring a datasource:
${LW_BATCH_SIZE}: Used with pagination by batch size. This variable represents the size query parameter defined in the property, Pagination By BatchSize → BatchSize.
${LW_INDEX_START}: Used with pagination by batch size. This variable represents the start-point query parameter defined in the property Pagination By BatchSize → IndexStart, which is used to traverse the pagination.
${LW_PARENT_DATA_KEY}: Used with the child request configuration. This variable is replaced with the id extracted from the root object using the property Parent Data Key.
This connector is compatible with Fusion 5.9.0 and later.
A new property, retry count, has been introduced to the S3 connector. This property value is passed in the AWS SDK, determining the number of retry attempts for retrieving the file from the AWS S3 bucket. The configurability of this property enhances the effectiveness of the connector, especially in situations of network instability.
Adds an Enable Stray Content Deletion property in the Fusion UI for the S3 connector to toggle stray content deletion on or off. When stray content deletion is enabled, content that was removed from the datasource is deleted from the index in Fusion. When stray content deletion is disabled, content that was removed from the datasource is not deleted from the index in Fusion. This property is enabled by default.
This version of the connector is compatible with Fusion 5.9.4 and later.
Updates the plugin to replace static Security Trimming with Graph Security Trimming, improving performance.
Fixes the invalid RegEx implementation that adjusts the Box URL to retrieve data from the Box datasource so the connector now accepts dashes (-) in Box start URLs.
Corrects relative path field data when indexing documents that are of type ‘File’ in nested folders.
Fixes a bug where new documents were not being indexed upon incremental crawls.
When the maximum number of referrals is reached, an exception is now thrown to handle the situation while ensuring the connector does not stop functioning.
Now you can configure the AEM connector to include child paths when indexing fields.
This option is off by default; enable it by selecting Index metadata by child path in your AEM datasource configuration.
Fixes a bug that prevented running the connector on Windows.
The basic authentication username and password fields have moved under Authentication Settings > Login Settings.
SharePoint Optimized V2 v2.1.0 is compatible with Fusion 5.9.1 and later. See the configuration reference page for the latest version and compatibility details.
Added the Events Reducer feature, which prevents performing multiple requests to the same SharePoint object and ensures index integrity. Specifically, if multiple change events are made over the same SharePoint object, the connector detects and requests the most relevant event and crawls it correctly.
Fixed Graph Security Trimming when using custom-level permissions.
Fixed an issue to reindex restored lists.
Errors from item/attachment are deleted from crawlDB after being successfully retried.
Fixed a pagination issue that limited the number of records returned.
Added support for pagination of IBM Db2 (version 11 and earlier) by using a template sub-query:
-- add any WHERE, ORDER BY clauses as needed following your_table\SELECT * FROM ( SELECT *, ROW_NUMBER() OVER () AS row_num FROM your_table ) AS subquery WHERE row_num BETWEEN :offset + 1 AND (:offset + :limit)
Fixed an issue where UTF-8 characters stored as CLOB data types didn’t index properly.
The JDBC V2 Connector is updated to v2.6.0 and implements a pagination feature. This allow the JDBC connector to ingest a specified number of rows during the crawl process.This feature is controlled by following newly added properties:
LIMIT - specifies the number of rows returned in the results.
OFFSET - dictates the number of rows to skip from the beginning of the returned data before presenting the results.
disableAutomaticPagination - disables automatic pagination to ignore limit and offset fields.
SharePoint Optimized V2 v2.0.0 is compatible with Fusion 5.9.1 and later. See the configuration reference page for the latest version and compatibility details.
LDAP ACLs V2 connector v2.0.0 is compatible with Fusion 5.9.1 and later. See the configuration reference page for the latest version and compatibility details.
The updated connector:
Supports remote configurations.
Uses the SDK that correctly processes the NOT-EQUAL phrase in delete-by-query.
If you are on Fusion 5.9.0 or later, you must use the SharePoint Optimized V2 v1.6.0 connector or later.
For SharePoint Online connector OAuth authentication and NTLM authentication, an issue with the SharePoint Optimized V2 connector crawl was fixed. Now the crawl fails if the data source is configured with incorrect credentials.
When the moderation status of a published document moves to unpublished, the document is deleted from the index.
The URL that requests folder metadata is now properly encoded and retrieves requested files correctly.
MapDB tables that are no longer needed after indexing are now automatically removed when the indexing job completes.
The connector no longer uses the index pipeline client when the onlyMetadata property is set to true or when Tika async parsing is enabled. Instead, documents are processed using the Connectors SDK’s .newDocument method.
A bug that affected JDBC datasources with large document sizes has been fixed. Previously, the indexing job would fail and produce a Solr error message similar to the following:
Document contains at least one immense term in field="FIELD_NAME"
This bug has been fixed and indexing large documents should now work as expected.
Added support for the graph security trimming stage for Active Directory in Azure.
This describes how to migrate your pre-Fusion 5.8 Graph Security Trimming query pipeline stage setup to Fusion 5.8 or later. It applies to deployments using:
SharePoint Optimized V2 connector v1.1.0 or later
LDAP ACLs V2 connector v1.4.0 or later to crawl Active Directory in Azure
The LDAP ACLs V2 connector v1.2.0 or later to crawl Active Directory in LDAP
The SharePoint Optimized V2 and LDAP ACLs V2 datasources must index the content documents and ACL documents to the same collection. Ensure both datasources use the same value, contentCollection, for the field ACL Collection ID.
If using SharePoint-Optimized and LDAP ACLs < v2.0.0
Update the ACL Collection Id in the datasource configuration.The SharePoint-Optimized and LDAP ACLs datasources must index their content_documents and acl_documents to the same collection. Make sure the property Security -> ACL Collection in both datasources have the same value. In both datasources, SharePoint-Optimized and LDAP ACLs, check the property Security -> ACL Collection Id and make sure it points to the same content-collection.
Navigate to Indexing > Datasources.
Open your SharePoint Optimized V2 or LDAP ACLs V2 datasource.
Under Security, update the configuration to use contentCollection as the ACL Collection ID. The Security checkbox must be checked for this field to appear.
Save the configuration.
Repeat this process for all required datasources.
If using SharePoint-Optimized and LDAP ACLs >= v2.0.0
Recreate or update the datasources. If only updated, it is not possible to go back to the configuration of a previous plugin version.
By default, the LDAP ACLs and SharePoint-Optimized V2 datasources will index the content_documents and acl_documents to the same collection.
Navigate to Indexing > Datasources.
Open your SharePoint Optimized V2 or LDAP ACLs V2 datasource.
Under Graph Security Filtering Configuration, select Enable security trimming.
Repeat this process for all required datasources.
A new field, _lw_document_type_s, has been added to differentiate between document types. SharePoint content documents use the value contentDocument. ACL documents use the value aclDocument.
A bug that caused a NullPointerException error when the listDefaultDisplayFormUrl value is null has been fixed.
A bug that prevented the connector from indexing SharePoint Online URLs containing certain special characters, such as +, &, and #, has been fixed.
A bug that stopped the indexing job unexpectedly when the digestValue request fails has been fixed.
Fixed a bug that removed access control lists (ACLs) during incremental crawls in Azure Active Directory. This bug resulted from a deviation from the LDAP ACLs V2 connector’s unique incremental crawl behavior for Azure Active Directory.
Fixed a bug that sometimes deleted ACLs that the indexing job failed to update during its previous run.
Known issue: The job may complete with a “Successful” status even when a network communication error occurred.
Fixed a bug that sometimes prevented the connector from parsing PDF, Word, Excel, and other file types.
Some conflicting dependencies were removed to resolve errors.
A missing dependency, guava-retrying, was added to resolve a NoClassDefFoundError error.
A bug was fixed that resulted in the connector performing full crawls instead of recrawling as expected.
Known issue: JavaScript evaluation is not functional at this time.
Known issue: The counter values in the Datasources > DATASOURCE_NAME > Job History view do not reflect actual values, when the datasource uses the Web V2 connector.
A bug that sometimes prevented files from being deleted as expected when the datasource was recrawled was fixed.
A bug that prevented the connector from indexing data within certain lists was fixed.
When indexing data to the MetaInfo_s field, an error could occur that prevented indexing data when the data was too large. Now, data larger than 32,766 bytes is indexed to the MetaInfo_txt field instead.
