Release Notes

• Fixed an authentication regression introduced in v2.0.0 that caused certain authenticated datasources to fail with a “Username may not be null” error, even when credentials were configured. Authentication now works as expected for these datasources.
• Restored support for viewport-related configuration properties used during JavaScript evaluation when indexing pages. Viewport width, viewport height, and device screen factor settings are now applied correctly by passing the configured values as command-line arguments to the browser.
• Updated the configuration UI to hide unused properties. Only relevant options are now visible, reducing clutter.

​

JavaScript evaluation

​

Improvements

​

Bug fix

• Fixed a bug where Web V2 v2.0.0 failed to handle non-HTML responses such as JSON. When a JSON response was returned, the connector would complete with a success response but without indexing any data due to a premature stream closure error. This issue occurred only with Web V2 v2.0.0 on Fusion 5.9.11 and did not affect Web V2 v1.4.0.
• Compatibility for this connector is extended to include all versions of Fusion 5.9.x.

• objectKeys configured
• enableStrayContent set to false

• The Link rewrite script option, which allows JavaScript to modify document links before fetching, is currently non-functional. A fix is planned for a future release.
• The Max items setting enforces a limit that is one less than the configured value. For example, if set to 10,000, only 9,999 documents are fetched. A fix is planned for a future release.

​

Summary

​

New recipe

• A new recipe is included as part of this release: Alfresco.
• This new recipe integrates with the Alfresco information management software improve to improve the search experience.
• The recipe implements the new hierarchical request feature to locate and index file content at multiple storage levels.

​

New features

Hierarchical request feature:

• Hierarchical object discovery allows requests to traverse multiple levels by following the natural structure of the data under the source.

• Retry Count: The number of attempts a request will be retried.
• Max Delay Time: The maximum wait time in milliseconds between retries.

Skip indexation of objects feature

• When enabled, the response is not indexed. This is useful when objects are requested only to discover their child objects without indexing the parent object.
• Example indexing a list of files with their binary content:
  • Given a parent request (objectType=FILE), retrieve a list of file metadata. This request helps discover the IDs of files to be downloaded in a follow-up request.
  • Given a child request (objectType=FILE-DOWNLOAD with parentObjectType=FILE), download the binary content from previously discovered file metadata.
  • The indexed documents will include file metadata (from the FILE request) joined with binary content (from the FILE-DOWNLOAD request).
  • Enable Skip Indexation in FILE request to prevent indexing file metadata objects.

Recursive requests

• Enables recursive retrieval of nested objects of the same ObjectType using the same request configuration. This is useful when the depth of nesting is unknown, automating the retrieval of all nested objects.
• Example: Detect all nested folders from a system path where the depth of nested folders is unknown. Enable Recursive Requests to retrieve all levels automatically.

Limit documents

• Exclude by RegEx allows specifying a list of key-value pairs to exclude objects from indexing.
  • Key references the field name of the object to exclude and supports JsonPath expressions for navigating nested objects (for example, objects.nested.path).
  • Value is a regular expression matched against the field value in the object. If the match succeeds, then the entire object is excluded.
• Exclude by File Size allows setting minimum and maximum file sizes (in bytes) to exclude files outside the specified range.
  • Key references the field name of the object containing the file size and supports JsonPath expressions (for example, objects.nested.path).
  • Minimum File Size: Files smaller than this value will be excluded.
  • Maximum File Size: Files larger than this value will be excluded. (Set to -1 for no limit.)

​

Improvements

• Improved logging to include query parameters in requests.
• Added index field _lw_rest_object_type_s to store the value of the ObjectType configuration property, which represents the name of the request. This helps track the number of documents indexed per request configuration.

​

Deprecation

• Property Service Endpoints used for object discovery through two-level requests is deprecated. Instead, use List of Requests Configuration for configuring multiple request levels.

• The REST V2 Connector relies on a public GitHub repository to store and manage recipes. Recipes are open-source and accessible to the community for use and contribution.
• Two recipes are included in the initial release: Jira and Confluence. Additional recipes are being developed and will be released as they are finalized.
• This release also includes two forms of authentication: OAuth and Basic Auth.

• Full crawl retrieves all objects from the datasource.
• Recrawl relies on the strayContentDeletion feature from the connectors-service to ensure deleted objects from the source are also removed from the index.

• Allows defining a root request to retrieve first-level objects.
• Supports a list of child requests (children of the main request) to retrieve second-level objects.
• By default, objects retrieved with the root request and child requests are indexed as individual Solr documents.

• Next page URL uses a URL to fetch the next page of results.
• Batch size and index start use a batch size and starting index to paginate through results.

• When parsing within the plugin, the response is parsed as a JSON object structure using JSONPath. This is the default behavior.
• When parsing with Fusion, the response is emitted directly to Fusion, where Fusion parsers handle the binary data. Enable this feature by setting the property Send as Binary Data.

• ${LW_BATCH_SIZE}: Used with pagination by batch size. This variable represents the size query parameter defined in the property, Pagination By BatchSize → BatchSize.
• ${LW_INDEX_START}: Used with pagination by batch size. This variable represents the start-point query parameter defined in the property Pagination By BatchSize → IndexStart, which is used to traverse the pagination.
• ${LW_PARENT_DATA_KEY}: Used with the child request configuration. This variable is replaced with the id extracted from the root object using the property Parent Data Key.

• The timeout limit to retrieve ACLs from LDAP services is extended.

• Incorporates OAuth for compatibility with Ping Identity and Azure.
• Fixes a bug where links listed under BULK START LINKS were not being indexed.

• A new property, retry count, has been introduced to the S3 connector. This property value is passed in the AWS SDK, determining the number of retry attempts for retrieving the file from the AWS S3 bucket. The configurability of this property enhances the effectiveness of the connector, especially in situations of network instability.

• Adds validation to establish a connection with Kaltura before commencing the crawling process.
• Updates the plugin to replace static Security Trimming with Graph Security Trimming, improving performance.

• Adds an Enable Stray Content Deletion property in the Fusion UI for the S3 connector to toggle stray content deletion on or off. When stray content deletion is enabled, content that was removed from the datasource is deleted from the index in Fusion. When stray content deletion is disabled, content that was removed from the datasource is not deleted from the index in Fusion. This property is enabled by default.

• Updates the plugin to replace static Security Trimming with Graph Security Trimming, improving performance.
• Fixed an issue where security trimming was enabled by default.

• Provides a convenient way to quickly ingest data from your local filesystem.
• Constructed using the V2 framework and serves as a replacement for the classic version.

• Updates the plugin to replace static Security Trimming with Graph Security Trimming, improving performance.
• Fixes the invalid RegEx implementation that adjusts the Box URL to retrieve data from the Box datasource so the connector now accepts dashes (-) in Box start URLs.
• Corrects relative path field data when indexing documents that are of type ‘File’ in nested folders.
• Fixes a bug where new documents were not being indexed upon incremental crawls.
• Migrates the Box-Java-SDK to version 4.8.0.

• When the maximum number of referrals is reached, an exception is now thrown to handle the situation while ensuring the connector does not stop functioning.

• Upgrades the plugin to use the latest SDK version.
• Updates the plugin to replace static Security Trimming with Graph Security Trimming, improving performance.
• Fixes a bug where an “Error validating datasource” message displayed after trying to save a datasource with the Enable DFS connection property.

• Now you can configure the AEM connector to include child paths when indexing fields.
  This option is off by default; enable it by selecting Index metadata by child path in your AEM datasource configuration.
• Fixes a bug that prevented running the connector on Windows.
• The basic authentication username and password fields have moved under Authentication Settings > Login Settings.

• Fixes Graph security trimming not working with Everyone except external access.
• Allows connector to reach maximum referral. The connector will throw the exception but will not stop.

• Added the Events Reducer feature, which prevents performing multiple requests to the same SharePoint object and ensures index integrity. Specifically, if multiple change events are made over the same SharePoint object, the connector detects and requests the most relevant event and crawls it correctly.
• Fixed Graph Security Trimming when using custom-level permissions.
• Fixed an issue to reindex restored lists.
• Errors from item/attachment are deleted from crawlDB after being successfully retried.

• Fixed a pagination issue that limited the number of records returned.
• Added support for pagination of IBM Db2 (version 11 and earlier) by using a template sub-query:
  Copy

  -- add any WHERE, ORDER BY clauses as needed following your_table\
  SELECT * FROM ( SELECT *, ROW_NUMBER() OVER () AS row_num FROM your_table ) AS subquery WHERE row_num BETWEEN :offset + 1 AND (:offset + :limit)
  

• Fixed an issue where UTF-8 characters stored as CLOB data types didn’t index properly.

• LIMIT - specifies the number of rows returned in the results.
• OFFSET - dictates the number of rows to skip from the beginning of the returned data before presenting the results.
• disableAutomaticPagination - disables automatic pagination to ignore limit and offset fields.

• Indexes the all-users acl document when indexing from AzureAD.
• Fixes Graph Security Trimming with Everyone except external access in SharePoint Online.

• Exposes configuration options for the validation timeout.
• Resolves an issue where documents were deleted on incremental crawls when stray delete was enabled.

• This adds back in the Kaltura External Media Entry Compare Attribute fields.

• The connector supports remote configurations.

• Supports remote configurations.
• Uses the SDK that correctly processes the NOT-EQUAL phrase in delete-by-query.
• Supports unauthenticated proxy.

• The connector allows OAuth 2 support for JWT token.

• This release expands the types of character large objects (CLOBs) which can be indexed to include IBM CLOBs.

• For SharePoint Online connector OAuth authentication and NTLM authentication, an issue with the SharePoint Optimized V2 connector crawl was fixed. Now the crawl fails if the data source is configured with incorrect credentials.
• When the moderation status of a published document moves to unpublished, the document is deleted from the index.
• The URL that requests folder metadata is now properly encoded and retrieves requested files correctly.

• The datasource indexing job will now return an error when invalid access credentials are provided instead of reporting a success.
• The Security parameter is now set to enabled by default.

• Fixed an issue that occurred when reaching Kaltura’s API limit of 10,000 documents. Indexing over 10,000 documents should now work as expected.

• The JDBC V2 connector now supports indexing character large objects (CLOBs).

• A NoClassDefFound error in the connector has been fixed.

• MapDB tables that are no longer needed after indexing are now automatically removed when the indexing job completes.
• The connector no longer uses the index pipeline client when the onlyMetadata property is set to true or when Tika async parsing is enabled. Instead, documents are processed using the Connectors SDK’s .newDocument method.

• A bug with the Rewrite URI configuration option that prevented the configuration from being applied during the indexing job has been fixed.
• The SAML, Form, and NTML authorization form fields are updated to match those present in the Web V1 connector.

• A bug that affected JDBC datasources with large document sizes has been fixed. Previously, the indexing job would fail and produce a Solr error message similar to the following:
  Copy

  Document contains at least one immense term in field="FIELD_NAME"
  

  This bug has been fixed and indexing large documents should now work as expected.

• Added support for the graph security trimming stage for Active Directory in Azure.

• A bug that prevented the connector from retrying after a connection error has been fixed.

• The statement.setMaxRows() field was added to resolve a timeout error for large queries on a JDBC data source.

• A new field, _lw_document_type_s, has been added to differentiate between document types. SharePoint content documents use the value contentDocument. ACL documents use the value aclDocument.
• A bug that caused a NullPointerException error when the listDefaultDisplayFormUrl value is null has been fixed.
• A bug that prevented the connector from indexing SharePoint Online URLs containing certain special characters, such as +, &, and #, has been fixed.
• A bug that stopped the indexing job unexpectedly when the digestValue request fails has been fixed.

• Fixed a memory leak that resulted in an OutOfMemoryError runtime error, causing recrawls to quickly fail.

• Fixed an issue where sometimes the connector did not index all attachments in full crawls.
• Fixed an error related to OAuth authentication where the body would not return the expected value.
• Folders with unique permissions no longer inherit list permissions instead of their own.

• Fixed a bug that sometimes resulted in an OutOfMemory: Java Heap Space error when the connector was run remotely.
• Fixed a dependency issue that sometimes resulted in a Error starting controllers error that caused the job to fail.

• Fixed a bug that removed access control lists (ACLs) during incremental crawls in Azure Active Directory. This bug resulted from a deviation from the LDAP ACLs V2 connector’s unique incremental crawl behavior for Azure Active Directory.
• Fixed a bug that sometimes deleted ACLs that the indexing job failed to update during its previous run.
• Known issue: The job may complete with a “Successful” status even when a network communication error occurred.

• Fixed a bug that sometimes generated a “User cannot be found” error statement and prevented the connector from indexing files.
• Updated the description of the Item size limit property to clarify its behavior.

• Fixed a bug that sometimes prevented the connector from parsing PDF, Word, Excel, and other file types.
• Some conflicting dependencies were removed to resolve errors.
• A missing dependency, guava-retrying, was added to resolve a NoClassDefFoundError error.
• A bug was fixed that resulted in the connector performing full crawls instead of recrawling as expected.
• Known issue: JavaScript evaluation is not functional at this time.
• Known issue: The counter values in the Datasources > DATASOURCE_NAME > Job History view do not reflect actual values, when the datasource uses the Web V2 connector.

• A bug that sometimes prevented files from being deleted as expected when the datasource was recrawled was fixed.
• A bug that prevented the connector from indexing data within certain lists was fixed.
• When indexing data to the MetaInfo_s field, an error could occur that prevented indexing data when the data was too large. Now, data larger than 32,766 bytes is indexed to the MetaInfo_txt field instead.
• The connector now accepts URLs with spaces.