LDAP ACLs V2 - Lucidworks documentation

Latest version: v2.1.1 Compatible with Fusion version: 5.9.1 and later The Active Directory Connector for ACLs indexes Access Control List (ACL) information so that it can be used by other connectors.

Verify your connector versionThis connector depends on specific Fusion versions. See the following table for the required versions:

Fusion version	Connector version
Fusion 5.6.1 and later	v1.0.0 and later
Fusion 5.9.0 and later	v1.5.0 and later
Fusion 5.9.1 and later	v2.0.0 and later

For connector downloads, see Download Connectors.

Pod limitThe LDAP ACLs V2 connector does not support running multiple instances. Don’t run the connector on more than one pod.

Full recrawls

Starting in Fusion 5.7, subsequent crawls work differently with the LDAP ACLs V2 connector than other connectors. Crawls follow this process:

Every time the connector crawl runs, all documents are indexed.
Each document is assigned a new field, _lw_job_id_s.
The connector job assigns the jobID value to this field.
When the crawl finishes, the job deletes documents that do not have the latest jobID value.

Remote connectors

You can configure the LDAP ACLs V2 connector (v2.0.0 and later) to running remotely in Fusion versions 5.9.1 and later.

Configure Remote V2 Connectors

If you need to index data from behind a firewall, you can configure a V2 connector to run remotely on-premises using TLS-enabled gRPC.

Prerequisites

Before you can set up an on-prem V2 connector, you must configure the egress from your network to allow HTTP/2 communication into the Fusion cloud. You can use a forward proxy server to act as an intermediary between the connector and Fusion.The following is required to run V2 connectors remotely:

The plugin zip file and the connector-plugin-standalone JAR.
A configured connector backend gRPC endpoint.
Username and password of a user with a remote-connectors or admin role.
If the host where the remote connector is running is not configured to trust the server’s TLS certificate, you must configure the file path of the trust certificate collection.

If your version of Fusion doesn’t have the remote-connectors role by default, you can create one. No API or UI permissions are required for the role.

Connector compatibility

Only V2 connectors are able to run remotely on-premises. You also need the remote connector client JAR file that matches your Fusion version. You can download the latest files at V2 Connectors Downloads.

Whenever you upgrade Fusion, you must also update your remote connectors to match the new version of Fusion.

The gRPC connector backend is not supported in Fusion environments deployed on AWS.

System requirements

The following is required for the on-prem host of the remote connector:

(Fusion 5.9.0-5.9.10) JVM version 11
(Fusion 5.9.11) JVM version 17
Minimum of 2 CPUs
4GB Memory

Note that memory requirements depend on the number and size of ingested documents.

Enable backend ingress

In your values.yaml file, configure this section as needed:

ingress:
  enabled: false
  pathtype: "Prefix"
  path: "/"
  #host: "ingress.example.com"
  ingressClassName: "nginx"   # Fusion 5.9.6 only
  tls:
    enabled: false
    certificateArn: ""
    # Enable the annotations field to override the default annotations
    #annotations: ""

Set enabled to true to enable the backend ingress.
Set pathtype to Prefix or Exact.
Set path to the path where the backend will be available.
Set host to the host where the backend will be available.
In Fusion 5.9.6 only, you can set ingressClassName to one of the following:
- nginx for Nginx Ingress Controller
- alb for AWS Application Load Balancer (ALB)
Configure TLS and certificates according to your CA’s procedures and policies.
TLS must be enabled in order to use AWS ALB for ingress.

Connector configuration example

kafka-bridge:
  target: mynamespace-connectors-backend.lucidworkstest.com:443 # mandatory
  plain-text: false # optional, false by default.
  proxy-server: # optional - needed when a forward proxy server is used to provide outbound access to the standalone connector
    host: host
    port: some-port
    user: user # optional
    password: password # optional
  trust: # optional - needed when the client's system doesn't trust the server's certificate
    cert-collection-filepath: path1

proxy: # mandatory fusion-proxy
  user: admin
  password: password123
  url: https://fusiontest.com/ # needed only when the connector plugin requires blob store access

plugin: # mandatory
  path: ./fs.zip
  type: #optional - the suffix is added to the connector id
    suffix: remote

Minimal example

kafka-bridge:
  target: mynamespace-connectors-backend.lucidworkstest.com:443

proxy:
  user: admin
  password: "password123"

plugin:
  path: ./testplugin.zip

Logback XML configuration file example

<configuration>
    <appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
        <encoder class="com.lucidworks.logging.logback.classic.LucidworksPatternLayoutEncoder">
            <pattern>%d - %-5p [%t:%C{3.}@%L] - %m{nolookups}%n</pattern>
            <charset>utf8</charset>
        </encoder>
    </appender>

    <appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <file>${LOGDIR:-.}/connector.log</file>
        <rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
            <!-- rollover daily -->
            <fileNamePattern>${LOGDIR:-.}/connector-%d{yyyy-MM-dd}.%i.log.gz</fileNamePattern>
            <maxFileSize>50MB</maxFileSize>
            <totalSizeCap>10GB</totalSizeCap>
        </rollingPolicy>
        <encoder class="com.lucidworks.logging.logback.classic.LucidworksPatternLayoutEncoder">
            <pattern>%d - %-5p [%t:%C{3.}@%L] - %m{nolookups}%n</pattern>
            <charset>utf8</charset>
        </encoder>
    </appender>

    <root level="INFO">
        <appender-ref ref="CONSOLE"/>
        <appender-ref ref="FILE"/>
    </root>
</configuration>

Run the remote connector

java [-Dlogging.config=[LOGBACK_XML_FILE]] \
  -jar connector-plugin-client-standalone.jar [YAML_CONFIG_FILE]

The logging.config property is optional. If not set, logging messages are sent to the console.

Test communication

You can run the connector in communication testing mode. This mode tests the communication with the backend without running the plugin, reports the result, and exits.

java -Dstandalone.connector.connectivity.test=true -jar connector-plugin-client-standalone.jar [YAML_CONFIG_FILE]

Encryption

In a deployment, communication to the connector’s backend server is encrypted using TLS. You should only run this configuration without TLS in a testing scenario. To disable TLS, set plain-text to true.

Egress and proxy server configuration

One of the methods you can use to allow outbound communication from behind a firewall is a proxy server. You can configure a proxy server to allow certain communication traffic while blocking unauthorized communication. If you use a proxy server at the site where the connector is running, you must configure the following properties:

Host. The hosts where the proxy server is running.
Port. The port the proxy server is listening to for communication requests.
Credentials. Optional proxy server user and password.

When you configure egress, it is important to disable any connection or activity timeouts because the connector uses long running gRPC calls.

Password encryption

If you use a login name and password in your configuration, run the following utility to encrypt the password:

Enter a user name and password in the connector configuration YAML.

Run the standalone JAR with this property:

-Dstandalone.connector.encrypt.password=true

Retrieve the encrypted passwords from the log that is created.
Replace the clear password in the configuration YAML with the encrypted password.

Connector restart (5.7 and earlier)

The connector will shut down automatically whenever the connection to the server is disrupted, to prevent it from getting into a bad state. Communication disruption can happen, for example, when the server running in the connectors-backend pod shuts down and is replaced by a new pod. Once the connector shuts down, connector configuration and job execution are disabled. To prevent that from happening, you should restart the connector as soon as possible.You can use Linux scripts and utilities to restart the connector automatically, such as Monit.

Recoverable bridge (5.8 and later)

If communication to the remote connector is disrupted, the connector will try to recover communication and gRPC calls. By default, six attempts will be made to recover each gRPC call. The number of attempts can be configured with the max-grpc-retries bridge parameters.

Job expiration duration (5.9.5 only)

The timeout value for irresponsive backend jobs can be configured with the job-expiration-duration-seconds parameter. The default value is 120 seconds.

Use the remote connector

Once the connector is running, it is available in the Datasources dropdown. If the standalone connector terminates, it disappears from the list of available connectors. Once it is re-run, it is available again and configured connector instances will not get lost.

Enable asynchronous parsing (5.9 and later)

To separate document crawling from document parsing, enable Tika Asynchronous Parsing on remote V2 connectors.

Security trimming

Starting in Fusion 5.9, you can configure the LDAP ACLs V2 connector with the SharePoint Optimized V2 connector to support security trimming.

Configure Security Trimming for SharePoint Optimized V2

You can configure the SharePoint Optimized V2 connector to use security trimming so that query results are filtered based on the roles and permissions assigned to the user.To configure security trimming, you’ll need to set up and run a SharePoint Optimized V2 datasource, an LDAP ACLs V2 datasource, and a Graph Security Trimming query stage in the same app and collection.When a crawl is run, the SharePoint Optimized V2 and LDAP ACLs V2 datasources must index the content documents and ACL documents to the same collection.

ACL documents: Users, Groups, and their Role Assignments.
Content documents: The SharePoint objects with metadata and content (Sites, Lists, Items). These documents have _lw_acl_ss fields which determines who can see the docs when searching.

Set up the SharePoint datasource

Navigate to Indexing > Datasources.
Install the datasource connector if not already installed.
Click Add and select SharePoint Optimized V2.
Fill in all required fields.
Configure only one authentication method. Enable NLTM Authentication Settings or SharePoint Online Authentication and configure the fields as explained below.

NTLM Authentication

This method connects to SharePoint on-premises server instances, such as SharePoint Server 2013, 2016, and 2019. When using this authentication method, the connector will index contentDocuments and the following aclDocuments: sharepointGroups, siteAdmins, roleDefinition, and roleAssignment.To use this authentication method, in your SharePoint Optimized V2 datasource, select the NTLM Authentication Settings checkbox and configure the following fields:

User
Password
Domain
Workstation

SharePoint Online Authentication

These methods connect to SharePoint Online server instances. When using one of these methods, the connector will index contentDocuments and the following aclDocuments: sharepointGroups, siteAdmins, roleDefinition, roleAssignment and sharepointUsers in which loginName ends with onmicrosoft.com.

Basic

To use this authentication method, in your SharePoint Optimized V2 datasource, select the SharePoint Online Authentication checkbox and configure the following fields:

Sharepoint online account
Password

App only (OAuth protocol)

To use this authentication method, in your SharePoint Optimized V2 datasource, select the SharePoint Online Authentication checkbox and configure the following fields:

Azure AD client ID
Azure AD tenant
Azure AD Client Secret
Azure AD login endpoint (advanced)
Azure AD Refresh Token (advanced)

App only with private key

To use this authentication method, in your SharePoint Optimized V2 datasource, select the SharePoint Online Authentication checkbox and configure the following fields:

Azure AD client ID
Azure AD tenant
Azure AD login endpoint
Azure AD PKCS12 Base64 Keystore
Azure AD PKCS12 Keystore Password

Set up the LDAP datasource

Navigate to Indexing > Datasources.
Install the datasource connector if not already installed.
Click Add and select LDAP and Azure ACLs Connector (V2).
Fill in all required fields.
Configure authentication methods. Enter LDAP login credentials and/or enable Azure AD Properties and configure the fields as explained below.

LDAP Authentication

This method connects to an LDAP AD server. When using this method, the connector will index the following aclDocuments: ldapUsers, and ldapGroups.To use this authentication method, in your LDAP and Azure ACLs Connector (V2) datasource, configure the following fields:

Login User Principal
Login Password

Azure AD Authentication

This method connects to an Azure AD server. When using this method, the connector will index the following aclDocuments: azureUsers, and azureGroupsTo use this authentication method, in your LDAP and Azure ACLs Connector (V2) datasource, select the Azure AD Properties checkbox and configure the following fields:

Azure AD Tenant ID
Azure AD Client ID
Azure AD Client Secret

Supported authentication methods for security trimming

	LDAP AD	Azure AD
SharePoint On-Premises	NTLM Authentication and LDAP Authentication	NTLM Authentication and Azure AD Authentication
SharePoint Online	N/A	Any SharePoint Online authentication method and Azure AD Authentication

Configure ACL collection

The SharePoint Optimized V2 and LDAP ACLs V2 datasources must index the content documents and ACL documents to the same collection. Ensure both datasources use the same value, contentCollection, for the field ACL Collection ID.

If using SharePoint-Optimized and LDAP-ACLs < v2.0.0

Update the ACL Collection Id in the datasource configuration.The SharePoint-Optimized and LDAP-ACLs datasources must index their content_documents and acl_documents to the same collection. Make sure the property Security -> ACL Collection in both datasources have the same value. In both datasources, SharePoint-Optimized and LDAP-ACLs, check the property Security -> ACL Collection Id and make sure it points to the same content-collection.

Navigate to Indexing > Datasources.
Open your SharePoint Optimized V2 or LDAP ACLs V2 datasource.
Under Security, update the configuration to use contentCollection as the ACL Collection ID.
The Security checkbox must be checked for this field to appear.
Save the configuration.

Repeat this process for all required datasources.

If using SharePoint-Optimized and LDAP-ACLs >= v2.0.0

Recreate or update the datasources. If only updated, it is not possible to go back to the configuration of a previous plugin version.By default, the LDAP-ACLs and SharePoint-Optimized V2 datasources will index the content_documents and acl_documents to the same collection.

Navigate to Indexing > Datasources.
Open your SharePoint Optimized V2 or LDAP ACLs V2 datasource.
Under Graph Security Filtering Configuration, select Enable security trimming.

Repeat this process for all required datasources.

Set up Graph Security Trimming

A Graph Security Trimming stage is used to pull all nested groups for a user. Then the Solr join query takes those ACL IDs found in the graph query and filters out everything that does not match one of the ACLs.

Navigate to Querying > Query Pipelines.
Open the query pipeline associated with your SharePoint Optimized V2 or LDAP ACLs V2 data.
Click Add a new pipeline stage and select Graph Security Trimming.
Configure the stage with the following settings:
Field Value
User ID source query_param or header
User ID key The key that contains the User ID


Field	Value
User ID source	`query_param` or `header`
User ID key	The key that contains the User ID

Test the configuration

To confirm that security trimming works as configured, run the following test:

First, run the SharePoint Optimized V2 and LDAP ACLs V2 datasources.
Run a series of queries to test user permissions are working as intended:
1. Run a query using a User ID key with no permissions. You should see no search results.
2. Run a query using a User ID key that has access to some documents. You should see some search results.
3. Run a query using a User ID key that has access to all documents. You should see all documents.
  Facet by _lw_document_type_s: contentDocument to see only the SharePoint docs, otherwise aclDocuments will be also shown.

Configuration

When entering configuration values in the UI, use unescaped characters, such as \t for the tab character. When entering configuration values in the API, use escaped characters, such as \\t for the tab character.

Fusion Connectors

​Full recrawls

​Remote connectors

​Security trimming

​Configuration

Full recrawls

Remote connectors

Security trimming

Configuration