Skip to main content
  • Latest version: v1.1.0
  • Compatible with Fusion version: 5.9.0 and later
The REST V2 connector enables users to crawl content by exposing its data through a REST API. It can be configured to communicate with a wide selection of external datasources by making API calls and indexing the responses. As an out-of-the-box V2 connector, it provides a low-code user experience for indexing data from REST API-compatible sources. This connector is the alternative solution for various deprecated and removed V1 connectors. Key concepts:
  • The REST V2 connector is a new approach to Fusion connectors: one connector that can index data from many different datasources.
  • The connector is powered by recipes, which are JSON configurations for the REST V2 connector that are customized for a specific datasource.
  • A supported datasource will have a corresponding recipe.
  • Over time, the deprecated V1 connectors may be replaced by comparable REST V2 connector recipes.
REST V2 connector features:
  • Indexes REST API-compatible datasources for data integration.
  • Supports initial full crawl and re-full crawl and object mapping.
  • Remote framework and pagination are supported.
  • Supports basic and OAuth authentication methods.
  • Enhances the datasource retrieval capability with a globally standardized format connector.
  • Saves DevOps time by using a preconfigured connector.

​
How REST V2 is different from other connectors

Most connectors are developed solely for a specific datasource. The REST V2 connector is able to work any datasources that has a supported recipe. This gives the REST V2 connector a great amount of flexibility and enhances its usefulness as part of the Lucidworks connectors offerings. An API call is made to the REST V2 connector containing a recipe as the body. API calls to set up the REST V2 connector go through the Connector Datasources API. This article lists and describes the Fusion properties used by REST V2. The recommended method of populating these fields is through the use of a recipe. For information on how to get a recipe from the public GitHub repository into Fusion, see How to use REST V2 recipes. The REST V2 connector performs a full crawl each time the connectors runs, as incremental crawling is not supported. When changes are made to the source, such as adding, deleting, or modifying content, those changes will reflect in the index after the next crawl is performed. You can check for new documents in the index by navigating to Querying > Query Workbench in the Fusion UI. Lucidworks no longer updates, maintains, or releases V1 connectors. A benefit of using this remote V2 connector over the classic V1 connectors is this V2 connector is regularly improved and updated, while the classic connectors are not. The REST V2 connector is replacing V1 connectors using comparable recipes. You can learn more about the benefits of V2 connectors by reading the differences between V1 and V2 connectors.

​
Recipes

Recipes are preconfigured datasource configurations that can be loaded into Fusion for quicker testing and setup of connectors. Parameter values are already entered into recipes for quick population of the required fields. Recipes are created in a JSON format and contain all of the parameters required to get a connector up and running. You can configure a connector using a recipe by issuing a POST request with the JSON as the body. Any minor adjustments required can be made directly in the JSON and sent to Fusion, where it will show in the UI. After the recipe has been sent to Fusion, you can also go into the Fusion UI and make changes.

​
Recipe repository

The REST V2 connector uses a public GitHub repository to store recipes. Here you will find all of the recipes that have been created and ready for use. The recipes are JSON files that can be added as the body in Postman requests or appended as the body in cURL calls. The parameter values can then be changed as necessary to match the data in the way you want it to be indexed. The repository is public and treated as an open source resource where everyone is welcome to contribute. Recipes are not considered a standard Lucidworks product. As such, Lucidworks does not guarantee that a recipe will be maintained or updated after it is created. If you encounter any issues or bugs, please report them in GitHub with detailed information to aid in troubleshooting.

​
Remote connectors

V2 connectors support running remotely in Fusion versions 5.7.1 and later.
If you need to index data from behind a firewall, you can configure a V2 connector to run remotely on-premises using TLS-enabled gRPC.

​
Prerequisites

Before you can set up an on-prem V2 connector, you must configure the egress from your network to allow HTTP/2 communication into the Fusion cloud. You can use a forward proxy server to act as an intermediary between the connector and Fusion.The following is required to run V2 connectors remotely:
  • The plugin zip file and the connector-plugin-standalone JAR.
  • A configured connector backend gRPC endpoint.
  • Username and password of a user with a remote-connectors or admin role.
  • If the host where the remote connector is running is not configured to trust the server’s TLS certificate, you must configure the file path of the trust certificate collection.
If your version of Fusion doesn’t have the remote-connectors role by default, you can create one. No API or UI permissions are required for the role.

​
Connector compatibility

Only V2 connectors are able to run remotely on-premises. You also need the remote connector client JAR file that matches your Fusion version. You can download the latest files at V2 Connectors Downloads.
Whenever you upgrade Fusion, you must also update your remote connectors to match the new version of Fusion.
The gRPC connector backend is not supported in Fusion environments deployed on AWS.

​
System requirements

The following is required for the on-prem host of the remote connector:
  • (Fusion 5.9.0-5.9.10) JVM version 11
  • (Fusion 5.9.11) JVM version 17
  • Minimum of 2 CPUs
  • 4GB Memory
Note that memory requirements depend on the number and size of ingested documents.

​
Enable backend ingress

In your values.yaml file, configure this section as needed:
ingress:
  enabled: false
  pathtype: "Prefix"
  path: "/"
  #host: "ingress.example.com"
  ingressClassName: "nginx"   # Fusion 5.9.6 only
  tls:
    enabled: false
    certificateArn: ""
    # Enable the annotations field to override the default annotations
    #annotations: ""
  • Set enabled to true to enable the backend ingress.
  • Set pathtype to Prefix or Exact.
  • Set path to the path where the backend will be available.
  • Set host to the host where the backend will be available.
  • In Fusion 5.9.6 only, you can set ingressClassName to one of the following:
    • nginx for Nginx Ingress Controller
    • alb for AWS Application Load Balancer (ALB)
  • Configure TLS and certificates according to your CA’s procedures and policies.
    TLS must be enabled in order to use AWS ALB for ingress.

​
Connector configuration example

kafka-bridge:
  target: mynamespace-connectors-backend.lucidworkstest.com:443 # mandatory
  plain-text: false # optional, false by default.  
    proxy-server: # optional - needed when a forward proxy server is used to provide outbound access to the standalone connector
    host: host
    port: some-port
    user: user # optional
    password: password # optional
  trust: # optional - needed when the client's system doesn't trust the server's certificate
    cert-collection-filepath: path1

proxy: # mandatory fusion-proxy
  user: admin
  password: password123
  url: https://fusiontest.com/ # needed only when the connector plugin requires blob store access

plugin: # mandatory
  path: ./fs.zip
  type: #optional - the suffix is added to the connector id
    suffix: remote

​
Minimal example

kafka-bridge:
  target: mynamespace-connectors-backend.lucidworkstest.com:443

proxy:
  user: admin
  password: "password123"

plugin:
  path: ./testplugin.zip

​
Logback XML configuration file example

<configuration>
    <appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
        <encoder class="com.lucidworks.logging.logback.classic.LucidworksPatternLayoutEncoder">
            <pattern>%d - %-5p [%t:%C{3.}@%L] - %m{nolookups}%n</pattern>
            <charset>utf8</charset>
        </encoder>
    </appender>

    <appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <file>${LOGDIR:-.}/connector.log</file>
        <rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
            <!-- rollover daily -->
            <fileNamePattern>${LOGDIR:-.}/connector-%d{yyyy-MM-dd}.%i.log.gz</fileNamePattern>
            <maxFileSize>50MB</maxFileSize>
            <totalSizeCap>10GB</totalSizeCap>
        </rollingPolicy>
        <encoder class="com.lucidworks.logging.logback.classic.LucidworksPatternLayoutEncoder">
            <pattern>%d - %-5p [%t:%C{3.}@%L] - %m{nolookups}%n</pattern>
            <charset>utf8</charset>
        </encoder>
    </appender>

    <root level="INFO">
        <appender-ref ref="CONSOLE"/>
        <appender-ref ref="FILE"/>
    </root>
</configuration>

​
Run the remote connector

java [-Dlogging.config=[LOGBACK_XML_FILE]] \
  -jar connector-plugin-client-standalone.jar [YAML_CONFIG_FILE]
The logging.config property is optional. If not set, logging messages are sent to the console.

​
Test communication

You can run the connector in communication testing mode. This mode tests the communication with the backend without running the plugin, reports the result, and exits.
java -Dstandalone.connector.connectivity.test=true -jar connector-plugin-client-standalone.jar [YAML_CONFIG_FILE]

​
Encryption

In a deployment, communication to the connector’s backend server is encrypted using TLS. You should only run this configuration without TLS in a testing scenario. To disable TLS, set plain-text to true.

​
Egress and proxy server configuration

One of the methods you can use to allow outbound communication from behind a firewall is a proxy server. You can configure a proxy server to allow certain communication traffic while blocking unauthorized communication. If you use a proxy server at the site where the connector is running, you must configure the following properties:
  • Host. The hosts where the proxy server is running.
  • Port. The port the proxy server is listening to for communication requests.
  • Credentials. Optional proxy server user and password.
When you configure egress, it is important to disable any connection or activity timeouts because the connector uses long running gRPC calls.

​
Password encryption

If you use a login name and password in your configuration, run the following utility to encrypt the password:
  1. Enter a user name and password in the connector configuration YAML.
  2. Run the standalone JAR with this property: 
    -Dstandalone.connector.encrypt.password=true
  3. Retrieve the encrypted passwords from the log that is created.
  4. Replace the clear password in the configuration YAML with the encrypted password.

​
Connector restart (5.7 and earlier)

The connector will shut down automatically whenever the connection to the server is disrupted, to prevent it from getting into a bad state. Communication disruption can happen, for example, when the server running in the connectors-backend pod shuts down and is replaced by a new pod. Once the connector shuts down, connector configuration and job execution are disabled. To prevent that from happening, you should restart the connector as soon as possible.You can use Linux scripts and utilities to restart the connector automatically, such as Monit.

​
Recoverable bridge (5.8 and later)

If communication to the remote connector is disrupted, the connector will try to recover communication and gRPC calls. By default, six attempts will be made to recover each gRPC call. The number of attempts can be configured with the max-grpc-retries bridge parameters.

​
Job expiration duration (5.9.5 only)

The timeout value for irresponsive backend jobs can be configured with the job-expiration-duration-seconds parameter. The default value is 120 seconds.

​
Use the remote connector

Once the connector is running, it is available in the Datasources dropdown. If the standalone connector terminates, it disappears from the list of available connectors. Once it is re-run, it is available again and configured connector instances will not get lost.

​
Enable asynchronous parsing (5.9 and later)

To separate document crawling from document parsing, enable Tika Asynchronous Parsing on remote V2 connectors.
Below is an example configuration showing how to specify the file system to index under the connector-plugins entry in your values.yaml file: 
additionalVolumes:
- name: fusion-data1-pvc
    persistentVolumeClaim:
    claimName: fusion-data1-pvc
- name: fusion-data2-pvc
    persistentVolumeClaim:
    claimName: fusion-data2-pvc
additionalVolumeMounts:
- name: fusion-data1-pvc
    mountPath: "/connector/data1"
- name: fusion-data2-pvc
    mountPath: "/connector/data2"
You may also need to specify the user that is authorized to access the file system, as in this example: 
securityContext:
    fsGroup: 1002100000
    runAsUser: 1002100000

​
Learn more

LucidAcademyLucidworks offers free training to help you get started.The Microlearning for The REST V2 Connector focuses on how to use the REST V2 connector and its recipes:
The REST V2 ConnectorPlay Button
Visit the LucidAcademy to see the full training catalog.
LucidAcademyLucidworks offers free training to help you get started.The Quick Learning for Using the REST V2 Connector focuses on how to ingest a datasource using REST V2 connector recipes:
Using the REST V2 ConnectorPlay Button
Visit the LucidAcademy to see the full training catalog.

​
Configuration