Develop a Custom Connector

Table of Contents

Fusion 4.2.6
Java SDK configuration
- About com.lucidworks.schema
Plugin client
- Java SDK / gRPC Wrapper
- Standalone Connector Plugin Application
Java SDK security
- Java Plugin Client
Random Connector
Quick start

Java SDK configuration

To build a valid connector configuration, you must:

Define an interface.
Extend ConnectorConfig.
Apply a few annotations.
Define connector "getter" methods and annotations.

All methods that are annotated with @Property and start with "get" are considered to be configuration properties. For example, @Property() String getName(); results in a String property called name. This property would then be present in the generated schema.

Here is an example of the most basic configuration, along with required annotations and a sample "getter" method:

@RootSchema(
    name = "com.my.connector",
    title = "My Connector",
    description = "My Connector description",
    category = "My Category"
)
public interface MyConfig extends ConnectorConfig<MyConfig.Properties> {

  @Property(
      title = "Properties",
      required = true
  )
  public Properties getProperties();

  /**
   * Connector specific settings
   */
  interface Properties extends FetcherProperties {

    @Property(
        title = "My custom property",
        description = "My custom property description"
    )
    public Integer getMyCustomProperty();

  }

}

The metadata defined by @RootSchema is used by Fusion when showing the list of available connectors. The ConnectorConfig base interface represents common, top-level settings required by all connectors. The type parameter of the ConnectorConfig class indicates the interface to use for custom properties.

Once a connector configuration has been defined, it can be associated with the ConnectorPlugin class. From that point, the framework takes care of providing the configuration instances to your connector. It also generates the schema, and sends it along to Fusion when it connects to Fusion.

About com.lucidworks.schema

The com.lucidworks.schema project (included in the connectors SDK) aims to simplify the process of creating JSON Schemas. It also provides utilities for building type-safe POJOs from schema definitions.

The basic idea here is that you create an interface that extends Model, add property getters, and then add a few annotations. Once your interface has been built, you can generate an ObjectType instance, which is the object that contains all of the JSON schema metadata. By then combining that schema object with a Map<String,Object>, you can create instances of your interface to use as configuration objects.

The configuration objects are based on java.lang.reflect.Proxy, which is what the ModelGenerator returns. Most method calls to these proxy instances result in method calls to the underlying Map<String,Object>. For example, if the interface defines a getId method, then a runtime call results in a call to the Map<String,Object>: data.get("id").

A few special cases exist:

toString is proxied directly to the Map<String,Object>
getClass returns the class of the interface provided to ModelGenerator#generate()
_data returns the underlying Map<String,Object> object
If the method starts with set, the arguments are captured and sent to the underlying map via put()

Here is a simple example:

interface MyConfig extends Model {

  @Property
  @StringSchema(minLength=1)
  String getId();

}

Create the schema:

class Runner {
  public static void main(String[] args){
    ObjectType schema = SchemaGenerator.generate(MyConfig.class);
  }
}

Generate an instance of the MyConfig class:

class Runner {
  public static void main(String[] args){
    Map<String,Object> data = new java.util.HashMap<>();
    data.put("id", 100);
    MyConfig config = ModelGenerator.generate(MyConfig.class, data);
    System.out.println("id -> " + config.getId());
  }
}

Schema metadata can be applied to properties using additional annotations. For example, applying limits to the min/max length of a string, or describing the types of items in an array.

Nested schema metadata can also be applied to a single field by using "stacked" schema based annotations:

interface MySetConfig extends Model {
    @SchemaAnnotations.Property(title = "My Set")
    @SchemaAnnotations.ArraySchema(defaultValue = "[\"a\"]")
    @SchemaAnnotations.StringSchema(defaultValue = "some-set-value", minLength = 1, maxLength = 1)
    Set<String> getMySet();
  }

Plugin client

The Fusion connector plugin client provides a wrapper for the Fusion Java plugin-sdk so that plugins do not need to directly talk with gRPC code. Instead, they can use high-level interfaces and base classes, like Connector and Fetcher.

The plugin client also provides a standalone "runner" that can host a plugin that was built from the Fusion Java Connector SDK. It does this by loading the plugin zip file, then calling on the wrapper to provide the gRPC interactions.

Java SDK / gRPC Wrapper

One of the primary goals of the plugin-client is to isolate plugin code from the underlying framework details. Specifically, the underlying message formats (protocol buffers) and native gRPC code. This makes it possible to make some changes to the base support layer, without having to make changes to the Java plugin implementation.

Standalone Connector Plugin Application

The second goal of the plugin-client is to allow Java SDK plugins to run remotely. The instructions for deploying a connector using this method are provided below.

Locating the UberJar

The uberjar is located in this location in the Fusion file system:

$FUSION_HOME/apps/connectors/connectors-rpc/client/connector-plugin-client-<version>-uberjar.jar

where $FUSION_HOME is your Fusion installation directory and <version> is your Fusion version number.

Starting the Host

To start the host app, you need a Fusion SDK-based connector, built into the standard packaging format as a .zip file. This zip must contain only one connector plugin.

Here is an example of how to start up using the web connector:

java -jar $FUSION_HOME/apps/connectors/connectors-rpc/client/connector-plugin-client-<version>-uberjar.jar fusion-connectors/build/plugins/connector-web-4.0.0-SNAPSHOT.zip

To run the client with remote debugging enabled:

java -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5010 -jar $FUSION_HOME/apps/connectors/connectors-rpc/client/connector-plugin-client-<version>-uberjar.jar fusion-connectors/build/plugins/connector-web-4.0.0-SNAPSHOT.zip

Java SDK security

Fusion connectors support SSL/TLS.

Java Plugin Client

The information here is specific to running a Java SDK plugin outside of Fusion.

The Fusion Connector Plugin Client supports several variations of SSL/TLS auth. The examples below show the relevant Java properties.

Example with Mutual TLS auth and private key passwords

-Dcom.lucidworks.apollo.app.hostname=myhost
-Dcom.lucidworks.fusion.tls.trustCertCollection=./sslcerts/ca.crt
-Dcom.lucidworks.fusion.tls.client.certChain=./sslcerts/client.crt
-Dcom.lucidworks.fusion.tls.client.privateKey=./sslcerts/client.pem
-Dcom.lucidworks.fusion.tls.client.privateKeyPassword=password123
-Dcom.lucidworks.fusion.tls.enabled=true</td>

Example without TLS auth and no private key passwords:

-Dcom.lucidworks.apollo.app.hostname=myhost
-Dcom.lucidworks.fusion.tls.trustCertCollection=./sslcerts/ca.crt
-Dcom.lucidworks.fusion.tls.enabled=true

Random Connector

This connector generates a configurable number of documents, all with random titles and body fields.

Quick start

This quick start assumes that Fusion is installed on the /opt path.

Clone the repo:

git clone https://github.com/lucidworks/connectors-sdk-resources.git
cd connectors-sdk-resources/java-sdk/connectors/
./gradlew assemblePlugins

This produces one zip file, named random-connector.zip, located in the build/plugins directory. This artifact is now ready to be uploaded directly to Fusion as a connector plugin.
See the following instructions on how to build, deploy, and run the plugin.