Connectors Configuration Reference

Fusion comes with a standard set of built-in connectors:

  • Local Filesystem connector

  • File Upload connector

  • JDBC connector

  • Web connector

Built-in connectors are in fusion/3.1.x/apps/connectors/bootstrap-plugins/.

Additional connectors are available for download at http://lucidworks.com/connectors/. You can look in fusion/3.1.x/apps/connectors/plugins/ to see which additional connectors are currently installed.

List of connectors

Database connectors

Couchbase

The Couchbase connector uses the Cross-Datacenter Replication (XDCR) feature of Couchbase to retrieve data stored in Couchbase continuously in real-time.

Download

JDBC

The JDBC connector fetches documents from a relational database via SQL queries. Under the hood, this connector implements the Solr DataImportHandler (DIH) plugin.

Built-in

MongoDb

Retrieve data from a MongoDB instance.

Download

Filesystem connectors

Box.com

The Box connector retrieves data from a Box.com cloud-based data repository. To fetch content from multiple Box users, you must create a Box app that uses OAuth 2.0 with JWT server authentication. For limited testing using a single user account, you can create a Box app that uses Standard OAuth 2.0 authentication.

Download

Dropbox

The Dropbox connector retrieves data from a Dropbox cloud-based data repository.

Download

File Upload

The File Upload connector provides a convenient way to quickly ingest data from your local filesystem. It’s available in the Quickstart interface in addition to the Index Workbench and the Datasources page.

Built-in

FTP

Retrieve documents using the File Transfer Protocol (FTP).

Download

Google Drive

The Google Drive connector is used to index the documents in a Google Drive account.

Download

HDFS

Hadoop Distributed File System (HDFS). It traverses the Hadoop file system as it would a regular Unix filesystem.

Download

Local Filesystem

A filesystem-based data store is a network of nodes to be traversed, where each node (such as a Unix file directory) provides information about its child nodes (such as the files in that directory) or references other nodes (such as links in an HTML document).

Built-in

S3

The S3 connector can access AWS S3 buckets in native format.

Download

SolrXML

The SolrXML connector indexes XML files formatted according to Solr’s XML structure. It is not a generic XML file crawler; it can only index SolrXML-formatted documents.

Download

Windows Share

The Windows Share connector can access content in a Windows Share or Server Message Block (SMB)/Common Internet File System (CIFS) filesystem.

Download

Hadoop cluster connectors

Hadoop

The Hadoop Connector is a MapReduce-enabled crawler which leverages the scaling qualities of Apache Hadoop.

Download

Push content connectors

Solr Push Endpoint

The Solr Push Endpoint accepts documents and pushes them to Solr using the Fusion index pipelines.

Download

Repository connectors

Alfresco

The Alfresco Connector is a crawler for the Alfresco server, which adheres to the Content Management Interoperability Services (CMIS) standard.

Download

Azure

The Azure connector is used to crawl an Azure instance. Its connector type is "lucid.azure" and its plugin type is "azure".

Download

Drupal

This connector uses Drupal’s Services 7.x­3.11Module REST API.

Download

GitHub

The GitHub connector retrieves data from GitHub repositories using the GitHub REST API.

Download

JIRA

The JIRA connector retrieves data from a instance of Atlassian’s JIRA issue tracking system.

Download

Salesforce

Salesforce REST API to extract data from a Salesforce repository via a Salesforce Connected App.

Download

ServiceNow

The ServiceNow Datasource retrieves data from ServiceNow repository via the ServiceNow REST API. ServiceNow records are stored in named tables.

Download

SharePoint

The SharePoint connector retrieves content and metadata from an on-premises SharePoint repository.

Download

SharePoint Online

The SharePoint Online connector retrieves data from cloud-based SharePoint repositories. Authentication requires a Sharepoint user who has permissions to access Sharepoint via the SOAP API. This user must be registered with the Sharepoint Online authentication server; it is not necessarily the same as the user in Active Directory or LDAP.

Download

Solr Index

A Solr connector pulls documents from an external standalone Solr instance or SolrCloud cluster using Solr’s javabin response type and streaming response parser.

Download

Subversion

This connector requires a Subversion client that is compatible which javahl.

Download

Zendesk

The Zendesk connector uses the Zendesk REST API to retrieve tickets and their associated comments and attachments from a Zendesk repository.

Download

Script connectors

Javascript

The Javascript connector allows users to write ad-hoc document retrieval routines to fetch content from filesystems and websites.

Download

Social media connectors

Jive

Retrieve content from a Jive instance.

Download

Web connectors

Web

The Web connector is used to retrieve data from a Web site using HTTP and starting from a specified URL.

Built-in

Installing a connector

Connectors are installed by uploading them to the blob store. You can do this using the Fusion UI or the Blob Store API. Both methods are explained below.

Installing a connector using the Fusion UI

  1. Download the connector zip file from http://lucidworks.com/connectors/.

    Do not expand the archive; Fusion consumes it as-is.

  2. In the Fusion UI, navigate to DevOps > Blobs.

  3. Click Add.

  4. Select Connector Plugin.

    Uploading a connector

    The "New 'Connector Plugin' Upload" panel appears.

  5. Click Choose File and select the downloaded zip file from your file system.

    Uploaded connector

  6. Click Upload.

    The new connector’s blob manifest appears.

    Uploaded connector

    From this screen you can also delete or replace the connector.

Installing a connector using the API

  1. Download the connector zip file from http://lucidworks.com/connectors/.

    Do not expand the archive; Fusion consumes it as-is.

  2. Upload the connector zip file to Fusion’s blob store.

    Specify an arbitrary blob ID, and a resourceType value of plugin:connector, as in this example:

    curl -H 'content-type:application/zip' -X PUT 'localhost:8765/api/v1/blobs/myplugin?resourceType=plugin:connector' —data-binary @myplugin.zip

    Fusion automatically publishes the event to the cluster, and the listeners perform the connector installation process on each node.

    Tip
    If the blob ID is identical to an existing one, the old connector will be uninstalled and the new connector will installed in its place. To get the list of existing blob IDs, run: curl -u user:pass localhost:8764/api/apollo/blobs
  3. Look in fusion/3.1.x/apps/connectors/plugins/ to verify that the new connector is installed.

Updating a connector

On Unix, you can update a connector by simply uploading the new one. Fusion overwrites the old one, and no restart is needed.

On Windows, a different procedure is needed:

How to update a Fusion connector on Windows
  1. Delete the old connector, as explained below.

  2. Restart Fusion.

  3. Upload the new connector.

Deleting a connector

You can delete a connector using the Fusion UI or the Blob Store API.

Deleting a connector using the Fusion UI

  1. In the Fusion UI, navigate to DevOps > Blobs.

  2. Under Connector Plugin, select the connector to delete.

  3. Click Delete Blob.

    Delete a connector

    Fusion prompts you to confirm that you want to delete the blob.

  4. Click Yes, Delete.

    The connector disappears from the blob list.

Deleting a connector using the REST API

  1. Get the list of blobs of the connector plugin type:

    curl -u user:pass http://localhost:8764/api/apollo/blobs?resouType=plugin:connector
  2. Locate the connector you want to delete, and copy its ID.

    For example, the Jive connector ID is lucid.jive:

{
  "name" : "lucid.jive",
  "contentType" : "application/zip",
  "size" : 125302,
  "modifiedTime" : "2017-06-13T17:49:20.171Z",
  "version" : 1570112704530612224,
  "md5" : "7032bf2c038bb2d1e27aee82c056c0fb",
  "metadata" : {
    "connectorBootstrapPluginName" : "lucid.jive",
    "resourceType" : "plugin:connector"
  }
}
  1. Delete the connector as follows:

    curl -u user:pass -X DELETE http://localhost:8764/api/apollo/blobs/<id>

    For example

    curl -u user:pass -X DELETE http://localhost:8764/api/apollo/blobs/lucid.jive

    A null response indicates success. You can verify that the connector is deleted like this:

    curl -u user:pass http://localhost:8764/api/apollo/blobs | grep lucid.jive