Connectors

    Connectors are the built-in mechanism for pulling your data into Fusion. Fusion comes with a wide variety of connectors, each specialized for a particular data type. When you add a datasource to a collection, you specify the connector to use for ingesting data. See the list of connectors for a complete list of connectors, with links to configuration reference information for each one.

    Fusion comes with a standard set of built-in connectors:

    • Local Filesystem connector

    • File Upload connector

    • JDBC connector

    • Web connector

    Built-in connectors are in fusion/3.1.x/apps/connectors/bootstrap-plugins/.

    Additional connectors are available for download at http://lucidworks.com/connectors/. You can look in fusion/3.1.x/apps/connectors/plugins/ to see which additional connectors are currently installed.

    Installing a connector

    Connectors are installed by uploading them to the blob store. You can install connectors:

    • By installing connectors as "bootstrap plugins", that is, by putting them in the bootstrap-plugins directory during initial installation or an upgrade

    • By using the Fusion UI after installation or an upgrade

    • By using the Blob Store API after installation or an upgrade.

    Note
    		 During upgrades, the migrator handles some aspects of installing connectors. Depending on the target version and the presence or absence of an Internet connection, there might be manual steps. Installing connectors during upgrades is explained where needed in the upgrade procedures.

    Installing a connector as a bootstrap plugin

    Fusion can install connectors as "bootstrap plugins." All this means is that you put the connector zip files in a specific directory named bootstrap-plugins, and Fusion installs the connectors the first time it starts during initial installation or an upgrade.

    How to install a connector as a bootstrap plugin

    1. Download the connector zip file from http://lucidworks.com/connectors/.

      Don’t expand the archive; Fusion consumes it as-is. Also, don’t start Fusion until instructed to do so by the installation or upgrade instructions.

    2. Under the version-numbered Fusion directory, place the connector in the directory apps/connectors/bootstrap-plugins/ (on Unix) or \apps\connectors\bootstrap-plugins\ (on Windows).

    3. Start Fusion when instructed to do so in the installation or upgrade procedure.

    Installing a connector using the Fusion UI

    1. Download the connector zip file from http://lucidworks.com/connectors/.

      Do not expand the archive; Fusion consumes it as-is.

    2. In the Fusion UI, navigate to DevOps > Blobs.

    3. Click Add.

    4. Select Connector Plugin.

      Uploading a connector

      The "New 'Connector Plugin' Upload" panel appears.

    5. Click Choose File and select the downloaded zip file from your file system.

      Uploaded connector

    6. Click Upload.

      The new connector’s blob manifest appears.

      Uploaded connector

      From this screen you can also delete or replace the connector.

    Installing a connector using the Blob Store API

    1. Download the connector zip file from http://lucidworks.com/connectors/.

      Do not expand the archive; Fusion consumes it as-is.

    2. Upload the connector zip file to Fusion’s blob store.

      Specify an arbitrary blob ID, and a resourceType value of plugin:connector, as in this example:

      curl -H 'content-type:application/zip' -X PUT 'localhost:8764/api/blobs/myplugin?resourceType=plugin:connector' --data-binary @myplugin.zip

      Fusion automatically publishes the event to the cluster, and the listeners perform the connector installation process on each node.

      Tip
      		 If the blob ID is identical to an existing one, the old connector will be uninstalled and the new connector will installed in its place. To get the list of existing blob IDs, run: curl -u user:pass localhost:8764/api/apollo/blobs

    3. Look in fusion/3.1.x/apps/connectors/plugins/ to verify that the new connector is installed.

    Updating a connector

    Note
    		 If you are updating the Jive connector from Fusion 2.0 or earlier, see Jive Connector and Datasource Configuration in the datasources and connectors reference.

    On Unix, you can update a connector by simply uploading the new one. Fusion overwrites the old one, and no restart is needed.

    On Windows, a different procedure is needed:

    How to update a Fusion connector on Windows

    1. Delete the old connector, as explained below.

    2. Restart Fusion.

    3. Upload the new connector.

    Deleting a connector

    You can delete a connector using the Fusion UI or the Blob Store API.

    Deleting a connector using the Fusion UI

    1. In the Fusion UI, navigate to DevOps > Blobs.

    2. Under Connector Plugin, select the connector to delete.

    3. Click Delete Blob.

      Delete a connector

      Fusion prompts you to confirm that you want to delete the blob.

    4. Click Yes, Delete.

      The connector disappears from the blob list.

    Deleting a connector using the REST API

    1. Get the list of blobs of the connector plugin type:

      curl -u user:pass http://localhost:8764/api/apollo/blobs?resouType=plugin:connector

    2. Locate the connector you want to delete, and copy its ID.

      For example, the Jive connector ID is lucid.jive:

    {
  "name" : "lucid.jive",
  "contentType" : "application/zip",
  "size" : 125302,
  "modifiedTime" : "2017-06-13T17:49:20.171Z",
  "version" : 1570112704530612224,
  "md5" : "7032bf2c038bb2d1e27aee82c056c0fb",
  "metadata" : {
    "connectorBootstrapPluginName" : "lucid.jive",
    "resourceType" : "plugin:connector"
  }
}

    1. Delete the connector as follows:

      curl -u user:pass -X DELETE http://localhost:8764/api/apollo/blobs/<id>

      For example

      curl -u user:pass -X DELETE http://localhost:8764/api/apollo/blobs/lucid.jive

      A null response indicates success. You can verify that the connector is deleted like this:

      curl -u user:pass http://localhost:8764/api/apollo/blobs | grep lucid.jive

    Connector configuration

    When you add a datasource to a collection, you select a connector and configure it. There are two ways to do this:

    • Using the API

    • Using the UI

    Configuring Connectors Using The API

    You can create or update a datasource with the Connector Datasources API, specifying the connector, its properties, and their values.

    Example: Create and configure a datasource to index Solr-formatted XML files 
    curl -u user:pass -X POST -H 'Content-type: application/json' -d '{"id":"SolrXML", "connector":"lucid.solrxml", "type":"solrxml", "properties":{"path":"/Applications/solr-4.10.2/example/exampledocs", "generate_unique_key":false, "collection":"MyCollection"}}' http://localhost:8764/api/apollo/connectors/datasources

    See the Connectors and Datasources Reference for details about configuration options.

    Tip
    		 Be sure the include the collection property; otherwise the datasource will not be available in the Fusion UI.
    Example: Change the max_docs value for the above datasource 
    curl -u user:pass -X PUT -H 'Content-type: application/json' -d '{"id":"SolrXML", "connector":"lucid.solrxml", "type":"solrxml", "properties":{"path":"/Applications/solr-4.10.2/example/exampledocs", "max_docs":10}}' http://localhost:8764/api/apollo/connectors/datasources/SolrXML

    Configuring Connectors Using The Fusion UI

    • To create and configure a new datasource and its connector:

      1. Click Applications > Collections.

      2. Select a Collection, or click Add a Collection to create a new one.

      3. Click Add a Datasource.

      4. Select a datasource type; these correspond to Fusion’s Connectors.

      5. Edit the configuration fields in the datasource panel that appears.

        See the Connectors and Datasources Reference for details about configuration options.

      6. Click Save.

    • To change the connector configuration for an existing datasource:

      1. Navigate to Applications > Collections.

      2. Click your collection name.

      3. Click the datasource you want to change.

      4. Edit the configuration fields as needed.

      5. Click Save.

    Connector logs

    You can find connector logs in fusion/3.1.x/var/log/connectors.