Connectors

Connectors are the built-in mechanism for pulling your data into Fusion. Lucidworks provides a wide variety of connectors, each specialized for a particular data type. When you add a datasource to a collection, you specify the connector to use for ingesting data. See the complete list of connectors, with links to configuration reference information for each one.

Connector architecture

As of Fusion Server 4.0, connector plugins can be hosted within Fusion, or can run remotely. The communication of messages between Fusion and a remote Connector or hosted Connector are identical; Fusion sees them as the same kind of Connector. This means you can implement a plugin locally, connect to a remote Fusion for initial testing, and when done, upload the same artifact into Fusion, so Fusion can host it for you.

The connectors architecture was designed to be scalable. Depending on the connector, jobs can now be scaled by adding new instances of the connector. The fetching process for these new types also supports distributed fetching, so that many instances can contribute to the same job.

Hosted connectors

In the hosted case, connectors are cluster aware. This means that when a new instance of Fusion starts up, the connectors on other Fusion nodes become aware of the new connectors, and vice versa. This makes scaling the crawling process very natural and simple.

Remote connectors

SDK connectors can be hosted within Fusion Server, or can run remotely. In the remote case, connectors become clients of Fusion. These clients run a very lightweight process and communicate to Fusion using a very efficient messaging format. This option makes it possible to put the connector wherever the data lives. This may be done for performance reasons, or for security/access reasons. See Remote Connectors for more details.

Built-in connectors

Fusion comes with a standard set of built-in connectors:

Additional connectors are available for download at http://lucidworks.com/connectors/. You can look in fusion/4.1.x/apps/connectors/connectors-classic/plugins/ and fusion/4.1.x/apps/connectors/connectors-rpc/plugins/ to see which additional connectors are currently installed.

Connector logs

You can find connector logs in fusion/4.1.x/var/log/connectors.