MongoDB Datasource and Connector Configuration

Retrieve data from a MongoDB instance.

This is an add-on connector that you can download and install.

At the first connection, the Fusion MongoDB connector crawls the entire MongoDB and saves the checkpoint.

If Process oplog is not selected, when you restart the datasource, the connector recrawls the entire MongoDB. In this mode the connector does not support incremental recrawling, nor does it delete entries that are deleted from MongoDB.

About reading from the MongoDB oplog

You can configure the Fusion MongoDB connector to read from the MongoDB oplog rather than from the entire MongoDB collection. In this mode, the connector crawls the full MongoDB collection, saves a checkpoint in ZooKeeper, then continues running indefinitely, grabbing updates from the oplog as they happen in real time. This way the connector can delete documents that are deleted from MongoDB.

If the connector stops for any reason, it stores a timestamp in ZooKeeper that shows what the latest update was. When the connector restarts, it continues reading from that checkpoint onward.

To read from the MongoDB oplog:

  1. Make sure your connector authenticates to Mongo as a user with read oplog permissions. See Role-based access control in the MongoDB documentation .

  2. Make sure Process Oplog is selected in the Fusion MongoDB connector UI.

Configuration

Tip
When entering configuration values in the UI, use unescaped characters, such as \t for the tab character. When entering configuration values in the API, use escaped characters, such as \\t for the tab character.