Product Selector

Fusion 5.11
    Fusion 5.11

    MongoDB V1 Connector Configuration Reference

    Table of Contents

    Retrieve data from a MongoDB instance.

    At the first connection, the Fusion MongoDB connector crawls the entire MongoDB and saves the checkpoint.

    If Process oplog is not selected, when you restart the data source, the connector recrawls the entire MongoDB. In this mode the connector does not support incremental recrawling, nor does it delete entries that are deleted from MongoDB.

    Configuration

    When entering configuration values in the UI, use unescaped characters, such as \t for the tab character. When entering configuration values in the API, use escaped characters, such as \\t for the tab character.

    Crawl a MongoDB database. For recrawls, the crawler can use the oplog in MongoDB to discover new content and updates to existing content (updated or removed documents). If a full re-synchronization is required, it can be done by de-selecting the oplog option and starting the crawl again.

    id - stringrequired

    Unique name for this datasource.

    >= 1 characters

    Match pattern: ^[a-zA-Z0-9_-]+$

    pipeline - stringrequired

    Name of an existing index pipeline for processing documents.

    >= 1 characters

    description - string

    Optional description for this datasource.

    properties - Properties

    Datasource configuration properties

    db - Connector DB

    Type and properties for a ConnectorDB implementation to use with this datasource.

    type - string

    Fully qualified class name of ConnectorDb implementation.

    >= 1 characters

    Default: com.lucidworks.connectors.db.impl.MapDbConnectorDb

    inlinks - boolean

    Keep track of incoming links. This negatively impacts performance and size of DB.

    Default: false

    aliases - boolean

    Keep track of original URI-s that resolved to the current URI. This negatively impacts performance and size of DB.

    Default: false

    inv_aliases - boolean

    Keep track of target URI-s that the current URI resolves to. This negatively impacts performance and size of DB.

    Default: false

    list_hosts - array[object]

    Host and ports of Mongo nodes

    Default: {"host":"localhost","port":27017}

    object attributes:{host : {
     display name: Host
     type: string
    }
    port : {
     display name: Port
     type: integer
    }
    }

    list_credentials - array[object]

    Credentials for Mongo databases

    object attributes:{database : {
     display name: Database
     type: string
    }
    username : {
     display name: Username
     type: string
    }
    password : {
     display name: Password
     type: string
    }
    id : {
     display name: Auth Config id
     type: string
    }
    }

    collections - string

    The MongoDB collections to index, in the format 'databaseName.collection'. Multiple collections can be separated by commas. The default '*.*' option crawls all databases (limited by user access) and their related collections.

    >= 1 characters

    Default: *.*

    process_oplog - boolean

    Process updates from the oplog. Disable this option to perform a full synchronization of content in MongoDB collections with the index.

    Default: true

    diagnosticMode - boolean

    Diagnostic mode enables more logging, including logging the ID of every document inserted, updated or deleted in the oplog.

    Default: false

    batch_size_solr_commit - integer

    The number of documents every time solr_commit will be made.

    Default: 1000

    enable_ssl - boolean

    When enabled, SSL connections will be used to communicate with the MongoDB server

    Default: false

    customized_timestamp - integer

    Customized timestamp in epoch format (e.g. 1557881001), it is used to overwrite the existing checkpoint in zookeeper, use it carefully. The checkpoint is overwritten as long as the oplog is enabled. This property is transient, it means: if you set a value and add/update the datasoure, after the checkpoint is replaced, this property will be removed; you must refresh the UI manually

    exclusiveMinimum: false

    oplog_listener_period_time - integer

    Period time in seconds when the checkpoint is updated in zookeeper. This option will work if oplog is enabled

    Default: 60

    read_preferences - string

    Read preference describes how MongoDB clients route read operations to the members of a replica set.

    Default: primary

    Allowed values: primaryprimary preferredsecondarysecondary preferrednearest

    tag_set_list - array[object]

    A list of Tag Sets used for non-primary read modes

    Default:

    object attributes:{tag_set : {
     display name: Tag Set
     type: array
    }
    }

    commit_on_finish - boolean

    Set to true for a request to be sent to Solr after the last batch has been fetched to commit the documents to the index.

    Default: true

    verify_access - boolean

    Set to true to require successful connection to the filesystem before saving this datasource.

    Default: true

    initial_mapping - Initial field mapping

    Provides mapping of fields before documents are sent to an index pipeline.

    skip - boolean

    Set to true to skip this stage.

    Default: false

    label - string

    A unique label for this stage.

    <= 255 characters

    condition - string

    Define a conditional script that must result in true or false. This can be used to determine if the stage should process or not.

    reservedFieldsMappingAllowed - boolean

    Default: false

    retentionMappings - array[object]

    Fields that should be kept or deleted

    object attributes:{field required : {
     display name: Field
     type: string
    }
    operation : {
     display name: Operation
     type: string
    }
    }

    updateMappings - array[object]

    Values that should be added to or set on a field. When a value is added, any values previously on the field will be retained. When a value is set, any values previously on the field will be overwritten.

    object attributes:{field required : {
     display name: Field
     type: string
    }
    value required : {
     display name: Value
     type: string
    }
    operation : {
     display name: Operation
     type: string
    }
    }

    translationMappings - array[object]

    Fields that should be moved or copied to another field. When a field is moved, the values from the source field are moved over to the target field and the source field is removed. When a field is copied, the values from the source field are copied over to the target field and the source field is retained.

    object attributes:{source required : {
     display name: Source Field
     type: string
    }
    target required : {
     display name: Target Field
     type: string
    }
    operation : {
     display name: Operation
     type: string
    }
    }

    unmappedRule - Unmapped Fields

    Fields not mapped by the above rules. By default, any remaining fields will be kept on the document.

    keep - boolean

    Keep all unmapped fields

    Default: true

    delete - boolean

    Delete all unmapped fields

    Default: false

    fieldToMoveValuesTo - string

    Move all unmapped field values to this field

    fieldToCopyValuesTo - string

    Copy all unmapped field values to this field

    valueToAddToUnmappedFields - string

    Add this value to all unmapped fields

    valueToSetOnUnmappedFields - string

    Set this value on all unmapped fields