Product Selector

Fusion 5.9
    Fusion 5.9

    JDBC V1Connector Configuration Reference

    Table of Contents

    The JDBC connector fetches documents from a relational database via SQL queries. Under the hood, this connector implements the Solr DataImportHandler (DIH) plugin.

    Deprecation and removal notice

    This connector is deprecated as of Fusion 5.3 and is removed or expected to be removed as of Fusion 5.4. Use the JDBC V2 connector instead.

    For more information about deprecations and removals, including possible alternatives, see Deprecations and Removals.

    Date field types

    Observe the date field type in the source database. Delta queries compare a date in the source database to the time of the most recent data import. The user selects a date field from their database to use for comparison (for example, a field like last_updated). The field type changes the behavior of the comparison:

    • DATE: It compares the day (for example, 2020-02-20)

    • TIMESTAMP: It compares both the day and time (for example, 2020-02-20 17:45:45)

    Note: When using a primary key other than id, remove UUIDUpdateProcessorFactory from solrconfig.xml. Also change the uniqueKey value in schema.xml to your primary key.

    Configuration

    When entering configuration values in the UI, use unescaped characters, such as \t for the tab character. When entering configuration values in the API, use escaped characters, such as \\t for the tab character.

    Connector for any JDBC database. This connector requires a JDBC driver to be loaded before creating the datasource.

    id - stringrequired

    Unique name for this datasource.

    >= 1 characters

    Match pattern: ^[a-zA-Z0-9_-]+$

    pipeline - stringrequired

    Name of an existing index pipeline for processing documents.

    >= 1 characters

    description - string

    Optional description for this datasource.

    properties - Properties

    Datasource configuration properties

    db - Connector DB

    Type and properties for a ConnectorDB implementation to use with this datasource.

    type - string

    Fully qualified class name of ConnectorDb implementation.

    >= 1 characters

    Default: com.lucidworks.connectors.db.impl.MapDbConnectorDb

    inlinks - boolean

    Keep track of incoming links. This negatively impacts performance and size of DB.

    Default: false

    aliases - boolean

    Keep track of original URI-s that resolved to the current URI. This negatively impacts performance and size of DB.

    Default: false

    inv_aliases - boolean

    Keep track of target URI-s that the current URI resolves to. This negatively impacts performance and size of DB.

    Default: false

    url - string

    A URL to the database, starting with jdbc and vendor name, e.g., 'jdbc:mysql://localhost/test'

    >= 1 characters

    driver - string

    The class name of the JDBC driver to use to connect to the database. Only JDBC4 drivers will appear in this list. If a JDBC3 driver has been uploaded but does not appear in the list, it may be possible to manually enter the class name.

    manually_uploaded_driver - string

    If you for whatever reason cannot use the blob store to upload your JDBC driver and needed to load it manually into the connectors apps/libs directory, you can specify the JDBC Driver class name here.

    username - string

    The username of a database account used for authentication and data access.

    password - string

    The password of the account used for authentication and data access.

    sql_select_statement - string

    A SQL SELECT statement to choose the records to be retrieved.

    primary_key - string

    The column name of the primary key for the table.

    nested_queries - array[string]

    A nested query to join data from multiple tables. The nested query will be used with the SQL Statement and must include the primary key with the $ character. For example, 'SELECT tag FROM tag WHERE document_tag.doc_id=$'.

    clean_in_full_import_mode - boolean

    Clean old records when doing a full import. This will remove all records from the index before re-indexing.

    Default: true

    convert_type - boolean

    If checked, the connector will automatically convert the data from a column to the target Solr data-type. Otherwise, the connector will use the field type based on the data type of the column from the JDBC result set.

    Default: true

    fetch_size - integer

    The number of documents to retrieve per batch.

    max_docs - integer

    The maximum number of documents to crawl. Use -1 to index all documents found.

    Default: -1

    delta_sql_query - string

    A SQL statement to select the delta (records added or changed) since the last run of the datasource. For example, 'SELECT id FROM customers WHERE last_modified > $'. The $ indicates the date and time of the last import.

    delta_import_query - string

    A SQL statement to retrieve the delta records. If not defined, the original SQL Statement will be used to retrieve delta records, which may be error prone in the case of complex statements. If that occurs, you can define a secondary SQL statement that will only be used for delta imports.

    kerberosAuthenticationProps - Kerberos Authentication

    Kerberos Authentication Properties

    kerberosServerPrincipal - string

    Kerberos Server Principal

    commit_on_finish - boolean

    Set to true for a request to be sent to Solr after the last batch has been fetched to commit the documents to the index.

    Default: true

    verify_access - boolean

    Set to true to require successful connection to the filesystem before saving this datasource.

    Default: true

    initial_mapping - Initial field mapping

    Provides mapping of fields before documents are sent to an index pipeline.

    skip - boolean

    Set to true to skip this stage.

    Default: false

    label - string

    A unique label for this stage.

    <= 255 characters

    condition - string

    Define a conditional script that must result in true or false. This can be used to determine if the stage should process or not.

    reservedFieldsMappingAllowed - boolean

    Default: false

    retentionMappings - array[object]

    Fields that should be kept or deleted

    object attributes:{field required : {
     display name: Field
     type: string
    }
    operation : {
     display name: Operation
     type: string
    }
    }

    updateMappings - array[object]

    Values that should be added to or set on a field. When a value is added, any values previously on the field will be retained. When a value is set, any values previously on the field will be overwritten.

    object attributes:{field required : {
     display name: Field
     type: string
    }
    value required : {
     display name: Value
     type: string
    }
    operation : {
     display name: Operation
     type: string
    }
    }

    translationMappings - array[object]

    Fields that should be moved or copied to another field. When a field is moved, the values from the source field are moved over to the target field and the source field is removed. When a field is copied, the values from the source field are copied over to the target field and the source field is retained.

    object attributes:{source required : {
     display name: Source Field
     type: string
    }
    target required : {
     display name: Target Field
     type: string
    }
    operation : {
     display name: Operation
     type: string
    }
    }

    unmappedRule - Unmapped Fields

    Fields not mapped by the above rules. By default, any remaining fields will be kept on the document.

    keep - boolean

    Keep all unmapped fields

    Default: true

    delete - boolean

    Delete all unmapped fields

    Default: false

    fieldToMoveValuesTo - string

    Move all unmapped field values to this field

    fieldToCopyValuesTo - string

    Copy all unmapped field values to this field

    valueToAddToUnmappedFields - string

    Add this value to all unmapped fields

    valueToSetOnUnmappedFields - string

    Set this value on all unmapped fields