Product Selector

Fusion 5.9
    Fusion 5.9

    Solr V1Connector Configuration Reference

    Table of Contents

    A Solr connector pulls documents from an external standalone Solr instance or SolrCloud cluster using Solr’s javabin response type and streaming response parser.

    V1 deprecation and removal notice

    Starting in Fusion 5.12.0, all V1 connectors are deprecated. This means they are no longer being actively developed and will be removed in Fusion 5.13.0.

    The replacement for this connector is in active development at this time and will be released at a future date.

    If you are using this connector, you must migrate to the replacement connector or a supported alternative before upgrading to Fusion 5.13.0. We recommend migrating to the replacement connector as soon as possible to avoid any disruption to your workflows.

    For Solr v4.7 and greater, cursorMark deep-paging is used. For earlier versions of Solr, standard paging (start+rows) is used.

    The following Solr components and parameters can be configured:

    • collection/core (also allows default/empty core)

    • query (*:* by default)

    • filter queries

    • query parser

    • request handler (defaults to /select)

    • stored fields to retrieve

    Also, since cursorMark deep paging should be used when possible:

    • sort spec (default: id asc)

    This connector can be configured to store information about datasources and the data ingested in a ConnectorDB crawldb instance.

    Limitations

    • Cannot do incremental crawls. (May be possible to do so in the future using source Solr docs' version field.)

    • Cannot do manual filtered deep paging.

    • Does not check that both sort spec and field list contain uniqueKey field.

    • Cannot handle encrypted connection to Solr.

    Configuration

    When entering configuration values in the UI, use unescaped characters, such as \t for the tab character. When entering configuration values in the API, use escaped characters, such as \\t for the tab character.

    A connector that pulls selected documents from an external Solr instance or cluster.

    id - stringrequired

    Unique name for this datasource.

    >= 1 characters

    Match pattern: ^[a-zA-Z0-9_-]+$

    pipeline - stringrequired

    Name of an existing index pipeline for processing documents.

    >= 1 characters

    description - string

    Optional description for this datasource.

    properties - Properties

    Datasource configuration properties

    db - Connector DB

    Type and properties for a ConnectorDB implementation to use with this datasource.

    type - string

    Fully qualified class name of ConnectorDb implementation.

    >= 1 characters

    Default: com.lucidworks.connectors.db.impl.MapDbConnectorDb

    inlinks - boolean

    Keep track of incoming links. This negatively impacts performance and size of DB.

    Default: false

    aliases - boolean

    Keep track of original URI-s that resolved to the current URI. This negatively impacts performance and size of DB.

    Default: false

    inv_aliases - boolean

    Keep track of target URI-s that the current URI resolves to. This negatively impacts performance and size of DB.

    Default: false

    solr_base_url - string

    If using a single Solr instance, enter the base URL, e.g., http://solrhost.example.com:8983/solr/.

    zk_host_string - string

    If using a SolrCloud instance, enter the ZooKeeper connect string, e.g., zkServerA:2181,zkServerB:2181,zkServerC:2181/solr.

    source_collection - string

    Collection or Core name in the source Solr instance/cluster. If not defined, the default core will be used.

    solr_query - string

    Query to select documents from the source Solr instance/cluster. If not defined, the default query *:* will be used.

    >= 1 characters

    Default: *:*

    solr_filter_queries - string

    Filter queries to select documents from the source Solr instance/cluster. Multiple filter queries should be separated with commas.

    solr_field_list - string

    Fields to fetch from the source Solr instance/cluster, which must be stored fields. Multiple field names should be separated with commas.

    >= 1 characters

    Default: *

    solr_page_size - integer

    Number of rows per request to Solr.

    Default: 100

    solr_sort_spec - string

    Sort order for the request. The uniqueKey field must be included as one of the sorted fields.

    >= 1 characters

    Default: id asc

    solr_request_handler - string

    The request handler to use for the request to the Solr instance/cluster.

    >= 1 characters

    Default: /select

    solr_query_parser - string

    The query parser to use for the request.

    commit_on_finish - boolean

    Set to true for a request to be sent to Solr after the last batch has been fetched to commit the documents to the index.

    Default: true

    verify_access - boolean

    Set to true to require successful connection to the filesystem before saving this datasource.

    Default: true

    initial_mapping - Initial field mapping

    Provides mapping of fields before documents are sent to an index pipeline.

    skip - boolean

    Set to true to skip this stage.

    Default: false

    label - string

    A unique label for this stage.

    <= 255 characters

    condition - string

    Define a conditional script that must result in true or false. This can be used to determine if the stage should process or not.

    reservedFieldsMappingAllowed - boolean

    Default: false

    retentionMappings - array[object]

    Fields that should be kept or deleted

    Default:

    object attributes:{field required : {
     display name: Field
     type: string
    }    operation : {
     display name: Operation
     type: string
    }    }

    updateMappings - array[object]

    Values that should be added to or set on a field. When a value is added, any values previously on the field will be retained. When a value is set, any values previously on the field will be overwritten.

    Default:

    object attributes:{field required : {
     display name: Field
     type: string
    }    value required : {
     display name: Value
     type: string
    }    operation : {
     display name: Operation
     type: string
    }    }

    translationMappings - array[object]

    Fields that should be moved or copied to another field. When a field is moved, the values from the source field are moved over to the target field and the source field is removed. When a field is copied, the values from the source field are copied over to the target field and the source field is retained.

    Default: {"source":"_version_","target":"external_version_s","operation":"move"}

    object attributes:{source required : {
     display name: Source Field
     type: string
    }    target required : {
     display name: Target Field
     type: string
    }    operation : {
     display name: Operation
     type: string
    }    }

    unmappedRule - Unmapped Fields

    Fields not mapped by the above rules. By default, any remaining fields will be kept on the document.

    keep - boolean

    Keep all unmapped fields

    Default: true

    delete - boolean

    Delete all unmapped fields

    Default: false

    fieldToMoveValuesTo - string

    Move all unmapped field values to this field

    fieldToCopyValuesTo - string

    Copy all unmapped field values to this field

    valueToAddToUnmappedFields - string

    Add this value to all unmapped fields

    valueToSetOnUnmappedFields - string

    Set this value on all unmapped fields