Drupal 7.x Connector and Datasource Configuration

Configuration

This connector uses Drupal’s Services 7.x3.11Module REST API.

Configuration

Tip	When entering configuration values in the UI, use unescaped characters, such as `\t` for the tab character. When entering configuration values in the API, use escaped characters, such as `\\t` for the tab character.

Connector-specific Properties

Property Description

f.cacheSize

Cache size (number of entries)

The number of entries to cache when making REST requests.

type: integer

default value: '2000'

f.comment

Drupal comment service

Name of the Comment resource to be able to index comment data. If you did not create an alias for the 'comment' object, keep the default.

type: string

default value: 'comment'

f.drupal_password

Drupal password

Password to access the REST service, if required.

type: string

f.drupal_username

Drupal username

Optional username, only required if the REST service requires authentication.

type: string

f.endpoint

Drupal Endpoint

Name of the REST endpoint defined when you added the REST service to Drupal.

type: string

default value: 'rest'

f.file

Drupal file service

Name of the File resource to be able to index file data. If you did not create an alias for the 'file' object, keep the default.

type: string

default value: 'file'

f.node

Drupal node service

Name of the Node resource to be able to index node data. If you did not create an alias for the 'node' object, keep the default.

type: string

default value: 'node'

f.pageSize

Node Page Size (number of nodes per page)

The number of items that will be returned. The Drupal default without this value is 20, this allows you to request more items and reduce the overall number of Node requests to fetch all content.

type: integer

default value: '100'

f.taxonomy_term

Drupal Taxonomy Term service

Name of the Taxonomy Term resource to be able to index taxonomy term data. If you did not create an alias for the 'taxonomy_term' object, keep the default.

type: string

default value: 'taxonomy_term'

f.taxonomy_vocabulary

Drupal Taxonomy Vocabulary service

Name of the Taxonomy Vocabulary resource to be able to index taxonomy data. If you did not create an alias for the 'taxonomy_vocabulary' object, keep the default.

type: string

default value: 'taxonomy_vocabulary'

f.timeoutMS

Connection timeout (ms)

Time in ms to wait for a server response.

type: integer

default value: '10000'

f.user

Drupal user service

Name of the User resource to be able to login, if authenticating to the REST service is required. If you did not create an alias for the 'user' object, keep the default.

type: string

default value: 'user'

Drupal URL Aliases

Property Description

f.endpoint

Drupal Endpoint

Name of the REST endpoint defined when you added the REST service to Drupal.

type: string

default value: 'rest'

f.node

Drupal node service

Name of the Node resource to be able to index node data. If you did not create an alias for the 'node' object, keep the default.

type: string

default value: 'node'

f.user

Drupal user service

Name of the User resource to be able to login, if authenticating to the REST service is required. If you did not create an alias for the 'user' object, keep the default.

type: string

default value: 'user'

f.file

Drupal file service

Name of the File resource to be able to index file data. If you did not create an alias for the 'file' object, keep the default.

type: string

default value: 'file'

f.comment

Drupal comment service

Name of the Comment resource to be able to index comment data. If you did not create an alias for the 'comment' object, keep the default.

type: string

default value: 'comment'

f.taxonomy_vocabulary

Drupal Taxonomy Vocabulary service

Name of the Taxonomy Vocabulary resource to be able to index taxonomy data. If you did not create an alias for the 'taxonomy_vocabulary' object, keep the default.

type: string

default value: 'taxonomy_vocabulary'

f.taxonomy_term

Drupal Taxonomy Term service

Name of the Taxonomy Term resource to be able to index taxonomy term data. If you did not create an alias for the 'taxonomy_term' object, keep the default.

type: string

default value: 'taxonomy_term'

Limit Documents

Property Description

depth

Max crawl depth

Number of levels in a directory or site tree to descend for documents.

type: integer

default value: '-1'

maxItems

Max items

Maximum number of documents to fetch. The default (-1) means no limit.

type: integer

default value: '-1'

includeExtensions

Included file extensions

File extensions to be fetched. This will limit this datasource to only these file extensions.

type: array of string

includeRegexes

Inclusive regexes

Regular expressions for URI patterns to include. This will limit this datasource to only URIs that match the regular expression.

type: array of string

excludeExtensions

Excluded file extensions

File extensions that should not to be fetched. This will limit this datasource to all extensions except this list.

type: array of string

excludeRegexes

Exclusive regexes

Regular expressions for URI patterns to exclude. This will limit this datasource to only URIs that do not match the regular expression.

type: array of string

delete

Delete dead URIs

Set to true to remove documents from the index when they can no longer be accessed as unique documents.

type: boolean

default value: 'true'

deleteErrorsAfter

Fetch failure allowance

Number of fetch failures to tolerate before removing a document from the index. The default of -1 means no fetch failures will be removed.

type: integer

default value: '-1'

Crawl Performance

Property Description

f.pageSize

Node Page Size (number of nodes per page)

The number of items that will be returned. The Drupal default without this value is 20, this allows you to request more items and reduce the overall number of Node requests to fetch all content.

type: integer

default value: '100'

f.cacheSize

Cache size (number of entries)

The number of entries to cache when making REST requests.

type: integer

default value: '2000'

f.timeoutMS

Connection timeout (ms)

Time in ms to wait for a server response.

type: integer

default value: '10000'

chunkSize

Fetch batch size

The number of items to batch for each round of fetching. A higher value can make crawling faster, but memory usage is also increased. The default is 1.

type: integer

default value: '1'

fetchThreads

Fetch threads

The number of threads to use during fetching. The default is 5.

type: integer

default value: '5'

fetchDelayMS

Fetch delay

Number of milliseconds to wait between fetch requests. The default is 0. This property can be used to throttle a crawl if necessary.

type: integer

default value: '0'

fetchDelayMSPerHost

Fetch delay per host

If true, the 'Fetch delay (ms)' property will be applied for each host.

type: boolean

default value: 'false'

emitThreads

Emit threads

The number of threads used to send documents from the connector to the index pipeline. The default is 5.

type: integer

default value: '5'

failFastOnStartLinkFailure

Fail crawl if start links are invalid

If true, when Fusion cannot connect to any of the provided start links, the crawl is stopped and an exception logged.

type: boolean

default value: 'true'

retryEmit

Retry emits

Set to true for emit batch failures to be retried on a document-by-document basis.

type: boolean

default value: 'true'

parserRetryCount

Max Parser Retries

The maximum number of times the configured parser will try getting content before giving up

type: integer

default value: '0'

exclusiveMaximum: true

exclusiveMinimum: false

maximum: 5

minimum: 0

Dedupe

Property Description

dedupe

Dedupe documents

If true, documents will be deduplicated. Deduplication can be done based on an analysis of the content, on the content of a specific field, or by a JavaScript function. If neither a field nor a script are defined, content analysis will be used.

type: boolean

default value: 'false'

dedupeSaveSignature

Save dedupe signature

If true,the signature used for dedupe will be stored in a 'dedupeSignature_s' field. Note this may cause errors about 'immense terms' in that field.

type: boolean

default value: 'false'

dedupeField

Dedupe field

Field to be used for dedupe. Define either a field or a dedupe script, otherwise the full raw content of each document will be used.

type: string

dedupeScript

Dedupe script

Custom javascript to dedupe documents. The script must define a 'genSignature(content){}' function, but can use any combination of document fields. The function must return a string.

type: string

Recrawl Rules

Property Description

refreshAll

Recrawl all items

Set to true to always recrawl all items found in the crawldb.

type: boolean

default value: 'true'

refreshStartLinks

Recrawl start links

Set to true to recrawl items specified in the list of start links.

type: boolean

default value: 'false'

refreshErrors

Recrawl errors

Set to true to recrawl items that failed during the last crawl.

type: boolean

default value: 'false'

refreshOlderThan

Recrawl age

Number of seconds to recrawl items whose last fetched date is longer ago than this value.

type: integer

default value: '-1'

refreshIDPrefixes

Recrawl ID prefixes

A prefix to recrawl all items whose IDs begin with this value.

type: array of string

refreshIDRegexes

Recrawl ID regexes

A regular expression to recrawl all items whose IDs match this pattern.

type: array of string

refreshScript

Recrawl script

A JavaScript function ('shouldRefresh()') to customize the items recrawled.

type: string

forceRefresh

Force recrawl

Set to true to recrawl all items even if they have not changed since the last crawl.

type: boolean

default value: 'false'

Crawl History

Property Description

retainOutlinks

Retain links in the crawldb

Set to true for links found during fetching to be stored in the crawldb. This increases precision in certain recrawl scenarios, but requires more memory and disk space.

type: boolean

default value: 'false'

aliasExpiration

Alias expiration

The number of crawls after which an alias will expire. The default is 1 crawl.

type: integer

default value: '1'

crawlDBType

Crawl database type

The type of crawl database to use, in-memory or on-disk.

type: string

default value: 'in-memory'

enum: { in-memory on-disk }

indexCrawlDBToSolr

Index crawl database to Solr

EXPERIMENTAL: Set to true to index the crawl-database into a 'crawldb_' collection in Solr.

type: boolean

default value: 'false'

reevaluateCrawlDBOnStart

Reevaluate crawldb on start?

Reevaluate exisiting crawldb entries for legality on startup?

type: boolean

default value: 'false'

Field Mapping

Property Description

mappings

Field Mappings

List of mapping rules

type: array of object

default value: '{"operation"=>"move", "source"=>"charSet", "target"=>"charSet_s"}{"operation"=>"move", "source"=>"fetchedDate", "target"=>"fetchedDate_dt"}{"operation"=>"move", "source"=>"lastModified", "target"=>"lastModified_dt"}{"operation"=>"move", "source"=>"signature", "target"=>"dedupeSignature_s"}{"operation"=>"move", "source"=>"contentSignature", "target"=>"signature_s"}{"operation"=>"move", "source"=>"length", "target"=>"length_l"}{"operation"=>"move", "source"=>"mimeType", "target"=>"mimeType_s"}{"operation"=>"move", "source"=>"parent", "target"=>"parent_s"}{"operation"=>"move", "source"=>"owner", "target"=>"owner_s"}{"operation"=>"move", "source"=>"group", "target"=>"group_s"}'

object attributes: {
  operation : {
    display name: Operation
    type: string
    default value: 'copy'
    description : The type of mapping to perform: move, copy, delete, add, set, or keep.
    enum: { copy move delete set add keep     }

    }
  source (required) : {
    display name: Source Field
    type: string
    description : The name of the field to be mapped.
    }
  target : {
    display name: Target Field
    type: string
    description : The name of the field to be mapped to.
    }
  }

reservedFieldsMappingAllowed

Allow System Fields Mapping?

type: boolean

default value: 'false'

unmapped

Unmapped Fields

If fields do not match any of the field mapping rules, these rules will apply.

type: object

    }
  source : {
    display name: Source Field
    type: string
    description : The name of the field to be mapped.
    }
  target : {
    display name: Target Field
    type: string
    description : The name of the field to be mapped to.
    }
  }

ConnectorDb Configuration

Property Description

aliases

Process Aliases?

Keep track of original URI-s that resolved to the current URI. This negatively impacts performance and size of DB.

type: boolean

default value: 'false'

inlinks

Process Inlinks?

Keep track of incoming links. This negatively impacts performance and size of DB.

type: boolean

default value: 'false'

inv_aliases

Process Inverted Aliases?

Keep track of target URI-s that the current URI resolves to. This negatively impacts performance and size of DB.

type: boolean

default value: 'false'

type

Implementation Class Name

Fully qualified class name of ConnectorDb implementation.

type: string

default value: 'com.lucidworks.connectors.db.impl.MapDbConnectorDb'

minLength: 1

General Configuration

Property

Description

parserId

Parser

The parser used to process raw content.

pipeline

Pipeline ID

The index pipeline used to process documents.