chunkSize - integer
The number of items to batch for each round of fetching. A higher value can make crawling faster, but memory usage is also increased. The default is 1.
Default: 1
commitAfterItems - integer
Commit the crawlDB to disk after this many items have been received. A smaller number here will result in a slower crawl because of commits to disk being more frequent; conversely, a larger number here will cause a resumed job after a crash to need to recrawl more records.
Default: 10000
crawlDBType - string
The type of crawl database to use, in-memory or on-disk.
Default: on-disk
Allowed values: on-diskin-memory
db - Connector DB
Type and properties for a ConnectorDB implementation to use with this datasource.
aliases - boolean
Keep track of original URI-s that resolved to the current URI. This negatively impacts performance and size of DB.
Default: false
inlinks - boolean
Keep track of incoming links. This negatively impacts performance and size of DB.
Default: false
inv_aliases - boolean
Keep track of target URI-s that the current URI resolves to. This negatively impacts performance and size of DB.
Default: false
type - string
Fully qualified class name of ConnectorDb implementation.
>= 1 characters
Default: com.lucidworks.connectors.db.impl.MapDbConnectorDb
dedupe - boolean
If true, documents will be deduplicated. Deduplication can be done based on an analysis of the content, on the content of a specific field, or by a JavaScript function. If neither a field nor a script are defined, content analysis will be used.
Default: false
dedupeField - string
Field to be used for dedupe. Define either a field or a dedupe script, otherwise the full raw content of each document will be used.
dedupeSaveSignature - boolean
If true,the signature used for dedupe will be stored in a 'dedupeSignature_s' field. Note this may cause errors about 'immense terms' in that field.
Default: false
dedupeScript - string
Custom javascript to dedupe documents. The script must define a 'genSignature(content){}' function, but can use any combination of document fields. The function must return a string.
delete - boolean
Set to true to remove documents from the index when they can no longer be accessed as unique documents.
Default: true
deleteErrorsAfter - integer
Number of fetch failures to tolerate before removing a document from the index. The default of -1 means no fetch failures will be removed.
Default: -1
depth - integer
Number of levels in a directory or site tree to descend for documents.
Default: -1
diagnosticMode - boolean
Enable to print more detailed information to the logs about each request.
Default: false
emitThreads - integer
The number of threads used to send documents from the connector to the index pipeline. The default is 5.
Default: 5
enable_security_trimming - Enable Security Trimming
f.cache_expiration_time - integer
Time in seconds before the local and global security filter cache entries expire.
Default: 7200
f.cache_max_size - integer
Maximum number of items to hold in the security filter cache.
Default: 1000
f.enable_sharepoint_security_trimming - boolean
Enable indexing and query-time security-trimming of Sharepoint users/groups (i.e. in addition to users/groups in Active Directory)
Default: true
f.global_security_filter_cache - boolean
Enable shared caching of access-control filters across datasources. This cache will be shared with other SharePoint datasources.
Default: false
f.ldap_follow_referrals - boolean
Indicates whether or not to follow LDAP referrals. Select this checkbox to "follow" referrals and un-select it to "ignore" referrals.
Default: true
f.ldap_host - string
Hostname of the LDAP or AD server where user information is stored.
f.ldap_port - integer
Port for the LDAP or AD server.
Default: 389
f.ldap_read_groups_type - string
Mode for reading groups from LDAP or AD.
Default: TOKEN_GROUPS
f.ldap_search_base - string
Base node for LDAP or AD user and group searches.
f.ldap_use_ssl - boolean
Enable to use SSL when connecting to the LDAP or AD server.
Default: false
f.security_filter_cache - boolean
Cache of document access control rules. This cache will not be shared with any other SharePoint datasources.
Default: false
f.user_group_cache - boolean
When this option is selected, the crawl process will store all user's LDAP and SharePoint groups a Solr sidecar collection such that a user's groups do not need to be looked up at query time. This is not to be mistaken with the other global and local security-filter caches of which cache the access-control filter of the user.
Default: true
f.user_group_cache_collection_name - string
The name of the solr collection that will store this datasources' user group cache. This user group cache collection can be shared with other datasources. There is a `ds_id_s` field that is used to query user/groups separately.
Default: sp_usr_grp_{datasource}
excludeExtensions - array[string]
File extensions that should not to be fetched. This will limit this datasource to all extensions except this list.
excludeRegexes - array[string]
Regular expressions for URI patterns to exclude. This will limit this datasource to only URIs that do not match the regular expression.
f.avoid_ssl_hostname_verification - boolean
Enable this in cases when the CN on the SSL certificate does not match the host name of the server.
Default: true
f.bulkStartLinks - string
If a large number of start links must be defined, you can provide them here. One link per line.
f.domain - string
Authentication domain for the Sharepoint user.
f.enable_http_headers_debugging - boolean
Prints DEBUG level information to the logs.
Default: false
f.enable_retry - boolean
Enable this to make the connector automatically try again if a request to sharepoint fails unexpectedly.
Default: false
f.firstLoginUrl - string
You will need to specify this property if your SharePoint instance has Forms authentication enabled. This will resolve the error: "Before opening files in this location you must first browse to the web site and select the option to login automatically". If your sharepoint url is http://sphost/sites/test_site the first login URL will be http://sphost/_windows/default.aspx?ReturnUrl=%2f_layouts%2f15%2fAuthenticate.aspx%3fSource%3d%252Fsites%252Ftest%255Fsite%252F&Source=%2Fsites%2Ftest_site%2F
f.index_items_discarded - boolean
Enable to index discarded document metadata
Default: true
f.kerberosEnabled - boolean
This tells the web connector to enable Spnego/Kerberos authentication scheme.
f.kerberosKeytabBase64 - string
This value is set as the base64 string of the to a keytab file of your principal that is used to authenticate. Optionally you can supply the keytab file path in the "Kerberos Keytab File Path" property, or the "Kerberos Password".
f.kerberosKeytabFile - string
This value is set as the file path on the server to a keytab file of the principal that is used to authenticate. Optionally you can supply the keytab as a base64 encoded string in the "Kerberos Keytab in Base64" property, or the "Kerberos Password".
f.kerberosLoginContextName - string
If you are using a login configuration file with the JVM property "java.security.auth.login.config" then this value will tell what login context name to choose from this file when authenticating with Kerberos.
f.kerberosPassword - string
The password of the principal to use when authenticating with kerberos. You can also specify a keytab in the "Kerberos Keytab File Path" or "Kerberos Keytab in Base64" properties.
f.kerberosPrincipal - string
When this value is specified along with a keytab, this value is the "Kerberos Principal" of the login user that has authorization to crawl your web site(s). If you do not specify this when kerberos/spnego is enabled, the default principal name will be used.
f.kerberosSpn - string
If you specify this value, this service principal name (SPN) will be used. If not specified, the resource you are accessing will be used to create this value from the client. An example of an SPN would be: "http@your.realm.com"
f.log_sharepoint_xml - boolean
When analyzing Sharepoint crawls, it can be helpful to log the Soap XML messages between Fusion and Sharepoint. By selecting this, Sharepoint will log the xml of all messages to the connectors log.
Default: false
f.maxSizeBytes - integer
Maximum size, in bytes, of a document to crawl.
Default: 4194304
f.minSizeBytes - integer
Minimum size, in bytes, of a document to crawl.
Default: 0
f.password - string
Password for the Sharepoint user.
f.proxyHost - string
The address to use when connecting through the proxy.
f.proxyPort - integer
The port to use when connecting through the proxy. (HTTP or SOCKS)
f.remove_prepended_ids - boolean
If fields have been defined to include PrependIds, this option will remove those IDs before indexing.
Default: true
f.replace_invalid_xml_entities - boolean
Replace invalid xml entities to avoid the error: \"Unmarshalling Error: Illegal character entity: expansion character ...\"
Default: false
f.retry_attempts - integer
If Retry requests is enabled, this number determines how many times the connector should retry a request before giving up.
Default: 4
f.sharepoint_services_timeout - integer
Time in milliseconds to wait for a server response.
Default: 600000
f.username - string
Name of a Sharepoint user who has the required permissions to access Sharepoint via the SOAP API.
failFastOnStartLinkFailure - boolean
If true, when Fusion cannot connect to any of the provided start links, the crawl is stopped and an exception logged.
Default: true
fetchDelayMS - integer
Number of milliseconds to wait between fetch requests. The default is 0. This property can be used to throttle a crawl if necessary.
Default: 0
fetchThreads - integer
The number of threads to use during fetching. The default is 5.
Default: 5
forceRefresh - boolean
Set to true to recrawl all items even if they have not changed since the last crawl.
Default: false
forceRefreshClearSignatures - boolean
If true, signatures will be cleared if force recrawl is enabled.
Default: true
includeExtensions - array[string]
File extensions to be fetched. This will limit this datasource to only these file extensions.
includeRegexes - array[string]
Regular expressions for URI patterns to include. This will limit this datasource to only URIs that match the regular expression.
indexCrawlDBToSolr - boolean
EXPERIMENTAL: Set to true to index the crawl-database into a 'crawldb_<datasource-ID>' collection in Solr.
Default: false
initial_mapping - Initial field mapping
Provides mapping of fields before documents are sent to an index pipeline.
condition - string
Define a conditional script that must result in true or false. This can be used to determine if the stage should process or not.
label - string
A unique label for this stage.
<= 255 characters
mappings - array[object]
List of mapping rules
Default: {"operation":"move","source":"charSet","target":"charSet_s"}{"operation":"move","source":"fetchedDate","target":"fetchedDate_dt"}{"operation":"move","source":"lastModified","target":"lastModified_dt"}{"operation":"move","source":"signature","target":"dedupeSignature_s"}{"operation":"move","source":"length","target":"length_l"}{"operation":"move","source":"mimeType","target":"mimeType_s"}{"operation":"move","source":"parent","target":"parent_s"}{"operation":"move","source":"owner","target":"owner_s"}{"operation":"move","source":"group","target":"group_s"}
object attributes:{operation : {
display name: Operation
type: string
}source required : {
display name: Source Field
type: string
}target : {
display name: Target Field
type: string
}}
reservedFieldsMappingAllowed - boolean
Default: false
skip - boolean
Set to true to skip this stage.
Default: false
unmapped - Unmapped Fields
If fields do not match any of the field mapping rules, these rules will apply.
operation - string
The type of mapping to perform: move, copy, delete, add, set, or keep.
Default: copy
Allowed values: copymovedeletesetaddkeep
source - string
The name of the field to be mapped.
target - string
The name of the field to be mapped to.
maxItems - integer
Maximum number of documents to fetch. The default (-1) means no limit.
Default: -1
parserRetryCount - integer
The maximum number of times the configured parser will try getting content before giving up
<= 99
exclusiveMinimum: false
exclusiveMaximum: true
Default: 0
refreshAll - boolean
Set to true to always recrawl all items found in the crawldb.
Default: true
refreshErrors - boolean
Set to true to recrawl items that failed during the last crawl.
Default: false
refreshIDPrefixes - array[string]
A prefix to recrawl all items whose IDs begin with this value.
refreshIDRegexes - array[string]
A regular expression to recrawl all items whose IDs match this pattern.
refreshOlderThan - integer
Number of seconds to recrawl items whose last fetched date is longer ago than this value.
Default: -1
refreshScript - string
A JavaScript function ('shouldRefresh()') to customize the items recrawled.
refreshStartLinks - boolean
Set to true to recrawl items specified in the list of start links.
Default: false
retryEmit - boolean
Set to true for emit batch failures to be retried on a document-by-document basis.
Default: true
rewriteLinkScript - string
A Javascript function 'rewriteLink(link) { }' to modify links to documents before they are fetched.
startLinks - array[string]
Documents that point to Sharepoint Sites.
trackEmbeddedIDs - boolean
Track IDs produced by splitters to enable dedupe and deletion of embedded content?
Default: true