|
chunkSize
Fetch batch size
|
The number of items to batch for each round of fetching. A higher value can make crawling faster, but memory usage is also increased. The default is 1.
type: integer
default value: '1'
|
|
collection
Collection
|
Collection documents will be indexed to.
type: string
pattern: ^[a-zA-Z0-9_-]+$
|
|
commitAfterItems
Commit After This Many Items
|
Commit the crawlDB to disk after this many items have been received. A smaller number here will result in a slower crawl because of commits to disk being more frequent; conversely, a larger number here will cause a resumed job after a crash to need to recrawl more records.
type: integer
default value: '10000'
|
|
crawlDBType
Crawl database type
|
The type of crawl database to use, in-memory or on-disk.
type: string
default value: 'on-disk'
enum: {
on-disk
in-memory
}
|
|
db
Connector DB
|
Type and properties for a ConnectorDB implementation to use with this datasource.
type: object
object attributes: {
}
object attributes: {
}
object attributes: {
}
|
|
dedupe
Dedupe documents
|
If true, documents will be deduplicated. Deduplication can be done based on an analysis of the content, on the content of a specific field, or by a JavaScript function. If neither a field nor a script are defined, content analysis will be used.
type: boolean
default value: 'false'
|
|
dedupeField
Dedupe field
|
Field to be used for dedupe. Define either a field or a dedupe script, otherwise the full raw content of each document will be used.
type: string
|
|
dedupeSaveSignature
Save dedupe signature
|
If true,the signature used for dedupe will be stored in a 'dedupeSignature_s' field. Note this may cause errors about 'immense terms' in that field.
type: boolean
default value: 'false'
|
|
dedupeScript
Dedupe script
|
Custom javascript to dedupe documents. The script must define a 'genSignature(content){}' function, but can use any combination of document fields. The function must return a string.
type: string
|
|
delete
Delete dead URIs
|
Set to true to remove documents from the index when they can no longer be accessed as unique documents.
type: boolean
default value: 'true'
|
|
deleteErrorsAfter
Fetch failure allowance
|
Number of fetch failures to tolerate before removing a document from the index. The default of -1 means no fetch failures will be removed.
type: integer
default value: '-1'
|
|
depth
Max crawl depth
|
Number of levels in a directory or site tree to descend for documents.
type: integer
default value: '-1'
|
|
diagnosticMode
Diagnostic mode
|
Enable to print more detailed information to the logs about each request.
type: boolean
default value: 'false'
|
|
emitThreads
Emit threads
|
The number of threads used to send documents from the connector to the index pipeline. The default is 5.
type: integer
default value: '5'
|
|
enable_security_trimming
Enable Security Trimming
|
type: object
object attributes: {
}
|
|
excludeExtensions
Excluded file extensions
|
File extensions that should not to be fetched. This will limit this datasource to all extensions except this list.
type: array of string
|
|
excludeRegexes
Exclusive regexes
|
Regular expressions for URI patterns to exclude. This will limit this datasource to only URIs that do not match the regular expression.
type: array of string
|
|
f.adPassword
Active Directory Password
|
The active directory account password.
type: string
|
|
f.adUsername
Active Directory Username
|
The active directory account username.
type: string
|
|
f.adfsStsIssuerURI
ADFS STS Issuer URI
|
The IssuerURI is used to by the authentication platform to locate the namespace that the token is designated for.
type: string
|
|
f.app_auth_azure_login_endpoint
App Auth Azure Login Endpoint
|
The azure login endpoint to use.
type: string
default value: 'https://login.windows.net'
|
|
f.app_auth_client_id
App Auth Client ID
|
When you want to use app authentication, this is the client ID of your application.
type: string
|
|
f.app_auth_client_secret
Azure AD Client Secret
|
Applicable to SharePoint Online OAuth App-Auth only. The Azure client ID of your application.
type: string
|
|
f.app_auth_pfx
App Base64 Encoded x509 PFX Certificate
|
The base64 encoded value of your X509 PFX certificate file. -- To get this in Linux (bash): base64 cert.pfx | tr -d '\n' -- To get this in Windows (powershell): [Convert]::ToBase64String([IO.File]::ReadAllBytes('cert.pfx'))
type: string
|
|
f.app_auth_pfx_password
App X509 Certificate Password
|
The password of the x509 pfx certificate.
type: string
|
|
f.app_auth_refresh_token
App auth refresh token
|
Applicable to SharePoint Online OAuth App-Auth only. This is a refresh token which is reusable for up to 12 hours. You must obtain a new tokenusing the OAuth login process if the token becomes expired.
type: string
|
|
f.app_auth_tenant
App Auth Tenant
|
The Office365 tenant of the app. E.g. exampleapp.onmicrosoft.com
type: string
|
|
f.avoid_ssl_hostname_verification
Avoid SSL hostname verification
|
Enable this in cases when the CN on the SSL certificate does not match the host name of the server.
type: boolean
default value: 'true'
|
|
f.connect_timeout
Connect Timeout
|
The async http connection timeout.
type: integer
default value: '5000'
|
|
f.dnsSuffix
Domain DNS suffix
|
The DNS suffix of the network. Example: lucidworks.com
type: string
|
|
f.domain
NetBIOS domain
|
The NETBIOS domain for the network. Example: LUCIDWORKS
type: string
|
|
f.enable_http_headers_debugging
Enable HTTP headers debugging
|
Prints DEBUG level information to the logs.
type: boolean
default value: 'false'
|
|
f.excludeContentsExtensions
Exclude file contents for file extensions
|
File extensions of files that will not have their contents downloaded when indexing this item. The list item metadata will still be indexed but the file contents will not. The comparison is not case sensitive, and you do not have to specify the '.' but it still work if you do. For example "zip" and ".zip" are both acceptable. The whitespace will also be trimmed.
type: array of string
|
|
f.fetch_all_site_collections
Fetch all site collections
|
When this is selected, obtain all site collections from the SharePoint web application and add them to the start links automatically.
type: boolean
default value: 'true'
|
|
f.includeContentsExtensions
Include file contents for file extensions
|
File extensions of files that will not have their contents downloaded when indexing this item. The list item metadata will still be indexed but the file contents will not. The comparison is not case sensitive, and you do not have to specify the '.' but it still work if you do. For example "zip" and ".zip" are both acceptable. The whitespace will also be trimmed.
type: array of string
|
|
f.list_view_threshold
List view threshold
|
Set this to your SharePoint farm's list item view threshold. This threshold sets the max amount of list item metadata elements you can fetch at one time. The typical default is 5000 but your SP administrators can set it to a different value. If you see the error "The attempted operation is prohibited because it exceeds the list view threshold enforced by the administrator", check with your SP admins to get the correct value.
type: integer
default value: '5000'
|
|
f.log_sharepoint_xml
Log Sharepoint Soap XML
|
When analyzing Sharepoint crawls, it can be helpful to log the Soap XML messages between Fusion and Sharepoint. By selecting this, Sharepoint will log the xml of all messages to the connectors log.
type: boolean
default value: 'false'
|
|
f.loginCookieRefreshRateMs
SharePoint Online Cookie Refresh Rate (ms)
|
The amount of time in milliseconds to re-use a SharePoint online cookie until fetching a new one. Ideally this will be set to a small bit less than the max of the Cookies.
type: integer
default value: '10800000'
|
|
f.maxSizeBytes
Maximum file size (bytes)
|
Maximum size, in bytes, of a document to crawl.
type: integer
default value: '4194304'
|
|
f.max_connections
Max Connections
|
Number of max async http connections.
type: integer
default value: '-1'
|
|
f.max_connections_per_host
Max Connections Per Host
|
The number of max connections per host.
type: integer
default value: '-1'
|
|
f.max_list_items_per_site_collection
Max number of list items per site collection
|
Setting this will cause the fetcher onlyfetch this many list items per site collection when crawling. Set to -1 for unlimited.
type: integer
default value: '-1'
|
|
f.max_prefetch_parallel_jobs
Max prefetch parallel jobs
|
This is the max number of prefetch jobs that can be run in parallel. If you are crawling small site collections, then you want to use a bigger number here such as 10 or 20. But if you are crawling large site collections, a smaller number is fine here because most of the work will be done within a each prefetch job.
type: integer
default value: '10'
|
|
f.max_site_collections
Maximium number of site collections
|
When the "Fetch all site collections" is checked, this will limit the number of site collections we fetch. If set to -1, all site collections will be fetched.
type: integer
default value: '-1'
|
|
f.max_sites_per_site_collection
Max number of sites per site collection
|
Setting this will cause the fetcher onlyfetch this many sites per site collection when crawling. Set to -1 for unlimited.
type: integer
default value: '-1'
|
|
f.minSizeBytes
Minimum file size (bytes)
|
Minimum size, in bytes, of a document to crawl.
type: integer
default value: '0'
|
|
f.password
Sharepoint password
|
Password for the Sharepoint user.
type: string
|
|
f.pooled_connection_idle_timeout
Pooled Connection Idle Timeout
|
The timeout for getting a connection from the pool.
type: integer
default value: '600000'
|
|
f.pooled_connection_ttl
Pooled Connection TTL
|
The time to live of a connection in the pool.
type: integer
default value: '360000'
|
|
f.prefetch_file_download_timeout_secs
Pre-Fetch file download timeout (seconds)
|
When should a file download timeout during the pre-fetch exporter (seconds).
type: integer
default value: '100'
|
|
f.prefetch_num_threads
Number of pre-fetch threads
|
How many threads to use when building the pre-fetch index. When crawling small site collections, you want to have a small number of prefetch threads such as 1 or 2 because the cost of allocating the prefetch threads will outweigh their benefit. But if you are crawling a very large site collection, you will want many threads like 25 - 50 because more work can be done faster with large number of fetch threads in that situation.
type: integer
default value: '5'
|
|
f.proxyHost
Proxy host
|
The address to use when connecting through the proxy.
type: string
|
|
f.proxyPort
Proxy port
|
The port to use when connecting through the proxy. (HTTP or SOCKS)
type: integer
|
|
f.proxyType
Proxy type
|
Type of proxy to use, if any. Allowed values are 'HTTP' and 'SOCKS'. Leave empty for no proxy.
type: string
default value: 'HTTP'
|
|
f.remove_prepended_ids
Remove prepended IDs
|
If fields have been defined to include PrependIds, this option will remove those IDs before indexing.
type: boolean
default value: 'true'
|
|
f.replace_invalid_xml_entities
Replace invalid xml entities
|
Replace invalid xml entities to avoid the error: \"Unmarshalling Error: Illegal character entity: expansion character ...\"
type: boolean
default value: 'false'
|
|
f.request_timeout
Request Timeout
|
The async http request timeout.
type: integer
default value: '300000'
|
|
f.retry_attempts
Retry Attempts
|
How many times should we fail retryable errors before giving up and throwing an error. Set this to 1 for no retries.
type: integer
default value: '4'
|
|
f.retry_backoff_delay_factor
Retryer Backoff Factor
|
The multiplicative factor in the backoff algorithm.
type: number
default value: '1.2'
|
|
f.retry_backoff_delay_ms
Retryer backoff delay (ms)
|
The number of milliseconds in the backoff retry algorithm.
type: integer
default value: '1000'
|
|
f.retry_backoff_max_delay_ms
Retryer backoff max delay (ms)
|
The maximum backoff time allowed in the backoff algorithm.
type: integer
default value: '15000'
|
|
f.retry_max_wait_ms
Retry maximum wait
|
The maximum amount of milliseconds to wait for retries to finish before giving up and stopping any more retries.
type: integer
default value: '600000'
|
|
f.sharepoint_online
Sharepoint Online
|
type: boolean
default value: 'true'
|
|
f.sharepoint_services_timeout
Sharepoint services timeout
|
Time in milliseconds to wait for a server response.
type: integer
default value: '600000'
|
|
f.user_agent
User agent
|
The user agent header decorates the http traffic. This is important for preventing hard rate limiting by SharePoint Online.
type: string
default value: 'ISV|Lucidworks|Fusion/4.2'
|
|
f.username
Sharepoint username
|
Name of a Sharepoint user who has the required permissions to access Sharepoint via the REST API. This user must be registered with the Sharepoint Online authentication server; it is not necessarily the same as the user in Active Directory or LDAP.
type: string
|
|
f.validate_on_save
Validate on Save
|
Validate when you save the datasource.
type: boolean
default value: 'true'
|
|
failFastOnStartLinkFailure
Fail crawl if start links are invalid
|
If true, when Fusion cannot connect to any of the provided start links, the crawl is stopped and an exception logged.
type: boolean
default value: 'true'
|
|
fetchDelayMS
Fetch delay
|
Number of milliseconds to wait between fetch requests. The default is 0. This property can be used to throttle a crawl if necessary.
type: integer
default value: '0'
|
|
fetchThreads
Fetch threads
|
The number of threads to use during fetching. The default is 5.
type: integer
default value: '5'
|
|
forceRefresh
Force recrawl
|
Set to true to recrawl all items even if they have not changed since the last crawl.
type: boolean
default value: 'false'
|
|
forceRefreshClearSignatures
Clear signatures
|
If true, signatures will be cleared if force recrawl is enabled.
type: boolean
default value: 'true'
|
|
includeExtensions
Included file extensions
|
File extensions to be fetched. This will limit this datasource to only these file extensions.
type: array of string
|
|
includeRegexes
Inclusive regexes
|
Regular expressions for URI patterns to include. This will limit this datasource to only URIs that match the regular expression.
type: array of string
|
|
indexCrawlDBToSolr
Index crawl database to Solr
|
EXPERIMENTAL: Set to true to index the crawl-database into a 'crawldb_' collection in Solr.
type: boolean
default value: 'false'
|
|
initial_mapping
Initial field mapping
|
Provides mapping of fields before documents are sent to an index pipeline.
type: object
object attributes: {
}
object attributes: {
}
object attributes: {
}
object attributes: {
}
unsafe: false
|
|
parserRetryCount
Max Parser Retries
|
The maximum number of times the configured parser will try getting content before giving up
type: integer
default value: '0'
|
|
refreshAll
Recrawl all items
|
Set to true to always recrawl all items found in the crawldb.
type: boolean
default value: 'true'
|
|
refreshErrors
Recrawl errors
|
Set to true to recrawl items that failed during the last crawl.
type: boolean
default value: 'false'
|
|
refreshIDPrefixes
Recrawl ID prefixes
|
A prefix to recrawl all items whose IDs begin with this value.
type: array of string
|
|
refreshIDRegexes
Recrawl ID regexes
|
A regular expression to recrawl all items whose IDs match this pattern.
type: array of string
|
|
refreshOlderThan
Recrawl age
|
Number of seconds to recrawl items whose last fetched date is longer ago than this value.
type: integer
default value: '-1'
|
|
refreshScript
Recrawl script
|
A JavaScript function ('shouldRefresh()') to customize the items recrawled.
type: string
|
|
refreshStartLinks
Recrawl start links
|
Set to true to recrawl items specified in the list of start links.
type: boolean
default value: 'false'
|
|
retryEmit
Retry emits
|
Set to true for emit batch failures to be retried on a document-by-document basis.
type: boolean
default value: 'true'
|
|
rewriteLinkScript
URI rewrite script
|
A Javascript function 'rewriteLink(link) { }' to modify links to documents before they are fetched.
type: string
|
|
startLinks
Start Links
|
Sharepoint Site Collections and Sites are allowed.
type: array of string
|
|
trackEmbeddedIDs
Track embedded IDs?
|
Track IDs produced by splitters to enable dedupe and deletion of embedded content?
type: boolean
default value: 'true'
|
General Configuration
| Property |
Description |
|
description Description
|
Optional description for datasource instance. |
|
id Datasource ID
|
Unique name for datasource instance. |
|
parserId Parser
|
The parser used to process raw content. |
|
pipeline Pipeline ID
|
The index pipeline used to process documents. |
