Fusion 5.12
Product Selector
This connector traverses a network file system (NFS), where a shared drive is mounted to the same location on all hosts in the cluster that are running this connector.
When entering configuration values in the UI, use unescaped characters, such as \t for the tab character. When entering configuration values in the API, use escaped characters, such as \\t for the tab character.
|
Connector for filesystems locally mounted to the Fusion server.
Optional description for this datasource.
Unique name for this datasource.
>= 1 characters
Match pattern: ^[a-zA-Z0-9_-]+$
Parser used when parsing raw content. Retry parsing setting is available under crawl performance (advance setting)
Default: _system
Name of an existing index pipeline for processing documents.
>= 1 characters
Datasource configuration properties
The number of crawls after which an alias will expire. The default is 1 crawl.
Default: 1
The number of items to batch for each round of fetching. A higher value can make crawling faster, but memory usage is also increased. The default is 1.
Default: 1
Commit the crawlDB to disk after this many items have been received. A smaller number here will result in a slower crawl because of commits to disk being more frequent; conversely, a larger number here will cause a resumed job after a crash to need to recrawl more records.
Default: 10000
The type of crawl database to use, in-memory or on-disk.
Default: on-disk
Allowed values: on-diskin-memory
Type and properties for a ConnectorDB implementation to use with this datasource.
Keep track of original URI-s that resolved to the current URI. This negatively impacts performance and size of DB.
Default: false
Keep track of incoming links. This negatively impacts performance and size of DB.
Default: false
Keep track of target URI-s that the current URI resolves to. This negatively impacts performance and size of DB.
Default: false
Fully qualified class name of ConnectorDb implementation.
>= 1 characters
Default: com.lucidworks.connectors.db.impl.MapDbConnectorDb
If true, documents will be deduplicated. Deduplication can be done based on an analysis of the content, on the content of a specific field, or by a JavaScript function. If neither a field nor a script are defined, content analysis will be used.
Default: false
Field to be used for dedupe. Define either a field or a dedupe script, otherwise the full raw content of each document will be used.
If true,the signature used for dedupe will be stored in a 'dedupeSignature_s' field. Note this may cause errors about 'immense terms' in that field.
Default: false
Custom javascript to dedupe documents. The script must define a 'genSignature(content){}' function, but can use any combination of document fields. The function must return a string.
Set to true to remove documents from the index when they can no longer be accessed as unique documents.
Default: true
Number of fetch failures to tolerate before removing a document from the index. The default of -1 means no fetch failures will be removed.
Default: -1
Number of levels in a directory or site tree to descend for documents.
Default: -1
Enable to print more detailed information to the logs about each request.
Default: false
The number of threads used to send documents from the connector to the index pipeline. The default is 5.
Default: 5
File extensions that should not to be fetched. This will limit this datasource to all extensions except this list.
Regular expressions for URI patterns to exclude. This will limit this datasource to only URIs that do not match the regular expression.
Set to true to add information about documents found in the filesystem to the document, such as document owner, group, or ACL permissions.
Default: true
Enable to index discarded document metadata
Default: false
Maximum size (in bytes) of documents to fetch or -1 for unlimited file size.
Default: 4194304
Minimum size, in bytes, of documents to fetch.
Default: 0
If true, when Fusion cannot connect to any of the provided start links, the crawl is stopped and an exception logged.
Default: true
Number of milliseconds to wait between fetch requests. The default is 0. This property can be used to throttle a crawl if necessary.
Default: 0
The number of threads to use during fetching. The default is 5.
Default: 5
Set to true to recrawl all items even if they have not changed since the last crawl.
Default: false
If true, signatures will be cleared if force recrawl is enabled.
Default: true
File extensions to be fetched. This will limit this datasource to only these file extensions.
Regular expressions for URI patterns to include. This will limit this datasource to only URIs that match the regular expression.
EXPERIMENTAL: Set to true to index the crawl-database into a 'crawldb_<datasource-ID>' collection in Solr.
Default: false
Provides mapping of fields before documents are sent to an index pipeline.
Define a conditional script that must result in true or false. This can be used to determine if the stage should process or not.
A unique label for this stage.
<= 255 characters
List of mapping rules
Default: {"operation":"move","source":"charSet","target":"charSet_s"}{"operation":"move","source":"fetchedDate","target":"fetchedDate_dt"}{"operation":"move","source":"lastModified","target":"lastModified_dt"}{"operation":"move","source":"signature","target":"dedupeSignature_s"}{"operation":"move","source":"length","target":"length_l"}{"operation":"move","source":"mimeType","target":"mimeType_s"}{"operation":"move","source":"parent","target":"parent_s"}{"operation":"move","source":"owner","target":"owner_s"}{"operation":"move","source":"group","target":"group_s"}
object attributes:{operation : {
display name: Operation
type: string
}source required : {
display name: Source Field
type: string
}target : {
display name: Target Field
type: string
}}
Default: false
Set to true to skip this stage.
Default: false
If fields do not match any of the field mapping rules, these rules will apply.
The type of mapping to perform: move, copy, delete, add, set, or keep.
Default: copy
Allowed values: copymovedeletesetaddkeep
The name of the field to be mapped.
The name of the field to be mapped to.
Maximum number of documents to fetch. The default (-1) means no limit.
Default: -1
Set to true to always recrawl all items found in the crawldb.
Default: true
Set to true to recrawl items that failed during the last crawl.
Default: false
A prefix to recrawl all items whose IDs begin with this value.
A regular expression to recrawl all items whose IDs match this pattern.
Number of seconds to recrawl items whose last fetched date is longer ago than this value.
Default: -1
A JavaScript function ('shouldRefresh()') to customize the items recrawled.
Set to true to recrawl items specified in the list of start links.
Default: false
If true, only documents found in a tree below the start links will be fetched. By default, this means limiting the crawl to the domain of the start links. For example, if the start link is 'http://host.com/US' then only links to the 'host.com' domain will be followed. Further options are available for modifying this behavior.
Default: true
Set to true for links found during fetching to be stored in the crawldb. This increases precision in certain recrawl scenarios, but requires more memory and disk space.
Default: false
Set to true for emit batch failures to be retried on a document-by-document basis.
Default: true
A Javascript function 'rewriteLink(link) { }' to modify links to documents before they are fetched.
An absolute path to a file or directory. If a directory, the connector traverses its subdirectories recursively, e.g., /path/to/files.json
Track IDs produced by splitters to enable dedupe and deletion of embedded content?
Default: true