Product Selector

Fusion 5.12
    Fusion 5.12

    SharePoint Online V1Connector Configuration Reference

    Table of Contents

    The SharePoint Online V1 connector retrieves data from cloud-based SharePoint repositories. Authentication requires a Sharepoint user who has permissions to access Sharepoint via the SOAP API. This user must be registered with the Sharepoint Online authentication server; it is not necessarily the same as the user in Active Directory or LDAP.

    Deprecation and removal notice

    This connector is deprecated as of Fusion 5.3 and is removed or expected to be removed as of Fusion 5.5. Use the SharePoint Optimized V2 connector instead.

    For more information about deprecations and removals, including possible alternatives, see Deprecations and Removals.

    Configuration

    When entering configuration values in the UI, use unescaped characters, such as \t for the tab character. When entering configuration values in the API, use escaped characters, such as \\t for the tab character.

    A crawler for SharePoint online

    id - stringrequired

    Unique name for this datasource.

    >= 1 characters

    Match pattern: ^[a-zA-Z0-9_-]+$

    pipeline - stringrequired

    Name of an existing index pipeline for processing documents.

    >= 1 characters

    description - string

    Optional description for this datasource.

    parserId - string

    Parser used when parsing raw content. For some connectors, a configuration to 'retry' parsing if an error occurs is available as an advanced setting

    properties - Properties

    Datasource configuration properties

    startLinks - array[string]

    SharePoint Sites, SharePoint Lists e.g. http(s)://<site_url>/Lists/<list_name>/AllItems.aspx and SharePoint Libraries e.g. http(s)://<site_url>/<library_name>/Forms/AllItems.aspx are allowed.

    db - Connector DB

    Type and properties for a ConnectorDB implementation to use with this datasource.

    type - string

    Fully qualified class name of ConnectorDb implementation.

    >= 1 characters

    Default: com.lucidworks.connectors.db.impl.MapDbConnectorDb

    inlinks - boolean

    Keep track of incoming links. This negatively impacts performance and size of DB.

    Default: false

    aliases - boolean

    Keep track of original URI-s that resolved to the current URI. This negatively impacts performance and size of DB.

    Default: false

    inv_aliases - boolean

    Keep track of target URI-s that the current URI resolves to. This negatively impacts performance and size of DB.

    Default: false

    dedupe - boolean

    If true, documents will be deduplicated. Deduplication can be done based on an analysis of the content, on the content of a specific field, or by a JavaScript function. If neither a field nor a script are defined, content analysis will be used.

    Default: false

    dedupeField - string

    Field to be used for dedupe. Define either a field or a dedupe script, otherwise the full raw content of each document will be used.

    dedupeScript - string

    Custom javascript to dedupe documents. The script must define a 'genSignature(content){}' function, but can use any combination of document fields. The function must return a string.

    dedupeSaveSignature - boolean

    If true,the signature used for dedupe will be stored in a 'dedupeSignature_s' field. Note this may cause errors about 'immense terms' in that field.

    Default: false

    delete - boolean

    Set to true to remove documents from the index when they can no longer be accessed as unique documents.

    Default: true

    deleteErrorsAfter - integer

    Number of fetch failures to tolerate before removing a document from the index. The default of -1 means no fetch failures will be removed.

    Default: -1

    fetchThreads - integer

    The number of threads to use during fetching. The default is 5.

    Default: 5

    emitThreads - integer

    The number of threads used to send documents from the connector to the index pipeline. The default is 5.

    Default: 5

    chunkSize - integer

    The number of items to batch for each round of fetching. A higher value can make crawling faster, but memory usage is also increased. The default is 1.

    Default: 1

    fetchDelayMS - integer

    Number of milliseconds to wait between fetch requests. The default is 0. This property can be used to throttle a crawl if necessary.

    Default: 0

    refreshAll - boolean

    Set to true to always recrawl all items found in the crawldb.

    Default: true

    refreshStartLinks - boolean

    Set to true to recrawl items specified in the list of start links.

    Default: false

    refreshErrors - boolean

    Set to true to recrawl items that failed during the last crawl.

    Default: false

    refreshOlderThan - integer

    Number of seconds to recrawl items whose last fetched date is longer ago than this value.

    Default: -1

    refreshIDPrefixes - array[string]

    A prefix to recrawl all items whose IDs begin with this value.

    refreshIDRegexes - array[string]

    A regular expression to recrawl all items whose IDs match this pattern.

    refreshScript - string

    A JavaScript function ('shouldRefresh()') to customize the items recrawled.

    forceRefresh - boolean

    Set to true to recrawl all items even if they have not changed since the last crawl.

    Default: false

    forceRefreshClearSignatures - boolean

    If true, signatures will be cleared if force recrawl is enabled.

    Default: true

    retryEmit - boolean

    Set to true for emit batch failures to be retried on a document-by-document basis.

    Default: true

    depth - integer

    Number of levels in a directory or site tree to descend for documents.

    Default: -1

    maxItems - integer

    Maximum number of documents to fetch. The default (-1) means no limit.

    Default: -1

    failFastOnStartLinkFailure - boolean

    If true, when Fusion cannot connect to any of the provided start links, the crawl is stopped and an exception logged.

    Default: true

    crawlDBType - string

    The type of crawl database to use, in-memory or on-disk.

    Default: on-disk

    Allowed values: in-memoryon-disk

    commitAfterItems - integer

    Commit the crawlDB to disk after this many items have been received. A smaller number here will result in a slower crawl because of commits to disk being more frequent; conversely, a larger number here will cause a resumed job after a crash to need to recrawl more records.

    Default: 10000

    initial_mapping - Initial field mapping

    Provides mapping of fields before documents are sent to an index pipeline.

    skip - boolean

    Set to true to skip this stage.

    Default: false

    label - string

    A unique label for this stage.

    <= 255 characters

    condition - string

    Define a conditional script that must result in true or false. This can be used to determine if the stage should process or not.

    reservedFieldsMappingAllowed - boolean

    Default: false

    retentionMappings - array[object]

    Fields that should be kept or deleted

    Default:

    object attributes:{field required : {
     display name: Field
     type: string
    }
    operation : {
     display name: Operation
     type: string
    }
    }

    updateMappings - array[object]

    Values that should be added to or set on a field. When a value is added, any values previously on the field will be retained. When a value is set, any values previously on the field will be overwritten.

    Default:

    object attributes:{field required : {
     display name: Field
     type: string
    }
    value required : {
     display name: Value
     type: string
    }
    operation : {
     display name: Operation
     type: string
    }
    }

    translationMappings - array[object]

    Fields that should be moved or copied to another field. When a field is moved, the values from the source field are moved over to the target field and the source field is removed. When a field is copied, the values from the source field are copied over to the target field and the source field is retained.

    Default: {"source":"charSet","target":"charSet_s","operation":"move"}{"source":"fetchedDate","target":"fetchedDate_dt","operation":"move"}{"source":"lastModified","target":"lastModified_dt","operation":"move"}{"source":"signature","target":"dedupeSignature_s","operation":"move"}{"source":"length","target":"length_l","operation":"move"}{"source":"mimeType","target":"mimeType_s","operation":"move"}{"source":"parent","target":"parent_s","operation":"move"}{"source":"owner","target":"owner_s","operation":"move"}{"source":"group","target":"group_s","operation":"move"}

    object attributes:{source required : {
     display name: Source Field
     type: string
    }
    target required : {
     display name: Target Field
     type: string
    }
    operation : {
     display name: Operation
     type: string
    }
    }

    unmappedRule - Unmapped Fields

    Fields not mapped by the above rules. By default, any remaining fields will be kept on the document.

    keep - boolean

    Keep all unmapped fields

    Default: true

    delete - boolean

    Delete all unmapped fields

    Default: false

    fieldToMoveValuesTo - string

    Move all unmapped field values to this field

    fieldToCopyValuesTo - string

    Copy all unmapped field values to this field

    valueToAddToUnmappedFields - string

    Add this value to all unmapped fields

    valueToSetOnUnmappedFields - string

    Set this value on all unmapped fields

    enable_security_trimming - Enable Security Trimming

    f.enable_sharepoint_security_trimming - boolean

    Enable indexing and query-time security-trimming of SharePoint users/groups (i.e. in addition to users/groups in Active Directory)

    Default: true

    f.security_filter_cache - boolean

    Cache of document access control rules. This cache will not be shared with any other SharePoint datasources.

    Default: false

    f.global_security_filter_cache - boolean

    Enable shared caching of access-control filters across datasources. This cache will be shared with other SharePoint datasources.

    Default: false

    f.cache_expiration_time - integer

    Time in seconds before the local and global security filter cache entries expire.

    Default: 7200

    f.user_group_cache - boolean

    When this option is selected, the crawl process will store all user's LDAP and SharePoint groups a Solr sidecar collection such that a user's groups do not need to be looked up at query time. This is not to be mistaken with the other global and local security-filter caches of which cache the access-control filter of the user.

    Default: true

    f.user_group_cache_collection_name - string

    The name of the solr collection that will store this datsources' user group cache. This user group cache collection can be shared with other datasources. There is a `ds_id_s` field that is used to query user/groups separately.

    Default: sp_usr_grp_{datasource}

    f.cache_max_size - integer

    Maximum number of items to hold in the security filter cache.

    Default: 1000

    f.ldap_host - string

    Hostname of the LDAP or AD server where user information is stored.

    f.ldap_port - integer

    Port for the LDAP or AD server.

    Default: 389

    f.ldap_use_ssl - boolean

    Enable to use SSL when connecting to the LDAP or AD server.

    Default: false

    f.ldap_search_base - string

    Base node for LDAP or AD user and group searches.

    f.ldap_read_groups_type - string

    Mode for reading groups from LDAP or AD.

    Default: TOKEN_GROUPS

    f.ldap_follow_referrals - boolean

    Indicates whether or not to follow LDAP referrals. Select this checkbox to "follow" referrals and un-select it to "ignore" referrals.

    Default: true

    excludeExtensions - array[string]

    File extensions that should not to be fetched. This will limit this datasource to all extensions except this list.

    excludeRegexes - array[string]

    Regular expressions for URI patterns to exclude. This will limit this datasource to only URIs that do not match the regular expression.

    includeExtensions - array[string]

    File extensions to be fetched. This will limit this datasource to only these file extensions.

    includeRegexes - array[string]

    Regular expressions for URI patterns to include. This will limit this datasource to only URIs that match the regular expression.

    rewriteLinkScript - string

    A Javascript function 'rewriteLink(link) { }' to modify links to documents before they are fetched.

    diagnosticMode - boolean

    Enable to print more detailed information to the logs about each request.

    Default: false

    trackEmbeddedIDs - boolean

    Track IDs produced by splitters to enable dedupe and deletion of embedded content?

    Default: true

    f.username - string

    Name of a SharePoint user who has the required permissions to access SharePoint via the SOAP API. This user must be registered with the SharePoint Online authentication server; it is not necessarily the same as the user in Active Directory or LDAP.

    f.password - string

    Password for the SharePoint user.

    f.loginCookieRefreshRateMs - integer

    The amount of time in milliseconds to re-use a SharePoint online cookie until fetching a new one. Ideally this will be set to a small bit less thank the max of the Cookies.

    Default: 120000

    f.adUsername - string

    The active directory account username.

    f.adPassword - string

    The active directory account password.

    f.domain - string

    NETBIOS domain for the SharePoint crawl account. Example: LUCIDWORKS

    f.dnsSuffix - string

    The Default fully qualified domain for users. Example: lucidworks.com

    f.maxSizeBytes - integer

    Maximum size, in bytes, of a document to crawl.

    Default: 4194304

    f.minSizeBytes - integer

    Minimum size, in bytes, of a document to crawl.

    Default: 0

    f.avoid_ssl_hostname_verification - boolean

    Enable this in cases when the CN on the SSL certificate does not match the host name of the server.

    Default: true

    f.sharepoint_services_timeout - integer

    Time in milliseconds to wait for a server response.

    Default: 600000

    f.enable_http_headers_debugging - boolean

    Prints DEBUG level information to the logs.

    Default: false

    f.remove_prepended_ids - boolean

    If fields have been defined to include PrependIds, this option will remove those IDs before indexing.

    Default: true

    f.log_sharepoint_xml - boolean

    When analyzing SharePoint crawls, it can be helpful to log the Soap XML messages between Fusion and SharePoint. By selecting this, SharePoint will log the xml of all messages to the connectors log.

    Default: false

    f.replace_invalid_xml_entities - boolean

    Replace invalid xml entities to avoid the error: \"Unmarshalling Error: Illegal character entity: expansion character ...\"

    Default: false

    parserRetryCount - integer

    The maximum number of times the configured parser will try getting content before giving up

    <= 5

    exclusiveMinimum: false

    exclusiveMaximum: true

    Default: 0

    f.proxyHost - string

    The address to use when connecting through the proxy.

    f.proxyPort - integer

    The port to use when connecting through the proxy. (HTTP or SOCKS)

    f.enable_retry - boolean

    Enable this to make the connector automatically try again if a request to SharePoint fails unexpectedly

    Default: true

    f.retry_attempts - integer

    If Retry requests is enabled, this number determines how many times the connector should retry a request before giving up

    Default: 4

    f.index_items_discarded - boolean

    Enable to index discarded document metadata

    Default: true