Product Selector

Fusion 5.9
    Fusion 5.9

    Slack V2Connector Configuration Reference

    The Slack V2 connector allows you to index Slack channels and messages.

    Due to rate limit restrictions in the Slack Web API, this connector has the following restrictions:

    • Only one connector-plugin instance should be started.

    • Only one fetching thread should be set when creating the datasource configuration.

    Authentication

    You must install a Slack application in the workspace that will be indexed before using the Slack V2 connector. See Slack’s Basic app setup for more information about installing a Slack application.

    Once the app is correctly installed in the workspace, Slack will provide an access token, which you will then use in the Slack V2 connecter configuration.

    Required application scopes

    The Slack application must provide access to the following User Token Scopes in order to work with Slack V2 connector:

    Slack User Token Scopes

    Initial actions

    The Slack V2 connector will perform the following actions at the beginning of each job:

    • Load all users in the users cache. This information will be used later to resolve user names when processing mentions in messages.

      It will also be useful for avoiding requests to retrieve user’s metadata.

    • Load the channels to be processed using the channel filtering properties from the configuration.

      This will be useful to establish a source of truth when determining whether a channel should be indexed or deleted on each job.

    Document filtering

    Profiles indexing

    Index profiles

    Determines whether users are indexed

    Index guest profiles

    Determines whether profiles from guest users are indexed

    Index inactive profiles

    Determines whether profiles of deactivated users are indexed

    Channels indexing

    Index channels

    Determines whether a document per channel is indexed

    Index archived channels

    Determines whether archived channels are indexed to the content collection

    Index public channels

    Determines whether public channels are indexed

    Index private channels

    Determines whether private channels are indexed

    Index direct messages

    Determines whether direct and multi-party direct message only are indexed

    Channels to include

    A list of the channel names to be included in the crawling process

    Messages indexing

    Index messages

    Determines whether messages from the channels are indexed

    Index bot messages

    Determines where messages from bots in channels are indexed

    Index messages replies (threads)

    Determines whether the replies (if applicable) to messages are indexed

    Crawling process

    Due to limitations in the Slack Web API, edited and deleted messages cannot be retrieved. You must clear the datasource and start a new connectors job to index edited messages and/or remove deleted messages.

    Endpoints used in the crawling process

    Endpoints Description

    https://slack.com/api/users.list

    Lists all users in the workspace. This is not controlled by the properties in Document Filtering. All users are retrieved at the beginning of each job.

    https://slack.com/api/conversations.list

    Lists the channels that should be indexed. This fetching is guided by the properties listed in Document Filtering.

    https://slack.com/api/conversations.members

    Lists the members of specific channels being indexed.

    https://slack.com/api/conversations.history

    Retrieves a page of messages from a specific channel

    https://slack.com/api/users.info

    Retrieves the information of a single user, if it’s not present in the users cache (message document generation).

    https://slack.com/api/conversations.info

    Retrieves the information of a single channel, if it’s not present in the channel’s cache (message document generation).

    https://slack.com/api/team.info

    Retrieves the information of a team (message document generation).

    https://slack.com/api/conversations.replies

    Retrieves a page of replies from a specific message from a channel.

    Message mentions processing

    The Slack V2 connector retrieves messages from the Slack API in the following format:

     "type": "message",
     "text": "this is a mention in a message: <@U01U580HSDN>",

    In this example, @U01U580HSDN is the user Id. The connector will resolve the user Id with the user’s display name, if it exists. Otherwise, it will use the full name in the user’s profile.

    Incremental Crawling processors

    Incremental crawling process

    Each incremental job will re-index the User ACL to update all users in the Access Control Collection, and the Channel ACL to update the members of each channel.

    Processor Description

    ProfilesCheckpointProcessor

    Performs the following actions:

    • Triggers the indexing of all user profiles and the profile’s ACL

    • Verifies that profiles should still be indexed. If not, it deletes them

    • Verifies guest and deleted profiles

    ChannelsCheckpointProcessor

    Detects new and deleted channels by comparison and the triggers the re-indexing of the channel ACL documents.

    ChannelCheckpointProcessor

    Verifies that the current channel should still be indexed.

    If the channel should still be indexed, then it triggers the indexing of new messages providing the timestamp (ts) of the last seen message from the previous job.

    Security filtering

    Security filtering processors

    Processor Description

    ProfileAclProcessor

    Builds a document representing a Slack user, which is then sent to the Access Control Collection.

    Regular users can access public channels, even if they are not members of the channel.

    Guest users can access only the channels they are members of.

    ChannelAclProcessor

    Retrieves the current members of the given channel, builds a document with those values, and sends the document to the Access Control Collection.

    In public channels, an extra document is sent to the Access Control Collection to provide access to regular users who are not members of public groups.

    Security trimming rules

    Security trimming uses the following rules:

    • All users (regular and guest) can see the profile of any user in the Slack workspace.

    • All regular users can access public channels and messages, even if they are not members of the channel.

    • Guest users can only access the public or private channels they are members of.

    • Only the members of private channels can access the channel and messages.

    • Only the users involved can access direct messages and multi-party direct messages.

    Remote connectors

    V2 connectors support running remotely in Fusion versions 5.7.1 and later. Refer to Configure Remote V2 Connectors.

    Below is an example configuration showing how to specify the file system to index under the connector-plugins entry in your values.yaml file:

    additionalVolumes:
    - name: fusion-data1-pvc
        persistentVolumeClaim:
        claimName: fusion-data1-pvc
    - name: fusion-data2-pvc
        persistentVolumeClaim:
        claimName: fusion-data2-pvc
    additionalVolumeMounts:
    - name: fusion-data1-pvc
        mountPath: "/connector/data1"
    - name: fusion-data2-pvc
        mountPath: "/connector/data2"

    You may also need to specify the user that is authorized to access the file system, as in this example:

    securityContext:
        fsGroup: 1002100000
        runAsUser: 1002100000

    Connector for Slack

    description - string

    Optional description

    <= 125 characters

    pipeline - stringrequired

    Name of the IndexPipeline used for processing output.

    >= 1 characters

    Match pattern: ^[a-zA-Z0-9_-]+$

    diagnosticLogging - boolean

    Enable diagnostic logging; disabled by default

    Default: false

    parserId - stringrequired

    The Parser to use in the associated IndexPipeline.

    coreProperties - Core Properties

    Common behavior and performance settings.

    fetchSettings - Fetch Settings

    System level settings for controlling fetch behavior and performance.

    indexingInactivityTimeout - number

    The maximum amount of time to wait for indexing results (in seconds). If exceeded, the job will fail with an indexing inactivity timeout.

    >= 60

    <= 691200

    exclusiveMinimum: false

    exclusiveMaximum: false

    Default: 86400

    Multiple of: 1

    pluginInactivityTimeout - number

    The maximum amount of time to wait for plugin activity (in seconds). If exceeded, the job will fail with a plugin inactivity timeout.

    >= 60

    <= 691200

    exclusiveMinimum: false

    exclusiveMaximum: false

    Default: 600

    Multiple of: 1

    indexMetadata - boolean

    When enabled the metadata of skipped items will be indexed to the content collection.

    Default: false

    indexContentFields - boolean

    When enabled, content fields will be indexed to the crawl-db collection.

    Default: false

    asyncParsing - boolean

    When enabled, content will be indexed asynchronously.

    Default: false

    indexingThreads - number

    Maximum number of indexing threads; defaults to 4.This setting controls the number of threads in the indexing service used for processing content documents emitted by this datasource.Higher values can sometimes help with overall fetch performance.

    >= 1

    <= 10

    exclusiveMinimum: false

    exclusiveMaximum: false

    Default: 4

    Multiple of: 1

    pluginInstances - number

    Maximum number of plugin instances for distributed fetching. Only specified number of plugin instanceswill do fetching. This is useful for distributing load between different instances.

    <= 500

    exclusiveMinimum: false

    exclusiveMaximum: false

    Default: 0

    Multiple of: 1

    fetchResponseScheduledTimeout - number

    The maximum amount of time for a response to be scheduled. The task will be canceled if this setting is exceeded.

    >= 1000

    <= 500000

    exclusiveMinimum: false

    exclusiveMaximum: false

    Default: 300000

    Multiple of: 1

    id - stringrequired

    A unique identifier for this Configuration.

    >= 1 characters

    Match pattern: ^[a-zA-Z0-9_-]+$

    properties - Slack properties

    Plugin specific properties.

    slackAppToken - string

    The Slack application access token, this token is provided after installing the application in the workspace

    profilesConfig - Profiles settings

    Settings to control the indexation of Slack profiles

    indexProfiles - boolean

    Should profiles be indexed

    Default: true

    indexGuestProfiles - boolean

    If indexing profiles, include profiles of guest users

    Default: true

    indexInactiveProfiles - boolean

    If indexing profiles, include profiles of inactive users

    Default: false

    channelsConfig - Channels and messages settings

    Settings to control the indexation of Slack channels and messages

    indexChannels - boolean

    Generate a document in the index for each channel

    Default: true

    indexMessages - boolean

    Should individual messages be indexed

    Default: true

    indexFromArchivedChannels - boolean

    If indexing messages or channel data, include data from archived channels

    Default: false

    indexFromPublicChannels - boolean

    If indexing messages or channel data, include data from public channels

    Default: true

    indexPrivateChannels - boolean

    If indexing messages or channel data, include data from private channels

    Default: false

    indexDirectMessages - boolean

    If indexing messages, include private messages

    Default: false

    indexBotMessages - boolean

    If indexing messages, include messages that came from bots

    Default: false

    indexThreadReplies - boolean

    If indexing messages, include messages that are replies to threads. Note: Each thread found will result in it's own API call, and is subject to rate limiting. Using this can significantly slow down indexing.

    Default: false

    channelFilter - array[string]

    If indexing messages or channel data, include data only from channels with names matching the specified regular expressions

    requests - Requests settings

    pageSize - number

    The number of objects requested per page from Slack

    >= 1

    <= 1000

    exclusiveMinimum: false

    exclusiveMaximum: false

    Default: 200

    Multiple of: 1

    itemRetryProperties - Item retry settings

    Options to configure the retry operation for items.

    maxRetries - number

    The maximum number of attempts for a failed item

    <= 20

    exclusiveMinimum: false

    exclusiveMaximum: false

    Default: 3

    Multiple of: 1

    retryDelayInSeconds - number

    The amount of time, in seconds, before process again a failed item

    >= 1

    <= 600

    exclusiveMinimum: false

    exclusiveMaximum: false

    Default: 30

    Multiple of: 1

    connectionsProperties - Connections Settings

    A set of options for configuring the HTTP client.

    maxConnections - number

    The maximum number of connections

    >= 1

    <= 2147483647

    exclusiveMinimum: false

    exclusiveMaximum: false

    Default: 20

    Multiple of: 1

    timeouts - Timeout Settings

    A set of options for configuring the HTTP client.

    readTimeoutMs - number

    <= 600000

    exclusiveMinimum: false

    exclusiveMaximum: false

    Default: 300000

    Multiple of: 1

    connectTimeoutMs - number

    <= 300000

    exclusiveMinimum: false

    exclusiveMaximum: false

    Default: 60000

    Multiple of: 1

    security - Security filtering configuration

    enabled - boolean

    Enable query-time security-trimming

    Default: true

    collectionId - string

    Id of the collection to be used for storing ACL records. If not specified, ACL collection name will be generated automatically using pattern '<datasource_id>_access_control_hierarchy'.