Twitter Stream Connector Configuration Reference

Configuration

The Twitter Stream connector uses Twitter’s streaming API to continually index Twitter. The datasource can be configured to limit tweets or it can be run indefinitely, until Twitter cuts off your access or you stop the datasource. This connector only retrieves tweets created after the datasource has been started.

Configuration

Tip	When entering configuration values in the UI, use unescaped characters, such as `\t` for the tab character. When entering configuration values in the API, use escaped characters, such as `\\t` for the tab character.

Property Description

access_token

Access Token

required

The OAuth Access Token is provided by Twitter when registering the application.

type: string

minLength: 1

collection

Collection

Collection documents will be indexed to.

type: string

pattern: ^[a-zA-Z0-9_-]+$

commit_on_finish

Solr commit on finish

Set to true for a request to be sent to Solr after the last batch has been fetched to commit the documents to the index.

type: boolean

default value: 'true'

consumer_key

Consumer Key

required

The OAuth Consumer Key is provided by Twitter when registering the application.

type: string

minLength: 1

consumer_secret

Consumer Secret

required

The OAuth Consumer Secret is provided by Twitter when registering the application.

type: string

minLength: 1

filter_follow

Filter Follow

Set of users (user ids) to track

type: array of string

filter_locations

Filter Locations

Set of bounding boxes (e.g. 'left,bottom,right,top' lat/long coordinates)

type: array of string

filter_track

Filter Track

Keywords or phrases to track

type: array of string

max_docs

Max Documents

The maximum number of documents to pull down, as a long. -1 for no limit

type: integer

default value: '-1'

sleep

Sleep

The amount of time, in milliseconds, to sleep when listening so as to not get throttled

type: integer

default value: '10000'

token_secret

Token Secret

required

The OAuth Token Secret is provided by Twitter when registering the application.

type: string

minLength: 1

url

URL

The URL used by the Twitter Stream

type: string

default value: 'https://stream.twitter.com'

Field Mapping

Initial Mappings Description

mappings

Field Mappings

List of mapping rules

type: array of object

object attributes: {
  operation : {
    display name: Operation
    type: string
    default value: 'copy'
    description : The type of mapping to perform: move, copy, delete, add, set, or keep.
    enum: { copy move delete set add keep     }

    }
  source (required) : {
    display name: Source Field
    type: string
    description : The name of the field to be mapped.
    }
  target : {
    display name: Target Field
    type: string
    description : The name of the field to be mapped to.
    }
  }

reservedFieldsMappingAllowed

Allow System Fields Mapping?

type: boolean

default value: 'false'

unmapped

Unmapped Fields

If fields do not match any of the field mapping rules, these rules will apply.

type: object

    }
  source : {
    display name: Source Field
    type: string
    description : The name of the field to be mapped.
    }
  target : {
    display name: Target Field
    type: string
    description : The name of the field to be mapped to.
    }
  }

ConnectorDb Configuration

Property Description

aliases

Process Aliases?

Keep track of original URI-s that resolved to the current URI. This negatively impacts performance and size of DB.

type: boolean

default value: 'false'

inlinks

Process Inlinks?

Keep track of incoming links. This negatively impacts performance and size of DB.

type: boolean

default value: 'false'

inv_aliases

Process Inverted Aliases?

Keep track of target URI-s that the current URI resolves to. This negatively impacts performance and size of DB.

type: boolean

default value: 'false'

type

Implementation Class Name

Fully qualified class name of ConnectorDb implementation.

type: string

default value: 'com.lucidworks.connectors.db.impl.MapDbConnectorDb'

minLength: 1

General Configuration

Property

Description

description

Description

Optional description for datasource instance.

Datasource ID

Unique name for datasource instance.

parserId

Parser

The parser used to process raw content.

pipeline

Pipeline ID

The index pipeline used to process documents.