REST V2Connector Configuration Reference
The REST V2 connector is a way to use one connector for multiple datasources. It works with recipes, which are premade JSON configurations.
An API call is made to the REST V2 connector containing a recipe as the body. API calls to set up the REST V2 connector go through the Connector Datasources API.
This article lists and describes the Fusion properties used by REST V2. The recommended method of populating these fields is through the use of a recipe. For information on how to get a recipe from the public GitHub repository into Fusion, see How to use REST V2 recipes.
The REST V2 connector performs a full crawl each time the connectors runs, as incremental crawling is not supported. When changes are made to the source, such as adding, deleting, or modifying content, those changes will reflect in the index after the next crawl is performed. You can check for new documents in the index by navigating to Querying > Query Workbench in the Fusion UI.
Below is an example configuration showing how to specify the file system to index under the connector-plugins entry in your values.yaml file:
additionalVolumes:
- name: fusion-data1-pvc
persistentVolumeClaim:
claimName: fusion-data1-pvc
- name: fusion-data2-pvc
persistentVolumeClaim:
claimName: fusion-data2-pvc
additionalVolumeMounts:
- name: fusion-data1-pvc
mountPath: "/connector/data1"
- name: fusion-data2-pvc
mountPath: "/connector/data2"
You may also need to specify the user that is authorized to access the file system, as in this example:
securityContext:
fsGroup: 1002100000
runAsUser: 1002100000
A connector designed to assist in setting up and indexing data from multiple REST APIs
properties - REST Connector properties
Plugin specific properties.
serviceURL - string
The base URL of the API for retrieving external data.
authenticationMode - Authentication
The method of authentication used to connect to the third-party service.
basicAuth - Basic Authentication
Basic Authentication Properties
user - string
The username for Basic Authentication with the REST API source.
>= 1 characters
password - string
The password or secret value to perform the Basic Authentication to the REST API source. For example: password or API Key.
>= 1 characters
customAuth - Custom Authentication
Custom Authentication Properties
authUrl - string
The URL used when authenticating to the third-party service.
httpMethod - string
The HTTP method to use for the request.
Default: POST
Allowed values: GETPOST
secureVariables - array[object]
Variables to use within the authentication request definition.
object attributes:{keyName : {
display name: Key Name
type: string
}secretValue : {
display name: Secret Value
type: string
}}
headers - array[object]
Headers to use within the authentication request.
object attributes:{headerKey : {
display name: Header Key
type: string
}headerValue : {
display name: Header Value
type: string
}}
queries - array[object]
Query parameters to include in the authentication request.
object attributes:{queryKey : {
display name: Query Key
type: string
}queryValue : {
display name: Query Value
type: string
}}
bodyAuthRequest - string
The body of the authentication request in JSON format.
oAuth - OAuth
OAuth Authentication Properties
oAuthServiceUrl - string
The URL of the service used to retrieve OAuth data.
oAuthEndpoint - string
Endpoint path for OAuth data. For example: /services/oauth2/token
httpMethod - string
HTTP method to be used in the request.
Default: POST
Allowed values: POSTGET
plainVariables - array[object]
Plain text variables to include in the authentication request definition.
object attributes:{keyName : {
display name: Key Name
type: string
}keyValue : {
display name: Key Value
type: string
}}
secureVariables - array[object]
Secret variables to use within the authentication request definition.
object attributes:{keyName : {
display name: Key Name
type: string
}secretValue : {
display name: Secret Value
type: string
}}
headerAuthorizationTemplate - string
A template for inserting authorization details, such as Bearer ${LW_ACCESS_TOKEN}
accessTokenPath - string
The path to the access token within the response. This property also accepts JsonPath expressions, such as objects, objects[*] or $.objects[*].
oAuthValuesLocation - string
Specify if OAuth key-value pair should be sent in request parameters, headers, the request body as JSON, or as URL-encoded data.
Default: REQUEST PARAM
Allowed values: REQUEST PARAMHEADERBODY JSONURL ENCODED
serviceEndpoints - array[object]
A list of request configurations for service endpoints. This property is deprecated.
object attributes:{rootRequest required : {
display name: Root Request Configuration
type: object
}childrenRequests : {
display name:
Child Request Configuration
type: array
}}
requestConfigurations - array[object]
A hierarchical list of requests to fetch data from the REST source.
object attributes:{request required : {
display name: Request Configuration
type: object
}}
limitDocuments - Limit Documents
Restrictions to be placed on documents fetched.
excludeByRegex - array[object]
Exclude documents by Regular Expression
object attributes:{key : {
display name: Key
type: string
}value : {
display name: RegEx value to exclude
type: string
}}
filterByFileSize - Exclude by File Size
Filter documents that are outside of the minimum/maximum size limits.
key - string
Specifies the field name of the object containing the file size. This property also supports JsonPath expressions, such as objects.nested.path
>= 1 characters
minimumSize - number
Excludes files with sizes smaller than the configured value.
>= -9223372036854776000
<= 9223372036854776000
exclusiveMinimum: false
exclusiveMaximum: false
Default: 1
Multiple of: 1
maximumSize - number
Excludes files with sizes larger than the configured value. Set to -1 when there is no limit.
>= -9223372036854776000
<= 9223372036854776000
exclusiveMinimum: false
exclusiveMaximum: false
Default: -1
Multiple of: 1
httpClientProperties - HTTP Client Settings
A set of options for configuring the HTTP client.
readTimeoutMs - number
<= 600000
exclusiveMinimum: false
exclusiveMaximum: false
Default: 300000
Multiple of: 1
connectTimeoutMs - number
<= 300000
exclusiveMinimum: false
exclusiveMaximum: false
Default: 60000
Multiple of: 1
retryProperties - Retry Properties
Configure the retry behavior for failed requests.
retryCount - number
Number of attempts a request will be retried
>= 1
<= 36
exclusiveMinimum: false
exclusiveMaximum: false
Default: 3
Multiple of: 1
maxDelayTime - number
The maximum wait time in milliseconds between retries.
>= 1000
<= 60000
exclusiveMinimum: false
exclusiveMaximum: false
Default: 60000
Multiple of: 1
id - stringrequired
A unique identifier for this Configuration.
>= 1 characters
Match pattern: ^[a-zA-Z0-9_-]+$
pipelineId - stringrequired
Name of the IndexPipeline used for processing output.
>= 1 characters
Match pattern: ^[a-zA-Z0-9_-]+$
parserId - string
The Parser to use in the associated IndexPipeline.
Match pattern: ^[a-zA-Z0-9_-]+$
description - string
Optional description
<= 125 characters
diagnosticLogging - boolean
Enable diagnostic logging; disabled by default
Default: false
coreProperties - Core Properties
Common behavior and performance settings.
fetchSettings - Fetch Settings
System level settings for controlling fetch behavior and performance.
numFetchThreads - number
Maximum number of fetch threads; defaults to 5. This setting controls the number of threads that call the Connectors fetch method. Higher values can, but not always, help with overall fetch performance.
>= 1
<= 500
exclusiveMinimum: false
exclusiveMaximum: false
Default: 5
Multiple of: 1
indexingThreads - number
Maximum number of indexing threads; defaults to 4. This setting controls the number of threads in the indexing service used for processing content documents emitted by this datasource. Higher values can sometimes help with overall fetch performance.
>= 1
<= 10
exclusiveMinimum: false
exclusiveMaximum: false
Default: 4
Multiple of: 1
pluginInstances - number
Maximum number of plugin instances for distributed fetching. Only specified number of plugin instances will do fetching. This is useful for distributing load between different instances.
<= 500
exclusiveMinimum: false
exclusiveMaximum: false
Default: 0
Multiple of: 1
fetchResponseScheduledTimeout - number
The maximum amount of time for a response to be scheduled. The task will be canceled if this setting is exceeded.
>= 1000
<= 500000
exclusiveMinimum: false
exclusiveMaximum: false
Default: 300000
Multiple of: 1
indexingInactivityTimeout - number
The maximum amount of time to wait for indexing results (in seconds). If exceeded, the job will fail with an indexing inactivity timeout.
>= 60
<= 691200
exclusiveMinimum: false
exclusiveMaximum: false
Default: 86400
Multiple of: 1
pluginInactivityTimeout - number
The maximum amount of time to wait for plugin activity (in seconds). If exceeded, the job will fail with a plugin inactivity timeout.
>= 60
<= 691200
exclusiveMinimum: false
exclusiveMaximum: false
Default: 600
Multiple of: 1
indexMetadata - boolean
When enabled the metadata of skipped items will be indexed to the content collection.
Default: false
indexContentFields - boolean
When enabled, content fields will be indexed to the crawl-db collection.
Default: false
asyncParsing - boolean
When enabled, content will be indexed asynchronously.
Default: false