> ## Documentation Index > Fetch the complete documentation index at: https://doc.lucidworks.com/llms.txt > Use this file to discover all available pages before exploring further. # REST V2 > The REST V2 connector is a way to use one connector for multiple datasources. It works with recipes, which are premade JSON configurations. export const schema = { "type": "object", "title": "REST (v2)", "description": "A connector designed to assist in setting up and indexing data from multiple REST APIs", "required": ["id", "properties", "pipeline"], "properties": { "id": { "type": "string", "title": "Configuration ID", "description": "A unique identifier for this Configuration.", "minLength": 1, "pattern": "^[a-zA-Z0-9_-]+$" }, "pipeline": { "type": "string", "title": "Pipeline ID", "description": "Name of the IndexPipeline used for processing output.", "minLength": 1, "pattern": "^[a-zA-Z0-9_-]+$" }, "parserId": { "type": "string", "title": "Parser ID", "description": "The Parser to use in the associated IndexPipeline.", "pattern": "^[a-zA-Z0-9_-]+$" }, "description": { "type": "string", "title": "Description", "description": "Optional description", "hints": ["lengthy"], "maxLength": 125 }, "modified": { "type": "string", "title": "Date Modified", "description": "The date at which this Configuration was last modified.", "hints": ["readonly", "hidden"] }, "created": { "type": "string", "title": "Date Created", "description": "The date at which this Configuration was created.", "hints": ["readonly", "hidden"] }, "type": { "type": "string", "title": "Type", "description": "A type ID for this connector.", "hints": ["readonly", "hidden"] }, "collection": { "type": "string", "title": "Collection ID", "description": "The associated content Collection.", "hints": ["readonly", "hidden"], "minLength": 1, "pattern": "^[a-zA-Z0-9_-]+$" }, "diagnosticLogging": { "type": "boolean", "title": "Diagnostic Logging", "description": "[Deprecated] Enables verbose diagnostic logging for troubleshooting. May increase log volume. Disabled by default.", "default": false }, "type_description": { "type": "string", "title": "Type Description", "default": "A connector designed to assist in setting up and indexing data from multiple REST APIs", "hints": ["hidden", "readonly"] }, "category": { "type": "string", "title": "Category", "default": "A connector designed to assist in setting up and indexing data from multiple REST APIs", "hints": ["hidden", "readonly"] }, "connector": { "title": "Connector Type", "description": "Connector type.", "minLength": 1, "type": "string", "hints": ["hidden"] }, "properties": { "type": "object", "title": "REST Connector properties", "description": "Plugin specific properties.", "required": ["serviceURL"], "properties": { "serviceURL": { "type": "string", "title": "Service URL", "description": "The base URL of the API for retrieving external data." }, "authenticationMode": { "type": "object", "title": "Authentication", "description": "The method of authentication used to connect to the third-party service.", "required": [], "properties": { "basicAuth": { "type": "object", "title": "Basic Authentication", "description": "Basic Authentication Properties", "required": ["user"], "properties": { "user": { "type": "string", "title": "Username", "description": "The username for Basic Authentication with the REST API source.", "minLength": 1 }, "password": { "type": "string", "title": "Secret", "description": "The password or secret value to perform the Basic Authentication to the REST API source. For example: password or API Key.", "hints": ["secret"], "minLength": 1 }, "passwordVaultLabel": { "type": "string", "title": "Secret from Vault", "description": "The Fusion managed secret to use for Secret.", "hints": ["vaultvaluetarget:password", "enumUrl:/apps-manager/vault/labels?type=secret", "vaultref"] } } }, "customAuth": { "type": "object", "title": "Custom Authentication", "description": "Custom Authentication Properties", "required": [], "properties": { "authUrl": { "type": "string", "title": "Authentication URL", "description": "The URL used when authenticating to the third-party service." }, "httpMethod": { "type": "string", "title": "HTTP Method Types", "description": "The HTTP method to use for the request.", "enum": ["GET", "POST"], "default": "POST" }, "secureVariables": { "type": "array", "title": "Secure Authentication Variables", "description": "Variables to use within the authentication request definition.", "items": { "type": "object", "required": [], "properties": { "keyName": { "type": "string", "title": "Key Name", "description": "The name of the key that saves the secret value, such as 'username'.", "minLength": 1 }, "secretValue": { "type": "string", "title": "Secret Value", "description": "Secret Value used for authentication, such as a password.", "hints": ["secret"], "minLength": 1 }, "secretValueVaultLabel": { "type": "string", "title": "Secret Value from Vault", "description": "The Fusion managed secret to use for Secret Value.", "hints": ["enumUrl:/apps-manager/vault/labels?type=secret", "vaultref", "vaultvaluetarget:secretValue"] } } } }, "headers": { "type": "array", "title": "Request Headers", "description": "Headers to use within the authentication request.", "items": { "type": "object", "required": [], "properties": { "headerKey": { "type": "string", "title": "Header Key" }, "headerValue": { "type": "string", "title": "Header Value" } } } }, "queries": { "type": "array", "title": "Request Queries", "description": "Query parameters to include in the authentication request.", "items": { "type": "object", "required": [], "properties": { "queryKey": { "type": "string", "title": "Query Key" }, "queryValue": { "type": "string", "title": "Query Value" } } } }, "bodyAuthRequest": { "type": "string", "title": "Request Body", "description": "The body of the authentication request in JSON format.", "hints": ["code", "lengthy"] } }, "hints": ["hidden"] }, "oAuth": { "type": "object", "title": "OAuth", "description": "OAuth Authentication Properties", "required": ["oAuthServiceUrl", "oAuthValuesLocation", "accessTokenPath", "oAuthEndpoint", "headerAuthorizationTemplate", "httpMethod"], "properties": { "oAuthServiceUrl": { "type": "string", "title": "OAuth Service URL", "description": "The URL of the service used to retrieve OAuth data." }, "oAuthEndpoint": { "type": "string", "title": "OAuth Endpoint path", "description": "Endpoint path for OAuth data. For example: /services/oauth2/token" }, "httpMethod": { "type": "string", "title": "HTTP Method", "description": "HTTP method to be used in the request.", "enum": ["POST", "GET"], "default": "POST" }, "plainVariables": { "type": "array", "title": "Plain Authentication Variables", "description": "Plain text variables to include in the authentication request definition.", "items": { "type": "object", "required": [], "properties": { "keyName": { "type": "string", "title": "Key Name", "description": "The name of the key used to store the value, such as username.", "minLength": 1 }, "keyValue": { "type": "string", "title": "Key Value", "description": "The value used for the key. For example: email@email.com.", "minLength": 1 } } } }, "secureVariables": { "type": "array", "title": "Secure Authentication Variables", "description": "Secret variables to use within the authentication request definition.", "items": { "type": "object", "required": [], "properties": { "keyName": { "type": "string", "title": "Key Name", "description": "The name of the key that saves the secret value, such as 'username'.", "minLength": 1 }, "secretValue": { "type": "string", "title": "Secret Value", "description": "Secret Value used for authentication, such as a password.", "hints": ["secret"], "minLength": 1 }, "secretValueVaultLabel": { "type": "string", "title": "Secret Value from Vault", "description": "The Fusion managed secret to use for Secret Value.", "hints": ["enumUrl:/apps-manager/vault/labels?type=secret", "vaultref", "vaultvaluetarget:secretValue"] } } } }, "headerAuthorizationTemplate": { "type": "string", "title": "Header Authorization Template", "description": "A template for inserting authorization details, such as Bearer ${LW_ACCESS_TOKEN}" }, "accessTokenPath": { "type": "string", "title": "Access token path", "description": "The path to the access token within the response. This property also accepts JsonPath expressions, such as objects, objects[*] or $.objects[*]. " }, "oAuthValuesLocation": { "type": "string", "title": "OAuth Values Location", "description": "Specify if OAuth key-value pair should be sent in request parameters, headers, the request body as JSON, or as URL-encoded data.", "enum": ["REQUEST PARAM", "HEADER", "BODY JSON", "URL ENCODED"], "default": "REQUEST PARAM" } } } } }, "serviceEndpoints": { "type": "array", "title": "Service Endpoints (deprecated)", "description": "A list of request configurations for service endpoints. This property is deprecated.", "items": { "type": "object", "required": ["rootRequest"], "properties": { "rootRequest": { "type": "object", "title": "Root Request Configuration", "description": "The request to retrieve the root data objects.", "required": ["dataRequest"], "properties": { "dataRequest": { "type": "object", "title": "Data Request", "description": "Configures the endpoint request.", "required": ["endpoint", "httpMethod"], "properties": { "endpoint": { "type": "string", "title": "Endpoint path. For example: /rest/api/object" }, "httpMethod": { "type": "string", "title": "HTTP Method", "description": "HTTP method to be used in the request.", "enum": ["GET", "POST"], "default": "GET" }, "headers": { "type": "array", "title": "Headers", "description": "HTTP headers to be included in the request.", "items": { "type": "object", "required": [], "properties": { "headerKey": { "type": "string", "title": "Header Key" }, "headerValue": { "type": "string", "title": "Header Value" } } } }, "queries": { "type": "array", "title": "Queries", "description": "Query parameters to be included in the request URL.", "items": { "type": "object", "required": [], "properties": { "queryKey": { "type": "string", "title": "Query Key" }, "queryValue": { "type": "string", "title": "Query Value" } } } }, "bodyRequest": { "type": "string", "title": "Body", "description": "Request body in JSON format.", "hints": ["code", "lengthy"] }, "pagination": { "type": "object", "title": "Pagination Options", "description": "Choose only one. If none of them are chosen, pagination will not be performed.", "required": [], "properties": { "paginationByNextUrl": { "type": "object", "title": "Pagination by Next Page URL", "description": "The URL will be extracted from the response and used to fetch the next page.", "required": ["paginationKey"], "properties": { "paginationKey": { "type": "string", "title": "Next Page URL Key", "description": "Key to access to the nextPageUrl in the response. This property also accepts JsonPath expressions. If the key is nested, use dot notation. Example: list.nextPageUrl or $.list.nextPageUrl" } } }, "paginationByBatchSize": { "type": "object", "title": "Pagination by Batch Size", "description": "Provide the startIndex and batchSize values.", "required": ["paginationStopConditionValue", "paginationStopConditionKey", "batchSize", "indexStart"], "properties": { "indexStart": { "type": "number", "title": "Index Start", "description": "The starting index value to be mapped to the ${LW_INDEX_START} variable configured in the data request's query parameters. This value will automatically increase during crawl-time. How to use: configure 'Index Start: 0' and configure queryParam 'startAt=${LW_INDEX_START}', where the 'startAt' is defined by the REST source.", "default": 0, "maximum": 2147483647, "exclusiveMaximum": false, "minimum": 0, "exclusiveMinimum": false, "multipleOf": 1 }, "batchSize": { "type": "number", "title": "Batch Size", "description": "The number of objects retrieved per page to be mapped to the ${LW_BATCH_SIZE} variable configured in the data request's query parameters. How to use: configure 'Batch Size: 25' and configure queryParam 'maxSize=${LW_BATCH_SIZE}', where 'maxSize' is defined by the REST source", "default": 0, "maximum": 2147483647, "exclusiveMaximum": false, "minimum": 0, "exclusiveMinimum": false, "multipleOf": 1 }, "paginationStopConditionKey": { "type": "string", "title": "Stop Condition Key", "description": "Key that specifies the condition for stopping pagination. This property also accepts JsonPath expressions." }, "paginationStopConditionValue": { "type": "string", "title": "Stop Condition Value", "description": "Value for the condition that must be met to stop pagination. If the condition value refers to a list, the stop condition must be []." } } } } } } }, "rootResponseMapping": { "type": "object", "title": "Root Response Mapping", "description": "Defines the mapping between the response and the data objects to be indexed.", "required": [], "properties": { "dataPath": { "type": "string", "title": "Data Path", "description": "The path to access a specific data object within a response. For example, to extract a list of elements named with key 'objects', the DataPath would be 'objects' and each element will be indexed as a separate Solr document. If not provided, the entire response body will be indexed as a single Solr document. This property also accepts JsonPath expressions, such as objects, objects[*] or $.objects[*]. " }, "dataId": { "type": "string", "title": "Data ID", "description": "The identifier key for the data objects extracted with the 'Data Path', whose value will be the Solr document's ID. If not provided, a random UUID will be used. This property also accepts JsonPath expressions." }, "binaryResponse": { "type": "boolean", "title": "Send as Binary Response", "description": "If true, the response will be sent as Binary Data to Fusion, properties 'Data ID' and 'Data Path' will be ignored, and pagination will not be performed.", "default": false } } } } }, "childrenRequests": { "type": "array", "title": "\nChild Request Configuration", "description": "The list of requests to retrieve additional information for the root data objects. The child requests will be performed per each root data object extracted with the root request configuration.", "items": { "type": "object", "required": ["dataRequest", "childResponseMapping"], "properties": { "dataRequest": { "type": "object", "title": "Data Request", "description": "Configuration of the endpoint request.", "required": ["endpoint", "httpMethod"], "properties": { "endpoint": { "type": "string", "title": "Endpoint path. For example: /rest/api/object" }, "httpMethod": { "type": "string", "title": "HTTP Method", "description": "HTTP method to be used in the request.", "enum": ["GET", "POST"], "default": "GET" }, "headers": { "type": "array", "title": "Headers", "description": "HTTP headers to be included in the request.", "items": { "type": "object", "required": [], "properties": { "headerKey": { "type": "string", "title": "Header Key" }, "headerValue": { "type": "string", "title": "Header Value" } } } }, "queries": { "type": "array", "title": "Queries", "description": "Query parameters to be included in the request URL.", "items": { "type": "object", "required": [], "properties": { "queryKey": { "type": "string", "title": "Query Key" }, "queryValue": { "type": "string", "title": "Query Value" } } } }, "bodyRequest": { "type": "string", "title": "Body", "description": "Request body in JSON format.", "hints": ["code", "lengthy"] }, "pagination": { "type": "object", "title": "Pagination Options", "description": "Choose only one. If none of them are chosen, pagination will not be performed.", "required": [], "properties": { "paginationByNextUrl": { "type": "object", "title": "Pagination by Next Page URL", "description": "The URL will be extracted from the response and used to fetch the next page.", "required": ["paginationKey"], "properties": { "paginationKey": { "type": "string", "title": "Next Page URL Key", "description": "Key to access to the nextPageUrl in the response. This property also accepts JsonPath expressions. If the key is nested, use dot notation. Example: list.nextPageUrl or $.list.nextPageUrl" } } }, "paginationByBatchSize": { "type": "object", "title": "Pagination by Batch Size", "description": "Provide the startIndex and batchSize values.", "required": ["paginationStopConditionValue", "paginationStopConditionKey", "batchSize", "indexStart"], "properties": { "indexStart": { "type": "number", "title": "Index Start", "description": "The starting index value to be mapped to the ${LW_INDEX_START} variable configured in the data request's query parameters. This value will automatically increase during crawl-time. How to use: configure 'Index Start: 0' and configure queryParam 'startAt=${LW_INDEX_START}', where the 'startAt' is defined by the REST source.", "default": 0, "maximum": 2147483647, "exclusiveMaximum": false, "minimum": 0, "exclusiveMinimum": false, "multipleOf": 1 }, "batchSize": { "type": "number", "title": "Batch Size", "description": "The number of objects retrieved per page to be mapped to the ${LW_BATCH_SIZE} variable configured in the data request's query parameters. How to use: configure 'Batch Size: 25' and configure queryParam 'maxSize=${LW_BATCH_SIZE}', where 'maxSize' is defined by the REST source", "default": 0, "maximum": 2147483647, "exclusiveMaximum": false, "minimum": 0, "exclusiveMinimum": false, "multipleOf": 1 }, "paginationStopConditionKey": { "type": "string", "title": "Stop Condition Key", "description": "Key that specifies the condition for stopping pagination. This property also accepts JsonPath expressions." }, "paginationStopConditionValue": { "type": "string", "title": "Stop Condition Value", "description": "Value for the condition that must be met to stop pagination. If the condition value refers to a list, the stop condition must be []." } } } } } } }, "childResponseMapping": { "type": "object", "title": "Child Response Mapping", "description": "Defines the mapping between the child response and child data objects to be indexed.", "required": ["parentIdKey"], "properties": { "parentIdKey": { "type": "string", "title": "Parent Data Key", "description": "Key to extract data from root/parent response for subsequent request. The extracted value will be used to replace the ${LW_PARENT_DATA_KEY} variable in the child request configuration (endpoint, query params or body).For example, endpoint: /api/path/${LW_PARENT_DATA_KEY}/additionalInfo." }, "parentNestedData": { "type": "string", "title": "Parent Nested Data Path", "description": "The path to a nested object within the Parent Data. When this is set, the 'Parent Data Key' should point to the 'id' or 'key' of the nested object. If the nested object is a list, this will be iterated to perform a request per each item." }, "childDataPath": { "type": "string", "title": "Child Data Path", "description": "The path to access a specific object within a child response. For example, to access a list of elements named with key 'objects', the childDataPath would be 'objects'. If not provided, the entire response body will be indexed." }, "childDataId": { "type": "string", "title": "Child Data ID", "description": "The identifier key for the child data object, used as the Solr document's ID. Use this when 'Custom Solr Field' is empty, otherwise the Solr document's ID will be a random UUID." }, "customSolrField": { "type": "string", "title": "Custom Solr Field", "description": "The field to use to store the child data within the root data objects. If not set, the child data object will be indexed as individual Solr documents." }, "binaryResponse": { "type": "boolean", "title": "Send as Binary Response", "description": "If true, the response will be sent as Binary Data to Fusion, properties 'Child Data ID' and 'Child Data Path' will be ignored, and pagination will not be performed.", "default": false } } } } } } } } }, "requestConfigurations": { "type": "array", "title": "List of Requests Configuration", "description": "A hierarchical list of requests to fetch data from the REST source.", "items": { "type": "object", "required": ["request"], "properties": { "request": { "type": "object", "title": "Request Configuration", "description": "Specifies the request configuration used to retrieve the data objects.", "required": ["linkRequest", "requestConfiguration"], "properties": { "linkRequest": { "type": "object", "title": "Parent-Child Request Link", "description": "Establishes the hierarchical relationship between parent and child requests.", "required": ["objectType"], "properties": { "objectType": { "type": "string", "title": "Object Type", "description": "Assign a unique name to identify the request. Use a descriptive name that reflects the type of data this request will retrieve." }, "parentObjectType": { "type": "string", "title": "Parent Object Type", "description": "References an existing Object Type to create a parent-child hierarchy, where the current request becomes the child of the specified Parent Object Type. If blank, the request is considered a parent or root request." } } }, "requestConfiguration": { "type": "object", "title": "Data Request", "description": "Configures the endpoint for the request. If configuring a child request, use the variable ${LW_PARENT_DATA_KEY} in the endpoint, query parameters, or body and this variable will be replaced with the value specified in 'Response Handling -> Parent Data Key'.", "required": ["endpoint", "httpMethod"], "properties": { "endpoint": { "type": "string", "title": "Endpoint path. For example: /rest/api/object" }, "httpMethod": { "type": "string", "title": "HTTP Method", "description": "HTTP method to be used in the request.", "enum": ["GET", "POST"], "default": "GET" }, "headers": { "type": "array", "title": "Headers", "description": "HTTP headers to be included in the request.", "items": { "type": "object", "required": [], "properties": { "headerKey": { "type": "string", "title": "Header Key" }, "headerValue": { "type": "string", "title": "Header Value" } } } }, "queries": { "type": "array", "title": "Queries", "description": "Query parameters to be included in the request URL.", "items": { "type": "object", "required": [], "properties": { "queryKey": { "type": "string", "title": "Query Key" }, "queryValue": { "type": "string", "title": "Query Value" } } } }, "bodyRequest": { "type": "string", "title": "Body", "description": "Request body in JSON format.", "hints": ["code", "lengthy"] }, "pagination": { "type": "object", "title": "Pagination Options", "description": "Choose only one. If none of them are chosen, pagination will not be performed.", "required": [], "properties": { "paginationByNextUrl": { "type": "object", "title": "Pagination by Next Page URL", "description": "The URL will be extracted from the response and used to fetch the next page.", "required": ["paginationKey"], "properties": { "paginationKey": { "type": "string", "title": "Next Page URL Key", "description": "Key to access to the nextPageUrl in the response. This property also accepts JsonPath expressions. If the key is nested, use dot notation. Example: list.nextPageUrl or $.list.nextPageUrl" } } }, "paginationByBatchSize": { "type": "object", "title": "Pagination by Batch Size", "description": "Provide the startIndex and batchSize values.", "required": ["paginationStopConditionValue", "paginationStopConditionKey", "batchSize", "indexStart"], "properties": { "indexStart": { "type": "number", "title": "Index Start", "description": "The starting index value to be mapped to the ${LW_INDEX_START} variable configured in the data request's query parameters. This value will automatically increase during crawl-time. How to use: configure 'Index Start: 0' and configure queryParam 'startAt=${LW_INDEX_START}', where the 'startAt' is defined by the REST source.", "default": 0, "maximum": 2147483647, "exclusiveMaximum": false, "minimum": 0, "exclusiveMinimum": false, "multipleOf": 1 }, "batchSize": { "type": "number", "title": "Batch Size", "description": "The number of objects retrieved per page to be mapped to the ${LW_BATCH_SIZE} variable configured in the data request's query parameters. How to use: configure 'Batch Size: 25' and configure queryParam 'maxSize=${LW_BATCH_SIZE}', where 'maxSize' is defined by the REST source", "default": 0, "maximum": 2147483647, "exclusiveMaximum": false, "minimum": 0, "exclusiveMinimum": false, "multipleOf": 1 }, "paginationStopConditionKey": { "type": "string", "title": "Stop Condition Key", "description": "Key that specifies the condition for stopping pagination. This property also accepts JsonPath expressions." }, "paginationStopConditionValue": { "type": "string", "title": "Stop Condition Value", "description": "Value for the condition that must be met to stop pagination. If the condition value refers to a list, the stop condition must be []." } } } } } } }, "recursiveRequest": { "type": "boolean", "title": "Enable Recursive Request", "description": "When enabled, extra requests are performed to retrieve nested objects within the initial objects found by the current request. For example, if the current request gets a list of folders, extra requests are made per folder to retrieve nested folders. This process will continue until no more nested folders are found.", "default": false }, "responseConfiguration": { "type": "object", "title": "Response Handling", "description": "Configures how to parse the response.", "required": [], "properties": { "dataPath": { "type": "string", "title": "Data Path", "description": "The path to access a specific data object within a response. For example, to extract a list of elements named with key 'objects', the DataPath would be 'objects', with each element indexed as a separate Solr document. If not provided, the entire response body will be indexed as a single Solr document. This property also accepts JsonPath expressions, such as objects, objects[*] or $.objects[*]. " }, "dataId": { "type": "string", "title": "Data ID", "description": "The identifier key for the data objects extracted with 'Data Path'. This value will be used to build the Solr document's ID. If not provided, a random UUID will be used. This property also accepts JsonPath expressions." }, "parentIdKey": { "type": "string", "title": "Parent Data Key", "description": "Set only if configuring a child request. Map to a key from the parent object, whose value will be used to replace the ${LW_PARENT_DATA_KEY} variable in the child request configuration (endpoint, query parameters, or body).For example, endpoint: /api/path/${LW_PARENT_DATA_KEY}/additionalInfo." }, "parentNestedData": { "type": "string", "title": "Parent Nested Data Path", "description": "Set only if configuring a child request. Specifies the path to a nested object within the Parent Data. When this is set, the 'Parent Data Key' should point to the 'id' or 'key' of the nested object. If the nested object is a list, this will be iterated to perform a request per each item.", "hints": ["advanced"] }, "customSolrField": { "type": "string", "title": "Custom Solr Field", "description": "Set only if configuring a child request. This field is used to store the child data within the parent data objects. If not set, the child data object will be indexed as an individual Solr document.", "hints": ["advanced"] }, "binaryResponse": { "type": "boolean", "title": "Parse as Binary Data", "description": "Enable to send the response as Binary Data to the Fusion Parsers. When enabled, properties 'DataPath, DataId, CustomSolrField' will be ignored, pagination will not be performed, and children requests will be ignored.", "default": false } } }, "skipIndexation": { "type": "boolean", "title": "Skip Indexation", "description": "When enabled, the response is not indexed. Useful when requests of objects are needed only to discover additional info (child objects) without indexing the object itself.For example, if a request is used to find folders along with a child request to retrieve files found within those folders, enable 'Skip Indexation' to index the files but not the folders", "default": false } } } } } }, "limitDocuments": { "type": "object", "title": "Limit Documents", "description": "Restrictions to be placed on documents fetched.", "required": [], "properties": { "excludeByRegex": { "type": "array", "title": "Exclude by RegEx", "description": "Exclude documents by Regular Expression", "items": { "type": "object", "required": [], "properties": { "key": { "type": "string", "title": "Key", "description": "Specifies the field name of the object to exclude. This property also accepts JsonPath expressions. For example, objects.nested.path", "minLength": 1 }, "value": { "type": "string", "title": "RegEx value to exclude", "description": "Regular expression value that matches the object to exclude.", "minLength": 1 } } } }, "filterByFileSize": { "type": "object", "title": "Exclude by File Size", "description": "Filter documents that are outside of the minimum/maximum size limits.", "required": [], "properties": { "key": { "type": "string", "title": "Key", "description": "Specifies the field name of the object containing the file size. This property also supports JsonPath expressions, such as objects.nested.path", "minLength": 1 }, "minimumSize": { "type": "number", "title": "Minimum File Size", "description": "Excludes files with sizes smaller than the configured value.", "default": 1, "maximum": 9223372036854776000, "exclusiveMaximum": false, "minimum": -9223372036854776000, "exclusiveMinimum": false, "multipleOf": 1 }, "maximumSize": { "type": "number", "title": "Maximum File Size", "description": "Excludes files with sizes larger than the configured value. Set to -1 when there is no limit.", "default": -1, "maximum": 9223372036854776000, "exclusiveMaximum": false, "minimum": -9223372036854776000, "exclusiveMinimum": false, "multipleOf": 1 } } } } }, "httpClientProperties": { "type": "object", "title": "HTTP Client Settings", "description": "A set of options for configuring the HTTP client.", "required": [], "properties": { "readTimeoutMs": { "type": "number", "title": "", "default": 300000, "maximum": 600000, "exclusiveMaximum": false, "minimum": 0, "exclusiveMinimum": false, "multipleOf": 1 }, "connectTimeoutMs": { "type": "number", "title": "", "default": 60000, "maximum": 300000, "exclusiveMaximum": false, "minimum": 0, "exclusiveMinimum": false, "multipleOf": 1 } } }, "retryProperties": { "type": "object", "title": "Retry Properties", "description": "Configure the retry behavior for failed requests.", "required": [], "properties": { "retryCount": { "type": "number", "title": "Retry Count", "description": "Number of attempts a request will be retried", "default": 3, "maximum": 36, "exclusiveMaximum": false, "minimum": 1, "exclusiveMinimum": false, "multipleOf": 1 }, "maxDelayTime": { "type": "number", "title": "Maximum Delay Time", "description": "The maximum wait time in milliseconds between retries.", "default": 60000, "maximum": 60000, "exclusiveMaximum": false, "minimum": 1000, "exclusiveMinimum": false, "multipleOf": 1 } } } } }, "coreProperties": { "type": "object", "title": "Core Properties", "description": "Common behavior and performance settings.", "required": [], "properties": { "fetchSettings": { "type": "object", "title": "Fetch Settings", "description": "System level settings for controlling fetch behavior and performance.", "required": [], "properties": { "fetchRequestCheckInterval": { "type": "number", "title": "Fetch request check interval(ms)", "description": "The amount of time to wait before check if a request is done", "default": 15000, "hints": ["hidden"], "maximum": 500000, "exclusiveMaximum": false, "minimum": 1000, "exclusiveMinimum": false, "multipleOf": 1 }, "numFetchThreads": { "type": "number", "title": "Fetch Threads", "description": "Maximum number of fetch threads; defaults to 5.This setting controls the number of threads that call the Connectors fetch method.Higher values can, but not always, help with overall fetch performance.", "default": 5, "maximum": 500, "exclusiveMaximum": false, "minimum": 1, "exclusiveMinimum": false, "multipleOf": 1 }, "fetchResponseScheduledTimeout": { "type": "number", "title": "Fetch response scheduled timeout(ms)", "description": "The maximum amount of time for a response to be scheduled. The task will be canceled if this setting is exceeded.", "default": 300000, "maximum": 500000, "exclusiveMaximum": false, "minimum": 1000, "exclusiveMinimum": false, "multipleOf": 1 }, "indexingThreads": { "type": "number", "title": "Index Subscription Threads", "description": "Maximum number of indexing threads; defaults to 4.This setting controls the number of threads in the indexing service used for processing content documents emitted by this datasource.Higher values can sometimes help with overall fetch performance.", "default": 4, "maximum": 10, "exclusiveMaximum": false, "minimum": 1, "exclusiveMinimum": false, "multipleOf": 1 }, "pluginInstances": { "type": "number", "title": "Number of plugin instances for distributed fetching", "description": "Maximum number of plugin instances for distributed fetching. Only specified number of plugin instanceswill do fetching. This is useful for distributing load between different instances.", "default": 0, "maximum": 500, "exclusiveMaximum": false, "minimum": 0, "exclusiveMinimum": false, "multipleOf": 1 }, "fetchItemQueueSize": { "type": "number", "title": "Fetch Item Queue Size", "description": "Size of the fetch item queue.Larger values result in increased memory usage, but potentially higher performance.Default is 10k.", "default": 10000, "hints": ["hidden"], "maximum": 500000, "exclusiveMaximum": false, "minimum": 1, "exclusiveMinimum": false, "multipleOf": 1 }, "fetchResponseCompletedTimeout": { "type": "number", "title": "Fetch response completion timeout(ms)", "description": "The maximum amount of time for a response to be completed. If exceeded, the task will be retried if the job is still running", "default": 300000, "hints": ["hidden"], "maximum": 600000, "exclusiveMaximum": false, "minimum": 1, "exclusiveMinimum": false, "multipleOf": 1 }, "indexingInactivityTimeout": { "type": "number", "title": "Indexing inactivity timeout(seconds)", "description": "The maximum amount of time to wait for indexing results (in seconds). If exceeded, the job will fail with an indexing inactivity timeout.", "default": 86400, "maximum": 691200, "exclusiveMaximum": false, "minimum": 60, "exclusiveMinimum": false, "multipleOf": 1 }, "pluginInactivityTimeout": { "type": "number", "title": "Plugin inactivity timeout(seconds)", "description": "The maximum amount of time to wait for plugin activity (in seconds). If exceeded, the job will fail with a plugin inactivity timeout.", "default": 600, "maximum": 691200, "exclusiveMaximum": false, "minimum": 60, "exclusiveMinimum": false, "multipleOf": 1 }, "indexMetadata": { "type": "boolean", "title": "Index metadata", "description": "When enabled the metadata of skipped items will be indexed to the content collection.", "default": false }, "indexContentFields": { "type": "boolean", "title": "Index content fields", "description": "When enabled, content fields will be indexed to the crawl-db collection.", "default": false }, "asyncParsing": { "type": "boolean", "title": "Async Parsing", "description": "When enabled, content will be indexed asynchronously.", "default": false } } }, "strayDeletionSettings": { "type": "object", "title": "Stray Content Deletion Settings", "description": "System level settings for controlling stray content deletion behavior", "required": [], "properties": { "enableStrayDeletion": { "type": "boolean", "title": "Enable Stray Deletion", "description": "When enabled, items not re-encountered in the current crawl are deleted from the content collection. Defaults to true.", "default": true }, "circuitBreakerSettings": { "type": "object", "title": "Circuit Breaker Settings", "description": "Settings that control the circuit breaker, which prevents mass deletion when an unusually high percentage of items are identified as stray in a crawl.", "required": [], "properties": { "percentageThreshold": { "type": "number", "title": "Stray Deletion Threshold (%)", "description": "Maximum percentage of indexed items that may be deleted as stray in a crawl. If the total stray item exceeds this percentage, the circuit breaker blocks the deletion to prevent accidental data loss. Accepts values from 0 to 100. Defaults to 80.", "default": 80, "maximum": 100, "exclusiveMaximum": false, "minimum": 0, "exclusiveMinimum": false, "multipleOf": 1 }, "enableCircuitBreaker": { "type": "boolean", "title": "Enable Circuit Breaker", "description": "When enabled, stray deletion is blocked if the percentage of items to be deleted exceeds the configured threshold. Disable only if unconditional stray deletion is required regardless of volume. Defaults to true.", "default": true } } } } }, "skipConfigValidation": { "type": "boolean", "title": "Skip Validation", "description": "Enable to skip configuration validation when it takes too long and causes timeout issue", "default": false } }, "hints": ["advanced"] } }, "category": "Repository", "categoryPriority": 1 }; export const SchemaParamFields = ({schema}) => { const sanitize = str => { if (typeof str !== "string") return str; return str.replace(/^"(.*)"$/s, "$1").replace(/\\/g, "").replace(/"/g, "'"); }; const formatDescription = str => { const s = sanitize(str); return (/[.!?]\)*$/).test(s) ? s : `${s}.`; }; const {description, properties = {}, required: requiredProps = []} = schema; const visibleProps = useMemo(() => Object.entries(properties).filter(([, prop]) => !prop.hints?.includes("hidden")), [properties]); return
{description &&

{formatDescription(description)}

} {visibleProps.map(([name, prop]) => { const isRequired = requiredProps.includes(name); const hasDefault = prop.default !== undefined; const rawDefault = prop.default; const isComplexDefault = hasDefault && (typeof rawDefault === "object" || typeof rawDefault === "string" && (rawDefault.length > 20 || rawDefault.includes('"'))); const fieldProps = { key: name, body: prop.title || name, type: prop.type, ...prop.title && ({ post: [<>API property: {name}] }), ...isRequired && ({ required: true }), ...!isComplexDefault && hasDefault ? { default: sanitize(String(rawDefault)) } : {} }; const isObject = prop.type === "object" && prop.properties; const isArrayOfObjects = prop.type === "array" && prop.items?.type === "object" && prop.items.properties; return {prop.description &&

{formatDescription(prop.description)}

} {isComplexDefault &&

Default: 

                  
                    {JSON.stringify(rawDefault, null, 2)}
} {isArrayOfObjects &&

Object attributes: 

                
                  {'{\n'}
                  {Object.entries(prop.items.properties).map(([iname, iprop]) => <>
                      {`  ${iname}`}
                      {prop.items?.required?.includes(iname) &&  required}
                      {`: {\n    display name: ${sanitize(iprop.title || '')}\n    type: ${iprop.type}\n  }\n`}
                    )}
                  {'}'}
} {isObject && } ; })}
; }; export const LwTemplate = ({title = "Key questions to get you started", icon = "sparkles", cta = "Powered by Agent Studio", linkHref = "https://lucidworks.com/demo/?utm_source=docs&utm_medium=referral&utm_campaign=docs_cta_ai"}) => { const [isLoaded, setIsLoaded] = useState(false); useEffect(() => { const timer = setTimeout(() => { setIsLoaded(true); }, 500); return () => clearTimeout(timer); }, []); return
{isLoaded && ` }} />} Powered by Lucidworks Agent Studio
; }; [localhost link]: http://localhost:3000/docs/fusion-connectors/connectors/rest-v2/overview [mintlify link]: https://doc.lucidworks.com/docs/fusion-connectors/connectors/rest-v2/overview [old doc.lw link]: https://doc.lucidworks.com/fusion-connectors/2fe4q4 * **Latest version:** v1.1.0 * **Compatible with Fusion version:** 5.9.0 and later The REST V2 connector enables users to crawl content by exposing its data through a REST API. It can be configured to communicate with a wide selection of external datasources by making API calls and indexing the responses. As an out-of-the-box V2 connector, it provides a low-code user experience for indexing data from REST API-compatible sources. This connector is the alternative solution for various deprecated and removed V1 connectors. Key concepts: * The REST V2 connector is a new approach to Fusion connectors: one connector that can index data from many different datasources. * The connector is powered by recipes, which are JSON configurations for the REST V2 connector that are customized for a specific datasource. * A supported datasource will have a corresponding recipe. * Over time, the deprecated V1 connectors may be replaced by comparable REST V2 connector recipes. REST V2 connector features: * Indexes REST API-compatible datasources for data integration. * Supports initial full crawl and re-full crawl and object mapping. * Remote framework and pagination are supported. * Supports basic and OAuth authentication methods. * Enhances the datasource retrieval capability with a globally standardized format connector. * Saves DevOps time by using a preconfigured connector. ## How REST V2 is different from other connectors Most connectors are developed solely for a specific datasource. The REST V2 connector is able to work any datasources that has a supported recipe. This gives the REST V2 connector a great amount of flexibility and enhances its usefulness as part of the Lucidworks connectors offerings. An API call is made to the REST V2 connector containing a [recipe](https://github.com/lucidworks/rest-configurations) as the body. API calls to set up the REST V2 connector go through the [Connector Datasources API](/api-reference/datasource-configuration-v2-api/list-all-v2-datasources). This article lists and describes the Fusion properties used by REST V2. The recommended method of populating these fields is through the use of a recipe. For information on how to get a recipe from the public GitHub repository into Fusion, see [How to use REST V2 recipes](/docs/fusion-connectors/connectors/rest-v2/recipes/overview). The REST V2 connector performs a full crawl each time the connectors runs, as incremental crawling is not supported. When changes are made to the source, such as adding, deleting, or modifying content, those changes will reflect in the index after the next crawl is performed. You can check for new documents in the index by navigating to **Querying > Query Workbench** in the Fusion UI. Lucidworks no longer updates, maintains, or releases V1 connectors. A benefit of using this remote V2 connector over the classic V1 connectors is this V2 connector is regularly improved and updated, while the classic connectors are not. The REST V2 connector is replacing V1 connectors using comparable recipes. You can learn more about the benefits of V2 connectors by reading the [differences between V1 and V2 connectors](/docs/fusion-connectors/concepts/v1-v2-connectors). ## Recipes Recipes are preconfigured [datasource configurations](/docs/fusion-connectors/connectors/rest-v2/recipes/overview) that can be loaded into Fusion for quicker testing and setup of connectors. Parameter values are already entered into recipes for quick population of the required fields. Recipes are created in a JSON format and contain all of the parameters required to get a connector up and running. You can configure a connector using a recipe by issuing a POST request with the JSON as the body. Any minor adjustments required can be made directly in the JSON and sent to Fusion, where it will show in the UI. After the recipe has been sent to Fusion, you can also go into the Fusion UI and make changes. ## Recipe repository The REST V2 connector uses a [public GitHub repository](https://github.com/lucidworks/rest-configurations) to store recipes. Here you will find all of the recipes that have been created and ready for use. The recipes are JSON files that can be added as the body in Postman requests or appended as the body in cURL calls. The parameter values can then be changed as necessary to match the data in the way you want it to be indexed. The repository is public and treated as an open source resource where everyone is welcome to contribute. Recipes are not considered a standard Lucidworks product. As such, Lucidworks does not guarantee that a recipe will be maintained or updated after it is created. If you encounter any issues or bugs, please [report them in GitHub](https://docs.github.com/en/issues/tracking-your-work-with-issues/using-issues/creating-an-issue#creating-an-issue-from-a-repository) with detailed information to aid in troubleshooting. ## Remote connectors V2 connectors support [running remotely](/docs/fusion-connectors/developers/remote-v2-connectors) in Fusion versions 5.7.1 and later. If you need to index data from behind a firewall, you can configure a V2 connector to run remotely on-premises using TLS-enabled gRPC. ## Prerequisites Before you can set up an on-prem V2 connector, you must configure the egress from your network to allow HTTP/2 communication into the Fusion cloud. You can use a [forward proxy server](#egress-and-proxy-server-configuration) to act as an intermediary between the connector and Fusion. The following is required to run V2 connectors remotely: * The [plugin zip file and the connector-plugin-standalone JAR](https://plugins.lucidworks.com/). * A configured connector backend gRPC endpoint. * Username and password of a user with a `remote-connectors` or `admin` role. * If the host where the remote connector is running is not configured to trust the server’s TLS certificate, you must configure the file path of the trust certificate collection. If your version of Fusion doesn’t have the `remote-connectors` role by default, you can create one. No API or UI permissions are required for the role. ## Connector compatibility Only V2 connectors are able to run remotely on-premises. You also need the remote connector client JAR file that matches your Fusion version. You can download the latest files at [V2 Connectors Downloads](/docs/fusion-connectors/downloads/v2-connectors-downloads). Whenever you upgrade Fusion, you must also update your remote connectors to match the new version of Fusion. The gRPC connector backend is not supported in Fusion environments deployed on AWS. ## System requirements The following is required for the on-prem host of the remote connector: * (Fusion 5.9.0-5.9.10) JVM version 11 * (Fusion 5.9.11) JVM version 17 * Minimum of 2 CPUs * 4GB Memory Note that memory requirements depend on the number and size of ingested documents. ## Enable backend ingress In your `values.yaml` file, configure this section as needed: ```yaml theme={"dark"} ingress: enabled: false pathtype: "Prefix" path: "/" #host: "ingress.example.com" ingressClassName: "nginx" # Fusion 5.9.6 only tls: enabled: false certificateArn: "" # Enable the annotations field to override the default annotations #annotations: "" ``` * Set `enabled` to `true` to enable the backend ingress. * Set `pathtype` to `Prefix` or `Exact`. * Set `path` to the path where the backend will be available. * Set `host` to the host where the backend will be available. * In Fusion 5.9.6 *only*, you can set `ingressClassName` to one of the following: * `nginx` for Nginx Ingress Controller * `alb` for AWS Application Load Balancer (ALB) * Configure TLS and certificates according to your CA’s procedures and policies. TLS must be enabled in order to use AWS ALB for ingress. ## Connector configuration example ```yaml theme={"dark"} kafka-bridge: target: mynamespace-connectors-backend.lucidworkstest.com:443 # mandatory plain-text: false # optional, false by default. proxy-server: # optional - needed when a forward proxy server is used to provide outbound access to the standalone connector host: host port: some-port user: user # optional password: password # optional trust: # optional - needed when the client's system doesn't trust the server's certificate cert-collection-filepath: path1 proxy: # mandatory fusion-proxy user: admin password: password123 url: https://fusiontest.com/ # needed only when the connector plugin requires blob store access plugin: # mandatory path: ./fs.zip type: #optional - the suffix is added to the connector id suffix: remote ``` ### Minimal example ```yaml theme={"dark"} kafka-bridge: target: mynamespace-connectors-backend.lucidworkstest.com:443 proxy: user: admin password: "password123" plugin: path: ./testplugin.zip ``` ### Logback XML configuration file example ```xml theme={"dark"} %d - %-5p [%t:%C{3.}@%L] - %m{nolookups}%n utf8 ${LOGDIR:-.}/connector.log ${LOGDIR:-.}/connector-%d{yyyy-MM-dd}.%i.log.gz 50MB 10GB %d - %-5p [%t:%C{3.}@%L] - %m{nolookups}%n utf8 ``` ## Run the remote connector ```java theme={"dark"} java [-Dlogging.config=[LOGBACK_XML_FILE]] \ -jar connector-plugin-client-standalone.jar [YAML_CONFIG_FILE] ``` The `logging.config` property is optional. If not set, logging messages are sent to the console. ## Test communication You can run the connector in communication testing mode. This mode tests the communication with the backend without running the plugin, reports the result, and exits. ```java theme={"dark"} java -Dstandalone.connector.connectivity.test=true -jar connector-plugin-client-standalone.jar [YAML_CONFIG_FILE] ``` ## Encryption In a deployment, communication to the connector’s backend server is encrypted using TLS. You should only run this configuration without TLS in a testing scenario. To disable TLS, set `plain-text` to `true`. ## Egress and proxy server configuration One of the methods you can use to allow outbound communication from behind a firewall is a proxy server. You can configure a proxy server to allow certain communication traffic while blocking unauthorized communication. If you use a proxy server at the site where the connector is running, you must configure the following properties: * **Host.** The hosts where the proxy server is running. * **Port.** The port the proxy server is listening to for communication requests. * **Credentials.** Optional proxy server user and password. When you configure egress, it is important to disable any connection or activity timeouts because the connector uses long running gRPC calls. ## Password encryption If you use a login name and password in your configuration, run the following utility to encrypt the password: 1. Enter a user name and password in the [connector configuration YAML](#configuration). 2. Run the standalone JAR with this property: ```java theme={"dark"} -Dstandalone.connector.encrypt.password=true ``` 3. Retrieve the encrypted passwords from the log that is created. 4. Replace the clear password in the configuration YAML with the encrypted password. ## Connector restart (5.7 and earlier) The connector will shut down automatically whenever the connection to the server is disrupted, to prevent it from getting into a bad state. Communication disruption can happen, for example, when the server running in the `connectors-backend` pod shuts down and is replaced by a new pod. Once the connector shuts down, connector configuration and job execution are disabled. To prevent that from happening, you should restart the connector as soon as possible. You can use Linux scripts and utilities to restart the connector automatically, such as [Monit](https://mmonit.com/monit/). ## Recoverable bridge (5.8 and later) If communication to the remote connector is disrupted, the connector will try to recover communication and gRPC calls. By default, six attempts will be made to recover each gRPC call. The number of attempts can be configured with the `max-grpc-retries` bridge parameters. ## Job expiration duration (5.9.5 only) The timeout value for irresponsive backend jobs can be configured with the `job-expiration-duration-seconds` parameter. The default value is `120` seconds. ## Use the remote connector Once the connector is running, it is available in the Datasources dropdown. If the standalone connector terminates, it disappears from the list of available connectors. Once it is re-run, it is available again and configured connector instances will not get lost. ## Enable asynchronous parsing (5.9 and later) To separate document crawling from document parsing, enable Tika Asynchronous Parsing on remote V2 connectors. Below is an example configuration showing how to specify the file system to index under the `connector-plugins` entry in your `values.yaml` file: ```yaml wrap theme={"dark"} additionalVolumes: - name: fusion-data1-pvc persistentVolumeClaim: claimName: fusion-data1-pvc - name: fusion-data2-pvc persistentVolumeClaim: claimName: fusion-data2-pvc additionalVolumeMounts: - name: fusion-data1-pvc mountPath: "/connector/data1" - name: fusion-data2-pvc mountPath: "/connector/data2" ``` You may also need to specify the user that is authorized to access the file system, as in this example: ```yaml wrap theme={"dark"} securityContext: fsGroup: 1002100000 runAsUser: 1002100000 ``` ## Learn more The microlearning for **The REST V2 Connector** focuses on how to use the REST V2 connector and its recipes. The quick learning for **Using the REST V2 Connector** focuses on how to ingest a datasource using REST V2 connector recipes. ## Configuration