> ## Documentation Index > Fetch the complete documentation index at: https://doc.lucidworks.com/llms.txt > Use this file to discover all available pages before exploring further. # AWS S3 Pro > The AWS S3 Pro connector enables Fusion to crawl and index content stored in Amazon S3 buckets. export const schema = { "type": "object", "title": "S3 Pro", "description": "Connector to index content in AWS S3 buckets.", "required": ["parserId", "id", "properties", "pipeline"], "properties": { "parserId": { "type": "string", "title": "Parser ID", "description": "The Parser to use in the associated IndexPipeline." }, "id": { "type": "string", "title": "Configuration ID", "description": "A unique identifier for this Configuration.", "minLength": 1, "pattern": "^[a-zA-Z0-9_-]+$" }, "pipeline": { "type": "string", "title": "Pipeline ID", "description": "Name of the IndexPipeline used for processing output.", "minLength": 1, "pattern": "^[a-zA-Z0-9_-]+$" }, "description": { "type": "string", "title": "Description", "description": "Optional description", "hints": ["lengthy"], "maxLength": 125 }, "modified": { "type": "string", "title": "Date Modified", "description": "The date at which this Configuration was last modified.", "hints": ["readonly", "hidden"] }, "created": { "type": "string", "title": "Date Created", "description": "The date at which this Configuration was created.", "hints": ["readonly", "hidden"] }, "type": { "type": "string", "title": "Type", "description": "A type ID for this connector.", "hints": ["readonly", "hidden"] }, "collection": { "type": "string", "title": "Collection ID", "description": "The associated content Collection.", "hints": ["readonly", "hidden"], "minLength": 1, "pattern": "^[a-zA-Z0-9_-]+$" }, "diagnosticLogging": { "type": "boolean", "title": "Diagnostic Logging", "description": "[Deprecated] Enables verbose diagnostic logging for troubleshooting. May increase log volume. Disabled by default.", "default": false }, "type_description": { "type": "string", "title": "Type Description", "default": "Connector to index content in AWS S3 buckets.", "hints": ["hidden", "readonly"] }, "category": { "type": "string", "title": "Category", "default": "Connector to index content in AWS S3 buckets.", "hints": ["hidden", "readonly"] }, "connector": { "title": "Connector Type", "description": "Connector type.", "minLength": 1, "type": "string", "hints": ["hidden"] }, "properties": { "type": "object", "title": "S3 properties", "description": "Plugin specific properties.", "required": ["application", "authenticationConfig"], "properties": { "application": { "type": "object", "title": "S3 Application", "required": ["bucketName", "region"], "properties": { "bucketName": { "type": "string", "title": "Bucket name", "description": "The name of the bucket to crawl" }, "objectLambdaAccessPointARNGetObject": { "type": "string", "title": "Object Lambda Access Point ARN - GetObject", "description": "The ARN of the Object Lambda Access Point to be used with the get-object api call. Only populate this when you have an Object Lambda Access Point setup that you want to use with the get-object part of the flow. Example: arn:aws:s3-object-lambda:::accesspoint/" }, "region": { "type": "string", "title": "Region", "description": "The AWS region in which the bucket is located.", "enum": ["ap-south-1", "eu-south-1", "us-gov-east-1", "ca-central-1", "eu-central-1", "us-west-1", "us-west-2", "af-south-1", "eu-north-1", "eu-west-3", "eu-west-2", "eu-west-1", "ap-northeast-2", "ap-northeast-1", "me-south-1", "sa-east-1", "ap-east-1", "cn-north-1", "us-gov-west-1", "ap-southeast-1", "ap-southeast-2", "us-iso-east-1", "us-east-1", "us-east-2", "cn-northwest-1", "us-isob-east-1", "aws-global", "aws-cn-global", "aws-us-gov-global", "aws-iso-global", "aws-iso-b-global"], "default": "us-west-2" }, "objectKeys": { "type": "array", "title": "Object keys or prefixes", "description": "Limit the crawl to a set of Files or Folders inside the bucket. Folders must end with '/'. Valid input examples: 'folderName/', 'folder/subFolder/', 'file.txt', 'folder/file.txt'", "items": { "type": "string" } } } }, "authenticationConfig": { "type": "object", "title": "S3 Authentication settings", "required": [], "properties": { "awsBasicAuthConfig": { "type": "object", "title": "AWS Basic Authentication settings", "required": [], "properties": { "accessKey": { "type": "string", "title": "AWS Key", "description": "An AWS Access Key ID that can access the content." }, "secretKey": { "type": "string", "title": "AWS Secret Key", "description": "The AWS Secret Key associated with the Access Key.", "hints": ["secret"] }, "secretKeyVaultLabel": { "type": "string", "title": "AWS Secret Key from Vault", "description": "The Fusion managed secret to use for AWS Secret Key.", "hints": ["vaultvaluetarget:secretKey", "enumUrl:/apps-manager/vault/labels?type=secret", "vaultref"] } } }, "awsSessionAuthenticationConfig": { "type": "object", "title": "AWS Session Authentication settings", "required": [], "properties": { "accessKey": { "type": "string", "title": "AWS Key", "description": "An AWS Access Key ID that can access the content." }, "secretKey": { "type": "string", "title": "AWS Secret Key", "description": "The AWS Secret Key associated with the Access Key.", "hints": ["secret"] }, "secretKeyVaultLabel": { "type": "string", "title": "AWS Secret Key from Vault", "description": "The Fusion managed secret to use for AWS Secret Key.", "hints": ["vaultvaluetarget:secretKey", "enumUrl:/apps-manager/vault/labels?type=secret", "vaultref"] }, "sessionToken": { "type": "string", "title": "AWS Session Token", "description": "The temporary AWS session token associated with the Access Key and Secret Key. Required for STS-issued temporary credentials." } } }, "awsInstanceCredentialsAuthConfig": { "type": "object", "title": "AWS Instance Credentials Authentication settings", "required": [], "properties": { "instanceCredentials": { "type": "boolean", "title": "Use Instance Credentials", "description": "Use AWS instance credentials rather than an AWS key. Requires that Fusion 5 be hosted in an EKS. You can specify another AWS region through Region property in S3 Application settings.", "default": false } } } } }, "proxyConfig": { "type": "object", "title": "S3 Proxy settings", "required": [], "properties": { "proxyEndpoint": { "type": "string", "title": "Proxy Host", "description": "The optional proxy protocol, host and endpoint through which the client will connect", "hints": ["advanced"] }, "proxyUsername": { "type": "string", "title": "Proxy Username", "description": "The optional username to use when connecting through a proxy", "hints": ["advanced"] }, "proxyPassword": { "type": "string", "title": "Proxy Password", "description": "The optional password to use when connecting through a proxy", "hints": ["advanced"] } }, "hints": ["advanced"] }, "enableStrayContentDeletion": { "type": "boolean", "title": "Enable Stray Content Deletion", "description": "When enabled, folders and files that are not found will be deleted.", "default": true, "hints": ["hidden"] }, "endpointOverride": { "type": "string", "title": "Endpoint Override", "description": "Configure the endpoint with which the Connector should communicate.", "hints": ["advanced"] }, "maximumItemLimitConfig": { "type": "object", "title": "Item Count Limits", "required": [], "properties": { "maxItems": { "type": "number", "title": "Maximum Output Limit", "description": "Limits the number of items emitted to the configured IndexPipeline. The default is no limit (-1).", "default": -1, "maximum": 2147483647, "exclusiveMaximum": false, "minimum": -2147483648, "exclusiveMinimum": false, "multipleOf": 1 } } }, "sizeLimitProperties": { "type": "object", "title": "Item Size Limits", "description": "Options for including or excluding items based on size, in bytes.", "required": [], "properties": { "maxSizeBytes": { "type": "number", "title": "Maximum", "description": "Used for excluding items when the item size is larger than the configured value.", "default": -1, "maximum": 2147483647, "exclusiveMaximum": false, "minimum": -2147483648, "exclusiveMinimum": false, "multipleOf": 1 }, "minSizeBytes": { "type": "number", "title": "Minimum", "description": "Used for excluding items when the item size is smaller than the configured value.", "default": 1, "maximum": 2147483647, "exclusiveMaximum": false, "minimum": -2147483648, "exclusiveMinimum": false, "multipleOf": 1 } } }, "regexConfig": { "type": "object", "title": "Regular expression rules", "required": [], "properties": { "inclusiveRegexes": { "type": "array", "title": "Inclusive regexes", "description": "Regular expressions for URI patterns to include. This will limit this datasource to only URIs that match the regular expression.", "default": [], "items": { "type": "string", "minLength": 1 } }, "exclusiveRegexes": { "type": "array", "title": "Exclusive regexes", "description": "Regular expressions for URI patterns to exclude. This will limit this datasource to only URIs that do not match the regular expression.", "default": [], "items": { "type": "string", "minLength": 1 } }, "regexCacheSize": { "type": "number", "title": "Regex Cache Size", "description": "The number of regex matches to cache when evaluating regular expressions. For example if you exclude files by filename, each filename's regex result will be cached so that if this same filename came up again, the regex matches would be remembered.", "default": 10000, "maximum": 2147483647, "exclusiveMaximum": false, "minimum": -2147483648, "exclusiveMinimum": false, "multipleOf": 1 } } }, "extensionConfig": { "type": "object", "title": "File Extension rules", "required": [], "properties": { "includedFileExtensions": { "type": "array", "title": "Included file extensions", "description": "Set of file extensions to be fetched. If specified, all non-matching files will be skipped.", "default": [], "items": { "type": "string", "minLength": 1 } }, "excludedFileExtensions": { "type": "array", "title": "Excluded file extensions", "description": "A set of all file extensions to be skipped from the fetch.", "default": [], "items": { "type": "string", "minLength": 1 } }, "regexCacheSize": { "type": "number", "title": "Regex Cache Size", "description": "The number of regex matches to cache when evaluating regular expressions. For example if you exclude files by filename, each filename's regex result will be cached so that if this same filename came up again, the regex matches would be remembered.", "default": 10000, "maximum": 2147483647, "exclusiveMaximum": false, "minimum": -2147483648, "exclusiveMinimum": false, "multipleOf": 1 } } }, "requestConfig": { "type": "object", "title": "Request Settings", "description": "Options to configure the client", "required": ["retryCount"], "properties": { "pageSize": { "type": "number", "title": "Request page size", "description": "Maximum number of items per paginated request. AWS S3 ListObjectsV2 caps this at 1000; larger values are silently clamped.", "default": 1000, "maximum": 1000, "exclusiveMaximum": false, "minimum": 1, "exclusiveMinimum": false, "multipleOf": 1 }, "retryCount": { "type": "number", "title": "Retry Count", "description": "Number of times the S3 client connection should retry. Set to 0 to disable retries entirely (useful for fail-fast diagnostics).", "default": 3, "hints": ["advanced"], "maximum": 10, "exclusiveMaximum": false, "minimum": 0, "exclusiveMinimum": false, "multipleOf": 1 }, "apiCallAttemptTimeoutSeconds": { "type": "number", "title": "API Call Attempt Timeout (seconds)", "description": "Maximum time for a single API call attempt including streaming the response body. Increase this value for large files. Capped at 3600 seconds (1 hour).", "default": 300, "hints": ["advanced"], "maximum": 3600, "exclusiveMaximum": false, "minimum": 1, "exclusiveMinimum": false, "multipleOf": 1 }, "socketTimeoutSeconds": { "type": "number", "title": "Socket Read Timeout (seconds)", "description": "Maximum time to wait for data between packets when reading from a socket. Increase this value for large files or slower network connections. Capped at 600 seconds (10 minutes).", "default": 30, "hints": ["advanced"], "maximum": 600, "exclusiveMaximum": false, "minimum": 1, "exclusiveMinimum": false, "multipleOf": 1 } } }, "documentsConfig": { "type": "object", "title": "Indexing settings", "description": "Options to control how documents will be indexed", "required": [], "properties": { "indexFolderMetadata": { "type": "boolean", "title": "Index Folder Metadata", "description": "Enable indexing of folder metadata. Each folder will be represented by a document in the collection.", "default": false } } } } }, "coreProperties": { "type": "object", "title": "Core Properties", "description": "Common behavior and performance settings.", "required": [], "properties": { "fetchSettings": { "type": "object", "title": "Fetch Settings", "description": "System level settings for controlling fetch behavior and performance.", "required": [], "properties": { "numFetchThreads": { "type": "number", "title": "Fetch Threads", "description": "Maximum number of fetch threads; defaults to 5.This setting controls the number of threads that call the Connectors fetch method.Higher values can, but not always, help with overall fetch performance.", "default": 5, "maximum": 500, "exclusiveMaximum": false, "minimum": 1, "exclusiveMinimum": false, "multipleOf": 1 }, "indexingThreads": { "type": "number", "title": "Index Subscription Threads", "description": "Maximum number of indexing threads; defaults to 4.This setting controls the number of threads in the indexing service used for processing content documents emitted by this datasource.Higher values can sometimes help with overall fetch performance.", "default": 4, "maximum": 10, "exclusiveMaximum": false, "minimum": 1, "exclusiveMinimum": false, "multipleOf": 1 }, "pluginInstances": { "type": "number", "title": "Number of plugin instances for distributed fetching", "description": "Maximum number of plugin instances for distributed fetching. Only specified number of plugin instanceswill do fetching. This is useful for distributing load between different instances.", "default": 0, "maximum": 500, "exclusiveMaximum": false, "minimum": 0, "exclusiveMinimum": false, "multipleOf": 1 }, "fetchItemQueueSize": { "type": "number", "title": "Fetch Item Queue Size", "description": "Size of the fetch item queue.Larger values result in increased memory usage, but potentially higher performance.Default is 10k.", "default": 10000, "hints": ["hidden"], "maximum": 500000, "exclusiveMaximum": false, "minimum": 1, "exclusiveMinimum": false, "multipleOf": 1 }, "fetchRequestCheckInterval": { "type": "number", "title": "Fetch request check interval(ms)", "description": "The amount of time to wait before check if a request is done", "default": 15000, "hints": ["hidden"], "maximum": 500000, "exclusiveMaximum": false, "minimum": 1000, "exclusiveMinimum": false, "multipleOf": 1 }, "fetchResponseScheduledTimeout": { "type": "number", "title": "Fetch response scheduled timeout(ms)", "description": "The maximum amount of time for a response to be scheduled. The task will be canceled if this setting is exceeded.", "default": 300000, "maximum": 500000, "exclusiveMaximum": false, "minimum": 1000, "exclusiveMinimum": false, "multipleOf": 1 }, "fetchResponseCompletedTimeout": { "type": "number", "title": "Fetch response completion timeout(ms)", "description": "The maximum amount of time for a response to be completed. If exceeded, the task will be retried if the job is still running", "default": 300000, "hints": ["hidden"], "maximum": 600000, "exclusiveMaximum": false, "minimum": 1, "exclusiveMinimum": false, "multipleOf": 1 }, "indexingInactivityTimeout": { "type": "number", "title": "Indexing inactivity timeout(seconds)", "description": "The maximum amount of time to wait for indexing results (in seconds). If exceeded, the job will fail with an indexing inactivity timeout.", "default": 86400, "maximum": 691200, "exclusiveMaximum": false, "minimum": 60, "exclusiveMinimum": false, "multipleOf": 1 }, "pluginInactivityTimeout": { "type": "number", "title": "Plugin inactivity timeout(seconds)", "description": "The maximum amount of time to wait for plugin activity (in seconds). If exceeded, the job will fail with a plugin inactivity timeout.", "default": 600, "maximum": 691200, "exclusiveMaximum": false, "minimum": 60, "exclusiveMinimum": false, "multipleOf": 1 }, "indexMetadata": { "type": "boolean", "title": "Index metadata", "description": "When enabled the metadata of skipped items will be indexed to the content collection.", "default": false }, "indexContentFields": { "type": "boolean", "title": "Index content fields", "description": "When enabled, content fields will be indexed to the crawl-db collection.", "default": false }, "asyncParsing": { "type": "boolean", "title": "Async Parsing", "description": "When enabled, content will be indexed asynchronously.", "default": false } } }, "strayDeletionSettings": { "type": "object", "title": "Stray Content Deletion Settings", "description": "System level settings for controlling stray content deletion behavior", "required": [], "properties": { "enableStrayDeletion": { "type": "boolean", "title": "Enable Stray Deletion", "description": "When enabled, items not re-encountered in the current crawl are deleted from the content collection. Defaults to true.", "default": true }, "circuitBreakerSettings": { "type": "object", "title": "Circuit Breaker Settings", "description": "Settings that control the circuit breaker, which prevents mass deletion when an unusually high percentage of items are identified as stray in a crawl.", "required": [], "properties": { "percentageThreshold": { "type": "number", "title": "Stray Deletion Threshold (%)", "description": "Maximum percentage of indexed items that may be deleted as stray in a crawl. If the total stray item exceeds this percentage, the circuit breaker blocks the deletion to prevent accidental data loss. Accepts values from 0 to 100. Defaults to 80.", "default": 80, "maximum": 100, "exclusiveMaximum": false, "minimum": 0, "exclusiveMinimum": false, "multipleOf": 1 }, "enableCircuitBreaker": { "type": "boolean", "title": "Enable Circuit Breaker", "description": "When enabled, stray deletion is blocked if the percentage of items to be deleted exceeds the configured threshold. Disable only if unconditional stray deletion is required regardless of volume. Defaults to true.", "default": true } } } } }, "skipConfigValidation": { "type": "boolean", "title": "Skip Validation", "description": "Enable to skip configuration validation when it takes too long and causes timeout issue", "default": false } }, "hints": ["advanced"] } }, "category": "FileSystem", "categoryPriority": 1 }; export const SchemaParamFields = ({schema}) => { const sanitize = str => { if (typeof str !== "string") return str; return str.replace(/^"(.*)"$/s, "$1").replace(/\\/g, "").replace(/"/g, "'"); }; const formatDescription = str => { const s = sanitize(str); return (/[.!?]\)*$/).test(s) ? s : `${s}.`; }; const {description, properties = {}, required: requiredProps = []} = schema; const visibleProps = useMemo(() => Object.entries(properties).filter(([, prop]) => !prop.hints?.includes("hidden")), [properties]); return
{description &&

{formatDescription(description)}

} {visibleProps.map(([name, prop]) => { const isRequired = requiredProps.includes(name); const hasDefault = prop.default !== undefined; const rawDefault = prop.default; const isComplexDefault = hasDefault && (typeof rawDefault === "object" || typeof rawDefault === "string" && (rawDefault.length > 20 || rawDefault.includes('"'))); const fieldProps = { key: name, body: prop.title || name, type: prop.type, ...prop.title && ({ post: [<>API property: {name}] }), ...isRequired && ({ required: true }), ...!isComplexDefault && hasDefault ? { default: sanitize(String(rawDefault)) } : {} }; const isObject = prop.type === "object" && prop.properties; const isArrayOfObjects = prop.type === "array" && prop.items?.type === "object" && prop.items.properties; return {prop.description &&

{formatDescription(prop.description)}

} {isComplexDefault &&

Default: 

                  
                    {JSON.stringify(rawDefault, null, 2)}
} {isArrayOfObjects &&

Object attributes: 

                
                  {'{\n'}
                  {Object.entries(prop.items.properties).map(([iname, iprop]) => <>
                      {`  ${iname}`}
                      {prop.items?.required?.includes(iname) &&  required}
                      {`: {\n    display name: ${sanitize(iprop.title || '')}\n    type: ${iprop.type}\n  }\n`}
                    )}
                  {'}'}
} {isObject && } ; })}
; }; export const LwTemplate = ({title = "Key questions to get you started", icon = "sparkles", cta = "Powered by Agent Studio", linkHref = "https://lucidworks.com/demo/?utm_source=docs&utm_medium=referral&utm_campaign=docs_cta_ai"}) => { const [isLoaded, setIsLoaded] = useState(false); useEffect(() => { const timer = setTimeout(() => { setIsLoaded(true); }, 500); return () => clearTimeout(timer); }, []); return
{isLoaded && ` }} />} Powered by Lucidworks Agent Studio
; }; [localhost link]: http://localhost:3000/docs/fusion-connectors/connectors/amazon-aws-s3-pro [mintlify link]: https://doc.lucidworks.com/docs/fusion-connectors/connectors/amazon-aws-s3-pro [old doc.lw link]: https://doc.lucidworks.com/fusion-connectors/8851 * **Latest version:** v2.0.0 * **Compatible with Fusion version:** 5.9.0 and later Connector flow The AWS S3 Pro connector crawls items in a single bucket. You must specify the **bucket name** and the **AWS region** in which that bucket is located. You may crawl specific items in a bucket. If no items are specified, the entire bucket will be crawled. Stray content deletion is enabled by default. When stray content deletion is enabled, content removed from the data source is deleted from the index in Fusion. When stray content deletion is disabled, content that was removed from the datasource is not deleted from the index in Fusion. When using this connector with Fusion 5.18.0 or later, the connector also includes a configurable circuit breaker setting that prevents accidental mass deletion by blocking the operation when deletions exceed a configurable threshold. Existing datasources created with previous versions of the connector retain their existing settings. If you are using Fusion 5.18.0 or later, you must upgrade to Amazon AWS S3 Pro v2.0.0. The connector can recursively crawl files and folders to retrieve content and metadata such as object size and the time it was last modified. You can also filter objects by file extension, object metadata, or by using regex. ## Prerequisites Perform these prerequisites to ensure the connector can reliably access, crawl, and index your data. Proper setup helps avoid configuration or permission errors, so use the following guidelines to keep your content available for discovery and search in Fusion. ### Connector installation The AWS S3 Pro connector in Fusion is named 'S3 Pro' and is not preceded by 'Amazon' or 'AWS.' When creating an S3 datasource using the UI, Fusion automatically verifies that the user information supplied has access to the bucket defined in the URL property. If the bucket is not in the list returned by S3, datasource creation may fail. At crawl time, if the bucket is not in the list returned by S3, the crawl will fail. Permission errors when trying to create or crawl the datasource may be caused by incorrect username or password, or they may be due to user account permissions. The user account must have List Bucket permissions for the account which owns the bucket that the crawler is trying to access. ### AWS Permissions The connector requires access to the following S3 operations: * `s3:ListBucket` lists objects in the specified bucket or that use a desired prefix. * `s3:GetObject` fetches the content and metadata of each object. The following is an example IAM policy. When you set permissions, replace `BUCKET_NAME` with the value used in your implementation. ```json theme={"dark"} "Statement": [ { "Action": [ "s3:GetObject" ], "Resource": [ "arn:aws:s3:::BUCKET_NAME/*" ], "Effect": "Allow" }, { "Action": [ "s3:ListBucket" ], "Resource": [ "arn:aws:s3:::BUCKET_NAME" ], "Effect": "Allow" } ] ``` ### Remote mode (optional) To run the connector remotely, you need the following: * A Fusion user with the `remote-connectors` or `admin` role for gRPC authentication. * The `connector-plugin-standalone.jar` alongside the plugin ZIP on the remote host. * A configured connector backend gRPC endpoint (`hostname:port`) in your YAML. * If the remote host doesn't trust Fusion's TLS cert, point to a truststore file path in your config. ## Authentication Setting up the correct authentication according to your organization's data governance policies helps keep sensitive data secure while allowing authorized indexing. The AWS S3 Pro connector supports multiple authentication methods to access your Amazon S3 bucket. Choose one of the following based on your environment and security model: * Basic authentication using an access key and secret access key * AWS session authentication using temporary credentials provided by AWS Security Token Service (STS) * AWS instance credentials for role-based authentication if Fusion is running inside AWS ### Basic authentication Enable **AWS Basic Authentication Settings** and enter your AWS access key and AWS secret key. ### Session authentication Session authentication uses temporary security credentials obtained from AWS STS. Enable **AWS Basic Authentication Settings** and enter your AWS access key, AWS secret key, and session token. These credentials must be unexpired at runtime. ### IAM Role for Fusion running in AWS If Fusion or the remote connector is running on an EC2 instance or ECS task with an attached IAM role, do not enter credentials in the connector configuration as the connector will automatically use the role assigned to the host. Enable **AWS Instance Credentials Authentication Settings** and **Use Instance Credentials**. Make sure the IAM role has permissions to read objects from the S3 bucket and access any required prefixes or object paths. ## Retry logic The `retryCount` field sets the number of times the S3 client connection should retry when a document fails to index. Issues with AWS connectivity might result in the S3 connector being unable to crawl all of the data. The default for this field is retrying three times. If you are having trouble with AWS connectivity, try setting this field to a higher value, for example, `10` retries. ## Remote connectors V2 connectors support [running remotely](/docs/fusion-connectors/developers/remote-v2-connectors) in Fusion versions 5.7.1 and later. If you need to index data from behind a firewall, you can configure a V2 connector to run remotely on-premises using TLS-enabled gRPC. ## Prerequisites Before you can set up an on-prem V2 connector, you must configure the egress from your network to allow HTTP/2 communication into the Fusion cloud. You can use a [forward proxy server](#egress-and-proxy-server-configuration) to act as an intermediary between the connector and Fusion. The following is required to run V2 connectors remotely: * The [plugin zip file and the connector-plugin-standalone JAR](https://plugins.lucidworks.com/). * A configured connector backend gRPC endpoint. * Username and password of a user with a `remote-connectors` or `admin` role. * If the host where the remote connector is running is not configured to trust the server’s TLS certificate, you must configure the file path of the trust certificate collection. If your version of Fusion doesn’t have the `remote-connectors` role by default, you can create one. No API or UI permissions are required for the role. ## Connector compatibility Only V2 connectors are able to run remotely on-premises. You also need the remote connector client JAR file that matches your Fusion version. You can download the latest files at [V2 Connectors Downloads](/docs/fusion-connectors/downloads/v2-connectors-downloads). Whenever you upgrade Fusion, you must also update your remote connectors to match the new version of Fusion. The gRPC connector backend is not supported in Fusion environments deployed on AWS. ## System requirements The following is required for the on-prem host of the remote connector: * (Fusion 5.9.0-5.9.10) JVM version 11 * (Fusion 5.9.11) JVM version 17 * Minimum of 2 CPUs * 4GB Memory Note that memory requirements depend on the number and size of ingested documents. ## Enable backend ingress In your `values.yaml` file, configure this section as needed: ```yaml theme={"dark"} ingress: enabled: false pathtype: "Prefix" path: "/" #host: "ingress.example.com" ingressClassName: "nginx" # Fusion 5.9.6 only tls: enabled: false certificateArn: "" # Enable the annotations field to override the default annotations #annotations: "" ``` * Set `enabled` to `true` to enable the backend ingress. * Set `pathtype` to `Prefix` or `Exact`. * Set `path` to the path where the backend will be available. * Set `host` to the host where the backend will be available. * In Fusion 5.9.6 *only*, you can set `ingressClassName` to one of the following: * `nginx` for Nginx Ingress Controller * `alb` for AWS Application Load Balancer (ALB) * Configure TLS and certificates according to your CA’s procedures and policies. TLS must be enabled in order to use AWS ALB for ingress. ## Connector configuration example ```yaml theme={"dark"} kafka-bridge: target: mynamespace-connectors-backend.lucidworkstest.com:443 # mandatory plain-text: false # optional, false by default. proxy-server: # optional - needed when a forward proxy server is used to provide outbound access to the standalone connector host: host port: some-port user: user # optional password: password # optional trust: # optional - needed when the client's system doesn't trust the server's certificate cert-collection-filepath: path1 proxy: # mandatory fusion-proxy user: admin password: password123 url: https://fusiontest.com/ # needed only when the connector plugin requires blob store access plugin: # mandatory path: ./fs.zip type: #optional - the suffix is added to the connector id suffix: remote ``` ### Minimal example ```yaml theme={"dark"} kafka-bridge: target: mynamespace-connectors-backend.lucidworkstest.com:443 proxy: user: admin password: "password123" plugin: path: ./testplugin.zip ``` ### Logback XML configuration file example ```xml wrap theme={"dark"} %d - %-5p [%t:%C{3.}@%L] - %m{nolookups}%n utf8 ${LOGDIR:-.}/connector.log ${LOGDIR:-.}/connector-%d{yyyy-MM-dd}.%i.log.gz 50MB 10GB %d - %-5p [%t:%C{3.}@%L] - %m{nolookups}%n utf8 ``` ## Run the remote connector ```java theme={"dark"} java [-Dlogging.config=[LOGBACK_XML_FILE]] \ -jar connector-plugin-client-standalone.jar [YAML_CONFIG_FILE] ``` The `logging.config` property is optional. If not set, logging messages are sent to the console. ## Test communication You can run the connector in communication testing mode. This mode tests the communication with the backend without running the plugin, reports the result, and exits. ```java theme={"dark"} java -Dstandalone.connector.connectivity.test=true -jar connector-plugin-client-standalone.jar [YAML_CONFIG_FILE] ``` ## Encryption In a deployment, communication to the connector’s backend server is encrypted using TLS. You should only run this configuration without TLS in a testing scenario. To disable TLS, set `plain-text` to `true`. ## Egress and proxy server configuration One of the methods you can use to allow outbound communication from behind a firewall is a proxy server. You can configure a proxy server to allow certain communication traffic while blocking unauthorized communication. If you use a proxy server at the site where the connector is running, you must configure the following properties: * **Host.** The hosts where the proxy server is running. * **Port.** The port the proxy server is listening to for communication requests. * **Credentials.** Optional proxy server user and password. When you configure egress, it is important to disable any connection or activity timeouts because the connector uses long running gRPC calls. ## Password encryption If you use a login name and password in your configuration, run the following utility to encrypt the password: 1. Enter a user name and password in the [connector configuration YAML](#configuration). 2. Run the standalone JAR with this property: ```java theme={"dark"} -Dstandalone.connector.encrypt.password=true ``` 3. Retrieve the encrypted passwords from the log that is created. 4. Replace the clear password in the configuration YAML with the encrypted password. ## Connector restart (5.7 and earlier) The connector will shut down automatically whenever the connection to the server is disrupted, to prevent it from getting into a bad state. Communication disruption can happen, for example, when the server running in the `connectors-backend` pod shuts down and is replaced by a new pod. Once the connector shuts down, connector configuration and job execution are disabled. To prevent that from happening, you should restart the connector as soon as possible. You can use Linux scripts and utilities to restart the connector automatically, such as [Monit](https://mmonit.com/monit/). ## Recoverable bridge (5.8 and later) If communication to the remote connector is disrupted, the connector will try to recover communication and gRPC calls. By default, six attempts will be made to recover each gRPC call. The number of attempts can be configured with the `max-grpc-retries` bridge parameters. ## Job expiration duration (5.9.5 only) The timeout value for irresponsive backend jobs can be configured with the `job-expiration-duration-seconds` parameter. The default value is `120` seconds. ## Use the remote connector Once the connector is running, it is available in the Datasources dropdown. If the standalone connector terminates, it disappears from the list of available connectors. Once it is re-run, it is available again and configured connector instances will not get lost. ## Enable asynchronous parsing (5.9 and later) To separate document crawling from document parsing, enable Tika Asynchronous Parsing on remote V2 connectors. Below is an example configuration showing how to specify the file system to index under the `connector-plugins` entry in your `values.yaml` file: ```yaml theme={"dark"} additionalVolumes: - name: fusion-data1-pvc persistentVolumeClaim: claimName: fusion-data1-pvc - name: fusion-data2-pvc persistentVolumeClaim: claimName: fusion-data2-pvc additionalVolumeMounts: - name: fusion-data1-pvc mountPath: "/connector/data1" - name: fusion-data2-pvc mountPath: "/connector/data2" ``` You may also need to specify the user that is authorized to access the file system, as in this example: ```yaml theme={"dark"} securityContext: fsGroup: 1002100000 runAsUser: 1002100000 ``` ## Configuration When entering configuration values in the UI, use *unescaped* characters, such as `\t` for the tab character. When entering configuration values in the API, use *escaped* characters, such as `\\t` for the tab character.