Google Cloud Storage (GCS) V2

Latest version: v2.1.0
Compatible with Fusion version: 5.2.0 and later

GCS connector can index:

CSV
JSON
PDF
Word docs
Other rich text formats

Features

Service account authentication
Full crawl of storage buckets and objects
Recrawl buckets and objects
Remove deleted objects
Update objects
Cascade deletion of objects in deleted buckets
Document parsing support
Bucket and object filtering

Prerequisites

Perform these prerequisites to ensure the connector can reliably access, crawl, and index your data. Proper setup helps avoid configuration or permission errors, so use the following guidelines to keep your content available for discovery and search in Fusion.

Set up a Google Cloud project with access to the target GCS bucket.
- Enable the Google Cloud Storage API.
Create a service account with appropriate permissions:
- The minimum permissions are storage.objects.list and storage.objects.get.
- For bucket metadata access, also add the permission storage.buckets.get.
Download a JSON key file for that service account to use in the connector authentication properties in Fusion.

Confirm network connectivity:

Fusion must have outbound internet access to reach https://storage.googleapis.com.
If you are using a proxy or VPC, make sure traffic to GCS is permitted.

Remote-connector setup if running as an on-prem process:

A Fusion user with the remote-connectors or admin role for gRPC authentication.
The connector-plugin-standalone.jar alongside the plugin ZIP on the remote host.
A configured connector backend gRPC endpoint (hostname:port) in your YAML.
If the remote host doesn’t trust Fusion’s TLS cert, point to a truststore file path in your config.

Authentication

Setting up the correct authentication according to your organization’s data governance policies helps keep sensitive data secure while allowing authorized indexing. The connector uses Google Cloud service account credentials to access buckets through JSON key authentication. The service account must have storage.objects.list and storage.objects.get permissions and the full JSON key content must be copied and pasted into the service account JSON key box. You can restrict the service account from accessing content you don’t want indexed using IAM roles or signed URLs. Create a service account in Google Cloud and authenticate the GCS V2 Connector:

Go to the Google Cloud Console and navigate to IAM & Admin > Service Accounts.
Click Create Service Account.
1. Give the service account a name such as fusion-gcs-crawler.
2. Grant the service account roles/storage.objectViewer or use custom roles with storage.objects.list and storage.objects.get.
Click Create Key and choose JSON. Download the JSON key file to paste the key’s value in Fusion under Authentication Settings.
Verify access and bucket permissions to ensure:
- The service account has access to the buckets and objects you’re trying to crawl.
- The service account is authorized in the GCS project to access the buckets.

Crawl specific buckets

If the account used has limited permissions, or to only crawl specific buckets, use the Specify buckets to crawl setting. Add the name of the buckets you would like to crawl to download bucket objects and metadata only for those buckets.

Recrawl

The GCS V2 connector updates the Solr index with any content changes since the previous crawl for items added, updated, or deleted.

Remote connectors

V2 connectors support running remotely in Fusion versions 5.7.1 and later. For more information, see Configure remote V2 connectors. Below is an example configuration showing how to specify the file system to index under the connector-plugins entry in your values.yaml file:

additionalVolumes:
- name: fusion-data1-pvc
    persistentVolumeClaim:
    claimName: fusion-data1-pvc
- name: fusion-data2-pvc
    persistentVolumeClaim:
    claimName: fusion-data2-pvc
additionalVolumeMounts:
- name: fusion-data1-pvc
    mountPath: "/connector/data1"
- name: fusion-data2-pvc
    mountPath: "/connector/data2"

You may also need to specify the user that is authorized to access the file system, as in this example:

securityContext:
    fsGroup: 1002100000
    runAsUser: 1002100000

Configuration

Name	Title	Description
`authenticationProperties`	Authentication settings	Connect to the bucket store using a service account. The service account requires the following permissions: storage.buckets.list to crawl all the available buckets; storage.objects.list and storage.objects.get to access to the objects in the buckets.
`applicationProperties`	Limit documents	Bucket and Object filtering options.
`jsonKey`	Service account Json key	Json key contents from authorized service account.
`buckets`	Bucket list	Add the bucket names to crawl. Leave blank to crawl all the available buckets.
`includedFileExtensions`	Included file extensions	Set of file extensions to be fetched. If specified all non-matching files will be skipped.
`excludedFileExtensions`	Excluded file extensions	A set of all file extensions to be skipped from the fetch.
`inclusiveRegexes`	Inclusive regexes	Regular expressions for bucket or object name patterns to include. This will limit this datasource to only items that match the regular expression.
`exclusiveRegexes`	Exclusive regexes	Regular expressions for bucket or object name patterns to exclude. This will limit this datasource to only items that do not match the regular expression.
`maxSizeBytes`	Maximum File Size	Used for excluding objects when the objects size is larger than the configured value.
`minSizeBytes`	Minimum File Size	Used for excluding objects when the objects size is smaller than the configured value.
`bucketPrefix`	Bucket prefix	Filter results to buckets whose names begin with this prefix. Useful only when ‘Bucket List’ property is empty.
`blobsPrefix`	Object prefix	Filter results to objects whose names begin with this prefix.
`pageSize`	Buckets and Objects page size	Maximum number of buckets or objects returned per page.

Concepts

Connectors

Developers

Downloads

Features

Prerequisites

Authentication

Crawl specific buckets

Recrawl

Remote connectors

Configuration

​Features

​Prerequisites

​Authentication

​Crawl specific buckets

​Recrawl

​Remote connectors

​Configuration

Features

Prerequisites

Authentication

Crawl specific buckets

Recrawl

Remote connectors

Configuration