Skip to main content
The ETL (Extract, Transform, Load) Indexer is a Lucidworks Search job that bulk-loads large volumes of structured catalog data, typically for ecommerce, from Google Cloud Storage into Lucidworks Search.
ETL indexing is available in Lucidworks Search 5.19 and later. It is turned off by default for existing clients upgrading to Lucidworks Search 5.19. Contact your Lucidworks representative to request ETL indexing.

​
Who should use this

The ETL Indexer is designed for ecommerce customers who bulk-load large product catalogs. The following use cases are suited for the ETL Indexer:
  • Indexing large volumes of catalog data.
  • Using a managed alternative to the Parallel Bulk Loader, or using bulk indexing for the first time as a Lucidworks Search customer.
  • Indexing CSV, JSON, JSONL, or Parquet files from a Google Cloud Storage bucket.
Connectors such as the GCS V2 connector are generally a better fit for knowledge management indexing environments.

​
How it works

When you run an ETL indexing job in the Lucidworks Search UI, the job writes an indexing request to the request path in your cloud storage bucket. The ETL Indexer, a Kubernetes operator, then performs the actual indexing. If the ETL Indexer pod fails or restarts, the request still completes. The ETL Indexer moves data using one of two flows:
  • From your cloud storage inbox to a Lucidworks Search indexing pipeline for processing
  • From your cloud storage inbox directly to Solr, bypassing the index pipeline
To index data that has been modified in your GCS bucket or to continually reindex data, rerun the ETL indexer. You can rerun the ETL indexer manually or by scheduling the job to run at a set interval.

​
Request lifecycle

Each indexing request moves through pending, processing, completed, or failed states as the ETL Indexer works through it. If the ETL Indexer pod restarts while a request is in the processing state, the indexer resumes that request automatically, resulting in no data loss.

​
Comparison to Parallel Bulk Loader

The Parallel Bulk Loader job remains available in Lucidworks Search 5.19 alongside the ETL Indexer and will be deprecated in the future.

​
Before you begin

Before configuring your ETL indexing job, ensure you have the following:
  • Lucidworks Search 5.19 or later with the ETL indexing job visible in the Lucidworks Search UI. Contact your Lucidworks representative if you do not have access.
  • A Lucidworks Search app configured with indexed content
  • A GCS bucket with content in CSV, Parquet, JSON, or JSONL format
  • A request path that Lucidworks Search can write to. Work with your Lucidworks representative to set up the request path.
  • A Lucidworks Search collection as a destination for the data

​
Configuration

1

Create the job

In the Lucidworks Search UI, go to Collections > Jobs. Click Add+ and select ETL Indexer.
2

Set the job ID and request path

Enter a Job ID for the job. In ETL Request Path, enter the cloud storage location where the job writes indexing requests.
3

Configure one or more collections

For each destination collection, enter the Collection Name, Data Source, and Data Format. To send documents through an indexing pipeline instead of directly to Solr, enter a Pipeline. Direct-to-Solr indexing is faster, so set a pipeline only if your data needs transformations at index time.To index data to more than one Lucidworks Search collection, click Add to add another collection configuration. Use the Indexing Group field to control whether collections are indexed serially or in parallel.
4

Save and run the job

Click Save.
For details about each field, indexing groups, and glob expressions in the Data Source field, see the ETL Indexer job.

​
Monitoring and troubleshooting

The ETL Indexer exports metrics per collection: documents indexed, failed documents, and throughput (documents per second). You can view these metrics in your Grafana dashboard. If an ETL Indexer job fails, it typically fails because of a malformed data source or because the ETL Indexer can’t find data files at the configured path. Verify the job configuration in the Lucidworks Search UI and confirm the cloud storage path is correct. If the job still fails, contact Lucidworks support for pod logs.

​
Learn more

For the ETL Indexer job configuration fields, see ETL Indexer.