> ## Documentation Index
> Fetch the complete documentation index at: https://doc.lucidworks.com/llms.txt
> Use this file to discover all available pages before exploring further.

# ETL indexing

[localhost link]: http://localhost:3000/docs/lucidworks-search/08-administration/jobs-scheduling/etl-indexing

[mintlify link]: https://doc.lucidworks.com/docs/lucidworks-search/08-administration/jobs-scheduling/etl-indexing.md

The ETL (Extract, Transform, Load) Indexer is a Lucidworks Search job that bulk-loads large volumes of structured catalog data, typically for ecommerce, from Google Cloud Storage into Lucidworks Search.

> [!NOTE]
>
> ETL indexing is available in Lucidworks Search 5.19 and later. It is turned off by default for existing clients upgrading to Lucidworks Search 5.19.
> Contact your Lucidworks representative to request ETL indexing.

> **Key questions to get you started**
>
> Powered by Lucidworks Agent Studio

## Who should use this

The ETL Indexer is designed for ecommerce customers who bulk-load large product catalogs.
The following use cases are suited for the ETL Indexer:

* Indexing large volumes of catalog data.
* Using a managed alternative to the Parallel Bulk Loader, or using bulk indexing for the first time as a Lucidworks Search customer.
* Indexing CSV, JSON, JSONL, or Parquet files from a Google Cloud Storage bucket.

Connectors such as the GCS V2 connector are generally a better fit for knowledge management indexing environments.

## How it works

When you run an ETL indexing job in the Lucidworks Search UI, the job writes an indexing request to the request path in your cloud storage bucket.
The ETL Indexer, a Kubernetes operator, then performs the actual indexing.
If the ETL Indexer pod fails or restarts, the request still completes.

The ETL Indexer moves data using one of two flows:

* From your cloud storage inbox to a Lucidworks Search indexing pipeline for processing
* From your cloud storage inbox directly to Solr, bypassing the index pipeline

```mermaid theme={"dark"}
%% Title: ETL Indexer Data Flow
flowchart LR
    Storage[(Cloud Storage)]
    ETL[ETL Indexer]

    subgraph "Indexing Targets"
        direction TB
        Pipeline[Indexing Pipeline]
        Solr[(Solr)]
    end

    Storage --> ETL
    ETL --> Pipeline
    ETL --> Solr
    Pipeline --> Solr
```

To index data that has been modified in your GCS bucket or to continually reindex data, rerun the ETL indexer. You can rerun the ETL indexer manually or by scheduling the job to run at a set interval.

### Request lifecycle

Each indexing request moves through pending, processing, completed, or failed states as the ETL Indexer works through it.
If the ETL Indexer pod restarts while a request is in the processing state, the indexer resumes that request automatically, resulting in no data loss.

```mermaid theme={"dark"}
%% Title: ETL Indexer Request Lifecycle
stateDiagram-v2
    RequestCreated: Request created

    RequestCreated --> Pending
    Pending --> Processing: ETL picks up the request
    Processing --> Completed: request succeeds
    Processing --> Failed: formatting error or no data found

    note right of Processing
        Resumes automatically
        if the pod restarts
    end note
```

## Comparison to Parallel Bulk Loader

The Parallel Bulk Loader job remains available in Lucidworks Search 5.19 alongside the ETL Indexer and will be deprecated in the future.

| | ETL Indexer | Parallel Bulk Loader |
| - | - | - |
| Runtime | Kubernetes operator that processes requests sent by the job | Spark job, run on demand or on a schedule |
| Availability | Lucidworks Search only | Lucidworks Search and self-hosted Fusion |
| Supported formats | CSV, JSON, JSONL, Parquet | Many formats, plus JDBC, MongoDB, and Solr datasources |
| Transformations | Field mapping only | Custom Scala, SQL, and ML model transforms |

## Before you begin

Before configuring your ETL indexing job, ensure you have the following:

* Lucidworks Search 5.19 or later with the ETL indexing job visible in the Lucidworks Search UI. Contact your Lucidworks representative if you do not have access.
* A Lucidworks Search app configured with indexed content
* A GCS bucket with content in CSV, Parquet, JSON, or JSONL format
* A request path that Lucidworks Search can write to. Work with your Lucidworks representative to set up the request path.
* A Lucidworks Search collection as a destination for the data

## Configuration

**1. Create the job**

In the Lucidworks Search UI, go to **Collections > Jobs**.
Click **Add+** and select **ETL Indexer**.

**2. Set the job ID and request path**

Enter a **Job ID** for the job.
In **ETL Request Path**, enter the cloud storage location where the job writes indexing requests.

**3. Configure one or more collections**

For each destination collection, enter the **Collection Name**, **Data Source**, and **Data Format**.
To send documents through an indexing pipeline instead of directly to Solr, enter a **Pipeline**.
Direct-to-Solr indexing is faster, so set a pipeline only if your data needs transformations at index time.

To index data to more than one Lucidworks Search collection, click **Add** to add another collection configuration.
Use the **Indexing Group** field to control whether collections are indexed serially or in parallel.

**4. Save and run the job**

Click **Save**.

For details about each field, indexing groups, and glob expressions in the **Data Source** field, see [the ETL Indexer job](/docs/lucidworks-search/09-developer-documentation/config-specs/jobs/etl-indexer.md).

## Monitoring and troubleshooting

The ETL Indexer exports metrics per collection: documents indexed, failed documents, and throughput (documents per second). You can view these metrics in your Grafana dashboard.

If an ETL Indexer job fails, it typically fails because of a malformed data source or because the ETL Indexer can't find data files at the configured path.
Verify the job configuration in the Lucidworks Search UI and confirm the cloud storage path is correct. If the job still fails, contact Lucidworks support for pod logs.

## Learn more

For the ETL Indexer job configuration fields, see [ETL Indexer](/docs/lucidworks-search/09-developer-documentation/config-specs/jobs/etl-indexer.md).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.