> ## Documentation Index
> Fetch the complete documentation index at: https://doc.lucidworks.com/llms.txt
> Use this file to discover all available pages before exploring further.

# OpenTelemetry Distributed Tracing

> How OpenTelemetry distributed tracing improves performance visibility and incident response for Fusion deployments

OpenTelemetry (OTel) distributed tracing gives Lucidworks engineers request-level visibility across the Fusion microservice stack. Where Prometheus provides aggregate metrics, OTel traces individual requests from the moment they enter the API Gateway through the Query Service and into Solr, making it possible to pinpoint exactly which service is responsible for a slowdown.

For you, this means faster resolution when you report a performance issue, more precise communication about what caused it, and fewer prolonged incidents.

<Info>
  OpenTelemetry distributed tracing is being rolled out to all Fusion environments. No action is required from you to enable it. The services that support OpenTelemetry are API Gateway, Query, and Indexing.
</Info>

## How it works

```mermaid theme={"dark"}
flowchart LR
    A["Incoming Request"] --> B["API Gateway"]
    B --"traceparent header propagated"--> C["Query Service"]
    C --> OTel["OTel Collector"]
    OTel --> Tempo["Grafana Tempo"]
    Tempo --> Dashboard["Grafana Dashboard"]
    C --> Kafka
    C --> Feign

    subgraph spans["Query Service child spans"]
        Kafka["Kafka span (if enabled)"] --> Solr["Solr span"]
        Feign["Feign/downstream span"] --> ItemStore["ItemStore span (if enabled)"]
    end
```

<Steps>
  <Step title="API Gateway creates the root span">
    The API Gateway receives the incoming request and creates the root trace <Tooltip tip="A span represents a single named operation within a distributed trace.">span</Tooltip>. It propagates trace context downstream using a `traceparent` header. Only query-path traffic is sampled, which keeps overhead minimal.
  </Step>

  <Step title="Query Service creates child spans">
    The Query Service picks up the trace context and creates child spans for each integration it calls. The following spans are created for every traced request:

    * **Solr span**: captures query execution time within Solr.
    * **Feign/downstream span**: captures HTTP calls made to downstream services.

    The following spans are optional and enabled per environment:

    * **Kafka span**: captures Kafka producer and consumer calls made during query processing.
    * **ItemStore span**: captures ItemStore read and write operations.
  </Step>

  <Step title="OTel Collector batches the spans">
    The <Tooltip tip="An agent that receives, processes, and exports telemetry data from instrumented services.">OTel Collector</Tooltip> receives spans from the Query Service and batches them for export.
  </Step>

  <Step title="Grafana Tempo stores the traces">
    The OTel Collector forwards the batched spans to Grafana Tempo, which stores them for search and analysis.
  </Step>

  <Step title="Grafana Dashboard surfaces the traces">
    <Tooltip tip="A trace is the complete record of a single request's journey across all services and operations.">Traces</Tooltip> stored in Grafana Tempo are available in the Grafana Dashboard, where Lucidworks engineers can search, filter, and drill into individual traces to identify performance bottlenecks.
  </Step>
</Steps>

## Viewing traces in Grafana

Lucidworks engineers use the Grafana Tempo dashboard to search and visualize traces.
A trace can be filtered by service, time window, latency threshold, or error status.
Drilling into a trace shows the full span tree: API Gateway root span, Query Service child spans, and Solr spans nested within them.

This makes the following workflow possible for Lucidworks engineers responding to an incident:

1. Open Grafana and navigate to the Tempo traces dashboard.
2. Select the time window that corresponds to the reported latency spike.
3. Identify the slowest trace for that window.
4. Drill into the span tree to see which service or integration was the bottleneck.

This full workflow is achievable in under five minutes, without code changes or log searches.

## Instrumented integrations

Tracing coverage can be enabled per integration. The integrations available are:

| Integration       | What it covers                                             |
| ----------------- | ---------------------------------------------------------- |
| Solr              | Query execution time within Solr for each request          |
| Kafka             | Kafka producer/consumer calls made during query processing |
| JDBC              | Database calls made by Fusion services                     |
| ItemStore         | ItemStore read/write operations                            |
| Feign HTTP client | HTTP calls between Fusion microservices                    |

Each integration can be enabled or disabled independently, so trace volume and overhead can be tuned without touching application code or requiring a Fusion release.