Skip to main content
OpenTelemetry (OTel) distributed tracing gives Lucidworks engineers request-level visibility across the Fusion microservice stack. Where Prometheus provides aggregate metrics, OTel traces individual requests from the moment they enter the API Gateway through the Query Service and into Solr, making it possible to pinpoint exactly which service is responsible for a slowdown. For you, this means faster resolution when you report a performance issue, more precise communication about what caused it, and fewer prolonged incidents.
OpenTelemetry distributed tracing is being rolled out to all Fusion environments. No action is required from you to enable it. The services that support OpenTelemetry are API Gateway, Query, and Indexing.

​
How it works

1

API Gateway creates the root span

The API Gateway receives the incoming request and creates the root trace . It propagates trace context downstream using a traceparent header. Only query-path traffic is sampled, which keeps overhead minimal.
2

Query Service creates child spans

The Query Service picks up the trace context and creates child spans for each integration it calls. The following spans are created for every traced request:
  • Solr span: captures query execution time within Solr.
  • Feign/downstream span: captures HTTP calls made to downstream services.
The following spans are optional and enabled per environment:
  • Kafka span: captures Kafka producer and consumer calls made during query processing.
  • ItemStore span: captures ItemStore read and write operations.
3

OTel Collector batches the spans

The receives spans from the Query Service and batches them for export.
4

Grafana Tempo stores the traces

The OTel Collector forwards the batched spans to Grafana Tempo, which stores them for search and analysis.
5

Grafana Dashboard surfaces the traces

stored in Grafana Tempo are available in the Grafana Dashboard, where Lucidworks engineers can search, filter, and drill into individual traces to identify performance bottlenecks.

​
Viewing traces in Grafana

Lucidworks engineers use the Grafana Tempo dashboard to search and visualize traces. A trace can be filtered by service, time window, latency threshold, or error status. Drilling into a trace shows the full span tree: API Gateway root span, Query Service child spans, and Solr spans nested within them. This makes the following workflow possible for Lucidworks engineers responding to an incident:
  1. Open Grafana and navigate to the Tempo traces dashboard.
  2. Select the time window that corresponds to the reported latency spike.
  3. Identify the slowest trace for that window.
  4. Drill into the span tree to see which service or integration was the bottleneck.
This full workflow is achievable in under five minutes, without code changes or log searches.

​
Instrumented integrations

Tracing coverage can be enabled per integration. The integrations available are:
IntegrationWhat it covers
SolrQuery execution time within Solr for each request
KafkaKafka producer/consumer calls made during query processing
JDBCDatabase calls made by Fusion services
ItemStoreItemStore read/write operations
Feign HTTP clientHTTP calls between Fusion microservices
Each integration can be enabled or disabled independently, so trace volume and overhead can be tuned without touching application code or requiring a Fusion release.