Grafana Tempo

Why It Exists

Before Tempo, distributed tracing at scale meant running Elasticsearch or Cassandra as the storage backend for Jaeger. Elasticsearch requires JVM tuning, shard management, ILM policies, and a dedicated team to operate. Cassandra requires topology planning, compaction tuning, and consistent hash ring management. Both are expensive to run at petabyte scale.

Grafana Labs built Tempo around a single insight: traces are write-heavy, read-light, and almost always queried by trace ID. If 90% of trace lookups are by a known trace ID (clicked from a metric exemplar or a log line), a full inverted index is unnecessary. A bloom filter per block is enough to locate the right Parquet file on S3. The remaining 10% of queries (search by attribute) are slower, but the operational and cost savings from eliminating the index are massive.

The result is a trace backend where storage cost scales with S3 pricing (~$0.023/GB/month for Standard, ~$0.004/GB/month for Glacier Instant Retrieval), not with compute cluster sizing. A petabyte of traces on Glacier Instant Retrieval costs ~$4,000/month. The same volume on Elasticsearch would require dozens of data nodes at 10-50x the cost.

How the Storage Engine Works

Tempo writes trace data as Apache Parquet columnar files on S3. Each file is called a "block" and contains all spans ingested during a time window.

Why Parquet? Parquet stores data column-by-column, not row-by-row. A trace span has dozens of attributes (service_name, operation, duration, http.method, http.status_code, custom attributes). A query filtering on service_name reads only the service_name column from the Parquet file, skipping all other columns entirely. This column pruning reduces the bytes read from S3 by 10-50x compared to reading full rows.

Block structure on S3:

s3://tempo-traces/
  single-tenant/
    <block-id>/
      data.parquet          # Span data in columnar format
      bloom-0.bloom         # Bloom filter shard (trace IDs)
      bloom-1.bloom         # Bloom filter shard
      meta.json             # Block metadata (time range, span count, size)

Bloom filter lookup flow:

1. User queries {traceid="abc123"}
2. Tempo reads meta.json for all blocks in the queried time range
3. For each block, Tempo checks the bloom filter: "Could trace abc123 be in this block?"
4. Bloom filter says "definitely not" for 9,997 out of 10,000 blocks → skipped
5. Bloom filter says "maybe" for 3 blocks → Tempo reads data.parquet from S3 for those 3
6. Parquet column pruning: read only the columns needed for the response
7. Return the matching trace

The bloom filter eliminates 99.97% of S3 reads in this example. The false positive rate depends on block size and bloom filter configuration. Well-compacted blocks (fewer, larger blocks) have more efficient bloom filters.

Ingester and Write Path

The write path follows this sequence:

1. Distributor receives spans via OTLP (gRPC or HTTP) from OTel Collectors
2. Distributor hashes each span's trace_id and routes it to the correct ingester (consistent hashing ring)
3. The ingester appends the span to an in-memory buffer and writes it to the local WAL (Write-Ahead Log) on SSD
4. When the buffer reaches the flush threshold (configurable: time-based or size-based), the ingester:
   • Sorts spans by trace_id
   • Builds a Parquet file with columnar encoding
   • Constructs bloom filter shards for all trace IDs in the block
   • Uploads data.parquet, bloom filter files, and meta.json to S3
   • Deletes the corresponding WAL segments

The trace_id-based routing in the distributor ensures all spans from a single trace arrive at the same ingester. This means the ingester has the complete trace (or most of it) when it builds the Parquet block, producing better compression and more accurate bloom filters.

WAL replay on crash: If an ingester crashes before flushing, the WAL on local SSD survives. On restart, the ingester replays the WAL, reconstructs the in-memory buffer, and flushes normally. No spans are lost unless the local SSD itself fails (mitigated by running multiple ingesters with replication factor 2-3).

Compactor

The compactor is Tempo's background maintenance process. Without it, query performance degrades over time.

What it does:

1. Merges small blocks. Frequent ingester flushes create many small Parquet files on S3. The compactor reads multiple small blocks, merges them into a single larger block, and deletes the originals. Fewer blocks = fewer bloom filter checks per query = faster queries.

2. Rebuilds bloom filters. Merged blocks get new, more efficient bloom filters. A bloom filter covering 1 million traces in one block is more space-efficient than 10 bloom filters covering 100K traces each.

3. Enforces retention. Blocks older than the configured retention period are deleted from S3.

4. Manages block lifecycle. Tracks block state (active, compacted, deleted) in a per-tenant block list stored on S3.

Compactor sizing: The compactor reads blocks from S3, merges them in memory, and writes the result back to S3. Memory usage scales with block size and concurrency. A single compactor handling a few hundred GB of daily traces needs 4-8 GB RAM and 2-4 CPU cores. At petabyte scale, run multiple compactor instances with sharded tenants.

TraceQL

TraceQL is Tempo's query language for trace analysis. It operates on spans and traces, not on time series.

Basic attribute queries:

# Find all traces from the checkout service
{resource.service.name = "checkout"}

# Find error spans longer than 2 seconds
{status = error && duration > 2s}

# Find database spans with slow queries
{span.db.system = "postgresql" && duration > 500ms}

Structural queries (unique to TraceQL):

# Find traces where a parent HTTP span contains a slow child DB span
{span.http.method = "POST"} >> {span.db.system = "postgresql" && duration > 1s}

# Find traces with more than 20 spans (complex request flows)
{rootSpan} | count() > 20

The >> operator means "is an ancestor of" — this finds traces where an HTTP POST span eventually calls a PostgreSQL query that takes over 1 second. No other trace query language supports this kind of structural query. Jaeger supports tag-based search. VictoriaTraces supports LogsQL flat field search. Only TraceQL can express parent-child span relationships.

Query execution: The query frontend receives a TraceQL query, splits it by time range, and fans it out to querier pods. Each querier checks bloom filters, reads relevant Parquet blocks from S3, evaluates the query against the columnar data, and returns matching spans. The query frontend merges results from all queriers.

Single-Node vs Distributed

Monolithic mode: A single binary runs all components (distributor, ingester, querier, query-frontend, compactor). Suitable for development, testing, and small deployments up to ~50K spans/sec. Simple to deploy: one process, one config file, one S3 bucket.

Microservices mode: Each component runs as a separate process (or Kubernetes deployment). Scale each dimension independently:

Component	Role	Scales with	Stateful?
Distributor	Accepts OTLP spans, routes by trace_id	Ingestion rate	No
Ingester	Buffers spans, writes Parquet to S3	Ingestion rate + trace cardinality	Yes (WAL on local SSD)
Querier	Reads Parquet from S3, evaluates TraceQL	Query concurrency	No
Query Frontend	Query scheduling, splitting, caching	Query concurrency	No (cache is external)
Compactor	Merges blocks, rebuilds bloom filters	Block count + retention volume	No (reads/writes S3)

The ingester is the only stateful component (local WAL). Everything else is stateless and horizontally scalable.

Decision Criteria

Criteria	Grafana Tempo	Jaeger	VictoriaTraces
Storage backend	S3/GCS (object storage)	Elasticsearch, Cassandra, or ClickHouse	Local NVMe/SSD
Index strategy	No index. Bloom filters for trace_id.	Full inverted index (ES) or column index (ClickHouse)	Bloom filters on all span fields
Query language	TraceQL (structural + attribute)	Tag-based search (limited)	Jaeger APIs + LogsQL
Storage cost (1 PB)	~$23K/month (S3 Standard)	~$200-500K/month (ES cluster)	Local NVMe provisioning
Resource efficiency	1.35 vCPU, 4.26 GiB at 10K spans/sec	Depends on backend	0.50 vCPU, 1.15 GiB at 10K spans/sec
Grafana integration	Native Tempo datasource	Jaeger datasource plugin	Jaeger datasource (+ experimental Tempo API)
Tiered storage	S3 lifecycle policies (automatic)	ES ILM or Cassandra TTL	NVMe → HDD → S3 via vmbackup (manual)
Operational complexity	Medium (S3 config, compactor monitoring)	High (external DB management)	Low (local disk, single binary or 3-component cluster)
Attribute search speed	Slow on large ranges (no index, Parquet scan)	Fast (inverted index)	Medium (bloom filter assisted)
Community	Large (Grafana Labs ecosystem)	Largest (CNCF graduated)	Growing (VictoriaMetrics ecosystem)

Choosing between them:

• TraceQL structural queries are required: Grafana Tempo. Only option.
• S3-native with automatic lifecycle tiering: Grafana Tempo. Zero local disk management.
• Resource efficiency is the priority: VictoriaTraces. 3.7x less RAM, 2.6x less CPU.
• Fast attribute search on large time ranges: Jaeger with Elasticsearch. Full inverted index.
• Already running VictoriaMetrics + VictoriaLogs: VictoriaTraces. Same operational model.
• Air-gapped environments: VictoriaTraces. No external storage dependencies.

Capacity Planning

Ingester sizing:

Spans/sec	Ingesters	CPU per ingester	RAM per ingester	Local SSD (WAL)
50,000	3	2 cores	8 GiB	50 GiB
200,000	10	4 cores	16 GiB	100 GiB
1,000,000	30	4 cores	16 GiB	100 GiB

S3 storage estimates (after tail-based sampling at 0.5% keep rate):

Raw spans/sec	After sampling	Daily S3 writes	Monthly S3 cost (Standard)
10M	50K	~4 TB	~$92
100M	500K	~40 TB	~$920
200M	1M	~86 TB	~$1,978

S3 request costs add 10-30% on top of storage costs at high query volume. Use query-frontend caching (memcached) to reduce S3 GETs.

Compactor sizing: 1 compactor instance per ~500 GB of daily ingestion. 4-8 GiB RAM, 2-4 CPU cores each. Monitor tempodb_compaction_outstanding_blocks — if this grows continuously, add compactor capacity.

Failure Scenarios

Scenario 1: Ingester Crash — WAL Replay

Trigger: An ingester runs out of memory during a traffic spike and is OOM-killed.

Impact: In-memory spans not yet flushed to S3 are lost from memory, but the WAL on local SSD has a copy. New spans for trace_ids hashed to this ingester are temporarily unavailable until it restarts.

Recovery: The ingester restarts, replays the WAL from local SSD, reconstructs the in-memory buffer, and flushes to S3. With replication factor 2+, the replica ingester continues accepting spans during the outage. Recovery time: 30 seconds to 2 minutes depending on WAL size.

Scenario 2: Compactor Backlog

Trigger: Compactor is under-resourced or temporarily down. Small blocks accumulate on S3 faster than the compactor can merge them.

Impact: Block count grows. Each query must check more bloom filters and potentially read more Parquet blocks. Query latency degrades gradually — from sub-second to multi-second for trace ID lookups, from seconds to tens of seconds for attribute searches.

Detection: Monitor tempodb_compaction_outstanding_blocks. Alert when outstanding blocks exceed 2x the expected count for the current ingestion rate.

Recovery: Scale up compactor resources (more instances, more memory). The compactor will catch up by merging the backlog. Once block count returns to normal, query latency recovers. No data loss occurs — the blocks are all valid, just not optimally merged.

Scenario 3: S3 Query Latency Spike

Trigger: A broad TraceQL query ({http.status_code=500} across 7 days) triggers thousands of S3 GET requests. S3 throttles the request rate or the querier exhausts its connection pool.

Impact: The broad query dominates querier resources. Other queries (including targeted trace ID lookups) queue behind it. Overall query latency spikes.

Recovery: Configure query-frontend limits: max_bytes_per_trace, max_search_duration (limit attribute search time range), and per-tenant query concurrency limits. Use a dedicated querier pool for broad analytical queries so they don't affect operational trace lookups. Add memcached query-frontend cache to avoid repeated S3 reads for the same blocks.