Observability Platform at Scale - Metrics, Traces, Logs, and Profiles
Goal: Build one observability platform that carries all four pillars at once.
- Metrics: ingest 500 million per second. Counters, gauges, histograms, and summaries.
- Traces: collect per-request call trees across 500K+ services, spanning service boundaries.
- Logs: centralize structured output from every container in the fleet, with full-text search.
- Profiles: profile application performance continuously. CPU, memory, and lock contention snapshots, correlated to traces.
On top of those four, the platform provides real-time dashboards and alerting. Alerting runs on SLO (Service Level Objective) burn rates plus ML-assisted anomaly detection. It also gives you multi-tenant isolation, and long-term retention with automatic downsampling.
Architecture in One Minute
🔒 Premium section
1. Problem Statement
A few clarifications before getting into the architecture:
Scale clarification: This design targets extreme hyperscale: a stress-test upper bound, not a typical deployment. Most organizations ingest thousands to low millions of metrics/sec. Only the largest enterprises (thousands of microservices, hundreds of thousands of hosts) reach 10M-50M.
Three public data points sit at the frontier:
- Uber's M3 aggregates 500M metrics/sec pre-aggregation across 6.6B active series.
- Datadog processes 100+ trillion events/day.
- Grafana Mimir benchmarked 1B active series at ~50M samples/sec.
The 500M figure here matches Uber's pre-aggregation rate to pressure-test every layer. Section 7.3.1 shows how the architecture scales down linearly to 10-50M with fewer components.
Assumptions:
- This is a multi-tenant platform (like Datadog, Grafana Cloud, or New Relic). Multiple teams and organizations push metrics into the same infrastructure.
- A metric is "ingested" once it's in durable storage and queryable, which takes under 2 seconds.
- Each metric is a (name, label set, timestamp, float64 value) tuple. Labels are key-value pairs like
{service="checkout", env="prod", region="us-east-1"}. - Services expose metrics via Prometheus exposition format or push via OTLP (OpenTelemetry Protocol).
Operating model:
Responsibility split: The platform team operates collection infrastructure (OTel Collectors, eBPF agents, Kafka, storage, dashboards). Tenant teams instrument their applications using OpenTelemetry SDKs, Prometheus client libraries, or rely on eBPF-based auto-instrumentation for baseline telemetry. Tenants either expose a
/metricsendpoint (pull) or push OTLP to the platform's ingestion endpoint (push). Tenants do not run their own Prometheus servers or storage. The platform handles everything after the application boundary.Two-tier instrumentation: The platform deploys Grafana Beyla (eBPF-based) as a DaemonSet alongside the OTel Collector DaemonSet. Beyla auto-instruments HTTP/gRPC/SQL at the kernel level, with zero application code changes. You get RED metrics (Rate, Errors, Duration) and basic trace spans for every service.
The split matters because eBPF and OTel SDK see different things.
- eBPF watches network boundaries. It knows "POST /checkout took 500ms" and "a SQL call to port 5432 took 200ms." It has no idea what happened inside the service. It can't tell you which function was slow, what the actual SQL query text was, or why the request failed.
- OTel SDK instrumentation lives inside the application code. So it captures the internal breakdown: which method took how long, and the exact query text.
- The SDK also captures what eBPF never sees. Custom business metrics like
orders_placedorpayment_amount. Span attributes likeuser_idandcart_size. CPU and memory profiles showing which line of code is the bottleneck. - Profiles come only from SDK instrumentation, not from eBPF.
Why not just use the SDK for everything? Because it requires code changes.
Importing libraries, adding instrumentation, redeploying. Across 500+ services in five languages, that takes months.
eBPF covers every service in an afternoon: deploy one DaemonSet and you have RED metrics and basic trace spans for the entire cluster.
When a team eventually adds SDK instrumentation to a service, the OTel Collector deduplicates the overlap. It keeps the richer SDK version, and eBPF steps back for those endpoints.
Data transport: For tenants on the same Kubernetes cluster, the platform deploys OTel Collectors as a DaemonSet, one per node. Each collector scrapes tenant pods locally over mTLS, with no cross-network hop. For tenants on separate infrastructure, the platform exposes an HTTPS ingestion endpoint (OTLP or Prometheus remote_write) authenticated via per-tenant API key. Both paths converge into the same Kafka-backed pipeline.
Scope: Four Pillars of Observability
A production monitoring platform handles four signals that look nothing alike:
- Metrics answer "how much?" Small numbers (CPU at 45%, 200 req/sec), arriving constantly.
- Traces answer "what happened to this one request?" A call tree across service boundaries with timing data.
- Logs answer "what did the system say?" Structured text messages with full-text search.
- Profiles answer "why is this code slow?" CPU flame graphs, memory allocations, lock contention snapshots showing exactly which function is the bottleneck.
Each needs a different storage engine because the query patterns are completely different:
| Signal | Data Shape | Size per Event | Storage Engine | Query Pattern |
|---|---|---|---|---|
| Metrics | (name, labels, timestamp, float64) | 1.5-2 bytes (Gorilla compression, Section 8.2) | TSDB (VictoriaMetrics) | Aggregate: rate(x[5m]) |
| Traces | (trace_id, span_id, parent_id, service, duration, attributes) | 500-2000 bytes/span | Columnar/object store (Grafana Tempo) | By trace ID, filter by service+latency |
| Logs | (timestamp, severity, message, structured_fields) | 200-1000 bytes | Log store (VictoriaLogs) | Full-text search, filter by severity |
| Profiles | (profile_id, span_id, service, type, stacktrace, values) | 50-200 KB/snapshot | Profile store (Grafana Pyroscope) | By span ID, by service+time range |
What NOT to do:
A single Prometheus instance typically handles 3-10 million active series, depending on available RAM and scrape interval. At 500M metrics/sec with potentially billions of unique series, that setup will OOM (an Out of Memory crash) within minutes.
Running 50,000 standalone Prometheus instances isn't a solution either. That's 50,000 things to manage, 50,000 things to fail, and no global query capability.
The other common mistake: storing raw metrics forever. At 500M samples/sec, that's 43.2 trillion samples/day.
Even at 1.37 bytes per sample (Gorilla compression), that's 54 TB/day of hot storage. Without downsampling, the storage bill alone kills the project.
Downsampling applies to metrics only. Traces, logs, and profiles retain full resolution or are discarded entirely (Sections 9-11 explain why).
2. Functional Requirements
| ID | Requirement | Priority |
|---|---|---|
| FR-01 | Ingest metrics via pull (Prometheus scrape) and push (OTLP remote write) | P0 |
| FR-02 | Support four metric types: counter, gauge, histogram, summary | P0 |
| FR-03 | Real-time dashboard queries with sub-second latency over 15-day window | P0 |
| FR-04 | Continuous alert rule evaluation with configurable thresholds | P0 |
| FR-05 | SLO-based alerting with multi-window burn rate detection | P0 |
| FR-06 | Automatic metric downsampling: raw (15d) → 5-min aggregates (90d) → 1-hour aggregates (forever) | P0 |
| FR-07 | Multi-tenant data isolation with per-tenant cardinality limits | P0 |
| FR-08 | Distributed trace collection via OTLP with tail-based sampling | P0 |
| FR-09 | Centralized log ingestion with full-text search (50M lines/sec) | P0 |
| FR-10 | eBPF-based baseline collection (Grafana Beyla) for automatic RED metrics and trace spans | P0 |
| FR-11 | Value-based data routing: full fidelity for errors/SLO-critical, sample/drop health checks and debug noise | P0 |
| FR-12 | ML-assisted anomaly detection with seasonal baselines, adaptive thresholds, and cross-service correlation | P0 |
P1 requirements (recording rules, RBAC, exemplar correlation, cross-region federation, continuous profiling) and P2 requirements (metric relabeling, cost attribution) are addressed in their respective deep-dive sections.
3. Non-Functional Requirements
| Requirement | Target |
|---|---|
| Ingestion throughput | 500M data points/sec sustained |
| Query latency (15-day window) | p50 < 50ms, p99 < 200ms |
| Query latency (90-day window, downsampled) | p50 < 200ms, p99 < 500ms |
| Ingestion-to-dashboard visibility | < 2 seconds |
| Alert rule evaluation interval | 15 seconds (configurable) |
| Alert firing latency (ingestion to notification) | < 30 seconds |
| ML anomaly detection latency | < 60 seconds from ingestion to anomaly signal |
| Profile ingestion throughput | 50K profiles/sec sustained |
| Availability | 99.99% (52 min downtime/year) |
| Data durability | 99.999% (no data loss on single node failure) |
| Active time series capacity | 10 billion+ |
| Retention | Raw: 15 days, 5-min: 90 days, 1-hour: indefinite |
| Tenant isolation | No cross-tenant data leakage, noisy neighbor protection |
Design Principles
🔒 Premium section
4. High-Level Approach & Technology Selection
🔒 Premium section
5. High-Level Architecture
🔒 Premium section
6. Data Model
🔒 Premium section
7. Back-of-the-Envelope Estimation
🔒 Premium section
What Breaks First at Scale
🔒 Premium section
8. Deep Dives: Metrics Pipeline
🔒 Premium section
9. Distributed Tracing at Scale
🔒 Premium section
10. Centralized Logging
🔒 Premium section
11. Continuous Profiling: The Fourth Pillar
🔒 Premium section
12. Identify Bottlenecks
🔒 Premium section
13. Failure Scenarios
🔒 Premium section
14. Observability (Meta-Monitoring)
🔒 Premium section
15. Deployment Strategy
🔒 Premium section
16. Security
🔒 Premium section
Explore the Technologies
🔒 Premium section
