Building a Fail-Safe, Scalable Top-K Streaming System
Goal: Compute the most-watched videos across multiple time windows (10 min, daily, monthly, lifetime), serve results in real time, and survive component failures with zero data loss. Scale: 7 billion events per day.
Here's the full architecture, from ingestion to serving, with failure scenarios and the math behind every decision.
1. Problem Statement
A video platform (YouTube / TikTok scale) needs to answer: "What's trending right now?"
Detailed functional requirements are in Section 2. But first, what NOT to build:
What NOT to do:
- Insert one DB row per watch event (7B rows/day = instant death)
- Compute Top-K with
SELECT ... ORDER BY count DESCover raw events - Use Redis as source of truth (it's volatile memory)
- Count in Redis with ZINCRBY (no replay, no correctness after restart)
- Build a monolith that does ingestion + computation + serving
2. Functional Requirements
| ID | Requirement | Priority |
|---|---|---|
| FR-1 | Top-K videos in the last 10 minutes (sliding window, what's hot right now) | P0 |
| FR-2 | Top-K videos today (daily tumbling window, daily trending) | P0 |
| FR-3 | Top-K videos this month (monthly tumbling window, monthly leaderboard) | P0 |
| FR-4 | Lifetime top videos (hall of fame, running count with no window) | P1 |
| FR-5 | Near-real-time freshness -- results update within seconds, not minutes | P0 |
| FR-6 | No data loss, even during failures of any component | P0 |
| FR-7 | Multi-tenant support -- per-region breakdowns (e.g., Top-K in India, US, EU) | P1 |
| FR-8 | Multi-tenant support -- per-category breakdowns (e.g., Top-K in music, gaming) | P1 |
| FR-9 | Combined dimensional queries (per-region AND per-category Top-K) | P2 |
| FR-10 | Write API for event ingestion (accept video watch events from clients) | P0 |
| FR-11 | Read API for Top-K queries (serve Top-K results with metadata) | P0 |
| FR-12 | Batched event ingestion for SDKs (up to 500 events per batch) | P1 |
| FR-13 | Freshness tracking in API responses (how old the data is) | P1 |
3. Non-Functional Requirements
| Requirement | Target |
|---|---|
| Throughput (average) | 81,000 events/sec (7B daily watches / 86,400s) |
| Throughput (peak, 10x) | ~810,000 events/sec |
| Throughput (viral spike) | ~2,000,000 events/sec (must plan for this) |
| API read latency (p99) | < 10ms |
| Data freshness | < 30 seconds (Top-K results reflect recent activity) |
| Availability | 99.99% (degrade gracefully, never hard-fail) |
| Durability | Zero data loss -- no event dropped even during component failures |
| Horizontal scalability | Scale linearly with traffic; handle 25x spikes without rearchitecture |
| Processing semantics | Exactly-once end-to-end -- every event counted exactly once |
Loading…
4. High-Level Approach & Technology Selection
🔒 Premium section
5. High-Level Architecture
🔒 Premium section
6. Back-of-the-Envelope Estimation
🔒 Premium section
7. Data Model
🔒 Premium section
8. API Design
🔒 Premium section
9. Deep Dives
🔒 Premium section
10. Identify Bottlenecks
🔒 Premium section
11. Failure Scenarios
🔒 Premium section
12. Deployment Strategy
🔒 Premium section
13. Observability
🔒 Premium section
14. Security
🔒 Premium section
Key Design Principles
🔒 Premium section
Explore the Technologies
🔒 Premium section
CrackingWalnuts