Design YouTube Top K

Each phase is what the interviewer expects you to do and say. Concrete steps, not topic hints. The diagrams are what you sketch on the board.

1. 1

   Clarify Requirements and Scale

   5 min

   GoalPin down exact-vs-approximate, the window taxonomy, and the exactly-once expectation before you draw a single box. This is a correctness problem disguised as a throughput problem, and the interviewer needs to see you frame it that way.

   Do & Say

   1. ASK·1ASK: “Are we computing exact top-K or approximate?” “Approximate (Count-Min Sketch, HeavyKeeper) is cheaper but undercounts.” “Exact requires per-key state in Flink, which is fine for 50M unique videos but breaks for top-K over billions of IPs.” Land on exact because video counts are bounded.
   2. SAY·2Pin the scale: “7B view events/day = ~81K events/sec sustained”, “~810K peak (10x)”, “~2M during a viral moment”. K=100 across 4 window types: “10-min sliding”, “daily tumbling”, “monthly tumbling”, “lifetime counter”. Write “81K avg, 2M spike, K=100” on the board.
   3. SAY·3Pin the freshness expectation: “Emit top-K every 30 seconds.” “End-user sees top-K with 30-60s staleness, which is fine for a leaderboard.” Write “emit every 30s” next to the QPS.
   4. SAY·4Pin the correctness requirement: “Exactly-once end-to-end.” “A view event must affect the count once and only once, even through Flink restarts and Kafka rebalances.” Write “exactly-once” under everything else. This is the load-bearing constraint.
   5. SAY·5Scope in: “ingestion”, “dedup”, “windowed aggregation”, “top-K selection”, “serving with fallback”. Out: “spam/bot filtering (assume upstream)”, “recommendation engine”, “monetization counts”. Wait for confirmation.

   Interviewer is grading: You ask exact-vs-approximate before drawing anything. You bring up exactly-once as the load-bearing constraint, not an afterthought. You name the 4 window types explicitly (sliding vs tumbling vs lifetime) and don't conflate them.

2. 2

   API and Data Model

   5 min

   GoalOne write endpoint (event ingest), one read endpoint (get top-K for a window). Justify Kafka as source of truth, Flink keyed state as the only place exact per-video counts live, Redis sorted set as a display board not a counter.

   Do & Say

   1. WRITE·1Write the ingest endpoint: “POST /v1/events {video_id, user_id, event_timestamp, watch_duration_sec, session_id, region}” returns “a 202 Accepted after a synchronous Kafka produce with acks=all”. “Latency target sub-50ms p99 at the API.”
   2. WRITE·2Write the read endpoint: “GET /v1/topk?window=10min&region=US&k=100” returns the current top-K with a “freshness_ms” field. “CDN-cached for 10 seconds with stale-while-revalidate=30”, so most traffic never hits the API.
   3. DRAW·3Sketch the Avro event schema (~150 bytes serialized): “event_id”, “video_id”, “user_id”, “event_timestamp (client clock)”, “server_timestamp (API clock, used for watermarks)”, “region”, “category”, “device”, “watch_duration_sec”, “session_id”. Two timestamps because “client clocks are skewed and watermarks need a reliable one”.
   4. SAY·4Justify Kafka as source of truth: “128 partitions on video.watches topic”, “RF=3, acks=all”, “partition key = video_id”. “Co-partitioning means Flink can keyBy(video_id) without repartitioning.” “7 days hot retention, 30 days tiered to S3.” “Everything downstream is derived state. If Flink dies we replay.”
   5. DRAW·5Sketch the Redis structure: “A sorted set per (window, dimension) at key video:topk:{window}:{dimension}:current.” “Members are video_ids, scores are exact counts.” “Flink writes via ZADD (overwrite), never ZINCRBY.” “Redis is a display board, not a counter.” “The counter lives in Flink RocksDB.”

   Interviewer is grading: You make Kafka the only source of truth and treat Redis as a derived cache. You insist on ZADD (overwrite) not ZINCRBY (increment) and can explain why. You volunteer the dual timestamp (event_timestamp + server_timestamp) for watermarks before being asked.

3. 3

   High-Level Design (draw the streaming pipeline)

   10 min

   GoalOne diagram showing the Kafka -> Flink -> dual sink architecture, with the 4 parallel window branches and the 3-tier read fallback. The exactly-once chain should be visible in the arrows.

   Draw on the board
   Do & Say

   1. DRAW·1Draw the ingest path. “API Gateway uses Kafka idempotent producer.” Settings: “acks=all”, “retries=MAX_INT”, “max.in.flight.requests=5 with idempotence on so retries don't duplicate”. “Partition by video_id, 128 partitions across 12 brokers.” “~15K events/sec per partition at viral spike, well within the single-thread ~50K/sec ceiling.”
   2. DRAW·2Draw the Flink topology. “Kafka source with checkpointed offsets (60-second incremental checkpoints to S3).” “Dedup operator keyed by event_id with 60-second state TTL catches producer retries that slipped through.” “Then keyBy(video_id), a no-op because the topic is already partitioned by video_id.”
   3. DRAW·3Draw the four window branches: “10-min sliding window slides every 30s with BoundedOutOfOrderness 30s and allowedLateness 5 min”, “daily and monthly tumbling with continuous-trigger every 30s so live count is visible mid-window”, “lifetime is a per-video atomic counter in RocksDB”.
   4. SAY·4Defend per-video counts: “Flink keyed state stores 50M (video_id, count) pairs per window”, “200 bytes each = 10 GB total in RocksDB”, “incremental checkpoints 2-5 GB”. “Top-K is recomputed every 30s via a min-heap of size K.” “Count-Min Sketch saves memory but at 50M keys we don't need to approximate.”
   5. SAY·5Defend min-heap over sorting: “O(log K) per update vs O(N log N) per emission.” “For K=100 and N=50M, that's 7 ops vs 1.3 billion.” “Heap runs in Flink, selects top-K from exact counts.” “Flink ZADD-overwrites Redis with the K=100 entries.” “Redis never runs the heap.”
   6. DRAW·6Draw the dual sink: “Flink writes to Redis via ZADD (at-least-once is fine because overwrite is idempotent)”, “Flink writes to Postgres via 2PC sink: pre-commit on barrier, commit on checkpoint success”. “Redis is the fast path, Postgres is the durable record and the API fallback.”
   7. DRAW·7Draw the 3-tier read fallback: “Read API hits Redis with 5ms timeout”, “Redis slow → fall through to Postgres at 50ms”, “Postgres down → serve stale cached response or 503”. Never hard-fail. “CDN with 10s max-age plus stale-while-revalidate=30s absorbs 99%+ of traffic anyway.”

   Interviewer is grading: You volunteer the exactly-once chain (idempotent producer + checkpointed offsets + ZADD + 2PC) without being asked. You draw all 4 windows in parallel branches, not as a single window with different sizes. You insist on min-heap inside Flink, not sorting in Redis. You volunteer the 3-tier read fallback.

4. 4

   Deep Dive: Exactly-Once, Watermarks, Failure Recovery

   15 min

   GoalThree sub-dives. First, exactly-once end-to-end with each hop's guarantee. Second, watermarks and late-event handling with the BoundedOutOfOrderness logic. Third, failure recovery: Flink crash, Redis crash, Kafka rebalance.

   Draw on the board
   Do & Say

   1. SAY·1Pivot to the exactly-once chain. “Producer to Kafka uses idempotent producer (enable.idempotence=true).” “Per-partition sequence numbers stop retry duplicates.” Kafka to Flink: “Flink commits offsets only on checkpoint success, bundling offsets with state atomically.” “A Flink crash replays from last checkpoint, every event processed exactly once.”
   2. SAY·2Continue: “Flink to Redis is technically at-least-once but ZADD is idempotent (overwrite, not increment), so re-emitting the same top-K is safe.” “Flink to Postgres uses a 2PC sink: pre-commit writes the row uncommitted, commit fires on checkpoint success.” “Crash between pre-commit and commit rolls back.” “End-to-end is effectively exactly-once.”
   3. SAY·3Cover the ZADD-not-ZINCRBY decision: “ZINCRBY makes Redis the counter.” “On Flink restart, replayed events would increment again.” “ZADD makes Flink the counter.” “Replay just rewrites the same value. Idempotent.” “This is the single most common bug in streaming systems. Make Redis a display board, never a counter.”
   4. SAY·4Pivot to watermarks: “Events arrive late.” “A view at 12:00:00 might reach Kafka at 12:00:45 due to mobile network delay.” “If the 10-min window [11:50:00, 12:00:00] has already closed, the event is too late.”
   5. SAY·5Defend BoundedOutOfOrderness 30s: “Flink waits 30s past latest event time before closing a window.” “99% arrive within 30s per real network data.” “allowedLateness=5min recomputes the window for late events.” “sideOutputLateData ships >5min events to a monitoring sink so we alarm on bot replays.”
   6. SAY·6Defend server_timestamp over event_timestamp for watermarks: “event_timestamp comes from the client clock, can be in the future, past, or jump”. “server_timestamp is when our API received the event, reliable.” “event_timestamp for windowing semantics, server_timestamp for watermarks.” “The 30s OutOfOrderness is on server_timestamp.”
   7. SAY·7Defend withIdleness 1min: “If one Kafka partition has no traffic for 60 seconds (low-traffic region at 3 AM), Flink advances the watermark anyway.” “Otherwise the whole pipeline stalls waiting for that partition.”
   8. SAY·8Failure recovery. Flink crash: “TaskManager dies.” “JobManager detects within 30s, restarts from last S3 checkpoint (~30s including RocksDB warmup).” “Kafka source resumes from checkpointed offset, events replay.” “ZADD re-overwrites Redis.” “2PC to Postgres rolls back uncommitted txns, replay produces the same commit.” Zero data loss.
   9. SAY·9Redis crash: “Redis Cluster failover within 15s.” “During the gap, Read API falls back to Postgres (slower but durable).” “Flink keeps writing top-K to both sinks.” “New primary up, next 30s emission repopulates fully.” Net effect: “degraded p99 for ~15-45s, no data loss.”
   10. SAY·10Kafka broker fail: “One broker dies.” “RF=3, leader election promotes a healthy replica within seconds.” “Producers buffer up to 5s during failover.” “Events land at new leader after reconnect.” “No loss since acks=all and min.insync.replicas=2 means writes already had 2 replicas.”
   11. SAY·11Cover the dedup caveat: “Producer retries can still happen if the network blips between client and our API Gateway before idempotence kicks in.” “Flink has a Dedup operator keyed by event_id with 60-second state TTL.” “Catches retries within 60s, which covers the realistic retry window.”

   Interviewer is grading: You walk the exactly-once chain hop-by-hop, naming the guarantee at each link (idempotent producer, checkpointed offsets, ZADD overwrite, 2PC). You volunteer server_timestamp over event_timestamp for watermarks. You walk through all three failure modes (Flink, Redis, Kafka) with concrete recovery times in seconds.

5. 5

   Trade-offs, Risks, and Wrap-up

   5 min

   GoalName three deliberate trade-offs (exact counts via per-key state, 30-second emit cadence, exactly-once checkpoint pause), give the multi-dimensional top-K cost, close in one breath.

   Do & Say

   1. SAY·1Exact counts trade-off: “We pick exact since 50M videos fit in 10 GB of Flink RocksDB state.” “Billions of IPs would force HeavyKeeper or Count-Min Sketch at ~1% error.” “Approximate is cheaper but harder to reason about (overcounts, no drill-down).” “10 GB buys exactness and debuggability.”
   2. SAY·230s emit cadence trade-off: “User sees top-K with 30-60s staleness, not real-time.” “We accept this because emit-on-every-event means 81K Redis writes/sec instead of 3.3.” “CDN (10s max-age plus stale-while-revalidate=30s) absorbs 99% of traffic and tolerates the staleness.”
   3. SAY·3Exactly-once checkpoint pause: “Exactly-once mode pauses processing briefly during barrier alignment, 100-500ms per 60s checkpoint.” “Negligible for a 30s emit cycle.” “At-least-once skips barrier alignment for a small latency win but forces us to handle duplicates downstream.” Not worth it.
   4. SAY·4Multi-dimensional cost: “Per-region or per-category top-K adds a Flink branch keyed by region+video_id or category+video_id.” “Each dimension doubles state.” “4 regions × 10 categories = 40x state.” “Limit to what the product actually ships.” “Avoid region × category × device = 200 branches.”
   5. SAY·5Close: “Kafka as source of truth”, “Flink RocksDB for exact per-video counts across 4 windows”, “min-heap K=100 emits every 30s”, “dual sink Redis (ZADD) plus Postgres (2PC)”. “Exactly-once via idempotent producer + checkpointed offsets + idempotent sinks.” “3-tier read Redis → Postgres → stale cache.” Risk: “late events past 30s.”

   Interviewer is grading: You name trade-offs as 'we accept X because Y'. You volunteer the multi-dimensional cost warning (don't multiply dimensions blindly). You close in one breath naming Kafka source-of-truth, Flink keyed state, min-heap, dual sink, exactly-once, 3-tier fallback, and the load-bearing risk.

Questions an interviewer is likely to ask after your walkthrough. Rehearse the short answer.

1. Why ZADD and not ZINCRBY?
2. Why exact counts instead of HeavyKeeper or Count-Min Sketch?
3. Walk me through the exactly-once chain hop by hop.
4. Why server_timestamp for watermarks instead of event_timestamp?
5. What happens when Flink crashes mid-window?
6. What if Redis goes down?
7. How do per-region or per-category top-K change the design?
8. How do late events get handled?