System Design: Ad Click Aggregator (10B Clicks/day, Lambda Architecture, Fraud Detection)
Goal: Process 10 billion clicks/day. Deduplicate, detect fraud, aggregate across 1-min/1-hour/1-day windows, reconcile stream and batch, produce billing-accurate reports. Sub-minute freshness for dashboards, penny-accurate reconciliation for billing.
1. Problem
A large advertising platform must answer a deceptively simple question: how many legitimate, non-fraudulent clicks did each ad receive, and how much does the advertiser owe?
Two competing requirements make this hard:
- Real-time dashboards need sub-minute freshness. Advertisers adjust bids and budgets on live counts. A 10-minute lag means wasted spend on underperforming campaigns.
- Billing needs penny-level accuracy. An advertiser who spent $1.2M last month will dispute a $50 discrepancy. Stream-processing edge cases (late events, checkpoint replays, watermark drift) are invisible at small scale but compound into real money at 10B clicks/day.
No single processing path satisfies both. Stream gives freshness but not financial accuracy. Batch gives accuracy but not real-time visibility.
Lambda architecture runs both paths in parallel: the speed layer serves dashboards, the batch layer serves billing, and batch wins when they disagree.
Simple example. A mobile user clicks an ad for running shoes at 14:32:07 UTC.
- Within 30 seconds. The dashboard shows the click.
- Within 5 minutes. The fraud pipeline scores it LEGITIMATE.
- At 15:15 (15 minutes past the next hour boundary, when the hourly Spark batch job runs), the batch reconciliation job writes it to the billing ledger.
- At month-end. An invoice includes the click at $0.25, computed in integer micros.
Scale. 10B clicks/day = ~115K/sec sustained, ~580K/sec peak US business hours, ~1M/sec Black Friday spikes. Every layer must be horizontally partitioned.
Assumptions:
- Internal platform for a single ad company, not multi-tenant SaaS
- "Click" = user clicking an ad unit. Impressions tracked separately.
- Billing = legitimate, deduplicated clicks only
- Real-time dashboards use the speed layer. Monthly invoices use the batch layer. Batch wins disputes.
- Late-arriving clicks (slow mobile connections) up to 10 minutes are accepted.
2. Requirements
Functional Requirements
| ID | Requirement | Priority |
|---|---|---|
| FR-01 | Ingest click events via HTTP and produce to durable event log | P0 |
| FR-02 | Deduplicate clicks: same user clicking same ad within 30 seconds counts as one click | P0 |
| FR-03 | Real-time aggregation across 1-minute tumbling windows for live dashboards | P0 |
| FR-04 | Hourly aggregation for campaign performance reports | P0 |
| FR-05 | Daily aggregation for billing reconciliation | P0 |
| FR-06 | Fraud detection: flag bot clicks, click farms, click injection, click flooding | P0 |
| FR-07 | Advertiser dashboard API: click counts, spend, CTR by campaign/ad group/ad/geo/device | P0 |
| FR-08 | Batch reconciliation: hourly Spark job produces billing-accurate aggregates | P0 |
| FR-09 | Billing pipeline: monthly invoice generation from batch-reconciled data | P0 |
| FR-10 | Late event handling: accept clicks arriving up to 10 minutes late | P1 |
| FR-11 | Click-level audit trail for billing disputes | P1 |
| FR-12 | Anomaly alerting: notify operations when click patterns are abnormal | P1 |
| FR-13 | Advertiser-level fraud reports: show detected fraud breakdown | P1 |
| FR-14 | Backfill support: reprocess historical data after fraud model updates | P1 |
| FR-15 | Multi-currency billing support | P2 |
Non-Functional Requirements
| ID | Requirement | Target |
|---|---|---|
| NFR-01 | Throughput | 10B clicks/day, 115K clicks/sec sustained, 1M clicks/sec peak |
| NFR-02 | Dashboard freshness | < 1 minute (speed layer aggregation latency) |
| NFR-03 | Batch accuracy | Penny-accurate: batch reconciliation must match to $0.01 per advertiser per day |
| NFR-04 | Availability | 99.99% for ingestion (52 min downtime/year), 99.9% for dashboards |
| NFR-05 | Durability | Zero click loss once accepted (at-least-once ingestion, exactly-once aggregation) |
| NFR-06 | Fraud detection latency | < 5 minutes from click to fraud verdict |
| NFR-07 | Late event tolerance | Accept events up to 10 minutes past their event timestamp |
| NFR-08 | Horizontal scalability | Linear scale-out, no single bottleneck |
| NFR-09 | Data retention | Hot: 7 days, Warm: 90 days, Cold: 3 years (legal/audit) |
| NFR-10 | Recovery Time Objective (RTO) | < 60 seconds for speed layer, < 1 hour for batch layer |
| NFR-11 | Recovery Point Objective (RPO) | 0 for raw events (Kafka RF=3), < 1 minute for aggregates |
| NFR-12 | Geographic distribution | Multi-region ingestion, single-region processing (with failover) |
3. Architecture Overview
🔒 Premium section
4. Why Lambda (not Kappa)
🔒 Premium section
5. Scale Estimation
🔒 Premium section
6. Why Naive Approaches Fail
🔒 Premium section
7. Technology Selection
🔒 Premium section
8. Data Model
🔒 Premium section
9. API Design
🔒 Premium section
10. Click Ingestion and Deduplication
🔒 Premium section
11. Click Fraud Detection Pipeline
🔒 Premium section
12. Exactly-Once Aggregation with Flink
🔒 Premium section
13. Stream/Batch Reconciliation
🔒 Premium section
14. Billing Pipeline
🔒 Premium section
15. Bottlenecks and Mitigations
🔒 Premium section
16. Failure Scenarios
🔒 Premium section
17. Deployment
🔒 Premium section
18. Observability
🔒 Premium section
19. Security
🔒 Premium section
20. Consistency Model
🔒 Premium section
21. Key Design Principles
🔒 Premium section
22. Explore the Technologies
🔒 Premium section
