System Design: Ad Exchange (Real-Time Bidding, Sub-100ms Auctions, DSP/SSP, Impression Serving)
Goal: an ad exchange running real-time bidding. The pieces it has to hit:
- Scale. 1M ad requests/sec, with top-5-of-50 smart DSP selection.
- Auctions. First-price, resolved inside 100ms.
- Rules. Publisher floor prices plus supply-chain rules (ads.txt, sellers.json, schain).
- Serving. CDN-served creatives and independent impression tracking.
- Money. $4B/month of spend reconciled between DSPs and publishers.
That's ~25B impressions a day across three regions (us-east, eu-west, ap-south).
1. How One Ad Actually Gets Served
🔒 Premium section
2. Problem Statement
Online advertising pays for most of the open web. Every page load triggers an auction that has to resolve before content finishes rendering, roughly 100ms from when the request leaves the publisher to when winning markup comes back.
Miss the budget and the slot stays empty: publisher loses revenue, advertiser loses reach.
The exchange sits in the middle. It takes supply from SSPs, picks DSPs to ask, runs a first-price auction, returns the winner, tracks the impression independently, and settles money at the end of the day. What makes it hard is the combination of latency, fan-out, and real money on every auction.
Latency is the dominant constraint. Naive fan-out to all 50 DSPs is 50KB × 1M QPS = 50 GB/sec outbound, and every auction waits on the slowest of fifty bidders. The realistic baseline is scoring each DSP per request and fanning out only to the top 5 most likely to bid competitively (see §10.2).
Scale matters even after that trim. 1M QPS sustained, 2M peak during US prime time, 5M outbound bid requests per second, ~100 auction-server pods across three regions. Every hot-path lookup must be sub-millisecond.
The money invariant: an auction bug that lets a $0.10 bid beat a $2.00 bid leaks $1.90 per impression. At 300K imps/sec that's $570/sec, $50M a day.
Auction integrity is the product, not a nice-to-have. Winning bids log at 100%, losing bids are sampled, and DSP-reported numbers are reconciled nightly.
Underneath all of that sit the long-running concerns. Each has to be handled in under 2ms of combined pre-bid overhead:
- Bot traffic, click farms, datacenter-hosted "users", pre-bid IP/ASN filters
- Domain spoofing, ads.txt, sellers.json, and the
schainobject close this loop - Headless browsers, WebGL/canvas fingerprints are the giveaway
- Consent (GDPR, CCPA), PII stripped on demand; a single GDPR violation can run 4% of global revenue
Quick numbers to anchor the rest of the post:
| Metric | Target |
|---|---|
| Ad requests per second (sustained) | 1,000,000 |
| Ad requests per second (peak) | 2,000,000 |
| Impressions per second (30% fill) | ~300,000 |
| Impressions per day | ~25 billion |
| DSPs registered | 50+ |
| DSPs per auction (top-N) | 5 |
| Auction latency p99 | < 100 ms |
| DSP response timeout | 60 ms |
| Monthly spend through exchange | ~$4 billion |
| Exchange take rate | 10–15% |
What we're deliberately avoiding
- Don't fan out to every DSP. Top-5 smart selection cuts outbound traffic 10× for a <2% fill-rate hit.
- Don't put Valkey in the hot path for every enrichment lookup. An in-process LRU fronts it and absorbs >95% of reads.
- Don't log every losing bid to Kafka. Sample losing bids at 1%; tee the full stream to S3/Iceberg for cheap durable storage.
- Don't call DSPs sequentially. Parallel HTTP/2 fan-out with a 60ms deadline.
- Don't trust DSP-reported impression counts for money. The exchange tracks its own and reconciles nightly.
3. Functional Requirements
| ID | Requirement | Priority |
|---|---|---|
| FR-01 | Accept bid requests from SSPs via OpenRTB 2.6 and run first-price sealed-bid auctions in < 100 ms p99 | P0 |
| FR-02 | Smart-select top 5 eligible DSPs per auction and fan out in parallel with 60 ms timeout | P0 |
| FR-03 | Enforce publisher floor prices and block-list (categories/advertisers) per publisher config | P0 |
| FR-04 | Return winning ad markup (HTML for banner, VAST XML for video) to the SSP within budget | P0 |
| FR-05 | Serve ad creatives through CDN edge nodes with cache headers for efficient delivery | P0 |
| FR-06 | Track impressions via server-side pixel (1×1 GIF) with deduplication | P0 |
| FR-07 | Track clicks via redirect URL with destination validation | P0 |
| FR-08 | Enforce exchange-level creative dedup (max N impressions of same creative per user per hour) as an ad-quality measure. Per-campaign frequency capping is a DSP responsibility. | P1 |
| FR-09 | Track per-DSP spend in near-real-time via Flink streaming for credit-limit enforcement and settlement | P0 |
| FR-10 | Validate supply chain: ads.txt, sellers.json, and schain object on every request | P0 |
| FR-11 | Check consent (TCF / US Privacy string) and strip PII from bid requests when required | P0 |
| FR-12 | Publish tracking events (impressions, clicks, viewability, auction results) to Kafka for billing and analytics | P0 |
| FR-13 | Reconcile exchange-tracked impressions with DSP-reported impressions daily; flag discrepancies > 0.01% | P1 |
| FR-14 | Provide publisher and DSP management APIs (floor prices, DSP onboarding, settlement reports) | P1 |
| FR-15 | Support banner, video (VAST 4.2), and native ad formats | P0 |
| FR-16 | Pre-bid fraud filtering: IP reputation, user-agent signature, datacenter detection, ASN reputation | P0 |
4. Non-Functional Requirements
| Dimension | Target |
|---|---|
| Auction latency (p50) | < 50 ms |
| Auction latency (p99) | < 100 ms |
| Fill rate | > 30% (varies by publisher and market) |
| Availability | 99.95% (4.4 hours/year planned + unplanned downtime) |
| Tracking pipeline loss | < 0.01% event loss end-to-end |
| Billing accuracy (reconciled) | ±0.01% of DSP-reported impressions |
| CDN cache hit rate | > 95% |
| DSP connection pool warm starts | All DSPs kept warm via periodic health pings |
| Multi-region failover | < 60 seconds (DNS-based geo failover) |
| Deployment rollback | < 5 minutes for any component |
5. High-Level Approach & Technology Selection
🔒 Premium section
6. High-Level Architecture
🔒 Premium section
7. Back-of-the-Envelope Sizing
🔒 Premium section
8. Data Model
🔒 Premium section
9. API Design
🔒 Premium section
10. Deep Dives
🔒 Premium section
11. Bottlenecks
🔒 Premium section
12. Failure Scenarios
🔒 Premium section
13. Deployment
🔒 Premium section
14. Observability
🔒 Premium section
15. Security
🔒 Premium section
Explore the Technologies
🔒 Premium section
