System Design: Rate Limiter (1M Requests/sec, Sub-ms Decision Latency)
1. The Problem in One Page
Picture a public API used by 10,000 companies, where we'll call each company a tenant and each tenant has many users calling many endpoints.
One tenant ships a bug that retries every failed call at once. In seconds that tenant is using most of your servers, and everyone else sees timeouts.
A rate limiter stops this. It counts requests per caller and answers 429 Too Many Requests when a caller goes over their share.
Counting is easy. Three things make it hard.
It sits in front of every request. Its time is added to every API call, so we aim for a median under one millisecond.
The count has to be shared. Say 40 gateway servers each count on their own, and a user may send 100 a minute. The user can send 100 to each server. That's 4,000 requests, and no server sees a problem.
So the count must live in one shared place, which means a network call on the hot path. Most of this design is about that conflict.
It must not become the outage. Block everything when the counter store is down and your API is down with it. Allow everything and the buggy tenant overloads you again. Section 10 shows the option in between.
The numbers we design for:
- 1M decisions per second at peak
- 10,000 tenants
- 100M live counter keys
- 40 gateway pods (running copies of the gateway server) at about 25,000 requests a second each
- Median decision under 1 ms
These are design inputs, not measurements from a real company.
Mistakes this design avoids:
- Counting in a SQL database. That's a million row locks a second.
- One network call per level. One script checks every level in one round trip.
- Trusting each server's clock. Clocks that disagree put the same request in different windows.
- One Valkey node. Valkey is an in-memory key-value store (an open-source fork of Redis). It keeps data in RAM, so reads and writes take microseconds. One node runs one command at a time, so it has a hard speed limit. And if it dies, every counter is gone.
- Sending a new rule everywhere at once. A wrong limit is the fastest way to block your own customers.
2. Requirements
2.1 Functional Requirements
| ID | Requirement | Priority |
|---|---|---|
| FR-01 | Decide allow or deny for each request | P0 |
| FR-02 | Limits per tenant | P0 |
| FR-03 | Limits per user | P0 |
| FR-04 | Limits per endpoint | P0 |
| FR-05 | A global limit across all tenants | P0 |
| FR-06 | Tell the client the limit, what is left, and when to retry | P0 |
| FR-07 | Change rules without restarting anything | P0 |
| FR-08 | Limits per IP for callers who aren't logged in | P1 |
| FR-09 | Burst allowance (token bucket) | P1 |
| FR-10 | Requests with different costs | P1 |
| FR-11 | Shadow mode, where a rule only logs what it would deny | P1 |
| FR-12 | Overrides and a bypass list | P1 |
| FR-13 | Analytics: who is limited, by which rule, how close each tenant is | P1 |
| FR-14 | A webhook when a tenant passes 80% of a limit | P2 |
2.2 Non-Functional Requirements
| ID | Requirement | Target |
|---|---|---|
| NFR-01 | Decision latency, median | under 1 ms |
| NFR-02 | Decision latency, p99 | under 2 ms |
| NFR-03 | Hard timeout on the counter call | 3 ms, then decide locally |
| NFR-04 | Throughput | 1M decisions a second |
| NFR-05 | Availability of the decision | The gateway always gets an answer |
| NFR-06 | Accuracy when healthy | within 5% for normal traffic (worst case in section 5.3) |
| NFR-07 | Accuracy when Valkey is down | a known maximum (section 10) |
| NFR-08 | Rule change reaches all pods | under 5 seconds by push, 30 seconds worst case |
| NFR-09 | Counter durability | Not needed for short windows |
| NFR-10 | Wrong denials | fewer than 1 in 10,000 denials (section 15.5) |
| NFR-11 | Decisions made with a real shared count | 99.9% over a month |
"p99" means 99 calls in 100 are faster than this. The p99 target is looser than the median because our counters sit in three zones (a zone is one data centre inside a cloud region; zones fail on their own, and a call between zones takes longer and costs money). A call to another zone is slower (section 8.2).
2.3 Out of Scope
- Stopping a large DDoS. That's the CDN's job. We only see traffic that reaches the gateway.
- Billing. A quota that turns into money needs a durable ledger. We show where it connects.
- Bot detection. A different system. The limiter only counts.
3. Where the Limiter Sits
🔒 Premium section
4. Architecture
🔒 Premium section
5. The Algorithms, All Eight
🔒 Premium section
6. The Counter Path
🔒 Premium section
7. Capacity
🔒 Premium section
8. How Should Valkey Be Set Up?
🔒 Premium section
9. The Rule Engine
🔒 Premium section
10. What Happens When Each Part Goes Down
🔒 Premium section
11. Bottlenecks and How to Fix Them
🔒 Premium section
12. Regions and Scaling
🔒 Premium section
13. Analytics
🔒 Premium section
14. Operations
🔒 Premium section
15. Cost, Migration and Ownership
🔒 Premium section
16. The Trade-off Ledger
🔒 Premium section
17. Questions You'll Get Asked
🔒 Premium section
Appendix: The GCRA Script
🔒 Premium section
Explore the Technologies
🔒 Premium section