System Design: LLM Inference Serving (Walnut Infer: Continuous Batching, KV Cache Tiers, Cache-Aware Routing, Fair Tiers, Stream Resume)
Goal: Design a platform that serves several large language models to many customers at once. Ours is Walnut Infer.
After reading, you can draw it, explain each choice in a system design interview, and size the fleet with light math in tokens.
Walnut Infer is our design. Ridgeline AI, the company that runs it, is made up. Real products appear only as dated evidence and in section 25, and every number of ours is labeled as ours.
Reading guide: problem and design (1 to 4), deep dives (5 to 14), running it (15 to 22), the interview (23, 24), five real projects checked November 2026 (25) and sources (26).
For a 45-minute interview, read 1 to 9, 21, 22 and 23 first.
1. What We Are Building
๐ Premium section
2. Requirements and a Quick Estimate
2.1 Functional Requirements
Walnut Infer must:
- Generate. Stream the answer token by token, or return it whole.
- Reconnect and cancel. Let a dropped client pick up its stream, and stop a request on demand.
- Safe retry. Treat a repeated call with the same idempotency key as the first one.
- Batch jobs. Finish a file of requests within 24 hours, at a lower price.
- Tiers and reservations. Sell four speed tiers and reserved capacity (section 8).
- Many models. Serve several models plus customer adapters (section 13.2).
- 2026 workloads. Prompts up to 128,000 tokens, reasoning, tool calls and schema-bound output (section 12).
- Usage. Report five token counts per request (section 3.2).
- Cache isolation. Never reuse one customer's prompt memory for another (section 18.1).
2.2 Non-Functional Requirements and Per-Tier Targets
An NFR (non-functional requirement) says how well the system must work. p50 and p99 are the values that 50% and 99% of measurements stay at or under.
TTFT (time to first token) runs from our server receiving a request to the first token leaving it. Time per output token is the gap between two tokens of one stream.
A tier is the speed class bought for a request (section 8.3). A zone is a data center group with its own power and network; a region, such as US East, has 3 (section 20.1).
| Target area | Target |
|---|---|
| Time to first token | TTFT, server side, for prompts up to 8,000 uncached tokens: priority p99 at most 1 s; standard p99 at most 2 s and p50 at most 0.5 s. Flex and batch have no TTFT target. |
| Time per token | Time per output token, per stream: priority p50 at most 12 ms and p99 at most 25 ms; standard p50 at most 25 ms and p99 at most 50 ms. |
| Success | In a month, 99.95% of priority and 99.9% of standard requests within the customer's limits finish without a server error. |
| Resume | 99% of streams broken by a replica failure resume within 15 s and keep every text token already sent; a thinking or tool-use block that was open restarts from its start. |
| Fairness | While one customer floods a pool, every other customer within its limits keeps its p99 TTFT within 1.5 times its normal value. |
| Billing | Each token is billed once, at one rate, for the tier that served it. The daily reconcile gap is at most 0.01%. Refused flex requests, over-limit refusals and expired batch lines are not billed. |
| Batch | 99% of batch jobs finish within 24 hours. |
| Reserved capacity | Within its reserved rate, a request gets a stream slot within 1 s. |
| Zones | Losing one zone changes nothing customers see except resumed streams and, at the busy hour, flex and batch shedding and slower standard first tokens for up to 10 minutes. |
| Regions | Losing a region moves its priority and standard traffic within 5 minutes; its new flex and batch are refused until it returns, and at the busy hour the other regions shed their own flex and batch. |
| Quality | Quality probes run every 30 minutes on every live and canary release; a release whose probe score falls more than 1 point below the live release stops within 5 minutes of that probe. |
| Cache isolation | No KV block is shared between customers, except blocks of Ridgeline's own public system prompts. |
| Cost | GPU cost per million output tokens at the busy hour: at most $1.20 for standard and $2.40 for priority. |
A KV block is a slice of a prompt's GPU memory (section 6.2); a quality probe is a prompt with a known answer (section 16.3).
Goodput, our headline signal, counts only requests that meet both speed targets (section 17.1). A 429 is a "too busy" reply (section 8.2).
| Tier | TTFT p99 | Time per token p50 | Price against standard | When the pool is full |
|---|---|---|---|---|
| Priority | 1 s | 12 ms | 2 times | Above its limit, served as standard |
| Standard | 2 s | 25 ms | 1 time | Waits up to 30 s, then 429 |
| Flex | none | none | half | 429 at once, no charge |
| Batch | none | none | half | Waits inside its 24-hour window |
Where the targets come from:
- Speed. MLPerf sets 450 ms TTFT and 40 ms per token for a 70-billion-weight model (vendor-reported, April 2025).
- Batch. OpenAI and Anthropic batch costs 50% less, inside 24 hours (checked November 2026).
2.3 The Load in Tokens
Load words:
- Daily average. A day's total over 86,400 seconds.
- Busy hour. 09:00 to 12:00 Pacific on weekdays, 2 times the daily average.
- Design capacity. 1.15 times the busy hour, always running: 5% for failed GPUs, 10% for zone loss and scaling lag.
Input tokens are the prompt. Output tokens are visible output plus thinking tokens, hidden reasoning (section 12.1).
The prefix cache hit rate is the share of input tokens computed earlier for another request (section 6.3). A live stream is a request still sending tokens.
We assume 10,000 requests a second at the busy hour. Each has 4,000 input and 600 output tokens (400 visible, 200 thinking). At 40 tokens a second, a stream lasts 15 s.
| What (world total) | One-line sum | Result |
|---|---|---|
| Requests a day, at the daily average | 5,000 a second x 86,400 | 432 million |
| Input tokens, busy hour | 10,000 x 4,000 | 40 million a second |
| Prompt tokens to compute, at 70% or 35% hits | 40 million x 0.3; x 0.65 | 12 or 26 million a second |
| Output tokens | 10,000 x 600 | 6 million a second |
| Live streams | 10,000 x 15 s | about 150,000 |
Requests are the wrong unit: one long document costs as much as hundreds of chat turns. The 70% hit rate needs cache-aware routing (section 7).
2.4 Fleet, Cost and the Speed-Cost Trade-Off
Our generic names:
- The inference GPU. The newest data-center GPU, about 180 GB, 8 per server.
- The older GPU. The previous generation, 80 GB, about half the price.
- The small model. About 8 billion weights, on one older GPU.
- The mid model. About 70 billion weights, on one inference GPU.
- The large MoE model. Hundreds of billions of weights, a few tens of billions used per token (section 11.1).
GPU equivalents count an older GPU as half. Our estimates: one inference GPU decodes 2,500 tokens a second at standard pace and prefills 20,000. Prefill GPUs add a quarter to decode GPUs (600 on 2,400), hence 1.25.
| What (world total) | One-line sum | Result |
|---|---|---|
| Decode GPUs | 6 million / 2,500 | about 2,400 |
| Prefill GPUs, cache-aware or cache-blind routing | 12 or 26 million / 20,000 | about 600 or 1,300 |
| Fleet, busy hour and design capacity | 2,400 + 600; x 1.15 | about 3,000; 3,450 GPU equivalents |
| Cost a month | 3,450 x $8.55 x 730 hours | about $21.5 million |
| Cost per million output tokens: decode only; plus prefill; at priority pace | $8.55 / 9 million tokens an hour; x 1.25; x 2 | about $0.95; $1.20; $2.40 |
Our assumed $8.55 is the GPU-cloud median for B200-class rentals in the week of 9 November 2026 (a GPU price index).
The speed-cost curve is the main trade-off: more streams on a GPU make more tokens in total, but each stream gets slower.
At 40 tokens a second per stream, one inference GPU holds 2,500 / 40 = about 62 streams. At 80, it holds about 16 and makes about 1,250 tokens a second, so a priority token costs 2 times a standard one.
Decode sets the fleet size, about 2,400 GPUs. Priority's 10% of tokens runs at half pace. That adds about 240 decode GPUs, about 3,240 in all with prefill, still inside the 3,450.
3. The API, the Data Model and the Stores
๐ Premium section
4. High-Level Design
๐ Premium section
5. Deep Dive: Inside One Model Server
๐ Premium section
6. Deep Dive: The KV Cache
๐ Premium section
7. Deep Dive: Cache-Aware Routing and Fair Queuing
๐ Premium section
8. Deep Dive: Admission, Tiers and Reserved Capacity
๐ Premium section
9. Deep Dive: Streams, Cancel, Resume and Safe Retry
๐ Premium section
10. Deep Dive: Splitting Prefill and Decode
๐ Premium section
11. Deep Dive: Mixture-of-Experts and Wide Expert Parallelism
๐ Premium section
12. Deep Dive: 2026 Workloads
๐ Premium section
13. Deep Dive: Many Models on Few GPUs
๐ Premium section
14. Deep Dive: Token Metering and Billing
๐ Premium section
15. How This Maps to Kubernetes in 2026
๐ Premium section
16. Releases, Config Changes and Silent Quality Regressions
๐ Premium section
17. Monitoring and Goodput
๐ Premium section
18. Security
๐ Premium section
19. Bottlenecks
๐ Premium section
20. Multi-Region
๐ Premium section
21. What Happens When Each Part Goes Down
๐ Premium section
22. Trade-Offs We Chose and Rejected
๐ Premium section
23. In the Interview (45 and 60 Minutes)
๐ Premium section
24. What to Remember
๐ Premium section
25. Walnut Infer vs llm-d, NVIDIA Dynamo, vLLM, SGLang and KServe
๐ Premium section
26. Further Reading
๐ Premium section
Explore the Technologies
๐ Premium section