System Design: ML Model Serving Platform (Walnut Serve: Ranking Funnel, Feature Fetch, GPU Batching, Load Shedding, Training-Serving Skew)
Goal: Design a platform that serves a company's ML models to live traffic. Ours is Walnut Serve.
Its main tenant, a home-feed ranking funnel, picks 20 items from 100 million for each user. A checkout fraud model shares it with a tighter deadline.
We plan for 100 million daily users, 45,000 feed requests a second at design capacity, and a 180 ms p99 for each page.
Walnut Serve is our design. Facts about real products are dated evidence; every number of ours is labeled as ours.
1. What We Are Building
๐ Premium section
2. Requirements and a Quick Estimate
2.1 Functional Requirements
| # | The system must | Example |
|---|---|---|
| FR1 | Return a ranked page of 20 of 100 million items for a user | Jenna's first page at 18:42 |
| FR2 | Run the whole funnel, from retrieval to re-ranking | fr_90231 runs every stage |
| FR3 | Serve other tenants' models on the same platform, including checkout fraud scoring | Sam's checkout score for tn_fraud |
| FR4 | Read features with a known age, and log what was served | it_88412's counters, with write times |
| FR5 | Make new items servable within minutes | Tyler's video it_88412 |
| FR6 | Send each call to a warm replica of one frozen release, and move funnel models together | Marcus's gen-311 |
| FR7 | Shed and fall back by priority, and say what was skipped | Thursday's zone loss in US East |
| FR8 | Change rules, deny lists and batcher settings through a checked path that starts in one zone; ship feature lists only inside releases | The rejected file items-v88 |
| FR9 | Show health per stage, per model and per pool | Nina's dashboards |
Training is out of scope (the training post), and so are LLM token serving and ads auctions. The feed's fan-out and storage belong to the news feed post. Feature definitions and offline stores belong to the feature store post.
2.2 Non-Functional Requirements
An NFR (non-functional requirement) says how well the system must work. p50 and p99 are the values that 50% and 99% of measurements stay at or below.
A zone is a group of data centers with its own power and network; each region (section 20.1) has 3.
A fallback floor is the least engagement a cheaper ranking must keep, compared with the heavy ranker. Feature age is the time since a feature value was written.
| NFR | Target |
|---|---|
| NFR1 | Feed latency, server side: p99 at most 180 ms, p50 at most 80 ms, measured from the Feed API receiving the request to the page leaving it. |
| NFR2 | Fraud score latency, server side: p99 at most 25 ms. |
| NFR3 | 99.95% of feed requests get a page ranked by the heavy ranker; 99.99% get some page, ranked or from a fallback list. |
| NFR4 | Fallback floors: light ranker order keeps at least 90% of the heavy ranker's engagement; a fallback list keeps at least 60%. Checked monthly against a holdout. |
| NFR5 | Feature age: item counters at most 2 minutes old; user history at most 10 seconds old. |
| NFR6 | New items: 95% servable within 3 minutes of upload. |
| NFR7 | Losing one zone changes nothing users can see. Losing a region degrades ranking for the moved users for at most 30 minutes, with no blank pages. |
| NFR8 | A bad model or config release stops within 2 minutes of its first errors; rollback takes at most 10 seconds while the old release is loaded. |
| NFR9 | Served features: the features each shown item was scored with are logged for at least 99.9% of feed requests, so training reads what serving used. |
| NFR10 | GPU pools at most 70% busy at design capacity; CPU pools at most 60%. |
The heavy ranker (section 5.5) is the last, costly model, and the light ranker (section 5.4) is the cheap one before it.
A fallback list is a ready page that needs no model (section 10.5). A holdout is a share of users kept on a fixed ranking (section 15.5).
Where the targets come from:
- NFR1. A recommender paper cites a request budget of "hundreds of milliseconds" (Pinterest, 2025), so our 180 ms covers server time only.
- NFR3. YouTube home pages stayed blank for about an hour after a recommendation fault (a February 2026 news report), so 99.99% of feed requests must get some page.
- NFR7. New GPUs may not come when they are needed most (section 9.1), so a region move may degrade ranking instead of waiting for GPUs.
- NFR9. Uber trains on the feature values its models used at serving time; key-feature mismatch fell from over 10% to 0% (Uber blog, September 2026). So we log what we serve.
2.3 Load
A feed request is one full run of the funnel for one user. A page is the 20 items it shows. A scroll refill is a later page as the user scrolls.
Most scroll refills read the list already ranked (section 5.6), so they are not feed requests. When that list runs out, the refill starts a new feed request (section 10.4).
Three load words:
- Daily average. A day's total spread over 86,400 seconds.
- Busy hour. 2 times the daily average, 17:00 to 20:00 Mountain on weekdays.
- Design capacity. 1.5 times the busy hour, ready all day.
No vendor publishes these for Larkspur; we assume 100 million daily users with 13 fresh feeds each a day.
| What (world totals) | One-line sum | Result |
|---|---|---|
| Feed requests a day | 100 million x 13 | 1.3 billion |
| Daily average | 1.3 billion / 86,400 s | about 15,000 a second |
| Busy hour | 15,000 x 2 | about 30,000 a second |
| Design capacity | 30,000 x 1.5 | about 45,000 a second |
| Fraud scores | our assumption | 4,000 a second at design capacity |
At 1.5 times the busy hour, two of a region's three zones still carry the busy hour when one is lost (NFR7).
2.4 Funnel Sizes and Fleets
An item score is one model output for one candidate. A CPU server is a 64-core machine.
The inference GPU is a small, cheap GPU with 24 GB of memory (section 8.3). The hot pool is the fixed set of them that runs the heavy ranker (section 9.3).
Per feed request, the light ranker scores 1,600 candidates and the heavy ranker 400 (section 5.2). Both speeds in the table are our estimates. The GPU price, $0.80 an hour, is one cloud's June 2026 list price, our assumption.
| What | One-line sum | Result |
|---|---|---|
| Light ranker item scores | 45,000 x 1,600 | 72 million a second |
| Heavy ranker item scores | 45,000 x 400 | 18 million a second |
| Heavy ranker GPUs, at 300 feed requests a second per GPU (estimate) | 45,000 / 300 | 150 inference GPUs |
| Their cost, on demand | 150 x $0.80 x 730 hours | about $88,000 a month |
| Per feed request, on demand | $88,000 / (1.3 billion x 30 days) | about $2.30 per million |
| Feed orchestrator servers, at 400 feed requests a second per server (estimate) | 45,000 / 400 | about 113, run as 120 CPU servers |
All fleets are world totals at design capacity. The model math is not the hard limit; feature reads are, since every light ranker score needs an item row (section 7.3). GPU supply sets the failure plan (sections 9 and 10).
3. The API, the Data Model and the Stores
๐ Premium section
4. High-Level Design
๐ Premium section
5. Deep Dive: The Ranking Funnel
๐ Premium section
6. Deep Dive: Retrieval and Embeddings
๐ Premium section
7. Deep Dive: Real-Time Feature Fetch
๐ Premium section
8. Deep Dive: Model Servers, Batching and GPU Sharing
๐ Premium section
9. Deep Dive: Autoscaling With Scarce GPUs
๐ Premium section
10. Deep Dive: Deadlines, Partial Results, Shedding and Fallbacks
๐ Premium section
11. Deep Dive: The Second Tenant: Fraud Scoring
๐ Premium section
12. Deep Dive: Logged Features and Training-Serving Skew
๐ Premium section
13. Deep Dive: Releases, Shadow Traffic and Funnel Versions
๐ Premium section
14. Deep Dive: Generative Recommenders
๐ Premium section
15. Monitoring and Evaluation
๐ Premium section
16. Security
๐ Premium section
17. Bottlenecks
๐ Premium section
18. What Happens When Each Part Goes Down
๐ Premium section
19. Trade-Offs We Chose and Rejected
๐ Premium section
20. Multi-Region
๐ Premium section
21. Walnut Serve vs Real Products in 2026
๐ Premium section
22. In the Interview (45 and 60 Minutes)
๐ Premium section
23. What to Remember
๐ Premium section
24. Further Reading
๐ Premium section
Explore the Technologies
๐ Premium section