System Design: LLM Evaluation Platform (Walnut Evals: Test Sets, Batch Runs, Model Judges, Paired Statistics, Release Gates, Online Evals)
Goal: Design the platform that decides whether a new model, prompt or tool may ship. Ours is Walnut Evals, at Walnut AI in Austin, for a support assistant and a coding agent.
We plan for 50 teams, 306,000 test items, 254 runs a day and about 5.65 billion input tokens a day (word pieces a model reads), with a gate decision in about 90 minutes and online checks on 1% of live conversations.
Walnut Evals is our design. Real facts from labs and vendors are dated evidence; every number of ours is labeled as ours.
1. What We Are Building
๐ Premium section
2. Requirements and Scale
2.1 What the System Must Do
| ID | Requirement |
|---|---|
| FR1 | Register and version suites, with owners and sealed access (section 5). |
| FR2 | Register release records: model snapshot, prompt, tools with versions, retrieval index version, settings, router rules and serving config (section 3.1). |
| FR3 | Run suites on one or more release records through the providers' batch and synchronous APIs, with trials and retries (section 6). |
| FR4 | Run agent and code tasks in isolated sandboxes and grade the state they leave (section 7). |
| FR5 | Grade with exact match, code, model judges and people, and check the judges against people (sections 8 and 9). |
| FR6 | Report paired differences with ranges, per suite and per slice, a tagged part of a suite such as one language (section 10). |
| FR7 | Apply versioned gate policies, return pass, review or block, and hand the decision to the rollout system (section 11). |
| FR8 | Sample production traffic, scrub it, judge it and alert on drops (section 12). |
| FR9 | Keep every answer, grade and trace for replay and dashboards (section 13). |
| FR10 | Serve 50 teams with quotas, access control, cost reports and deletion (section 14). |
2.2 How Well It Must Do It
p50 is the time half of all runs beat; p95, the time 95 in 100 beat. Availability is the share of time a service works. An NFR, a non-functional requirement, says how well the system works.
The candidate is the release under test; the baseline is the release in production. A rule's margin is the largest drop in pass rate it accepts, in percentage points. Its power is the chance it passes a candidate with no real change.
| ID | Requirement | Target |
|---|---|---|
| NFR1 | Gate latency | "A release run reaches a gate decision within 2 hours of its start in 95 of 100 runs. The run waits at most 60 minutes on the providers' batch tier; after that, its unfinished calls move to the synchronous path. A run starts when it takes a release slot (section 6.4)." |
| NFR2 | PR checks | A PR check reports within 15 minutes in 95 of 100 checks. |
| NFR3 | Decision quality | "For each main rule, a candidate that is truly worse than the baseline by the rule's margin passes at most 5% of the time. A candidate with no real change passes the whole policy at least 80% of the time, so every main rule needs a suite of its planned size (section 10.5)." |
| NFR4 | Results | "Each answer and each grade is stored once per idempotency key. A retried or duplicated provider call can be billed twice; it is counted once. One decision never mixes judge versions or suite versions." |
| NFR5 | Online detection | "In US busy hours, a drop of 2.5 points or more in an online check over all US Walnut Assist traffic is flagged by the first hourly check on a full hour of it, in at least 95 of 100 cases. EU checks need larger drops (section 12.4)." |
| NFR6 | Judges | "A judge may grade a gate rule only while its Cohen's kappa with people, on each anchor set dimension the rule uses, is at least 0.70 and no more than 0.08 below the kappa between two people. Every new judge version is checked before use, and the anchor set is re-judged every day." |
| NFR7 | Privacy | "Production traces are scrubbed of personal data before they are stored, and kept at most 90 days. Customers whose contracts forbid retention are never sampled. A deletion request removes the data from the Trace store and the Review app within 24 hours, and from every stored copy, promoted test items and the second region included, within 30 days." |
| NFR8 | Reproducibility | "Regrading or re-reporting a stored run from its stored answers under the same policy version gives the same decision. A fresh rerun lands inside the reported 90% range about 9 times in 10; when it does not, the report says why." |
| NFR9 | Isolation | "Each agent trial starts in a fresh sandbox with no network except the package mirror and the Model gateway, and its grader runs outside it. A trial that fails for an infrastructure reason is retried and never counted as a model failure; a trial out of time or steps is a model failure; if more than 2% of a suite's trials fail this way, that suite's rule goes to review." |
| NFR10 | Availability | Release runs can be started 99.9% of each month. |
Later sections explain PR checks, release runs and busy hours (2.3), the synchronous path (2.4), idempotency keys (3.1), the Model gateway (4.3), the Review app and package mirror (4.4), the batch tier (6.3), kappa (9.3), ranges (10.2), the Trace store (13.3) and regions (19.1).
Where the targets come from:
- NFR1. A morning run should decide before lunch; one provider says most batches finish in under 1 hour (2026).
- NFR3. A 5% chance of passing a worse candidate, and 80% for a harmless one on the whole policy, our choice (section 10.5).
- NFR6. One company required its judge's kappa to reach 0.7, where people agreed with each other at 0.78 to 0.83 (a 2026 paper).
- NFR10. 99.9% of 43,200 minutes leaves 43.2 minutes a month.
- The rest. Our choices; NFR5 follows from the US hourly sample size (section 12.4).
2.3 Workload: Teams, Suites, Runs and Traffic
No lab or vendor publishes an eval workload like ours, so every number here is our assumption.
A team is one tenant of Walnut Evals, with its own suites, runs and budget. A customer is a business using Walnut Assist. A conversation is one Walnut Assist chat; a session is one Walnut Coder task.
Four kinds of run:
- PR check. A quick run on 500 items for each code change, or pull request, an engineer opens.
- Nightly run. Each team's run on all its items, every night.
- Release run. The full test of a candidate against the baseline.
- Bake-off. One run that compares many new models, on release day.
Three load words:
- Daily average. A day's total spread evenly over 24 hours.
- Busy hours. 10:00 to 16:00 Central on weekdays, when live traffic is 2 times the daily average.
- Release day. A day a provider ships new models.
A trial is one attempt at one item (section 6.5). The sample rate is the share of live traffic we judge (section 12.2).
| What | Number | How we got it | Section |
|---|---|---|---|
| Teams and suites | 50; 400 | Our assumption for a mid-size company; 50 ร 8 | 14.3, 5 |
| Items | 306,000; 765 a suite | 300,000 text items + 6,000 agent tasks; รท 400 | 5 |
| Runs | 254 a day; a month, 6,000 PR, 1,500 nightly, 120 release | 200 PR + 50 nightly + 4 release; ร 30 | 6 |
| Production calls | 70 million a day | Assist 10 million conversations ร 4 = 40 million; Coder 1 million sessions ร 30 = 30 million | 2.4 |
| Assist conversations | 6,944 a minute; 13,889 in busy hours, 833,333 an hour; US 11,111 | 10 million รท 1,440; ร 2; ร 60; 80% in the US, 20% in the EU | 12.5 |
| Assist samples | 100,000 a day; 4,167 an hour; 8,333 in busy hours: US 6,667, EU 1,667 | 1%, our choice; รท 24; ร 2; 80% and 20% | 12.4 |
| Coder samples | 5,000 a day; 208 an hour | 0.5% of 1 million; รท 24 | 12.2 |
| Audit by a stronger judge | 5,250 a day | 5% of all samples: 5,000 + 250 | 12.3 |
A PR check or nightly run tests 1 setup, a release run 2 (candidate and baseline), a bake-off 7.
2.4 Calls, Tokens and Time
A token is a word piece. Input tokens go to a model; output tokens come back. The synchronous path sends one request and waits.
With prompt caching, the provider keeps a shared prompt start, and a cache read of it costs a twentieth of normal input on our mid model.
A rate limit caps tokens a minute (section 6.4).
Our assumptions: a text item takes 2,000 input tokens, 1,500 of them cached, and 400 output. A judge call takes 2,600, with a 1,200-token cached rubric, and 150 out.
An agent trial makes 20 calls, 120,000 input tokens (80% cache reads) and 10,000 output, in 5 minutes.
| Run | Calls | Input tokens | Output tokens |
|---|---|---|---|
| Release run, text | 42,000 items ร 2 setups ร 1 trial = 84,000 | ร 2,000 = 168 million | ร 400 = 33.6 million |
| Release run, judge | 24,000 ร 2 ร 1 = 48,000 | ร 2,600 = 124.8 million | ร 150 = 7.2 million |
| Release run, agents | 1,200 ร 2 ร 1 = 2,400 trials ร 20 = 48,000 | 2,400 ร 120,000 = 288 million | ร 10,000 = 24 million |
| One release run | 180,000 | 580.8 million | 64.8 million |
| Nightly, all teams | 300,000 text + 150,000 judge + 6,000 trials ร 20 | 600 + 390 + 720 = 1,710 million | 120 + 22.5 + 60 = 202.5 million |
| PR checks a day | 200 ร (500 + 250) | 200 ร (500 ร 2,000 + 250 ร 2,600) = 330 million | 200 ร (500 ร 400 + 250 ร 150) = 47.5 million |
| Online judge a day | samples and audits: 105,000 Assist + 5,250 Coder = 110,250 | ร 9,200 and ร 60,000 = 1,281 million | ร 300 = 33.1 million |
| Anchors a day | 1,200 | 3.1 million | 0.2 million |
| A day (4 release runs) | 4 ร 580.8 + 1,710 + 330 + 1,281 + 3.1 = 5.65 billion | 4 ร 64.8 + 202.5 + 47.5 + 33.1 + 0.2 = 0.54 billion |
Judge calls a day: 150,000 + 4 ร 48,000 + 200 ร 250 + 110,250 + 1,200 = 503,450. Other eval calls: 528,000 + 420,000 + 100,000 = 1,048,000.
In all 1.55 million, 2.2% of production's 70 million. The check judge, a second judge from another provider (section 8.5), adds 30,720 calls there.
Our eval account limits are our assumption, matching one provider's middle tier (2026). Each model class gets 10,000 requests, 5 million input tokens and 1 million output tokens a minute.
Cache reads do not count. Per account, 300,000 batch requests may wait, 100,000 a batch.
Evals never share an account with production. We split the per-minute limits by run kind: release runs 60%, PR checks 25%, the rest 15%. Unused share is lent out; a release run in fallback may take 90%.
A second eval account serves only the second release slot (section 6.4).
A release run, in minutes:
- 0 to 5. Freeze the manifest, the run's list of versions (section 3.1).
- 5 to 60 at most. Text answers come from the batch tier.
- 5 to 65. Agents: 2,400 ร 5 minutes รท 200 sandboxes = 60 minutes, using 24 million รท 60 = 0.4 million output tokens a minute (40% of the limit) and 288 million ร 20% รท 60 = 0.96 million uncached input.
- Judging. 48,000 ร 1,400 uncached = 67.2 million รท 3 million a minute (60% of 5 million) = 22.4 minutes.
- Statistics. 10 minutes. The decision comes near 90 minutes.
If the batch tier finishes nothing in 60 minutes, 84,000 ร 400 = 33.6 million + 7.2 million judge = 40.8 million output tokens remain.
At 0.9 million a minute (90%) that is 45.3 minutes: 60 + 45.3 + 10 = about 115 minutes, still inside NFR1's 2 hours.
A PR check sends 50 calls at a time, at about 10 s an answer and 4 s a judge call (estimates).
That is 500 รท 50 ร 10 s = 100 s plus 250 รท 50 ร 4 s = 20 s, about 3 minutes with queueing.
Nightly runs and bake-offs share 200,000 of the account's 300,000 waiting batch slots. The other 100,000 stay free for release runs (our choice).
So nightly text goes in two waves, 200,000 at 01:00 and 100,000 at about 01:50. Agents take 6,000 ร 5 minutes รท 200 sandboxes = 150 minutes (section 6.7).
2.5 Bytes, Storage and Money
A sandbox-hour is one sandbox running one hour. Compression stores data in fewer bytes; our columnar files shrink 4 times. Erasure coding splits an object into pieces plus spare pieces across 3 zones, 1.5 times its size.
A result row is one small row per answer or grade. A payload is the large text behind it (section 13.2).
Prices are our assumption, matching one provider's November 2026 list. A mid model costs $2 per million input tokens and $10 per million output, half on batch.
The mid model's cache reads cost $0.10, or $0.05 on batch. A small model costs $0.10 and $0.50, cache reads $0.01.
We also assume $0.10 a sandbox-hour, $300 a server a month and $0.02 a GB a month. Per call:
- Text, batch. (500 ร $1 + 1,500 ร $0.05 + 400 ร $5) รท 1 million = $0.002575.
- Text, synchronous. (500 ร $2 + 1,500 ร $0.10 + 400 ร $10) รท 1 million = $0.00515.
- Judge. (1,400 ร $1 + 1,200 ร $0.05 + 150 ร $5) รท 1 million = $0.00221 on batch; $0.00442 synchronous.
- Agent trial. (24,000 ร $2 + 96,000 ร $0.10 + 10,000 ร $10) รท 1 million = $0.1576.
- Online small judge. (8,000 ร $0.10 + 1,200 ร $0.01 + 300 ร $0.50) รท 1 million = $0.000962 (Assist); with 58,800 uncached tokens, $0.006042 (Coder).
- Online audit, mid model. (8,000 ร $2 + 1,200 ร $0.10 + 300 ร $10) รท 1 million = $0.01912 (Assist); with 58,800 uncached tokens, $0.12072 (Coder).
| Run | Per run | Runs a month | Per month |
|---|---|---|---|
| Release run | 84,000 ร $0.002575 = $216.30; + 48,000 ร $0.00442 = $212.16; + 2,400 ร $0.1576 = $378.24; + 2,400 ร 5 minutes = 200 sandbox-hours ร $0.10 = $20; + 7,680 check-judge calls ร $0.00442 = $33.95 (section 8.5); = $860.65 | 120 | $103,278 |
| Nightly, all teams | 300,000 ร $0.002575 = $772.50; + 150,000 ร $0.00221 = $331.50; + 6,000 ร $0.1576 = $945.60; + 6,000 ร 5 minutes = 500 sandbox-hours = $50; = $2,099.60 | 30 | $62,988 |
| PR check | 500 ร $0.00515 + 250 ร $0.00442 = $3.68 | 6,000 | $22,080 |
| Online, a day | 100,000 ร $0.000962 + 5,000 ร $0.006042 + 5,000 ร $0.01912 + 250 ร $0.12072 = $252.19 | 30 | $7,566 |
| Anchors, a day | 1,200 ร $0.00442 = $5.30 | 30 | $159 |
| Servers | 44 ร $300 (section 15.3) | $13,200 | |
| Total | about $209,000 |
Agent trials cost 120 ร $378.24 + 30 ร $945.60 = $73,757, about 35% of the month; release runs about 49%. A stalled batch tier costs a release run 84,000 ร ($0.00515 โ $0.002575) = $216.30 more.
People's hours a month:
- Judge audit. 500 verdicts ร 2 labelers ร 2 minutes + 75 third labels ร 2 minutes = 35.8 hours a week, ร 30 รท 7 = 153.6.
- Gate reviews. 120 ร 20% ร 1 hour = 24; judge checks on blocks, 120 ร 10% ร 1 hour = 12 (section 9.3).
- Anchor refresh. 200 ร 2 ร 2 + 30 ร 2 minutes = 14.3.
- In all. About 204 hours, 1.3 full-time labelers at 160 hours a month.
Bytes, at 10 KB a text answer, 3 KB a grade and 500 KB an agent trial:
- A release run. 84,000 ร 10 KB + 48,000 ร 3 KB + 2,400 ร 500 KB = 840 + 144 + 1,200 = 2,184 MB; ร 4 = 8.74 GB a day.
- Nightly and PR. 300,000 ร 10 KB + 150,000 ร 3 KB + 6,000 ร 500 KB = 6.45 GB; 200 ร (500 ร 10 KB + 250 ร 3 KB) = 1.15 GB.
- Evals a day. 8.74 + 6.45 + 1.15 + 0.0036 for anchors = 16.3 GB.
- Samples a day. 100,000 Assist conversations ร 32 KB + 5,000 Coder sessions ร 2 MB + 110,250 judge records ร 3 KB = 3.2 + 10 + 0.33 = 13.5 GB.
- Evals stored 730 days. 16.3 รท 4 = 4.08 GB a day ร 730 = 2.98 TB ร 1.5 = 4.47 TB.
- Samples stored 90 days. 13.5 รท 4 = 3.38 GB ร 90 = 304 GB ร 1.5 = 457 GB.
- Cost. About 4.9 TB in all: 4,930 GB ร $0.02 = about $99 a month.
- Rows. About 1.7 million result rows a day, 1.25 billion over 730 days.
So the limits are tokens a minute, sandboxes, money and people. Models and sandboxes take about $196,000 of the $209,000 month; storage and servers are small.
2.6 Out of Scope
- Training models. We test models; we do not make them.
- Serving production traffic. See our LLM inference serving post.
- Moving traffic during a rollout. That is the rollout post.
- The retrieval system. See our RAG platform and vector database posts.
- Red-team programs and public leaderboards. We run public benchmarks only for reference.
3. The API and the Core Data
๐ Premium section
4. High-Level Design
๐ Premium section
5. Deep Dive: The Test Set Registry
๐ Premium section
6. Deep Dive: The Run Orchestrator
๐ Premium section
7. Deep Dive: Sandboxes for Agent and Code Tasks
๐ Premium section
8. Deep Dive: Graders
๐ Premium section
9. Deep Dive: Judge Calibration
๐ Premium section
10. Deep Dive: Paired Statistics and Sample Size
๐ Premium section
11. Deep Dive: Release Gates and Policy
๐ Premium section
12. Deep Dive: Online Evals and Sampling
๐ Premium section
13. Deep Dive: The Trace Store
๐ Premium section
14. Deep Dive: Privacy, Many Teams, Dashboards and Cost
๐ Premium section
15. Peaks: Release Day
๐ Premium section
16. What Happens When Each Part Goes Down
๐ Premium section
17. Bottlenecks
๐ Premium section
18. Trade-Offs We Chose and Rejected
๐ Premium section
19. Multi-Region
๐ Premium section
20. Walnut Evals and the Tools of 2026
๐ Premium section
21. In the Interview (45 and 60 Minutes)
๐ Premium section
22. Further Reading
๐ Premium section
Explore the Technologies
๐ Premium section