System Design: LLM Safety Pipeline (Walnut Guard: Guard Cascade, Streaming Checks, Agent Tool Gates, Thresholds, Human Review)
Goal: Design the system that checks what goes into and comes out of a company's LLM products. Ours is Walnut Guard.
It checks every prompt, streamed answer and agent tool call for an assistant app with teen accounts, a developer API and business agents.
We plan for 60 million daily users, 42,000 checked calls a second at design capacity, at most 30 ms added to most turns, and about 0.5 s of hold-back once per answer.
Walnut Guard is our design. Facts about real products and laws are dated; every number of ours is labeled as ours.
1. What We Are Building
๐ Premium section
2. Requirements and a Quick Estimate
2.1 Functional Requirements
- Check inputs. Check every prompt, image and file before the chat model's answer is shown, including text pulled in by retrieval and tools.
- Check answers. Check every answer while it streams, and stop it when it breaks the policy.
- Check tool calls. Check every Fernway Agents tool call outside the model, with the user's own rights. On the API, check tool-call text and return signals (section 9.2).
- Pick the policy. Choose it by surface (chat, API or agents), age band, region and tenant, a business customer such as Ridgeway Supply.
- Change without code. Change the policy and guard model versions without a chat product deploy.
- Act in ten ways. Allow, allow with care, answer with care, redact, refuse, block, stop, hold for review, deny and ask the user (section 6.2).
- Hear users. Let users report and appeal, and send hard cases to trained reviewers.
- Keep records. Record every decision with its versions and reasons, with as little personal data as possible.
- Meet the law. Send child safety reports, take down intimate images, show AI notices and marks, and file incident reports.
The chat product and its abuse controls, inference internals and model training are out of scope. So are fraud and spam that are not about model content, and the fine-tuning API.
2.2 Non-Functional Requirements
An NFR (non-functional requirement) says how well the system must work. p95 is the value that 95% of measurements stay at or below. The first visible token is the moment the user sees the first word of an answer.
A zone is a group of data centers with its own power and network. A region is 3 zones in one area (section 19.1).
Later sections own some table words. Escalated calls go on to the guard model (section 7.2). Hold-back keeps tokens until checked (section 8.3). Reasoning text is the model's working (section 8.4). The Action gate checks tool calls (section 9.2).
A policy bundle is the signed rules file (section 6.3). Each first runs in shadow, scoring live traffic without acting (section 11.4). Recall is the share of harm caught (section 11.2). A legal hold keeps records for a court (section 14.2).
| Target | |
|---|---|
| NFR1 | Input checks: on the normal path they add at most 30 ms p95 to the first visible token, because they run while the chat model reads the prompt. On the escalated path (about 3% of calls) they add at most 150 ms p95. |
| NFR2 | Output hold-back adds about 0.5 s once, before the first visible token, and nothing after. Reasoning text is shown only after its check. |
| NFR3 | Every turn and every agent action gets a verdict under the current policy. When a layer is down, each category's fail rule applies, or the surface's strictest rule when no layer could name the category. No tool action runs without an Action gate verdict. |
| NFR4 | Misses: on the held-out severe set, recall is at least 95% at the enforced thresholds, reported with a 90% range and checked before each policy or guard release. |
| NFR5 | Over-blocking: harmless refusals stay at or under 0.1% of turns, overall, in each of the top 10 languages and on each surface, measured weekly on reviewed live samples. |
| NFR6 | Policy changes: an approved bundle is live everywhere within 30 minutes of its last rollout check; an emergency rule within 10 minutes; a rollback to the last good bundle within 1 minute. |
| NFR7 | The Action gate adds at most 20 ms p95 to each tool call. Tool calls with free text or code add at most 150 ms p95 for the guard model's read (section 9.2). |
| NFR8 | People: 95% of appeals get a human-checked answer within 24 hours; confirmed child-safety cases are reported within 24 hours of confirmation. |
| NFR9 | Records: a decision record exists for at least 99.99% of checked calls and holds no raw text; raw text is kept only for flagged or reported checks and a small random sample, for 30 days unless a legal hold applies; EU users' records stay in EU regions. |
| NFR10 | Losing one zone changes no verdict. New turns see no change; answers streaming through that zone pause with "answer interrupted" and resume after a recheck. Losing a region moves its users to the other region of the same pair; Walnut Guard adds no "busy, try again" reply of its own, and no turn skips its checks. |
NFR5 is possible: Anthropic reported harmless refusals of 0.05% over one month of live traffic (January 2026).
2.3 Load
These numbers are our assumptions. A turn is one user message and the answer to it. An agent step is one model call inside an agent task that may propose a tool call.
A checked call is a turn or an agent step, the unit of load. Fernway makes 1.2 billion checked calls a day: 1.0 billion turns and 0.2 billion agent steps.
- Daily average. A day's total spread over 86,400 seconds.
- Busy hour. 2 times the daily average, 17:00 to 21:00 Pacific on weekdays, while the EU sleeps.
- Design capacity. 1.5 times the busy hour, ready all day.
| Checked calls a second | Sum | |
|---|---|---|
| Daily average | about 14,000 | 1.2 billion รท 86,400 s |
| Busy hour | 28,000 | 14,000 ร 2 |
| Design capacity | 42,000 | 28,000 ร 1.5 |
These are world totals. Design capacity splits into 35,000 turns and 7,000 agent steps a second (42,000 ร 1.0 รท 1.2 = 35,000).
With 1.5 times the busy hour, two of a region's three zones still carry the busy hour if one is lost. An average answer of 400 tokens streams for about 12 s (section 8.6).
2.4 Layer Sizes and Fleets
The inference GPU is the one GPU type all our guards use, 48 GB class. A pool is a group of them running one guard. A CPU server is a 64-core machine.
To escalate is to send a call on to the next, costlier layer (section 7.2). We escalate 3% of checked calls to the guard model, our choice. Per-GPU speeds are estimates.
Section 4 builds these layers. For now: cheap checks (rules, a probe and a small classifier) read every call and pass doubtful ones to a costly guard model; other guards read answers, images and tool results.
| Layer | Runs on | Calls checked | Inference GPUs | Sum |
|---|---|---|---|---|
| Rules layer | Guard API CPU servers | Every call | 0 | About 1 ms each |
| Probe | Fernway's chat model servers | Turns to models Fernway serves | 0 of ours | About 1% of their compute (our assumption) |
| Small classifier | Its pool | Every turn | about 18 | 35,000 รท 2,000 a GPU |
| Injection classifier | Its pool | One tool result per agent step, twice a prompt's length | 7 | 7,000 ร 2 รท 2,000 a GPU |
| Guard model | Its pool | 3% of calls | about 25 | 42,000 ร 3% รท 50 a GPU |
| Stream guard | Its pool | Every answer token | 70 | 35,000 ร 400 tokens รท 200,000 a GPU |
| Image guard | Its pool | 2% of turns (our assumption) | about 18 | 35,000 ร 2% รท 40 a GPU |
| Total | about 140 | 18 + 7 + 25 + 70 + 18 = 138 |
These are world totals of compute, spread over every zone by region share. Without the cascade, the guard model would need 42,000 รท 50 = 840 GPUs.
The table leaves out three loads. Tool-call text on 1 in 10 agent steps adds about 2 guard model GPUs, since it is short.
File checks, the random sample and long-chat reads use spare capacity at low priority (sections 7.2, 7.7, 10.3). The last two are about 1.5% of calls (our estimate).
At $2 an hour per GPU (our assumption), 140 ร $2 ร 730 hours = about $200,000 a month of compute. The 840 would cost about $1.2 million a month.
Buying checks instead, a 1,500-token prompt is about 6,000 characters, or about 6 text units. At one cloud's list price (Dec 2024), 6 ร $0.00015 = about $0.0009 a check. For 1.2 billion checks, that is about $1.1 million a day.
At that price, the hot path runs our own guards.
3. The API, the Data Model and the Stores
๐ Premium section
4. High-Level Design
๐ Premium section
5. Deep Dive: The Threat Model and the Trust Boundary
๐ Premium section
6. Deep Dive: Policy as Data
๐ Premium section
7. Deep Dive: The Input Cascade
๐ Premium section
8. Deep Dive: Output Checks While the Answer Streams
๐ Premium section
9. Deep Dive: Agents and Tools
๐ Premium section
10. Deep Dive: Jailbreaks and Prompt Injection in Depth
๐ Premium section
11. Deep Dive: Thresholds, Evaluation and Shadow Mode
๐ Premium section
12. Deep Dive: Human Review, Appeals and Reviewer Wellbeing
๐ Premium section
13. Deep Dive: Legal Duties That Change the Design
๐ Premium section
14. Deep Dive: Decision Records and Privacy
๐ Premium section
15. Monitoring and Evaluation
๐ Premium section
16. Security of the Safety System
๐ Premium section
17. Bottlenecks
๐ Premium section
18. What Happens When Each Part Goes Down
๐ Premium section
19. Multi-Region
๐ Premium section
20. Cost
๐ Premium section
21. Trade-Offs We Chose and Rejected
๐ Premium section
22. Walnut Guard vs Guard Models and Managed Services in 2026
๐ Premium section
23. In the Interview (45 and 60 Minutes)
๐ Premium section
24. What to Remember
๐ Premium section
25. Further Reading
๐ Premium section
Explore the Technologies
๐ Premium section