System Design: LLM Evaluation Platform
Goal: Decide whether a candidate LLM is good enough and safe enough to replace the model in production.
The platform runs versioned test suites, compares the candidate with the current model, explains where behavior changed and applies the release policy. After deployment, it watches real traffic for failures the offline tests did not cover.
This article designs the evaluation platform around one question: should we ship the candidate? It does not design model training or the inference service. It assumes both model versions can be called through a stable API.
How the article builds the answer:
- Define what must be measured and what evidence a release decision needs.
- Build the execution path that generates, grades and stores results.
- Design private evaluation datasets and choose a valid grader for each task.
- Compare the candidate with the production baseline using uncertainty, not only averages.
- Apply release gates, then continue measuring the deployed system on production traffic.
1. The Release Decision
A new model version arrives. Its training team says it is better at code and instruction following. A higher benchmark score is encouraging, but it is not enough to replace a production model.
The release owner needs evidence for four questions:
- Did the capabilities we intended to improve actually improve?
- Did an unrelated capability, language or customer group regress?
- Did safety, privacy, latency or cost move outside policy?
- Is the measured difference large and reliable enough to matter?
An evaluation platform produces the evidence. People and release policy still decide what evidence is sufficient. A single overall score cannot safely replace that decision because an average can hide a severe failure in one important slice.
1.1 Running Example: Atlas Support Assistant
Assume Atlas v2 serves customer-support conversations today. The team wants to replace it with Atlas v3. The claim is that v3 writes more useful answers and handles tool calls better without weakening safety.
The platform evaluates both versions on the same immutable snapshot of support cases:
- Tool correctness: Did the model call the correct tool with valid arguments? A deterministic scorer checks the call and the final state.
- Answer quality: Is the response correct, relevant and clear? A calibrated judge compares blinded v2 and v3 answers against a rubric. Humans review a sample and disagreements.
- Safety: Did the model expose secrets, follow a malicious instruction or give disallowed advice? Dedicated safety suites measure each policy category.
- Critical slices: Results are reported separately for languages, products, new customers and long conversations. An average improvement cannot cancel a serious regression in one of these groups.
The run produces paired results because v2 and v3 answer the same cases. The gate doesn't ask only whether v3's score is higher.
It checks the size and uncertainty of the change. It also checks important slices and required-result completeness. A passing v3 moves to shadow traffic. A failure report names the responsible examples and slices.
We will return to this example throughout the article.
1.2 Terms Used Throughout the Design
| Term | Meaning in this article |
|---|---|
| Eval item | One test case: input, optional reference data, metadata and scoring instructions. |
| Dataset snapshot | An immutable, versioned collection of eval items. A golden set is a carefully curated snapshot used as trusted release evidence. |
| Suite | A dataset snapshot plus the model settings, grader configuration, metrics and slice definitions needed to evaluate one behavior. |
| Trial | One model attempt on one eval item. A stochastic or agent task may need several trials. |
| Grader or scorer | Code, a model or a human process that turns an output or final state into structured measurements. These terms are used interchangeably here. |
| Rubric | The written criteria a model or human grader applies. |
| Run | One execution of a pinned set of suites against one or more model versions. |
| Candidate | The model or application version being considered for release. |
| Baseline | The version used for comparison, usually the current production version. |
| Slice | A meaningful subset such as Hindi conversations, billing questions or prompts longer than 8,000 tokens. |
| Gate | A versioned release policy that converts metrics and uncertainty into pass, fail or review. |
1.3 The Evaluation Types Fit Together
The platform supports several kinds of evaluation. They answer different questions and should not be collapsed into one score.
| Evaluation | When it runs | What it answers |
|---|---|---|
| Capability | During development and before release | Can the system perform the intended task? |
| Regression | Candidate versus a pinned baseline | What became better or worse? |
| Safety and abuse | Before release and continuously | Does behavior remain inside policy under normal and adversarial inputs? |
| Agent or tool-use | In an isolated test environment | Did the model take valid actions and leave the environment in the correct state? |
| Human evaluation | On calibrated samples and disputed cases | Does the automated measurement match informed human judgment? |
| Online evaluation | Shadow, canary and production traffic | Does the deployed system behave well on the traffic it actually receives? |
Public benchmarks can provide context and comparability. Private, task-specific suites are usually stronger release evidence because they reflect the product, policy and traffic being changed. Private does not automatically mean uncontaminated; provenance and leakage checks are still required.
1.4 Why This Is Hard
- Quality has several dimensions. Correctness, usefulness, safety, tone, latency and cost can move in different directions.
- Open-ended answers have no single correct string. The platform needs task-valid deterministic checks, calibrated model judges or human review.
- Evaluation data can leak or become stale. Public and private cases can overlap training data, and teams can overfit to a release suite through repeated use.
- Averages hide important regressions. Every release needs pre-declared critical slices and enough evidence to evaluate them.
- Measurements contain uncertainty. Model sampling, judge behavior, human disagreement and finite datasets all contribute different kinds of variation.
- Production changes the input distribution. Prompts, retrieval results, tools and user behavior move after the offline snapshot was created.
1.5 Reference Scale
| Metric | Target |
|---|---|
| Models in the registry | 50+ (every version of every model) |
| Eval suites | 200+ |
| Atlas release items | 1 million paired items |
| Automated release evidence | Complete within 4 hours |
| Human calibration and review | Complete within 24 hours when required |
| Online Atlas traffic | 1,000 eligible responses/second at peak |
| Online evaluation | Risk-based sample, continuous, 10 responses/second baseline |
| Regression detection lag | under 1 hour |
These numbers define the design exercise, not universal defaults. For example, the production sample rate should follow traffic volume, risk, privacy policy and evaluation cost.
1.6 Canonical Atlas Release Run
One million logical items do not all cost the same. The reference run pins this mix:
| Suite class | Items | Execution | Primary grader |
|---|---|---|---|
| Contract, retrieval and tool checks | 650,000 | One response per model | Schema, facts and final-state rules |
| Open-ended answer quality | 250,000 | One response per model | Blinded pairwise judge |
| Safety and policy challenges | 80,000 | One response per model | Policy rules plus calibrated safety grader |
| Multi-step agent fixtures | 20,000 | Five model turns per model on average | Sandboxed final state and policy trace |
| Total | 1,000,000 | 2.16 million model calls | Mixed |
The 250,000 open-ended pairs produce one model-judge call each. The automated path also selects 2,000 cases for blinded human review using disagreement, severity and slice coverage. Human review does not block the four-hour automated evidence target unless the gate returns REVIEW. Its service objective is 24 hours.
All values are teaching workload measurements. A suite-mix, average-turn, token-length, grader or provider change invalidates the capacity evidence.
2. Requirements
Functional
| ID | Requirement | Priority |
|---|---|---|
| F1 | Run a versioned suite against any registered candidate and baseline | P0 |
| F2 | Compare paired results overall, by critical slice and by example | P0 |
| F3 | Support deterministic, model, human and environment-based graders | P0 |
| F4 | Add datasets, rubrics, graders and gates without changing platform code | P0 |
| F5 | Preserve the configuration and lineage needed to explain every result | P0 |
| F6 | Apply a versioned release gate and record approvals or waivers | P0 |
| F7 | Evaluate privacy-approved samples from shadow and production traffic | P0 |
| F8 | Alert on meaningful production regressions and grader failures | P0 |
| F9 | Route calibration samples and disputed cases to blinded human review | P1 |
Non-functional
| ID | Requirement | Target |
|---|---|---|
| NFR-01 | Automated decision latency | Complete required automated evidence within 4 hours |
| NFR-02 | Human-review latency | Complete required calibration or adjudication within 24 hours |
| NFR-03 | Logical-result integrity | Exactly one committed result per work-unit idempotency key |
| NFR-04 | Traceability | Every metric resolves to immutable input, output, grader and policy versions |
| NFR-05 | Statistical validity | Report effect size, uncertainty, missingness and pre-declared slices |
| NFR-06 | Privacy and isolation | Enforce tenant, region, retention and sandbox policy before dispatch |
| NFR-07 | Run recoverability | Resume after worker, provider or control-plane failure without regenerating durable outputs |
| NFR-08 | Budget safety | Admit only after reserving generation, grading, sandbox and review budgets |
| NFR-09 | Online detection | Detect a declared production regression within 1 hour when sample-size requirements are met |
| NFR-10 | Judge health | Meet the suite's calibrated false-pass, consistency and invalid-output limits |
3. High-Level Architecture
🔒 Premium section
4. Capacity and Execution
🔒 Premium section
5. Evaluation Datasets
🔒 Premium section
6. Graders: Measure the Intended Behavior
🔒 Premium section
7. Statistics Before Gates
🔒 Premium section
8. Regression Gates
🔒 Premium section
9. Online Evaluation and Drift
🔒 Premium section
10. Reproducibility and Lineage
🔒 Premium section
11. Reliability and Failure Recovery
🔒 Premium section
12. Observability
🔒 Premium section
13. Security and Privacy
🔒 Premium section
14. Cost and Quality Controls
🔒 Premium section
15. Trade-offs
🔒 Premium section
16. Common Failure Patterns
🔒 Premium section
17. Reference Design and Feasibility
🔒 Premium section
18. What an Interviewer Is Grading
🔒 Premium section
19. Follow-up Questions
🔒 Premium section
Related AI Engineering Chapters
🔒 Premium section
Further Reading
🔒 Premium section
Related System Designs
🔒 Premium section
