System Design: LLM Inference Serving
From one prompt to a fleet of multi-GPU model servers
🔒 Premium section
1. Problem Statement
We are designing real-time inference for a large language model. A parameter is one learned number inside that model. A 70-billion-parameter model stores about 70 billion of them.
Training changes those parameters. Inference uses the trained parameters to answer a request.
The running request is a report summary:
Model: assistant-70b v42
Input after chat-template rendering: 1,500 tokens
Maximum output: 300 tokens
Actual output in this example: 247 tokens
Service tier: interactive
Server-capped timeout: 30 seconds
Reference worker: one four-GPU tensor-parallel group (TP=4)Its lifecycle is:
- The gateway authenticates the caller and applies product/model policy.
- The pinned chat template and tokenizer produce 1,500 tokens.
- Admission atomically reserves tenant tokens, an active-sequence slot, expected KV blocks, and spending budget.
- The router chooses a ready TP=4 group using prefix overlap and live load.
- Prefill processes the prompt, possibly reusing an exact cached prefix, and creates the initial per-request KV state.
- Decode repeatedly selects and streams one next token. Continuous batching allows other sequences to enter or leave between decode iterations.
- Generation stops after 247 tokens. The engine frees KV blocks, reconciles reserved versus actual usage, and emits one terminal usage event.
At the reference SLO, the first token should arrive within two seconds at p95. Later tokens should remain inside the declared speed and streaming-gap targets.
The article will calculate whether enough complete four-GPU groups can satisfy that promise.
Terms used in the request path
| Term | Simple meaning |
|---|---|
| Token | A word, part of a word, punctuation mark, or whitespace unit processed by the model |
| TTFT | Time from accepting the request until the client receives the first token |
| TPOT | Average time per generated token after the first one |
| ITL | The measured gap between individual streamed tokens; tail ITL exposes pauses hidden by an average |
| SLO | A measurable latency or reliability promise |
| Prefill | Read the prompt and create its initial attention state |
| Decode | Repeatedly generate the next token |
| KV cache | Attention state retained for tokens already processed |
| Context window | The validated limit for input and generated tokens together |
| Sequence | One active generation with its prompt, output, sampling state, and KV cache |
| Logits and sampling | Next-token scores and the policy that chooses a token from them |
| Chat template | Versioned formatting that converts messages and roles into the model's exact token sequence |
| Pinned release | One immutable set of weights, tokenizer, template, runtime settings, and related artifacts |
| Rank | One process or device participating in distributed model execution |
| Logical replica group | Every rank required to serve one request together; the router treats the group as one worker |
| Adapter | A small set of extra weights, such as a LoRA customization, applied to a compatible base model |
A GPU is a processor built for highly parallel numerical work. H100 and H200 are examples of data-centre GPUs used for LLM inference. Operational terms such as fencing and admission leases are defined when their mechanisms appear.
Scope boundaries
This is the inference-serving layer of a ChatGPT-like product. It covers the path from an accepted prompt to streamed model output. It does not claim to reproduce ChatGPT.
A complete conversational AI product contains more than inference:
Conversation storage, retrieval, tool orchestration, training, human feedback, product UI, safety, billing, and evaluations touch the inference API. Their boundaries appear here, but their internal designs belong in separate articles.
The worked capacity model covers interactive, decoder-only text generation. Embedding models, rerankers, diffusion, and large offline jobs have different execution and SLO models.
A multimodal extension needs its own contract. It must validate and preprocess media, run the pinned encoder and projector, and convert media into model-specific embeddings or token-equivalent cost. Admission must include that memory and compute. It can't silently reuse the 1,500-text-token benchmark.
2. Functional Requirements
| ID | Requirement | Priority |
|---|---|---|
| FR-01 | Accept text and stream generated tokens; expose multimodal input only through a separately validated encoder and capacity contract | P0 |
| FR-02 | Select a pinned model and version; optionally choose from policy-approved model tiers | P0 |
| FR-03 | Enforce input, output, context, deadline, and spending limits | P0 |
| FR-04 | Cancel queued and running work when the client disconnects | P0 |
| FR-05 | Support sampling controls, stop sequences, log probabilities, and deterministic seeds where the runtime allows | P1 |
| FR-06 | Support constrained structured output such as a JSON schema | P1 |
| FR-07 | Support tool-call output while leaving tool authorization and execution outside the model process | P1 |
| FR-08 | Enforce per-tenant request, token, concurrency, KV-memory, and adapter quotas | P0 |
| FR-09 | Roll out immutable model versions and adapters without dropping accepted streams | P0 |
| FR-10 | Preserve the last known-good serving configuration during control-plane failure | P0 |
| FR-11 | Record usage for billing without storing raw prompts by default | P0 |
| FR-12 | Join sampled generations to delayed safety and quality outcomes | P1 |
3. Non-Functional Requirements
| ID | Signal | Reference target | Why it matters |
|---|---|---|---|
| NFR-01 | Availability | 99.9% per model tier and region | One rarely used model must not hide failure of the default model |
| NFR-02 | TTFT | Under 2 seconds at p95 | Measures queueing plus prompt processing |
| NFR-03 | TPOT | Under 80ms at p95 for the reference output class | Controls perceived generation speed |
| NFR-04 | Streaming gap | No unexplained gap over 1 second at p99 | Averages can hide visible stalls |
| NFR-05 | Cancellation | Stop GPU work within 250ms at p95 after disconnect | Prevents paying for tokens nobody receives |
| NFR-06 | Overload | Bounded queues and explicit rejection | Slow failure consumes more GPU than early failure |
| NFR-07 | Zone resilience | Continue within SLO after one-zone loss | Requires reserved capacity, not only replicated control services |
| NFR-08 | Regional recovery | New default-model admissions recover within 5 minutes; RPO is zero for acknowledged deployment, quota, and billing state | Live streams may fail because KV state isn't regionally replicated |
| NFR-09 | Privacy | Tenant-scoped cache and logging policy | Prompts and generated text may contain sensitive data |
| NFR-10 | Completion latency | Under 26 seconds at p95 for up to 1,500 input and 300 generated tokens | TTFT and TPOT alone don't describe the whole request |
Targets must be segmented by input length, requested output length, model, hardware, and service tier. A single global p95 hides long-context starvation and small-tenant failures.
Here, p95 means 95% of measured requests are at or below the target; p99 means 99% are. Tail percentiles expose the slow requests that averages hide.
Don't add independent p95 values and call the result a p95 completion time. The slowest TTFT and TPOT samples may belong to different requests.
The limits still need to be mutually plausible. At the declared ceilings, a 300-token response could take roughly:
2 seconds + 299 × 80 ms ≈ 25.9 secondsThe running request therefore uses a 30-second server-capped timeout. NFR-10 measures completion directly. It doesn't add unrelated p95 samples and call the result a percentile.
Overload Contract
Overload is normal. A request can be rejected before execution when:
- The deadline cannot cover expected queue, prefill, and first-token time.
- The tenant has exhausted its concurrent-token or KV-memory allocation.
- The model replica group is unavailable or warming.
- The requested context exceeds the validated model/runtime limit.
- The bounded queue is full.
- The requested model, adapter, or structured-output mode is incompatible with the selected runtime.
Return a stable error type and a retry hint. Do not accept work into an unbounded queue and hope that autoscaling catches up.
Keep the System Map in View
Before doing the capacity mathematics, keep this request path in mind:
client → gateway and tokenizer → admission → router
→ one complete multi-GPU replica group → token streamer → clientThe deployment control plane loads and warms complete replica groups, then publishes only ready groups to the router. The next section calculates how much memory one group needs and how many groups the service must reserve. Section 5 expands this map into the full architecture.
4. Back-of-the-Envelope Estimation
🔒 Premium section
5. High-Level Architecture
🔒 Premium section
6. Execution Deep Dive: Prefill, Decode, and Continuous Batching
🔒 Premium section
7. KV Cache, Paged Memory, and Prompt Caching
🔒 Premium section
8. When the Model Does Not Fit on One GPU
🔒 Premium section
9. Request Routing, Admission, and API Semantics
🔒 Premium section
10. Runtime and Optimization Strategy
🔒 Premium section
11. Deployment, Placement, and Autoscaling
🔒 Premium section
12. Failure Modes and Disaster Recovery
🔒 Premium section
13. Observability, SLOs, and Quality
🔒 Premium section
14. Security, Privacy, and Multi-Tenancy
🔒 Premium section
15. Trade-Offs and Cost Decisions
🔒 Premium section
16. Reference Design and Feasibility
🔒 Premium section
Related AI Engineering Chapters
🔒 Premium section
Further Reading
🔒 Premium section
Related System Designs
🔒 Premium section
