System Design: ChatGPT End-to-End (Inference, Memory, RAG, Streaming, RLHF)
Goal: Build ChatGPT, properly. The thing serves around 100 million people a day and roughly a billion messages, with every answer streaming out word by word.
It has to remember a conversation, look things up when its training data has gone stale, call tools when it needs them, turn down the requests it should turn down, and still land at a few cents per chat.
This post traces the whole path, from the API call at the top down to the GPU generating each token.
How to read this: The first six sections build the system up, from capacity and APIs to what actually happens to a single request. Section 7, the inference engine, is the part that matters most.
Get prefill vs decode, KV-cache, and continuous batching, and the rest of the design stops looking arbitrary.
1. The Problem
Someone types "explain quantum computing like I'm five" and hits Enter. Two seconds later, words start appearing one at a time. Behind those two seconds is one of the most complex distributed systems running in production today.
Five things make this hard:
Inference is expensive. A 70B-parameter model (parameters are the learned numbers that make up the model; at 2 bytes each, 70 billion of them weigh ~140GB) needs 140GB of GPU memory just for the weights. Running it means billions of matrix multiplications per token, for thousands of concurrent users. One A100 GPU (a high-end NVIDIA datacenter chip built for AI, with 80GB of on-board memory) costs about $2/hour, and serving this thing takes around a thousand of them.
Conversations have memory. The user said "my dog is named Max" in message 3. In message 47, they ask "what should I feed him?" The model has to connect "him" to "Max" to "dog" across 44 messages. The context window (the fixed amount of text the model can look at in one go) is finite, so older messages get compressed without losing the facts that matter.
Streaming is not optional. A 500-token response (a token is a chunk of text, roughly a word or word-piece; models read and write in tokens, not letters) takes 5 to 10 seconds to generate fully. If users wait for the whole thing before seeing anything, the product feels broken. Tokens have to flow to the browser the instant they are generated, with time to first token (TTFT, the gap between hitting Enter and the first word showing up) under 2 seconds.
Knowledge goes stale. Training data has a cutoff date. Ask "who won yesterday's game?" and the weights cannot help. The system needs to retrieve fresh information and feed it into the prompt, which is a whole subsystem of its own.
Alignment is fragile. Straight out of pre-training, before any safety work, the model is just a text predictor that has read most of the internet. Ask it to help with something harmful, like writing a convincing scam email, and it will usually just do it, because nothing has taught it to refuse. RLHF and DPO (the training steps that teach a raw model to be helpful and safe, covered in Section 16), plus safety classifiers, are what add that judgment, and they have to hold up on every request, even when someone is deliberately trying to trick the model, without turning it into a useless refusal machine.
2. Requirements
Functional
| ID | Requirement | Priority |
|---|---|---|
| FR-01 | Multi-turn conversation with persistent per-conversation context | P0 |
| FR-02 | Streaming token-by-token responses via SSE (a one-way server-to-client stream, detailed in Section 13) | P0 |
| FR-03 | Multi-model routing by query complexity | P0 |
| FR-04 | Input and output safety filtering | P0 |
| FR-05 | Tool/function calling with a multi-step agent loop (web search, code, APIs) | P0 |
| FR-06 | Retrieval (RAG) over web, user files, and a vector store | P0 |
| FR-07 | Conversation storage, listing, retrieval, and deletion | P0 |
| FR-08 | User-level persistent memory, separate from conversation memory | P1 |
| FR-09 | Image and file understanding (multimodal input) | P1 |
| FR-10 | Custom system prompts per user | P1 |
| FR-11 | Response and prompt-prefix caching for repeated queries | P1 |
| FR-12 | Rate limiting and priority by subscription tier | P1 |
FR-06 (RAG), FR-08 (user memory), FR-11 (caching), and the multi-step agent loop in FR-05 are what turn this from a plain inference service into ChatGPT. Each gets its own section below.
Non-Functional
| Requirement | Target |
|---|---|
| Time to first token | P50 < 1.5s, P99 < 4s |
| Token throughput | 50 to 80 tokens/sec per stream |
| Retrieval latency (RAG) | < 300ms added to TTFT |
| Safety classifier latency | < 50ms |
| Cache hit rate (response + prefix) | > 25% of eligible queries |
| Agent loop bound | <= 6 tool iterations before forced finalize |
| Availability | 99.9% |
P50 and P99 are percentiles: P50 is the median request, and P99 < 4s means even the slowest 1% of requests still finish under 4 seconds.
The agent-loop bound matters: without a hard cap, a confused model can loop tool calls forever and burn GPU time.
The retrieval budget matters because RAG sits on the critical path before generation starts.
3. Capacity Estimation
🔒 Premium section
4. APIs
🔒 Premium section
5. High-Level Architecture
🔒 Premium section
6. Request Lifecycle
🔒 Premium section
7. The Inference Engine
🔒 Premium section
8. Context and Memory Management
🔒 Premium section
9. Retrieval Layer (RAG)
🔒 Premium section
10. Multimodal Pipeline
🔒 Premium section
11. Tool Calling and the Agent Loop
🔒 Premium section
12. Caching
🔒 Premium section
13. Streaming
🔒 Premium section
14. Model Routing and the GPU Scheduler
🔒 Premium section
15. Safety Architecture
🔒 Premium section
16. Alignment: RLHF and DPO
🔒 Premium section
17. Storage Design
🔒 Premium section
18. Scalability and Reliability
🔒 Premium section
19. Monitoring
🔒 Premium section
20. Tradeoffs
🔒 Premium section
21. Where Designs Go Wrong
🔒 Premium section
Conclusion
🔒 Premium section
Go Deeper
🔒 Premium section
