System Design: ChatGPT End-to-End (Inference, Memory, RAG, Streaming, RLHF)
Goal: Design the whole ChatGPT product for about a billion people a week. That's about 3.6 billion messages a day, and 103,000 a second in the busiest hour.
Every answer streams word by word. The product remembers the user, reads their files, searches the web and runs code. It can think for a minute before it answers, and it must survive a lost data centre.
Reading guide: Sections 1 to 3 give the problem, the requirements and the architecture. Sections 4 to 12 explain each part.
Section 13 follows one message in code, and sections 14 to 16 cover capacity, storage and safe change. Sections 17 to 23 cover failures and scale, then operations, cost and trade-offs.
TL;DR:
- Three models in four pools on NVIDIA B200 GPUs. A fast model (gpt-oss-120b class) takes 65% of messages and a dense 70B model 15%.
- The frontier model. A large mixture-of-experts model (DeepSeek-V3 class) takes 20%. It thinks first on 12.5% of messages, in its own pool.
- Thinking decides the bill. A thinking message costs 26 times the GPU time of a fast one, so 12.5% of messages use 58% of GPU time. Hidden thinking runs at a slower step, which saves about 13,700 GPUs.
- The fleet. About 37,700 B200s at the busiest hour (21,800 to 54,700), and 16,600 on average over the day.
- Keep the start of every prompt the same. The model servers keep what they computed for earlier turns (the prefix cache). Our prompt order lets them reuse 79% of all prompt tokens. Changing text early drops that to 47%, for 31% more GPUs.
- The answer survives a dropped connection. Every answer goes into a short-lived log with numbered entries. A phone that reconnects gets the rest from where it stopped.
- Cost. About $151M a month, or $0.0014 per message, if every GPU-hour can be rented by the hour. A reserved base plus rented peaks gives $190M, and reserving everything gives $285M.
1. The Problem in One Page
A large language model (LLM) reads text and writes the next piece of it, in tokens about three quarters of a word long. ChatGPT is a product built around such models, where the answer appears a few words at a time.
Take one message. Priya has saved a few facts about herself, and an hour ago she uploaded her lease in this chat. Now she asks:
"Can my landlord raise the rent mid-lease in Bengaluru? Check the current rules."
The system has seven jobs.
- Keep her message safe on disk before anything else happens.
- Build the prompt. It holds the system instructions, her saved facts and the chat so far, plus the right part of the lease.
- Pick a model. This needs care, so a stronger model is a good choice.
- Search the web, because the rules may have changed after the model was trained.
- Check the input and the output for harm, without making her wait.
- Stream the answer at reading speed, even if her phone drops off Wi-Fi halfway.
- Save everything, so the next turn, her chat list and her memory all work.
Here's what makes this hard.
Answers are expensive. A large model does billions of operations for every token it writes. One answer from the biggest model keeps eight GPUs busy, and we need tens of thousands of GPUs.
Every turn carries the whole chat. The model has no memory of its own. Each turn sends the system prompt, her memories, every earlier message and the new one. The model must read them all again unless something cached them.
Some answers take a whole minute. A thinking (reasoning) model writes thousands of hidden tokens before its answer, which is great for hard questions and expensive.
Text from outside can't be trusted. A web page or a file can hide instructions that the model can't reliably tell apart from the user's.
Failure must not lose messages. A sent message must never vanish, and a dropped connection on a train shouldn't waste a minute of paid GPU work.
The answer in short:
- Four model pools. A router picks the cheapest pool that can answer well, before the first token.
- A context builder that puts fixed text first and changing text last, so most prompt tokens are reused from earlier turns.
- Memory as two kinds of data. Saved facts the user controls, and notes about past chats fixed at the start of each conversation.
- Search, files and code as tools the model chooses to use, inside an agent loop with four limits.
- A per-answer log in Valkey, an in-memory store. Any stream server can resume any answer from it.
- The safety pipeline and the inference platform from our sibling posts, summarized here and linked.
- Post-training (DPO and GRPO, section 12) on the same GPUs.
- Numbers from maths. We had no GPU to test on. So GPU numbers come from the inference post's speed model with low, mid and high cases.
Mistakes this design avoids:
- Summarizing the chat every turn. The new summary breaks the cache for the turns after it, for about the same GPUs and less detail (section 5).
- Putting fresh memory notes at the top of every prompt. That needs 31% more GPUs.
- One model for every message. Thinking on every message would need about 4.7 times the GPUs.
- Sizing by output tokens. It gives about half the GPUs we need (section 14).
- Safety that fails closed everywhere. A guard outage would become a full outage.
- A stream that dies with the connection. The answer keeps going and the phone catches up.
- One Postgres for every message. Message writes need a sharded store.
- Changing the system prompt for everyone at once. Every live chat loses its cache together (section 16).
Design inputs. OpenAI said ChatGPT had 900 million weekly users in February 2026, and reports since say it's nearing a billion.
An OpenAI study counted about 25 messages per user per week in July 2025.
We plan for 1 billion weekly users and 25 messages each a week, about 3.6 billion messages a day or 41,000 a second. The busiest hour is 2.5 times the average, so the design point is about 103,000 messages a second.
OpenAI doesn't publish its model mix, prompt sizes or fleet. Those numbers below are our inputs. Section 22 shows how wrong they could be.
2. Requirements
2.1 Functional Requirements
A conversation is one chat. A message is one user turn plus the answer to it, and a run is the work that writes one answer. A tool is something the model can ask the system to do, like a web search.
| ID | Requirement | Priority |
|---|---|---|
| FR-01 | Multi-turn chat. Each answer sees the conversation so far | P0 |
| FR-02 | Stream answers as they're written. Resume after a dropped connection without losing or repeating text | P0 |
| FR-03 | Pick a model for each message. Fast, 70B or frontier, with thinking on or off. The user can also choose | P0 |
| FR-04 | Check inputs, outputs and tool actions for harm | P0 |
| FR-05 | Tools: web search, search over the user's files, a code sandbox and memory writes. Up to 8 model steps per message | P0 |
| FR-06 | Store, list and search conversations. Export and delete them. Edit a message or regenerate an answer, which makes a branch | P0 |
| FR-07 | Saved memories the user can see and delete. Notes from past chats, which the user can turn off | P1 |
| FR-08 | Images and files in messages | P1 |
| FR-09 | Plans (free and paid) with different limits, models and priority | P1 |
| FR-10 | Change prompts, tools, router settings and models without a deploy. Every change goes through safe steps | P0 |
| FR-11 | Post-train new model versions from preference data and checked rewards | P1 |
| FR-12 | Record usage, feedback and quality signals for analysis and training, with user opt-out | P1 |
2.2 Non-Functional Requirements
TTFT (time to first token) is the time from sending to the first visible word. TPOT (time per output token) is the gap between words after that.
p99 is the time 99% of cases beat. p50 and p95 work the same way. A zone is one data centre in a cloud region.
| ID | Requirement | Target |
|---|---|---|
| NFR-01 | Peak load | About 103,000 messages a second |
| NFR-02 | First visible token, plain chat | p99 under 1 s when the model reads up to 4,000 fresh tokens |
| NFR-03 | First visible token, longer fresh prompts | p99 under 3 s. During a zone loss, up to 8 s on the 70B pool |
| NFR-04 | First token with web search | "Searching" within 1.2 s. First answer token p95 under 4.5 s |
| NFR-05 | Thinking | "Thinking" within 1.2 s. The answer starts within about a minute |
| NFR-06 | TPOT, p99 | 30 ms on the fast and frontier pools, 40 ms on the 70B and thinking pools |
| NFR-07 | Safety added to TTFT | p95 under 60 ms on the normal path |
| NFR-08 | Durability | A zone loss loses no accepted message. A region loss loses at most about 6 s of messages |
| NFR-09 | Availability | 99.9% of messages get an answer from some pool |
| NFR-10 | Zone loss | Paid plans keep their models. Free plans lose thinking and the frontier model. No pool over 85% of its tested rate, except the thinking pool for about a minute |
| NFR-11 | Resume | While the answer runs, plus 60 s. Later, the saved message |
| NFR-12 | Deletion | 24 hours from live stores, 30 days from backups, unless a legal hold applies |
| NFR-13 | Cost | At most $0.002 per message, all costs counted, with peak GPUs rented by the hour |
Why 1 second? It matches the inference platform's target, but we count only fresh tokens and include the output guard's hold-back (section 11.2). Section 13.1 shows the budget.
Why does NFR-03 exist? A conversation whose cache was dropped (a cold one) needs its whole prompt read again, so prefill speed sets a minimum time there.
Why up to 8 s in a zone loss? After a zone loss, long 70B prompts wait up to about 7 s. Removing that wait costs about $2.3M a month (section 18.2), and a zone loss is rare.
Why does NFR-08 allow a loss? Messages copy to a paired region without waiting, so the newest seconds can be lost with the region. The app keeps each message until it reads it back, and re-sends any that are missing (section 17.1).
Why does NFR-13 mention renting? The cheapest mix rents part of the fleet by the hour at the daily peak. Reserving the whole peak fleet all day fails NFR-13 (section 21.2).
Why does NFR-05 give a thinking time? Users wait for hidden tokens, and our slower thinking step (section 4.6) makes the wait longer. So the wait is a target too. Thinking also stops at 8,000 tokens or 10 minutes.
2.3 Out of Scope
- Pre-training. See the 70B training post. We cover post-training only.
- Inside one model server. We summarize the inference post and link to it.
- Inside the safety pipeline. We summarize the safety post.
- How evals are run. See the evals post.
- Voice, image generation and video. Voice runs over a separate real-time API. Image generation has its own GPU pools.
- Long agent tasks, like deep research. They reuse the loop here with longer limits.
- Billing and payments.
3. Architecture
Here's the whole system.
Some terms first:
- A replica is one running copy of a model, on one or more GPUs.
- A pool is all the replicas of one model.
- Prefill is the model reading the prompt. Decode is writing the answer, one token per step.
- The KV cache holds what the model computed for each token it read. The prefix cache keeps it after a call ends, so the next turn can reuse it.
- A cache hit means a prompt token didn't need to be read again. The hit rate is the share of prompt tokens that hit.
- Valkey is an open-source in-memory store, a fork of Redis. Kafka is a durable log for events.
- A shard is one slice of a store's data. A copy of a shard on another machine is a shard copy.
- A lease is a lock with a time limit, so a crashed holder can't keep it forever.
- Idempotent means doing it twice has the same effect as doing it once.
The message path. Each step waits on the one before, except the input check. That one runs beside prefill.
- Gateway. The app sends the message with its remaining time, a client message id and the parent id. The gateway checks the session and the plan's limits.
- Save the message. The chat service takes the conversation's lease and writes the message to 2 of 3 store copies.
- Build the context. The context builder reads the recent chat and saved memories, plus first-turn notes and any file passages. Then it packs the prompt.
- Check the input. Safety checks run cheap ones first (a cascade), in parallel with step 6. No token is shown until they pass.
- Route and admit. The router picks a pool. Admission reserves tokens for the plan, or refuses when the pool is full.
- Prefill and decode. The prefix-aware router picks the replica holding this conversation's cache. The model reads the prompt and writes.
- Tools, if asked. The agent loop runs a search or code run, adds the result and calls the model again.
- Check the output. A token-level guard reads every token and holds back the newest 16.
- Log. The run writer appends released text to the run's log every 50 ms, with a sequence number.
- Stream. A stream server reads the log and sends server-sent events (SSE) to the app.
- Save the answer. When the run ends, the answer is saved and the lease released.
Work beside the message path:
- A. Notes. When a conversation goes quiet, the fast model writes a short note and its embedding, a vector that captures its meaning.
- B. Titles. The fast model names new conversations.
- C. Memory writes. The model saves a fact when the user asks, and the user sees it.
- D. Files. Uploads are parsed, cut into passages and embedded.
- E. Events. Usage, feedback, safety decisions and costs go to Kafka. From there they reach ClickHouse and S3.
- F. Releases. New models, prompts and settings move through replay, canary and A/B steps (section 16).
The parts:
| Part | What it does | Technology |
|---|---|---|
| Gateway | Sessions, plan limits, A/B assignment | Envoy plus our own service |
| Chat service | Leases, message writes, runs | Our own Go service |
| Context builder | Reads chat, memory and files. Packs the prompt | Our own Go service, with a tokenizer |
| Router | Picks the pool, and the replica inside it | A small classifier, plus the inference post's prefix-aware router |
| Model pools | Three models in four pools on B200 GPUs | vLLM or SGLang, with NVIDIA Dynamo for routing |
| Agent loop and tools | Runs tool calls with limits | Our own service. Code runs in Firecracker microVMs |
| Safety | Input cascade, output guard, action gate | The safety post's design |
| Token log | One log per run, numbered entries | Valkey streams |
| Stream servers | SSE to the apps | Our own Go service |
| Conversation store | Conversations, messages, notes and memories | A sharded wide-column store (Cassandra or ScyllaDB class) |
| Accounts | Users, plans, settings | Postgres, one primary and read replicas |
| Events | Usage, feedback, analytics | Kafka, ClickHouse, S3 |
Hardware. An NVIDIA B200 has 180 GB of memory, 8 TB/s of memory speed and 4.5 PFLOPS in 8-bit numbers. Eight share a server, joined by NVLink.
An H100 has 80 GB, so the frontier model wouldn't fit on one 8-GPU H100 server with room for the KV cache.
We price the B200 at $8.55 per GPU-hour, the median dedicated price in our sibling posts. AWS lists $14.24 on demand.
3.1 Why These Choices?
- Four pools, not one. Running every message on the biggest model would waste most of the fleet.
- A separate thinking pool. Hidden tokens don't need reading speed, so a slower step fits more requests per replica.
- Mixture-of-experts for the fast and frontier models. Each token uses only a few weights, so a 671B model computes like a 37B one.
- The context builder is its own service. It reads three stores in parallel, and its order decides the cache hit rate.
- A log per run. It separates writing an answer from delivering it, so the GPU never waits for a slow phone.
- A wide-column store for messages. OpenAI also moved its write-heavy data off Postgres.
3.2 Where It Sits, and Build or Buy
ChatGPT is the product layer. Under it sit the safety pipeline, the inference platform and the GPUs. Those, pre-training and evals each have a sibling post that we reuse.
| Option | Good at | Weak at |
|---|---|---|
| A managed model API behind a chat app | Fast start. No GPUs to run | At our volume, list prices are many times our GPU cost. Little control of routing and caches |
| Open-source chat apps (Open WebUI, LibreChat) | A ready interface and plug-ins | Built for one team or company, not for a billion users. No resumable streams or shared caches |
| Our own product layer on our own inference platform (chosen) | Full control of cost, prompts, caches and failure modes | We build and run all of it |
Why build? At OpenAI's listed flagship API prices, our average message would cost about $0.03, over 20 times our cost. A retail price includes a margin. But it shows why a company at this scale serves its own models.
3.3 The API
The app talks to three groups of endpoints. Sending is the busy one.
POST /v1/conversations/{conv_id}/messages
X-Remaining-Ms: 15000 # remaining time, not a clock time
{ "client_msg_id": "cm_7f3a", "parent_id": "m_41", "text": "Can my landlord...",
"attachments": ["file_9c2"], "choice": "auto" }
202 Accepted
{ "run_id": "run_c81_cm_7f3a", "stream": "/v1/runs/run_c81_cm_7f3a/events" }GET /v1/runs/run_c81_cm_7f3a/events
Last-Event-ID: 1.118 # resume after entry 118, written under fence 1
id: 1.119
event: text
data: "the rent can only change"
id: 1.120
event: citation
data: {"url": "https://...", "title": "..."}client_msg_idmakes sending idempotent. The run id is made from it, so every retry gets the same run, and a retry starts a run the first try never started.parent_idsays which message this answers. If it isn't the newest and not an edit, the server answers409with the newest id.choiceisauto,fast,thinkor a named model.- Event ids are
<fence>.<seq>. The fence changes only when the run moves to a new writer (section 10.2). - Events.
textcarries the answer,statussays "Searching…" or "Thinking…", andtoolandcitationcarry tools and sources.donehas usage and a finish reason.
Other endpoints list, search, export and delete conversations and memories.
4. The Inference Engine
🔒 Premium section
5. Context and Memory
🔒 Premium section
6. Retrieval, Files and Images
🔒 Premium section
7. Tools and the Agent Loop
🔒 Premium section
8. Routing, Admission and the GPU Scheduler
🔒 Premium section
9. Caching
🔒 Premium section
10. Streaming and Resumable Answers
🔒 Premium section
11. Safety in This Product
🔒 Premium section
12. Alignment and Post-Training
🔒 Premium section
13. One Message in Detail
🔒 Premium section
14. Capacity
🔒 Premium section
15. Data Stores
🔒 Premium section
16. Config and Safe Rollout
🔒 Premium section
17. What Happens When Each Part Goes Down
🔒 Premium section
18. Bottlenecks and How to Fix Them
🔒 Premium section
19. Regions and Bigger Scale
🔒 Premium section
20. Analytics and Operations
🔒 Premium section
21. Cost, Migration and Ownership
🔒 Premium section
22. Trade-offs, and What Is Standard
🔒 Premium section
23. Questions You'll Get Asked
🔒 Premium section
Related AI Engineering Chapters
🔒 Premium section
Related System Designs
🔒 Premium section
Further Reading
🔒 Premium section