System Design: Distributed 70B Training (Walnut Train: Parallelism Layout, FP8, Checkpoints, Failure Recovery, Gang Scheduling)
Goal: Design the platform that pre-trains a large language model, and run one big job on it. Ours is Walnut Train.
The job trains a dense 70-billion-parameter model on 15 trillion tokens. Evals and research jobs share the cluster.
We plan for 1,024 GPUs, about 58 days, a stop every 8 to 49 hours, and 97% of wall-clock time spent on saved progress.
Walnut Train is our design. Facts about real systems are dated evidence; every number of ours is labeled as ours.
Reading guide: sections 1 to 4 set the problem and the design, 5 to 15 go deep, 16 to 23 cover cost, operations and real systems, and 24 to 26 are for the interview.
1. What We Are Building
๐ Premium section
2. Requirements and a Quick Estimate
2.1 Functional Requirements
| # | The system must | Example |
|---|---|---|
| FR1 | Run one training job on about 1,000 GPUs as one unit | job-7014 on 1,024 GPUs |
| FR2 | Split the model across GPUs by a declared layout | The layout in mf-bs70-v4 |
| FR3 | Feed every rank its part of the data in a fixed order that can be replayed | Every restart reads the same next samples |
| FR4 | Save state often, and restore it after any stop, onto the same or a different GPU count | Wednesday's restore at step 26,700 |
| FR5 | Find crashes, hangs, slow servers and silent errors, and replace the bad server | gs-0417 swapped for gs-0901 |
| FR6 | Pause, roll back with a skip rule, and change the job only through a new manifest version | mf-bs70-v5 on Thursday |
| FR7 | Evaluate checkpoints during training without slowing the job | Kevin's core evals |
| FR8 | Share the cluster with evals and research by queues and quotas | Rachel's job-7102 gives up a server |
| FR9 | Hand the trained weights to the rollout system | The final export |
Collecting and tokenizing the data belongs to another system. Serving classic models belongs to the model serving post. Safety filters are out, and so is the rollout of a new model (the rollout post).
Post-training and reinforcement learning are out too. They put inference engines inside the training loop, which needs its own design.
2.2 Non-Functional Requirements
An NFR (non-functional requirement) says how well the system must work. Some words first:
- ETTR. The effective training time ratio: the share of wall-clock time spent making progress that ends up saved.
- Pilot. The planned short runs before the full run, which replace our estimates.
- A crash stops with an error, a hang stops with none, and a straggler runs slowly (sections 12.3 and 12.5).
- Silent data corruption, or SDC, is wrong numbers with no error (section 12.6).
- A loss spike is a sudden rise in the loss (section 13.2). A NaN is a "not a number" value from broken math.
- A snapshot is a copy of the state in server memory, and a durable checkpoint is a copy in long-term storage (section 11.3).
- The eval pool is a set of servers kept for evals (section 13.5).
| NFR | Target |
|---|---|
| NFR1 | Saved progress: the effective training time ratio is at least 95% over every 7 days; we aim at 97%. |
| NFR2 | Speed: after the pilot, 95% of steps run within 10% of the pilot's step time. |
| NFR3 | Recovery: one failed server costs at most 15 minutes of job time, lost work and restart together. |
| NFR4 | Checkpoints: each in-memory snapshot stalls training at most 5 s; a snapshot at least every 10 minutes; a durable checkpoint at least every 60 minutes; any durable checkpoint restores onto a different GPU count. |
| NFR5 | Data order: after any restart the job reads the same sample order from its saved position; samples are skipped only by a recorded skip rule. |
| NFR6 | Detection: a crash or GPU error within 2 minutes; a hang within 5 minutes; a straggler within 15 minutes. |
| NFR7 | Training health: a NaN pauses the job at once; a loss spike is flagged within 10 steps; a rollback is a recorded human decision. |
| NFR8 | Evals: the core suite finishes within 6 hours of each evaluated checkpoint, on the eval pool, never on the training GPUs. |
| NFR9 | Tenants: other queues never preempt the run's quota; a borrowed server is returned within 5 minutes of the request. |
| NFR10 | Time and cost: 15 trillion tokens in at most 60 days on 1,024 GPUs plus spares. |
Where the targets come from:
- NFR1. A 3-month run of a dense model of over 70B parameters on 9,600 GPUs reached 97% ETTR (ByteDance, October 2025 paper).
- NFR3 and NFR6. Llama 3 405B saw 419 unexpected stops in 54 days on 16,384 GPUs (Meta, July 2024 paper). Fast detection and recovery matter more than raw speed.
- NFR10. Apertus 70B trained a dense 70B model on 15T tokens (a September 2025 paper), the closest public run to ours.
2.3 The Numbers That Shape the Design
The GPU is a training chip with 180 GB of memory and about 4.5 PF of dense 8-bit math. The GPU server holds 8 of them, joined by NVLink, with one 50 GB/s network card per GPU.
No lab publishes these numbers for bluestem-70b. They are our assumptions and estimates. The terms first:
- Training state. Everything training keeps per parameter: an FP32 (32-bit) master weight, its FP32 gradient and two Adam moments, the optimizer's running averages, also FP32.
- The 6 x parameters x tokens rule. Training costs about 6 FLOPs per parameter per token, forward and backward together.
- Tokens a second per GPU. Job speed divided by the GPU count.
- MFU. Model FLOPs utilization: the share of a stated peak spent on model math. The dense 8-bit peak and the BF16 peak are the GPU's top speeds in 8-bit and in BF16 (a 16-bit format, section 7.2).
| What | One-line sum | Result |
|---|---|---|
| Training state | 70.6B x 16 bytes | about 1.13 TB |
| Checkpoint (no gradient) | 70.6B x 12 bytes | about 0.85 TB |
| Training math | 6 x 70.6B x 15T | about 6.4 x 10^24 FLOPs |
| Speed per GPU (estimate) | our planning speed | 3,000 tokens a second per GPU |
| Job speed | 1,024 x 3,000 | about 3.07 million tokens a second |
| Pure training time | 15T / 3.07 million / 86,400 | about 56.5 days |
| MFU | 3,000 x 6 x 70.6B = about 1.27 PF a GPU, against each peak | 28% of the dense 8-bit peak, 56% of the BF16 peak |
The 3,000 is our planning estimate. It sits below the published 64-GPU rows for the next GPU class (section 23), and the pilot replaces it (section 6.7).
All our times assume this planning speed. An MFU means nothing without its peak, and many published ones use the BF16 peak.
2.4 Days, GPUs and Failures
| What | One-line sum | Result |
|---|---|---|
| Wall-clock time at 97% ETTR | 56.5 / 0.97 | about 58 days |
| Servers | 1,024 / 8, plus 4 hot spares (section 12.4) | 128 GPU servers plus 4 (32 GPUs) |
| How often the job stops | sums in section 12.2 | once every 8 to 49 hours |
| Training data read | sum in section 10.6 | about 12 MB/s for the whole job |
The math sets the days, about 58. The network between servers sets the layout (section 6). The failure rate sets checkpoints, spares and restart speed (sections 11 and 12). So the design spends its effort on placement, checkpoints and recovery.
3. The Job API, the Data Model and the Stores
๐ Premium section
4. High-Level Design
๐ Premium section
5. Deep Dive: The Network: NVLink, Rails and Racks
๐ Premium section
6. Deep Dive: Splitting the Model Across 1,024 GPUs
๐ Premium section
7. Deep Dive: Number Formats: FP8 Now, FP4 Later
๐ Premium section
8. Deep Dive: One Training Step, End to End
๐ Premium section
9. Deep Dive: Memory on One GPU
๐ Premium section
10. Deep Dive: The Data Pipeline and Deterministic Resume
๐ Premium section
11. Deep Dive: Checkpoints
๐ Premium section
12. Deep Dive: Failures, Hot Spares and Fast Restart
๐ Premium section
13. Deep Dive: Loss Spikes, Rollback and Evals During Training
๐ Premium section
14. Deep Dive: Scheduling: Gangs, Topology, Quotas and Preemption
๐ Premium section
15. Deep Dive: When the Model Is a Mixture of Experts
๐ Premium section
16. Cost
๐ Premium section
17. Monitoring and Evaluation
๐ Premium section
18. Security
๐ Premium section
19. Bottlenecks
๐ Premium section
20. What Happens When Each Part Goes Down
๐ Premium section
21. Trade-Offs We Chose and Rejected
๐ Premium section
22. Multi-Cluster and Multi-Region
๐ Premium section
23. Walnut Train vs Real Frameworks and Schedulers in 2026
๐ Premium section
24. In the Interview (45 and 60 Minutes)
๐ Premium section
25. What to Remember
๐ Premium section
26. Further Reading
๐ Premium section
Explore the Technologies
๐ Premium section