System Design: Distributed 70B Training
Goal: You are training a 70-billion-parameter language model starting from random weights. The training run uses about 1024 GPUs for roughly a month. It will process around 3 trillion tokens of text.
The dollar cost of the run is in the millions. If a single bug halves your training speed, your boss asks why the bill doubled.
Design the training stack so the run actually finishes, the loss curve looks healthy, and you can recover from hardware failures without losing days of work.
Reference workload: A dense decoder-only transformer, 8,192-token training sequences, approximately 4.19 million tokens per optimizer step, 1024 H100 SXM 80GB GPUs, and BF16 training with FP32 optimizer state. The numbers are a worked design, not a vendor benchmark.
How to read this: Sections 1-3 define the problem, requirements, and one optimizer step. Sections 4-7 prove the architecture, topology, memory, network, and performance model. Sections 8-13 cover checkpoints, data, evaluation, failures, cost, preflight, and security. Section 14 closes with the feasibility verdict.
TL;DR: Split each model layer, long sequence, pipeline, and persistent training state across GPU groups chosen to match the physical network. Freeze the data and run manifests, save complete sharded checkpoints atomically, keep compatible spare nodes, and measure learning health alongside useful compute. Sections 5 and 6 name the exact parallelism and sharding mechanisms.
Terms Needed Before the First Calculation
🔒 Premium section
1. The Problem
A 70-billion-parameter model stored with two bytes per parameter needs about 140 GB for weights. The weights alone do not fit on one 80 GB GPU.
Then the optimizer arrives. Under one common mixed-precision AdamW policy, BF16 weights and gradients use four bytes per parameter together. An FP32 master parameter and two FP32 moment tensors use another twelve.
That policy needs about 1.12TB of persistent state. It doesn't include activations, gathered parameters, communication buffers, or framework overhead.
The exact byte count is a configuration property, not a law. Some implementations keep FP32 gradients, some use FP32 parameters as the optimizer's source of truth, and some use lower-precision optimizer state. Record the real dtypes and measure peak allocated memory for the pinned framework build.
So the model state, layers, sequences, and data batch are split across different GPU groups. The hard part is choosing groups that fit memory, respect the physical network, and leave enough work on every GPU to hide communication.
Five hard problems:
- The model is too big for one GPU. You must split the model itself, the data, or both.
- GPUs are fast, networks are slow (relatively). Every step requires GPUs to communicate gradients and activations. If the network is the bottleneck, throwing more GPUs at the problem makes things worse, not better.
- Failures are normal at this scale. Meta reported 419 unplanned interruptions during 54 days of Llama 3 405B training on 16,384 H100 GPUs. Treat its rate as a stress-planning reference, not as the predicted hardware failure rate of a different cluster. If this fleet experienced one interruption per 50,000 GPU-hours, 1024 GPUs would see one roughly every two days. Measure the real cluster rate and revise the checkpoint interval from it.
- Cost is enormous. A 30-day run at 1024 H100s costs something on the order of a few million dollars. A 10 percent training inefficiency is hundreds of thousands of wasted dollars.
- The loss curve must look right. If the loss spikes and never recovers, days of compute are wasted. You need monitoring tight enough to catch divergence within hours, not days.
Scale:
| Metric | Target |
|---|---|
| Model size | 70 billion parameters |
| GPUs | 1024 H100s, in groups of 8 per node, 128 nodes |
| Training tokens | 3 trillion |
| Training duration | 28 to 35 days |
| Training sequence length | 8,192 tokens |
| Effective batch size | 4,194,304 tokens, or 512 sequences per optimizer step |
| GPU compute utilization (MFU) | over 45 percent |
| Training availability while the reservation is active | at least 96 percent |
Where the 30 days comes from.
A transformer forward-plus-backward pass costs about 6 FLOPs per parameter per token. So the whole run is:
6 x 70e9 params x 3e12 tokens = 1.26e24 FLOPsFor this teaching profile, use 989 TFLOP/s of dense BF16 Tensor Core peak for an H100 SXM. Do not use a structured-sparsity figure unless the actual training kernels exploit that sparsity. At 1024 GPUs, the declared peak is about 1.01e18 FLOP/s.
MFU = achieved FLOP/s / peak FLOP/s
at 45%: 0.45 x 1.01e18 = 4.56e17 FLOP/s
time = 1.26e24 / 4.56e17 = 2.77e6 seconds = 32 daysThat is active training time, not calendar time. With 96 percent training availability:
calendar time = 32 active days / 0.96 = about 33.3 daysAvailability covers restarts, checkpoint stalls, planned evaluations, and other periods in which the reservation is paid for but useful training does not advance. Let active-run MFU fall from 45 to 35 percent and compute time rises from about 32 to 41 days before those delays are added.
The batch also connects the token goal to the training loop:
sequences per optimizer step = 4,194,304 / 8,192 = 512
optimizer steps = 3e12 / 4,194,304 ≈ 715,256
average active step time = 32 days / 715,256 ≈ 3.9 secondsThe release benchmark must show that the selected topology can sustain that step time without violating memory or numerical-stability limits.
1.1 Distributed-System Terms
| Term | Meaning in this design |
|---|---|
| Node | One server containing eight GPUs, host CPUs, memory, storage, and network interfaces |
| Rank | One trainer process controlling one GPU and holding one coordinate in the parallel topology |
| World size | Total rank count, 1024 here |
| Process group | The exact ranks participating in one kind of collective communication |
| Collective | A coordinated operation such as all-reduce, all-gather, or reduce-scatter |
| Microbatch | The small unit sent through the pipeline at one time |
| Global batch | All tokens contributing to one optimizer update across every data-parallel worker |
| Gradient accumulation | Process several microbatches before applying one optimizer update |
| HBM | High-bandwidth memory attached to a GPU |
| NVLink/NVSwitch | High-bandwidth GPU interconnect inside the reference node |
| InfiniBand | The reference high-speed network between nodes |
| Straggler | A slow rank that makes synchronized peers wait |
2. Requirements
Functional
| ID | Requirement | Priority |
|---|---|---|
| F1 | Train the full 70B model end to end on 1024 GPUs | P0 |
| F2 | Automatically resume from the last complete checkpoint after a single GPU or node failure | P0 |
| F3 | Log training metrics (loss, gradient norms, throughput) every step | P0 |
| F4 | Save a model checkpoint at least every 30 minutes of wall-clock time | P0 |
| F5 | Run small-scale eval suite every few thousand steps | P1 |
| F6 | Detect and alert on loss spikes within 15 minutes | P0 |
| F7 | Resume the exact global sample stream without unintended replay or omission | P0 |
| F8 | Pin immutable code, container, model, tokenizer, data, and parallel-layout manifests | P0 |
| F9 | Run held-out evaluations without pausing the full training fleet | P1 |
| F10 | Verify that every published checkpoint can be restored | P0 |
Non-functional
| ID | Requirement | Target |
|---|---|---|
| NFR-01 | Active-run efficiency | At least 45 percent MFU for the validated workload |
| NFR-02 | Average active step time | At most 3.9 seconds at the declared global batch |
| NFR-03 | GPU memory safety | Measured peak at most 72GB on an 80GB H100 |
| NFR-04 | Checkpoint recovery point | One complete checkpoint at least every 30 minutes |
| NFR-05 | Checkpoint completion | Under 5 minutes with bounded training interference |
| NFR-06 | Single-node recovery time | Under 30 minutes from detection through validated resume |
| NFR-07 | Training availability | At least 96 percent while the reservation is active |
| NFR-08 | Data reproducibility | Every committed step maps to one manifest and global sample range |
| NFR-09 | Checkpoint integrity | Restore only after every shard and the completion manifest verify |
| NFR-10 | Fabric safety | No sustained oversubscription on the validated TP, CP, PP, or SDP paths |
MFU uses the declared dense hardware peak. A lower absolute value is not automatically a defect, but a regression from the pinned release benchmark needs an explanation. The 72GB admission limit deliberately leaves 8GB outside the operating budget for fragmentation spikes and emergency diagnostics.
3. Follow One Optimizer Step
🔒 Premium section
4. High-Level Architecture
🔒 Premium section
5. Parallelism: Split Different Dimensions for Different Reasons
🔒 Premium section
6. FSDP2 Inside the Reference Topology
🔒 Premium section
7. Memory, Optimizer, and Precision
🔒 Premium section
8. Checkpointing and Exact Recovery
🔒 Premium section
9. Data Pipeline and Deterministic Sampling
🔒 Premium section
10. Evaluation and Run Validation
🔒 Premium section
11. Observability, Failures, and Response
🔒 Premium section
12. Cost and Capacity Decisions
🔒 Premium section
13. Preflight, Reproducibility, and Security
🔒 Premium section
14. Reference Design and Feasibility
🔒 Premium section
15. What an Interviewer Is Grading
🔒 Premium section
16. Follow-up Questions
🔒 Premium section
Related AI Engineering Chapters
🔒 Premium section
Further Reading
🔒 Premium section
Related System Designs
🔒 Premium section
