Many GPUs, One Model
DDP, ZeRO, FSDP2 — what training adds to the parallelism you already know.
You already built ring all-reduce and Megatron TP in the shared Parallelism quest; this quest adds what's training-specific. Data parallelism looks trivial (average the gradients) until you build real DDP: gradient bucketing and overlapping communication with the still-running backward pass are where the actual engineering lives, and building it from your own ring all-reduce closes the loop on that whole arc.
ZeRO is the memory argument you set up in The Backward Pass made structural: optimizer state, gradients, and parameters sharded in three stages — and FSDP2 is its PyTorch-native present, with torchtitan as the reference open stack that shows how FSDP2, TP, and float8 compose in a real pretraining codebase. The Ultra-Scale Playbook is the text that holds it all together; read it end to end here even though you met its appendices earlier.
Gradient buckets, comm/compute overlap with backward hooks, 2–4 CPU processes (gloo). Logits must match single-process training — same bar as the Megatron task.
build+150 XPThe three sharding stages and their memory math, then fully_shard on your model. Verify the per-rank memory drop matches the arithmetic.
How FSDP2 + TP + PP + float8 compose in one production-shaped codebase. The training-side analog of reading nano-vllm.
5D parallelism, ZeRO, kernels, and the comm/compute overlap math — the training analog of the scaling book, and the interview text for training-infra roles.