Many GPUs, One Model
DDP, ZeRO, FSDP2 — what training adds to the parallelism you already know.
You already built ring all-reduce and Megatron TP in the shared Parallelism quest; this quest adds what's training-specific. Data parallelism looks trivial (average the gradients) until you build real DDP. Gradient bucketing and overlapping communication with the still-running backward pass are where the actual engineering lives, and building it on top of your own ring all-reduce closes that arc.
ZeRO takes the memory argument from The Backward Pass and makes it structural: optimizer state, gradients, and parameters sharded in three stages. FSDP2 is its PyTorch-native present, and torchtitan is the reference open stack showing how FSDP2, TP, and float8 compose in a real pretraining codebase. The Ultra-Scale Playbook holds it all together; read it end to end here even though you met its appendices earlier.
Gradient buckets, comm/compute overlap with backward hooks, 2–4 CPU processes (gloo). Logits must match single-process training — same bar as the Megatron task.
build+150 XPThe three sharding stages and their memory math, then fully_shard on your model. Verify the per-rank memory drop matches the arithmetic.
How FSDP2 + TP + PP + float8 compose in one production-shaped codebase. The training-side analog of reading nano-vllm.
5D parallelism, ZeRO, kernels, and the comm/compute overlap math — the training analog of the scaling book, and the interview text for training-infra roles.
The two schedulers training fleets actually run; Kubernetes you'll meet on the serving side. Enough to read an sbatch script and to say what a Ray cluster is (docs.ray.io/en/latest/ray-overview/index.html). Job posts ask for exactly this level.