Parallelism

DeepMind's systems-math text. TPU-flavored, universally applicable. The best thing written on this.

Column/row-parallel splits and where the all-reduces land. Pair with GPU MODE lecture 17 (NCCL).

Ultra-Scale Playbook appendices A0 and A3 — the analytical grounding under everything in this phase.

Reduce-scatter + all-gather over point-to-point send/recv across 4 processes (CPU, gloo — no fleet needed). Collectives are monkeypatched to raise, so the ring is yours. Three peer courses grade exactly this before letting students near NCCL.

Shard your Phase 1 GPT's attention + MLP column/row-parallel across 2+ processes, placing the f/g all-reduces yourself (torch.distributed, CPU is fine). Logits must match single-process.

It will not be 2×. Explaining exactly why (NVLink vs PCIe, all-reduce cost per layer) is the lesson.

What gets communicated when, TP vs PP vs EP tradeoffs, interconnect math. 75% to pass.