The Backward Pass
Gradients, optimizers, and a loop that converges.
You built the forward pass in Phase 1; this quest is the other half, and it's anchored on Stanford CS336 because that course is the strongest from-scratch training curriculum in the open — its first assignment is essentially this quest with a grader. Backprop by hand isn't nostalgia: deriving the attention and cross-entropy gradients yourself, then checking against autograd, is the difference between using an optimizer and understanding one — and backprop-from-scratch is a reported ML-fundamentals round at OpenAI.
The AdamW-from-scratch task exists because the optimizer state is where training memory actually goes (two moments per parameter — the arithmetic that makes ZeRO necessary later), and the schedule details — warmup, cosine decay, gradient clipping — are exactly the knobs you'll watch when a run diverges. Finish with mixed precision measured, not assumed: bf16 autocast is the unquestioned default, and knowing what it does to throughput and loss on your own hardware is the habit this whole path drills.
Stanford's Language Modeling from Scratch (Spring 2026) — public lectures and assignments. Assignment 1 is the reference target for this quest.
On paper first, then in code without autograd; verify gradients against torch.autograd.gradcheck-style comparisons. A reported OpenAI ML-fundamentals interview round, verbatim.
build+120 XPNo torch.optim. Train your Phase-1 GPT on TinyStories with it. Count the optimizer-state bytes per parameter — that number is why ZeRO exists.
Autocast your loop, compare throughput and final loss vs fp32, and know why bf16 doesn't need the loss-scaling dance fp16 did.
The harness owns the workload — a ~1M-param GPT with fixed init on a synthetic corpus with a known entropy floor — and YOUR loop (your optimizer, your schedule) must reach the calibrated val-loss band on exactly one pass of the 1M-token budget. Token budgets, not wall-clock: any GPU passes in under a minute, CPU in ~10.
build+150 XPauto-verified