The Recipe
Scaling laws are the interview math of the training world.
Chinchilla is paired with the Epoch replication because the replication teaches you to read scaling-laws papers critically: it corrected the original's parametric fit and strengthened the ~20-tokens-per-parameter headline. Then Beyond Chinchilla-Optimal breaks the spell. Once you serve a model at volume, compute-optimal is the wrong target, and training small models far past it (SmolLM3: 3B parameters, 11.2T tokens) is the industry default. Being able to argue both sides with arithmetic is what xAI lists as a basic qualification.
The Smol Training Playbook is the modern synthesis, the SmolLM team's actual decisions with their reasoning. It's the closest thing available to shadowing a pretraining team. μP is framed as the optional deep end on purpose: know what hyperparameter transfer buys and who uses it, and know that the flagship open recipes mostly don't. The drill at the end is pure arithmetic, like the KV-cache math on the other path: 6ND, token budgets, epoch limits, precision effects.
Then the replication (arxiv.org/abs/2404.10102) — how the Approach-3 fit was corrected, and why ~20 tokens/param survived the audit.
Why deployable small models train 100×+ past compute-optimal — Llama 3 8B at ~1,875 tok/param, SmolLM3 at ~3,700.
The SmolLM team's decision log for building world-class small models: architecture, data mixture, schedule — with reasons attached.
Low-precision training as an effective-parameter discount — and the finding that overtrained models quantize worse afterward, which ties back to Precision Games.
Tune on a small proxy, transfer the LR to the big run. Know the idea and its successors (u-μP, CompleteP) — and that most flagship open recipes ship without it.
6ND arithmetic, Chinchilla vs overtraining, epochs on limited data, precision effects. 75% to pass.
drill+80 XPauto-verified