The 124M
Pretrain a real GPT-2. Yours. On your hardware or fifty dollars.
This is the Paul Graham artifact made literal: a GPT-2-class model, trained by you, as good as you can make it on the cheapest hardware you can get. nanochat is the blueprint (Karpathy's ~$100 full stack; its leaderboard drove time-to-GPT-2 from OpenAI's 168 hours to 1.65), and llm.c's discussions carry the cost math. Both honest routes are first-class here: ~$50 of rented 8×H100 time, or a multi-day run on your own card — ~28 hours on a 4090, a patient week on smaller — with checkpoint/resume discipline doing the work a cluster babysitter would.
Write the run plan before you spend a token: data shard, config, token budget, checkpoint cadence, cost both ways. That document is the difference between training a model and having trained one — and the public worklog at the end is the genre (per the speedrun-to-OpenAI pipeline) that training-side hiring actually reads.
The full pipeline in one hackable repo, and the community discussions where the real cost/quality tradeoffs live — including single-5090 runs.
FineWeb-Edu shard choice, model config, token budget vs the scaling math from The Recipe, checkpoint cadence, and the cost estimate for both routes (own GPU vs rented node).
write+60 XPTo the GPT-2 loss band, with resumable checkpoints and full telemetry kept. Auto-verifier (loss band + downstream probe + checkpoint chain) is in calibration — this converts to the path's flagship graded milestone.
build+400 XPThe run plan, the curves, what broke, what it cost, and honest evals. The training-side analog of the kernel worklog — a known door-opener.
write+200 XPauto-verified