Library

The hands-on companion to the Transformer Internals phase. What it adds over Karpathy's videos: BPE tokenization done properly, loading real GPT-2 checkpoints into your own code and matching their outputs — the milestone where “my code” and “a real model” become the same thing — and fine-tuning for classification and instruction following.

The repo (github.com/rasbt/LLMs-from-scratch) is fully open; the book buys you the prose and the ordering, which matter most from chapter 5 on.

The one book that covers the inference path's whole map — models, hardware, software, techniques, production — written by someone who spent four years at an inference shop explaining these tradeoffs to customers. It's a survey, not a deep dive: FlashAttention and FLOPs accounting get a paragraph where the quests here make you build them.

Use it as the connective tissue between quests: the chapter that tells you why a technique exists before the paper that tells you how, and the vocabulary reference for the interview gauntlet. Its Techniques chapter has the clearest short account of EAGLE-style speculative decoding in print. The PDF, EPUB, and audiobook are free from Baseten; only print costs money. Appendix B's reading list overlaps this library heavily — treat disagreements between the two as things to reconcile, not ignore.

Karpathy building neural nets from scratch on camera, from a scalar autograd engine to a small GPT. It's the on-ramp task in Go Brrrr for anyone who hasn't trained a model before, and it earns a library entry because you come back: The Forward Pass rebuilds the same GPT with real weights, and the training path re-derives the backward pass you first met here.

The canonical tour of strides, views, autograd, and the operator registry — the parts of PyTorch that haven't changed in seven years. What it can't cover is the compiler stack (torch.compile: Dynamo, AOTAutograd, Inductor), which is now the other half of “internals”; the author's podcast below picks that up.

The successor to the 2019 post's dispatch chapter: DispatchKeys as a stack of functionality (autograd, autocast, tracing) that intercepts an op before its kernel runs. Read it immediately after the internals post; together they explain why a one-line PyTorch call does so much more than launch a kernel.

The compute/memory/overhead taxonomy the whole map leans on — it's the first assigned reading in Go Brrrr, and it sits in the library because you'll reread it: before profiling in the GPU phase, and again when decode bandwidth math shows up in the inference engine. Every perf conversation in this field silently assumes this post.

The living edition of the internals post, by the same author, covering the compile stack the 2019 post predates. Commute-sized: each episode is one mechanism explained by someone who built it.

100+ models with layer mix, attention flavor (GQA/MLA/sliding window), KV-cache footprint, and links to each config.json and tech report. Read it as diffs from a model you've built: nearly every dense entry is the GPT skeleton plus the Llama recipe (RMSNorm, RoPE, SwiGLU, GQA); MoE, MLA, and the recurrent hybrids are the genuine departures.

SMs, warps, occupancy, HBM, TMA, NVLink — the device vocabulary, written by people who run GPUs for a living and kept current. Skim it once early so the words are loaded, then keep it open through the CUDA and kernel phases; it answers “wait, what's a TMA” faster than the whitepapers do.