The Forward Pass

From scratch in PyTorch — embeddings, attention, MLP, layernorm placement. No nn.Transformer anything.

Implement scaled dot-product attention (causal + non-causal, numerically stable) and pass the harness — checked against torch SDPA including a large-logit stability case.

Pull the HF checkpoint into your implementation. If it rambles, your shapes are wrong somewhere.

BPE from scratch. Explains half of all 'weird LLM behavior' — and why streaming detokenization is subtle.

Min-p (ICLR 2025) ships in every engine now — read arxiv.org/abs/2407.01082 alongside. Include beam search: Perplexity hands candidates its exact signature and unit tests, and Mistral asks top-k/top-p from scratch with no libraries.