The Architecture Zoo
MHA → MQA → GQA → MLA, RoPE, and MoE — through the serving lens.
Read every architecture here through one lens: what it does to KV bytes per token. MQA and GQA exist because the cache, not the weights, is what caps batch size, so actually do the arithmetic for each variant. The arithmetic is the content, not a chore attached to it. Slow down on MLA: latent compression changes both the cache math and what an attention kernel has to do, and DeepSeek-shaped models have made it table stakes.
EleutherAI's RoPE explainer is assigned alongside RoFormer because the paper's notation is heavier than the idea deserves. YaRN is here because context extension is a serving-time concern, not a training curiosity. Read the MoE material as systems papers: routing is an all-to-all communication problem, and it comes back hard in the disaggregation quest. The build task is the checkpoint. Verifying logits against a real Llama-family model catches the head-interleaving and rotation details that prose lets you gloss over.
- Build a Large Language Model (From Scratch)Sebastian Raschkabook↗
- Neural Networks: Zero to HeroAndrej Karpathycourse↗
- LLM Architecture GallerySebastian Raschkareference↗
Shazeer 2019 (one write-head) + GQA. Work out the KV-cache arithmetic for each — that IS the point of these designs.
Latent KV compression — mandatory 2026 knowledge. It changes cache math AND kernel design.
Then skim YaRN (arxiv.org/abs/2309.00071) — long-context serving depends on it.
Modify your implementation, verify logits against a HF reference model that uses them (e.g. a small Llama).
build+120 XPRouting, capacity factors, fine-grained + shared experts. Then skim the DeepSeek-V3 report (arxiv.org/abs/2412.19437) — it doubles as a systems paper.