← Quest map
Phase 1 · Foundations
The Architecture Zoo
MHA → MQA → GQA → MLA, RoPE, and MoE — through the serving lens.
0/5 · 380 XP
You’re previewing the curriculum. Sign in (top right) to track progress, take knowledge checks, and unlock the verifiers.
Tasks
Shazeer 2019 (one write-head) + GQA. Work out the KV-cache arithmetic for each — that IS the point of these designs.
Latent KV compression — mandatory 2026 knowledge. It changes cache math AND kernel design.
Then skim YaRN (arxiv.org/abs/2309.00071) — long-context serving depends on it.
Modify your implementation, verify logits against a HF reference model that uses them (e.g. a small Llama).
build+120 XPRouting, capacity factors, fine-grained + shared experts. Then skim the DeepSeek-V3 report (arxiv.org/abs/2412.19437) — it doubles as a systems paper.