Disaggregation & MoE at Scale

Prefill/decode disaggregation and per-request goodput — the framing the industry adopted.

Cache tiering across DRAM/SSD, SLO-aware scheduling, real production numbers.

KV movement is a subsystem with job reqs attached now. Know what each piece does and why it exists.

DeepEP, EPLB, two-batch overlap — the single best writeup of modern MoE serving.

UCSD CSE 234-style: a sharded-linear expert layer both ways — TP (shard every expert) vs EP (all-to-all token routing) — and measure where each wins. CPU processes are fine; the communication pattern is the lesson.

The fourth parallelism axis — vLLM's decode context parallelism is production-real for long-context agents.

KV-aware load balancing is a K8s primitive now (GA). DRA replaced device-plugin-only thinking in 1.34.