Elasticity
Cold starts, autoscaling signals, and the build-vs-buy math.
The cold-start task teaches a method that generalizes: measure the whole path (provision, image pull, weight load, engine init) as a stacked bar, then attack the biggest bar. On most stacks that's weight loading, which is why streaming loaders and snapshotting exist. The autoscaling reading has a one-sentence takeaway worth the whole task: CPU/GPU utilization and QPS are the wrong scaling signals for LLM workloads, in-flight concurrency is the right one, and Little's Law is the queueing intuition behind why.
The last two tasks are the architecture-interview layer: serverless-versus-self-hosted crossovers, GPU selection, routing that knows about prefix caches and KV utilization, and the standing fact that offline batch inference is the cheapest tokens you'll ever serve. None of it is deep on its own. The skill is having the whole decision tree loaded when a design prompt opens with “you have a spiky workload and a budget.”
- Inference EngineeringPhilip Kielybook↗
Time vLLM vs SGLang from process start to first token on your fleet, break it down by stage, then attack the biggest bar (weight loading, compile time, snapshotting techniques).
Why CPU/GPU-util and QPS are the wrong signals and in-flight concurrency is the right one; queueing intuition via Little's Law; when scale-to-zero pays.
The LLM Inference Handbook's getting-started part: cost crossovers, procurement economics (hyperscaler vs neocloud), and mapping model sizes to GPUs. Standard architecture-interview material.
read+80 XPPrefix-aware / KV-utilization-aware / PD-aware routing; app-level gateways (LiteLLM-class) with exact + semantic caching; and why offline batch inference is the cheapest tokens you'll ever serve.
read+80 XP