The Serving Surface
Where production bugs actually live: templates, tools, adapters, modalities.
The least glamorous quest on the map, and the one most correlated with being useful on day one. Templates, tool parsing, and adapter management are where production incidents actually come from. Take the render-by-hand-and-diff exercise seriously: a chat-template mismatch doesn't error, it silently degrades quality, which makes it the worst class of bug to find. Tool calling is the same story one layer up — model-specific markup turned into OpenAI tool_calls JSON by per-model parsers, streaming deltas included, and every parser is a place things break.
The multi-LoRA and multimodal tasks are here to break the one-model-one-server assumption. Batched adapters share a base, and an embedding server is a different serving profile entirely: no KV cache, latency-critical. The security task's prefix-cache timing side channel is worth the read on its own. A shared cache leaks whether someone else's prompt shared your prefix — that's why cache salting exists — and raising it unprompted reads as senior in a design interview.
- Inference EngineeringPhilip Kielybook↗
Template mismatch is a top source of silent quality regressions. Render a template by hand for one model and diff it against apply_chat_template.
How model-emitted markup becomes OpenAI tool_calls JSON — per-model parsers (hermes/llama/mistral), and where they break. Skim MCP as the emerging layer above.
Deploy one reasoning model on your fleet: separate reasoning_content, cap thinking budgets, and measure what long decodes do to your ITL and cost math.
S-LoRA/LoRAX-lineage batched adapters. Serve ≥2 adapters on one vLLM instance, hot-load a third, and check adapters appear in /v1/models.
Image inputs via content parts (encoder scheduling + image-token budgets), plus a TEI-style embedding server — a different serving profile: no KV cache, latency-critical.
API keys (and what stays unauthenticated: /health, /metrics), prefix-cache timing side channels + cache salting, and a guardrails/prompt-injection primer for the gateway layer.