Profiling & the Roofline

The two Nsight tools answer different questions, and people conflate them constantly. Systems shows the timeline, where the GPU sits idle between kernels — the silent killer in inference. Compute shows the inside of one kernel; learn to read the SOL section and the memory charts, because that screenshot is the shared language of every performance discussion. The roofline drill is the most reliable interview filter in the field: given a workload's arithmetic intensity, say which side of the ridge it lands on and what that implies. Prefill and decode land on opposite sides. That's the entire field in one picture.

CUDA graphs close the loop on the overhead bottleneck from the very first quest. A decode step launches hundreds of tiny kernels, launch overhead swamps them, so engines capture the step once and replay it. And do the memory-snapshot task for real; the caching allocator and fragmentation OOMs are genuine on-call work, not curriculum filler.