Profiling, benchmarking, and correctness
A blocked or tiled program is, by design, no longer readable as a literal description of what one thread does. That means neither speed nor correctness can be checked by re-reading the source the way chapter 1’s per-thread CUDA code could be: both need to be measured against what actually ran on the GPU. NVIDIA splits those two jobs across separate tools. Nsight Compute answers “how fast, and where is the time going”; Compute Sanitizer answers “did it read or write anything it shouldn’t have.”
Nsight Compute profiles by inserting measurement libraries into the running application process, intercepting its communication with the CUDA driver, and collecting metrics whenever it detects a kernel launch. Its most direct performance readout is occupancy: “occupancy is the ratio of the number of active warps per multiprocessor to the maximum number of possible active warps,” and a large gap between the theoretical and the achieved value “typically indicates highly imbalanced workloads.” Its GPU Speed Of Light section reports the achieved percentage of compute and memory throughput against the theoretical maximum for each, which is the same flop-bound-versus-memory-bound question from chapter 1, now measured per kernel instead of estimated from a datasheet.
Collecting all of that is not free, and not always possible in one pass: the GPU has a limited number of hardware counters it can read concurrently, and some metrics require patch-based software counters whose overhead would itself distort the measurement. When a requested set of metrics cannot be collected together, Nsight Compute uses kernel replay , saving all memory the kernel can reach before the first pass and restoring whatever it wrote before each subsequent one. A profiled run is therefore not the same execution as an unprofiled one, which is a reason to treat wall-clock numbers taken while profiling as approximate.
Compute Sanitizer is the correctness half, shipped as part of the CUDA toolkit. Its own documentation states the reason it exists plainly: “every programmer invariably encounters memory access errors and thread ordering hazards that are hard to detect and time consuming to debug,” and “the number of such errors increases substantially when dealing with thousands of threads.” None of its four tools measure speed at all; they instrument actual memory and synchronization behavior and report a violation the moment one occurs, which is precisely the check a profiler has no reason to perform. A kernel can report high occupancy and near-peak memory throughput in Nsight Compute while still failing racecheck, because occupancy says nothing about whether two warps are racing on the same shared-memory address.
Beyond a single kernel#
Nsight Compute answers its questions one kernel at a time, and that is also its blind spot: it cannot say whether the kernel it is dissecting is the one worth dissecting. That is Nsight Systems’s job. By default it “collects a profile over the entire run of your application,” laying out CPU threads, CUDA API calls, and GPU activity on one timeline, and its guide recommends narrowing collection to the performance-critical region with cudaProfilerStart() and cudaProfilerStop() rather than profiling a test harness’s setup and validation. NVTX markers and ranges added to the application appear in the Timeline View and are projected onto the GPU timeline, “allowing you to see what GPU activity was launched within each CPU range.” The workflow this implies runs in one direction: find the expensive kernel, or the gap where no kernel is running, on the Systems timeline first, then drill into that kernel with Compute.
Even with the right kernel in hand, a number measured casually is not a number worth reporting. NVIDIA’s GEMM performance measurement methodology, published with the CUTLASS documentation, prescribes what a reproducible benchmark harness must do: separate warmup and profiling loops, delimited by cudaProfilerStart/Stop or CUDA events; no allocations, copies, or extra kernels between launches; and buffer rotation, cycling each tensor through duplicate buffers whose total footprint is at least twice the L2 capacity, so every iteration starts from DRAM instead of inheriting the last iteration’s cache. It distinguishes fixed frequency tests, which measure architectural and software efficiency at a locked clock, from fixed power tests, which mimic the dynamically-scaled clocks of real-world use and vary far more. The scale it recommends is sobering for anyone timing a kernel in a loop of ten: earlier GEMM studies needed more than 1,000 iterations for stable results on a 4096 by 4096 by 4096 problem, and for large GEMMs on Blackwell the document uses 10,000 warmups, 4,000 profiling iterations, and a second of cool-down between tests.
AMD’s counterpart to the kernel drill-down is ROCm Compute Profiler, also known by its package name rocprofiler-compute: “a kernel-level profiling tool for machine learning and high performance computing (HPC) workloads running on AMD Instinct GPUs.” It is built on ROCprofiler-SDK to monitor hardware performance counters, acquires those counters via application replay, and runs accelerator-specific microbenchmarks to build hierarchical roofline data. Its analysis vocabulary maps almost term for term onto the NVIDIA one: system Speed-of-Light and hardware-block-level Speed-of-Light summaries, memory chart and roofline analysis, and baseline comparisons between runs, all for the CDNA GPUs whose waves and chiplets HipKittens schedules around in the previous section.