Compute-bound and memory-bound
Every kernel eventually hits one of two limits: the time it spends waiting on arithmetic units, or the time it spends waiting on memory traffic. NVIDIA calls this boundary math bandwidth. A kernel is flop bound when there is never a moment where nothing is moving through memory; it is memory bound when there is never a moment where no floating-point operation is in flight. The ratio between a chip’s peak arithmetic throughput and its peak memory bandwidth sets exactly where that boundary falls, in operations performed per byte moved.
On an A100, that works out to 208 floating-point operations per byte moved. Below that arithmetic intensity, a kernel is memory bound: it finishes exactly as fast as its bytes can be streamed in, no matter how many arithmetic units sit idle. Above it, the kernel is flop bound, and streaming data faster would not help. This is the same reasoning the roofline model formalizes across an entire chip, not just one kernel.
The bandwidth in that ratio is a peak, and whether a kernel gets it depends on the shape of its accesses. Global memory loads and stores by the threads of a warp are coalesced by the device into as few transactions as possible, and on devices of compute capability 6.0 or higher the rule is simple: a warp’s concurrent accesses coalesce into however many 32-byte transactions are needed to service them all. Thirty-two threads reading adjacent 4-byte floats need four 32-byte transactions, and every byte fetched is a byte used. NVIDIA’s best practices guide marks keeping accesses coalesced as a high-priority recommendation, and the penalty for breaking the pattern is arithmetic: when adjacent threads access memory with a stride of two elements, half of every fetched segment goes unused, a 50 percent load and store efficiency, and as the stride grows the decay continues until a warp of 32 threads loads 32 separate 32-byte segments. A strided kernel is not paying the streaming rate the roofline promises; it is paying for bytes it never touches.
The other lever is keeping the memory system busy at all. Thread instructions execute sequentially, so when a warp stalls on a load, executing other warps is the only way to hide the latency, and occupancy names how much of that capacity is in play. What limits it is resource sharing: registers are allocated to a whole block at once out of a file that every resident thread shares, so on a compute capability 7.0 device with 65,536 registers per SM and up to 2,048 resident threads, full occupancy leaves each thread at most 32 registers. The guide is careful about the direction of the claim: higher occupancy does not always buy more performance, but low occupancy always interferes with hiding memory latency.
The same guide prescribes a workflow for applying any of this: Assess, Parallelize, Optimize, Deploy, a cycle rather than a checklist. Assess profiles the application to find the code responsible for the bulk of the execution time and uses Amdahl’s and Gustafson’s laws to bound what accelerating it can possibly buy. Parallelize exposes the parallelism, sometimes as simply as calling an existing GPU library. Optimize is explicitly iterative (identify an opportunity, apply and test it, verify the speedup, repeat), so no one needs to memorize every strategy before seeing gains. Deploy ships each partial speedup to production before the next hotspot is tackled, so every pass around the loop pays for itself.