Why a GPU looks like that
A GPU is not a faster CPU. The two are built to different briefs, and almost everything in this book follows from that difference. NVIDIA states the trade directly: a CPU “is designed to excel at executing a serial sequence of operations (called a thread) as fast as possible and can execute a few tens of these threads in parallel,” while a GPU “is designed to excel at executing thousands of threads in parallel, trading off lower single-thread performance to achieve much greater total throughput.”
Read that carefully, because it contains a concession. The GPU is not better at everything. It is worse at the thing a CPU is for, and it gives that up deliberately in exchange for doing far more work in total. A single GPU thread is slow. There are simply an enormous number of them.
That trade shows up as silicon. GPUs “are specialized for highly parallel computations and devote more transistors to data processing units, while CPUs dedicate more transistors to data caching and flow control.” A CPU spends its area on the machinery that makes one instruction stream fast when the next instruction is unpredictable: branch predictors that guess which way an if will go, out-of-order execution that keeps working while one instruction waits, and large caches that keep recently used data close. All of that is area spent on managing work rather than doing it, and it is the right trade when there is only one instruction stream to manage. A GPU spends the same area on arithmetic units instead. The result, in the guide’s summary, is “much higher instruction throughput and memory bandwidth than a CPU within a similar price and power envelope.”
None of this was designed for neural networks. The GPU was “born as a special-purpose processor for 3D graphics,” fixed-function hardware for shading millions of pixels, which is the original embarrassingly parallel problem: every pixel is independent, and none of them individually matters much. Parts of that pipeline became programmable, and in 2006 NVIDIA introduced CUDA “to enable any computational workload to use the throughput capability of GPUs independent of graphics APIs.” Matrix multiplication turned out to want exactly what pixel shading wanted, which is why the hardware was already waiting when deep learning arrived.
Two budgets, not one#
A chip that is mostly arithmetic units has an obvious consequence and a less obvious one. The obvious one is that it can do a great deal of arithmetic. The less obvious one is that it therefore runs out of data long before it runs out of arithmetic.
Every kernel in this book is spending from two separate budgets. The first is arithmetic: how many floating-point operations per second the hardware can perform. The second is bandwidth: how many bytes per second it can move between memory and the chip. These are independent quantities with independent limits, and a given piece of work exhausts one of them first. Which one it exhausts is the single most useful thing to know about a kernel, because it tells you which optimizations can possibly help. Work limited by bandwidth does not get faster when you give it faster arithmetic.
The ratio between those two budgets has been moving in one direction for a long time, and it is not the flattering one. Arithmetic has grown faster than bandwidth, generation over generation, which means the amount of arithmetic you must perform per byte fetched, simply to keep the arithmetic units occupied, keeps rising. This is why a book about making GPUs fast spends so much of its length on data movement rather than on arithmetic. Compute-bound and memory-bound turns this into a model with numbers in it, the roofline, and works out where the boundary sits on real hardware. This section only establishes that there are two budgets and that they are not interchangeable.
There is one more mechanism worth naming here, because it explains why a GPU tolerates slow memory at all. It does not avoid the wait; it hides it. With many threads resident on a core at once, the hardware switches to a different thread whenever the current one stalls on a memory access, so the arithmetic units keep issuing as long as some thread is ready. This is why a GPU wants to be given far more work than it has cores to run it on, and why handing one a small job leaves it mostly idle. The specific version of this idea, occupancy, is the next chapter’s business.
When not to write a kernel#
A book about kernels should say early that most people should not write one. NVIDIA’s own guide puts libraries first: “When a library has already been implemented, especially those provided by NVIDIA, using it is often more productive and performant than reimplementing algorithms from scratch,” naming cuBLAS, cuFFT, cuDNN and CUTLASS, and noting that these “have the added benefit of being optimized for each GPU architecture.”
That is not a disclaimer, it is the actual economics. A vendor library is tuned per architecture by people with access to the hardware before it ships. Beating it requires knowing something about your specific problem that the library cannot assume: a fixed shape, a fusion opportunity across operations it sees separately, a precision you can tolerate that it cannot assume you can. Every genuine kernel win in the later chapters has that shape. Nobody beats cuBLAS at being cuBLAS.
The reason to understand kernels anyway, even if you never ship one, is that the same reasoning decides everything above them. Whether a batch is large enough, whether a cache is worth keeping, whether two operations should be fused, whether a request should be split across machines: these are all the two-budget question asked at a larger scale. The vocabulary is the point, not the C++.