1 of 3

Why kernels matter

A modern accelerator is not one fast processor. It is a very wide machine with a deep memory hierarchy, and almost every disappointing performance number comes from the same root cause: the arithmetic units finished early and spent the rest of their time waiting for data. A kernel that reads its inputs badly can be an order of magnitude slower than one that reads them well, running the exact same arithmetic on the exact same chip.

The gap this book is about. The top bar is what the hardware can do at peak; the bottom bar is what naive code gets on the same chip. The lengths are not measurements, the shape is the point: a kernel that reads its inputs badly can be an order of magnitude slower than one that reads them well.Illustrative numbers

That is why this field has a distinctive shape. The interesting work is rarely inventing new mathematics. It is arranging a known computation so that the data it needs arrives in the right place, in the right order, in large enough pieces, while the arithmetic units stay busy. Tiling, coalescing, pipelining, and quantization are all answers to that one question.

The same logic scales outward. A single kernel worries about the trip from global memory to registers. An inference engine worries about the trip from one request to a batch of them. A distributed system worries about the trip across an interconnect. The unit of analysis changes; the reasoning does not.