Chapter 2 · Kernel optimization
From naive matmul to fused, low-precision kernels
Chapter 2

Kernel optimization

From naive matmul to fused, low-precision kernels

Getting a kernel correct is the easy part. This chapter follows one matrix multiply from 1.3% of a GPU’s advertised throughput to 93.7% of it through nothing but memory-hierarchy discipline, then turns to the two techniques that multiply that hard-won bandwidth further: tensor cores that trade numeric precision for raw throughput, and the fused attention kernels that turned a quadratic memory-traffic problem into a solved one.

4 sections

  1. 2.1Foundational kernel exercises
  2. 2.2Matrix multiplication
  3. 2.3Tensor cores and low precision
  4. 2.4Attention