Chapter 3 · Kernel optimization
From naive matmul to fused, low-precision kernels
Chapter 3

Kernel optimization

From naive matmul to fused, low-precision kernels

Getting a kernel correct is the easy part. This chapter follows one matrix multiply from 1.3% of cuBLAS’s throughput on the same GPU to 93.7% of it through nothing but memory-hierarchy discipline, then turns to the two techniques that multiply that hard-won bandwidth further: tensor cores that trade numeric precision for raw throughput, and the fused attention kernels that turned a quadratic memory-traffic problem into a solved one.

4 sections

  1. 3.1Foundational kernel exercises
  2. 3.2Matrix multiplication
  3. 3.3Tensor cores and low precision
  4. 3.4Attention