The KV cache

This is where GPU fundamentals stop being an abstract exercise and start explaining why inference engines look the way they do. Autoregressive sampling generates one token at a time, and at every step, self-attention needs the key and value vectors for every token already in the sequence, not just the new one. Recomputing them from scratch each step would mean redoing the same matrix multiplies over and over as the sequence grows. Instead, engines store those vectors in a KV cache the first time they are computed, and read them back on every later step.

That trade only pays off because of the boundary from the previous section. Computing the key and value vectors for one new token is cheap; the same weights are needed to run a full forward pass over the whole context, which is why a decode step spends most of its time waiting on memory bandwidth to stream the model’s weights through, rather than on arithmetic. The KV cache avoids paying that same memory cost again for tokens already seen, at the price of storing one more matrix per layer that grows with sequence length and batch size.