2 of 6

KV cache systems

The KV cache established that every decode step has to read the cached keys and values for every token that came before, so the size of that cache is a direct tax on the memory-bound step. One multiplier on that size is the number of key-value heads the model keeps. Multi-query attention collapses them to a single key-value head shared by all query heads, which drastically speeds up decoder inference but can degrade quality. Grouped-query attention sits between the two extremes, and the GQA paper showed it does not require training a model from scratch: an existing multi-head checkpoint can be uptrained into a grouped-query one using 5 percent of the original pre-training compute, reaching quality close to multi-head attention at speed comparable to multi-query attention.

Reducing the head count is not the only way to shrink what each token leaves behind. Multi-head latent attention, introduced with DeepSeek-V2, compresses the KV cache into a latent vector rather than storing full per-head keys and values. Combined with the rest of that model’s architecture, DeepSeek-V2 reduces the KV cache by 93.3 percent relative to its dense 67B predecessor while supporting a 128K token context and raising maximum generation throughput to 5.76 times.

A third lever is where the cache lives at all. Mooncake, the serving platform behind the Kimi chatbot, treats the KV cache as the center of the whole system: it separates prefill and decoding onto different clusters and pools the underexploited CPU, DRAM, SSD, and NIC resources of the GPU cluster into a disaggregated KV cache, so a prefix computed once can be stored and reused far beyond a single GPU’s memory. On real traces this KV-cache-centric design increased effective request capacity by 59 to 498 percent over baseline methods while meeting latency service-level objectives, and the deployed system runs across thousands of nodes processing over 100 billion tokens a day.

What each attention variant leaves in the cache. Eight query heads in every panel; the shaded shapes underneath are what a decode step must read back per past token. MHA caches keys and values for every head, MQA shares one key-value head across all of them, GQA gives each group of queries its own, and MLA stores only a compressed latent vector that all heads reconstruct from.Illustrative numbers

The cache can also be compressed in place, the same lever the next section applies to weights. KIVI starts from a study of how the cached elements of popular models are actually distributed, and the finding is asymmetric: the key cache should be quantized per-channel, grouping elements along the channel dimension, while the value cache should be quantized per-token. Built on that split, KIVI is a tuning-free 2-bit KV cache quantization algorithm, and with a hardware-friendly implementation it lets Llama-2, Falcon, and Mistral models keep almost the same quality while using 2.6 times less peak memory. The freed memory admits up to 4 times larger batches, which translates to 2.35 to 3.47 times the throughput on real inference workloads.

Once a cache outlives a single machine, as in Mooncake’s disaggregated store, moving it becomes its own problem: reusing a stored prefix avoids recomputing it, but the KV cache is a large tensor, and fetching it over the network can add enough delay to undo the savings. CacheGen treats this as a transmission problem. A custom tensor encoder leverages the distributional properties of the KV cache to encode it into compact bitstreams with negligible decoding overhead, and the compression level of different parts of the cache adapts to the bandwidth available while it streams, so loading delay stays low as conditions change. Against recent systems that reuse KV caches without encoding them this way, CacheGen reduced the cache size 3.5 to 4.3 times and the total delay of fetching and processing contexts 3.2 to 3.7 times, with negligible impact on response quality.

Source

Multi-query and grouped-query attention and the 5 percent uptraining figure from GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints (Ainslie et al., 2023). Multi-head latent attention and the 93.3 percent and 5.76× figures from DeepSeek-V2 (DeepSeek-AI, 2024). The disaggregated cache and capacity figures from Mooncake: Trading More Storage for Less Computation (Qin et al., FAST 2025). Per-channel key and per-token value quantization and the memory and throughput figures from KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache (Liu et al., ICML 2024). The KV cache encoding and transfer figures from CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving (Liu et al., SIGCOMM 2024). More in KV cache systems in the reading list.