Long context and multimodal inference

Everything so far assumed a request fits on the devices serving it. Long inputs break that assumption twice: the KV cache for a single very long sequence can outgrow any one device, and prefill compute grows quadratically with input length. The memory side is exactly what the Ring Attention paper targets, motivated by videos, actions, and other long-form sequences and modalities whose token counts overwhelm a single accelerator. Ring attention distributes blockwise computation of self-attention and feedforward across devices while fully overlapping the communication of key-value blocks with the computation of blockwise attention. The result is exact, not an approximation, and it scales the feasible sequence length by up to the number of devices in the ring, which the paper demonstrates at context sizes in the millions of tokens.

The compute side of long context is a prefill problem. The MInference paper measures the cost concretely: because attention is quadratic in the prompt length, an 8B parameter model takes 30 minutes to prefill a 1 million token prompt on a single A100. Its observation is that long-context attention matrices are not densely important; they exhibit three recurring structures, called A-shape, Vertical-Slash, and Block-Sparse. Dynamic sparse attention exploits this by assigning each attention head its best-fitting pattern offline, then building the sparse indices for that pattern dynamically at inference time and running optimized sparse kernels over them. Applied to existing models with no change to pre-training and no fine-tuning, this cuts prefill latency by up to 10 times on an A100 while maintaining accuracy across long-context benchmarks. Between them, the two techniques bracket the long-context problem: one spreads exact attention over more hardware, the other spends less compute per unit of hardware, and both leave the model’s output distribution intact enough to serve the same requests.

MInference recovers sparsity from a model trained with full attention; Native Sparse Attention, from DeepSeek, builds the sparsity in from the start. Natively trainable sparse attention uses a dynamic hierarchical strategy that combines coarse-grained token compression, which preserves global context awareness, with fine-grained token selection, which preserves local precision. The design is hardware-aligned, balancing arithmetic intensity so the sparse kernels actually run fast on modern GPUs, and because it is trainable end to end it reduces pretraining computation as well as inference cost. A model pretrained with NSA maintains or exceeds its full-attention counterpart across general benchmarks, long-context tasks, and instruction-based reasoning, while reaching substantial speedups over full attention on 64k-length sequences across decoding, forward propagation, and backward propagation.

Multimodal requests arrive through the same serving stack. In vLLM, a multimodal model takes its text prompt together with a separate multimodal data dictionary carrying the other modalities, and the documented input types cover images, video, and audio. Images can be passed as URLs, as image objects, or as pre-computed embeddings, and the maximum number of frames taken per video can be configured. The engine machinery this chapter has covered is what serves those requests too: a multimodal request enters the same scheduler as a text-only one, carrying extra inputs alongside its prompt rather than taking a separate path.