# Inference engines

<!-- https://learn-kernels.com/chapters/inference-engines -->

A single kernel and a single request’s KV cache only explain what happens to one sequence on one GPU. An inference engine has to spread that same memory and compute across hundreds of concurrent requests, shrink the weights it streams every step, and turn spare memory bandwidth into extra tokens per second, all without changing what the model would have said on its own.

## 6 sections

1.  [4.1Scheduling and continuous batching](https://learn-kernels.com/chapters/inference-engines/scheduling-and-continuous-batching)
2.  [4.2KV cache systems](https://learn-kernels.com/chapters/inference-engines/kv-cache-systems)
3.  [4.3Quantization](https://learn-kernels.com/chapters/inference-engines/quantization)
4.  [4.4Speculative decoding](https://learn-kernels.com/chapters/inference-engines/speculative-decoding)
5.  [4.5Structured decoding and fairness](https://learn-kernels.com/chapters/inference-engines/structured-decoding-and-fairness)
6.  [4.6Long context and multimodal inference](https://learn-kernels.com/chapters/inference-engines/long-context-and-multimodal)