Speculative decoding
Autoregressive decoding is serial by construction: generating K tokens takes K sequential runs of the model, each one waiting on the last token before it can start. But because a decode step is memory bound rather than compute bound, as the previous chapter established, the GPU is usually sitting on spare arithmetic capacity while it waits for weights to stream in. Speculative decoding spends that spare capacity on parallel verification instead of serial generation. A small, fast draft model proposes several tokens ahead, autoregressively, and the large target model then scores all of those candidates in a single parallel forward pass, the same cost as generating just one token normally.
The verification step is not a simple accept-or-reject on whether the two models agree. It uses a modified rejection sampling scheme: a draft token is kept whenever the target model would have assigned it at least as much probability as the draft model did, and when the target assigns it less probability, the token is only kept with probability equal to that ratio; a rejected token is replaced by resampling from a distribution built out of the leftover probability mass. That correction is what makes the method exact rather than an approximation: the sequence that comes out has exactly the distribution the target model would have produced sampling on its own, regardless of what draft model was used or how often it was wrong. What varies with the draft model’s quality is the acceptance rate, since every rejection means one parallel scoring pass produced only one or a few tokens instead of many.
Both papers that introduced this measured real speedups without retraining or changing the target model at all. Leviathan et al. report a 2 to 3 times latency improvement sampling from an 11 billion parameter T5-XXL model, walltime-tested against the standard T5X implementation; in one worked example, a 38-token sentence was produced from only 9 serial runs of the target model. Chen et al. report a 2 to 2.5 times decoding speedup on the 70 billion parameter Chinchilla model in a distributed serving setup, using the same draft-then-verify structure under the name speculative sampling.
The draft model itself is the operational weak point of this design: a separate model has to be acquired and maintained, and the Medusa paper names that as the obstacle impeding adoption. Medusa removes it. Instead of a second model, it adds extra decoding heads on top of the backbone’s last hidden states, each a single feed-forward layer with a residual connection predicting a token several positions ahead. The heads emit multiple top predictions per position, which are assembled into several candidate continuations and verified simultaneously in one decoding step through a tree-based attention mechanism, a simple adjustment to the attention mask. Fine-tuning only the heads on a frozen backbone, called Medusa-1, keeps the acceleration lossless and reaches over 2.2 times speedup; training the heads and backbone together, Medusa-2, raises that to 2.3 to 2.8 times.
EAGLE keeps a small drafting network but moves the drafting to a different level of the model. Its starting observations are that autoregression at the feature level, the second-to-top layer of the transformer, is more straightforward than at the token level, and that the inherent uncertainty in feature-level autoregression is what constrains its performance. EAGLE resolves that uncertainty by feeding the draft network a token sequence advanced by one time step alongside the features, which makes precise feature prediction possible with minimal overhead. On LLaMA2-Chat 70B this reaches latency speedups of 2.7 to 3.5 times and doubles throughput, while still maintaining the distribution of the generated text, the same exactness guarantee the original draft-and-verify schemes provide.