# Generating one token at a time · Prerequisites A model of the kind just described maps a sequence to a set of scores over the next token. Producing a paragraph means running it many times: predict a token, append it to the sequence, and run again on the sequence that now includes it. This is what **autoregressive** (Generating a sequence one element at a time, where each new element is conditioned on all the elements produced before it.) generation means, and the dependency is real rather than an implementation choice. The transformer’s own masking enforces it: the paper modifies self-attention in the decoder “to prevent positions from attending to subsequent positions,” so that “the predictions for position i can depend only on the known outputs at positions less than i.” That constraint has an awkward consequence for hardware built to run thousands of threads at once. Training can process a whole sequence in parallel, because every position’s correct output is already known and masking alone prevents cheating. Generation cannot: token five does not exist until token four has been produced. The machine from the first section, which is fast only when saturated, is being asked to run a strictly sequential loop. ### Choosing the next token The model produces scores, not text, and something has to turn one into the other. The simplest rule is to take the highest-scoring token every time. That is greedy decoding, and it is deterministic: the same prompt yields the same output forever. Every other rule introduces randomness deliberately, and the three knobs that appear in almost every engine API are worth knowing before you meet them in a listing. vLLM’s own documentation defines them compactly. Temperature “controls the randomness of the sampling. Lower values make the model more deterministic, while higher values make the model more random. Zero means greedy sampling.” Top-k “controls the number of top tokens to consider,” discarding everything outside that shortlist. Top-p “controls the cumulative probability of the top tokens to consider,” keeping however many tokens are needed to reach that share of the total probability, which lets the shortlist grow and shrink with the model’s confidence. These matter here for one reason beyond vocabulary: they are cheap. Sampling touches one vector of vocabulary scores per step, against a forward pass that touched every parameter in the model. No amount of tuning them changes the performance picture, which is why this book discusses them only where they interact with something structural, as constrained decoding does in [Structured decoding and fairness](https://learn-kernels.com/chapters/inference-engines/structured-decoding-and-fairness). ### Two phases wearing one model Serving a request splits into two phases that look nothing alike. The first is *prefill*. Given the prompt, the engine processes “all the tokens in the prompt at the same time,” saving the resulting key and value projections in a KV cache along with the scores for the last token. Every prompt token is available up front, so this phase has as much parallel work in it as the prompt is long. The second is *decode*. It samples one token, passes that single token through the model attending to the cache, writes its own key and value projections back, and repeats. The scaling book puts the conclusion plainly: transformer inference is “two tasks in disguise.” > Figure. The same model, two shapes of work. Why prefill and decode behave differently on the same hardware. The weight matrix is identical in both, but prefill multiplies it by a block of every prompt token at once while a decode step multiplies it by a single row. The weights must be read from memory either way, so the second shape pays the same memory cost for a fraction of the arithmetic. (Illustrative numbers.) The reason the two behave differently is visible in that figure and follows from the two budgets of the first section. The weights are the same in both cases and must be read from memory in both cases. Prefill divides that reading across every token in the prompt; a decode step pays all of it for one token. The arithmetic per byte fetched therefore differs by orders of magnitude between two phases of the same request, running the same model. ### Counting the cost Two numbers make that argument concrete, and both are worth memorizing because the later chapters lean on them constantly. Storage first. Given a parameter count, “we can multiply by two to get bytes” at 16-bit precision, which is the arithmetic of the previous section applied to a whole model: two bytes per parameter, so a 70-billion-parameter model needs roughly 140 GB simply to hold its weights, before any activation or cache. This is the number that decides whether a model fits on a GPU at all, and it is why narrowing the format is a serving decision and not only a speed one. Arithmetic second. A forward pass does “2 P flops of operations, which can be intuited by the fact that we matmul through all the parameters,” where P is that same parameter count. Two floating-point operations per parameter per token, because a multiply-accumulate is a multiply and an add. Now put them beside each other, because the comparison is the whole point. A decode step performs about two floating-point operations for every parameter, and to do so it must read every one of those parameters out of memory. Two operations per two bytes fetched is an extraordinarily poor ratio on hardware whose arithmetic units outnumber its memory paths by the margin the first section described. Prefill does the identical reading but spreads it over every token in the prompt at once, so the ratio improves by roughly the prompt length. It is worth being precise about what inference does *not* pay for, because a reader arriving from training will expect costs that are simply absent. At inference time the weights are frozen and read-only. The scaling book lists what that removes: “during inference, we store one copy of our parameters,” and “there’s no optimizer state or gradients to keep track of.” Nor is there a backward pass to feed, so “because we don’t checkpoint (keep activations around for the backwards pass), our activation footprint is negligible.” A training system holds parameters, gradients, and optimizer state at once; an inference system holds parameters and a growing cache. Two consequences follow, and both shape the later chapters. Because the weights never change, the expensive work of converting them to a narrower format can be done once, offline, and the result reused by every request forever, which is what makes the [quantization](https://learn-kernels.com/chapters/inference-engines/quantization) of the previous section economically obvious in a way it never is during training. And because the weights are read-only, every request in a batch can share the same copy. Nothing has to be duplicated per request except that request’s own cache. That is the entire reason batching exists. If one decode step for one request reads all the weights to produce one token, then running sixty-four requests in the same step reads those same weights once and produces sixty-four tokens. The reading is amortized, and nothing else about the work changes. Almost every technique in [Inference engines](https://learn-kernels.com/chapters/inference-engines) is a variation on getting more useful tokens out of one pass over the weights. One structural point falls out of this and is worth carrying into the later chapters. A request runs prefill exactly once and a decode step once per token it generates, so a reply of five hundred tokens is five hundred sequential passes over the weights, each one of them the poorly-proportioned shape described above. For any response longer than a sentence, almost all of the wall-clock time and almost all of the memory traffic belong to decode. That is why the phase which looks like the trivial one, a single row through a matrix, is the phase most of this book is about. Which of these phases is limited by arithmetic and which by memory, and at what batch size the answer flips, is exactly the question [Compute-bound and memory-bound](https://learn-kernels.com/chapters/foundations/compute-bound-and-memory-bound) takes up next, with the numbers worked out on real hardware. The storage that the decode phase keeps accumulating, which grows with every token and is not covered by the parameter count at all, gets its own section in [The KV cache](https://learn-kernels.com/chapters/foundations/the-kv-cache). You now have everything those sections assume. Source The masking rule and its consequence for prediction order are from [Attention Is All You Need](https://arxiv.org/abs/1706.03762), section 3.1. The temperature, top-k and top-p definitions are quoted from the [vLLM SamplingParams](https://docs.vllm.ai/en/latest/api/vllm/sampling_params.html) reference. The prefill definition and the description of inference as two tasks are from [How to Scale Your Model: Inference](https://jax-ml.github.io/scaling-book/inference/), which is also where [the next chapter](https://learn-kernels.com/chapters/foundations/compute-bound-and-memory-bound) gets its critical batch size, and which is the source for the absence of optimizer state, gradients, and retained activations at inference time. The bytes-per-parameter rule and the 2 P flop count are from [Transformer Inference Arithmetic](https://kipply.github.io/blog/transformer-inference-arithmetic/), which derives the second by walking the transformer step by step. The 70-billion-parameter illustration applies that source’s rule and is not a figure it reports. All three documents are in [Start here](https://learn-kernels.com/chapters/reading#start-here).