Prefill and decode disaggregation
Generating a response happens in two phases with almost opposite performance characteristics. The prefill phase processes the entire prompt at once to produce the first output token; DistServe measures this as compute-bound, noting that for a 13B-parameter model, “processing a single sequence of 512 tokens can fully engage an A100 GPU.” The decode phase then generates one token per step, each step reading the full set of weights and the growing KV cache to produce a single new token, which is the same memory-bound pattern this book has already covered. Existing serving systems, DistServe observes, batch both phases together on the same GPUs to maximize throughput, but that colocation means “adding a single prefill job to a batch of decoding requests significantly slows down both processes,” stretching out both time to first token (TTFT) and time per output token (TPOT).
Disaggregation is the fix DistServe proposes: assign prefill and decode to separate GPU pools entirely. A prefill instance runs only the prefill computation for a request and forwards its KV cache to a decode instance, which owns everything after the first token. Because the two phases no longer share hardware, each can be scaled and parallelized on its own terms; DistServe finds that intra-op (tensor) parallelism lowers TTFT more effectively at low request rates by cutting execution time directly, while inter-op (pipeline) parallelism scales better at high request rates since it adds capacity linearly per GPU without the same communication cost. Tuned this way, DistServe reports serving “7.4× more requests or 12.6× tighter SLO” than a colocated baseline on the same hardware, while keeping 90% of requests within their latency target.
The cost disaggregation introduces is moving the KV cache: every request now requires a transfer of prefill’s output state to whichever GPU is running decode, over whatever network sits between the two pools. NIXL, part of NVIDIA’s Dynamo stack, is purpose-built for that transfer: it “offers a unified abstraction across various memory types, including HBM, DRAM, local or remote SSDs” and picks a backend, such as UCX over InfiniBand or NVLink, based on where the source and destination actually live, so the inference engine issues one transfer request without hardcoding a transport. Disaggregation only pays off if that hand-off is cheap relative to the interference it removes, which is why DistServe pairs its placement algorithm with the cluster’s actual bandwidth rather than assuming the transfer is free.
DistServe is not the only system to land on this split. Splitwise, from Microsoft, reached the same architecture from a power and cost angle: its characterization finds that every request runs through “a compute-intensive prompt computation phase and a memory-intensive token generation phase, each with distinct latency, throughput, memory, and power characteristics,” and that even with state-of-the-art batching and scheduling the token generation phase underutilizes compute. The conclusion Splitwise draws is about hardware selection: “token generation does not need the compute capability of the latest GPUs and can be run with lower power and cost,” so the two pools do not even have to be built from the same GPU generation.
That freedom turns disaggregation into a cluster design problem. Splitwise moves request state between machines using optimized network libraries over the fast back-plane interconnects already present in GPU clusters, the role NIXL fills in the Dynamo stack, and uses the phase split to design both homogeneous clusters and heterogeneous ones where each phase gets hardware suited to it. Compared with an existing design, it reports clusters with “up to 1.4x higher throughput at 20% lower cost,” or “2.35x more throughput under the same power and cost budgets.” Prefill and decode were always two different workloads sharing one machine; once they are separated, hardware can be purchased for each, not just scheduled for each.