Prefill and decode disaggregation
Generating a response happens in two phases with almost opposite performance characteristics. The prefill phase processes the entire prompt at once to produce the first output token; DistServe measures this as compute-bound, noting that for a 13B-parameter model, “processing a single sequence of 512 tokens can fully engage an A100 GPU.” The decode phase then generates one token per step, each step reading the full set of weights and the growing KV cache to produce a single new token, which is the same memory-bound pattern this book has already covered. Existing serving systems, DistServe observes, batch both phases together on the same GPUs to maximize throughput, but that colocation means “adding a single prefill job to a batch of decoding requests significantly slows down both processes,” stretching out both time to first token (TTFT) and time per output token (TPOT).
Disaggregation is the fix DistServe proposes: assign prefill and decode to separate GPU pools entirely. A prefill instance runs only the prefill computation for a request and forwards its KV cache to a decode instance, which owns everything after the first token. Because the two phases no longer share hardware, each can be scaled and parallelized on its own terms; DistServe finds that intra-op (tensor) parallelism lowers TTFT more effectively at low request rates by cutting execution time directly, while inter-op (pipeline) parallelism scales better at high request rates since it adds capacity linearly per GPU without the same communication cost. Tuned this way, DistServe reports serving “7.4× more requests or 12.6× tighter SLO” than state-of-the-art systems, which colocate the two phases, while keeping more than 90% of requests within their latency target.
The cost disaggregation introduces is moving the KV cache: every request now requires a transfer of prefill’s output state to whichever GPU is running decode, over whatever network sits between the two pools. NIXL, part of NVIDIA’s Dynamo stack, is purpose-built for that transfer: it “offers a unified abstraction across various memory types, including HBM, DRAM, local or remote SSDs” and picks a backend, such as UCX over InfiniBand or NVLink, based on where the source and destination actually live, so the inference engine issues one transfer request without hardcoding a transport. Disaggregation only pays off if that hand-off is cheap relative to the interference it removes, which is why DistServe pairs its placement algorithm with the cluster’s actual bandwidth rather than assuming the transfer is free.
DistServe is not the only system to land on this split. Splitwise, from Microsoft, reached the same architecture from a power and cost angle: its characterization finds that every request runs through “a compute-intensive prompt computation phase and a memory-intensive token generation phase, each with distinct latency, throughput, memory, and power characteristics,” and that even with state-of-the-art batching and scheduling the token generation phase underutilizes compute. The conclusion Splitwise draws is about hardware selection: “token generation does not need the compute capability of the latest GPUs and can be run with lower power and cost,” so the two pools do not even have to be built from the same GPU generation.
That freedom turns disaggregation into a cluster design problem. Splitwise moves request state between machines using optimized network libraries over the fast back-plane interconnects already present in GPU clusters, the role NIXL fills in the Dynamo stack, and uses the phase split to design both homogeneous clusters and heterogeneous ones where each phase gets hardware suited to it. Compared with an existing design, it reports clusters with “up to 1.4x higher throughput at 20% lower cost,” or “2.35x more throughput under the same power and cost budgets.” Prefill and decode were always two different workloads sharing one machine; once they are separated, hardware can be purchased for each, not just scheduled for each.
What the split looks like at launch#
Structurally, disaggregation is one worker becoming two. NVIDIA’s Dynamo ships presets for exactly that shape: a frontend, plus a prefill worker that computes the prompt’s KV cache and a decode worker that receives it over NIXL and generates tokens, each pinned to its own GPU. The two-GPU vLLM preset is short enough to read.
# run ingress
python -m dynamo.frontend &
CUDA_VISIBLE_DEVICES=0 python3 -m "$WORKER_MODULE" \
--model "$MODEL" \
--enforce-eager \
--disaggregation-mode decode \
--kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_both"}' &
VLLM_NIXL_SIDE_CHANNEL_PORT=20097 \
CUDA_VISIBLE_DEVICES=1 python3 -m "$WORKER_MODULE" \
--model "$MODEL" \
--enforce-eager \
--disaggregation-mode prefill \
--kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_both"}' &The two workers differ in three things: the GPU they are pinned to, the value of --disaggregation-mode, and a side-channel port that has to be unique per worker. Everything else matches, including the transfer configuration, which names the same NIXL connector on both sides rather than a sender and a receiver. That symmetry is the point. The pools are not two different programs, they are the same engine told which half of a request it owns.
What turns two processes into a system is the router. Dynamo’s prefill router picks a prefill worker using KV-aware routing, scored on cache overlap and load, or plain load balancing; the prefill worker computes the cache and hands back transfer metadata; the router injects that into the decode request and picks a decode worker, which uses the metadata to coordinate with the prefill worker directly. NIXL then moves the cache from VRAM to VRAM over “the optimal available transport (NVLink, InfiniBand/UCX, etc.),” and the transfer is non-blocking, so forward passes keep serving other requests while it is in flight. The hand-off DistServe pairs with a placement algorithm is, in a deployment, a routing decision plus a metadata blob.
The metadata is where the engines stop agreeing, and the difference costs latency. Under vLLM the payload is a set of block IDs plus remote worker connection information, and prefill runs synchronously: decode waits for it to finish. Under SGLang the payload is bootstrap information for an RDMA rendezvous, a host, a port, and a room ID published to the discovery service at startup, which lets prefill run as a background task so decode can begin immediately while the transfer proceeds in parallel. Neither pool is fixed in size either: workers register with the discovery service and publish their KV capacity, drain their active requests and deregister on the way out, and the router picks up the change, so the ratio of prefill workers to decode workers is a runtime property rather than a deployment constant.
Dynamo’s own performance notes are the useful corrective to the headline multipliers. Disaggregation, they say, “changes where compute and memory are consumed; it does not make either phase free.” At low concurrency an aggregated worker often avoids the transfer and the fragmentation of keeping two pools. Separate pools can improve latency isolation when prefill and decode contend under sustained load. And when decode KV capacity is what limits the deployment, adding prefill replicas “can reduce total serving capacity unless the latency benefit compensates for the GPUs removed from decode.” Even the prompt length at which a request should be sent to a dedicated prefill pool is refused as a constant: it depends on the model, the quantization, the GPU, the backend, and the request-length distribution, and “should not be encoded as a universal token threshold.”