Production systems and serving benchmarks

Everything so far in this chapter happens inside one deployment of one model. A production serving system is the layer above that: it decides which replica a request lands on, when it runs, and how the fleet scales, and its defining problem is the tail rather than the average. Clockwork, a model serving system that predates LLMs, made the case that this layer can be built on prediction instead of reaction: existing serving architectures used “well-known reactive techniques” against common-case latency but could not curtail the tail latency caused by unpredictable execution times, while DNN inference itself “has deterministic performance.” Built bottom-up from those predictable execution times, with a centralized scheduler in place of reactive workers, Clockwork supported thousands of models while meeting “100 ms latency targets for 99.997% of requests.” The lesson carries directly into LLM serving: latency targets are met by scheduling, not only by fast kernels.

llm-d is what that layer looks like for LLMs today. Engines like vLLM and SGLang handle running the model on accelerators; llm-d provides “orchestration and optimizations above model servers,” deployed on Kubernetes, and its feature list reads like this chapter turned into routing decisions: prefix-cache and load-aware request balancing, tiered KV cache offloading with global indexing of cache state, prefill/decode disaggregation and wide expert parallelism for the largest models, and SLO-aware autoscaling driven by real-time inference signals. Where a request lands now matters as much as how fast the engine runs it: llm-d reports 3x higher output throughput and 2x faster TTFT from prefix-cache-aware routing compared to round-robin, measured on Llama 3.1 70B across 4 AMD MI300X GPUs, a gain that comes from routing each request to a replica that already holds its prefix in cache.

That routing layer is also becoming a standard rather than a product. Gateway API Inference Extension, an official Kubernetes project, extends any gateway that supports the Gateway API and Envoy’s external processing protocol into an inference gateway: a load balancer coupled with an endpoint picker that chooses a replica using metrics and capabilities the model servers themselves report, such as prefix cache status or which LoRA adapters a replica has loaded. The same layer carries policy that a plain load balancer has no vocabulary for: serving priority, so a latency-sensitive chat model can outrank a latency-tolerant summarization model, and incremental rollouts of new model versions by splitting traffic on model names.

Scaling the fleet has a cold start problem of its own: a new replica must load its model’s weights from a checkpoint before it can serve a single token. ServerlessLLM, an OSDI 2024 system, attacks that startup path by harnessing “the substantial near-GPU storage and memory capacities of inference servers” to keep checkpoints close to the accelerators. Its three mechanisms are a loading-optimized checkpoint format with a multi-tier loading system that uses the full bandwidth of the storage hierarchy, live migration of running inference so a new request can claim a server that already holds its checkpoint, and a scheduler that places each model on the server that minimizes its time to start. The paper reports “reducing latency by 10 - 200X across various LLM inference workloads” against state-of-the-art serverless systems.

Measuring such a system takes more than a throughput number. The etalon benchmark framework works with the per-request latency metrics this chapter has already met: time to first token, defined as “the time taken between arrival and first output token,” which includes “both scheduling delay and prompt processing time”; time between tokens, the gap between two consecutive output tokens of a streaming response; and time per output token, total generation time divided by the number of output tokens. Etalon’s starting observation is that even these “fail to fully capture the nuances of LLM inference,” leaving an incomplete picture of user-facing performance in real-time applications like chat. What a production system optimizes is goodput under service-level objectives: DistServe defines per-GPU goodput as “the maximum request rate that can be served adhering to the SLO attainment goal (say, 90%)” for each GPU provisioned. By these definitions a system can raise its throughput while lowering its goodput, simply by letting a slice of requests blow through their latency targets.

The load those metrics are measured under matters as much as the metrics themselves. Clockwork was evaluated “using production trace workloads,” and BurstGPT makes the same possible for LLM serving research: a public trace of real GPT-3.5 and GPT-4 traffic served on Azure, covering 121 consecutive days and roughly 5.29 million requests, each entry recording request and response token counts and whether the call came through a conversation or the API. The trace’s own overview plots weekly and daily periodicity in request arrivals, structure that a uniform synthetic load has none of, and its usage notes suggest scaling the trace’s average request rate to the evaluation setup rather than discarding the arrival pattern.

ServeGen carries that argument further. Built on a characterization of workloads “collected from our worldwide cloud inference serving service” at Alibaba, covering language, multimodal, and reasoning models, it argues that prior analyses were too limited in scale and scope to capture how real traffic behaves, and generates realistic workloads “by composing them on a per-client basis” rather than drawing from one aggregate distribution. Its production use case makes the stakes concrete: benchmarking with ServeGen “avoids 50% under-provisioning compared to naive workload generation.” A capacity plan validated against uniform synthetic load can simply be wrong.

Comparing systems across vendors needed a referee, and MLPerf became it. The MLPerf Inference paper (ISCA 2020) describes a field where over 100 organizations were building inference chips and existing systems spanned “at least three orders of magnitude in power consumption and five orders of magnitude in performance,” and answers with rules and best practices “to ensure comparability across systems with wildly differing architectures”; its first call for submissions drew more than 600 reproducible measurements from 14 organizations. MLPerf Endpoints is the same idea aimed at LLM serving: a submitted system is a measured curve rather than a single number, each point one real test at a fixed concurrency, relating system throughput, interactivity in tokens per second per user, and 95th-percentile TTFT. As MLCommons puts it, as load rises “total throughput goes up and per-user speed comes down. That is the core tradeoff in serving AI,” which is goodput’s tradeoff restated as a purchasing decision.