Chapter 5
Distributed inference
When one machine is not enough
A single GPU cannot hold a frontier model, and a single request cannot use a whole cluster efficiently. Everything in this chapter is about the seam between those two facts: how a model gets split across chips, how experts get routed to and dispatched across nodes, and how the two arithmetic phases of generation end up running on different hardware entirely.