Mixture-of-experts serving

A mixture-of-experts model replaces a single dense feed-forward block with many smaller ones and a router that picks which few run for each token. DeepSeek-V3 is a concrete example of the shape this takes: each MoE layer has 1 shared expert, always active, plus 256 routed experts, of which 8 are activated for any given token through expert routing, a sigmoid affinity score per expert with a per-expert bias term adjusted over training to keep load balanced without an auxiliary loss that would otherwise hurt model quality. The result is a model with 671B total parameters where only 37B are activated per token, which is the entire economic argument for MoE: most of the parameters sit idle for any single forward pass.

That sparsity has to be realized physically, and that is where expert parallelism and its communication pattern take over. DeepSeek-V3’s prefilling deployment unit spans 4 nodes and 32 GPUs, combining 4-way tensor parallelism for attention with 32-way expert parallelism for the MoE blocks, so that “each expert processes a sufficiently large batch size.” Because a token’s chosen experts can live on any GPU in that group, every MoE layer needs an all-to-all exchange: each GPU sends token activations out to the GPUs hosting its routed experts (dispatch) and receives the computed results back (combine). DeepEP, DeepSeek’s open-source dispatch and combine library, describes itself as providing “high-throughput and low-latency all-to-all GPU kernels” for exactly this exchange, built to route dispatch traffic across nodes over InfiniBand and then forward it to the right GPU within a node over NVLink.

Because the router decides per token which experts get used, expert load is never guaranteed even, and an overloaded expert stalls the GPU that hosts it while the rest of the all-to-all waits. DeepSeek-V3 addresses this by deploying redundant copies of the experts that online traffic statistics show are hottest, rebalancing that set roughly every 10 minutes; its decoding deployment goes further, spanning 40 nodes and 320 GPUs with one expert per GPU and 64 GPUs dedicated to hosting redundant and shared experts, using direct point-to-point transfers over InfiniBand to keep dispatch and combine latency low. The load-balancing problem and the communication problem are really the same problem seen from two sides: a router that is free to send every token anywhere needs both a placement strategy that keeps GPUs evenly loaded and a communication kernel fast enough that the resulting all-to-all does not dominate the layer’s runtime.

DeepSeek open-sourced the placement half of that answer as EPLB, its expert parallelism load balancer, which computes “a balanced expert replication and placement plan based on the estimated expert loads”; predicting those loads is left to the deployer, with a moving average of historical statistics named as the common method. It ships two policies. Hierarchical load balancing, meant for the prefilling stage with a smaller expert-parallel size, packs whole expert groups onto nodes evenly and then replicates within each node, exploiting DeepSeek-V3’s group-limited routing to place “the experts of the same group to the same node to reduce inter-node data traffic.” Global load balancing, for the decoding stage with a larger expert-parallel size, replicates experts across the whole fleet regardless of groups. Either way, replication and placement fall out of measured load rather than being fixed at deployment: the plan is an output of traffic.

MegaScale-Infer pushes the same disaggregation logic inside the MoE layer itself. Its starting observation is that sparse activation “shifts feed-forward networks (FFNs) from being compute-intensive to memory-intensive during inference, leading to substantially lower GPU utilization,” because each expert sees only its routed slice of the batch. The system therefore disaggregates the attention and FFN modules within each layer onto separate GPUs, giving each module its own parallelism strategy and hardware, and runs what it calls ping-pong pipeline parallelism: a request batch is partitioned into micro-batches that shuttle between the attention side and the expert side, so one side computes while the other’s traffic is in flight. Backed by an M2N communication library that strips out “unnecessary GPU-to-CPU data copies, group initialization overhead, and GPU synchronization,” MegaScale-Infer reports up to 1.90× higher per-GPU throughput than state-of-the-art systems.