Quantization

A decode step is memory bound because it spends most of its time streaming the model’s weights through memory rather than doing arithmetic on them, as established in the memory-bound and compute-bound boundary. Shrinking those weights shrinks that streaming cost directly. GPTQ does this with weight-only quantization: it is a one-shot, post-training method based on approximate second-order information that compresses the weights of a model, including ones with 175 billion parameters, down to 3 or 4 bits each with negligible accuracy loss, and without any retraining. Because the compression touches only the weights, the arithmetic itself still runs at higher precision: the low-bit weights have to be dequantized back up before each matrix multiply, so the benefit is concentrated in the memory-bound decode step, where fewer bytes have to move, rather than in the compute-bound prefill step.

SmoothQuant instead quantizes both operands of the matrix multiply, weight and activation quantization. That is harder than quantizing weights alone because, as the paper puts it, weights are easy to quantize while activations are not: activation values in large language models develop large outliers concentrated in a handful of channels, and forcing those outliers into an 8-bit range destroys the precision available to every other value. SmoothQuant’s fix is a mathematically equivalent transformation, applied offline before serving, that migrates this quantization difficulty from the activations into the weights, making both sides easy enough to quantize to INT8 together. Because both operands end up in low precision, the matrix multiply itself can run on faster low-precision hardware paths, not just save memory traffic.

AWQ sits in the same weight-only family as GPTQ but reads the problem through the activations. Its core observation is that not all weights in a model are equally important: protecting only 1 percent of salient weights can greatly reduce quantization error, and the way to find those salient channels is to look at the activation distribution, not at the weights themselves. Keeping that 1 percent at higher precision would leave a hardware-inefficient mixed-precision layout, so AWQ instead derives an equivalent transformation that scales up the salient channels before quantizing, with the scale determined by activation statistics collected offline. Because the method relies on no backpropagation and no reconstruction, it does not overfit its calibration set, and it generalizes across domains and modalities, including instruction-tuned and multimodal models.

Low-bit weights only pay off if the kernels serving them are fast, which is why the AWQ paper ships with TinyChat, an inference framework built for 4-bit models. With kernel fusion and platform-aware weight packing, TinyChat runs more than 3 times faster than the Huggingface FP16 implementation on both desktop and mobile GPUs, and the compression is what lets a 70B Llama-2 model be deployed on a mobile GPU at all.