# Quantization · Inference engines A decode step is memory bound because it spends most of its time streaming the model’s weights through memory rather than doing arithmetic on them, as established in [the memory-bound and compute-bound boundary](https://learn-kernels.com/chapters/foundations/compute-bound-and-memory-bound). Shrinking those weights shrinks that streaming cost directly. GPTQ does this with **weight-only quantization** (Compressing only the model's weights to a lower bit-width while leaving activations at their original precision.): it is a one-shot, post-training method based on approximate second-order information that compresses the weights of a model, including ones with 175 billion parameters, down to 3 or 4 bits each with negligible accuracy loss, and without any retraining. Because the compression touches only the weights, the arithmetic itself still runs at higher precision: the low-bit weights have to be dequantized back up before each matrix multiply, so the benefit is concentrated in the memory-bound decode step, where fewer bytes have to move, rather than in the compute-bound prefill step. SmoothQuant instead quantizes both operands of the matrix multiply, **weight and activation quantization** (Compressing both the model's weights and its activations to a low bit-width, most commonly 8 bits each (W8A8), so the matrix multiply itself can run in low precision instead of just being read from memory in low precision.). That is harder than quantizing weights alone because, as the paper puts it, weights are easy to quantize while activations are not: activation values in large language models develop large outliers concentrated in a handful of channels, and forcing those outliers into an 8-bit range destroys the precision available to every other value. SmoothQuant’s fix is a mathematically equivalent transformation, applied offline before serving, that migrates this quantization difficulty from the activations into the weights, making both sides easy enough to quantize to INT8 together. Because both operands end up in low precision, the matrix multiply itself can run on faster low-precision hardware paths, not just save memory traffic. > Figure. Weight-only versus weight-and-activation quantization. Two dataflows into the same matrix multiply. Weight-only quantization shrinks what streams from memory but dequantizes back to FP16 before the multiply, so the arithmetic stays high precision. Quantizing activations too lets the multiply itself run in INT8, at the cost of surviving activation outliers.Illustrative numbers AWQ sits in the same weight-only family as GPTQ but reads the problem through the activations. Its core observation is that not all weights in a model are equally important: protecting only 1 percent of **salient weights** (The small fraction of a model's weight channels whose quantization error most damages output quality, identified in AWQ by the distribution of the activations flowing through them rather than by the weights' own magnitudes.) can greatly reduce quantization error, and the way to find those salient channels is to look at the activation distribution, not at the weights themselves. Keeping that 1 percent at higher precision would leave a hardware-inefficient mixed-precision layout, so AWQ instead derives an equivalent transformation that scales up the salient channels before quantizing, with the scale determined by activation statistics collected offline. Because the method relies on no backpropagation and no reconstruction, it does not overfit its calibration set, and it generalizes across domains and modalities, including instruction-tuned and multimodal models. Low-bit weights only pay off if the kernels serving them are fast, which is why the AWQ paper ships with TinyChat, an inference framework built for 4-bit models. With kernel fusion and platform-aware weight packing, TinyChat runs more than 3 times faster than the Huggingface FP16 implementation on both desktop and mobile GPUs, and the compression is what lets a 70B Llama-2 model be deployed on a mobile GPU at all. Source Weight-only quantization and the 3 to 4 bit, 3.25× figures from [GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers](https://arxiv.org/abs/2210.17323) (Frantar et al., 2022). Weight-and-activation quantization and the outlier framing from [SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models](https://arxiv.org/abs/2211.10438) (Xiao et al., 2022). Salient-weight protection, activation-aware scaling, and the TinyChat figures from [AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration](https://proceedings.mlsys.org/paper_files/paper/2024/file/42a452cbafa9dd64e9ba4aa95cc1ef21-Paper-Conference.pdf) (Lin et al., MLSys 2024). More quantization methods in [Quantization](https://learn-kernels.com/chapters/reading#quantization) in the reading list.