LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale

Chapter 1: Quantization 101

Before we can understand LLM.int8(), we need to understand what quantization means and why it's hard. The core idea is simple: represent numbers with fewer bits.

A 16-bit floating point number (FP16) can represent values from approximately -65,504 to +65,504 with varying precision. It uses 1 sign bit, 5 exponent bits, and 10 mantissa bits. This gives you fine-grained precision near zero and coarser precision for large values — exactly what you want for neural network weights, which tend to cluster near zero.

An 8-bit signed integer (INT8) can represent exactly 256 values: the integers from -128 to +127. That's it. No fractions, no exponents. Just 256 evenly spaced integers.

To feel the difference: FP16 can distinguish between 1.0000 and 1.0010. INT8 can only distinguish between 1 and 2. If a weight has value 1.3, INT8 must round it to either 1 or 2 — losing the fractional part. This rounding error is quantization error, and the entire challenge of model quantization is keeping it small enough that the model doesn't notice.

Think of it like compressing a high-resolution photograph into a 256-color palette. You need a mapping function that assigns each original color to its closest palette entry, and a reverse mapping (dequantization) that converts back. The quality of the compressed image depends entirely on how cleverly you choose those 256 colors.

For neural networks, there are two main approaches:

Let's build these from scratch.

Why does quantization matter for matrix multiplication?

A transformer's core operation is matrix multiplication: hidden states X (shape [sequence_length, hidden_dim]) multiplied by weight matrices W (shape [hidden_dim, output_dim]). In FP16, each element of X and W is a 16-bit float. The multiplication produces a 32-bit output that's then cast back to FP16.

If we quantize both X and W to INT8, we halve the memory for storing weights and can use INT8 tensor cores, which are roughly 2x faster than FP16 cores on modern GPUs. The INT8 multiplication produces INT32 outputs (because two 8-bit integers multiplied can be up to 127 × 127 = 16,129, needing more than 8 bits), which we dequantize back to FP16. The key question: does this round-trip — quantize, multiply, dequantize — preserve the original FP16 result?

Where S_f16 is a scaling factor that converts the INT32 result back to FP16. The "approximately equals" is the crux — how close is the approximation? That depends on the quantization scheme. A good scheme minimizes the error between C_f16 (the true FP16 matmul) and the dequantized INT8 result. A bad scheme introduces enough error that the model's outputs become garbage.

Chapter 2: Absmax Quantization — Deriving It

Let's derive absmax quantization from first principles. We have a floating-point tensor X_f16 with shape [s, h] (sequence length by hidden dimension). We want to map every value into the integer range [-127, +127].

The strategy: find the largest absolute value in the tensor, and scale everything so that value maps to 127 (or -127). All other values are scaled proportionally and rounded to the nearest integer.

Step 1: Compute the scaling factor

Find the infinity norm of the tensor — the maximum absolute value across all elements:

This is the scaling factor. If the largest value in X is 5.0, then s_x = 127 / 5.0 = 25.4. Every value gets multiplied by 25.4, so 5.0 maps to 127, 2.5 maps to ~64, and 0.1 maps to ~3.

Step 2: Quantize

Where ⌊·⌉ denotes rounding to the nearest integer. That's the entire quantization formula. Multiply by the scaling factor, round to integer, clamp to [-127, 127].

Step 3: Dequantize

To recover the approximate original value, divide by the scaling factor:

Worked example

Suppose we have a small tensor X_f16 = [-0.8, 1.5, 0.3, -2.1, 0.7].

Not bad! The worst error is 0.006. But now imagine what happens if one value in the tensor is an outlier with magnitude 60:

This is foreshadowing. Remember this: a single large outlier destroys quantization precision for an entire tensor. We'll see that this is exactly what happens in large transformers.

python
import torch

def absmax_quantize(X_f16):
    """Absmax quantization: FP16 -> INT8"""
    scale = 127.0 / X_f16.abs().max()     # s_x = 127 / ||X||_inf
    X_i8 = (X_f16 * scale).round().clamp(-127, 127).to(torch.int8)
    return X_i8, scale

def absmax_dequantize(X_i8, scale):
    """Dequantize: INT8 -> FP16"""
    return X_i8.float() / scale

# Example: quantize a weight matrix
W = torch.randn(4096, 4096, dtype=torch.float16)
W_i8, scale = absmax_quantize(W)
W_recovered = absmax_dequantize(W_i8, scale)
print(f"Max error: {(W.float() - W_recovered).abs().max():.6f}")
# Typical output: Max error: 0.012

Chapter 3: Vector-wise Quantization

In the previous chapter, we used a single scaling factor for the entire tensor. One outlier anywhere ruins precision everywhere. The natural fix: use more scaling factors.

The key insight from Dettmers et al. is to view matrix multiplication as a sequence of independent inner products. Consider multiplying X ∈ R^s×h by W ∈ R^h×o. Each element of the output C[i, j] is the inner product of row i of X with column j of W. These inner products are independent — they don't share any computation.

Three levels of quantization granularity

The math

For vector-wise quantization, we assign a scaling constant c_xⁱ to each row i of X (computed as 127 / max|X[i, :]|) and a constant c_w^j to each column j of W (computed as 127 / max|W[:, j]|). The quantized matrix multiplication becomes:

Where c_x ∈ R^s is the vector of row-wise scaling constants, c_w ∈ R^o is the vector of column-wise constants, and ⊗ denotes the outer product. The dequantization matrix S = (c_x ⊗ c_w)⁻¹ has shape [s, o] — one dequantization factor per output element.

Worked example

Suppose X is 2×3 and W is 3×2:

python
import torch

def vector_wise_quantize(X, W):
    """Vector-wise quantization for matrix multiplication."""
    # Row-wise scaling for X: one constant per row
    cx = 127.0 / X.abs().amax(dim=1, keepdim=True).clamp(min=1e-8)  # shape [s, 1]
    X_i8 = (X * cx).round().clamp(-127, 127).to(torch.int8)

    # Column-wise scaling for W: one constant per column
    cw = 127.0 / W.abs().amax(dim=0, keepdim=True).clamp(min=1e-8)  # shape [1, o]
    W_i8 = (W * cw).round().clamp(-127, 127).to(torch.int8)

    # INT8 matmul (accumulate in INT32)
    C_i32 = X_i8.int() @ W_i8.int()  # shape [s, o]

    # Dequantize: outer product of inverse scaling constants
    S = 1.0 / (cx @ cw)  # shape [s, o] — one factor per output
    C_f16 = (C_i32.float() * S).half()

    return C_f16

The paper shows that vector-wise quantization preserves perplexity up to about 2.7B parameters. Beyond that, even vector-wise quantization starts to degrade — because of the emergent outlier features we'll see in Chapter 4.

Study this table carefully. Look at the 13B column. Tensor-wise absmax: 19.08 — worse than the 6.7B model. Vector-wise absmax: 16.48 — still terrible. Even zeropoint: 13.47, slightly degraded. Only LLM.int8() achieves 12.45, matching the FP32 baseline exactly. Something catastrophic happens at the 6.7B+ scale that only LLM.int8() can handle.

Chapter 4: Emergent Outlier Features

This is the paper's most important contribution: the discovery and characterization of emergent outlier features in large transformer hidden states. This section is fascinating because it reveals something genuinely surprising about how large language models work internally.

Dettmers et al. examined transformer hidden states X ∈ R^s×h across models from 125M to 13B parameters. They tracked individual feature dimensions h_i (columns of the hidden state matrix) across all layers, looking for values with magnitude ≥ 6.0.

What they found

At small scales (125M-1.3B parameters), outliers are rare and sporadic — maybe 1-2 dimensions with large values, appearing in about 25% of layers, affecting 6-18% of sequence positions. Quantization handles these fine because they're not systematic enough to dominate the scaling factor.

At medium scales (2.7B-6B), outliers become more common — 5-6 dimensions, appearing in 50-62% of layers. Quantization starts to struggle.

Look at the magnitude column. At 6.7B, the outlier values reach -44. At 13B, they reach -63. Normal feature values in a transformer hidden state are typically in the range [-3.5, +3.5]. These outliers are 10-20x larger than everything else.

Why do outliers matter for the model?

Dettmers et al. tested what happens when you remove outlier features by setting them to zero before they enter the attention layers. The results are dramatic:

These features make up only about 0.1% of all input dimensions, but they carry an outsized fraction of the model's "decision-making" information. They dominate the attention softmax, steering which tokens attend to which. The model has learned to encode critical information in just a handful of dimensions with extreme magnitudes.

Why does this happen?

The paper observes that emergence correlates more closely with perplexity than raw model size — a better-trained smaller model could potentially trigger the phase shift. The outliers appear to be a strategy the model develops to create sharp attention patterns. When a dimension has magnitude 60 and everything else is < 3, the softmax becomes extremely peaked on the tokens with the large outlier values. This is the mechanism the model uses to "pay attention" to specific positions.

The outliers are also highly asymmetric — mostly one-sided (either all positive or all negative across the sequence dimension). This is important: it explains why zeropoint (asymmetric) quantization outperforms absmax (symmetric) quantization for models at the 6.7B+ scale.

Chapter 5: Why Outliers Break INT8

Now we understand the two pieces of the puzzle: quantization schemes (Chapter 2-3) and emergent outliers (Chapter 4). Let's put them together and see exactly why standard quantization fails at scale.

The geometry of the problem

Remember that vector-wise quantization assigns one scaling constant per row of the hidden state X. A row of X is one token's hidden state — all h feature dimensions for a single sequence position. Outliers occur in specific feature columns — the same handful of hidden dimensions across all tokens.

Let's trace through a concrete example. Consider a hidden state with h=6 dimensions, where dimension 3 is an outlier dimension:

This is why the perplexity table shows degradation at 6.7B+. Every row that contains an outlier (75% of rows) has its normal-valued features crushed. These normal values — 99.9% of the tensor — carry the bulk of the semantic information. Destroying their precision destroys the model's ability to distinguish between similar tokens.

Why zeropoint helps but isn't enough

Recall that outlier features are almost always one-sided (all negative or all positive). Absmax quantization maps [-max, +max] to [-127, +127], wasting half the range when values are one-sided. Zeropoint quantization uses the asymmetric range [min, max] to [-127, +127], utilizing all 254 levels.

This explains why zeropoint outperforms absmax in the perplexity table at 6.7B (13.38 vs 14.13 for vector-wise). But by 13B, even zeropoint fails (13.47 vs 12.45 baseline) — the outlier magnitudes have grown so large (-63) that no amount of clever scaling within a single row can preserve precision for the normal values alongside them.

The fundamental impossibility

The fundamental impossibility

The problem is mathematically inescapable as long as we try to represent both outliers and normal values with the same 8-bit integers in the same row. We have 254 quantization levels. The range of values in a contaminated row is [-63, +3.5] or so. That's a span of 66.5. Each quantization step covers 66.5/254 = 0.26. For normal values that differ by 0.1 or less, they all quantize to the same integer. Information is permanently lost.

Why column-wise quantization isn't the answer

You might wonder: why not quantize by column instead of by row? Then each column gets its own scaling constant, and the outlier columns get their own large scale. The problem is that column-wise quantization for W means row-wise for X and vice versa — you can't independently optimize both. The standard inner-product structure of matrix multiplication requires one axis for X and the orthogonal axis for W. Vector-wise quantization already uses the best combination (rows of X, columns of W), but the outliers are in the columns of X — the axis that doesn't get its own scaling constant.

The solution? Don't try. Separate the outliers from the normal values and handle them differently. This is exactly what mixed-precision decomposition does.

Chapter 6: Mixed-Precision Decomposition

The insight is elegant in its simplicity: if outlier features live in a few columns and destroy quantization for everything they touch, just remove them before quantizing. Handle the outlier columns in full 16-bit precision. Quantize everything else to INT8. Then combine the results.

The algorithm step by step

Given input hidden states X_f16 ∈ R^s×h and weights W_f16 ∈ R^h×o:

The formula

In Einstein notation where indices are superscripts and h indexes the hidden dimension:

The first sum handles outlier dimensions in FP16. The second sum handles everything else in INT8 with vector-wise quantization (S_f16 is the dequantization scaling matrix from the outer product of row/column constants).

Why the memory cost is negligible

The critical fact: for transformers up to 13B parameters, |O| ≤ 7. Out of a hidden dimension of ~5120 at the 13B scale, only 7 dimensions are outliers. That's 7/5120 = 0.14% of the dimensions handled in FP16. The remaining 99.86% are in INT8. The extra memory for storing the FP16 outlier columns is negligible — about 0.1% additional memory on top of the INT8 baseline.

python
import torch

def llm_int8_matmul(X_f16, W_f16, threshold=6.0):
    """LLM.int8() mixed-precision matrix multiplication."""
    # Step 1: Find outlier columns in X
    outlier_mask = X_f16.abs().max(dim=0).values > threshold  # [h]
    outlier_cols = outlier_mask.nonzero().squeeze(-1)         # indices
    regular_cols = (~outlier_mask).nonzero().squeeze(-1)      # indices

    # Step 2: Decompose
    X_outlier = X_f16[:, outlier_cols]          # [s, |O|]
    W_outlier = W_f16[outlier_cols, :]          # [|O|, o]
    X_regular = X_f16[:, regular_cols]          # [s, h-|O|]
    W_regular = W_f16[regular_cols, :]          # [h-|O|, o]

    # Step 3: Outliers in FP16
    C_outlier = X_outlier @ W_outlier           # [s, o] in FP16

    # Step 4: Regular values — vector-wise INT8 quantization
    # Row-wise scaling for X_regular
    cx = 127.0 / X_regular.abs().amax(dim=1, keepdim=True).clamp(min=1e-8)
    X_i8 = (X_regular * cx).round().clamp(-127, 127).to(torch.int8)

    # Column-wise scaling for W_regular
    cw = 127.0 / W_regular.abs().amax(dim=0, keepdim=True).clamp(min=1e-8)
    W_i8 = (W_regular * cw).round().clamp(-127, 127).to(torch.int8)

    # INT8 matmul + dequantize
    C_i32 = X_i8.int() @ W_i8.int()
    S = 1.0 / (cx @ cw)                       # outer product dequant
    C_regular = (C_i32.float() * S).half()      # [s, o] back to FP16

    # Step 5: Combine
    return C_outlier + C_regular

# Usage: drop-in replacement for any linear layer
X = torch.randn(2048, 4096, dtype=torch.float16, device='cuda')
W = torch.randn(4096, 4096, dtype=torch.float16, device='cuda')
output = llm_int8_matmul(X, W)

Chapter 7: The Full LLM.int8() Algorithm

Let's put it all together. LLM.int8() is the combination of two techniques: vector-wise absmax quantization and mixed-precision decomposition. Together, they form a complete procedure that converts a 16-bit transformer checkpoint to 8-bit at inference time with zero performance degradation.

The complete data flow

For every linear layer in the transformer (feed-forward up-projection, down-projection, attention Q/K/V/O projections), the following happens at inference time:

What layers get quantized?

LLM.int8() targets the matrix multiplications in:

This covers 95% of all model parameters and 65-85% of all computation. The remaining operations (embeddings, normalization, attention scores) stay in FP16.

Memory savings

Since 99.9% of values are stored in INT8 (1 byte) instead of FP16 (2 bytes), the memory savings are approximately 2x for the quantized layers. For BLOOM-176B, the total memory reduction is 1.96x — from ~352 GB to ~180 GB. The slight shortfall from a perfect 2x comes from the FP16 outlier columns, the scaling constants, and the non-quantized layers (embeddings, norms).

Runtime performance

The quantization and decomposition overhead matters for runtime speed. For models smaller than 6.7B, LLM.int8() is actually slower than FP16 due to the overhead of outlier detection and two separate matmuls. But for large models (6.7B+), INT8 tensor cores provide enough speedup to overcome the overhead:

For BLOOM-176B end-to-end inference, LLM.int8() on 3x A100 80GB is comparable to FP16 on 8x A100 80GB — same speed, but using fewer than half the GPUs.

Chapter 8: Quantization Explorer

Now let's see the full LLM.int8() algorithm in action. This interactive simulation lets you experience the mixed-precision decomposition on a real(istic) hidden state matrix. You can control the outlier magnitude, the number of outlier dimensions, and see how the quantization error changes with and without mixed-precision decomposition.

Notice how in standard INT8 mode, increasing the outlier magnitude causes the error for normal values to spike dramatically. Switch to LLM.int8() mode and the normal-value error drops to near-zero because outliers are handled separately in FP16.

What happens inside the simulation

The simulation does exactly what the paper's algorithm does:

Standard INT8: All values are quantized with a single row-wise scaling factor. The outlier dominates the scale, crushing precision for normal values. Try setting the outlier magnitude to 70 and observe the red error bars — almost every token row has substantial error.

LLM.int8(): Outlier columns are extracted and multiplied in FP16 (zero error). The remaining values are quantized with their own row-wise scaling factor — now undistorted by outliers. The two results are summed. Total error is orders of magnitude lower. The dashed red lines show where the standard INT8 error would be — the improvement is dramatic.

Experiments to try

1. Increase outlier magnitude from 5 to 70. In standard mode, watch the error grow linearly. In LLM.int8() mode, the error for normal values stays flat — because the outlier magnitude doesn't affect the INT8 quantization path at all.

2. Set outlier columns to 0. Both modes become identical — there's nothing to decompose. This is why LLM.int8() has no benefit for models below 6.7B (few or no outliers).

3. Set outlier columns to 4. More outlier columns means more values handled in FP16, but the paper shows that even at 13B scale, there are at most 7 outlier dimensions. The overhead of the FP16 path stays negligible.

Chapter 9: Connections

LLM.int8() was published in August 2022 at NeurIPS and immediately changed the landscape of LLM deployment. Let's place it in the broader context of model compression and efficient inference.

What this paper established

What came next

Limitations acknowledged by the authors

INT8 only. The paper does not study FP8 data types. Since 2022, NVIDIA H100 and later GPUs added native FP8 support, which may provide better precision-performance tradeoffs than INT8 for the same bit width.

Inference only. LLM.int8() is designed for inference. The authors' initial experiments with INT8 training showed degradation for attention projections at scale (Appendix E of the paper). Training requires different techniques — this gap was partially filled by QLoRA in 2023.

Attention scores not quantized. The QK^T attention computation stays in FP16. For long-context models where attention memory dominates, this is a significant limitation.

Overhead at small scale. For models under 6.7B, LLM.int8() is slower than FP16. But models under 6.7B generally fit in GPU memory without quantization, so this is rarely a practical concern.

The deeper insight

Perhaps the most lasting impact of this paper isn't the quantization algorithm itself — it's the discovery of emergent features. The finding that large transformers spontaneously develop a handful of extreme-magnitude dimensions, that these dimensions are critical for attention, and that their emergence follows a phase transition — this tells us something fundamental about how neural networks organize information at scale. Subsequent work on model interpretability has built on this observation, studying what these outlier dimensions encode and why they emerge.

Using LLM.int8() today

python
# One line to load any model in 8-bit via bitsandbytes + HuggingFace
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-2-70b",
    load_in_8bit=True,          # <-- that's it
    device_map="auto"
)
# Model is now ~35 GB instead of ~140 GB
# Zero performance degradation