How to shrink a 32-bit floating-point network into 8-bit integers — 4× smaller, 2–3× faster — while losing almost no accuracy. The whitepaper that defined the per-channel, symmetric/asymmetric, PTQ-vs-QAT vocabulary every modern inference engine now speaks.
You trained a great convolutional network on a server full of GPUs. Now you have to run it on a phone, a doorbell camera, or a hearing aid. Suddenly the rules change. That device has a few hundred megabytes of RAM, a battery that drains, no GPU, and a CPU that processes 8-bit integers far faster than 32-bit floats. Your beautiful float32 model is the wrong shape for the place it has to live.
Concretely: a ResNet-50 stores ~25.6 million weights. At float32 — 4 bytes each — that is ~102 MB just for the weights. Downloading it over a cell connection is slow; holding it in cache is wasteful; and every multiply-accumulate during inference moves 4-byte operands across the memory bus, which is where most of the energy goes.
An int8 can hold only 256 distinct values. A float32 can hold ~4 billion, spread across a vast dynamic range. When you map a continuous, wide-ranging weight distribution onto 256 buckets, every weight gets rounded to the nearest bucket. That rounding is quantization error, and it accumulates through dozens of layers. Done naively, the network's predictions drift and accuracy falls off a cliff.
The paper's whole job is to make that mapping smart: choose the buckets well (scale and zero-point), choose the granularity (one mapping for a whole tensor, or one per output channel), and — when post-hoc rounding isn't enough — teach the network during training to tolerate the rounding it will suffer at inference.
Let's make the savings concrete with a single dense layer of 1,000,000 weights:
| Quantity | float32 | int8 | Effect |
|---|---|---|---|
| Bytes per weight | 4 | 1 | 4× smaller storage |
| Layer size | 4.0 MB | 1.0 MB | 4× less to download / cache |
| Bytes moved per MAC | 4 | 1 | memory bandwidth & energy ↓ |
| Distinct representable values | ~4×109 | 256 | this is what we must spend wisely |
That last row is the tension of the entire paper: 256 buckets is plenty if you place them well, and a disaster if you don't.
You could — that is the MobileNet/SqueezeNet line of work. But quantization is attractive for a different reason: it is broadly applicable with little or no retraining. You take an existing trained float model and convert it. No new architecture search, no waiting for new hardware. The paper's headline is that weight-only post-training quantization needs zero data, and full 8-bit quantization needs only ~100 mini-batches of calibration.
Quantization is just a linear map between a real range and an integer range. Pick the smallest and largest real value you need to represent, stretch that interval evenly across the 256 integer slots, and you have a dictionary that converts back and forth. Two numbers define the whole dictionary:
Everything in the whitepaper is a choice along four axes. Memorize these; the rest of the lesson fills them in:
| Knob | Options | Tradeoff |
|---|---|---|
| Symmetry | symmetric (z=0) vs asymmetric (affine) | symmetric is cheaper in hardware; asymmetric fits skewed (e.g. ReLU) ranges better |
| Granularity | per-tensor vs per-channel | per-channel costs a few extra scalars but rescues skewed-channel layers |
| When | post-training (PTQ) vs quantization-aware training (QAT) | PTQ is instant; QAT recovers accuracy at 4 bits but needs a training loop |
| What | weights only vs weights + activations | weights-only needs no data; activations need calibration |
The whitepaper's central practical verdict — the one line every inference engine adopted — is:
Now the math — but motivated, not dropped from the sky. We have a real range (xmin, xmax) and we want to map it onto integers (0, Nlevels−1), where Nlevels=256 for 8 bits. The map is a line: shift, then scale, then round.
First, what is the step size? We have Nlevels−1 = 255 gaps to cover a width of (xmax−xmin). So each integer step is worth:
Next, which integer should real 0.0 land on? We want zero quantized exactly (padding and ReLU produce exact zeros all over the place; an error there is poison). So we define the zero-point as the integer index of 0.0:
Now quantize a real value x: divide by the step, round to the nearest integer, shift by the zero-point, then clamp into the valid integer range so outliers don't overflow:
And de-quantize — recover an approximate real value — by reversing it: subtract the zero-point, multiply by the step:
Quantize a single weight to unsigned int8 (Nlevels=256). Suppose a layer's weights span xmin=−0.6, xmax=0.8, and we want to quantize the weight x = 0.35.
| Step | Computation | Result |
|---|---|---|
| Scale | Δ = (0.8 − (−0.6)) / 255 = 1.4 / 255 | Δ ≈ 0.005490 |
| Zero-point | z = round(0.6 / 0.005490) = round(109.3) | z = 109 |
| Quantize | round(0.35 / 0.005490) + 109 = round(63.75) + 109 | xQ = 173 |
| De-quantize | (173 − 109) × 0.005490 = 64 × 0.005490 | x̂ ≈ 0.3514 |
| Quantization error | |0.3514 − 0.35| | ≈ 0.0014 |
The recovered value 0.3514 differs from the true 0.35 by ~0.0014 — at most half a step (Δ/2 ≈ 0.0027). That bounded, sawtooth-shaped error is the price of 256 buckets. The interactive widget below lets you feel it.
Drag bits down and watch the staircase coarsen — fewer, fatter steps means bigger gaps between representable values and a taller error sawtooth. Widen the range and the same number of steps must stretch further, so error grows everywhere. This is the central tradeoff: range vs resolution.
The affine quantizer of Chapter 2 carries a zero-point z. That extra integer offset is flexible — but it costs you in the inner loop. The whitepaper studies a simpler cousin: the symmetric quantizer, which forces z = 0.
With z=0 the conversions lose a term — no offset to add or subtract:
For signed int8 the integer range is [−128, 127], centred on 0. The scale is set from the largest absolute value: Δ = max(|x|)/127. Real zero sits at integer zero automatically. No zero-point means the de-quant is a single multiply — and, crucially, the cross-terms in a convolution disappear.
This is the "concept + realization" payoff. A convolution of quantized weight wQ and activation xQ, after substituting the de-quant formula, expands to:
Look at the four terms. The first, conv(wQ, xQ), is the cheap 8-bit dot product you wanted. The other three are correction terms created purely by the zero-points. They mean extra sums over the inputs — the paper notes a naive affine implementation costs ~3× more operations than the pure 8-bit dot product.
The interactive lets you see the waste: a symmetric quantizer on a one-sided distribution literally throws away the negative half of its 256 buckets.
Push skew all the way right to simulate a post-ReLU activation (all values positive). In symmetric mode, count the wasted buckets below zero — resolution thrown away. Switch to asymmetric and the zero-point slides over so all 256 buckets cover the real data — at the cost of the conv cross-terms.
Here is where the paper's most important practical finding lives. When you quantize a weight tensor, how many (Δ, z) pairs do you use? The simple answer is one per whole tensor — per-tensor (also called per-layer) quantization. The better answer, for weights, is one per output channel — per-channel quantization.
A conv weight tensor has shape [Cout, Cin, K, K] — it is a stack of Cout separate 3-D kernels, one per output feature map. The problem: different kernels can have wildly different magnitude ranges. One kernel's weights might span ±0.05; another's ±2.0 — a 40× difference.
A single per-tensor scale must cover the widest kernel (±2.0). When you apply that coarse scale to the small kernel (±0.05), all its weights collapse into just a handful of the 256 buckets — near-total loss of resolution for that channel. The paper found this is exactly why per-tensor quantization devastates MobileNet (top-1 crashing from 70.9% to 0.1%).
The fix is almost embarrassingly simple. Give each of the Cout kernels its own Δ (and z, if affine). The small kernel gets a fine scale; the large kernel gets a coarse one. Each kernel now uses its full 256 buckets.
| Scheme | #scales for weights | Extra storage | Inner-loop cost |
|---|---|---|---|
| Per-tensor | 1 | negligible | baseline |
| Per-channel | Cout (e.g. 256) | ~Cout floats — tiny vs the weight tensor | same dot product; scales applied per output map — still SIMD-friendly |
The cost is Cout extra float scalars — trivial next to millions of weights — and because the scale is fixed per output map, the integer dot product is unchanged. You pay almost nothing and rescue the accuracy. Note: per-channel is used for weights, not activations — activations are quantized per-tensor because a per-channel activation scale would break the shared accumulator in the matmul inner loop.
Each row is one output channel's kernel; brightness = effective resolution (how many of the 256 buckets that channel actually uses). Crank channel spread up to mimic BN-induced variation, then toggle per-tensor vs per-channel. In per-tensor mode the small-magnitude rows go dim — collapsed into a few buckets. Per-channel lights every row back up. The mean-squared error readout drops accordingly.
You have a trained float model and you want it quantized now, with minimal fuss. That is PTQ: convert after training, no gradient steps. The paper splits it into two flavours by how much you're willing to do.
Quantize only the weights to int8; keep activations in float at runtime. Because weights are fixed constants known at convert time, you compute their (Δ, z) directly from their min/max — no calibration data needed at all. A command-line tool reads the checkpoint and writes int8 weights.
This buys you the full 4× model-size reduction (great for download/storage) but not the inference speedup, since the math still runs in float after de-quantizing. Data flow: int8 weights → dequant to float → float conv with float activations.
To actually run integer arithmetic you must also quantize activations — and unlike weights, you don't know an activation's range until you run data through. So you calibrate: feed ~100 mini-batches through the float model and record each activation tensor's running min/max.
With per-channel weights + per-tensor activations, PTQ lands within ~2% of float across the board — without retraining. The failure mode is per-tensor weights on MobileNet-style nets (the BN problem from Chapter 4). And almost all the remaining error comes from weight quantization, not activations: activations quantize cleanly to 8 bits because ReLU6 and BN already keep their dynamic range small.
PTQ at 8 bits is great. But push to 4 bits (only 16 levels!) and post-hoc rounding falls apart — especially on small nets. The fix is quantization-aware training (QAT): let the network see the rounding during training and learn weights that are robust to it. But there's a problem: rounding has zero gradient almost everywhere. How do you backprop through it?
QAT inserts fake-quant nodes: a quantize immediately followed by a de-quantize. The tensor stays in float, but its values are snapped to the quantization grid — the forward pass now experiences exactly the rounding error inference will suffer:
The weights you keep and update are still full-precision float; fake-quant just makes the loss "feel" the quantization on every forward pass, so SGD steers toward a flat, quantization-tolerant region of the loss landscape.
round() is a staircase — flat between steps (derivative 0), with vertical jumps (derivative undefined). If you used the true derivative, no gradient would ever flow and training would freeze. The trick, due to Bengio, is the straight-through estimator (STE): in the backward pass, pretend the quantizer was the identity — pass the gradient straight through, except where the value was clamped (saturated), where you pass zero.
In words: inside the range, gradient flows unchanged (as if round() weren't there); outside the range, gradient is killed (a saturated value can't usefully move). This is a deliberate lie in the backward pass that makes the forward simulation trainable.
The SGD update is applied to the float weights, and only then re-quantized for the next forward pass:
If you updated the quantized weights directly, a tiny gradient (smaller than Δ) would round to zero and vanish — the weight could never move. Keeping a float master lets many tiny updates accumulate until they cross a quantization step. This is the same insight as mixed-precision training's master weights.
In Forward view, the orange staircase is what the loss sees — the float input (dashed) snapped to the grid. Switch to Backward: the true derivative of the staircase is 0 everywhere (useless), so STE substitutes the green line — gradient = 1 inside the range, 0 outside (the clamped tails). Drop levels to see the staircase coarsen toward 4-bit / 3-bit regimes.
One subtlety can quietly wreck QAT: batch normalization. At inference, BN is a fixed affine rescale per channel, so it gets folded into the preceding conv's weights and bias — one fused op. But during training, BN uses live batch statistics that wobble from step to step. If you fake-quant the conv weights using wobbling batch statistics, the quantizer's range jitters and training destabilizes.
BN applies, per output channel c: y = γc(conv(w,x) − μc)/√(σc²+ε) + βc. Because that is linear in conv(w,x), you can absorb the scale into the weights and the shift into the bias:
Each output channel's weights are scaled by γc/√(σc²+ε) — and this per-channel factor is precisely the source of the cross-channel range variation from Chapter 4. Now you see why per-channel weight quantization is not optional for BN networks: the BN fold creates the spread that per-tensor quantization can't handle.
The paper folds BN into the conv for the quantized forward pass, but uses moving-average (inference-time) statistics for the fold — not the noisy current-batch statistics — with a correction so the training math still matches what runs at inference. This makes the quantizer ranges stable and ensures the trained-with-BN network and the deployed-folded network compute the same thing.
Left bars: raw conv weight ranges per channel (similar). Right bars: after folding each channel's BN factor γc/√varc. Increase the spread and watch the folded ranges fan out — that fan-out is exactly what defeats a single per-tensor scale.
The paper's tables are dense, so let's extract the story. Three findings matter most, and the interactive below lets you toggle between them on real ImageNet top-1 numbers.
Weight-only PTQ, MobileNet-v1 (float = 70.9%):
| Scheme | Top-1 | Read |
|---|---|---|
| Asymmetric, per-layer | 0.1% | total collapse — the BN range problem |
| Symmetric, per-channel | 59.1% | per-channel alone recovers most of it |
| Asymmetric, per-channel | 70.3% | ~float — per-channel + affine ranges |
| Floating point | 70.9% | baseline |
ResNet and Inception barely move under 8-bit PTQ (within tenths of a percent) regardless of scheme — they have redundancy to spare. MobileNets, engineered to be lean, have no slack, so every quantization choice shows up in the accuracy. The more efficient your float model, the more carefully you must quantize it.
At 8 bits, QAT closes the gap to float to ~1% — and even makes per-tensor schemes viable (the network learns around the coarse scale). The real payoff is at 4 bits, where PTQ fails badly but QAT keeps the loss to 2–10% (worse for smaller nets). QAT is the tool when you need sub-8-bit.
Toggle the three findings. The dashed line is the float baseline; bars show top-1 accuracy. Watch the per-layer MobileNet bar sit at the floor, then leap up under per-channel — the single most important result in the paper.
Per-channel symmetric weight quantization — the paper's recommended weight scheme — in a dozen lines:
# Per-channel symmetric int8 quantization of a conv weight tensor # W shape: [C_out, C_in, K, K] — one scale per output channel import numpy as np def quantize_per_channel(W, bits=8): qmax = 2**(bits-1) - 1 # 127 for int8 # max abs per output channel -> one scale each amax = np.abs(W).reshape(W.shape[0], -1).max(axis=1) scale = amax / qmax # shape [C_out] s = scale.reshape(-1, 1, 1, 1) # broadcast over kernel Wq = np.clip(np.round(W / s), -qmax, qmax).astype(np.int8) return Wq, scale # store int8 + C_out floats def dequantize(Wq, scale): return Wq.astype(np.float32) * scale.reshape(-1, 1, 1, 1) # Compare granularities W = np.random.randn(64, 3, 3, 3).astype(np.float32) W[0] *= 40 # one channel with huge range (BN-like) Wq, s = quantize_per_channel(W) err_pc = np.abs(W - dequantize(Wq, s)).mean() # per-tensor: one global scale collapses the small channels s_pt = np.abs(W).max() / 127 err_pt = np.abs(W - np.clip(np.round(W/s_pt),-127,127)*s_pt).mean() print(err_pt, err_pc) # per-tensor error >> per-channel error
This whitepaper is the Rosetta Stone of inference quantization. The vocabulary it standardized — scale, zero-point, per-channel, symmetric/asymmetric, PTQ/QAT, fake-quant, STE, BN folding — is exactly the API surface of PyTorch quantization, TensorFlow Lite, ONNX Runtime, and TensorRT today.
| Symbol / term | Meaning | Plain analogy |
|---|---|---|
| Δ (scale) | (xmax−xmin)/(Nlevels−1) | how many real units per integer tick |
| z (zero-point) | integer that maps to real 0.0 | where "zero" sits on the ruler |
| xQ = clamp(round(x/Δ)+z) | quantize | snap to nearest tick, pin in range |
| x̂ = (xQ−z)Δ | de-quantize | read the real value back off the ruler |
| Symmetric (z=0) | range centred on 0 | balanced ruler — good for weights |
| Asymmetric (affine) | z≠0, fits skewed ranges | ruler slid to cover ReLU's one-sidedness |
| Per-tensor | one (Δ,z) for whole tensor | one ruler for everyone — cheap, fragile |
| Per-channel | one (Δ,z) per output kernel | a custom ruler per channel — rescues BN spread |
| PTQ | quantize after training | convert the finished model |
| QAT + fake-quant | simulate rounding in training | practice with the handicap you'll face |
| STE | backprop through round() as identity | a useful lie so gradients flow |
| BN fold | absorb BN into conv weights | one op at inference; source of channel spread |
Quantization is a linear map from a real range to 256 integer buckets, defined by a scale and a zero-point. Pick the scale per channel for weights (because BN folding fans each channel's range out) and per tensor for activations (to keep the matmul accumulator shared). Symmetric for weights (zero-centred, no costly cross-terms), affine for skewed activations. For 8 bits, post-training conversion with ~100 calibration batches is enough; for 4 bits, train with fake-quant nodes and push gradients through the rounding with a straight-through estimator while keeping float master weights. The result: 4× smaller, 2–3× faster, within ~1–2% of float.