HAWQ-V3 — Veanors

Chapter 0: The Problem

You quantize a ResNet to 8-bit integers. The weights are now int8, the activations are int8, the model file is 4× smaller. You ship it, expecting it to run roughly 4× faster on your edge chip. It runs barely faster. What happened?

Open the inference graph and you find the culprit. Almost every "quantized" framework actually does simulated quantization (also called fake quantization): the weights are stored as integers, but right before every convolution they are cast back to FP32, the convolution runs in floating point, batch-norm runs in floating point, the residual addition runs in floating point, and the result is cast back to integer. The integers are a storage format, not a compute format.

The hidden cost: Every layer pays a round-trip tax — int→float→int. The expensive FP32 ALUs are still doing the work. On a chip that has fast INT4/INT8 logic but slow or absent floating-point units (phones, microcontrollers, dedicated accelerators, NVIDIA Tensor Cores in INT mode), you get almost none of the speed you paid for in accuracy.

Why is it so hard to delete the floats?

Three things keep dragging floating point back into the graph, and they are exactly the three places prior "integer-only" work quietly kept FP32:

Operation	Why a float sneaks in	What HAWQ-V3 must fix
Rescaling after a matmul	The scale factor S_wS_h/S_a is an arbitrary real number	Force it to a dyadic number b/2^c → integer multiply + bit shift
Batch normalization	μ, σ are floats; quantizing them wrecks accuracy	Fold BN into the conv weights, quantize the folded result
Residual addition	Q(a+b) ≠ Q(a)+Q(b); naive int add corrupts the signal	Dyadic-scale both branches to a common scale, add in INT32

And there is a second, separate problem layered on top. Uniformly quantizing every layer to INT4 saves the most memory and runs fastest, but some layers are sensitive — slam them to 4 bits and accuracy collapses. You want a mixed-precision assignment: 4 bits where it's cheap, 8 bits where it matters. But with L layers and 2 choices each there are 2^L assignments. For ResNet50 that's 2⁵⁰ ≈ a quadrillion. You cannot grid-search that.

This paper answers two questions at once. (1) How do you run a network with literally zero floating-point ops — including BN and residuals? (2) Given a hard latency or memory budget, which layers should be 4-bit and which 8-bit, decided in under a second?

Why does storing weights as int8 often fail to deliver the expected speedup?

int8 weights take the same memory as float32 "Simulated" quantization casts back to FP32 for every conv/BN/add — the slow float ALUs still do the work int8 multiplication is slower than float multiplication on all hardware

Chapter 1: The Key Insight

The whole paper rests on one observation about a single number. After a quantized matmul, you have an INT32 accumulator, and to pass it to the next layer you must multiply by a real-valued scale and round. That scale is the only floating point that has to exist. Kill it, and the floats are gone everywhere.

HAWQ-V3 kills it by forcing every rescale factor to be a dyadic number: a rational of the form b/2^c, where b and c are integers. Multiplying by b/2^c is exactly "multiply by integer b, then shift right by c bits." No float, no division — and on a CPU, division costs roughly 10× a multiply, so removing it is a real win.

The insight: Real arbitrary scales need an FPU. Dyadic scales need only an integer multiplier and a barrel shifter — hardware every chip already has. So pick the nearest dyadic to each true scale, and the entire forward pass becomes integer arithmetic with INT32 accumulation.

The four ideas, stacked

Everything else in the paper is making that one trick work in the three places floats normally hide, plus a way to choose bit-widths:

1 · Dyadic rescale

Replace the float scale S_wS_h/S_a after every matmul with b/2^c → integer multiply + shift.

↓

2 · Integer BN folding

Merge σ into the weight scale and μ into an INT32 bias, so BN disappears into the conv.

↓

3 · Integer residuals

Dyadic-scale main and skip branches to one common scale, add in INT32 — no FP32 accumulation.

↓

4 · ILP bit-width search

An integer linear program minimizes Hessian sensitivity subject to size / BOPS / latency budgets — solves in <1s.

Chapters 2–3 build the dyadic core. Chapter 4 folds BN, Chapter 5 fixes residuals, Chapter 6 is the ILP showcase, and Chapters 7–8 are hardware and results. Let's start by being precise about what "quantize" even means here.

What is the single floating-point operation that HAWQ-V3 eliminates, and how?

The matmul itself — by replacing it with a lookup table The post-matmul rescale — by forcing the scale to a dyadic b/2^c (integer multiply + bit shift) The softmax — by approximating it with integers

Chapter 2: Uniform Quantization, Precisely

To map a real number r (a weight or an activation) onto an integer grid, the paper uses the standard affine rule:

Q(r) = Int(r / S) − Z

Three symbols, each with a job:

S — the scale, a real number, the size of one quantization step. Think of it as "how many real units does one integer tick represent." If your weights live in [−0.5, 0.5] and you have 8 bits (256 levels), S ≈ 1/256.
Z — the zero-point, an integer, chosen so the real value 0 maps to an exact integer. Critical because padding and ReLU produce a lot of exact zeros and you don't want zero to become noisy.
Int(·) — round-to-nearest then clamp into the representable range.

To go back: r ≈ S·(q + Z). The pair (S, Z) is the dictionary between the integer world and the real world.

Symmetric vs asymmetric — and why HAWQ-V3 mixes them. Weights are roughly centered at zero, so the paper uses symmetric quantization for weights (Z=0, range [−2^b-1, 2^b-1−1]) — this makes Z vanish from the matmul, removing a cross-term. Activations after ReLU are non-negative and lopsided, so it uses asymmetric quantization for them (Z ≠ 0) to not waste half the codes on negative values that never occur.

Worked example: quantize one weight

Take a single INT4 weight (4 bits, symmetric, signed range [−8, 7]). Suppose this channel's weights span [−0.4, 0.4], so the step is S = 0.4 / 7 ≈ 0.0571. Quantize r = 0.21:

q = Int(0.21 / 0.0571) = Int(3.67) = 4 → dequantized = 4 × 0.0571 = 0.2286

The error is 0.0186 — less than half a step, as it must be for round-to-nearest. Channel-wise quantization (a separate S per output channel) keeps that step small: a channel with tiny weights gets a tiny S and doesn't have its resolution stolen by a louder neighbor.

The Quantization GridInteractive

Drag the bit-width and range. Watch how few INT4 grid lines there are versus INT8 — and how a single fat weight forces a coarse step that blurs every small weight near zero.

Bits: 4 Max |w|: 0.50

Why does HAWQ-V3 use symmetric quantization for weights but asymmetric for activations?

Symmetric is always more accurate Activations are easier to quantize than weights Weights center on zero (Z=0 removes a matmul cross-term); post-ReLU activations are non-negative & lopsided, so Z≠0 avoids wasting codes

Chapter 3: The Dyadic Rescale

Now the core mechanism. A quantized layer takes activation h = S_hq_h and weight W = S_wq_w. The convolution multiplies the integers and accumulates in INT32:

a = S_wS_h (q_w ∗ q_h)

Here q_w ∗ q_h is a pure INT4×INT4 multiply-accumulate landing in an INT32 register — fast, no floats. But the next layer wants its input as INT4, so we must requantize: divide by the next activation's scale S_a and round.

q_a = Int( a / S_a ) = Int( (S_wS_h / S_a) · (q_w ∗ q_h) )

That factor S_wS_h/S_a is a product/quotient of three arbitrary reals — it is itself an arbitrary real. Multiplying an INT32 by it naively needs an FPU. This is the float we kill.

Dyadic numbers. Force the scale to the nearest number of the form b/2^c with integers b, c. Then "multiply by b/2^c" = "multiply INT32 by b (still integer), then arithmetic-shift right by c bits." Both are single hardware instructions. No division, no float. The function DN(·) finds (b, c) for a given real scale.

How DN finds b and c — with numbers

Say the true scale is S_wS_h/S_a = 0.1875. Pick a precision, e.g. c = 8 bits of fraction. Then:

b = round(0.1875 × 2⁸) = round(48.0) = 48, so b/2^c = 48/256 = 0.1875

Exact here. For a messier scale like 0.137, b = round(0.137×256) = 35, giving 35/256 = 0.13672 — an error of 0.0003, negligible after rounding to an integer code. At runtime: q_a = (acc * 35) >> 8. One multiply, one shift. That single line replaces a floating-point rescale on every layer of the network.

# Dyadic approximation of a real scale, then integer-only rescale def dyadic(scale, c=16): # c = bits of fractional precision b = round(scale * (1 << c)) # nearest integer numerator return b, c # scale ≈ b / 2**c def requantize(acc_int32, Sw, Sh, Sa): b, c = dyadic(Sw * Sh / Sa) # done ONCE, offline — scales are static return (acc_int32 * b) >> c # integer mul + arithmetic shift. no float, no div.

Float scale → nearest dyadic b/2⁰Interactive

Set any real scale and the fractional bit-budget c. See the dyadic ladder b/2⁰ snap to it, and the rounding error shrink as c grows.

Scale: 0.137 c (bits): 6

What does "multiply by the dyadic number b/2^c" compile down to in hardware?

An integer multiply by b, then an arithmetic right-shift by c bits A floating-point multiply followed by a round An integer division by 2^c

Chapter 4: Batch-Norm Folding, the Integer Way

Batch norm is the second place floats hide. At inference, BN is an affine map with fixed statistics:

BN(a) = γ · (a − μ) / σ + β

where μ, σ are the running mean/std and γ, β are the learned scale/shift. Prior quantizers keep these in FP32 because quantizing μ, σ directly is brutal — the std appears in a division, and small errors blow up. But FP32 BN is illegal on integer-only hardware.

The fold: BN dissolves into the convolution

The trick: a convolution followed by BN is itself an affine map, so the two collapse into one conv with adjusted weights and bias. For a conv weight W and bias, folding gives:

Ŵ = (γ / σ) · W, b̂ = β − γμ/σ (+ folded conv bias)

Now there is no separate BN layer at all. HAWQ-V3 quantizes the folded result, and routes the float pieces into integer slots:

σ → weight scale

The 1/σ factor is absorbed into the per-channel weight scale S_w. It never becomes a runtime float — it's baked into the quantized weights offline.

↓

μ → INT32 bias

The folded bias b̂ is quantized to an INT32 value (b₃₂) and simply added into the INT32 accumulator — biases are cheap, so 32 bits costs nothing.

Why INT32 for the bias, INT4 for weights? The bias is added once per output, not multiplied in the inner loop, so its bit-width barely affects speed or memory. Spending 32 bits on it preserves accuracy for free. The weights, multiplied millions of times, must stay at 4 bits to get the speedup. Match each tensor's bit-width to its role in the data flow.

The subtle scheduling bug HAWQ-V3 fixes

The earlier integer-only work (Jacob et al., 2018) folded BN while still updating the running statistics during training, which forced computing each conv twice per step (once without BN to get stats, once with). HAWQ-V3 found this both wasteful and accuracy-hurting. Their fix is simpler:

Two-phase folding. Train for several epochs with Conv and BN unfolded (let BN statistics settle). Then freeze the running μ, σ, fold once, and continue. This avoids the double-conv and gives +2.68% Top-1 on ResNet50 INT8 over Jacob et al. — the BN fold alone is most of that gap.

Conv + BN → one folded integer convInteractive

Click a stage to trace where each BN parameter goes. Drag σ to see the weight scale absorb it; drag μ to see the INT32 bias shift.

σ: 1.00 μ: 0.00

In folded integer BN, where do the BN std (σ) and mean (μ) end up?

Both stay in FP32 and run separately σ is absorbed into the per-channel weight scale; μ becomes an INT32 bias added in the accumulator σ becomes the zero-point and μ becomes the dyadic shift

Chapter 5: Residual Connections in Pure Integers

The third hiding place is the residual add in a ResNet block: out = main + skip. It looks trivial — just add two tensors. Prior work added them in FP32 and re-quantized. HAWQ-V3 shows this is a disaster on integer hardware, and the reason is a one-line fact:

Q(a + b) ≠ Q(a) + Q(b)

Quantization is not linear. If you accumulate the residual in float and then quantize, you get a different answer than if you accumulate the already-quantized integers. So a model trained/calibrated assuming FP32 residual accumulation, when actually deployed with integer accumulation, produces a different signal. The paper measures the mismatch at over 90% of feature-map values for low-precision residuals. The signal is gone.

The common misconception: "Keeping the residual in FP32 is fine, it's just an add." No. Because Q is nonlinear, an FP32 residual is fundamentally incompatible with integer deployment — you cannot just swap it out at the end. The accumulation domain is part of the model's definition.

The fix: align both branches to one scale, add in INT32

The two branches arrive with different scales: main is m = S_mq_m, skip is r = S_rq_r. You can't add q_m and q_r directly — they're in different units. Rescale both into the output scale S_a using two dyadic factors, then add the integers:

q_a = DN(S_m/S_a) · q_m + DN(S_r/S_a) · q_r

Each DN(·) is a precomputed (b, c) pair — an integer multiply and shift. The addition is INT32. Every scale is statically known, so all the (b, c) pairs are baked in offline. The residual is now genuinely integer-only.

# Integer-only residual add. Sm, Sr, Sa are static scales known offline. bm, cm = dyadic(Sm / Sa) br, cr = dyadic(Sr / Sa) def residual_add(qm_int32, qr_int32): main = (qm_int32 * bm) >> cm # main branch rescaled to output units skip = (qr_int32 * br) >> cr # skip branch rescaled to the SAME units return main + skip # INT32 add — both now in S_a units

Why FP32-residual deployment corrupts the signalInteractive

Same two branches, two ways. Left: quantize-then-add (HAWQ-V3, integer). Right: add-in-float-then-quantize (prior work). Drag the bit-width down and watch the mismatch (red) explode at low precision.

Bits: 3

Why can't a quantized model keep its residual add in FP32 and still deploy on integer hardware?

FP32 adds are too slow Q is nonlinear, so Q(a+b)≠Q(a)+Q(b) — an FP32-accumulated residual gives a different signal than integer accumulation (90%+ mismatch) Residual connections aren't needed in quantized networks

Chapter 6: Mixed Precision as an Integer Program · SHOWCASE

Now the second half of the paper. We can run any layer at INT4 or INT8. INT4 is smaller and faster but more lossy. Which layers get which? With L layers and 2 choices, the search space is 2^L — intractable. We need to score each layer's sensitivity and then optimize under a budget.

Sensitivity = Hessian sharpness

How much does quantizing layer i to b bits hurt the loss? The answer is the curvature of the loss in that layer's directions. A layer sitting in a flat valley tolerates the perturbation — the loss barely moves. A layer in a sharp ravine is sensitive — a small weight nudge spikes the loss. The paper measures this with the Hessian (second derivative): per-layer perturbation

Ω_i^(b) ≈ (top Hessian eigenvalue of layer i) · ‖quantization error of layer i at b bits‖²

Big eigenvalue (sharp) × big rounding error (few bits) = high cost. Assuming layers perturb independently, total damage is the sum — so we only need BL sensitivity probes, not 2^L.

The flat-vs-sharp intuition is the whole sensitivity story. Quantizing = jiggling the weights by up to half a step. In a flat basin the loss surface is forgiving; in a sharp one the same jiggle launches you up a wall. Give the sharp layers their 8 bits; spend 4 bits on the flat ones.

The ILP: minimize sensitivity under a hard budget

Let b_i ∈ {4, 8}. Minimize total sensitivity subject to whatever the hardware demands:

min_{{b_i}} Σ_i Ω_i^(b_i) s.t. Σ_i M_i^(b_i) ≤ Size, Σ_i G_i^(b_i) ≤ BOPS, Σ_i Q_i^(b_i) ≤ Latency

M_i — memory of layer i at b_i bits (params × bits/8).
G_i = b_w·b_a·MAC_i — BOPS, Bit Operations: a hardware-agnostic compute proxy. INT4×INT4 is 4× cheaper than INT8×INT8 (16 vs 64).
Q_i — the directly measured latency of layer i at b_i bits on the target chip. This is what makes it hardware-aware.

Because everything is linear in binary choices, an off-the-shelf ILP solver (PuLP) cracks it in under one second. The RL-based HAQ took tens of hours to do the same job — and wasn't hardware-aware.

Why latency ≠ BOPS ≠ size. Some layers don't speed up at all from INT4 (memory-bound, or the kernel underutilizes the cores); some superlinearly speed up. A small model can be slow; FLOPs ignore cache misses and bandwidth. So the paper plugs in measured per-layer latency. The ILP then refuses to 4-bit a layer that gives no speedup but costs accuracy.

ILP Bit-Width Solver — tighten the budget, watch it chooseInteractive Showcase

8 layers, each with a sensitivity (sharpness) and a measured INT4 speedup. Drag the latency budget: the solver greedily 4-bits the layers that are least sensitive per unit of latency saved. Toggle "hardware-aware" off to see it ignore measured speedups and make worse choices.

Latency budget: 80%

What makes HAWQ-V3's ILP "hardware-aware," and why does it matter?

It runs on a GPU It uses more bits everywhere It constrains on directly-measured per-layer latency, so it won't 4-bit a layer that gives no speedup but costs accuracy

Chapter 7: Hardware Deployment

A quantization paper that never touches silicon is a paper of promises. HAWQ-V3 actually deploys, on NVIDIA T4 Turing Tensor Cores, which natively support both INT8 and INT4 matmul. The catch: the only API is the low-level WMMA micro-kernel, and no compiler existed that could map an INT4 network onto it. So the authors extended Apache TVM — the first framework to add INT4 support to TVM.

The 4-bit packing problem. Machines are byte-addressable; you can't store a single 4-bit value cleanly. TVM packs eight INT4 values into one INT32 word and moves data in those chunks. All graph-level passes (memory planning, constant folding, operator fusion) operate on the packed INT32 representation, and only the final codegen unpacks to the INT4 lanes the Tensor Cores want.

The verification discipline that makes the numbers trustworthy

The training happens in PyTorch (with fake quantization); the deployment happens in TVM. A tiny mismatch between the two could silently break the model and never show up in the PyTorch validation accuracy. So the authors forced layer-by-layer, machine-precision agreement between PyTorch and TVM for every stage, then confirmed the final Top-1 with the actual integer-only hardware execution — not a simulation.

Concept → realization, made literal. Appendix G shows that deploying a model with FP32 residuals on integer hardware accumulates feature-map error until 90%+ of values are wrong by the deep layers. The integer-only residual of Chapter 5 is exactly what closes that gap. The architecture choice and the hardware result are the same fact seen twice.

Packing 8×INT4 into one INT32 wordInteractive

Drag the lanes. Eight 4-bit values share a single 32-bit machine word so the hardware can move and multiply them as a chunk on Tensor Cores.

Lane: 0 Value: 11

Why does TVM pack eight INT4 values into a single INT32 word?

Machines are byte-addressable, so individual 4-bit storage/movement is inefficient; packing lets data move and compute as 32-bit chunks INT4 multiplication requires 32-bit inputs To make the model file larger

Chapter 8: The Results

The headline: integer-only, no-FP32-anywhere quantization that matches or beats prior work which cheated by keeping BN and residuals in float — while running measurably faster on real hardware.

Accuracy (ImageNet Top-1)

Model	FP32 baseline	HAWQ-V3 INT8	INT4/8 mixed	Uniform INT4
ResNet18	71.47%	71.56%	70.22% (70.38 +distill)	67.50% (68.45 +distill)
ResNet50	77.72%	77.58%	75.39% (76.73 +distill)	73.70% (74.24 +distill)
InceptionV3	78.88%	78.76%	74.65% (74.72 +distill)	70.39%

Read the ResNet50 row carefully. INT8 lands at 77.58% — just 0.14% below FP32 and +2.68% over the prior integer-only work (Jacob et al.), almost entirely thanks to the two-phase BN fold from Chapter 4. The INT8 model is also 4× smaller (97.8 MB → 24.5 MB).

The mixed-precision payoff. For ResNet50, mixed INT4/8 with distillation reaches 76.73% while cutting INT8 latency by 23%. Uniform INT4 is fastest of all but drops to 73.70% — the ILP buys back ~3% accuracy by spending 8 bits only on the handful of sharp layers it identified. That is the flat-vs-sharp principle paying rent.

Speed (T4 GPU, the part most papers skip)

Uniform INT4 vs uniform INT8: 1.45× average speedup (ResNet50).
Per-layer INT4 vs INT8: up to 1.47× — but crucially not uniform across layers, which is exactly why the latency-constrained ILP beats a naive "INT4 everywhere."
Mixed INT4/8: 23% lower latency than INT8 at near-INT8 accuracy.

The accuracy ↔ latency Pareto frontier (ResNet50)Interactive

Slide along the frontier the ILP traces out by varying its latency budget. Each point is a real configuration; see what accuracy each precision recipe buys.

Latency budget → precision:

HAWQ-V3 INT8 beats prior integer-only work by +2.68% on ResNet50. What is the main reason?

A bigger network The two-phase BN folding (settle stats, then freeze and fold once) instead of folding while updating statistics Using floating point for some layers

Chapter 9: Connections & Cheat Sheet

HAWQ-V3 sits at the end of a lineage: Jacob et al. (2018) gave us INT8 integer-only inference and dyadic rescaling; HAWQ / HAWQ-V2 gave us Hessian-based sensitivity for mixed precision; HAWQ-V3 fuses both, pushes to INT4, makes BN and residuals truly integer, and adds a fast hardware-aware ILP with a real TVM deployment.

The whole forward pass, in one breath

INT4 input

activations packed 8-per-INT32

→

INT4×INT4 conv

folded BN weights, accumulate in INT32

→

+ INT32 bias

folded BN mean

→

dyadic rescale

(acc × b) >> c

→

INT32 residual

both branches DN-scaled, added

→

INT4 out

to next layer

Cheat sheet

Symbol / term	Meaning	Where
Q(r)=Int(r/S)−Z	affine quantizer; S=step, Z=zero-point	Ch 2
b/2^c = DN(scale)	dyadic number: integer multiply b + right-shift c, no float/division	Ch 3
q_a=Int((S_wS_h/S_a)·q_w∗q_h)	requantize after matmul via dyadic scale	Ch 3
Ŵ=(γ/σ)W, b̂=β−γμ/σ	BN fold; σ→weight scale, μ→INT32 bias	Ch 4
q_a=DN(S_m/S_a)q_m+DN(S_r/S_a)q_r	integer residual (Q is nonlinear, must add in int)	Ch 5
Ω_i^(b)	Hessian sensitivity = sharpness × quant-error²	Ch 6
G_i=b_wb_aMAC_i	BOPS: bit-operation compute proxy	Ch 6
min ΣΩ s.t. size/BOPS/latency	the ILP, solved in <1s with PuLP	Ch 6

Related lessons

If you want the foundations under this paper: a Quantization gleam for the affine grid and calibration, a BatchNorm lesson for the fold algebra, and the Hessian / loss-landscape material for why sharpness predicts sensitivity. Forward from here: Parallel/Post-training quantization, GPTQ/AWQ for LLM-era weight-only quantization, and QLoRA for fine-tuning quantized models.

What the paper quietly assumes. (1) Layer sensitivities are independent — a convenient fiction that makes BL probes enough, but cross-layer interactions exist. (2) Per-layer latency is measured once on one chip; the ILP's answer is chip-specific. (3) It targets CNNs (ResNet/Inception); transformers add layernorm and attention with their own quantization headaches not covered here. (4) Generation-time error from the residual fix is shown for vision; the dyadic philosophy generalizes, but each new op (softmax, gelu, layernorm) needs its own integer-only treatment.

In one sentence, what are HAWQ-V3's two coupled contributions?

Faster training and a new optimizer Truly integer-only inference (dyadic rescale + integer BN/residuals) AND a fast hardware-aware ILP for mixed-precision bit assignment A larger model and a new dataset

"...the entire computational graph is performed only with integer multiplication, addition, and bit shifting, without any floating point operations or even integer division."

— Yao, Dong et al., HAWQ-V3 (2021)

HAWQ-V3: Dyadic NeuralNetwork Quantization