ZeroQ: Mixed-Precision Quantization
With No Data At All

Chapter 1: The Key Insight

Here is the whole paper in one sentence. The model already contains a compressed description of its training data — in its batch-norm statistics — so we can synthesize fake images that "look like" the real data to the network, and use those fake images to do everything quantization needs.

Every batch-norm layer stored a running mean μ_i and variance σ_i² of its input activations, averaged over the real training set during training. Those numbers say: "when real data flows through me, layer i's activations have this mean and this spread." That is a statistical fingerprint of the data, frozen into the weights.

Three moves, in order

What flows where

Notice what is being optimized at each stage, because it is easy to confuse:

• Stage 1 optimizes the input pixels x^r. The network weights are frozen. Gradients flow into the image, not the model.
• Stages 2–3 never optimize anything by gradient descent on the model. They measure (sensitivity) and search (bit assignment). The original weights are never updated — that is what "no fine-tuning" means.

The entire procedure for ResNet50 takes under 30 seconds on an 8-GPU box: ~3s to distill data, ~12s to measure sensitivity across all layers and bit-widths, ~14s to run the Pareto search. That is 0.5% of a single training epoch. We build each stage from scratch next.

Chapter 2: Quantization, Concretely

Before the clever part, let's be precise about what "quantize a tensor to k bits" actually means, because the rest of the paper rests on it. ZeroQ uses asymmetric uniform quantization. "Uniform" means the integer levels are evenly spaced; "asymmetric" means the range need not be centered at zero.

Take any tensor (a weight matrix or a layer's activations). First clip it to a range [a, b]. Then chop that range into 2^k−1 equal intervals of width

Each real value v maps to the nearest grid point, stored as an unsigned integer in [0, 2^k−1]:

Here v is the original 32-bit value, a, b are the clipping bounds, k is the bit-width (e.g. 4), Δ is the step size (the gap between adjacent representable values), q is the stored integer, and v̂ is the dequantized approximation. The error |v − v̂| is at most Δ/2.

Worked example: 4-bit weights

Suppose a weight tensor lives in [a, b] = [−0.8, 1.2] and we quantize to k = 4 bits. Then there are 2⁴−1 = 15 intervals:

A weight v = 0.37 maps to q = round((0.37 + 0.8)/0.1333) = round(8.78) = 9, which dequantizes to v̂ = −0.8 + 9·0.1333 = 0.40. The error is |0.37 − 0.40| = 0.03 — under Δ/2 = 0.067. The whole tensor now needs 4 bits/value instead of 32: an 8× shrink.

Drag the bit-width below and watch a smooth signal get snapped onto the grid. Notice how few levels 2-bit gives you, and how the error explodes — this is exactly why some layers cannot go that low.

Chapter 3: Distilled Data

This is the heart of the paper. We need synthetic input that the network treats as if it were real training data — without ever seeing real data. The signal we exploit is sitting in every batch-norm layer.

What batch norm secretly stored

During training, the i-th BN layer watched activations z_i (the output of conv layer i) and kept a running average of their mean and variance. After training, the model file contains, for every BN layer:

• μ_i — the mean of layer i's activations, averaged over the real training set.
• σ_i — the standard deviation of layer i's activations, same.

These were computed from millions of real images. They are a low-dimensional summary of "what the data looks like at every depth of the network."

The distillation objective

We start with a batch of random noise x^r (drawn from a standard Gaussian) and treat the pixels themselves as the variables to optimize. We push x^r through the frozen network and read off the actual mean μ̃_i^r and std σ̃_i^r of the activations our noise produces at each layer. Then we minimize the gap between those and the stored values:

Symbol by symbol: x^r is the reconstructed (distilled) image batch we are solving for; μ̃_i^r, σ̃_i^r are the mean/std of x^r's activations measured at layer i right now in the forward pass; μ_i, σ_i are the BN stats stored in the model; L is the number of BN layers. The i=0 term pins the input itself to zero mean, unit variance (standard preprocessing).

Algorithm 1, in plain steps

Below: watch noise turn into structure as the statistic-matching loss falls. Each "iteration" nudges the synthetic batch's activation statistics toward the stored BN targets. The loss curve is the ‖μ̃−μ‖²+‖σ̃−σ‖² sum dropping.

Chapter 4: Sensitivity — Which Layers Can Take the Hit?

Now we have synthetic data that the network experiences as real. Time for job two: figure out which layers can be aggressively quantized and which must stay high-precision. Not all layers are equal — quantizing a fragile early layer can destroy accuracy, while a robust layer shrugs it off.

Defining sensitivity

The metric is beautifully direct. Quantize only layer i to k bits, leave everything else at full precision, and ask: how much did the model's output change? Measure that change as KL-divergence between the full-precision output distribution and the partially-quantized one, averaged over the Distilled Data:

where M(θ; x_j) is the full-precision model's softmax output on Distilled image x_j; M(θ̃_i(k); x_j) is the same model with only layer i quantized to k bits; KL measures how far the quantized output distribution drifted from the original; N_dist is the number of Distilled images. Small Ω_i(k) ⇒ layer i barely cares about k-bit quantization ⇒ safe to compress.

The experiment that justifies everything

The paper's pivotal plot (Figure 2, left): measure each ResNet50 layer's sensitivity to 4-bit quantization three ways — using real training data, using Gaussian noise, and using Distilled Data. If Distilled Data is a good stand-in, its sensitivity curve should track the real-data curve. It does. Gaussian noise does not — for the first few layers its ordering is literally reversed.

Toggle the data source below and watch the per-layer sensitivity curves. Notice how the Distilled curve hugs the real-data curve, while Gaussian diverges — especially in early layers where the ordering flips.

Chapter 5: The Pareto Frontier (Showcase)

We can now rank layers. But mixed-precision means choosing a bit-width for every layer, and the choices interact through a budget. For ResNet50 with bit options {2,4,8} over ~50 layers, the search space is 3⁵⁰ ≈ 7.2×10²³ configurations. You cannot brute-force that.

The key simplifying assumption

ZeroQ assumes layer sensitivities are independent: the cost of quantizing layer i to k bits is Ω_i(k), regardless of what the other layers do. This is an approximation (errors can compound), but it makes the total perturbation a simple sum, and that is what makes the search cheap.

The optimization: for a target model size S_target, pick the bit assignment {k_i} that minimizes total sensitivity while fitting the budget:

k_i is the bit-width chosen for layer i; Ω_i(k_i) is that layer's sensitivity at that bit-width (from Ch 4); P_i is layer i's parameter count, so P_i·k_i is its size in bits; S_target is the size budget. Total cost of building the sensitivity table is only O(m·L) — m bit options times L layers — because each Ω_i(k) is measured independently.

Worked mini-example: 3 layers

Suppose three layers with parameter counts P = [4, 2, 1] (millions), bit options {2,4,8}, and a sensitivity table (lower = more robust):

Budget S_target = 30 Mbit. All-8-bit costs (4+2+1)·8 = 56 Mbit — too big. The optimizer wants to spend its precious bits where they reduce sensitivity most. The big, sensitive L1 is where high bits pay off; the tiny robust L3 can go to 2-bit nearly free. A good assignment: L1=8 (32 Mbit)… too big. Try L1=4, L2=4, L3=2 → 16+8+2 = 26 Mbit ≤ 30, total Ω = 0.20+0.05+0.08 = 0.33. The interactive sim below runs this exact search live — drag the budget and watch the assignment snap to the lowest-sensitivity point that fits.

Chapter 6: The Full Pipeline

Let's assemble the three stages into the end-to-end ZeroQ flow and trace exactly what data crosses each boundary. This is the diagram you should be able to redraw from memory.

The boundaries, spelled out

One subtlety the paper buries

The independence assumption in the Pareto step is not exactly true — quantization errors do interact across layers. The appendix relaxes this without an exponential blowup, but the headline method uses the independent approximation and it works because the errors are small enough that the additive surrogate ranks configurations correctly. The lesson: a "wrong" model that ranks options correctly is good enough for a search.

Chapter 7: The Core Idea as Code

Here is the heart of ZeroQ — Distilled Data generation — as runnable PyTorch. The trick that makes it tiny: hook the BN layers to capture their inputs, compare measured stats to the stored running stats, and backprop into the image.

import torch, torch.nn as nn

def distill_data(model, batch=32, iters=500, lr=0.5, img=(3,224,224), dev='cuda'):
    model.eval().to(dev)

    # 1) Grab the stored BN targets and register hooks that capture inputs.
    bn_targets, captured = [], []
    def make_hook(slot):
        def hook(mod, inp, out):
            captured[slot] = inp[0]          # activations entering this BN layer
        return hook
    for m in model.modules():
        if isinstance(m, nn.BatchNorm2d):
            bn_targets.append((m.running_mean.detach(),
                               m.running_var.detach().sqrt()))   # (mu_i, sigma_i)
    captured = [None] * len(bn_targets)
    bns = [m for m in model.modules() if isinstance(m, nn.BatchNorm2d)]
    for i, m in enumerate(bns):
        m.register_forward_hook(make_hook(i))

    # 2) The IMAGE is the variable we optimize; weights stay frozen.
    x = torch.randn(batch, *img, device=dev, requires_grad=True)
    opt = torch.optim.Adam([x], lr=lr)

    for it in range(iters):
        opt.zero_grad()
        model(x)                                  # forward fills `captured`
        loss = 0.0
        # input layer: pin to zero-mean / unit-var (standard preprocessing, i=0)
        loss += (x.mean()**2) + (x.var() - 1.0)**2
        for i, a in enumerate(captured):
            mu_t, sig_t = bn_targets[i]
            mu_r  = a.mean(dim=(0, 2, 3))         # per-channel mean of synthetic activations
            sig_r = a.var(dim=(0, 2, 3)).sqrt()   # per-channel std
            loss += ((mu_r - mu_t)**2).sum() + ((sig_r - sig_t)**2).sum()
        loss.backward()                           # gradient flows into x, not weights
        opt.step()
    return x.detach()                             # the Distilled Data

Read the three load-bearing lines. requires_grad=True on x (not the weights) makes the image the optimization variable. The per-channel mean/std over dims (0,2,3) reduces a [N,C,H,W] activation to a [C] statistic, matching the shape of the stored running_mean. And loss.backward() sends gradients all the way back to the pixels.

Sensitivity is then a short loop: for each layer i and bit k, quantize just that layer, run the Distilled Data through both versions, and accumulate KL(full_softmax, quant_softmax). The Pareto step is a dynamic program over the resulting table.

Chapter 8: Experiments

Does data-free actually compete with data-hungry methods? The headline numbers say yes — often beating methods that do use real data and fine-tuning.

The result that surprised people

On MobileNetV2, a notoriously quantization-fragile compact model, ZeroQ at W8A8 lands within 0.12% of the FP32 baseline and beats DFQ (the previous data-free SOTA) by 1.71% — with no data and no fine-tuning. On ResNet18 at the brutal W4A4 setting, ZeroQ hits 69.05%, beating DFC's fine-tuned 68.06%. Read that again: no-data-no-finetune beat with-data-with-finetune.

The numbers, exact

ZeroQ† uses percentile clipping (instead of min/max) to set [a,b] — helpful at low precision because it ignores activation outliers. It even extends beyond classification: on COCO object detection with RetinaNet, ZeroQ at W8A8 matched the 36.4 mAP baseline exactly, and DFQ's weight-equalization trick doesn't obviously transfer to detection.

Chapter 9: Connections & Cheat Sheet

ZeroQ sits at the intersection of three ideas you've probably met separately: post-training quantization, mixed-precision (Hessian/HAWQ-style) bit allocation, and feature-visualization-by-inversion. Its contribution was gluing them so that no real data is needed at any point.

Where it fits

Related lessons

• SQuant: data-free quantization — a later data-free method; compare its calibration approach to Distilled Data.
• Knowledge Distillation — "distilling" here means matching statistics, not soft labels; contrast the two senses.
• Model Compression (Gleam) — the broader landscape: pruning, quantization, distillation.
• TinyML Quantization I — the quantization fundamentals this paper builds on.

Cheat sheet — every key equation