Quantize and fine-tune a neural network to 4-bit precision without ever touching the training data — by mining synthetic images directly out of the BatchNorm statistics already baked into the trained model.
You have a trained network — say a ResNet-18 classifier — sitting in float32. You want to ship it to a phone, an edge accelerator, a camera. The accelerator runs 4-bit integers, not 32-bit floats. So you must quantize: replace every weight and activation with a low-precision integer approximation. Done well, this is an 8× memory cut and a huge speedup. Done blindly, it destroys accuracy.
Here is the catch that nobody warns you about. To quantize well, you need the training data. You need it to calibrate (measure how big each layer's activations get, so you can set the integer scale), and under aggressive compression you need it to fine-tune (nudge the weights to recover from the rounding error). Both steps feed real data through the network.
This is not a contrived scenario. A hospital trains a tumor classifier and hands you the weights to deploy on a portable scanner — they will never hand you the patient images. A bank trains a fraud model; the transactions are sealed. A vendor sells you a pretrained face model with the faces stripped out by policy. In every case you receive a model, not a dataset. This decoupling of training and deployment is exactly what the paper calls the data-free regime.
The obvious escape — "train a GAN to imitate the data" — fails on its own terms: training a generative model also needs the real data. You would be solving the problem by assuming it away.
So the only resource is the trained network. The whole paper is a bet that this is enough — that a network which classifies cats has, somewhere inside its weights and its normalization layers, a usable fingerprint of what cats look like. The job is to extract that fingerprint and turn it into synthetic data good enough to calibrate and fine-tune a quantized copy.
| Compression step | Needs data for… | Severity if skipped |
|---|---|---|
| 8-bit calibration | Activation range per layer | Mild — often survives |
| 4-bit calibration | Tighter ranges, sensitive layers | Large accuracy drop |
| 4-bit / 2-bit fine-tune (KD) | Recover from rounding error | Catastrophic — model unusable |
That bottom row is the real target. Calibration alone gets you to ~8 bits. Pushing to 4 bits and below requires fine-tuning — and that is where the data-free constraint bites hardest, because you must run thousands of gradient steps through the network with some data standing in for the real thing.
Where, inside a trained network, is the data hiding? The paper's answer is almost embarrassingly specific: in the BatchNorm layers.
Recall what BatchNorm does during training. For every channel of every BN layer, it keeps a running estimate of the mean and variance of the activations flowing through it — averaged over all the real training batches it ever saw. These running statistics, μ̂ and σ̂, are then frozen and shipped inside the model so inference can normalize consistently.
If the model "remembers" that the activations of layer 7, channel 3 should average 0.4 with standard deviation 0.9 (because that is what real cats produced), then we can run the search in reverse: start from random noise and optimize the input pixels until the network's internal activations reproduce those remembered statistics at every layer.
An input that makes every BN layer light up with its real, training-time statistics must — by the network's own internal logic — look like the data the network was trained on. Not pixel-for-pixel, but statistically indistinguishable from the network's point of view, which is all calibration and distillation care about.
The trained model is run forward to read its internal statistics, and gradients flow backward into the input pixels (not the weights — the weights are frozen). The model is simultaneously the data source and the judge.
Before we synthesize anything, let us make the need concrete. Why does quantization want data at all? Build the intuition first, then we will know exactly what our synthetic data has to reproduce.
Uniform quantization maps a real value x in some range [−r, r] onto a small grid of integers. With b bits you have 2b levels. The scale (step size) is Δ = 2r / (2b − 1). Picking r is the whole game.
Set r too large and most of your precious 16 (for 4-bit) levels sit in empty range that the data never visits — you waste resolution. Set r too small and you clip the loud activations, throwing away their magnitude. There is a sweet spot, and finding it requires knowing how big the activations actually get. That knowledge comes from pushing data through and watching.
That watching step is calibration: run a small batch, record the min/max (or a smoothed version) of each layer's activations, and set r from it. The paper uses a smoothed absolute dynamic range — running estimates of mean abs-min/max over chunks of 16 samples — over just 1% of the training set. The point of synthetic data is to make that step possible without the 1%.
The grey curve is a real activation distribution. The orange staircase is the quantization grid for your chosen range and bit-width. Too wide and the steps yawn over empty space; too narrow and the tails get clipped flat. Watch the reconstruction-error readout find its minimum.
4w4a = 4-bit weights, 4-bit activations; 2w4a = 2-bit weights, 4-bit activations. Weights use a per-channel scale; activations a per-tensor scale. BN layers and the first/last layers are often kept at higher precision because they are most sensitive. The compression level is deliberately set so that calibration or fine-tuning is genuinely needed — otherwise the data-free question would be trivial.Start with the dumbest thing that could possibly work. We know (or can be told) the first two moments of the raw input — the mean and std of the pixel values the dataset was normalized with. So just sample noise from a Gaussian with that mean and std and call it data.
This generator (denoted G) is gloriously cheap: no optimization, no forward passes, infinite samples on demand. And — surprise — for mild compression it is good enough to calibrate. If all you need is a rough activation range, random noise with the right overall scale produces activations in roughly the right ballpark.
Now try to fine-tune with Gaussian noise and the wheels come off. The problem is precisely the BN statistics from Chapter 1. Random noise does not preserve the spatial structure of real images, so the activations it induces have the wrong internal statistics — far from what real cats produced.
If you let gradient descent fine-tune on this noise, the BN layers will happily re-estimate their running mean/variance to fit the noise. You have just overwritten the network's memory of the real data with garbage. The moment you switch back to real test images, those corrupted BN stats mis-normalize everything and accuracy collapses.
Toggle to compare the internal activation distribution at a deep layer. Real data (matching the stored BN target, dashed line) sits where the network expects; Gaussian noise drifts off — wrong mean, wrong spread. Fine-tuning on the right panel is what corrupts BN.
If random pixels give wrong statistics, maybe we should ask the network to draw for us. This is the old "Inceptionism" / DeepDream idea, recast for our purpose. Pick a target class — say "goldfish" — and optimize the input pixels to maximize that class's logit. Whatever image makes the network shout "goldfish!" loudest is, in some sense, the network's idea of a goldfish.
The paper's Inception scheme (denoted I) maximizes a single chosen output neuron, but with a twist: instead of the raw logit it uses a decaying exponential of it, the Inception loss:
Why the exponential rather than just −logit? Plain logit maximization runs away — the optimizer drives the logit to infinity and produces hallucinatory high-frequency garbage. The e−logit/scale form exponentially decays the gradient as the logit grows: once the class is confidently present, pushing harder buys almost nothing, so the optimizer stops inflating the output. The scale (temperature) tunes how confident is "confident enough."
Logit maximization alone produces adversarial-looking static — pixels that fool the network but look nothing like a photo. So the scheme adds an image prior: real images are smooth (nearby pixels correlate). The prior applies a Gaussian blur to the current image and penalizes the difference from the blurred version, gently pulling neighboring pixels together.
The curve is the loss as a function of the target logit. Plain −logit (orange) keeps a strong negative slope forever — the optimizer never stops inflating. The exponential (teal) flattens once the logit is high, so the gradient vanishes and the image settles. Raise scale to widen the "good enough" zone.
We have diagnosed both failures. Gaussian noise: wrong internal statistics. Inception: constrains input and output but leaves the middle to chance. The cure writes itself — directly optimize the input so that every layer's internal statistics match what the trained model remembers. This is the paper's central contribution: the BN-Stats objective (denoted BNS).
We saw in Chapter 1 that each BN layer l stores a target mean μ̂l and std σ̂l per channel. The plan: start from random pixels, push them through the frozen model, measure the induced per-channel statistics μ̃l, σ̃l, and gradient-descend the pixels until induced matches stored, at every layer simultaneously.
This is the method, made fully interactive. Below, a small frozen network has stored target statistics at three layers (read off "real data"). You start from noise and run the BN-Stats optimizer. Watch the per-layer statistics gap close as the synthetic input is sculpted to satisfy the model's internal memory.
Each row is one BN layer's per-channel statistics: the hollow markers are the stored targets (μ̂, σ̂ from real data), the filled markers are what the current synthetic input induces. Hit Run and watch the filled markers slide onto the targets and the BNS loss fall. Lower the learning rate to see it crawl; raise it too high to watch it overshoot and oscillate — a real failure mode.
"Match the statistics" needs a precise loss. The paper measures the gap between the stored target distribution and the induced distribution with the Kullback-Leibler (KL) divergence, under a simple assumption: treat each channel's activations as a 1-D Gaussian (isotropic, no cross-channel covariance). Target P = 𝒩(μ̂, σ̂2), induced Q = 𝒩(μ̃, σ̃2).
The closed-form KL between two univariate Gaussians is a standard, derivable result. The paper's BNS measure is exactly this:
The total objective averages this over all N BN layers (and, within a layer, over channels):
Suppose for some channel the target is μ̂=0, σ̂=1 and the synthetic batch currently gives μ̃=0.5, σ̃=2. Plug in:
Now suppose the optimizer pulls it to μ̃=0.1, σ̃=1.1: BNS = log(1.1) + (1+0.01)/(2·1.21) − 0.5 = 0.0953 + 0.417 − 0.5 = 0.013. The loss fell, confirming the gradient is steering the pixels toward the target distribution. A tiny ε=10−8 is added to σ̃2 to survive zero-variance channels.
Hollow curve = stored target 𝒩(μ̂,σ̂). Filled curve = your induced 𝒩(μ̃,σ̃). Drag the sliders to move/widen it and watch BNS fall toward 0 as the curves overlap. The loss reading separates the location term from the spread term — see which one dominates.
The three schemes are not rivals; they are loss terms. The paper's full generator, BNS + I, simply adds them with weights — and the authors note that I and BNS alone are just special cases (set the other weights to zero). The combined loss is:
Why combine? BNS gives correct internal statistics but is class-agnostic — it does not know which classes the batch should represent. Inception injects class identity. Together, the class constraint nudges the batch composition, and the BNS term keeps the internal statistics honest while accommodating those chosen classes. The trade-off: more hyper-parameters (the loss weights) to tune.
# Frozen model with BN layers. We optimize the INPUT, not the weights. import torch, torch.nn.functional as F bn_stats = [] # hooks capture (mu_hat, sig_hat) targets + induced (mu_t, sig_t) def hook(m, inp, out): induced_mu = out.mean(dim=[0,2,3]) # per-channel mean of THIS synthetic batch induced_var = out.var(dim=[0,2,3]) + 1e-8 # +eps for zero-variance channels bn_stats.append((m.running_mean, m.running_var, induced_mu, induced_var)) for m in model.modules(): if isinstance(m, torch.nn.BatchNorm2d): m.register_forward_hook(hook) X = torch.randn(256, 3, 224, 224, requires_grad=True) # the TRAINABLE input opt = torch.optim.Adam([X], lr=0.1) model.eval() # freeze weights + BN running stats for step in range(1000): bn_stats.clear() logits = model(torch.clamp(X, 0, 1)) # forward; hooks fill bn_stats loss = 0. for mu_hat, var_hat, mu_t, var_t in bn_stats: # BNS: closed-form Gaussian KL loss += (torch.log(var_t.sqrt()/var_hat.sqrt()) + (var_hat + (mu_hat-mu_t)**2)/(2*var_t) - 0.5).mean() loss = loss/len(bn_stats) # J_KL averaged over BN layers opt.zero_grad(); loss.backward(); opt.step() # gradient flows into X only
We now have synthetic data. What do we do with it? Two jobs: calibrate the quantized model (set the ranges from Chapter 2), and — for aggressive compression — fine-tune it back to accuracy via knowledge distillation.
The distillation setup is clean: the full-precision model is the teacher, its quantized copy is the student. Feed both the same synthetic batch; the student is trained to match the teacher's output distribution (KL on the logits). No ground-truth labels are needed — and indeed none are available in the data-free regime.
Under extreme quantization (2-bit, 4-bit) the student's internal features drift far from the teacher's, and matching only the final logits leaves training unstable. The authors add an Intermediate Quantization (IQ) loss: align the teacher's and student's intermediate features (outputs of several blocks) with a smooth-L1 distance.
The intuition: a good low-precision proxy lives near the teacher in feature space, so explicitly pulling the student's intermediate activations toward the teacher's stabilizes the descent. They also use in-batch MixUp on the inputs (without mixing teacher outputs) for further regularization. The teacher's BN stays in full precision; the student's activation ranges are frozen during fine-tuning (following McKinstry et al.).
Real Table-1 numbers (CIFAR-10 accuracy vs number of fine-tune samples). Toggle the tricks on/off to see how IQ and MixUp lift accuracy most when samples are scarce — exactly the data-free regime. With just 1 sample, KD alone = 76.3%; KD+IQ+Mix = 83.7%.
The headline claim: synthetic BNS data calibrates and fine-tunes quantized models on par with real data. Let us look at the actual numbers, side by side, the way the paper presents them.
ResNet-44 on CIFAR-10 (fp32 baseline 93.23%), Wide-ResNet-28-10 on CIFAR-100 (fp32 83.69%). Across calibration and KD, BNS tracks the Reference (real-data) column closely, while Gaussian (G) and Inception (I) lag badly under hard compression.
Bars are validation accuracy per scheme (Reference / BNS / I / BNS+I / Gaussian). Switch between Calibration and KD, and between settings. Notice: under the hard 2w4a KD setting, Gaussian craters (70%) and Inception too (80%), while BNS reaches ~88.5% — within ~1.7 points of real data's 90.3%.
This is the punchline. On ImageNet (ResNet-18 fp32 69.75%, MobileNet-V2, DenseNet-121 from torchvision), the authors generate 10K–100K synthetic images and fine-tune the quantized student for 44,000 steps — and recover accuracy with no real data at all. For 8w8a calibration, BNS lands at 69.55% vs the real-data 69.63% — a rounding error apart. The paper states this is, to their knowledge, the first successful data-free fine-tuning of a compressed model at ImageNet scale.
| Model / setting | fp32 | Real data | BNS (synthetic) | Gap |
|---|---|---|---|---|
| ResNet-18 · 8w8a · calib | 69.75 | 69.63 | 69.55 | 0.08 |
| ResNet-18 · 4w4a · calib | 69.75 | 54.72 | 55.29 | +0.57 |
| MobileNet-V2 · 8w8a · calib | 71.88 | — | on par | ~0 |
One honest caveat the authors surface: KD has no labels, so the student inherits whatever biases the teacher has — semi-supervised KD shows a ~1.5% relative gap to label-supervised cross-entropy when full data is available. But when samples are scarce (the realistic data-free case), KD wins. They are upfront that calibration/quantization improvements (bias correction, equalization, bit allocation) are orthogonal and would stack on top.
This paper sits at the head of a small lineage. It crystallized the idea that BatchNorm statistics are a recoverable data prior — an idea that exploded into the broader data-free toolkit.
| Symbol / term | Meaning | Where it lives |
|---|---|---|
| μ̂, σ̂ | Target per-channel mean/std (real data) | BN running stats, stored in model |
| μ̃, σ̃ | Induced per-channel mean/std (synthetic) | Measured live from synthetic batch |
| BNS | log(σ̃/σ̂) + (σ̂²+(μ̂−μ̃)²)/(2σ̃²) − ½ | Per-channel Gaussian KL loss |
| JKL | Mean of BNS over all BN layers | The statistics objective |
| lossI | e−logit/scale (class injection) | Inception term |
| BNS+I | αJKL + βlossI + γlossprior | Full generator |
| IQ loss | smooth-L1 on intermediate teacher/student features | Distillation stabilizer |
| #w#a | e.g. 4w4a = 4-bit weights, 4-bit activations | Quantization config |