SQuant: On-the-Fly
Data-Free Quantization

Chapter 1: The Key Insight

When you round a weight, you make an error. Round-to-nearest minimizes the size of each individual error — but it treats every weight in isolation. SQuant's insight is that what actually hurts the network is not the individual errors; it is what they add up to.

Picture a single neuron computing a dot product: y = ∑_i w_i x_i. If you perturb each weight by Δw_i, the output changes by ∑_i Δw_i x_i. Now here is the trick: if the inputs x_i are all roughly the same on average (they are — a kernel scans nearby pixels of one feature map), then the output error is roughly proportional to ∑_i Δw_i — the signed sum of the rounding errors.

The quantity that runs the whole paper

The paper names this signed sum the Absolute Sum of Error (ASE), and minimizing it under a rounding constraint is the Constrained Absolute Sum of Error, or CASE:

That is it. No dataset appears in that formula. No activations. No gradients. It depends only on the weights you are rounding. That is why SQuant is data-free: the entire optimization target was derived from the loss, but every data-dependent term got absorbed into a constant and dropped — leaving a quantity built purely from the weights' own rounding errors.

What the whole method is, in one breath

SQuant is: (1) round every weight to nearest — call the leftover error its perturbation; (2) for each kernel, compute the signed sum of perturbations; (3) flip a few weights (round the other way) so that signed sum drops below 0.5; (4) repeat the same flipping trick one level up, across whole kernels in an output channel. Three nested passes — element, kernel, channel — each one canceling the residual error the previous pass left behind. We will derive every piece, and you will watch the flips happen.

Chapter 2: Quantization from Scratch

Before any of SQuant's cleverness, we need to be precise about what "quantize a weight" even means. Quantization maps a continuous float to one of a small, evenly spaced set of integers, then scales back.

The standard uniform quantizer for a weight w is:

Symbol by symbol, with intuition:

• w — the original float weight, e.g. 0.137.
• s — the scale: the real-world size of one integer step. Think of it as the spacing between grid lines on a ruler. w/s measures the weight in "ruler ticks."
• ⌊ · ⌉ — round-to-nearest integer. This snaps the weight onto the grid. The only place information is destroyed.
• clip(·, min, max) — clamp to the integer range the bit-width allows. For 4-bit signed weights, that range is roughly [−8, 7].
• ŵ — the dequantized weight: the integer multiplied back by s so it lives in the same units as w again. This is what the network actually uses.

A fully worked int8 example

Let us quantize a real weight to 8-bit. Suppose a channel's weights have a max magnitude of 0.50, and we use symmetric int8 with range [−127, 127]. The scale is the max magnitude divided by the largest integer:

Now quantize w = 0.137:

The rounding error (perturbation) is Δw = ŵ − w = 0.13780 − 0.137 = +0.00080. In tick units, w/s = 34.80 rounded to 35 — an error of +0.20 ticks. SQuant works entirely in tick units, where the rounding error always lies in [−0.5, +0.5]. Our weight is at +0.20: rounded up, with room to flip down to −0.80 ticks if SQuant decides cancellation is worth it.

Drag bits to see the grid get coarser (fewer levels) as precision drops — at 4 bits the gaps are huge. Drag weight and watch the orange marker snap to the nearest blue grid line; the red bar is the rounding error Δw. Notice how a small drop in bits makes that error explode.

Chapter 3: Why Rounding Is Not Enough

Round-to-nearest feels optimal — it minimizes each weight's error. So why does it collapse to 0.1% accuracy at 4 bits? To see the flaw we need to ask the right question: what does quantization actually do to the loss?

Start with a second-order Taylor expansion of the task loss around the trained weights, as a function of the weight perturbation ΔW:

Decode it:

• L — the task loss (e.g. cross-entropy on the classification it was trained for).
• ΔW — the perturbation quantization introduces: ŵ − w for every weight, stacked into a vector.
• g — the gradient of the loss w.r.t. weights. Key move: the model is already trained to convergence, so the gradient is essentially zero. We drop this term.
• H — the Hessian, the matrix of second derivatives. It encodes the curvature of the loss: how much the loss bends as you move weights, and crucially how weights interact.

With the gradient gone, the loss damage from quantization is governed entirely by the quadratic term ½ ΔW^T H ΔW. This is the heart of every Hessian-aware quantizer (AdaRound, HAWQ, BRECQ).

Reducing the quadratic to something concrete

Under the standard layer-wise assumption, the per-output-channel Hessian factors as H_m ≈ l_m · E[x x^T], where x are the layer inputs and l_m a positive scalar. Substituting, the objective for one output channel becomes:

Read the right-hand side: it is the mean-squared output error of the channel — the error in y = ΔW·x. This is precisely what AdaRound minimizes, but it needs x (the data!) to estimate E[x x^T]. And minimizing it exactly is NP-hard combinatorial optimization. SQuant's two contributions are: approximate E[x x^T] so the data vanishes (Ch 4–5), and solve it cheaply without back-prop (Ch 7–8).

Chapter 4: The Diagonal Hessian Decomposition (the showcase)

The whole game is the matrix E[x x^T]. It is dense — every weight error couples to every other through it — and computing it needs data. SQuant's masterstroke is to approximate this dense matrix as the sum of three simple structured matrices, each capturing the coupling at a different granularity of the weight tensor:

Recall the weight tensor shape: an output channel m has N input channels (kernels), each with K elements (kernel height × width). So one output channel is an N×K grid of weights. The three matrices cover three nested scopes of that grid:

• E (element-wise) — a diagonal matrix. Captures each weight on its own. This is the part round-to-nearest already handles.
• K (kernel-wise) — a block-diagonal matrix of all-ones blocks J_K, one block per kernel. Captures the coupling among the K weights within a kernel — they all scan the same feature map, so their inputs are nearly equal.
• C (channel-wise) — a single all-ones matrix c_m J_NK over the whole output channel. Captures the common coupling across all weights in the channel.

The objective, expanded

Substituting E + K + C into the channel objective and writing it out term by term:

Three terms, three scopes. The first is per-element squared error (what rounding minimizes). The second squares the sum within each kernel. The third squares the sum over the whole channel. The coefficients e_n,i, k_n, c_m are positive constants that weight the three scopes — and they are the only thing in this expression that depends on data. That is the door SQuant walks through in the next chapter.

This is the paper's Figure 1, made interactive. The grid is the NK×NK matrix E[xx^T] for one output channel (here N=3 kernels of K=3 elements). Click E to see only the diagonal (rounding); K to add the per-kernel blocks; C to fill in the whole-channel coupling; E+K+C to see how the three structured pieces together cover the full dense matrix that rounding ignored.

Chapter 5: Going Data-Free

We have the objective from Chapter 4 with three coefficients e_n,i, k_n, c_m. Those coefficients are the only place the data x still hides — they come from E[xx^T]. SQuant's audacious move: just drop them.

Why is that allowed? The paper proves (Appendix A) that you can always decompose E[xx^T] so that all three coefficients are strictly positive: e_n,i > k_n > c_m > 0. If every coefficient is positive, then driving each squared-sum term toward zero always reduces the true objective — regardless of the exact coefficient values. You do not need to know the weights of the three terms to know which direction is downhill.

Is dropping the coefficients actually safe?

This is the move that should make you suspicious — so the authors checked it empirically. They computed the true coefficients from real data, ran the flipping optimization, and measured how often a flip that helps the approximate (data-free) objective also helps the true (data-driven) one. They call this the approximation precision (AP):

Across ResNet18 layers the AP is ~95–100% for SQuant-E&K and essentially 100% for the channel-level flips. In other words: dropping the coefficients almost never sends a flip the wrong way. The progressive, term-by-term reduction (each term positive) is what makes the approximation so faithful in practice.

Each bar is one layer's approximation precision — the fraction of weight flips chosen by the data-free CASE objective that also reduce the true, data-dependent loss. Near-100% means dropping the coefficients was safe.

Chapter 6: SQuant-E, -K, -C & Relaxation

The three terms of the data-free objective become three optimization sub-problems, solved one after another. Each one minimizes a signed sum under a constraint, and each constraint is 0.5 — because the quantization grid is discrete and a perturbation past 0.5 ticks means the weight should have rounded the other way.

SQuant-E: rounding (the floor)

The first term, ∑ ΔW², is minimized by making each |ΔW| ≤ 0.5 — which is exactly round-to-nearest. SQuant-E introduces zero error and is O(1) per weight. It is the starting point, not the goal.

SQuant-K: cancel within each kernel

The second term wants each kernel's signed sum small: |∑_i ΔW_m,n,i| ≤ r_k = 0.5. This is the kernel's ASE. To shrink it, you flip some elements (round the other way), trading larger per-element error for a smaller sum.

SQuant-C: cancel across the channel

The third term wants the whole channel's signed sum small: |∑_n,i ΔW_m,n,i| ≤ r_c = 0.5. Same idea, one scope up: now you flip whole kernels.

Composing in order: E → K → C

The composition is strict and one-directional. First round everything (E). Then, holding the rounding as a baseline, flip elements to fix kernel sums (K). Then, holding the kernel solution, flip kernels to fix the channel sum (C). The progressive order matters: it lets each pass calibrate the error the previous pass left behind, and it is what makes the algorithm linear-time instead of NP-hard.

Chapter 7: The Flip Algorithm

Here is the engine. Given a set of rounded weights and their perturbations, how do you cancel their signed sum with the fewest, smallest flips? The answer is a beautiful four-line routine that we will run by hand.

Say a kernel of weights, after rounding, has perturbations (in tick units):

Step 1: accumulate the error

The signed sum is e = ∑_i p_i = 0.30 + 0.40 − 0.10 + 0.35 − 0.20 = +0.75. This exceeds r_k=0.5, so the kernel needs fixing. The sum is positive — the kernel rounded up too much on net.

Step 2: disable the wrong-sign elements

To reduce a positive e, we want to flip elements that currently round up (positive p) down by one tick — each such flip subtracts about 1 from the sum. Flipping a negative-perturbation element would make things worse, so we disable them (set their p to 0 as candidates). Surviving candidates: [+0.30, +0.40, 0, +0.35, 0].

Step 3: how many to flip

We need to remove |e| = 0.75 of error, and each flip removes ~1 tick. So k = ⌊|e|⌉ = ⌊0.75⌉ = 1 flip suffices to push |e| below 0.5.

Step 4: flip the largest perturbations

Which one? The element with the largest |p| — here +0.40. Why the largest? Because it is closest to the flip boundary (0.5), so flipping it incurs the smallest new per-element error. Flipping +0.40 → +0.40 − 1 = −0.60. New sum: e = 0.75 − 1.0 = −0.25, and |{-0.25}| < 0.5. Done. The kernel's output error is now near zero, at the cost of one element rounding to its second-nearest grid point.

Each bar is one weight's perturbation p_i (in ticks). The teal line is the running signed sum e; the dashed band is the ±0.5 target. Hit Step to walk the four phases: accumulate → disable wrong-sign (greyed) → pick top-k (highlighted) → flip. Watch the sum snap inside the band. New random kernel reshuffles.

Chapter 8: On-the-Fly — The Full Algorithm

The full SQuant is the flip routine wrapped in the progressive E→K→C composition, run independently per output channel. Here it is as the paper's Algorithm 1, with the data flow annotated:

# Progressive SQuant: quantize weight tensor W of one layer
# W: [M, N, K]  (out-channels M, in-channels N, kernel size K)
def squant_flip(w, p):                 # the Ch.7 engine
    e = p.sum()                          # 1. accumulate signed error
    p = p.clone(); p[e * p < 0] = 0      # 2. disable wrong-sign
    k = round(abs(e))                    # 3. how many to flip
    f = p.abs().topk(k).indices         # 4. top-k largest |p|
    w[f] = flip(w[f])                    #    flip them (±1 tick)
    return w

def squant(W, s):                       # s: per-channel scale [M]
    for m in range(M):                    # SQuant-C: each output channel
        E = round(W[m] / s[m])           # SQuant-E: round to nearest
        dE = E - W[m] / s[m]              # element perturbation [N,K]
        for n in range(N):                # SQuant-K: each kernel
            E[n] = squant_flip(E[n], dE[n])
        dK = kernel_perturbation(dE)    # one residual per kernel
        E = squant_flip(E, dK)            # SQuant-C: flip whole kernels
        C[m] = E * s[m]                  # dequantize
    return C

Complexity — why it is sub-second

Each kernel's flip is O(K) (a sum and a top-k over K elements). There are MN kernels and M channels, so the layer is linear in the number of weights — no iteration, no gradient, no back-prop. Contrast with AdaRound/GDFQ, which run hundreds of gradient steps over generated data.

What happens to the data flow

Trace the tensors: in goes W ∈ ℝ^M×N×K (fp32) and a scale vector s ∈ ℝ^M. SQuant-E produces integer codes ∈ [−2^b−1, 2^b−1−1] and float perturbations ΔE ∈ [−0.5,0.5]. SQuant-K rewrites a few codes (their |Δ| grows to [0.5,1.0)). SQuant-C rewrites a few whole kernels. Out comes the integer tensor C ∈ ℤ^M×N×K at b bits, plus the same s. For a 1×1 conv or FC layer (K=1), there is nothing to sum inside a kernel, so SQuant-K is skipped entirely — the design degrades gracefully to just E&C.

Chapter 9: Experiments

Two numbers matter for any DFQ method: accuracy after quantization, and time to quantize. SQuant wins both, and the margin widens as the bits drop.

Accuracy vs. bit-width

At 8 bits everything is easy — SQuant loses only ~0.1% on average. The story is at 4 bits, where rounding collapses and even data-generative methods struggle:

ResNet18 ImageNet top-1. Grey = float baseline (71.47%). DFQ is the only other true data-free method (no back-prop, no synthetic data); ZeroQ/DSG generate data; GDFQ even fine-tunes. SQuant (orange) tops them all at every bit-width — and at 4-bit it beats the next-best by >5 points. Toggle bits to watch the gap open as precision drops.

Speed

The accuracy is impressive; the speed is almost absurd. Quantizing 4-bit ResNet18 end to end:

That is roughly a 70,000× speedup over GDFQ — while being more accurate. A single layer averages 3 ms.

Ablation: do you need all three of E, K, C?

The ablation is the cleanest proof that the cancellation idea, not the rounding, is what carries the accuracy. On 4-bit ResNet18 (weights only):

Each cancellation scope you add buys accuracy, and they stack. Note E&K beats E&C — the kernel-level approximation is tighter than the channel-level one (the channel scope is a stronger, looser assumption), exactly as the Hessian decomposition predicted in Chapter 4.

Chapter 10: Connections & Cheat Sheet

SQuant sits in a lineage of Hessian-aware quantizers and pushes one idea to its limit: if you approximate the Hessian cleverly enough, the data falls out of the objective entirely.

• AdaRound (Nagel 2020) — the direct ancestor. Also Hessian-based, also "round up or down," but it minimizes the true output MSE and therefore needs data and a per-layer gradient-based solve. SQuant's CASE is the data-free, closed-form cousin.
• HAWQ / HAWQ-v2/v3 — use the Hessian trace to decide mixed-precision bit allocation. SQuant uses Hessian structure to decide rounding direction.
• DFQ (Nagel 2019) — the original data-free method (weight equalization + bias correction). SQuant keeps its data-free activation handling but replaces rounding with CASE.
• ZeroQ / DSG / GDFQ — the data-generative camp. SQuant makes them unnecessary at the weight-quantization step by removing the data dependence analytically rather than synthesizing fake data.

The cheat sheet

The one-sentence summary

Related Engineermaxxing lessons

If you came here for compression, also see the quantization and model-compression material in the Gleams and the broader Veanors catalog — SQuant pairs naturally with lessons on the Hessian, post-training quantization, and the AdaRound "up or down" objective it descends from.