Optimal Bit Allocation for
Compressing Weights & Activations

Chapter 1: The Key Insight

Strip the paper to its skeleton and it is one analogy: allocating bits to network layers is the same problem as allocating bits to frequency bands in JPEG or to subbands in an audio codec. Compression engineers solved this decades ago with rate–distortion theory. This paper imports that machinery wholesale.

The argument has exactly three moving parts, and once you see them the rest is bookkeeping:

Why this is the right framing

The reason additivity (part 2) holds is the quiet miracle that makes everything tractable. A neural network is continuously differentiable from input to output. Quantization adds a tiny error to each tensor. For small errors, a first-order Taylor expansion says the output perturbation is approximately linear in those input errors — and linear perturbations add. We will verify this empirically in Chapter 4 and derive it in Chapter 5.

What the whole pipeline is, in one breath

For each weight layer and each activation layer: sweep the bit-rate, measure the output error at each rate, fit a distortion-vs-rate curve. Then run a tiny allocator that, given a total budget, picks the bit-rate for every layer so all slopes match. Quantize at those rates, entropy-code the indices, done. No gradients, no training data, polynomial time.

Chapter 2: Quantization & the Rate Knob

Before we can allocate bits, we need to be precise about what a "bit-rate" buys us. The paper uses uniform scalar quantization followed by entropy coding. Let's unpack both, because the rate knob lives here.

Uniform scalar quantization

Take a tensor of floats. Pick a step size Δ. Round every value to the nearest multiple of Δ. The integer index of that multiple is what you store. Smaller Δ means finer bins, less rounding error, but more distinct indices → more bits. Larger Δ means coarse bins, big rounding error, few indices → few bits. The step size IS the rate knob.

• w — one weight (or activation) value, a float.
• Δ — the quantizer step size, chosen individually for each layer. This is the lever the allocator pulls.
• round(·) — nearest integer. The stored value is the index round(w/Δ).
• q(w) — the reconstructed value the network actually uses at inference.

Why entropy coding, not just "use B bits"

If a layer's quantized values land in, say, 30 distinct bins, naive fixed-width storage needs ⌈log₂ 30⌉ = 5 bits each. But the bins are not used equally — values cluster near zero. Entropy coding (e.g. Huffman/arithmetic coding) gives common indices short codes and rare indices long codes, so the average bits per value drops to the empirical entropy of the index distribution.

A bell-shaped weight distribution (gray) and its quantization bins (orange grid). Drag Δ. Watch the readout: small Δ → many bins, high entropy (high rate), small rounding error (low distortion); large Δ → few bins, low rate, big distortion. This single curve — distortion vs rate — is what the allocator reasons about.

import numpy as np

def quantize(w, delta):
    idx = np.round(w / delta)              # integer index per value
    return idx * delta, idx               # reconstructed value, index

def entropy_bits(idx):
    """average bits/value after ideal entropy coding"""
    _, counts = np.unique(idx, return_counts=True)
    p = counts / counts.sum()
    return -np.sum(p * np.log2(p))      # Shannon entropy H

w = np.random.randn(100000)            # a layer's weights
qw, idx = quantize(w, delta=0.2)
rate = entropy_bits(idx)               # avg bits/value
dist = np.mean((w - qw) ** 2)         # quantization MSE

Chapter 3: Output Error — the Right Thing to Measure

Here is a subtle but crucial choice. When you quantize layer i's weights, the obvious thing to measure is how much those weights changed — the quantization MSE of that tensor. The paper measures something else: how much the network's final output changed.

Why? Because a big change in a robust layer's weights might leave the output almost untouched, while a tiny change in a sensitive layer can flip the prediction. The thing you actually care about is the output, so measure error there.

The definition, symbol by symbol

Given network ℱ and input image I, the clean output is the vector V = ℱ(I) (think: the pre-softmax logits). Quantize the weights of layer i only, and the output becomes V̂. The output error from quantizing those weights is:

• D_ℱ^W_i — the output distortion caused by quantizing layer i's weights. (Superscript A_i for activations.)
• d(V, V̂) — squared Euclidean distance between clean and perturbed output: ∑_j(V_j − V̂_j)².
• E[·] — expectation over input images (in practice, averaged over a few hundred validation images).
• dim(V) — the output dimension, dividing out so the number is per-component, comparable across networks.

A convex, decreasing distortion-vs-rate curve. Drag rate to slide the operating point and read the slope (the orange tangent) — this slope, ∂D/∂R, is the quantity the Lagrangian equalizes across layers. Drag sensitivity to see fragile layers (steep) vs robust layers (flat). A fragile layer's slope stays steep longer, so it deserves more bits.

Chapter 4: Additivity — the Load-Bearing Assumption

Everything rests on one claim: if you quantize many layers at once, the total output error equals the sum of the errors you'd get from quantizing each layer alone.

Read it carefully. The left side is the error when all weights and activations are quantized simultaneously. The right side is a sum of terms, each measured by quantizing one tensor in isolation. If this holds, you can characterize each layer independently and never have to test combinations — that's the difference between polynomial and exponential work.

The paper's verification (Figure 1)

The authors don't just assert it — they test it. Quantize layer 5's weights and layer 10's weights together, plot the true joint output error (y-axis) against the sum of the two individual errors (x-axis). If additivity holds, every point lands on the 45° diagonal. In VGG-16 they did this for weight–weight, activation–activation, and weight–activation pairs. The points hug the diagonal.

Each dot is a pair of quantized layers: x = (error of layer A alone) + (error of layer B alone), y = true error of quantizing both. The gray line is the perfect-additivity diagonal. Drag interaction up to simulate large quantization noise (where the Taylor approximation breaks and points drift off the line). At small noise — the regime the paper operates in — additivity is tight.

Chapter 5: The Lagrangian — Deriving the Equal-Slope Rule

Now the payoff. We have a sum of per-layer errors (additivity) and a sum of per-layer rates. We want to minimize total error subject to a fixed total budget. This is a constrained optimization, and the tool is a Lagrange multiplier.

The constrained problem

In words: choose the bit-rates R^W_i, R^A_i for every weight and activation layer to make total output error smallest, while the total bits used equals your budget R_total.

Fold the constraint into a single cost

A Lagrange multiplier λ turns the constrained problem into an unconstrained one. We build a cost J that subtracts λ times the rate — spending bits is "penalized" at price λ:

Think of λ as the exchange rate between error and bits: "how much output error am I willing to accept to save one bit?" High λ → bits are expensive → aggressive compression. Low λ → bits are cheap → keep more precision. Sweep λ and you trace the entire rate–distortion frontier.

Set the derivatives to zero

At the optimum, J is flat in every variable. Take the partial derivative of J with respect to one layer's rate, say R^W_i. Only that layer's distortion term and that layer's rate term depend on it:

Rearrange, and do the same for activations. The result is the entire paper in one line:

Three layers with different sensitivities, each shown as its own D–R curve. The operating point on each is placed where the slope equals −λ. Drag λ: as bits get cheaper (small λ), every point slides right to higher rate, but they always sit at the same slope. The total bits and total distortion update live — this is one point on the rate–distortion frontier.

Chapter 6: The Allocator Algorithm (the showcase)

The equal-slope condition tells us what the answer looks like. Now: how do we actually compute it? The paper finds the Pareto-optimal allocation in polynomial time with a simple λ-sweep. Here is the recipe, and then you'll run it.

Six layers, sorted left-to-right by sensitivity (most sensitive first). Bars = bits assigned. Drag budget to set the average bits/layer. Toggle Optimal vs Equal: the optimal allocator (a binary search on λ) front-loads bits onto the steep, sensitive layers and starves the flat, robust ones — while the equal scheme paints every bar the same. The total output distortion readout shows how much the optimal allocation saves.

The allocator as runnable code

import numpy as np

# curves[i] = list of (rate, distortion) points for layer i, rate ascending
def slope_at(points, k):
    """local slope dD/dR between point k and k-1 (negative)"""
    (r0,d0),(r1,d1) = points[k-1], points[k]
    return (d1 - d0) / (r1 - r0)

def alloc_for_lambda(curves, lam):
    """each layer picks the highest rate whose slope is still steeper than -lam"""
    chosen = []
    for pts in curves:
        k = 0
        while k+1 < len(pts) and -slope_at(pts, k+1) > lam:
            k += 1
        chosen.append(pts[k])
    return chosen                               # (rate, dist) per layer

def allocate(curves, budget):
    """binary-search lambda so total rate == budget"""
    lo, hi = 1e-6, 1e6
    for _ in range(60):
        lam = (lo + hi) / 2
        pts = alloc_for_lambda(curves, lam)
        rate = sum(r for r,_ in pts)
        if rate > budget: lo = lam       # too many bits -> raise price
        else: hi = lam                      # too few -> lower price
    return alloc_for_lambda(curves, hi)

Chapter 7: Joint Weight + Activation Allocation

A key contribution that's easy to skim past: the paper allocates bits to weights and activations in one shared budget, not two separate ones. This is not just tidiness — it measurably lowers output error.

Separate vs joint — what changes

The naive approach (separate) gives weights their own budget and activations their own budget, and optimizes each pool independently. The paper's approach (joint) throws all weight tensors and all activation tensors into one pool and applies a single λ across everything.

Why does joint win? Because the equal-slope condition must hold across the entire network. If activations happen to be more sensitive than weights at the current allocation, a separate scheme can't react — it's locked into its weight/activation split. A joint scheme simply moves a few bits from weights to activations until all slopes match.

The data-flow reason activations matter

Remember the tensor accounting from Chapter 0: ResNet-50 is 22M weights + 8M activations. If you only ever compress weights (as most prior work did), the 8M activations — over a quarter of the memory traffic — stay in full float and dominate the remaining footprint. Joint allocation is what lets a single knob squeeze both, which is the only way to actually hit an 8× total-memory target.

Left group = weight layers, right group = activation layers (activations here are more sensitive). Separate fixes each group's total bits, so activations stay starved. Joint uses one λ everywhere — watch bits migrate from weights to the hungrier activations and the total distortion drop. Same total bits, lower error.

Chapter 8: Experiments & Results

The proof is in the numbers. The authors evaluate on VGG-16 (70.8% baseline top-1) and ResNet-50 (75.2% baseline) over the ImageNet validation set, comparing against a wall of contemporary methods: BWN, TWN, INQ, FGQ, IAOI, AQ, CLIP-Q, SYQ, TBSQ.

Headline numbers

The remarkable line is VGG-16 at 3 average bits — over 10× smaller — with under 1% accuracy loss, and no retraining. The method outperforms every baseline on both networks except TBSQ on VGG-16, and TBSQ only wins after retraining (which can even exceed the original accuracy). Without retraining, this framework is state-of-the-art.

What the per-layer allocation revealed

When the authors plotted the learned bit allocation (Figures 4–5), a pattern emerged: the allocation is strongly unequal, and it differs between weights and activations and across depth. They also note one practical choice — they do not quantize the last layer's activations, because the final logits feed directly into the prediction and are too sensitive to coarsen. This is the equal-slope rule talking: that tensor's slope stays steep, so it always demands more bits.

A stylized version of the paper's Figure 2 (VGG-16). The orange curve is the proposed method's accuracy–size frontier; the dim curve is equal allocation; dots are individual baselines. Drag total size to slide along the budget axis and read the accuracy gap — the unequal-allocation curve sits well above equal allocation across the whole range.

Chapter 9: Connections & Cheat Sheet

You can now re-derive this paper on a whiteboard. Let's place it in the landscape and leave you with a reference card.

Where this sits

The cheat sheet

The recipe, memorized: (1) profile each tensor's distortion-vs-rate curve on a few hundred images; (2) binary-search λ so every layer picks the operating point at slope −λ and the total rate hits your budget; (3) quantize at those step sizes, entropy-code, ship. No gradients. No retraining.

Limitations to keep honest about

• Low-precision breakdown: additivity is first-order, so at binary/ternary precision the Taylor approximation frays. This is a moderate-precision (3–8 bit) method.
• Profiling cost: you must run inference many times per tensor to build the curves — cheap vs retraining, but not free.
• Convexity assumed: the equal-slope rule needs convex D–R curves; in practice they're operated on the lower convex hull of the measured points.
• Hardware reality: truly arbitrary per-layer bit-widths need flexible hardware/codecs to realize the memory savings — the paper measures values stored, not always wall-clock speedup.

Related lessons

If quantization is new, start with the foundations; if the optimization felt fast, slow down on the math.

• The Lagrangian and equal-slope condition are the same tool used throughout constrained optimization — the "price of a constraint" intuition transfers directly.
• The Taylor/Jacobian sensitivity argument is the bridge to Hessian-based methods (HAWQ) that estimate sensitivity analytically instead of by sweeping.

"The rate-distortion approach is intuitively pleasing: the slopes of error-vs-rate must be equal — exactly the well-known Pareto condition for optimal resource allocation."