A2Q+ — Veanors

Chapter 0: The Problem

Picture the single most-repeated operation inside a neural network: the multiply-accumulate (MAC). Take a weight, multiply it by an activation, add the product to a running total. Do that thousands of times, and the running total is one output of a layer. A dot product is just a pile of MACs.

For a decade the quantization community has obsessed over the multiply half. Squeeze the weights to 4 bits, the activations to 4 bits, and the multiplier shrinks dramatically — because the cost of an integer multiply scales with the product of the operand bit widths. But almost nobody touched the accumulate half. The running total — the accumulator — is still kept at a comfortable 32 bits, "just to be safe."

The overlooked cost: The cost of integer multiplication scales quadratically with weight and activation bit widths, but the cost of accumulation scales only linearly with the accumulator width. So as you push weights and activations down to 3 or 4 bits, the multiplier becomes cheap and the 32-bit adder suddenly dominates. Ni et al. (2021) measured a 3-bit×1-bit MAC unit where 32-bit accumulation ate 75% of the power and 90% of the area.

Why not just use a smaller accumulator?

Because of overflow. An accumulator is a fixed-width register. A signed P-bit register holds integers in [−2^P−1, 2^P−1−1]. If your dot product sum wanders outside that range, the bits silently wrap around: a large positive sum reappears as a large negative number. That is not a small rounding error — it is a catastrophic, sign-flipping corruption injected into the middle of a forward pass.

And the danger grows fast. The widest dot product in a layer has K terms; each term is bounded by the operand ranges. The worst-case magnitude of the sum grows with K and with the operand bit widths. Halving the accumulator width exponentially increases the chance that some input pushes the sum out of range.

The data type bound (Eq. 20): The smallest accumulator that provably never overflows, for the worst possible inputs, is
P* = ⌈ α + φ(α) + 1 ⌉, where α = log₂(K*) + N + M − 1.
Here K* is the largest dot-product size, M the weight bits, N the activation bits. For a ResNet50 layer with a wide dot product, this can demand 25+ bits. The paper's mission: train networks that hold accuracy far below this "safe" number.

A worked sense of scale

Concrete numbers. Suppose a layer has dot-product size K = 4096, 4-bit weights (M=4), 4-bit unsigned activations (N=4).

Quantity	Value	Meaning
log₂(K)	12	4096 terms can add up 2¹²×
α = 12 + 4 + 4 − 1	19	worst-case exponent
Safe accumulator P*	~21 bits	provably no overflow, no constraints
A2Q+ target	down to 12 bits	achieved by constraining the weights

That last line is the whole paper in one row. Instead of accepting a 21-bit accumulator, A2Q+ changes the weights during training so that the dot product physically cannot exceed a 12-bit register — and it does so while keeping accuracy. The accumulator stops being a passive number and becomes a hardware budget the training loop must respect.

As you push weight and activation precision very low (e.g. 3-bit), why does the accumulator suddenly become the dominant cost?

Accumulators are always the most expensive part of any chip Multiplier cost scales quadratically with operand bits while accumulation scales only linearly with accumulator width — so low-bit multipliers shrink and the 32-bit adder dominates Low-bit weights require more accumulator passes

Chapter 1: The Key Insight

There are two philosophies for surviving a small accumulator. The first is to accept that overflow will sometimes happen and make the model robust to the wrap-around (train with it, clip scale factors, estimate overflow rates from data). That works, but it needs to know your input distribution in advance — fragile, and a security hole if an attacker feeds you adversarial inputs.

The second philosophy — the one A2Q (2023) pioneered and A2Q+ improves — is far cleaner: make overflow mathematically impossible. If you can prove that for any legal input, the dot product can never leave the register's range, you need no input statistics, no robustness tricks, and no luck.

The lever: the dot-product magnitude bound. The absolute value of a dot product is bounded by Hölder's inequality:
|x^Tq| ≤ ‖x‖_∞ · ‖q‖₁.
The activations x are given by the data, but the weights q are ours to shape. So if we cap the ℓ₁-norm of the weights (the sum of their absolute values), we directly cap how large the dot product can ever get — and therefore which accumulator it fits in.

That single inequality is the seed of the entire method. Read it slowly: the worst the accumulator can ever swing is "the biggest possible activation" times "the total absolute weight." We can't shrink the activations without hurting accuracy elsewhere, but we can learn weights with a small ℓ₁-norm. A2Q makes ‖q‖₁ a learnable, constrained quantity.

So what does A2Q+ add?

A2Q proved the idea works but left two efficiencies on the table. A2Q+ fixes both:

Fix 1 — the bound was too tight

A2Q's ℓ₁ budget wasted half the accumulator's range and depended on the sign of the inputs. A2Q+ zero-centers the weights and derives a bound up to 4× looser — same overflow guarantee, more room for the weights to be expressive.

↓

Fix 2 — the starting point was bad

A2Q couldn't initialize cleanly from a pretrained float model. A2Q+ frames initialization as a projection onto an ℓ₁-ball — the closest weights that still satisfy the budget — so training starts with minimal quantization error.

↓

Result

Combine both with weight normalization → A2Q+. New state of the art on the accumulator-bits-vs-accuracy Pareto frontier. ResNet50 keeps 95% of float accuracy at a 12-bit accumulator.

What this lesson builds: By the end you will be able to (1) derive A2Q's ℓ₁ bound from Hölder's inequality, (2) derive the improved zero-centered bound and see exactly why it's looser, (3) compute the ℓ₁-ball projection by hand, and (4) assemble the full A2Q+ quantizer and read its PyTorch. The whole thing rests on one idea: the accumulator is a budget, and the ℓ₁-norm of the weights is how you spend it.

What is the core mechanism A2Q/A2Q+ use to guarantee no overflow?

Constrain the ℓ₁-norm of the weights so that, via Hölder's inequality, the dot product provably fits the accumulator range Estimate the input distribution and clip outliers Use floating point for the accumulator only

Chapter 2: Background — Quantizers & Accumulators

Before the new math, two pieces of machinery we must be precise about: how a quantizer maps real weights to integers, and what overflow in a fixed register actually looks like.

The uniform affine quantizer

A quantizer turns a real value into a low-bit integer and back. The standard form is:

Q(w) = s · ( clip( ⌊ w/s ⌉ + z ; n, p ) − z )

Symbol by symbol:

s — the scale factor, a strictly positive real. It's the "resolution" of one integer step. Think of it as the size of each tick mark on the integer ruler.
z — the zero-point, an integer that makes real-zero land exactly on an integer. For a symmetric quantizer (which A2Q+ uses) z=0.
⌊·⌉ — rounding. A2Q+ rounds toward zero (denoted ⌊·⌋), not to nearest. We'll see in Ch.5 why this seemingly minor choice is load-bearing.
clip(·;n,p) — clamp into the representable range. For b bits signed: n=−2^b−1, p=2^b−1−1. Unsigned: n=0, p=2^b−1.

So a quantized weight is really Q(w) = s · q, where q is an integer vector. The hardware multiplies integers x^Tq and accumulates them; the scale s is applied once at the end. That's why the overflow analysis is about q, not w.

Two's-complement wrap-around — push the sum past the edge

Accumulator bits P: True sum:

Drag True sum past the register's edge and watch it wrap. A sum of +20 in a 5-bit register (range −16…15) doesn't saturate to 15 — it reappears as −12. Sign-flipped, magnitude wrong. That corruption flows straight into the next layer.

Why per-channel?

Each output channel has its own dot product and its own accumulator. So the ℓ₁ budget is enforced per output channel. A convolution producing 64 channels has 64 independent constraints. Data flow: input tile x ∈ Z^K_N (K = kernel·in-channels) dotted with each channel's weight vector q_c ∈ Z^K_M → 64 separate accumulators, each of which must stay inside P bits.

Key vocabulary locked in: M = weight bits, N = activation bits, P = accumulator bits, K = dot-product length, q = the integer weight vector, x = the integer activation vector. The whole game is choosing q so that x^Tq fits in P bits for every legal x.

In a quantized layer, what number actually flows into the integer accumulator hardware?

The real-valued weight w directly The integer dot product x^Tq; the scale s is applied once afterward The floating-point product before rounding

Chapter 3: The A2Q Baseline

To improve A2Q we must first derive A2Q. The goal: find the largest ℓ₁-norm a weight vector may have so its dot product with any legal activation still fits a signed P-bit register.

The derivation, step by step

We need |x^Tq| ≤ 2^P−1−1 for every x. Start from Hölder's inequality:

|x^Tq| ≤ ‖x‖_∞ · ‖q‖₁

Here ‖x‖_∞ is the largest absolute activation. For N-bit integers the worst case is ‖x‖_∞ = 2^{N−1_signed(x)} — that exponent is N−1 for signed inputs (range straddles zero) and N for unsigned. Substituting and demanding the bound hold:

2^{N−1_signed(x)} · ‖q‖₁ ≤ 2^P−1 − 1

Solve for the weight norm:

‖q‖₁ ≤ (2^P−1 − 1) / 2^{N−1_signed(x)} (A2Q bound, Eq. 5)

Read what this says. The right-hand side is a budget. For fixed activation bits N, shrinking the accumulator P exponentially tightens the budget — it halves every time you drop one accumulator bit. A tighter budget pulls all the weights toward zero, which limits how expressive the layer can be. That is the fundamental accumulator-vs-accuracy trade-off, born directly out of this inequality.

A worked example

Let P=12 (accumulator), N=4 with unsigned activations (so 1_signed=0).

Step	Compute	Result
Numerator 2^P−1−1	2¹¹−1	2047
Denominator 2^N−1_signed	2⁴⁻⁰	16
A2Q budget ‖q‖₁ ≤	2047 / 16	127.9

So with a 12-bit accumulator and 4-bit unsigned activations, the total absolute integer weight per channel may not exceed ~128. If a channel has K=64 weights, that's an average magnitude of just 2 per weight — extremely tight.

How A2Q enforces it during training

You can't just clip after the fact — gradients must flow. A2Q borrows weight normalization: re-parameterize the weight vector w as a direction v and a magnitude g:

w = g · v / ‖v‖₁ ⇒ ‖w‖₁ = g

Now the ℓ₁-norm is a single learnable scalar g. To enforce the budget, A2Q clips g to an upper bound T (the budget, times the scale), and crucially rounds the scaled weights toward zero so that quantization can only ever shrink the norm — never push it back over the budget. The integer norm is guaranteed: ‖Q(w)‖₁ ≤ min(g,T).

Two flaws the authors spotted. (1) The numerator 2^P−1−1 uses only half the register's range — it assumes the sum could be as negative as it is positive, which is overly pessimistic. (2) The denominator's dependence on 1_signed(x) tightens the budget by 2× whenever activations are unsigned. Both throw away ℓ₁ budget the weights could have used. Chapter 4 reclaims it.

With P=12, N=4, signed activations (1_signed=1), what is the A2Q ℓ₁ budget on ‖q‖₁?

2047 / 8 ≈ 255.9 2047 / 16 ≈ 127.9 4096

Chapter 4: The Improved Bound via Zero-Centering (the showcase)

Here is the paper's first contribution and its cleverest trick. The A2Q bound was pessimistic because it had to defend against a worst case where every weight conspires to push the sum maximally in one direction. A2Q+ removes that worst case by forcing one extra property on the weights: zero-centering, i.e. Σ_i q_i = 0.

Why zero-centering helps — the intuition. Imagine the activations as living in a range [c, d]. The dot product x^Tq is the worst when positive weights all hit the max activation and negative weights all hit the min. But if the weights sum to zero, you can subtract a constant from every activation for free — it contributes (const)·Σq_i = 0. Subtracting the midpoint re-centers the activations on zero, halving their effective magnitude. That recovered factor of 2 is the whole win.

The derivation

Let activations lie in [c,d] with d−c = 2^N−1 (the full N-bit span). Because Σq_i=0, shift the activations by the midpoint m=(c+d)/2:

x^Tq = (x − m·1)^Tq + m·(1^Tq) = (x − m)^Tq

The shifted activations x−m now lie in [−(2^N−1)/2, +(2^N−1)/2], so ‖x−m‖_∞ = (2^N−1)/2 — independent of sign. Apply Hölder again and demand the FULL register range [−2^P−1, 2^P−1−1] (the proof shows you can use it all, not half):

((2^N−1)/2) · ‖q‖₁ ≤ 2^P−1 − 1

Solve for the norm (the algebra collapses to a beautifully clean form):

‖q‖₁ ≤ (2^P − 2) / (2^N − 1) (A2Q+ bound, Eq. 7)

How much looser is it?

Divide the new bound by the old one. The ratio is:

2^{N+1−1_signed(x)} / (2^N − 1)

For unsigned activations (1_signed=0) this approaches 4× at low bit width; for signed it approaches 2×. And the lower the activation precision N, the bigger the gain — exactly the sub-4-bit regime that matters most for tiny accumulators.

A2Q vs A2Q+ — ℓ₁ budget & relative gain (Fig. 1)

Accumulator P:

The bars show the ℓ₁ budget at each activation bit width N. Orange = A2Q, teal = A2Q+. The dashed line is the ratio (right axis): notice it climbs toward 4× (unsigned) as N shrinks. More budget = weights free to be more expressive = higher accuracy at the same accumulator size. Drag P and toggle signed/unsigned to watch the whole frontier move.

Worked: the same 12-bit, 4-bit-unsigned case

Bound	Compute	Budget on ‖q‖₁
A2Q (Eq. 5)	2047 / 16	127.9
A2Q+ (Eq. 7)	(2¹²−2)/(2⁴−1) = 4094/15	272.9
Ratio	272.9 / 127.9	2.13×

The weights now get more than twice the budget for the identical 12-bit overflow guarantee. (At N=3 unsigned the ratio approaches the full 4×.) Nothing about the hardware changed — only the math defending it got sharper.

The guarantee is intact. This is not "looser = riskier." The zero-centering constraint earns the extra room: the proof (Prop. 3.1) shows the dot product still cannot leave the register for any legal input. You traded one degree of freedom (forcing the weights to sum to zero) for up to 4× more ℓ₁ budget. That trade is overwhelmingly worth it.

Why does zero-centering the weights (Σq_i=0) let us loosen the ℓ₁ bound?

It makes the weights smaller Any constant shift of the activations contributes (const)·Σq_i=0, so we can re-center activations on zero — halving ‖x‖_∞ and removing the sign dependence It increases the accumulator width

Chapter 5: The Better Initialization via ℓ₁-Ball Projection

The second contribution is subtler but just as important. When you re-parameterize w = g·v/‖v‖₁, you now have two new things to initialize: the direction v and the magnitude g. Crucially, you usually want to start from a good pretrained float model w_float — not from scratch.

The naïve init fails

The obvious move: set v=w_float and g=‖w_float‖₁, recovering w=w_float exactly. But A2Q immediately clips g to the budget T. If the pretrained channel had ‖w_float‖₁ > T (very common — float models aren't accumulator-aware), the clip uniformly crushes every weight by the ratio T/‖w_float‖₁, injecting huge quantization error before training even starts. The network then wastes epochs recovering.

The reframing. Don't shrink uniformly. Find the weight vector that satisfies the budget and is as close as possible to the pretrained one. That's a projection:
v* = argmin_v ½‖v − w_float‖₂² subject to ‖v‖₁ ≤ T.
This is the classic Euclidean projection onto an ℓ₁-ball — a solved convex problem (Duchi et al., 2008).

The closed-form solution — soft-thresholding

If ‖w_float‖₁ ≤ T, the answer is trivially v*=w_float — already inside the ball, no change. Otherwise the optimum lies on the boundary (‖v*‖₁=T) and has a gorgeous form:

v* = sign(w_float) · ( |w_float| − θ )₊

where (·)₊ is ReLU (clamp negatives to 0) and θ ≥ 0 is a single threshold chosen so the surviving weights sum (in absolute value) to exactly T. This is soft-thresholding: subtract θ from every magnitude, kill anything that goes negative.

Why soft-threshold beats uniform-scale. Uniform scaling multiplies everything by 0.5, blurring the strong weights that carry the signal. Soft-thresholding instead sparsifies: it removes the small, near-noise weights entirely and barely touches the large important ones. Same total budget T, but the model's discriminative structure survives. This is exactly why LASSO uses the ℓ₁ ball — it preserves big coefficients and prunes small ones.

ℓ₁-ball projection — soft-threshold vs uniform scale

Budget T:

Faint bars = the original pretrained weights. Solid = after projection to budget T. Drag T down and compare the two strategies. Soft-thresholding (teal) keeps the tall important weights nearly intact and zeros out the small ones; uniform scaling (orange) shrinks everything, including the weights that matter. The reported ℓ₂ error is what training has to undo.

Worked example by hand

Pretrained channel w_float = [5, 3, 1, −4], so ‖w_float‖₁=13. Budget T=9. We must remove 4 of total magnitude. Try θ=1: magnitudes become [4,2,0,3] (the "1" hit zero), summing to 9 = T. ✓ So v* = [4, 2, 0, −3].

Compare the naïve uniform scale by 9/13≈0.69: [3.46, 2.08, 0.69, −2.77]. The largest weight dropped from 5 to 3.46 (lost 30%), and the noise weight 1 survived as 0.69. Soft-thresholding instead kept 5→4 (lost only 20%) and cleanly pruned the 1. The strong directions are preserved.

An honest caveat the paper flags. Extending this clean projection to the zero-centered A2Q+ constraint is hard — the projection's sign-preservation assumption breaks once you subtract the mean. So in practice the authors initialize A2Q+ networks using the A2Q projection and find it still slashes the initial error dramatically. A reminder that real papers ship the 95%-clean version, not the mythical perfect one.

Why is soft-thresholding a better initialization than uniformly scaling the weights down to fit the budget?

It sparsifies — prunes small near-noise weights while barely touching the large important ones, minimizing the ℓ₂ distance to the pretrained weights It makes all weights equal It increases the ℓ₁ norm

Chapter 6: Building A2Q+ — the Full Quantizer

Now assemble the pieces. A2Q+ keeps weight-normalization re-parameterization but bolts on (a) the zero-centering that unlocks the looser bound, and (b) the new bound itself. Here is the full quantizer, then a line-by-line read.

Q(w) = s · clip( ⌊ w/s ⌋ ; n, p )
where w = ( (v − μ_v) / ‖v − μ_v‖₁ ) · min(g, T₊)
and μ_v = (1/K) Σ_i v_i
and T₊ = s · (2^P − 2) / (2^N − 1)

Reading it like the hardware does

1 · subtract the mean

v − μ_v — center the direction vector. This is what makes Σw_i=0, the property Chapter 4's looser bound depends on. Enforced on v, inherited by w.

↓

2 · normalize to unit ℓ₁

/ ‖v − μ_v‖₁ — make the direction have ℓ₁-norm 1, so the magnitude is controlled entirely by g.

↓

3 · clip the magnitude to the budget

min(g, T₊) — the learned norm g is capped at the new (looser) budget T₊. This is where the overflow guarantee is physically enforced.

↓

4 · round toward zero, clip, rescale

s·clip(⌊w/s⌋) — round-to-zero ensures ‖Q(w)‖₁ ≤ ‖w‖₁ ≤ T₊, so quantization can never re-inflate past the budget.

The two exponential parameterizations. A2Q+ learns s = 2^d and g = 2^t, with d, t learned per output channel by gradient descent. Why exponential? Because scale factors span many orders of magnitude and gradients behave far better in log space — and a power-of-two scale is also hardware-friendly (a bit-shift). Rounding is non-differentiable, so the straight-through estimator passes gradients through ⌊·⌋ as if it were the identity: ∇_x⌊x⌋ = 1 everywhere.

The core insight as runnable code

# A2Q+ weight quantizer — the load-bearing forward pass.
import torch

def a2q_plus_quantize(v, t, d, P, N, K):
    """v: [C_out, K] direction params. t,d: [C_out] log-norm / log-scale."""
    s = 2.0 ** d                              # scale = 2^d  (per channel)
    g = 2.0 ** t                              # learned L1 norm = 2^t

    # 1. zero-center the direction  ->  enables the looser bound
    vc = v - v.mean(dim=1, keepdim=True)

    # 2. the IMPROVED A2Q+ accumulator budget (Eq. 7), scaled by s
    T_plus = s * (2.0**P - 2) / (2.0**N - 1)

    # 3. unit-L1 direction * clipped magnitude
    w = vc / vc.abs().sum(1, keepdim=True) * torch.min(g, T_plus)

    # 4. round TOWARD ZERO so ||Q(w)||_1 can only shrink, never exceed T_plus
    q = torch.trunc(w / s)                   # trunc = round-to-zero
    n, p = -2**(31), 2**(31)-1          # weight-int clip range
    q = q.clamp(n, p)
    return s * q                            # dequantized; q is what HW accumulates

# Guarantee check: for any N-bit x, |x @ q.T| provably fits a signed P-bit register.

The contract this code upholds: after step 4, every channel's integer weight vector q satisfies ‖q‖₁ ≤ T₊/s = (2^P−2)/(2^N−1) and Σq_i=0. Those two facts together are exactly the hypotheses of Prop. 3.1 — so the dot product x^Tq is mathematically incapable of overflowing a signed P-bit accumulator, for every N-bit input. No input statistics required.

Why does A2Q+ round weights toward zero rather than to nearest?

It's faster on GPUs Round-to-zero guarantees ‖Q(w)‖₁ ≤ ‖w‖₁, so the quantized integer norm can never grow back past the budget T₊ It removes the need for a scale factor

Chapter 7: Experiments — the Pareto Frontier

The natural way to evaluate an accumulator-vs-accuracy method is a Pareto frontier: for every target accumulator width P, plot the best achievable accuracy (over all weight/activation bit-width combinations). A method "dominates" if its curve sits above and to the left of the others everywhere.

The authors sweep 3–8 bit weights and activations (64 combinations) and, for each, evaluate up to a 10-bit reduction below the safe accumulator — 640 configurations per model, repeated over 3 seeds. Models: MobileNetV1 & ResNet18 on CIFAR-10 (classification), ESPCN & U-Net on BSD300 (super-resolution), plus ResNet18/34/50 on ImageNet.

Accumulator bits vs accuracy — Pareto frontier (Fig. 2)

Each curve is the best accuracy at a given accumulator width. Teal = A2Q+, orange = A2Q, red = baseline QAT (heuristic bit-width manipulation), dashed = float reference. A2Q+ dominates everywhere — and the gap widens as the accumulator shrinks, exactly where it counts. Switch models to see the same story on classification and super-resolution.

The headline numbers

Real values from Tables 1–2 (CIFAR-10) — accuracy at a fixed accumulator width:

Model / P	Baseline	A2Q	A2Q+
MobileNetV1, P=11	—	71.8%	78.5%
MobileNetV1, P=12	62.2%	76.9%	83.5%
ResNet18, P=11	—	81.0%	89.9%
ResNet18, P=12	—	89.1%	92.8%

And the marquee result: on ImageNet, ResNet50 keeps 95% of its float top-1 accuracy at a 12-bit accumulator with no overflow — a +17% top-1 improvement over A2Q. This is the first time anyone showed accumulator-constrained training holding up at this scale and aggressiveness.

Where does the gain come from — attribution. Two independent levers stack: the looser bound (Ch.4) gives the weights more room to be expressive, and the projection init (Ch.5) starts training with far less error to undo. The ablations show both matter, and their effect is largest in the low-P, low-N corner — precisely the regime where A2Q was most starved.

In the Pareto plots, where is A2Q+'s advantage over A2Q largest?

At very large accumulator widths, where everything already works At small accumulator widths (and low activation precision), where the tighter A2Q budget was most starving the weights Only on super-resolution, not classification

Chapter 8: The Tradeoffs & Failure Modes

The fourth contribution is honest characterization: accumulator constraints don't just trade off against accuracy — they reshape the whole quantization design space. Three tradeoffs worth internalizing.

1 · The activation-precision surprise

You'd expect higher activation precision N to always help accuracy. But look at the budget: ‖q‖₁ ≤ (2^P−2)/(2^N−1). Raising N shrinks the weight budget. So under a tight accumulator there is an optimal N — and the paper finds the Pareto-optimal activation bit width decreases as the accumulator shrinks. Counterintuitive, and a direct consequence of the bound.

The competing pressures. Lowering N relieves ℓ₁ pressure on the weights (good for a small accumulator) but also discretizes the activations more aggressively (bad for accuracy in general). The optimum balances the two — and the balance point slides with P. This is why uniform "always use 8-bit activations" advice is wrong in the accumulator-constrained regime.

2 · The exponential cliff

The constraint tightens exponentially as P drops (every bit halves the budget). Accuracy degrades gracefully for a while, then falls off a cliff once the budget gets so small that the weights can't represent the function at all. The frontiers in Ch.7 show this: flat, then a knee, then collapse. The method buys you several extra bits of headroom before the cliff — but it cannot move the cliff to zero.

Budget vs accumulator bits — feel the exponential cliff

Activation bits N: Dot size K:

The teal curve is the per-weight budget (budget ÷ K) as a function of accumulator bits P. Drag P via the curve's reading line, change N and the dot-product size K. Below the red line (~1 unit of magnitude per weight) the channel can barely represent anything — that's the cliff. Bigger K and bigger N both push the cliff to the right.

3 · The cost it does not remove

A2Q+ guarantees overflow avoidance, but it is a training-time method: it changes the weights you ship, not the inference math. It assumes uniform-precision layers, worst-case input bounds (so it can be conservative for benign data), and per-channel accumulators. And it only addresses integer accumulation — float accumulators have their own (different) numerics.

The mental model to keep. The accumulator width is a hardware budget. The ℓ₁-norm of each weight channel is the currency you spend against it. A2Q+ gives you (a) a more generous exchange rate (looser bound) and (b) a better opening balance (projection init). It does not make the budget infinite — it makes you spend it wisely.

Why does the Pareto-optimal activation bit width decrease as the accumulator shrinks?

Because the ℓ₁ budget (2^P−2)/(2^N−1) shrinks as N rises, so under a tight accumulator it pays to lower N to free up weight budget Lower activation precision is always better The accumulator width has no effect on activation choice

Chapter 9: Connections & Cheat Sheet

A2Q+ sits at the intersection of quantization, convex optimization, and hardware-aware ML. The threads it pulls on:

Weight normalization (Salimans & Kingma, 2016) — the re-parameterization w=g·v/‖v‖ that makes the norm a learnable scalar. A2Q+ adapts it to the ℓ₁ norm and adds zero-centering.
ℓ₁-ball projection (Duchi et al., 2008) — the soft-thresholding solution reused for initialization. The same machinery underlies LASSO and sparse coding.
Straight-through estimator (Bengio et al., 2013) — how gradients flow through the non-differentiable rounding. Universal to quantization-aware training.
Hölder's inequality — the one-line bound that the entire overflow guarantee rests on. Worth memorizing: |x^Tq| ≤ ‖x‖_∞‖q‖₁.
Brevitas (Pappalardo, 2021) — AMD's PyTorch quantization library where A2Q and A2Q+ ship, targeting FPGA/ASIC deployment via FINN.

The cheat sheet

Quantity	Formula	Meaning
Dot-product bound	\|x^Tq\| ≤ ‖x‖_∞‖q‖₁	Hölder — the lever
A2Q budget (Eq. 5)	(2^P−1−1)/2^N−1_signed	pessimistic, sign-dependent
A2Q+ budget (Eq. 7)	(2^P−2)/(2^N−1)	zero-centered, up to 4× looser
Gain ratio (Eq. 8)	2^{N+1−1_signed}/(2^N−1)	→4× unsigned, →2× signed
Re-param	w = (v−μ_v)/‖v−μ_v‖₁ · min(g,T₊)	learnable norm + centering
ℓ₁ projection (Eq. 15)	v* = sign(w)(\|w\|−θ)₊	soft-threshold init
Safe accumulator (Eq. 20)	P* = ⌈α+φ(α)+1⌉	unconstrained worst case

The one sentence to remember. The accumulator is a fixed-width budget; the ℓ₁-norm of each weight channel is what you spend against it; A2Q+ earns up to 4× more budget by zero-centering the weights, and starts training closer to optimal by projecting the pretrained weights onto the ℓ₁-ball instead of crushing them uniformly.

Where to go next

If you want to deploy this, the path is: pretrain in float → re-parameterize with weight-norm + zero-centering → initialize via ℓ₁-ball projection → quantization-aware fine-tune with round-to-zero and the STE → export to a low-precision-accumulator backend (Brevitas → FINN for FPGAs). The accuracy you keep at, say, 12 bits is the accuracy your hardware gets for free in power, area, and bandwidth.

What single idea unifies A2Q+'s two contributions?

Treat the accumulator as a budget and the weights' ℓ₁-norm as the currency — get a looser exchange rate (better bound) and a better opening balance (projection init) Use floating-point accumulation Make the network deeper

A2Q+: Accumulator-AwareWeight Quantization