PowerQuant: Automorphism Search
for Non-Uniform Quantization

Chapter 1: The Key Insight

Two ideas were already on the table when this paper landed. Uniform quantization: evenly-spaced levels, maps multiply-to-multiply (hardware loves it), but a poor fit to bell-shaped weights. Logarithmic quantization: levels bunched near zero (great fit to bell-shaped weights!), but it secretly turns multiplications into bit-shifts — which need special hardware and break on most chips.

PowerQuant asks a sharper question. What is the full space of non-uniform grids that keep multiplication as multiplication? If we never leave that space, we get the accuracy benefit of non-uniformity AND the hardware compatibility of uniform — no bit-shifts, no lookup tables.

The three-word fix

Each piece of "search over automorphisms" solves a specific failure of the alternatives:

What the whole method is, in one breath

PowerQuant is: pick exponent a by minimizing the per-layer reconstruction error (no data needed) → transform each weight by w→sign(w)·|w|^a → quantize that with a uniform grid → at inference, raise activations to the power a, multiply in integers as usual, then undo the power. We will build every one of those pieces from scratch and watch the actual numbers flow.

Chapter 2: Uniform Quantization, From Scratch

Before we can improve on uniform quantization we have to be able to do it by hand. Let's quantize a real weight tensor to b bits, symmetrically (signed, centered on zero — what the paper uses for weights).

The recipe has three numbers: a scale (how big one integer step is, in real units), the rounded integer code, and the reconstructed de-quantized value.

The symmetric uniform map, defined

Let W be the weight tensor and b the bit-width. A signed b-bit integer ranges over [−(2^b−1−1), 2^b−1−1]. For b=4 that is [−7, 7] — 15 usable levels.

Where s is the scale — think of it as "real-world meters per integer tick." max|W| is the largest-magnitude weight (it defines the range we must cover). W_int is the stored integer code. Ŵ ("W-hat") is what you get back when you de-quantize — almost certainly not equal to the original W, and that gap is the error we fight all lesson.

See it: drag the scale, watch a weight snap to its grid

Below is a single layer's worth of weights (a Gaussian, like real networks). The horizontal ticks are the 15 uniform levels for 4-bit. Drag the highlighted weight — watch which level it snaps to and how the reconstruction error changes. Notice how lonely the levels are out in the tails and how starved they are at the peak.

Chapter 3: The Mismatch — Why Uniform Hurts

You just felt the problem in the last simulation. Neural-network weights are not spread uniformly. Layer after layer, they pile up into a sharp bell shape centered at zero: a huge mass of tiny weights, a thin tail of large ones. Uniform quantization spends its precious 16 levels evenly across the whole range — so the tails, where almost no weights live, get just as many levels as the dense peak.

That is exactly backwards. The peak is where most of the action is. Crushing a thousand small weights to a coarse grid does far more total damage than mis-rounding a handful of large outliers.

Side-by-side: the same weights, two grids

Here is the comparison that motivates the entire method. Same Gaussian weights, same 4-bit budget. Left: a uniform grid. Right: a grid that bunches near zero (a power grid with a<1, previewing Chapter 5). Toggle the bit-width and watch the total reconstruction error. The power grid wins precisely because it puts its ticks where the histogram is tall.

Chapter 4: Automorphisms — The Algebra That Picks the Grid

So we want a non-uniform grid. But not any non-uniform grid — only ones that keep multiplication intact, because matrix multiply (integer multiply on real hardware) is the operation a network spends 99% of its time on. Let's make "keeps multiplication intact" precise, and watch it force our hand.

Manufacturing the requirement

Quantization is a map Q from real values to a discrete space, with a de-quantization inverse Q⁻¹. For the quantized network to compute the same kind of arithmetic, we need: whatever the quantized values do under the quantized multiply, de-quantizing must recover the true product. Setting the quantized multiply to be ordinary multiplication ×, the requirement is:

Read it slowly. "Quantize, then multiply" must equal "multiply, then quantize." A map that satisfies this — and is invertible (bijective, so Q⁻¹ exists) — is by definition a group automorphism of (R^*₊, ×): the positive reals under multiplication. ("Automorphism" = a structure-preserving map from a set to itself. Here the structure being preserved is the multiplication.)

Why power functions, derived in two lines

Here is the classic argument so it doesn't appear by magic. Take logs: let g(u) = ln Q(e^u). The multiplicative law Q(xy)=Q(x)Q(y) becomes the additive law g(u+v) = g(u)+g(v) — Cauchy's functional equation. Its only continuous solutions are linear: g(u) = a·u. Undo the logs: ln Q(x) = a·ln x, so Q(x) = x^a. The exponent a is the slope of the only line that fits — that is where our single free parameter comes from.

See it: turn the dial, watch the grid breathe

The map x→x^a sends a uniform grid in the warped space back to a non-uniform grid in the original space. Drag the exponent a and watch where the levels land. a=1 is plain uniform; a<1 bunches levels near zero (fine for small weights); a>1 bunches them near the max.

Chapter 5: PowerQuant — The Operator (SHOWCASE)

Now we assemble the full operator. The recipe is "warp, then quantize uniformly, then un-warp." Because the warp is |w|^a, the only new ingredient over uniform quantization is raising magnitudes to a power.

The quantization operator, every symbol defined

Walk through it. |W|^a — warp every magnitude by the power (this is the automorphism). max|W|^a — the largest warped magnitude, used to normalize into [−1,1]. (2^b−1−1) — scale up to fill the signed integer range (7 for 4-bit). sign(W) — store the sign separately so we can warp magnitudes safely (powers of negatives are trouble). round — snap to the nearest integer level.

And the de-quantization that recovers a real value (un-warp with the inverse power 1/a):

SHOWCASE: the PowerQuant grid, fully interactive

This is the paper's Figure 2 made live. A layer of Gaussian weights (the histogram), the b-bit power grid drawn as vertical ticks, and a live reconstruction-error readout. Drag a to reshape the grid; toggle bits; change the weight distribution to a heavy-tailed one to see when a<1 stops helping. Find the a that minimizes the error — you are doing by hand what Nelder-Mead does in Chapter 6.

Chapter 6: Finding a* — Data-Free Calibration

We have one knob, a. How do we set it without data? The paper's move: use the reconstruction error — how much the weights change when you quantize and de-quantize them — as a stand-in for accuracy loss. Weights are all we have, and weights are all this needs.

The objective, defined

Symbol by symbol: F is the network, L its number of layers, W_l the weights of layer l. The inner term is the original weights minus the round-tripped (quantized then de-quantized) weights — the per-weight damage. ‖·‖_p is the L_p norm (the paper uses p=2, ordinary Euclidean length). Sum over all layers. We pick a^* = argmin_a ε(F,a).

Nelder-Mead: optimization without derivatives

Nelder-Mead is a derivative-free simplex search. In 1-D it just keeps a bracket of points, evaluates the error, and reflects/contracts toward the minimum — perfect here because round makes the gradient zero (or undefined) almost everywhere. It typically converges in a handful of evaluations.

See it: watch the solver hunt for a*

The blue curve is reconstruction error ε(a) over the real network's weights; the faint orange curve is the (hidden, data-needing) accuracy — note they bottom/peak at the same place. Press Step to advance the search one probe at a time, or Run to animate it converging. Change the bit-width and the optimum a^* shifts — which is exactly why the paper re-optimizes a per network and per bit-width.

Chapter 7: Inference — Keeping Integer Multiply Intact

Here is the payoff of choosing automorphisms. At inference time, the quantized layer must still be ordinary integer matrix multiplication — that's the whole point. So where does the power go? Onto the activations, fused into the activation function, with essentially no overhead.

The data flow, traced with shapes

For a layer with input activations X and weights W, the simulated inference (Algorithm 2 in the paper) is:

Handling signed activations (SiLU, GeLU)

ReLU outputs are positive, so X^a is fine. But Transformers and EfficientNets use SiLU/GeLU, which go negative — and powers of negatives break the automorphism. The fix is asymmetric quantization: shift activations up by a constant so they're positive. SiLU is provably bounded below by C_SiLU=0.27846, GeLU by C_GeLU=0.16997. Shift by that constant (the zero-point), warp, and the extra C·W bias term is absorbed cheaply — standard asymmetric-quant bookkeeping.

Chapter 8: The Core Insight as Runnable Code

The entire method fits in a few dozen lines of NumPy. Here is the weight quantizer, the de-quantizer, the data-free objective, and the 1-D search — exactly the loop from Algorithm 1. Read it as the executable summary of everything above.

pythonimport numpy as np
from scipy.optimize import minimize_scalar

def powerquant(W, a, b=4):
    """Quantize weight tensor W with power exponent a to b bits."""
    qmax = 2**(b-1) - 1                 # e.g. 7 for 4-bit
    sgn  = np.sign(W)
    mag  = np.abs(W)
    warped = mag ** a                   # the automorphism: |w|^a
    s = warped.max() / qmax             # scale in WARPED space
    W_int = np.round(warped / s) * sgn  # integer codes (with sign)
    return W_int, s, sgn

def dequant(W_int, s, a):
    """Recover real weights: un-warp with inverse power 1/a."""
    warped = np.abs(W_int) * s          # back to warped magnitudes
    return np.sign(W_int) * (warped ** (1.0 / a))

def recon_error(W, a, b=4):
    """Data-free objective: how much do the weights change? (L2)"""
    W_int, s, _ = powerquant(W, a, b)
    W_hat = dequant(W_int, s, a)
    return np.linalg.norm((W - W_hat).ravel(), ord=2)

# ---- data-free calibration: find a* (Nelder-Mead style 1-D search) ----
W = np.random.randn(512, 512).astype(np.float32) * 0.1   # a layer's weights
res = minimize_scalar(lambda a: recon_error(W, a, b=4),
                      bounds=(0.2, 2.0), method='bounded')
a_star = res.x
print(f"optimal exponent a* = {a_star:.3f}")   # ~0.55 in practice

# compare to plain uniform (a = 1)
print("uniform  err:", recon_error(W, 1.0,    b=4))
print("power    err:", recon_error(W, a_star, b=4))   # lower!

The inference-time activation power (integer-friendly)

pythondef quant_layer_forward(x, W_int, sW, a, qmax=7):
    # x is the (positive, ReLU) activation tensor [B, C_in]
    x_warped = x ** a                       # step 1: warp activations
    sx = x_warped.max() / qmax
    x_int = np.round(x_warped / sx)         # step 2: to integers
    o_int = x_int @ W_int.T                 # step 3: PLAIN integer matmul
    o = (o_int * sx * sW)                   # rescale accumulator
    return o ** (1.0 / a)                   # step 4: un-warp for next layer
    # the only extra ops vs uniform quant are ** a and ** (1/a) on activations

Chapter 9: Experiments — How Much It Buys

The headline is the W4/A4 column on ResNet-50. Uniform quantization there is a disaster (54.68%); PowerQuant recovers most of the loss (70.29%) — and it is still fully data-free, no fine-tuning. At W8/A8 PowerQuant is lossless (76.15%, equal to FP32), matching or beating every data-free competitor including methods that do generate synthetic data.

The numbers (ResNet-50, ImageNet)

Two facts to sit with. (1) PowerQuant beats SQuant — the prior best data-free operator — by +1.93 points at W4/A4, and beats the synthetic-data GDFQ by +14.88. (2) It can only ever beat uniform, never lose to it, because uniform (a=1) is inside its search space — at worst the optimizer returns a=1.

See it: PowerQuant vs the field, swept across bit-widths

This reconstructs the paper's Figure 1. Pick a bit-width regime and watch the accuracy bars. The gap between PowerQuant and uniform/log widens dramatically as bits drop — non-uniformity matters most when levels are scarce.

Chapter 10: Connections & Cheat Sheet

PowerQuant sits at a clean intersection: it borrows the fit of non-uniform/log quantization and the hardware story of uniform quantization, by recognizing that "preserve multiplication" is an algebraic constraint with a one-parameter solution.

Where it fits in the landscape

Cheat sheet — every equation

Related lessons on Engineermaxxing

• Lloyd-Max quantization — the optimal scalar quantizer; PowerQuant trades optimality for a hardware-preserving closed form.
• Pruning — the "small weights don't matter" assumption PowerQuant directly challenges.
• LLM inference — where low-bit quantization is now mandatory for deployment.
• LLM deployment — the edge-device pressure that makes data-free PTQ valuable.