EvoQ — Veanors

Chapter 0: The Problem

You have a trained network — say a ResNet — that works beautifully on the GPU you trained it on. Now someone wants it on a phone, or a doorbell camera, or a $3 microcontroller. The weights are stored as 32-bit floating-point numbers. There are tens of millions of them. That is too much memory, too much bandwidth, and too much energy for the target device.

The obvious fix is quantization: replace each 32-bit float with a small integer — 8 bits, 4 bits, maybe even 2. Fewer bits means a smaller model and cheaper arithmetic. An 8-bit model is 4× smaller than a 32-bit one. A 4-bit model is 8× smaller.

The core question: How few bits can each layer tolerate before the network's accuracy collapses — and can we find that answer without retraining the model and using only a handful of unlabeled images?

Why "just use 4 bits everywhere" is the wrong answer

The lazy approach is uniform quantization: pick one bit-width — say 4 — and apply it to every layer. It is simple, but it leaves enormous value on the table, for one stubborn reason:

Layers are not equally fragile. The first convolution that touches raw pixels, and the final classifier that produces logits, are often extremely sensitive — drop them to 4 bits and accuracy craters. But a wide middle layer with thousands of redundant channels might survive at 2 bits with no visible damage. Forcing a single bit-width on all of them means you either over-spend bits on the robust layers or destroy the fragile ones.

The combinatorial wall

So we want mixed precision: a possibly different bit-width per layer. But now count the choices. Suppose each of L layers can be 2, 4, or 8 bits. The number of assignments is 3^L.

Layers L	Bit choices	Configurations 3^L	If each took 1 sec to evaluate
10	{2,4,8}	59,049	~16 hours
20	{2,4,8}	~3.5 billion	~110 years
50 (a ResNet-50)	{2,4,8}	~7×10²³	longer than the age of the universe

You cannot try them all. You cannot even try a millionth of them. And every "evaluation" means running the quantized network on data to see how much accuracy you lost — not free. This is the wall EvoQ is built to get around.

The "limited data, no training" twist

One more constraint makes EvoQ's setting realistic. In post-training quantization (PTQ), you often do not have the full training set, the training pipeline, or the GPU-weeks to retrain. You have a finished model and maybe a few hundred unlabeled images. So every trick has to work cheaply, from little data, with no gradient-based fine-tuning.

Why is uniform quantization (one bit-width for all layers) usually suboptimal?

It is slower to run on hardware Layers differ in fragility — fragile ones break at low bits while robust ones waste bits at high precision It requires retraining the whole network

Chapter 1: The Key Insight

We are trapped between two facts: the search space is astronomically large (Chapter 0), and a brute-force search is impossible. EvoQ's answer is a two-part idea, and each part attacks one of those facts.

Part 1 — Sensitivity analysis

Measure, per layer, how much the network "hurts" when that layer alone is quantized to a low bit-width. This turns a blind search into an informed one: spend bits where they matter, save bits where they don't.

↓ feeds

Part 2 — Evolutionary search

Run a population of candidate bit-assignments through mutation, crossover, and selection. It evolves toward configs that meet the size budget with minimal accuracy loss — never needing to enumerate the whole space.

The insight that ties them together: Don't start the evolutionary search from random guesses. Seed it with sensitivity. Give fragile layers more bits and robust layers fewer bits in the initial population. The search then spends its effort polishing a good guess instead of crawling out of a random hole — dramatically fewer evaluations to a good answer.

Why each part is necessary

Sensitivity alone is not enough. Knowing each layer's fragility tells you a ranking, but not the exact joint assignment that hits a specific size budget. Sensitivities are measured one layer at a time; the real network's accuracy under combined quantization has interactions a per-layer score can't capture.

Evolution alone is not enough. A genetic search over 3⁵⁰ configurations, started from random noise, wastes thousands of expensive evaluations before it stumbles onto anything decent. The whole point of PTQ is to be cheap — a slow search defeats the purpose.

Together they are complementary: sensitivity gives a warm start and a bias for mutation; evolution finds the joint assignment and handles the interactions. That synergy — "sensitivity-guided evolutionary search" — is the whole name of the paper.

What you can verify from the abstract: EvoQ does mixed-precision PTQ with limited data, uses per-layer quantization sensitivity to make the evolutionary search efficient, optimizes bit allocation under a resource budget with no extra training/computation, and reports beating prior post-training methods. Everything else in this lesson is the standard, well-understood machinery behind those words — taught rigorously, with any concrete numbers clearly labeled illustrative.

What is the role of sensitivity in EvoQ's search?

It seeds and biases the evolutionary search so it starts from a good guess instead of random noise It replaces the search entirely — sensitivity gives the final answer directly It is used to retrain the network after quantization

Chapter 2: Quantization from Zero

Before we can talk about how many bits, we need to be precise about what quantizing a tensor actually does to the numbers. The idea is simple: take a continuous range of real values and snap each one to the nearest entry in a small grid.

Uniform affine quantization

For a weight tensor with values in [w_min, w_max], choose a bit-width b. That gives 2^b integer levels (e.g. b=4 → 16 levels). The step size (also called scale) is how far apart two adjacent grid points are:

s = (w_max − w_min) / (2^b − 1)

Here s is the scale (the gap between levels), b is the bit-width, and the denominator 2^b−1 is the number of gaps between the 2^b grid points. To quantize a value w, you round it onto the grid and back:

q(w) = s · round( (w − w_min) / s ) + w_min

Read it left to right: divide by the step to get an integer index, round to the nearest grid index, multiply back by s to return to real units, shift by w_min. The gap between w and q(w) is the quantization error — the noise we inject.

Bits halve the step, twice over: drop from 8 bits to 4 and the number of levels falls from 256 to 16 — a 16× coarser grid, so the step s grows 16× and the typical error grows with it. That is exactly why low bits are dangerous: error scales like s ∝ 2^−b.

See it: the same weights, fewer levels

Quantization grid — drag the bit-width

A bell-shaped cloud of real weights (gray) snapped onto a b-bit grid (orange ticks). Watch the error bars grow as you cut bits. The number under the canvas is the average absolute quantization error — the noise injected into this layer.

bit-width b 4

levels = 2^b = 16 · mean |error| = --

The size payoff

A layer with n weights at b bits costs n · b bits of storage. The whole-model size is just the sum over layers — and that sum is the budget the search must respect. We will track it precisely in Chapter 5.

If you go from 8-bit to 4-bit quantization, roughly what happens to the quantization step size s (and thus the error)?

It stays about the same It grows ~16× — levels drop from 256 to 16, so the grid is 16× coarser It halves

Chapter 3: Sensitivity — Which Layers Are Fragile?

Chapter 2 showed that low bits inject more noise. But the same amount of injected noise hurts different layers by wildly different amounts. Sensitivity is the answer to one clean question:

The sensitivity question: If I quantize this one layer to b bits and leave every other layer at full precision, how much does the network's behavior change?

A layer is sensitive if that single change causes a big shift in the outputs; it is robust if the outputs barely move. We will make "shift in outputs" precise in Chapter 4. For now, build the intuition.

Why some layers are fragile and others aren't

First layer. It transforms raw input (pixels) directly. Errors here propagate through every later layer, amplified. Usually very sensitive.
Last layer (classifier). Its outputs are the logits. A small perturbation can flip the argmax between two classes. Usually very sensitive.
Wide middle layers. Thousands of channels, lots of redundancy. One channel's error gets averaged away. Often robust — survive 2–4 bits comfortably.
Depthwise convolutions (in mobile nets). Few parameters per channel, no cross-channel averaging to hide error. Often surprisingly sensitive.

The bit-allocation principle that falls out of this: give more bits to sensitive layers and fewer bits to robust layers. That single rule, applied well, is the entire payoff of mixed precision — you keep accuracy where it's fragile and harvest compression where it's free.

See it: a per-layer sensitivity profile

Sensitivity profile of a (illustrative) 8-layer network

Each bar is one layer's sensitivity at the bit-width you choose — taller means more fragile. Notice the classic "U": the first and last layers spike, the middle dips. Lower the bit-width and every bar rises, but the fragile ones rise fastest. (Profile is illustrative, shaped like real ResNet/MobileNet measurements.)

bit-width b 4

A layer's sensitivity is high. What should the bit-allocation do with it?

Quantize it to the fewest bits to save the most memory Give it more bits — it's fragile, so spending precision here protects accuracy Remove the layer entirely

Chapter 4: Measuring Sensitivity from a Few Images

"How much does the network's behavior change?" needs a number. And the constraint is brutal: we have no labels and only a small calibration set — a few dozen to a few hundred unlabeled images. We cannot measure accuracy directly (no labels) and we cannot afford the full validation set. So we measure how the output distribution shifts.

Three standard sensitivity metrics

Let P be the network's output distribution (the softmax over classes) on the calibration data at full precision, and Q_ℓ,b the output when layer ℓ alone is quantized to b bits. The sensitivity is a distance between P and Q. Three metrics are standard (the closely-related literature evaluates exactly these kinds):

Metric	Formula (per sample, then averaged)	What it captures
KL divergence	Σ_i P_i log(P_i / Q_i)	How much the predicted class distribution shifts. Punishes confident-but-wrong moves heavily.
MSE / L2	Σ_i (P_i − Q_i)²	Raw squared output drift. Simple, cheap, no log issues.
SQNR (signal-to-quant-noise)	10 log₁₀( ‖P‖² / ‖P−Q‖² )	Error relative to signal size, in decibels. Higher = more robust; we negate it to make "bigger = more sensitive."

Symbols, in plain terms: P_i is the full-precision probability for class i; Q_i is the same probability after quantizing layer ℓ; the sum runs over all classes. KL is the divergence of the language-model-style "next token" sense — how surprised P is by Q.

Why a distribution distance, not accuracy? With no labels you can't compute accuracy. But you can compute the full-precision output and the quantized output on the same image and compare them. If the quantized layer barely perturbs the output distribution, accuracy will barely move either. This is the trick that lets sensitivity be measured from a tiny, unlabeled calibration set.

The data flow, with shapes

Calibration batch

X ∈ R^{[N, 3, 224, 224]} — N≈32 unlabeled images

→ full-precision forward

Reference P

P ∈ R^{[N, 1000]} — softmax outputs, computed once, cached

↓ for each layer ℓ, each bit b

Quantize ℓ→b, re-forward

Q_ℓ,b ∈ R^{[N, 1000]}, then restore ℓ to full precision

↓ distance

Sensitivity table

S[ℓ][b] = mean_N distance(P, Q_ℓ,b) — an L×B matrix, computed in O(L·B) forward passes, no gradients

That last box is the deliverable of this chapter: a small L × B table where entry S[\ell][b] says "how badly layer \ell hurts at b bits." It costs only L \cdot B forward passes — cheap — and every later step reads from it instead of re-running the network.

Worked example (illustrative numbers). Take layer 3 (a wide middle conv). On 32 images: full-precision mean output entropy is stable; quantizing it to 4 bits gives KL = 0.012, to 2 bits gives KL = 0.21. Take layer 0 (first conv): 4 bits gives KL = 0.18, 2 bits gives KL = 1.4. Same bit drop, ~7× more damage at the fragile layer. The table now knows layer 0 is fragile and layer 3 is robust — before any search has run.

Why does EvoQ measure a distribution distance (KL/MSE/SQNR) instead of validation accuracy?

Distribution distances are always more accurate than accuracy The calibration set is small and unlabeled, so accuracy can't be computed — but P vs Q drift can, and it tracks accuracy loss Because gradients are needed and accuracy has none

Chapter 5: The Search Space and the Budget

Now we formalize what the evolutionary search is actually searching over, and what "good" means. This is the contract between the search and the world.

A candidate is a genome

A single candidate solution is a vector — one bit-width per layer. We'll call it a genome (borrowing the genetic-algorithm word we'll need in Chapter 6):

g = [b₁, b₂, ..., b_L], b_ℓ ∈ {2, 4, 8}

Each gene b_\ell is the bit-width assigned to layer \ell. A genome for an 8-layer net might be [8,4,2,2,4,2,4,8] — full precision on the fragile ends, squeezed in the robust middle.

The two numbers every genome has

Size. If layer \ell has n_\ell weights, the model's quantized size is:

Size(g) = Σ_ℓ=1..L n_ℓ · b_ℓ (bits)

Cost / damage. We need a cheap proxy for accuracy loss. The natural one reuses the sensitivity table from Chapter 4 — sum the per-layer damage of the chosen bits:

Cost(g) ≈ Σ_ℓ=1..L S[ℓ][b_ℓ]

This is an approximation — it ignores cross-layer interactions — but it's nearly free to compute (just table lookups), so the search can evaluate millions of genomes with it. The true accuracy of promising genomes is checked occasionally with a real forward pass on calibration data.

The constraint that makes it a real optimization: minimize Cost(g) subject to Size(g) ≤ Budget. The budget is the device's memory limit (e.g. "model must fit in 2× less space than 8-bit"). Without it, the trivial answer is "8 bits everywhere." The budget forces real tradeoffs.

See it: build a genome by hand, watch the two numbers move

Genome builder — click a layer to cycle its bits (2→4→8)

Eight layers, sized like a real net (fat middle). Click any bar to change its bit-width. Watch Size and Cost trade off, and whether you're under the dashed budget line. Try to get Cost low while staying under budget — that's the search's job, by hand.

budget 55%

Size = --% of 8-bit · Cost = -- · feasible

What does the size budget constraint accomplish?

It makes the search faster by reducing the number of layers It forces a real tradeoff — without it the optimum is just "8 bits everywhere" It guarantees zero accuracy loss

Chapter 6: Evolution — The Genetic Operators

We have a space of genomes and a way to score them. An evolutionary (genetic) algorithm finds good genomes by mimicking natural selection: keep a population of candidates, let the good ones reproduce, occasionally mutate, and repeat. No gradients, no enumeration.

The four operators

1. Selection

Rank the population by fitness (low Cost, within budget). Keep the top performers — they become parents. Bad genomes die.

↓

2. Crossover

Take two parents, splice their genomes: child = [parent A's first half] + [parent B's second half]. Combines good "sub-assignments" from each.

↓

3. Mutation

With small probability, randomly bump one layer's bit-width up or down. Injects new variation so the search doesn't get stuck.

↓

4. Repair / evaluate

If a child violates the budget, repair it (drop bits on robust layers until feasible). Score the survivors. Go back to step 1.

Fitness: folding the budget in

The search maximizes fitness. The clean way to honor the constraint is a penalty on infeasible genomes:

Fitness(g) = −Cost(g) − λ · max(0, Size(g) − Budget)

Here \lambda is a large penalty weight. The first term rewards low damage; the second term punishes any overshoot of the budget. A feasible, low-cost genome has high fitness; an over-budget genome is crushed by the penalty so it loses the selection step.

Why crossover and mutation are different tools. Crossover is exploitation: it recombines pieces that already work, jumping to new but plausible genomes. Mutation is exploration: it makes small random moves to discover options crossover would never reach. Too much mutation and the search is random noise; too little and it stalls on a local optimum. Real GAs tune the mutation rate carefully.

Why this beats brute force and beats pure greedy

vs brute force: a population of, say, 50 genomes over 40 generations evaluates ~2,000 candidates — not 3⁵⁰.
vs greedy (lower the most-robust layer one at a time): greedy commits early and can't undo a bad choice. Crossover lets the search escape the order it started in; mutation lets it back out of dead ends.

In a genetic algorithm, what is the difference between crossover and mutation?

Crossover recombines existing good genomes (exploit); mutation makes small random changes (explore) They are the same operation with different names Crossover deletes layers; mutation adds layers

Chapter 7: One Generation, by Hand

Let's run a single generation on a tiny 4-layer network so every operator is concrete. All numbers below are illustrative but internally consistent.

The setup

Four layers with weight counts n = [100, 400, 400, 100] (fat middle, like a real net). Bit choices \{2,4,8\}. Budget: model must use ≤ 50% of the all-8-bit size. The all-8-bit size is (100+400+400+100)\times 8 = 8000 bits, so Budget = 4000 bits.

From Chapter 4's sensitivity table (illustrative), the damage S[\ell][b] is:

Layer	S at 4b	S at 2b
L1 (first, fragile)	0.30	1.50
L2 (wide, robust)	0.04	0.20
L3 (wide, robust)	0.05	0.22
L4 (last, fragile)	0.35	1.60

Step 1 — the starting population (2 genomes)

Genome	bits	Size = Σ n·b	Feasible? (≤4000)	Cost = Σ S
A	[8, 4, 4, 8]	800+1600+1600+800 = 4800	NO (4800>4000)	0+0.04+0.05+0 = 0.09
B	[4, 2, 2, 4]	400+800+800+400 = 2400	YES	0.30+0.20+0.22+0.35 = 1.07

A is low-cost but over budget; B is feasible but pricey (it cut the fragile layers). Neither is great. Evolution mixes them.

Step 2 — crossover (splice A's front, B's back)

Child C = [A₁, A₂ | B₃, B₄] = [8, 4, 2, 4].

Size = 800 + 1600 + 800 + 400 = 3600 ≤ 4000 → feasible
Cost = S[1][8] + S[2][4] + S[3][2] + S[4][4] = 0 + 0.04 + 0.22 + 0.35 = 0.61

C is already better than B (cost 0.61 vs 1.07) and feasible. Crossover combined "keep L1 precise" from A with "squeeze L3" from B.

Step 3 — mutation

Mutate C at layer 4: 4→8 bits (give the fragile last layer more precision). Child C' = [8, 4, 2, 8].

Size = 800 + 1600 + 800 + 800 = 4000 ≤ 4000 → exactly on budget
Cost = 0 + 0.04 + 0.22 + 0 = 0.26

The payoff in one line. We went from the best starting genome (B, cost 1.07) to C' (cost 0.26) in a single generation — a 4× reduction in predicted damage, still inside budget, by spending precision exactly where sensitivity said it mattered (L1 and L4) and harvesting it where it didn't (L2, L3). That is the entire algorithm in miniature.

In the worked example, why did mutating L4 from 4→8 bits lower the cost so much despite increasing size?

L4 is a fragile layer (high S at low bits), so more bits there sharply cuts its damage — and it still fit the budget More bits always lowers cost regardless of the layer Because L4 is the smallest layer so it doesn't matter

Chapter 8: Sensitivity-Guided Seeding — the Showcase

This is the heart of EvoQ, made live. We run the full evolutionary search over per-layer bit configs for an illustrative network, and you control the one knob the paper is named for: does the initial population get seeded by sensitivity, or start random?

The plot is the accuracy–size plane — every dot is one genome. Up-and-left is the dream (small model, high accuracy). The frontier of best-possible tradeoffs is the Pareto front: genomes where you can't improve one axis without sacrificing the other. Watch the population evolve toward that front. Then toggle seeding off and watch how many more generations a random start needs.

EvoQ search — population evolving toward the accuracy/size Pareto front

● population · ● best-so-far · — Pareto front discovered · | budget line. Press Step for one generation, Run to animate, Reset to restart. Flip seeding and compare generation counts to reach the front. (Accuracy proxy is illustrative, built from a U-shaped sensitivity profile.)

population 30

mutation rate 0.20

budget (% of 8-bit) 50%

generation 0 · best feasible: acc --% @ size --% · evaluations 0

What to look for: with seeding ON, the very first generation's cloud already hugs the upper-left near the budget line — the warm start put fragile layers at high bits immediately. With seeding OFF, generation 0 is a random smear and it takes many more generations (and evaluations) to crawl up to the same front. That gap is the contribution: sensitivity doesn't change where the optimum is, it changes how fast you reach it — which is everything when each evaluation is a forward pass.

Try to break it. Set mutation rate to 0 — the population freezes into whatever crossover can reach and stalls early (no exploration). Set it to 0.6 — the search turns into a random walk and the best-so-far jitters instead of converging. The sweet spot in the middle is why real GAs tune this knob. Tighten the budget to 30% and watch accuracy fall: with so few total bits, even the sensitivity-guided search can't protect every fragile layer.

With seeding ON vs OFF, what mainly changes in the search?

Seeding moves the location of the optimal Pareto front Seeding reaches a near-optimal front in far fewer generations/evaluations — the front itself is the same Seeding eliminates the need for mutation and crossover

Chapter 9: The Code — the Evolutionary Loop

Here is the whole method as runnable Python. It reads the precomputed sensitivity table, seeds the population with it, and runs the genetic loop. Read it as the executable summary of Chapters 4–8.

python
import numpy as np

# --- Inputs from Chapter 4 / Chapter 5 ---
# n[l]      : weight count of layer l        (size accounting)
# S[l][b]   : sensitivity/damage of layer l at b bits (lookup table)
# BITS      : allowed bit-widths, e.g. [2, 4, 8]
BITS = [2, 4, 8]

def size_bits(g, n):
    return sum(n[l] * g[l] for l in range(len(g)))

def cost(g, S):
    # cheap accuracy-loss proxy: sum of per-layer damage
    return sum(S[l][g[l]] for l in range(len(g)))

def fitness(g, n, S, budget, lam=1e3):
    over = max(0, size_bits(g, n) - budget)
    return -cost(g, S) - lam * over   # penalize budget violation

def seed_genome(n, S, budget):
    # SENSITIVITY-GUIDED START: fragile layers -> more bits.
    # rank layers by how much they hurt at the lowest bit-width
    frag = sorted(range(len(n)), key=lambda l: -S[l][BITS[0]])
    g = [BITS[0]] * len(n)        # start everyone at the cheapest bits
    for l in frag:                  # spend bits on the fragile first
        for b in reversed(BITS):     # try high bits down to low
            g[l] = b
            if size_bits(g, n) <= budget: break
            g[l] = BITS[0]            # can't afford -> stay low
    return g

def mutate(g, rate):
    g = list(g)
    for l in range(len(g)):
        if np.random.rand() < rate:
            g[l] = np.random.choice(BITS)   # explore
    return g

def crossover(a, b):
    cut = np.random.randint(1, len(a))
    return a[:cut] + b[cut:]              # exploit: splice parents

def evoq(n, S, budget, pop=30, gens=40, mut=0.2, seeded=True):
    L = len(n)
    # --- initialize population ---
    if seeded:
        base = seed_genome(n, S, budget)
        P = [mutate(base, 0.15) for _ in range(pop)]  # jitter the warm start
    else:
        P = [[np.random.choice(BITS) for _ in range(L)] for _ in range(pop)]
    for _ in range(gens):
        P.sort(key=lambda g: -fitness(g, n, S, budget))   # 1. select
        elite = P[:pop // 2]                              # survivors = parents
        children = []
        while len(children) < pop - len(elite):
            a, b = elite[np.random.randint(len(elite))], elite[np.random.randint(len(elite))]
            child = mutate(crossover(a, b), mut)            # 2-3. cross + mutate
            children.append(child)
        P = elite + children                              # 4. next gen
    P.sort(key=lambda g: -fitness(g, n, S, budget))
    return P[0]   # best per-layer bit-width assignment

Concept → realization, line by line. seed_genome is Part 1 of the key insight: everyone starts at the cheapest bits, then we walk the fragile-first ranking and raise bits while the budget allows. fitness folds the budget in as a penalty so over-budget genomes lose selection. crossover exploits, mutate explores. The seeded flag is the exact toggle from the Chapter 8 showcase. Swap the cheap cost() proxy for a real calibration-set forward pass on the top-k genomes each generation and you have the production version.

In seed_genome, why iterate layers in order of most fragile first?

So the limited bit budget is spent on the layers that need precision most, before it runs out Because fragile layers have fewer weights It makes the code run faster

Chapter 10: Failure Modes & What the Paper Doesn't Say

Every method has edges where it breaks. Knowing them is what separates understanding from memorizing. Here are the honest limits of sensitivity-guided evolutionary PTQ.

1. The per-layer cost proxy ignores interactions

Cost(g) = \Sigma S[\ell][b_\ell] assumes layer damages add. They don't always. Quantizing two adjacent layers can compound — the second layer receives already-corrupted activations and amplifies them. The fix is to occasionally validate promising genomes with a real forward pass on calibration data, not just the table. EvoQ's "limited data" is exactly for these spot-checks.

2. Calibration set too small or unrepresentative

Sensitivity is measured on a handful of images. If those images don't cover the input distribution (e.g. all daytime photos for a model that also sees night), the sensitivity table is wrong, and the search confidently optimizes for the wrong thing. Garbage calibration → garbage bit allocation.

The silent failure: a bad calibration set doesn't crash — the search still runs, still reports a "feasible, low-cost" genome. The damage only shows up later on real, diverse inputs. Always validate the final config on held-out data if you have any.

3. Hardware doesn't support arbitrary bit-widths

The search happily proposes [8,2,4,2,8,...]. But many accelerators only run 8-bit or 4-bit kernels efficiently; a 2-bit or 3-bit layer may fall back to slow emulation, erasing the speed gain. The bit set \{2,4,8\} must match what the target hardware actually accelerates, or "smaller" doesn't mean "faster."

4. PTQ has a floor; QAT goes lower

Because EvoQ does not retrain, it can only reallocate the precision the trained weights already tolerate. At very aggressive budgets, quantization-aware training (QAT) — which fine-tunes weights to be robust to quantization — can reach bit-widths PTQ cannot. EvoQ trades that last bit of accuracy for not needing the training pipeline, full data, or GPU-weeks. That trade is the value proposition, and it's the right call when retraining isn't available.

Situation	Symptom	Mitigation
Layer interactions	Predicted cost < real accuracy loss	Validate top genomes with real forward pass
Tiny/biased calibration	Looks great, fails on real data	Larger, representative calibration set
Unsupported bit-widths	Smaller but not faster	Restrict BITS to hardware-native widths
Extreme budget	Accuracy floor hit	Use QAT, or relax the budget

Why might a "smaller" mixed-precision model from EvoQ not actually run faster on real hardware?

Because mixed precision always increases compute Because the search uses too much memory The accelerator may not have efficient kernels for the chosen bit-widths, falling back to slow emulation

Chapter 11: Connections & Cheat Sheet

EvoQ sits at the intersection of three big ideas: quantization, search/optimization, and the realities of deploying models without retraining. Here's where it fits and what to remember.

How it relates to neighboring methods

Method	How bits are chosen	Needs training?	Tradeoff
Uniform PTQ	One bit-width for all layers	No	Simplest; wastes/destroys precision unevenly
HAWQ (Hessian-aware)	Hessian eigenvalues rank sensitivity	No (PTQ) / yes (QAT variants)	Principled sensitivity, heavier to compute
HAQ (RL-based)	Reinforcement-learning agent picks bits	Trains a policy	Powerful but expensive to set up
EvoQ	Sensitivity-seeded evolutionary search	No	Cheap, limited data, gradient-free; PTQ accuracy floor
QAT	Often uniform; weights fine-tuned	Yes (full pipeline)	Best low-bit accuracy; needs data + compute

The cheat sheet — every symbol in one place

Symbol	Meaning	Plain analogy
b_ℓ	bit-width of layer ℓ	how fine the ruler is for that layer
s = range / (2^b−1)	quantization step (scale)	gap between ruler ticks
g = [b₁..b_L]	genome: a full bit assignment	one candidate recipe
S[ℓ][b]	sensitivity table	how much pain layer ℓ feels at b bits
Size(g) = Σ n_ℓb_ℓ	model size in bits	total luggage weight
Cost(g) = Σ S[ℓ][b_ℓ]	accuracy-loss proxy	total predicted pain
Fitness = −Cost − λ·over	what the GA maximizes	low pain, inside the bag
KL / MSE / SQNR	distance(P, Q) metrics	how far outputs drifted

The recipe, end to end

1. Calibrate

Run a few unlabeled images at full precision → reference outputs P

↓

2. Profile

For each layer×bit, quantize alone → distance(P,Q) → sensitivity table S

↓

3. Seed

Build a warm-start genome: fragile layers high bits, robust layers low bits, within budget

↓

4. Evolve

Selection → crossover → mutation → repair/evaluate, repeated until converged

↓

5. Deploy

Quantize each layer to its evolved bit-width — smaller model, no retraining

Related lessons on Engineermaxxing

• Transformer — the kind of large model that most needs compression to deploy
• Veanors hub — more paper walkthroughs, including other Model Compression methods
• MDPs & RL — the framework behind RL-based bit-allocation methods like HAQ

The mastery test. You should now be able to: (1) explain why one bit-width for all layers is wasteful; (2) measure per-layer sensitivity from unlabeled data; (3) write the size and cost of any genome; (4) run one generation of the GA by hand; and (5) say exactly why seeding the search with sensitivity makes it converge faster without moving the optimum. If you can do those five, you can implement EvoQ from scratch.

"Spend your bits where they're scarce, save them where they're free — and let the right search find the line between."

— the one-sentence soul of mixed-precision quantization

EvoQ: Mixed-Precision Quantizationvia Sensitivity-Guided Evolutionary Search