A Survey of Neural Network
Compression

Chapter 1: The Map of Compression

A survey's real gift is a taxonomy — a single picture that tells you where each technique lives and why they're different. There are five families, and they attack the parameter count from genuinely different angles.

The five families at a glance

Two cross-cutting distinctions that decide everything

Before any specific method, the survey hammers two axes that recur in every family:

Chapter 2: Weight Sharing

Start with the simplest idea. A weight matrix has, say, a million distinct floating-point values. What if many of those weights could share a single value? Then you only store the few shared values plus a tiny index per weight saying which shared value it uses.

This is the cheapest form of compression because it doesn't change the network's shape at all — same layers, same forward pass — it just reduces how many distinct numbers you must store.

Clustering-based sharing (the workhorse)

Run k-means on all the weights of a layer. Pick k cluster centroids. Replace every weight with the centroid of its cluster. Now instead of a million 32-bit floats you store: k centroids (the "codebook") plus a million log₂(k)-bit indices.

Worked example. A layer has 1,000,000 weights at FP-32 = 4,000,000 bytes. Cluster into k = 16 centroids:

The codebook is negligible; the win comes from 4 bits per index instead of 32 bits per float. This is exactly the same lever quantization pulls (Ch 7) — sharing is "learned, non-uniform quantization."

Other flavors of sharing

• Parameter hashing (HashedNets): a hash function maps each weight connection into one of K buckets; all weights in a bucket share one value. Backprop sums gradients into the shared value. No clustering needed — sharing is decided before training by a random hash.
• Tied layers: reuse the same weight matrix across multiple layers. Universal Transformers and ALBERT tie all Transformer blocks together, slashing parameters while keeping depth. Tying input and output word embeddings saves parameters proportional to vocabulary size.
• Recursive reuse: feed a layer's output back into its own input (a recurrent conv block). More compute, same parameters — but watch for vanishing/exploding gradients.

The grey dots are raw weights. Each colored band snaps its weights to one shared centroid (the vertical line). Fewer centroids = more compression but bigger error (the grey-to-line gaps). This is the size-vs-error tradeoff every compression method faces.

Chapter 3: Pruning — Removing Weights

If sharing reuses values, pruning deletes them. The survey calls pruning "perhaps the most commonly used technique." The premise: most weights in a trained network are nearly useless — set them to exactly zero and the network barely notices.

There's even a neuroscience parallel the survey draws: as humans learn, the brain prunes synaptic connections, keeping the salient ones. The key word is salient — random pruning hurts; selective pruning works.

Magnitude-based pruning (MBP) — the default

The simplest rule that actually works: the smaller |w|, the less it matters. Set a threshold γ; any weight with |w| < γ becomes zero. Or pick a target sparsity (say 90%) and zero out the 90% of weights closest to zero.

Then retrain (fine-tune the survivors) to recover the accuracy lost. Iterative MBP repeats: prune a little, retrain, prune more, retrain — until the size/accuracy tradeoff is met. Iterating beats one-shot pruning because each retraining lets the network compensate.

Pruning via regularization (push weights toward zero first)

Instead of pruning whatever happens to be small, train the network to make weights small. Add an L2 penalty:

But the survey flags a subtle flaw: plain L2 decays all weights at the same exponential rate and over-penalizes large ones. Weigend et al.'s fix uses a saturating penalty that approximates "count of non-zero weights":

For small |w| this term ≈ w², penalizing it; for large |w| it saturates to 1, so big weights aren't crushed. The derivative 2w/(1+w²)² stops pulling hard on already-large weights — exactly what you want when large weights are the important ones.

The lottery ticket twist: prune before training

Each square is a weight; brighter = larger |w|. Pruned weights go dark. Watch how global mode keeps the dense, salient column while gutting the weak one — per-layer mode forces equal sparsity everywhere, killing useful weights in the strong column.

Chapter 4: Second-Order Pruning

Magnitude is a crude proxy. A weight can be small yet important (a tiny lever on a sensitive output), or large yet redundant. The right question isn't "how big is w?" but "how much does the loss change if I delete w?" That is loss-sensitivity pruning, and the survey's centerpiece here is Optimal Brain Damage.

Manufacturing the need: a Taylor expansion of the loss

We want δL, the change in loss when we perturb the weights by δw. Expand L around the trained weights with a Taylor series:

Define every symbol with its plain meaning:

• g_i = ∂L/∂w_i — the gradient. How steeply the loss rises if you nudge w_i.
• h_ij = ∂²L/∂w_i∂w_j — the Hessian. The curvature: how the slope itself changes. h_ii is the curvature along w_i alone.
• δw_i — the perturbation. To delete weight i we set δw_i = −w_i (drive it to zero).

LeCun's three assumptions (this is the derivation)

OBD makes the messy expansion tractable with three honest approximations:

What survives is beautifully simple:

To delete weight i, δw_i = −w_i, so its saliency — the loss damage from removing it — is:

Worked example: when magnitude lies

Two weights. w₁ = 0.10 with curvature h₁₁ = 50. w₂ = 0.30 with curvature h₂₂ = 1.

Magnitude pruning and OBD make opposite decisions here. OBD is right: deleting w₁ costs 0.25 of loss; deleting w₂ costs only 0.045. The curvature carried the information magnitude missed.

Each dot is a weight at position (|w|, curvature). Toggle the ranking: MBP prunes by the x-axis alone (leftmost dots). OBD prunes by the product ½·h·w² (the contour). Notice the small-but-curved weights OBD rescues from the chopping block.

Chapter 5: Low-Rank Decomposition

Pruning makes the grid sparse. Decomposition makes it small by exploiting a different kind of redundancy: a big matrix often has low rank — its rows are nearly linear combinations of a few basis rows. If so, you can factor it into the product of two thin matrices and store far fewer numbers, with the grid staying dense (so it's a real hardware speedup).

The arithmetic of why this saves so much

A fully-connected layer is a matrix multiply W·x where W is m×n. Storing W costs m·n parameters; computing W·x costs m·n multiply-adds. Approximate W ≈ U·V where U is m×r and V is r×n, with rank r ≪ min(m,n):

Now storage is m·r + r·n = r·(m+n), and W·x = U·(V·x) costs r·(m+n) FLOPs too. The ratio:

Worked example: a 1024×1024 layer (1,048,576 params). Pick rank r = 64:

Tensors, not just matrices

Conv layers and attention are 4-D / higher-order tensors, not flat matrices, so the survey covers tensor decompositions (Tucker, CP, Block-Term / Tensor-Train). The CNN trick: a k×k convolution over C channels is a 4-D tensor that decomposes into a sequence of cheaper convs — e.g. a filter decomposition (separable spatial convs) and a channel-wise decomposition (1×1 convs that mix channels). MobileNet's depthwise-separable convolution is exactly this idea baked into the architecture from the start.

The bars are singular values (energy), sorted. The shaded region is what rank-r keeps; the rest is the reconstruction error. A fast-decaying spectrum means a small r already captures most energy — that's when low-rank compression pays off.

Chapter 6: Knowledge Distillation

The four families so far all edit the existing grid. Distillation does something different and clever: it trains a brand-new small network (the student) to imitate the big trained network (the teacher). The student inherits the teacher's behavior without ever being big.

Manufacturing the need: why not just train the student on the labels?

You could train the small network directly on the one-hot ground-truth labels. It works worse. The reason is that the teacher's output is richer than a hard label. When a teacher classifies a "2", it might say 90% "2", 8% "7", 2% "3" — encoding that 2s look more like 7s than 3s. Those relative probabilities across wrong classes are the "dark knowledge." One-hot labels throw it all away.

The distillation loss (Hinton et al.)

Train the student with a blend of two losses:

Symbol by symbol:

• H(y, y^S) — ordinary cross-entropy between the student output y^S and the true one-hot label y. The "get it right" term.
• z^T, z^S — the teacher's and student's logits (pre-softmax). φ is the softmax.
• ρ (temperature) — softens both distributions to expose dark knowledge. Same ρ for teacher and student.
• KL(·‖·) — pulls the student's soft distribution toward the teacher's. The "mimic the teacher" term.
• α — balances the two; ρ² rescales the soft-target gradients (they shrink by ~1/ρ² when softened, so multiplying by ρ² restores their magnitude relative to the hard-label term).

The student learns both the answer (hard labels) and the teacher's whole worldview (soft targets). Buciluǎ et al. did the original version (training a small net on a big ensemble's pseudo-labels); Hinton et al. formalized the temperature-softmax recipe.

Same fixed teacher logits, different ρ. At ρ=1 the softmax is spiky (one-hot-ish) and the wrong-class probabilities — the dark knowledge — are invisible. Raise ρ and those small bars grow into a usable training signal. That softened shape is what the student is taught to match.

Chapter 7: Quantization — The Showcase

Here is the family most likely to ship in production, and the survey's most hardware-concrete section. The other families change how many numbers you store; quantization changes how many bits each number takes. Go from FP-32 to INT-8 and you've cut storage 4× and unlocked integer-only inference on phones, FPGAs, and TPUs — with the grid staying dense, so it's a genuine speedup.

The math: affine (asymmetric) quantization

To map real values r in [r_min, r_max] onto integers q in [0, 2ⁿ−1], you need a scale S (the size of one integer step in real units) and a zero-point Z (which integer represents real 0.0):

S is a single FP-32 number stored per tensor (or per channel). This is why real quantization is mixed precision: integer weights/activations plus a handful of FP-32 scale factors. The dot product runs in integers; only the rescale touches floating point.

Worked example — quantize a weight to INT-8

Tensor range [r_min, r_max] = [−0.5, 1.5], n = 8 → 256 levels.

Note real 0.0 maps exactly to the integer Z = 64 — that's the whole point of the zero-point. Zero must be exact because padding and ReLU produce a lot of true zeros, and you don't want quantization noise on them.

The grey curve is the real weight distribution (bell-shaped, like a real layer). The vertical lines are the 2ⁿ quantization levels; dots show where sample values snap. Drop bits and the levels thin out (error rises). Tighten clip range to spend levels where the mass is. Turn on μ-law warp to space levels logarithmically — denser near zero, where most weights live. This single sim is the entire chapter: fewer bits + smarter placement = same accuracy, smaller model.

The core insight as runnable Python

import numpy as np

def quantize(r, n=8):
    rmin, rmax = r.min(), r.max()
    S = (rmax - rmin) / (2**n - 1)        # scale: real units per int step
    Z = np.round(-rmin / S).astype(np.int32) # zero-point: which int == 0.0
    q = np.round(r / S).astype(np.int32) + Z
    q = np.clip(q, 0, 2**n - 1)            # INT-n, dense, no FP
    return q, S, Z

def dequantize(q, S, Z):
    return S * (q.astype(np.float32) - Z)   # back to ~real for the math

w = np.random.randn(1000) * 0.3           # a layer's weights
q, S, Z = quantize(w, n=8)
err = np.abs(w - dequantize(q, S, Z)).mean()
# 4x smaller storage; mean abs error ~ S/4 (half a quantization step)

Chapter 8: Clipping, Retraining & the STE

Two problems stand between you and aggressive quantization. First: outliers blow up the range, starving the bulk of your values of precision. Second: the round() function has zero gradient almost everywhere, so you cannot backprop through it to retrain. The survey's quantization section is largely about these two fixes.

Problem 1 — outliers waste your bits, so CLIP

Most weights cluster near zero; a few outliers sit far out. If S spans the full [min, max], those rare outliers stretch the range and every level lands far apart — the dense middle gets coarse quantization. The fix (Banner et al.): clip at some threshold c < max, accept large error on the few clipped outliers, win small error on the many central values.

Non-uniform spacing — put levels where the mass is

Uniform quantization spaces levels evenly. But weights are bell-shaped, so most of the action is near zero. Non-uniform / logarithmic quantization (the same μ-law idea from telephony and WaveNet) uses fine spacing near zero and coarse spacing in the tails — fewer levels wasted on rare large values. This is precisely what the μ-law toggle in the Ch 7 showcase demonstrates.

Problem 2 — round() has no gradient, so use the STE

To retrain a quantized network (quantization-aware training), you must backprop through the quantizer. But round() is a staircase: flat everywhere (gradient 0) with jumps (gradient ∞). Useless for SGD. The Straight-Through Estimator (Bengio; used by Zhou et al.) is the elegant cheat: do the real rounding on the forward pass, but pretend it was the identity on the backward pass.

The extreme end: binary and ternary

Push n all the way down. Binary weights {−1, +1}: a multiply becomes a sign flip — addition/subtraction only, no multipliers. Ternary {−1, 0, +1}: adds a "skip." With binary activations too, the dot product becomes XNOR + popcount — pure bit operations. Rastegari et al.'s XNOR-Net hit 58× speedup and 32× memory savings. The cost is real accuracy loss; this is the bleeding edge of the range/precision tradeoff.

The orange staircase is the real forward quantizer round(); its true gradient is zero on every flat step (useless). The teal line is the STE's pretend gradient — the identity — that backprop uses instead. Toggle it on/off to see the lie that makes quantization-aware training possible.

Chapter 9: Picking a Method

A survey is only useful if it ends with judgment. The five families are not interchangeable — they trade different things, stack in different orders, and pay off on different hardware. Here is the practitioner's decision map distilled from the survey's recommendations.

The decision table

The trap the survey keeps returning to

Three failure modes that catch everyone:

• Sparsity ≠ speed. Unstructured pruning to 90% can run slower on a GPU than the dense model. Measure latency, not parameter count.
• PTQ ≠ free on lean models. MobileNet-class nets fall apart under post-training INT-8. Budget for quantization-aware retraining.
• Accuracy on the wrong metric. Activations are more sensitive to quantization than weights; quantizing weights alone may not shrink your run-time memory footprint at all.

Each bubble is a family, placed by typical compression and by accuracy retention (y-axis). Toggle the x-axis between "size reduction" and "real hardware speedup" and watch unstructured pruning collapse on the speedup view — the survey's central practical lesson in one picture.

Chapter 10: Connections & Cheat Sheet

This survey is a 2020 snapshot, but every family it maps is alive and central today — the techniques that put 70B-parameter LLMs onto a single consumer GPU are direct descendants of these five ideas.

The cheat sheet — every key equation

The one-paragraph summary you could whiteboard

Related lessons on Engineermaxxing

• Quantization deep-dives — the bit-level math here is the foundation for LLM.int8(), GPTQ, and QLoRA.
• WaveNet (Veanors) — μ-law companding, the same non-uniform quantization trick, applied to audio.
• Knowledge Distillation & DistilBERT — the student-teacher recipe scaled to Transformers.
• Backpropagation & the Hessian — needed to truly understand OBD's curvature term.