Your network wins the leaderboard and weighs 240 MB. The phone that has to run it has 2 GB of RAM and a 30 ms latency budget. This survey lays out the four families of techniques that shrink a network by 5–50× without throwing away the accuracy you fought for — pruning & quantization, low-rank factorization, compact filters, and knowledge distillation.
You spend three weeks training a convolutional network. It tops the validation leaderboard. You ship it. And then the mobile team writes back: the model is 240 MB, it allocates 1.2 GB of RAM at inference, and a single forward pass takes 600 ms on the target phone. The product needs 30 ms. The model that won is the model you cannot deploy.
This is not a corner case. It is the default. VGG-16 has 138 million parameters and performs ~15.5 billion multiply-accumulate operations (FLOPs) for one 224×224 image. AlexNet is 240 MB on disk. These numbers were fine for a server with a beefy GPU. They are hopeless on a watch, a drone, a self-driving car's embedded board, or a battery-powered sensor.
Here is the surprising fact that makes compression possible at all: most of the weights in a trained network do almost nothing. Networks are massively over-parameterized. The extra capacity helps during training (it makes the loss landscape easier to optimize), but once trained, a huge fraction of weights are near zero, redundant with each other, or contribute negligibly to the output. Compression is the art of finding and removing that redundancy.
The survey defines the two yardsticks every compression paper reports. Let a be the parameter count of the original model M, and a* the parameter count of the compressed model M*. The compression rate is:
A compression rate of 10 means the small model has one-tenth the parameters. Separately, let s be the runtime of M and s* the runtime of M*. The speedup rate is:
One more fact that drives every design decision: the cost is not evenly spread across a network.
| Layer type | Where it dominates | Why |
|---|---|---|
| Fully-connected (dense) | Parameters (memory) | A 4096→4096 layer is 16.7M weights — one matrix, all stored |
| Convolutional | FLOPs (compute) | Each small filter is reused across millions of spatial positions |
In VGG, the fully-connected layers hold ~90% of the parameters but the convolutional layers do ~99% of the compute. So if you need a smaller download, attack the dense layers; if you need lower latency, attack the convolutions. This single asymmetry is why no compression method is universally best — and why this is a survey, not a single algorithm.
Faced with hundreds of compression papers, the survey's central contribution is organization: every method falls into one of four families, distinguished by what kind of redundancy it exploits.
The mental model: each family attacks a different kind of waste.
The survey itself ends with practical guidance. Here is the gist, which we'll earn chapter by chapter:
| Family | Works on | Best when you need | Main drawback |
|---|---|---|---|
| Pruning & Quant. | Conv & FC | Reliable size cut, stable accuracy | Sparse speedup needs special hardware |
| Low-rank | Conv & FC | End-to-end FLOP reduction | Decomposition is costly; layer-by-layer only |
| Compact filters | Conv only | From-scratch mobile model | Helps wide nets, not thin/deep ones |
| Distillation | Conv & FC | Small/medium data, softmax tasks | Softmax-only; often less competitive |
Start with the simplest idea, and the one that started the whole field (LeCun's "Optimal Brain Damage," 1989): if a weight is near zero, it barely affects the output. So set it exactly to zero and stop storing it. Do this to enough weights and the network becomes sparse — mostly zeros — which compresses beautifully.
The workhorse recipe (Han et al., 2015) is almost embarrassingly simple:
| Step | What you do | Why |
|---|---|---|
| 1. Train | Train the dense network normally | Get a good model first; you can only prune what you understand |
| 2. Prune | Remove every weight with |w| below a threshold τ | Small magnitude ≈ small influence (a learned proxy for importance) |
| 3. Fine-tune | Retrain the surviving weights | The remaining weights compensate for the deleted ones — this recovers almost all lost accuracy |
| 4. Repeat | Raise τ, prune more, fine-tune again | Iterative pruning beats one-shot — the network adapts gradually |
Take a fully-connected layer mapping 4096→4096. Its weight matrix is 4096×4096 = 16,777,216 weights at 4 bytes each = 67 MB for one layer. Prune to 90% sparsity (keep 10%):
# Dense layer dense_params = 4096 * 4096 # 16,777,216 dense_bytes = dense_params * 4 # 67.1 MB (float32) # Pruned: keep 10% of weights. But sparse storage needs # the VALUE *and* the INDEX of each surviving weight. nnz = dense_params * 0.10 # 1,677,721 nonzeros sparse_bytes = nnz * (4 + 4) # value(4B) + index(4B) = 13.4 MB # Effective compression rate: ratio = dense_bytes / sparse_bytes # ≈ 5.0×, NOT 10×
The survey flags this as pruning's biggest practical issue. Two flavors:
Drag the sparsity slider. Watch the weight matrix empty out (gray = pruned). The accuracy curve shows the difference between one-shot pruning (brittle) and iterative prune+finetune (graceful). Toggle fine-tuning to see the cliff move.
Pruning removes weights. Quantization keeps every weight but stores each one in fewer bits. A float32 weight uses 32 bits to express a number like 0.0173629. But do you really need that precision? Empirically: no. Vanhoucke et al. showed 8-bit weights barely dent accuracy. The extreme: BinaryConnect uses 1 bit per weight (just +1 or −1).
Take that same 16.7M-weight FC layer. Cluster all weights into k = 16 groups (so each weight becomes a 4-bit index, since 2⁴=16):
# Original N = 16_777_216 orig_bits = N * 32 # 536.9 Mbit (67.1 MB) # Weight sharing with k=16 clusters → log2(16)=4 bits/index index_bits = N * 4 # 67.1 Mbit codebook_bits = 16 * 32 # 512 bits — negligible shared_bits = index_bits + codebook_bits ratio = orig_bits / shared_bits # ≈ 8.0×
From 32 bits to 4 bits per weight is an 8× compression, and the codebook (16 numbers) is rounding error. With k=256 (8-bit indices) you get 4×; with k=8 (3-bit) you get ~10.7×. You pick k to trade accuracy for size.
Binary networks (weights ∈ {−1, +1}) turn multiplies into additions/XNORs — potentially 32× smaller and dramatically faster. But the survey is honest about the cost: aggressive low-precision quantization does drop accuracy on large datasets (ImageNet), and the non-differentiable rounding step complicates training (you need a "straight-through estimator" to push gradients past the round operation). Binarization shines on small models / small datasets and on hardware where the energy win is decisive.
Each dot is a weight on the number line. Change the cluster count k — every weight snaps to its nearest centroid (the colored bars). Fewer clusters = fewer bits = more compression, but each weight drifts further from its true value (the quantization error, shown below). Watch the bits-per-weight and compression rate update live.
Pruning and quantization treat weights as a bag of independent numbers. Low-rank factorization treats a weight matrix as linear algebra and asks: is this matrix really full-rank, or is it secretly the product of two thin matrices? If a 1000×1000 matrix has rank 50, you don't need a million numbers — you need two 1000×50 matrices.
Any matrix W ∈ Rm×n factors via the Singular Value Decomposition as W = U Σ VT, where Σ is diagonal with the singular values sorted big-to-small. Keep only the top k singular values (and corresponding columns):
Symbol by symbol: Uk is m×k (the dominant output directions), VkT is k×n (the dominant input directions), Σk is the k biggest singular values (how important each direction is). We fold Σ into U to get two factors A (m×k) and B (k×n). The original one layer y = Wx becomes two smaller layers y = A(Bx) with no nonlinearity between them.
Take a 4096→4096 layer (m=n=4096) and pick rank k=256:
# Original: one big matmul orig = 4096 * 4096 # 16,777,216 multiply-adds # Factored W ≈ A(4096×256) · B(256×4096) k = 256 factored = k*4096 + 4096*k # 1,048,576 + 1,048,576 = 2,097,152 speedup = orig / factored # ≈ 8.0× (both FLOPs AND params) params_orig = 4096*4096 # 16.7M params_new = 256*4096 + 4096*256 # 2.1M → same 8× on storage
A conv kernel is a 4D tensor W ∈ RC_out×C_in×k×k, not a 2D matrix — so plain SVD doesn't apply directly. The survey covers tensor-decomposition variants: CP decomposition and Tucker / BN low-rank, which break the big convolution into a sequence of tiny convolutions (e.g. a 1×1, then a depthwise k×k, then another 1×1). The reported results on ImageNet:
| Model | Method | Top-5 acc | Speed-up | Compression |
|---|---|---|---|---|
| AlexNet | baseline | 80.03% | 1.0× | 1.0× |
| AlexNet | CP low-rank | 79.66% | 1.82× | 5.0× |
| VGG-16 | baseline | 90.60% | 1.0× | 1.0× |
| VGG-16 | CP low-rank | 90.31% | 2.05× | 2.75× |
A weight matrix has a few large singular values and a long tail of tiny ones (left bars). Slide the rank k — you keep the k largest singular values and reconstruct W from them. Watch the reconstruction error fall as k grows, while the FLOP cost rises. The sweet spot is where the curves cross: most energy captured, big compute saved.
The first three families take a big network and shrink it. Compact filters flip the philosophy: don't compress after the fact — design the convolution to be cheap in the first place. This family applies only to conv layers, but conv layers are where the FLOPs live, so it matters enormously for mobile vision.
The motivating observation (group equivariance theory): CNNs learn redundant filters. The survey writes the equivariance condition as:
Symbol by symbol: x is an input, Φ(·) is a layer, T(·) is a transform (rotation, flip, negation). The equation says: transforming the input then running the layer equals running the layer then transforming the output. If that holds, you don't need to learn separate filters for the transformed versions — learn one base filter and generate the rest by applying T. Example (CRELU): lower conv layers learn filters in positive-negative pairs, so define T(W) = −W and store half the filters — an instant 2× compression that also acts as a regularizer (it improves accuracy).
The more impactful, more famous idea — and the foundation of SqueezeNet and MobileNet. The key move: replace a 3×3 convolution with mostly 1×1 convolutions. A 1×1 conv is 9× cheaper than a 3×3 (it touches one spatial position instead of nine).
The cost of a standard conv is: (output positions) × (kernel size) × (input channels) × (output channels). Take an H×W feature map with C_in=C_out=C channels:
# Standard 3×3 conv, C in / C out, on H×W map std = H * W * (3*3) * C * C # = 9·H·W·C² # Depthwise-separable (MobileNet): 3×3 depthwise + 1×1 pointwise depthwise = H * W * (3*3) * C # 9·H·W·C (each channel separately) pointwise = H * W * (1*1) * C * C # H·W·C² (mix channels) sep = depthwise + pointwise # 9·H·W·C + H·W·C² # Speedup ratio for C=128: ratio = std / sep # 9C²/(9C+C²) = 9C/(9+C) ≈ 8.4× for C=128
Slide the channel count C. The left bar is a standard 3×3 conv; the right is depthwise (3×3 per channel) + pointwise (1×1 mixing). Watch how the gap explodes as C grows — this is exactly why MobileNet stacks separable convs.
The previous four families all manipulate weights — deleting, coarsening, factoring, or constructing them. Distillation changes the target entirely: it ignores the big model's weights and copies its behavior. Train a small "student" network to imitate the outputs of a big "teacher" network. The student ends up far better than if it had been trained alone on the raw labels.
Here is the deep idea (Hinton et al.). A hard label says "this image is a dog" — a one-hot vector [0,0,1,0]. But the teacher's softmax output says something richer: "90% dog, 8% wolf, 1.5% cat, 0.5% car." Those small numbers — the "dark knowledge" — encode similarity structure: the teacher has learned that dogs look more like wolves than like cars. A one-hot label throws all of that away.
To expose the dark knowledge, you "soften" the teacher's softmax with a temperature T:
Here zi are the teacher's logits (pre-softmax scores), and T is the temperature. T = 1 is the normal softmax. T > 1 flattens the distribution — the small probabilities grow, revealing the relative similarities the teacher learned. (At T→∞ everything is uniform; at T→0 it's one-hot.) The student is trained with the same high T to match these softened targets, then deployed at T=1.
import torch.nn.functional as F def distill_loss(student_logits, teacher_logits, labels, T=4.0, alpha=0.7): # 1. Soft target loss: student matches teacher's softened distribution soft_teacher = F.softmax(teacher_logits / T, dim=-1) soft_student = F.log_softmax(student_logits / T, dim=-1) # KL divergence, scaled by T² to keep gradient magnitudes right soft_loss = F.kl_div(soft_student, soft_teacher, reduction='batchmean') * (T * T) # 2. Hard target loss: student also matches the true labels hard_loss = F.cross_entropy(student_logits, labels) # 3. Blend: mostly imitate the teacher, partly trust the labels return alpha * soft_loss + (1 - alpha) * hard_loss
Two terms: soft_loss pulls the student toward the teacher's full distribution (the dark knowledge); hard_loss keeps it grounded in the actual ground truth. The T² factor compensates for the 1/T² shrinkage that dividing logits by T introduces into the gradient — without it, the soft term would be drowned out as T grows.
The teacher's logits for one image are fixed. Slide the temperature T. At T=1 the distribution is sharp (almost one-hot) — the student would learn almost nothing beyond the label. As T rises, the small probabilities swell, exposing that "dog" is also a bit "wolf" — the similarity structure the student actually learns from.
Now the payoff. The whole point of a survey is to let you compare the families on the same footing. Here is one network — pick a family, dial its single knob, and watch where it lands on the size–accuracy plane. The bottom-left is the dream (tiny and accurate); the top-right is where you started.
Pick a family, slide its compression strength, and read the dot's position. The dashed line marks your minimum acceptable accuracy — anything below it is a failed compression. Toggle "stack" to chain pruning + quantization (the Deep Compression idea) and reach corners no single family can.
| Your situation | Reach for |
|---|---|
| Have a pretrained net, need a smaller file, can't change accuracy | Pruning & quantization — most reliable, stable accuracy |
| Need a true end-to-end speedup on any hardware | Low-rank or compact filters — dense, smaller ops |
| Building a mobile model from scratch | Compact filters (depthwise-separable) ± distillation |
| Small/medium dataset, softmax task, have a strong teacher | Distillation — transfers knowledge, robust on little data |
| Domain prior available (e.g. rotation symmetry in medical images) | Transferred/compact filters — bakes in the prior |
| Want the maximum squeeze | Stack them — they're orthogonal (Deep Compression: 35–49×) |
You can now read any compression paper and immediately place it: which redundancy does it exploit? Does it start from a pretrained model or train end-to-end? Does it optimize size or speed? Those three questions cover the whole field.
| Family | Exploits | Mechanism | Size? | Speed (any HW)? |
|---|---|---|---|---|
| Pruning | Sparsity | Zero out small weights, fine-tune | Yes | Only if structured |
| Quantization | Precision | Fewer bits / weight sharing | Yes | Yes (int math) |
| Low-rank | Linear-algebra rank | SVD / tensor decomposition | Yes | Yes (dense) |
| Compact filters | Architectural waste | Depthwise-separable, 1×1, transforms | Yes | Yes (dense) |
| Distillation | Functional behavior | Student mimics teacher's soft outputs | Yes | Yes (small net) |
| Quantity | Formula | Meaning |
|---|---|---|
| Compression rate | α = a / a* | Original params ÷ compressed params |
| Speedup rate | δ = s / s* | Original runtime ÷ compressed runtime |
| Low-rank FLOPs | k(m+n) vs mn | Win when rank k < mn/(m+n) |
| Weight sharing bits | log₂(k) per weight | k clusters → codebook + indices |
| Separable conv | 9C+C² vs 9C² | Depthwise + pointwise vs standard 3×3 |
| Temperature softmax | pᵢ = e^(zᵢ/T) / Σ e^(zⱼ/T) | T>1 softens, reveals dark knowledge |