If two filters in a layer always light up together, one of them is redundant. PFA measures that redundancy directly from the layer's responses — using a covariance spectrum — and hands you a per-layer recipe for how many filters to keep. One intuitive knob, or none at all.
You trained a network and it works. Great. But you designed it the way everyone designs networks: you guessed. You picked 64 filters in this layer, 256 in that one, because those were round numbers and the model trained fine. Nobody told you the right number — there is no theory that says layer 3 of a VGG needs exactly 256 filters. So you over-provisioned, the way you'd over-provision a server you weren't sure how to size.
Now you want to ship that model to a phone. A 138-million-parameter VGG-16 will not fit comfortably in a mobile app, and even if it did, every forward pass costs battery and latency. You need it smaller. The question is brutally concrete: which filters can I delete, and how many, in each of the dozens of layers, without the accuracy collapsing?
You could just prune a fixed percentage from every layer — say, "delete 50% everywhere." But layers are not equally redundant. The first conv layer of a vision model learns edge and color detectors that are nearly all distinct; gut it and accuracy craters. A deep layer might have huge redundancy and tolerate 80% removal. A uniform percentage ignores this completely.
You could prune filters whose weights have small magnitude (a popular trick). But weight magnitude is a property of the filter in isolation. It tells you nothing about whether that filter is doing the same job as its neighbor. Two large-magnitude filters can be near-duplicates — keeping both is waste, and weight-based pruning never notices.
You could solve a heavy optimization that minimizes reconstruction error layer by layer. That works, but it is slow, fiddly, and usually has to process one layer at a time. Adoption dies on friction.
The paper sets a high bar for a practical method. Watch this scenario, then we'll name the goal.
Drag the slider to set a target footprint (% of original filters kept). The three curves show what accuracy you'd typically recover after fine-tuning, for a clumsy uniform cut, a random cut, and a redundancy-aware cut like PFA. Watch where each one breaks.
Notice that all three methods agree when you barely prune (90%+) — there is so much slack that even random cuts survive. The story is at the aggressive end: a redundancy-aware method holds accuracy far longer because it removes the filters that were truly doing duplicate work, not the ones that happened to be small or unlucky.
Here is the idea the whole paper rests on, in one sentence: if two filters in a layer always produce the same response, they are redundant — and you can detect that redundancy without ever looking at the filters' weights.
Think about what a filter does. Run an image through layer ℓ and each filter produces a feature map. Max-pool that feature map down to a single number — "how strongly did this filter fire on this image?" Do that for thousands of images and each filter becomes a long vector of firing strengths, one entry per image. We call that vector the filter's response.
Now ask: are two filters' response vectors correlated? If filter A fires high exactly when filter B fires high, across all images, then A and B carry the same information. The network is paying for two filters and getting one filter's worth of signal. That is the smell of waste — and correlation is exactly the tool that detects "these two vary together."
Correlation between every pair of filters is a whole matrix — messy. We want one clean summary of "how redundant is this layer." The trick is the covariance spectrum: take the covariance matrix of the responses and look at its eigenvalues. Each eigenvalue is the variance captured along one independent direction. Their shape tells the story:
That single observation — flat means keep, spiky means cut — is the engine. The two algorithms in the paper (PFA-En and PFA-KL) are just two different, principled ways to turn the spectrum's shape into a number: "keep k filters." Everything else is plumbing.
We keep saying "response." Let's make it a tensor with a shape, because the whole method is an operation on one specific matrix and you cannot implement what you cannot name.
Push image Xi through the network. At layer ℓ it produces an output tensor Ti[ℓ] ∈ R1×H×W×C — that's C feature maps, each H×W pixels. We don't want all those spatial pixels; we want one number per filter that says "did this filter fire on this image." So we spatially max-pool each feature map down to its single largest activation. The result is the response vector:
One entry per filter. For fully-connected layers there's no spatial extent, so the response is just the layer output directly (ai = Ti). Every symbol:
Now run M images through (M = thousands; the paper often uses the training set). Stack their response vectors as rows:
Read the shape: M rows (images), C columns (filters). Each column is one filter's life story — how it fired across the whole dataset. Two filters being "correlated" means two columns of A being correlated. That is the object PFA analyzes.
Each row of squares is one image's feature maps (one square per filter); brightness = activation. We max-pool each to a scalar (the dot on the right) and these scalars become a row of A. Click "New batch" to stream more images and watch columns 2 & 3 always brighten together — they're correlated.
0 images pooledWe have A ∈ RM×C. To measure how the C filters vary together, take the covariance matrix of the columns:
Entry Σjk is "how much do filters j and k move together." Now diagonalize it — find its eigenvalues λ1 ≥ λ2 ≥ … ≥ λC, sorted descending and normalized to sum to 1. This is PCA on the responses. Each eigenvalue is the fraction of total response variance that lives along one independent direction (principal component).
Counting pairwise correlations misses higher-order redundancy. Three filters can each be only mildly correlated pairwise, yet any one of them is a linear combination of the other two — collectively they span just two directions. PCA catches this automatically: the third eigenvalue would be ~0. Eigenvalues measure intrinsic dimensionality, which is the honest notion of "how many filters' worth of information is here."
Slide the correlation up. Left: 6 filter-response vectors drawn as wiggles; as correlation rises they synchronize. Right: the resulting normalized eigenvalue spectrum λ. Watch it morph from flat (uncorrelated, keep all) to a spike (redundant, keep few). The dashed line is the uniform "ideal."
Now we turn the spectrum into a number of filters to keep. The first algorithm, PFA-En ("Energy"), borrows the oldest trick in PCA: keep enough components to explain a target fraction of the variance.
Define the cumulative energy kept when you retain a fraction γ of the filters (the top ones, since λ is sorted):
Plain words: walk down the sorted eigenvalues adding them up; E(γ) is the total you've accumulated by the time you've covered γ·C filters. The recipe is then: in every layer, keep the fewest filters whose energy still reaches a user-chosen threshold τ:
Symbols, each with its job:
"95% energy" is intuitive but you may actually care about megabytes or FLOPs. Because evaluating E(γ) from a precomputed spectrum is instant, you can sweep τ downward until the resulting model hits your target footprint F:
So the practitioner says "make it fit in 10 MB" and PFA finds the τ that does it. No retraining in the loop — just arithmetic on the cached eigenvalues.
A 5-layer network. Each bar = a layer; its height = original filter count, its internal redundancy is shown by the mini-spectrum. Set the energy threshold τ and PFA recomputes, per layer, how many filters survive (the filled portion). The footprint readout updates live. Drag τ down and watch spiky layers collapse while the decorrelated first layer barely moves.
Play with it: at τ = 1.00 nothing is removed (you demand every drop of energy). Drop to τ = 0.90 on the redundant net and watch the deep layers lose most of their filters while accuracy-relevant energy is preserved. That asymmetry — different cuts per layer from one global knob — is the entire value proposition.
PFA-En still asks you for one number, τ. The authors wanted a version with zero parameters — hand it a model, get a recipe, done. That's PFA-KL, and the idea is elegant: measure how far the layer's spectrum is from the ideal flat spectrum, and let that distance directly decide the compression.
We frame the eigenvalue vector λ as a probability distribution (it sums to 1). Then:
Measure how dissimilar the actual λ is from the ideal u using KL divergence — the standard "how many extra bits to describe λ if you assumed it was u." Two quantities matter:
The actual divergence ranges from 0 (λ already uniform → keep all) up to uKL (λ is a Dirac → keep one). KL is asymmetric, so we always compute it in the direction "from λ to u" — and since d has a single support point, uKL is well-defined as the maximum possible.
Define a mapping ψ from "how far from ideal" to "what fraction to keep." The paper uses the simplest sensible choice — a linear ramp:
Read it: divergence near 0 (decorrelated) → γ near 1 → keep everything. Divergence near the max (Dirac-like) → γ near 0 → keep almost nothing. No threshold, no tuning — the data picks the compression. The full recipe is just this applied to every layer:
Move correlation. The bar shows where this layer's KL(λ,u) lands between 0 and the upper bound u_KL. The needle below it reads off γ on the linear ramp ψ = 1 − KL/u_KL. Notice: as the spectrum spikes, divergence climbs and γ — the keep-fraction — drops automatically. No knob touched.
Both recipes tell you how many filters to keep in each layer — F[ℓ] = ⌈γ[ℓ] · C[ℓ]⌉. They do not say which ones. You could pick randomly and just retrain. But the authors found you do measurably better by keeping the actual trained filters and using their weights to warm-start the small model — provided you keep the right ones.
The goal when removing a filter is: remove the one whose job is most covered by the others. Procedure, per layer:
Why recompute after each removal? Because correlations are about relationships. If filters A and B are twins, both have high redundancy scores — but once you remove A, B is suddenly unique and its score must drop, or you'd wrongly delete both halves of the only copy. The greedy update protects the survivors. (Tie-break: if two filters share an ℓ1-norm, keep the one with the higher single peak correlation.)
8 filters as a correlation graph (thicker/brighter edge = more correlated). Click "Remove worst" to delete the filter with the highest total correlation; the graph updates and scores recompute. Notice that removing one twin makes its partner safe. Keep going until only the decorrelated core survives.
Let's run PFA by hand on a tiny layer so the formulas stop being abstract. Suppose a layer has C = 4 filters, and after pooling responses over a dataset and doing PCA we get this sorted, normalized spectrum:
Accumulate eigenvalues until we reach 0.90:
| Keep top k | Cumulative energy E | ≥ τ = 0.90? |
|---|---|---|
| 1 | 0.70 | no |
| 2 | 0.70 + 0.20 = 0.90 | yes ✓ |
| 3 | 0.97 | (already satisfied) |
We hit 0.90 at k = 2. So γ = 2/4 = 0.5 — keep 2 of 4 filters, a 2× cut in this layer. If τ were 0.95 instead, we'd need k = 3 (E = 0.97), giving γ = 0.75.
The uniform ideal is u = (0.25, 0.25, 0.25, 0.25). KL divergence (using log base 2, in bits):
Term by term: 0.70·log₂(0.70/0.25) + 0.20·log₂(0.20/0.25) + 0.07·log₂(0.07/0.25) + 0.03·log₂(0.03/0.25)
= 0.70·(1.485) + 0.20·(−0.322) + 0.07·(−1.837) + 0.03·(−3.059)
= 1.040 − 0.064 − 0.129 − 0.092 ≈ 0.755 bits.
Upper bound uKL = KL(Dirac, u) = 1·log₂(1/0.25) = log₂(4) = 2 bits. So γ = 1 − 0.755/2 ≈ 0.62 → keep ⌈0.62·4⌉ = 3 filters. PFA-KL is a touch gentler here than PFA-En at τ=0.90, with no knob chosen.
python import numpy as np def layer_spectrum(A): # A: (M images, C filters) pooled responses Sigma = np.cov(A, rowvar=False) # (C, C) covariance of the columns lam = np.linalg.eigvalsh(Sigma)[::-1] # eigenvalues, descending lam = np.clip(lam, 0, None) return lam / lam.sum() # normalized spectrum, sums to 1 def pfa_en(lam, tau=0.95): # energy recipe: how many filters to keep keep = np.searchsorted(np.cumsum(lam), tau) + 1 return min(keep, len(lam)) def pfa_kl(lam): # parameter-free recipe C = len(lam); u = np.full(C, 1.0/C) p = np.clip(lam, 1e-12, None) # avoid log(0) kl = np.sum(p * np.log2(p / u)) u_kl = np.log2(C) # KL(Dirac, uniform) gamma = 1.0 - kl / u_kl # linear mapping psi return max(1, int(np.ceil(gamma * C))) lam = layer_spectrum(A) # run a forward pass first to get A print("PFA-En keeps", pfa_en(lam, 0.90), "of", len(lam)) print("PFA-KL keeps", pfa_kl(lam), "of", len(lam))
That is genuinely the core of the paper. The cost is one forward pass to build A, an eigendecomposition per layer (O(MC² + C³)), and a one-line recipe. No backprop, no loss function, no per-layer tuning.
An intuitive method is worthless if it loses to brute force. PFA was tested on VGG-16 and ResNet on CIFAR-10, CIFAR-100, and ImageNet, against several pruning families. The headline numbers from the paper, for VGG-16:
| Dataset | Compression | Accuracy change |
|---|---|---|
| CIFAR-10 | 8× smaller | +0.4 pp (better!) |
| CIFAR-100 | 3× smaller | +1.4 pp |
| ImageNet | 1.4× smaller | +2.4 pp |
Read that again: the compressed networks were not just nearly as good — several were more accurate than the original. The authors attribute this to the smaller model generalizing better; the over-parameterized original was carrying redundant capacity that hurt it. Compression here doubles as regularization.
To separate "PFA's recipe is smart" from "any pruning works after fine-tuning," they ran a SimpleCNN study with three references at each footprint: (1) PFA recipes, (2) 300 random per-layer cuts, and (3) an empirical upper bound = the best of all those random cuts. The result that matters:
Recreation of the paper's ablation shape. X = footprint kept, Y = accuracy change (pp). Toggle each series. The gap between "average random" and "upper bound" shows pruning is hard; PFA-En and PFA-KL hug the upper bound. Hover the slider to inspect a footprint.
How does it compare to the heavyweights? PFA is competitive with — sometimes better than — much more complex methods like Network Slimming (NS), Variational Information Bottleneck (VIB), and Filter Group Approximation (FGA). VIB edges PFA by ~1 pp at comparable size, but at far higher computational cost and with a loss-function modification PFA never needs. The trade PFA makes: give up the last point of accuracy, win on simplicity and zero adoption friction.
Here's the consequence almost nobody expects from a compression paper. Because PFA decides what to cut from responses on a dataset — not from weights — the dataset you feed it changes the recipe. Show it images from a different domain and it sculpts a different compressed network, specialized for that domain.
The setup: train VGG-16 on CIFAR-100 (domain DA). Then run PFA-KL using responses gathered on a target domain DZ — say, a 2-class subset. Filters that were useful for the full 100-class problem but produce flat, redundant responses on the 2-class subset now show up as compressible. PFA carves them away. The result is "compress and specialize" in a single shot.
Across adaptation experiments, "PFA fine" (compress on the target, then fine-tune with filter selection) matched the accuracy of fully fine-tuning the full model — while using architectures more than 4× smaller — and clearly beat training a full model from scratch on the small target. You inherit the source model's learned features and get a right-sized network for the new task, for the price of one forward pass plus fine-tuning.
Pick a target domain. The same source VGG (gray outline = original per-layer width) gets a different PFA-KL recipe (filled bars) depending on the task's complexity. Easier targets (2 classes) → thinner networks; harder targets (10 classes) → wider. The total footprint readout updates.
PFA sits in the structured pruning family — it removes whole filters, so the compressed model is a standard dense network that runs fast on any hardware (unlike sparse pruning, which leaves scattered zeros that need special kernels). Its distinguishing trait is reasoning from responses, not weights, which is what unlocks domain adaptation.
| Family | Idea | vs PFA |
|---|---|---|
| Quantization | Fewer bits per weight | Orthogonal — stack it on top of PFA |
| Knowledge distillation | Small net mimics big net's outputs | Needs a training loop; PFA needs only a forward pass |
| Weight pruning (e.g. FP) | Drop small-magnitude filters | Ignores redundancy & domain; PFA reads correlation |
| Sparse pruning (NS, VIB) | Zero individual weights via modified loss | Modifies training; PFA is post-hoc, loss-agnostic |
| Tensor factorization (FGA) | Replace tensors with cheaper factors | Per-layer optimization; PFA is closed-form, all layers at once |
Related lessons on the site: the TinyML pruning series (structured pruning), transformer efficiency, and the linear-algebra foundations of PCA in the linear algebra workbook.