Perch 2.0: The Bittern Lesson
for Bioacoustics

Chapter 1: The Bittern Lesson

Rich Sutton's famous essay The Bitter Lesson (2019) argues that general, compute-hungry methods (search, learning) always win over hand-crafted human knowledge. Perch 2.0's title is a pun: the bittern lesson (a bittern is a shy, booming heron). And the lesson is almost the opposite flavour: here, a simple, well-tuned supervised model is the thing that's "difficult to beat."

Why fine-grained labels are the secret weapon

Imagine two training regimes for the same network:

• Coarse task: "Is this a bird or a mammal?" — 2 classes. The network only needs to learn crude features (birds are high-pitched, mammals low). Easy. Lazy.
• Fine-grained task: "Which of these 40 warbler species?" — 40 nearly-identical classes. The network is forced to notice subtle timing, the exact shape of a frequency sweep, the micro-texture of a trill. It must build a rich representation just to survive the loss.

Section C.1 of the paper makes this concrete: if you deliberately coarsen the labels (merge species into genera, genera into families), transfer performance gets worse. The harder the original problem, the better the leftover features. This echoes Hong et al. (2024), "Why fine-grained labels in pretraining benefit generalization."

Watch the embedding cloud: with 2 coarse labels the points blur into two soft blobs; with 14,795 fine labels the model carves clean, separable clusters. A linear probe on the right-hand picture is trivial; on the left it's hopeless.

Why birds transfer to everything

Two facts make avian audio a freakishly good pre-training source:

1. Birdsong is insanely diverse (Kroodsma, 2004) — buzzes, whistles, trills, clicks, harmonics. A model that handles all of it has effectively seen most of the acoustic primitives animals use.
2. Sound production is physically universal (Elemans et al., 2015): birds and mammals make sound with the same fluid-dynamic mechanisms. A feature that captures a bird's frequency sweep also captures a whale's.

Chapter 2: From Audio to Embedding

Before any learning happens, raw sound must become a tensor a CNN can chew on. Let's trace one 5-second clip through the front of the model, shape by shape.

Step 1 — The frontend: waveform → log-mel spectrogram

The model ingests monaural audio at 32 kHz. A 5-second clip is therefore:

A raw waveform is a terrible input — it's a 1-D wiggle where the same bird call shifted by one millisecond looks completely different numerically. So we convert to a spectrogram: chop the signal into short overlapping frames, take the Fourier transform of each, and stack the frequency content over time. The result is a picture: time across, frequency up, brightness = energy.

The exact recipe the paper uses:

Two design choices worth understanding. The mel scale spaces frequency bins the way the ear does — fine resolution low down (where most animal calls live), coarse up high. The log (of energy) compresses dynamic range, so a loud call and a faint call have comparable contrast — exactly like how we perceive loudness. The bins span 60 Hz–16 kHz, which is also why the authors throw out all bat recordings: bats vocalize above 16 kHz and simply don't exist in this representation.

Step 2 — The embedding model: EfficientNet-B3

That [500, 128] picture goes into an EfficientNet-B3 — a convolutional residual network with depthwise convolutions, 12M parameters. (Perch 1.0 used the smaller B1, 7.8M; the extra data justified scaling up.) The CNN downsamples spatially and grows the channel axis, producing a spatial embedding:

Then we average over the two spatial axes (the 5×3 grid) to get a single vector — the mean embedding:

This 1536-dim vector is the product. Everything the ecologist uses — nearest-neighbour search, clustering, the linear probe — operates on E_A. The whole rest of the paper is machinery to make E_A as good as possible, then thrown away at deployment. The heads exist only during training.

Chapter 3: Multi-Source Mixup

Here's a problem buried in the data. Recordings range from under a second to over an hour, and the labels are weak: a 2-minute file tagged "Cerulean Warbler" doesn't tell you which 5 seconds actually contain the bird. The bird might be silent for most of it. So when we crop a random 5-second window, we might get pure wind.

Perch 1.0 fixed this with energy-peak selection — a wavelet detector that finds the loudest moment and crops there, assuming the labelled species is the loudest thing. Perch 2.0 keeps this option but also asks a sharper question: how do we make the classification problem harder, on purpose, to force better features? (Remember Chapter 1: harder task → better embedding.)

Manufacturing difficulty with mixup

Standard mixup (Zhang et al., 2018) blends two training examples into one and asks the model to predict both. Perch 2.0 generalizes it to more than two sources. The procedure, step by step:

1. Pick how many components to blend. Sample a count from a beta-binomial, then add one so you always have at least one:
   N ∼ BetaBin(n, α, β) + 1
   In the winning configuration N ranged over {2,…,5}. (beta-binomial = a binomial whose success-probability is itself drawn from a Beta, giving a flexible, over-dispersed count.)
2. Pick blend weights from a symmetric Dirichlet (a distribution over things that sum to 1):
   w = (w₁, …, w_N) ∼ SymDir(N, ω)
   The concentration ω (best around 10–30) controls how equal the weights are — high ω → all clips roughly equal loudness.
3. Build the composite signal as the weighted sum, then renormalize the gain:
   x_mix = ( Σ_i w_i x_i ) / √( Σ_i w_i² )
   The denominator keeps total loudness roughly constant no matter how many clips you stack (the energy of a sum of independent signals grows like the root-sum-of-squares of the weights).
4. Build the target as a MULTI-HOT vector, not a weighted average. Every species in the blend gets a 1 (well, an equal share — see Ch 4), regardless of how loud it is.

Worked numerical example

Say N=3 with weights w = (0.6, 0.3, 0.1) for a robin, a thrush, and a faint warbler. The normalizer is

So x_mix = (0.6·robin + 0.3·thrush + 0.1·warbler) / 0.678. The robin enters at 0.6/0.678 ≈ 0.88× its original amplitude. And the target? [robin=1, thrush=1, warbler=1] — the warbler at 0.1 weight is demanded just as confidently as the robin at 0.6.

Drag any bar to change its weight. Watch two things: the gain normalizer keeps the composite loudness flat as you add sources, and the target row stays all-1 no matter how quiet a source gets. That's the multi-hot rule made visible.

Chapter 4: The Three Heads (showcase)

Now the heart of the architecture. From the shared embedding, training drives three output heads, each with its own loss. None of them survive to deployment — they're scaffolding that shapes E_A. Let's meet them, then watch gradients flow.

Why three? Each shapes the embedding differently

Step through it. On the forward pass the embedding lights up all three heads. On the backward pass, notice the red ✕ on the prototype→embedding edge — that's the stop-gradient. The teacher's gradients are not allowed to update the shared embedding. Why that matters is the whole next chapter.

Chapter 5: Self-Distillation from Prototypes

We have two species classifiers reading the same embedding: a plain linear head and a fancier prototype head. The paper makes the prototype head teach the linear head. This is self-distillation: a model improving by learning from a slightly-better version of itself.

What a prototype classifier is

ProtoPNet (Chen et al., 2019) reasons by analogy: it stores a handful of learned prototypes per class — little reference patterns — and classifies a new clip by "this region looks like that stored prototype." Perch learns 4 prototypes per class and the prediction uses the maximum activation across them. An orthogonality loss pushes the 4 prototypes apart so they capture different aspects of a call (the intro buzz vs. the terminal trill) rather than collapsing to one.

Because it consumes the spatial E_S and matches local patterns, the prototype head tends to make sharper, better-calibrated predictions than the global linear head. It's the better classifier — so it's the teacher.

The mechanism: soft targets + stop-gradient

Instead of training the linear head against the hard multi-hot label, in Phase II we train it against the prototype head's soft probability distribution:

Two pieces decode this. pproto is the teacher's softmax — a soft target like "92% Cerulean, 5% Black-throated Blue, 3% other," which carries dark knowledge: the relative similarity between classes that a hard one-hot label throws away. And stopgrad[·] means: treat the teacher's output as a fixed constant when computing gradients.

The two-phase schedule (and why it's two phases)

You can't distill from a teacher that doesn't exist yet. Phase I builds a competent prototype teacher. Phase II then uses it. By Phase II the embedding is already good, so the model wants gentler treatment: small learning rate, almost no mixup, less dropout.

Chapter 6: Source Prediction (DIET)

The third head is the strangest and most clever. It asks the model to predict: which of the 1.5 million source recordings did this 5-second window come from? Each recording is literally its own class. This is the DIET objective (Balestriero, 2023): "Datum IndEx as Target."

The trick hidden in plain sight

A one-hour recording can be sliced into dozens of completely non-overlapping 5-second windows. They share no audio samples — a window from minute 3 and a window from minute 47 are acoustically unrelated. Yet they have the same source label. So to predict the source from a single window, the model must learn the stable, recording-level fingerprint — the individual bird's voice, the habitat's ambient signature, the microphone's character — that persists across the whole file.

Engineering: why a low-rank head

A normal linear head from a 1536-dim embedding to 1.5M classes would be a weight matrix of

— larger than the entire rest of the model by 100×. Absurd. So the source head is low-rank, rank 512:

This factorization says "there are really only ~512 latent dimensions that distinguish recordings" — and it shrinks the head's parameters dramatically while keeping the gradient signal. One footnote-level gotcha: with 1.5M classes the cross-entropy loss is numerically large, so its loss weight is tuned carefully (Vizier picked 0.1–0.9 in Phase I, dropping toward 0 in Phase II) to avoid drowning out the species loss.

How the three losses combine

All minimized with Adam. The λ weights were swept with Vizier (Google's black-box hyperparameter optimizer): 100 models per stage, two stages per phase. Each model took 20–30 hours on a TPUv3-8.

Chapter 7: Probing & Evaluation

The model is trained. Now the embedding model is frozen — no more gradients ever touch it. How do we use it, and how do we know it's good? Both the deployment workflow and the model-selection procedure are built around three frozen-embedding tasks.

Three ways to use a frozen embedding

The reason this list exists: practitioners have little labelled data and no GPUs. Embedding the dataset once and running a linear probe is cheap, requires no ML expertise, and reuses the same embeddings for clustering and search. This is the "agile modeling" workflow (Dumoulin et al., 2025).

Model selection: optimizing for the right thing

You can't pick the best model on a single held-out accuracy — you'd pick one that's good at that domain and brittle elsewhere. So Perch validates on 19 datasets across three task types (pre-trained classification, retrieval, linear transfer) covering bird soundscapes, call-types, dialects, and non-avian taxa.

The aggregation choice is deliberate: a geometric mean of per-task scores, not an arithmetic mean.

Chapter 8: The Results

Two benchmarks. BirdSet: six fully-annotated bird soundscapes (US, Hawai‘i, Peru, Colombia). BEANS: twelve cross-taxa tasks — birds, land & marine mammals, anurans, insects. Crucially, Perch 2.0 uses no fine-tuning: on BirdSet it applies its prototype head directly; on BEANS it trains only a linear/prototype probe on frozen embeddings.

On the AUROC metric the authors trust most (the most stable, per van Merriënboer et al.), Perch 2.0 reaches 0.908 on BirdSet, beating AudioProtoPNet-5 (0.896), Bird-MAE-L (0.886), and its own predecessor Perch 1.0 (0.839) — all without touching the embedding weights.

The headline numbers

The result that stunned even the authors

Three quieter but important findings

• Random windows ≈ energy-peak selection. Contrary to Perch 1.0 and the wider literature, careful window selection stopped mattering — self-distillation absorbs the label noise. One less moving part.
• Prototype probing > linear probing on detection. On BEANS detection tasks (find the call in noise) the prototype head's part-based matching wins; on classification they're roughly tied.
• Supervision > self-supervision, again. Bird-MAE, BirdAVES and the authors' own MAE/HuBERT/SimCLR attempts all lose. The bittern lesson holds.

Why supervision keeps winning here — the hypotheses

1. Enough labels. Self-supervision wins when labels are scarce. Bioacoustics has 1.5M labelled recordings — Cole et al. (2022) show that past hundreds of thousands of labels, supervised becomes very hard to beat.
2. Self-supervision is augmentation-fragile. SSL leans heavily on hand-tuned augmentations; the right ones for audio aren't known yet (Morningstar et al., 2024).
3. Fine-grained labels are gold. 15,000 species, many in the same genus → the supervised features are unusually rich (Hong et al., 2024).

Chapter 9: Connections & Cheat Sheet

Perch 2.0 sits at a crossroads of audio ML, transfer learning, and interpretable deep learning. Here's how it links to ideas you may already know — and a one-page summary of every symbol.

The core algorithm in runnable Python

The whole training objective — three heads, the stop-gradient, the multi-hot mixup target — fits in one readable forward pass:

The single most important line is p_proto.softmax(-1).detach() — .detach() is the stop-gradient. Remove it and the teacher would warp the embedding to flatter itself. Everything else is bookkeeping.

Cheat sheet — every symbol

The one sentence to remember

If your domain has hundreds of thousands of fine-grained labels, don't reach for self-supervision — train a small supervised model on the hardest classification task you can construct, freeze it, and let its embeddings be the foundation. The bittern is shy, but the lesson is loud.