PANNs: Pretrained Audio Neural Networks

Chapter 1: AudioSet & Weak Labels

Everything in PANNs is downstream of one dataset, so we have to understand its shape. AudioSet is ~2.1M ten-second clips drawn from YouTube, each tagged with one or more of 527 sound classes organised in an ontology (Speech, Music, Animal → Dog → Bark, Vehicle → Car, Engine, Glass, and so on). After excluding clips that had been removed from YouTube, the authors trained on roughly 1.9M clips.

Two properties of these labels shape the whole architecture:

Why "weak" forces global pooling

Because labels are clip-level, the network must produce one prediction per clip, even though internally it sees a sequence of time frames. The standard fix — and the one PANNs uses — is to let the CNN produce a per-frame feature map and then pool over time to collapse it to a single clip vector. The label supervises only the pooled output. The network never gets told which frame contained the dog bark; it has to figure that out by getting the clip-level answer right. This is a multiple-instance learning setup, and the pooling operator is the heart of it (Chapter 4).

The audio resampling pipeline (real numbers)

Before any learning, every clip is normalised to a fixed shape so it can be batched. PANNs uses:

So a clip becomes a single-channel image of shape roughly [1000, 64] — time down the rows, mel-frequency across the columns. That is exactly what a CNN wants. (Hop 320 at 32 kHz is 10 ms per frame, which is why ~10 s gives ~1000 frames.)

Worked example: how big is the training matrix?

Take 1.9M clips, each [1000, 64] in log-mel space. That's 1.9e6 × 1000 × 64 ≈ 1.2e11 floats. At 4 bytes each that's ~486 GB if you precomputed and stored all spectrograms. PANNs instead computes the STFT and mel filterbank on the GPU, on the fly, from the raw 16-bit waveform (which is ~640 KB/clip, ~1.2 TB total). Storing waveforms and transforming live is both smaller and lets the mel front end be part of the network — which is what makes the learnable Wavegram in Chapter 5 possible.

Chapter 2: The Log-Mel Front End

This is the layer that turns sound into a picture, and it's the component you most need to understand to use PANNs. We'll derive each step with real numbers.

Step 1: Short-Time Fourier Transform (STFT)

A raw waveform is just amplitude over time — 320,000 numbers for our clip. The ear, though, hears frequency. The STFT slides a window over the signal and computes a Fourier transform inside each window, giving "how much of each frequency is present in this 32 ms slice."

Here N = 1024 is the window length, H = 320 is the hop, and w is a Hann window (a smooth bell that tapers the edges so we don't get spectral leakage). With N = 1024 the FFT returns N/2 + 1 = 513 frequency bins per frame. With hop 320 over 320,000 samples we get ~1000 frames. We keep the power |X[f,t]|² and throw away the phase.

Step 2: The mel filterbank

513 linear-frequency bins is too many and the wrong scale — human pitch perception is roughly logarithmic. The mel scale warps frequency so that equal mel steps sound like equal pitch steps. We compress the 513 power bins down to 64 mel bands using a fixed matrix of overlapping triangular filters M of shape [64, 513]:

Each row M[m] is a triangle: zero everywhere except over a small band of FFT bins, rising to 1 at the band's centre and falling back to 0. Low mel bands are narrow (fine resolution in the bass, where the ear is picky) and high mel bands are wide (coarse in the treble). The mel-to-Hz mapping is:

We place 64 + 2 mel points evenly in mel space between 50 Hz and 14 kHz, convert back to Hz, snap to the nearest FFT bins, and build a triangle between each triple of points. That triple-of-points construction is exactly what you'll implement in Code Lab 1.

Step 3: Log compression

Sound energy spans an enormous dynamic range — a whisper and a jet differ by a factor of ~10¹² in power. A linear network would be swamped by the loud bins. We take the log:

The small ε (e.g. 10⁻¹⁰) prevents log(0) = −∞ on silent bins. Log compression mirrors the ear's roughly logarithmic loudness perception (decibels) and turns multiplicative gain changes into additive shifts — so making the whole clip louder just adds a constant, which the following batch-norm absorbs.

Worked example: one mel band by hand

Suppose mel band m = 5 covers FFT bins 8, 9, 10, 11, 12 with triangle weights M[5] = [0, 0.5, 1.0, 0.5, 0] over those bins. Let the power at those bins in frame t be |X|² = [4, 4, 9, 16, 1]. Then:

And L[5, t] = log(19 + 10⁻¹⁰) ≈ 2.944. The centre bin (10) dominates because its weight is 1.0; the wings contribute half. Repeat for all 64 bands and all ~1000 frames and you have the log-mel image. That is the entire front end.

python
# The PANNs front end, conceptually (torchlibrosa does this on-GPU)
import torch, torchaudio

wav, sr = torchaudio.load("clip.wav")        # [1, 320000] at 32 kHz

spec = torchaudio.transforms.Spectrogram(
    n_fft=1024, hop_length=320, power=2.0)(wav)  # [1, 513, ~1000]

melfb = torchaudio.transforms.MelScale(
    n_mels=64, sample_rate=32000, n_stft=513,
    f_min=50, f_max=14000)                       # fixed [64, 513] matrix
mel = melfb(spec)                              # [1, 64, ~1000]

logmel = torch.log(mel + 1e-10)               # compress dynamic range
logmel = logmel.transpose(1, 2)               # [1, ~1000, 64]: time, mel
# a BatchNorm over the 64 mel bins follows, learned during training

Chapter 3: The CNN14 Backbone

With the log-mel "image" [1, 1000, 64] in hand, we need a network to read it. PANNs evaluates a ladder of CNNs (CNN6, CNN10, CNN14) and a few variants; CNN14 is the workhorse. It is a deliberately plain VGG-style stack — the paper's point is partly that a clean, well-trained vanilla CNN beats the fancier prior approaches.

The convolutional block

CNN14 is built from six conv blocks. Each block is two 3×3 convolutions, each followed by batch-norm and ReLU, then a 2×2 average-pool that halves both time and frequency:

The channel widths grow as the maps shrink — the classic CNN bargain of trading spatial resolution for semantic depth:

After block 6 we have a [2048, 15, 1] feature map: 2048 channels, 15 time steps, 1 frequency. The frequency axis has been pooled away entirely — each channel now answers "how strongly is my learned pattern present at each of 15 time steps?"

Why 3×3 convolutions?

A 3×3 kernel looks at a tiny patch — 3 frames × 3 mel bands. Stacking two of them per block gives an effective 5×5 receptive field with fewer parameters and an extra nonlinearity. After six blocks of pooling, a single late-layer neuron sees the whole clip in time and the whole spectrum in frequency. Depth buys global context cheaply — the same reason VGG used stacks of 3×3 in vision.

Worked example: parameter count of one conv

Block 3's first conv maps 128 input channels to 256 output channels with a 3×3 kernel. Its weight tensor is [256, 128, 3, 3], so it has 256 × 128 × 9 = 294,912 weights plus 256 biases ≈ 295K parameters in that single layer. CNN14 totals ~81M parameters — most of them concentrated in the wide late blocks (block 6's convs alone hold tens of millions because 1024→2048 channels × 3×3 is huge). This is why the "make it deeper/wider" ablation hits diminishing returns: you pay quadratically in the late blocks.

python
import torch.nn as nn

class ConvBlock(nn.Module):
    def __init__(self, c_in, c_out):
        super().__init__()
        self.conv1 = nn.Conv2d(c_in,  c_out, 3, padding=1, bias=False)
        self.conv2 = nn.Conv2d(c_out, c_out, 3, padding=1, bias=False)
        self.bn1, self.bn2 = nn.BatchNorm2d(c_out), nn.BatchNorm2d(c_out)

    def forward(self, x):
        x = torch.relu(self.bn1(self.conv1(x)))
        x = torch.relu(self.bn2(self.conv2(x)))
        return nn.functional.avg_pool2d(x, 2)   # halve T and F

# CNN14: 6 conv blocks, widths 64..2048
widths = [64, 128, 256, 512, 1024, 2048]
blocks = nn.ModuleList(
    [ConvBlock(c_in, c_out)
     for c_in, c_out in zip([1] + widths[:-1], widths)])
# x: [B, 1, 1000, 64] -> ... -> [B, 2048, 15, 1]

Chapter 4: The Tagging Head & Loss

This is the part of PANNs you reuse most often, and it's a direct consequence of the weak labels from Chapter 1. We have a [2048, 15, 1] map. We need a single 527-vector of probabilities, with weak supervision. Three operations get us there.

Step 1: Embedding (squeeze frequency, keep time)

The frequency axis is already 1. A 1×1 conv (or a linear layer) maps the 2048 channels to a 2048-d embedding per time step, giving a sequence of 15 frame-embeddings, each 2048-d. This sequence is the reusable representation — the thing you transfer to other tasks.

Step 2: Framewise logits → sigmoid

A linear layer W of shape [527, 2048] projects each frame-embedding to 527 logits, and a sigmoid (not softmax!) turns each logit into an independent probability:

Why sigmoid? Softmax forces the 527 probabilities to sum to 1 — that's correct for mutually exclusive classes, but a clip can be "Speech" AND "Music" AND "Applause" simultaneously. Sigmoid treats each class as an independent yes/no, which is exactly what multi-label means. Each p_frame[c, t] is "how strongly does class c fire at frame t?"

Step 3: Clipwise aggregation (pool over time)

The labels are clip-level, so we collapse the 15 frame-probabilities of each class into one clip-probability. PANNs' default is average pooling:

Average pooling says "a class is present to the degree it fires across the whole clip." Max pooling ("present if it fires in any frame") is an alternative, but tends to be noisy on weak labels. PANNs also explores attention pooling, where learned weights decide how much each frame contributes — a soft, learned interpolation between average and max. The clipwise vector p_clip is the final 527-d prediction.

The loss: binary cross-entropy, summed over classes

Because each class is an independent Bernoulli, the loss is binary cross-entropy (BCE) on every class, summed (or averaged) over the 527:

where y_c ∈ {0, 1} is the weak label for class c. Note the gradient only ever touches the clip-level prediction — the network must learn on its own which frames to fire on to make p_clip right, since no frame-level supervision exists. That is multiple-instance learning in one equation.

Worked example: BCE on three classes by hand

Suppose for one clip the true labels are y = [1, 0, 1] (Speech yes, Dog no, Music yes) and the clip predictions are p = [0.8, 0.3, 0.6]. Then:

If the model improves Music to p = 0.9, the Music term becomes −log 0.9 = 0.105 and L drops to 0.685. The loss rewards pushing present-class probabilities up and absent-class probabilities down — independently per class. There's no competition between Speech and Music, unlike softmax.

python
import torch, torch.nn as nn

class TaggingHead(nn.Module):
    def __init__(self, d=2048, n_classes=527):
        super().__init__()
        self.fc = nn.Linear(d, n_classes)

    def forward(self, feat):           # feat: [B, 2048, T]  (T=15)
        e = feat.transpose(1, 2)        # [B, T, 2048]
        frame_logits = self.fc(e)       # [B, T, 527]
        frame_prob  = torch.sigmoid(frame_logits)   # multi-label!
        clip_prob   = frame_prob.mean(dim=1)         # avg over time -> [B, 527]
        return clip_prob, frame_prob

# weak clip-level targets y in {0,1}^527
loss = nn.functional.binary_cross_entropy(clip_prob, y)

Chapter 5: Wavegram-Logmel-CNN

Log-mel is a fixed, hand-designed front end. It bakes in human psychoacoustics — which is mostly good, but it also throws away information (it discards phase, and the mel warping and band limits are guesses). What if the network could learn its own time-frequency representation directly from the raw waveform, and we used it alongside log-mel? That is the Wavegram-Logmel-CNN, the paper's best-performing model.

The Wavegram: a learned spectrogram from 1-D convs

The Wavegram replaces the STFT+mel pipeline with a small stack of 1-D convolutions over the raw waveform. The first conv has a large stride that downsamples the 320,000 samples to a frame rate matching the log-mel (~100 frames/s); subsequent 1-D convs deepen the per-frame features. The output channels are then reshaped to act as a "frequency" axis, producing a learned 2-D map of shape [channels, frames] — a spectrogram the network designed for itself.

Fusing the two front ends

The Wavegram and the log-mel spectrogram are both [·, frames]-shaped images. PANNs concatenates them along the channel axis and feeds the stacked input into the same CNN14 backbone:

So the backbone sees both a representation tuned to human hearing (log-mel) and one the data chose (Wavegram). The two are complementary: log-mel is a strong prior that stabilises early training, while the Wavegram can recover detail the mel warping discarded. Fused, they edge out either alone.

Worked example: the downsampling stride

To match log-mel's ~10 ms hop (hop 320 at 32 kHz → 100 frames/s), the Wavegram's first Conv1d needs an effective stride that turns 320,000 samples into ~1000 frames — i.e. a total stride of ~320. PANNs achieves this with a strided first conv plus pooling (e.g. stride 4 conv, then more downsampling), so the Wavegram's time axis lines up frame-for-frame with the log-mel's. Without that alignment you couldn't concatenate them — the frames wouldn't correspond to the same moments.

python
import torch, torch.nn as nn

class Wavegram(nn.Module):
    """Learned time-frequency front end from raw waveform."""
    def __init__(self):
        super().__init__()
        # strided 1-D conv downsamples 320k samples toward the frame rate
        self.pre = nn.Conv1d(1, 64, kernel_size=11, stride=5, padding=5)
        self.blocks = nn.Sequential(   # further 1-D convs + pooling
            ConvBlock1d(64, 64), ConvBlock1d(64, 128), ConvBlock1d(128, 64))

    def forward(self, wav):              # wav: [B, 1, 320000]
        x = self.blocks(self.pre(wav))   # [B, 64, ~1000]
        # reshape the 64 channels into a learned "frequency" axis
        return x.unsqueeze(1)             # [B, 1, 64, ~1000]  (acts like a spectrogram)

# Wavegram-Logmel-CNN: stack the two front ends on the channel axis
wavegram = Wavegram()(wav)            # [B, 1, 64, T]
logmel   = logmel_frontend(wav)        # [B, 1, 64, T]
x = torch.cat([wavegram, logmel], dim=1)  # [B, 2, 64, T] -> CNN14

Chapter 6: Results & Ablations

A method is only as good as the number it produces, and on AudioSet the number is mAP. Let's understand it, then read the results.

What mAP measures

For each of the 527 classes you rank all test clips by their predicted probability for that class, then compute average precision (AP) — the area under that class's precision-recall curve. Because classes are wildly imbalanced and multi-label, AP (which is threshold-free and per-class) is far more honest than accuracy. mAP is the mean of the 527 per-class APs. A perfect ranker scores 1.0; random scores roughly the class prevalence.

Here P_k and R_k are precision and recall at the k-th ranked clip. The earlier baseline trained only on Google's released embeddings scored around mAP 0.31–0.36; PANNs, training end-to-end on raw audio, pushed past it decisively.

The headline numbers

Two stories sit in this table. First, depth helps: CNN6 → CNN10 → CNN14 climbs steadily as you add blocks. Second, the learned front end helps a little more: the Wavegram-Logmel hybrid edges CNN14 from 0.431 to 0.439. The gain from the learnable branch is real but modest — the dominant lever is "train a good deep CNN end-to-end on raw audio at scale."

Ablations the paper ran

Chapter 7: Transfer Learning

The AudioSet mAP is impressive, but the lasting contribution is transfer — the claim that a PANN is a good starting point for other audio tasks, the way an ImageNet ResNet is for vision. This is where "pretrained" in the name earns its keep.

The transfer recipe

Two modes: frozen vs fine-tuned

You can use a PANN two ways, and the choice is an engineering tradeoff:

Fine-tuning with a low backbone learning rate is the usual winner: it adapts the AudioSet features to the new domain without destroying them. Freezing is the safe choice when you have only a few hundred examples. PANNs reports that fine-tuned CNN14 set new states of the art across the downstream suite.

Where it transferred

The pattern is the vision pattern: pretraining helps most when the target dataset is small, and even on larger targets it speeds convergence and lifts the ceiling. A 50-clip-per-class problem that was hopeless from scratch becomes tractable because the backbone already knows what "sounds" are.

Worked example: how few labels does transfer rescue?

ESC-50 has 50 classes × 40 clips = 2000 clips total. Training CNN14's 81M parameters from scratch on 2000 clips would catastrophically overfit — far more parameters than data points. But with AudioSet pretraining, those 81M weights are already in a good basin; the 2000 clips only have to nudge them. That's why frozen-feature linear probes on ESC-50 already beat many from-scratch deep models, and fine-tuning pushes further. The lesson: pretraining converts a data-starved problem into a data-sufficient one.

python
# Transfer CNN14 to ESC-50 (50 classes) — the canonical PANNs use
import torch, torch.nn as nn
from panns_inference import Cnn14

model = Cnn14(sample_rate=32000, ..., classes_num=527)
model.load_state_dict(torch.load("Cnn14_mAP=0.431.pth")["model"])

# Swap the 527-class head for ESC-50's 50 classes
model.fc_audioset = nn.Linear(2048, 50)

# Fine-tune: low LR on the pretrained backbone, higher on the new head
opt = torch.optim.Adam([
    {"params": model.fc_audioset.parameters(), "lr": 1e-3},
    {"params": [p for n, p in model.named_parameters()
                if "fc_audioset" not in n], "lr": 1e-4},
])
# 2000 ESC-50 clips now suffice — the backbone already understands audio

Chapter 8: The Front-End Explorer

Here is the whole pipeline in one interactive instrument. Build a sound from components, watch it pass through the log-mel front end, the (schematic) CNN14 backbone, and the multi-label tagging head, and read the clipwise tag probabilities at the end. This is PANNs end-to-end: waveform → log-mel → features → sigmoid tags.

Things to try in the explorer:

• Add Speech AND Music together. Both tag bars rise independently — neither suppresses the other. That is the multi-label sigmoid head; a softmax would force them to share one unit of probability.
• Add the Glass click alone. Its energy is a brief vertical streak in the mel panel — a transient, broadband. The Glass tag fires sharply while sustained-sound tags stay low.
• Crank the mel bands down to 16. The mel panel coarsens; fine harmonic detail blurs. This is the sample-rate/band ablation made visible — fewer bands, less resolution, slightly worse separation.
• Raise the sharpness (pooling temperature). Confident tags get more confident; the pooling concentrates on the frames that fire. This mirrors the avg-vs-attention pooling choice.

Chapter 9: Limitations & Connections

PANNs gave audio its ImageNet backbone, but it is a 2020 CNN, and the field moved. Knowing its limits tells you when to reach for something newer.

Limitations

The lineage

Related Veanors & Gleams

Cheat sheet