CLAP: Audio Concepts from Natural Language

Chapter 1: Captions Are Supervision

Chapter 0 ended with a promise: replace the label with a sentence. That sounds like a small edit — a richer annotation format. It is not. It changes what the loss function can possibly teach, and it changes what the model can possibly be asked at inference. This chapter is about why.

Here are four real captions from CLAP's training data, sampled in the paper's own appendix table:

Now compress each into one integer from a taxonomy. "Music." "Siren." "Bird." "Speech." Look at what evaporates.

The geometry point, made precisely

This is worth slowing down for, because it is the load-bearing idea of the whole paradigm. Take the one-hot targets for three classes:

Compute the distance between any two: the squared Euclidean distance is 2 for every pair. Dog-to-wolf: 2. Dog-to-trombone: 2. The loss function is structurally incapable of knowing that a dog and a wolf are related. If the model confuses a wolf howl for a dog bark, it is penalised exactly as hard as if it had said trombone.

Now take the caption route. "A dog barking in a yard" and "a wolf howling in the distance" are two token sequences that a language encoder maps to two vectors that are genuinely close, because they share a, in, the animal-vocalisation frame, and the outdoor-scene frame. The target for the audio is no longer an arbitrary corner of a simplex; it is a point in a space where semantic neighbours are geometric neighbours.

That is the entire trick. The text encoder donates its geometry to the audio encoder. Everything else — the contrastive loss, the projections, the temperature — is plumbing to make the donation happen.

Where 128,010 pairs came from

The paper did not commission a new dataset. It assembled one out of four public corpora, and the assembly is itself instructive:

Two details in that table are easy to skim past and both are load-bearing.

Pairs exceed unique audios by 41%. 128,010 pairs over 90,947 clips. ClothoV2 contributes five captions for each of 5,929 clips; MACS multiplies similarly. So the same audio appears repeatedly with different descriptions. That is free regularisation: the model is being told, five ways, that one waveform has many valid verbalisations. It cannot latch onto one phrasing. It is nudged toward the meaning that all five share.

FSD50K's class label was thrown away on purpose. FSD50K ships with clean sound-event labels — the paper's own Table 1 evaluates on them. And the authors chose the messy human-written title and description instead. That is a bet: that noisy natural language beats clean categories as a training signal. Chapter 8 will show you the one place that bet failed spectacularly.

Counting the bits, properly

"Captions are richer" deserves an actual number rather than a gesture. Information theory gives one.

A label drawn uniformly from C categories carries log₂(C) bits. For ESC50's fifty classes that is 5.64 bits. For FSD50K's 200 classes, 7.64 bits — and even that is optimistic, because real label distributions are skewed, which lowers the entropy further.

A caption is a sequence of words. If you model English at roughly 8–10 bits per word for content-bearing text (Shannon's classic estimate for English is about 1 bit per character after redundancy, so a 5-letter word plus a space runs about 6 bits, and audio captions are less predictable than newspaper prose), then a 15-word caption carries on the order of a hundred bits. Two orders of magnitude more than the label.

But the bit count is the less important half. The label's 5.6 bits are unstructured: they identify a corner of a simplex and say nothing about which corners are near which. The caption's hundred bits arrive already embedded in a space where synonyms are close, negations are far, and compositional phrases land between their parts. That structure is what CLAP is really harvesting.

How the pairs are actually assembled

python — constructing CLAP's 128,010 pairspairs = []

# FSD50K: 36,796 clips. The class label is available and DELIBERATELY IGNORED.
for clip in fsd50k_train_and_val:
    caption = clip.title + ". " + clip.description      # raw Freesound metadata
    pairs.append((clip.audio, caption))                   # 1 pair per clip

# ClothoV2: 5,929 clips, each with 5 human captions -> 5 pairs each.
for clip in clotho_v2:
    for caption in clip.captions:                         # len == 5
        pairs.append((clip.audio, caption))               # 29,646 total

# AudioCaps: 44,292 ten-second AudioSet clips, 1 crowdsourced caption each.
# MACS: 3,930 clips captioned by multiple annotators -> 17,276 pairs.

len(pairs)                       # 128,010
len({p[0] for p in pairs})       # 90,947 unique clips  <- 41% more pairs than audio
len({p[1] for p in pairs})       # 128,010 unique captions <- every caption is distinct

Look at the last two lines together. Every caption is unique; the clips are not. That asymmetry is a design decision with a consequence we will meet again in Chapter 8: at large batch sizes, two entries in the same batch can be the same audio with different captions, and the contrastive loss will insist they are negatives of each other.

CLIP, Wav2CLIP, AudioCLIP — and the difference that matters

Two audio models had already borrowed CLIP's machinery by 2022, and understanding why CLAP is not just a third one clarifies the contribution.

The paper is measured about the prior work — "the performance was promising" — but names the gap precisely: those models are "trained with audio and class labels from AudioSet instead of audio and natural language," so "it is yet to be explored how natural language can benefit flexibility and generalization to new classes and tasks."

Why does training on label-text hurt? Because of a distribution mismatch that will come back in Chapter 5. If every training text is a bare category name, the text encoder's useful region is a small island of noun phrases. Ask it about "a small dog barking indoors" and you are off the island. If instead the training text is real sentences, the model has seen the whole continent, and the class names at inference are the odd case — which, delightfully, is exactly the problem the prompt templates of Chapter 5 exist to fix.

What "natural language supervision" is not

The phrase is loose enough to cover several different things, and CLAP means exactly one of them. Pin the boundaries.

The narrowness is the point. CLAP does one thing: it makes audio and its description land in the same place. Everything in the rest of this lesson is a consequence of that single commitment.

Reading four captions like an engineer

Take the paper's own samples and ask, of each, what the model is being asked to solve.

Two of the four demand things the architecture provably cannot represent. That is not a flaw in the data; it is how the ceiling gets set. The model extracts what it can and treats the rest as noise, which is exactly why the resulting embedding is a good event detector and a poor sequence model.

The one-sentence method

Before the plumbing chapters, here is the whole model in a sentence you should be able to reconstruct by Chapter 4: encode the audio, encode the caption, project both into one shared space, and train so that each clip's own caption is the nearest text in the batch and every other caption is far.

Chapter 2: From Pressure Wave to Tensor

Chapter 1 gave us pairs: a clip and a sentence. The sentence is easy — text encoders eat text. The clip is not. A microphone hands you a list of numbers describing air pressure over time, and a convolutional network wants a two-dimensional array with meaningful local structure. Getting from one to the other is a sequence of deliberate, lossy decisions, and the paper specifies every one of them in three lines of Section 3.2. This chapter unpacks those three lines completely, because if you cannot state the shape of the tensor entering the audio encoder, you do not understand the model.

Here is the paper's own specification, verbatim in substance: log Mel spectrogram, sampling rate 44.1 kHz, hop size 320, window size 1024, 64 Mel bins in the range 50–8000 Hz, each clip randomly truncated to a continuous 5-second segment or padded if shorter.

Step 1 — the raw signal, and why it is not enough

At 44.1 kHz, five seconds of audio is 5 × 44,100 = 220,500 floating-point numbers. That is a one-dimensional array. You could hand it straight to a network — some models do — but for a convolutional encoder it is a bad substrate for three reasons.

The features are non-local in time. A 440 Hz tone is a pattern that repeats every 100 samples. To notice it, a filter needs a receptive field spanning hundreds of samples. To notice a 50 Hz rumble it needs 880. Pitch information lives at scales that a small convolution kernel cannot see.

Phase is mostly nuisance. Shift the whole waveform by one sample and every number changes, while the sound is perceptually identical. You would be asking the network to learn shift-invariance from scratch, using capacity you would rather spend elsewhere.

It is enormous. 220,500 numbers per clip, times a batch, times 40 epochs.

So we transform: chop time into overlapping frames, take the frequency content of each frame, and stack the results into an image-like array where one axis is time and the other is frequency. Convolutions are good at images. That array is the spectrogram.

Step 2 — framing: window 1024, hop 320

The window length is how many consecutive samples go into one frequency analysis; the hop is how far you slide before the next one. CLAP uses 1024 and 320.

Notice the tension the two numbers resolve. Make the window longer and you get sharper frequency resolution but you smear transients — a sudden bark becomes a blur. Make it shorter and onsets get crisp but pitch turns to mush. This is the time-frequency uncertainty tradeoff, and 23 ms is the conventional compromise for general audio.

Then hop is chosen separately and smaller, which is the part beginners often miss. Overlap does not improve the frequency resolution of any single frame — that is set by the window. What it buys is temporal sampling density: the analysis is re-run every 7.26 ms so that whatever the window sees, it sees at many alignments.

Step 3 — the FFT, and 513 bins nobody wants

Each 1024-sample frame goes through a discrete Fourier transform, yielding 1024 complex coefficients. Real signals have conjugate-symmetric spectra, so only 1024/2 + 1 = 513 of them are independent. Take their magnitudes and discard phase.

Each of those 513 bins covers 44,100 / 1024 = 43.07 Hz, uniformly. That uniformity is the problem. Uniform spacing means bin 1 (43 Hz) to bin 2 (86 Hz) is one octave — a gigantic perceptual leap — while bin 400 (17,226 Hz) to bin 401 (17,269 Hz) is a difference no human can hear. The representation spends most of its dimensions on distinctions nobody perceives.

Step 4 — Mel: 513 bins to 64, warped to hearing

The Mel scale is an empirical mapping from frequency in hertz to perceived pitch, built so that equal steps on the Mel axis sound like equal pitch steps. The standard form:

Let us actually run CLAP's numbers rather than admire the formula. The band is 50 to 8000 Hz with 64 filters.

The last filter is 10.8 times wider than the first. That single ratio is the entire content of the Mel warp: fine resolution where hearing is fine, coarse where it is coarse. Each of the 64 outputs is a weighted sum of the 513 FFT magnitudes, with triangular weights over that filter's band. 513 numbers become 64, and the ones we kept are aligned with perception.

Step 5 — the logarithm

Mel energies span an enormous dynamic range: a loud transient can be a million times the energy of a quiet background. Feed that to a network and the loud bins dominate every gradient. So take the logarithm.

Two things happen, and both are useful. First, dynamic range compression: six orders of magnitude become a span of about 14 in natural log, comfortable for a network. Second, and more elegant, gain becomes an offset. Record the same sound twice as loud and every energy is multiplied by 4; in the log domain every value gains a constant log 4. A network with any translation tolerance — or a batch-norm layer — can shrug that off. Volume stops being a confound.

This also mirrors hearing: perceived loudness is roughly logarithmic in intensity, which is exactly why we measure it in decibels.

The tensor, finally

This is the paper's own notation from Section 2.1: "Let the processed audio be X_a such that X_a ∈ R^F×T where F are the number of spectral components (e.g. Mel bins) and T are the number of time bins." Now you can fill in both letters: F = 64, T = 690.

That is 44,160 floats, down from 220,500 raw samples — a 5.0× compression, and every surviving number is perceptually meaningful. In a batch of N pairs the audio tensor is (N, 64, 690), or (N, 1, 64, 690) once you add the channel axis a 2-D convolution expects.

The uncertainty tradeoff, with actual numbers

"Longer window, sharper frequency, blurrier time" is easy to say and worth quantifying, because the numbers explain why 1024 and not 256 or 4096.

Two things to notice in that table. First, the frame count does not change with the window — it is set entirely by the hop. Window and hop are genuinely independent knobs, controlling resolution and sampling density respectively. Second, the product of duration and bin width is constant at about 1.0: 5.8 ms × 172.3 Hz ≈ 1, 92.9 ms × 10.8 Hz ≈ 1. That is the time-frequency uncertainty principle, and no amount of engineering escapes it. You can only choose where to spend it.

One Mel filter, weight by weight

The filterbank is often treated as a black box. It is not; it is 64 triangles. Build filter 0 by hand.

Its three edge frequencies, from the Mel spacing computed above, are 50.0, 78.8, and 108.7 Hz. Convert each to an FFT bin index by dividing by the 43.07 Hz bin width:

The triangle rises from bin 1 to its peak near bin 1.83 and falls to bin 2.52. With only two or three FFT bins underneath it, the weights come out roughly 0.63 on bin 1 and 0.37 on bin 2. Compare filter 63, spanning 7368 to 8000 Hz: bins 171 through 186, sixteen bins wide, with a smooth triangular ramp over all of them.

That contrast is the Mel warp made concrete. Low filters are starved — barely more than one FFT bin each — while high filters average sixteen. If you wanted better low-frequency resolution you would need a longer window, and you would pay for it in time smearing. The filterbank does not create resolution; it allocates the resolution the FFT gave you.

Why not just feed the raw waveform?

Learned front ends exist and work — SincNet, LEAF, and the convolutional stems of wav2vec-style models all take samples directly. So the log-Mel choice is a decision, not a law. Weigh it honestly:

With 128,010 pairs, the strong prior wins. CLAP has nowhere near enough data to learn a front end; it barely has enough to learn an alignment. The log-Mel front end is 44,160 free, perceptually-organised numbers, and that is exactly the deal a small-data project should take.

The same pipeline, three ways

First by hand, on one frame, so nothing is magic:

the arithmetic for ONE frame# frame 0 = samples 0..1023 of a 220,500-sample clip
x = signal[0:1024]                  # 1024 floats
x = x * hann_window                  # taper the edges so the FFT sees no cliff
X = fft(x)                           # 1024 complex numbers
mag = abs(X[0:513])                 # keep the non-redundant half, drop phase
power = mag ** 2                     # 513 energies, one per 43.07 Hz bin

# mel filter 0 covers 50.0 -> 108.7 Hz, i.e. FFT bins 1..2 (43.07 Hz each)
# with triangular weights peaking at 78.8 Hz:
mel[0] = 0.63*power[1] + 0.37*power[2]
# mel filter 63 covers 7368 -> 8000 Hz, i.e. FFT bins 171..186 -- 16 bins wide
mel[63] = sum(w[i] * power[i] for i in range(171, 187))
logmel = np.log(mel + 1e-10)          # epsilon so log(0) is finite
# repeat for frames 1..689, hopping 320 samples each time -> (64, 690)

Then the honest loop, which is what the hand version generalises to:

python — numpy, from scratchimport numpy as np

SR, WIN, HOP, N_MEL, FMIN, FMAX = 44100, 1024, 320, 64, 50, 8000

def hz_to_mel(f): return 2595 * np.log10(1 + f / 700)
def mel_to_hz(m): return 700 * (10 ** (m / 2595) - 1)

def mel_filterbank():
    edges = np.linspace(hz_to_mel(FMIN), hz_to_mel(FMAX), N_MEL + 2)   # 66 points
    hz    = mel_to_hz(edges)                                       # 50.0 ... 8000.0
    bins  = np.floor((WIN + 1) * hz / SR).astype(int)
    fb    = np.zeros((N_MEL, WIN // 2 + 1))                        # (64, 513)
    for m in range(N_MEL):
        l, c, r = bins[m], bins[m + 1], bins[m + 2]
        fb[m, l:c] = np.linspace(0, 1, c - l)                       # triangle up
        fb[m, c:r] = np.linspace(1, 0, r - c)                       # triangle down
    return fb

def log_mel(sig):                       # sig: (220500,) float32
    win  = np.hanning(WIN)
    n    = 1 + (len(sig) - WIN) // HOP
    S    = np.stack([np.abs(np.fft.rfft(sig[i*HOP : i*HOP+WIN] * win)) ** 2
                      for i in range(n)], axis=1)          # (513, T)
    return np.log(mel_filterbank() @ S + 1e-10)          # (64, T)

And the one-liner you would actually ship:

python — torchaudio, the one-linerimport torchaudio.transforms as T

front_end = torch.nn.Sequential(
    T.MelSpectrogram(sample_rate=44100, n_fft=1024, win_length=1024,
                     hop_length=320, f_min=50, f_max=8000, n_mels=64),
    T.AmplitudeToDB())
X_a = front_end(waveform)      # (B, 64, 690)  <- the paper's X_a, exactly

The 5-second crop, and a cost the paper does not price

"During training, each audio clip is randomly truncated to a continuous segment of 5 secs, or padded if shorter." Two clauses, two consequences.

Random truncation is augmentation and it is also label noise. ClothoV2 clips run 15 to 30 seconds and their captions describe the whole clip. Take a random 5-second window of "a bow playing a stringed instrument in a one note tone repeatedly before violins join to create the melody" and there is a good chance your window contains only the bow, or only the violins. The caption is then partly wrong for the crop you are training on. Over 40 epochs the crop re-randomises, so on average the model sees every part of the clip paired with the full caption — the noise averages toward a weak-labelling signal rather than a systematic bias. But the per-step gradient is noisier than the pairing suggests, and this is one reason a large batch matters (Chapter 8).

Padding is worse than it looks, on exactly the tasks that failed. The paper never mentions this, but run the arithmetic on the downstream datasets and a pattern jumps out:

This is a correlation on a handful of points, not a proof, and Chapter 8 will give a much stronger explanation for the speech failures — the training captions simply do not describe speech content. But note the mechanism, because it is real: a CNN14 that global-pools over the time axis is averaging over 690 frames of which 552 are silence for a Speech Commands clip. Whatever evidence exists is diluted five-fold before the classifier ever sees it.

Chapter 3: Two Towers, One Space

We have X_a ∈ R^64×690 and a caption string. Section 2.1 of the paper compresses the next stage into two equations. This chapter expands them into every tensor shape, every parameter count, and every design decision behind them — because "two encoders and two projections" is a diagram, and a diagram is not understanding.

Where, in the paper's words, X̂_a ∈ R^N×V are the audio representations of dimensionality V, and X̂_t ∈ R^N×U the text representations of dimensionality U. N is the batch size. From Section 3.2 we can name V and U: V = 2048 and U = 768.

Tower A: CNN14, the borrowed ear

The audio encoder f_a is CNN14, taken from PANNs (reference [23] in the paper). The paper gives its vital statistics: 80.8 million parameters, embedding size 2048, and — the important part — "pretrained with 2M audio clips from AudioSet."

Read that last clause slowly. CLAP does not learn to hear from 128k captioned clips. It starts from an encoder that already learned to hear from two million labelled AudioSet clips, and then re-purposes that ear to speak to language. The 128k pairs are not building an audio front end; they are building a bridge. This is why 128k pairs is enough where CLIP needed 400M — CLIP's vision tower was trained from scratch.

The paper also states its own reason for the choice: CNN14 was picked "to provide a fair comparison to previous SoTA models." PANNs, Wav2CLIP, and AudioCLIP all use it, so any difference in the results is attributable to the training signal, not to a fancier encoder. That is good experimental hygiene and worth copying.

CNN14's internals come from the PANNs paper rather than this one, but they matter for tracing shapes. It is six convolutional blocks; each block is two 3×3 convolutions with batch norm and ReLU, followed by 2×2 average pooling. Channels double each block. Then a global pooling over what remains, and a fully-connected layer to the 2048-dim embedding:

Sit with the last row. Every temporal detail — "the horn honked twice", "the violins joined after the bow" — is pooled into a single 2048-vector. A clip-level embedding cannot, in principle, distinguish "dog then siren" from "siren then dog". CLAP is a bag-of-events model. That limitation is invisible in the ESC50 number and very visible if you ever try to use CLAP for temporal localisation, and it is precisely what later work (Chapter 9) attacks.

Tower B: BERT, the donated geometry

The text encoder f_t is BERT base uncased via HuggingFace: 110 million parameters. Two implementation details from Section 3.2 deserve attention.

"We limited the max text sequence length to 100 chars for computational efficiency." Not 100 tokens — 100 characters. Roughly 20 English words. Check that against the paper's own sample captions in Table 7 and you find genuine casualties: "A bow playing a stringed instrument in a one note tone repeatedly before violins join to create the melody" is 106 characters, so BERT never sees "the melody". "Leaves falling in a forest near a pond. Recorded in October 2017 in a German forest using a Zoom H2n." is 101 characters. Two of the twelve sampled captions get clipped — about 17%. In those cases the tail of the sentence, which is often where the second sound source appears, is discarded.

"The [CLS] token from the final layer of BERT is used as the text embedding with a size of 768." BERT prepends a special [CLS] token to every input; its final-layer vector is conventionally used as a whole-sequence summary, because self-attention lets it read every other token. So a 20-word caption collapses to one 768-dim vector, by exactly the same logic that collapsed 690 frames to one audio vector. Both towers are summarisers.

Dual encoder versus cross-attention — the decision that defines the family

There is another way to compare audio and text: let them attend to each other. Feed the 690 audio frames and the 20 text tokens into a joint transformer and let every token look at every frame. That is a cross-attention or fusion model, and it is strictly more expressive. CLAP does not do it. Understanding why is understanding the family.

The last row is decisive and often overlooked. The contrastive trick of Chapter 4 — getting N²−N negatives for free — only works because the two towers are independent. If scoring a pair required a joint forward pass, computing the full similarity matrix would cost N² passes instead of 2N, and a batch of 768 would need 589,824 of them. Contrastive pretraining at scale is not merely compatible with the dual-encoder design; it is enabled by it.

The projections: 2.88 million parameters that do the actual joining

Equation (2) is the entire "joint multimodal space." Two learnable linear maps, both landing in d = 1024:

Two questions worth asking here, both of which the paper leaves implicit.

Why linear and not an MLP? A linear map can rotate, scale, shear, and project — it can re-express the encoder's geometry in new coordinates, but it cannot fold or tear it. Neighbours stay neighbours. That is exactly what you want when your goal is to align two existing geometries rather than invent a new one. An MLP could align the training pairs more tightly while distorting the space in ways that do not generalise to unseen class names — and unseen class names are the entire product.

Why does text go up (768 → 1024) while audio comes down (2048 → 1024)? Because d is a choice about the shared space, not about either tower. Meeting in the middle keeps both projections cheap. Note the asymmetry it creates: L_a must discard information (2048 dims into 1024), while L_t cannot possibly add any (a 768-dim input can only ever span a 768-dim subspace of the 1024-dim output). The text embeddings live on a lower-dimensional sheet inside the shared space, and the audio embeddings must learn to land on that sheet.

Trace the whole flow

The model in 30 lines of PyTorch

python — the whole architectureimport torch, torch.nn as nn
from transformers import AutoModel, AutoTokenizer

class CLAP(nn.Module):
    def __init__(self, d=1024):
        super().__init__()
        self.audio_enc = cnn14_pretrained_on_audioset()   # 80.8M, out 2048
        self.text_enc  = AutoModel.from_pretrained("bert-base-uncased")  # 110M, out 768
        self.tok       = AutoTokenizer.from_pretrained("bert-base-uncased")
        self.La = nn.Linear(2048, d)                   # 2,097,152 weights
        self.Lt = nn.Linear(768,  d)                   #    786,432 weights
        # temperature: stored in log space so it stays positive under SGD
        self.logit_scale = nn.Parameter(torch.tensor(np.log(1 / 0.007)))

    def encode_audio(self, x):          # x: (N, 64, 690) log-mel
        h = self.audio_enc(x)             # (N, 2048)   <- X_hat_a
        e = self.La(h)                   # (N, 1024)   <- E_a
        return e / e.norm(dim=-1, keepdim=True)   # unit sphere

    def encode_text(self, captions):    # list[str], truncated to 100 chars
        b = self.tok([c[:100] for c in captions], padding=True, return_tensors="pt")
        h = self.text_enc(**b).last_hidden_state[:, 0]   # [CLS] -> (N, 768)
        e = self.Lt(h)                   # (N, 1024)   <- E_t
        return e / e.norm(dim=-1, keepdim=True)

Note the norm calls. The paper's Equation (3) writes a bare dot product, but a dot product between unnormalised vectors mixes "do these mean the same thing" with "how long are these vectors" — and vector length has no semantics here. Normalising to the unit sphere makes every entry of the similarity matrix a cosine in [−1, 1], which is what makes a single global temperature meaningful. Chapter 4 shows exactly why that matters.

Why L2-normalise, derived rather than asserted

Equation (3) writes a bare dot product. Every implementation normalises first. Here is the argument, in three steps.

Write the dot product in polar form: E_t · E_a = ‖E_t‖ · ‖E_a‖ · cosθ. Only the angle carries semantics — two embeddings mean the same thing when they point the same way, regardless of length. The two magnitudes are free parameters the model can inflate at will.

Now watch what the loss does with that freedom. The loss wants the diagonal large relative to its row. Doubling ‖E_a,i‖ doubles every entry in column i — useless for a column softmax, but it inflates the diagonal in row i relative to that row's other entries, which are unchanged. So the cheapest way to lower the loss is not to learn better semantics; it is to make the vectors of easy examples longer. The model discovers a shortcut, and length starts encoding confidence instead of meaning.

Normalising kills the shortcut. Every embedding sits on the unit sphere, every entry of C is exactly a cosine in [−1, 1], and the only way to raise the diagonal is to rotate. Two further benefits fall out: the temperature is now comparable across the whole matrix (a single τ is meaningful only if the raw scale is fixed), and cosine similarity at inference is just a dot product, so zero-shot needs no per-query normalisation.

What happens when an input degrades

A model is defined as much by its behaviour on bad inputs as on good ones. Trace each degradation through the shapes.

What it costs to run

How it was trained

Compare the two supervised evaluation setups the paper also runs, because they clarify what "the encoder learned" means. Supervised Feature Extraction freezes CLAP and trains 1 or 3 fully-connected layers on top (LR 10⁻³, 30 epochs) — that measures representation quality. Supervised Finetune unfreezes the audio encoder along with the attached layers (LR 10⁻⁴, 30 epochs) — that measures ceiling performance. The paper notes it did no hyperparameter grid search "due to computation constraints," which makes the numbers in Chapter 6 conservative rather than cherry-picked.

Chapter 4: The Similarity Matrix

We have two vectors per pair: E_a from the clip, E_t from its caption, both 1024-dimensional, both on the unit sphere. Nothing yet forces them to be near each other. This chapter builds the objective that does — and it builds it by hand, on a batch of three, with every intermediate number written down.

This is the load-bearing chapter of the paper. If you can reconstruct the 3×3 example below on a whiteboard, you can implement CLAP.

First, the objective that does not work

The naive move: maximise the similarity between each clip and its own caption.

Run gradient descent on that for five minutes and you get a model with a perfect score and zero value. It maps every clip and every caption to the same point on the sphere. Similarity 1.0 for all pairs, matched or not. This failure has a name — representational collapse — and it happens because the objective only ever says "closer", never "not that one."

The fix has to introduce a comparison. A pairing is only meaningful if it beats alternatives. So the objective must be: this caption should match this clip more than the other captions do.

The paper says exactly this, in the sentence right after Equation (3): "The similarity matrix C ∈ R^N×N has N correct pairs in the diagonal and N²−N incorrect pairs in the off-diagonal."

Equation (3): building C

Shapes first, always. E_t is (N, d) and E_a is (N, d), so E_a^T is (d, N) and the product is (N, N). Entry C[i][j] is a scaled inner product between text i and audio j.

Since both E's are L2-normalised, each raw entry is a cosine: 1 means "same direction", 0 means "unrelated", −1 means "opposite". That bounded range is exactly what makes a single global temperature sensible.

The temperature puzzle — read this before you implement

Section 3.2 says: "The temperature parameter τ is learnable and initialised to 0.007. To prevent training instability, the logits scaled by τ are clipped to a maximum value of 100."

Take Equation (3) literally with τ = 0.007 and multiply your cosines by 0.007. Every logit now lies in [−0.007, 0.007]. Softmax over a batch of 768 such logits is indistinguishable from uniform: every probability is about 1/768, the loss sits at ln(768) = 6.644 forever, and the gradients are microscopic. The model cannot learn. Also, no product of a number in [−1,1] with 0.007 will ever approach 100, so the stated clipping would never fire.

Both problems vanish under the convention CLIP established and CLAP inherits: 0.007 is the temperature, and the multiplier applied to the cosines is its reciprocal, 1/0.007 = 142.86. Now the numbers all make sense at once:

This is reasoning about the paper, not a quotation from it — but the arithmetic only closes one way, and the reference implementations of both CLIP and CLAP store a learnable logit_scale initialised to log(1/0.007) and exponentiate it.

And the clip? Once the model is right — diagonal larger than every off-diagonal entry in its row — the loss decreases monotonically as the scale grows. Gradient descent notices. Left alone, the learnable scale runs away, logits explode, exp() overflows in mixed precision, and training dies. Clipping the scaled logits at 100 is a hard ceiling on that runaway. It is a numerical guardrail, not a modelling choice.

Equation (4): the symmetric cross-entropy

with, in the paper's notation, ℓ_k = (1/N) ∑_i log diag(softmax(C)) taken "along text and audio axes respectively."

Unpack that in three moves.

Move 1 — softmax along an axis turns a similarity row into a choice. Take row i of C: the similarities between caption i and all N clips. Softmax it and you have a probability distribution over "which clip does this caption describe?" The right answer is clip i, the diagonal entry.

Move 2 — the diagonal is the label, so no labels are needed. Cross-entropy with a one-hot target reduces to −log(probability of the correct class). The correct class for row i is index i. The "labels" are literally [0, 1, 2, …, N−1]. That is the whole reason this is called self-supervised.

Move 3 — do it both ways. Softmax along rows answers "given this caption, which clip?" (text-to-audio retrieval). Softmax along columns answers "given this clip, which caption?" (audio-to-text retrieval). These are different questions with different answers, and optimising only one leaves the other free to be bad. Averaging the two gives the symmetric loss.

The 3×3 batch, worked by hand

Everything above becomes concrete now. To make the arithmetic doable on paper we shrink the shared space from d = 1024 to d = 2 and the batch from N = 768 to N = 3. Nothing else changes — the equations are identical.

Our toy space has two axes: horizontal is impulsive / percussive energy, vertical is sustained / tonal energy. Three pairs:

Step 0 — normalise. The projections L_a and L_t output arbitrary-length vectors; we put them on the unit circle. Do one by hand so the step is not a black box. Suppose L_t emits (3.984, 0.348) for caption 1:

Doing that for all six gives our batch:

Step 1 — the raw similarity matrix. Nine dot products. Here are three of them written out in full; the rest follow identically.

All nine, with rows indexed by text and columns by audio:

Before scaling, read the structure. The diagonal is the largest entry in every row and every column — this batch is already mostly right. But look at cell (1,2): 0.819. Caption "a dog barks" is nearly as close to the knock audio as to the dog audio. Both are impulsive. That is our hard negative, and it will dominate the loss.

Step 2 — scale to logits. The real model multiplies by about 143, which would saturate this well-behaved toy instantly. For hand arithmetic take a teaching scale of s = 4, and we will study the real value in a moment. C = 4 · Ĉ:

Step 3 — softmax along the text axis (rows). Exponentiate each entry, then divide by the row sum.

Step 4 — softmax along the audio axis (columns). Same matrix, same numbers, different normalisation direction. Column 1 asks: given the dog audio, which of the three captions describes it?

Step 5 — the symmetric loss.

Compare to the guessing baseline ln 3 = 1.099. The batch is learning.

Now turn the temperature knob

Take the same Ĉ and vary the scale s. Nothing about the model changes — only how sharply we read its similarities.

That last row is the single most common confusion about CLIP-family temperatures, so state it plainly: a temperature is not "sharp" or "soft" in the abstract. It is sharp or soft relative to the typical spread of your similarities, and that spread shrinks as 1/√d.

Press Show collapse to see the failure from the top of this chapter play out: with only the diagonal in the objective, every embedding migrates to one point, the whole matrix turns bright, and the model has learned nothing. The off-diagonal is not a nuisance term. It is the entire supervision.

The same loss, three ways to write it

Hand arithmetic first — the exact steps you just did, transcribed:

the arithmetic, for row 1 of the 3x3 batchlogits  = [ 4*0.985 , 4*0.819 , 4*(-0.088) ]  = [ 3.940 , 3.276 , -0.352 ]
exps    = [ 51.42   , 26.47   , 0.703 ]
sum     = 78.59
probs   = [ 51.42/78.59 , 26.47/78.59 , 0.703/78.59 ] = [ 0.654 , 0.337 , 0.009 ]
loss_1  = -ln(0.654) = 0.424            # the diagonal entry is index 0 for row 0
# repeat for rows 2,3 -> 0.591, 0.086 ; average -> l_text = 0.367
# repeat down the COLUMNS         -> 0.359, 0.493, 0.295 ; l_audio = 0.382
# L = 0.5 * (0.367 + 0.382) = 0.375

Then step-by-step numpy, which is the same thing with loops:

python — numpy, every intermediate exposedimport numpy as np

Ea = np.array([[ 0.966, 0.259], [ 0.766, 0.643], [-0.174, 0.985]])   # (3, 2)
Et = np.array([[ 0.996, 0.087], [ 0.574, 0.819], [-0.342, 0.940]])   # (3, 2)
Ea = Ea / np.linalg.norm(Ea, axis=1, keepdims=True)      # unit sphere
Et = Et / np.linalg.norm(Et, axis=1, keepdims=True)

s  = 4.0                          # the real model uses 1/0.007 = 142.86
C  = s * Et @ Ea.T             # (3, 3)  <- Equation (3)
C  = np.minimum(C, 100.0)         # the paper's stability clip

def softmax(x, axis):
    x = x - x.max(axis=axis, keepdims=True)   # subtract max: exp never overflows
    e = np.exp(x)
    return e / e.sum(axis=axis, keepdims=True)

P_text  = softmax(C, axis=1)      # each ROW sums to 1: caption -> which clip?
P_audio = softmax(C, axis=0)      # each COL sums to 1: clip -> which caption?

l_text  = -np.log(np.diag(P_text )).mean()      # 0.367
l_audio = -np.log(np.diag(P_audio)).mean()      # 0.382
L = 0.5 * (l_text + l_audio)                    # 0.375   <- Equation (4)
print(L, "vs guessing baseline", np.log(len(Ea)))   # 0.375 vs 1.0986

And the one-liner you would actually train with. The insight that makes it a one-liner: cross-entropy against the identity is cross-entropy against the index vector [0, 1, …, N−1].

python — PyTorch, production formimport torch.nn.functional as F

logits  = model.logit_scale.exp().clamp(max=100) * E_t @ E_a.T   # (N, N)
targets = torch.arange(len(logits), device=logits.device)          # [0,1,...,N-1]
loss    = 0.5 * (F.cross_entropy(logits,   targets) +      # rows: text -> audio
                 F.cross_entropy(logits.T, targets))        # cols: audio -> text

Three lines. logits.T is the entire content of "symmetric". torch.arange is the entire content of "no labels needed".

What the gradient actually does to the geometry

Differentiate the row term for pair i and the structure is beautifully simple. Writing p_ij for the softmax probability that caption i matches clip j:

Read that as a tug-of-war. The first term pulls caption i toward its own clip. The second pushes it away from a probability-weighted average of all clips in the batch — so the clips the model currently finds confusable pull hardest. In our batch, p₁₂ = 0.337, so the knock audio exerts a third of the repulsion budget, while the violin at p₁₃ = 0.009 exerts almost none.

Two consequences follow immediately, and both reappear in Chapter 8. Larger batches mean more candidates in that weighted average, so a harder and more informative push. And the temperature multiplies the whole gradient by s, which is a second, sneakier reason the learnable scale wants to grow.

Three things that break this loss, and how to spot them

Chapter 5: Zero-Shot Is Retrieval in Disguise

Training is over. You have two encoders and two projections. You have no classifier. Someone hands you 2,000 five-second clips and asks you to sort them into fifty environmental categories — categories that appear nowhere in your training objective.

Section 2.2 of the paper answers in three sentences, and this chapter is the unpacking of those three sentences into something you could implement in an afternoon.

The reframe that makes it obvious

Return to Chapter 0's prison. A classifier's last layer is z = W h with W ∈ R^C×D, one learned row per class. Now write CLAP's zero-shot computation the same way:

These are the same equation. In both cases a matrix of shape (C, d) multiplies a vector of shape (d,) to give C logits. The only difference — and it is the whole paper — is where the matrix comes from.

Worked, with numbers

Same 2-D toy space as Chapter 4 (horizontal = impulsive, vertical = tonal). One test clip — a dog barking, embedded at E_a = (0.966, 0.259). Four candidate classes, each turned into a prompted caption and embedded:

Multiply by the logit scale (s = 4 again, for readable arithmetic) and softmax:

Prediction: dog, 58.5% confident. And look at the runner-up: rooster, at 34.8%. Nothing told the model that dogs and roosters are both impulsive animal vocalisations — the language geometry did. A one-hot classifier's second-place guess carries no such information.

The multilabel case is different and the paper is careful about it. FSD50K and AudioSet clips can contain several events at once, so "which one class" is the wrong question. There you apply a sigmoid to each logit independently:

On the same logits that gives 0.982, 0.970, 0.844, 0.414 — no longer summing to one, because "dog" being present says nothing about whether "rain" is also present. Which is exactly why FSD50K and AudioSet are scored with mAP rather than accuracy in Table 1: they are ranking problems, not choice problems.

What it costs

Two experiments worth running in that widget before moving on. First, turn off the correct label and watch the probabilities redistribute over the survivors — the model is still forced to sum to one, so zero-shot inherits Chapter 0's abstention problem unless you threshold the raw cosine. Second, add the deliberately unusual candidate and see it rank sensibly anyway. That is generalisation to a class that exists only as a string.

The prompt problem, and the paper's honest table

There is a distribution mismatch hiding in the procedure, and Section 4.4 names it: "The training data of CLAP consists of natural language captions containing one or more sentences. However, the vast majority of datasets have class labels defined by a few words — 'dog barking' and 'sneezing'. Using single words instead of language description affects how Zero-Shot learning transfers."

The text encoder spent 40 epochs reading full sentences. At inference you hand it the fragment "dog". That fragment lands in a region of the text space the model rarely visited. The fix is a prompt template: wrap the class name in a sentence shaped like the training data. The paper's default is "This is a sound of [class label]".

Here is Table 3, complete, and it does not say what you expect:

One more honest note. The paper writes that prompting "leads to a 5% acc increase." The table's own span is 0.826 − 0.786 = 4.0 points, and against the more natural baseline of the bare class label it is 1.4 points. The effect is real and free, but it is a percentage point or two, not a transformation. Do not let a rounded claim in prose overwrite a table you can read.

The three domains that needed a different prompt

Section 3.3 records a detail most readers skim: "The prompt was kept the same for all the domains except three."

These three exceptions are not tuning noise; they are a map of where the caption distribution runs out. Captions describe what happened acoustically. They rarely annotate a speaker's emotional state, almost never transcribe words, and essentially never count talkers. In Chapter 6 those same three domains produce CLAP's three worst zero-shot numbers. The prompts were the first warning.

Run the same machine backwards: retrieval

Nothing in Chapter 5 says the class list must be short and the audio list long. Swap which axis you loop over and the same model does something else entirely.

All five are the same two lines of code with the loops exchanged. That is what "joint embedding space" actually buys, and it is why CLAP's descendants show up inside data pipelines as often as inside classifiers.

The multilabel case, worked

Softmax forces a choice. FSD50K clips genuinely contain several events, so the paper switches to sigmoid there — and the switch has consequences worth walking.

With the same four logits (3.985, 3.464, 1.690, −0.349), sigmoid gives:

Every score is high, including "rain" at 0.844 and "violin" at 0.414 — and their sum is 3.21, not 1. That is not a bug; independence is the point. But it exposes a real difficulty: to turn these into predictions you need a threshold, and cosine similarities between unit vectors are bunched. Because raw cosines rarely go below zero for related concepts, σ(s·cos) sits above 0.5 for almost everything, and the useful threshold depends on both s and the dataset.

This is exactly why FSD50K and AudioSet are scored with mAP — mean average precision — rather than accuracy. mAP evaluates the ranking and never asks you to pick a threshold. When you see a mAP number, read it as "how well does the model order the classes", not "how often is it right".

Zero-shot in three forms

the arithmetic, one clip against four classescos     = [ 0.996 , 0.866 , 0.423 , -0.087 ]     # dot products, both sides unit-norm
logits  = 4 * cos = [ 3.985 , 3.464 , 1.690 , -0.349 ]
exps    = [ 53.77 , 31.95 , 5.42 , 0.71 ]
sum     = 91.85
probs   = [ 0.585 , 0.348 , 0.059 , 0.008 ]     # multiclass: softmax
sigmoid = [ 0.982 , 0.970 , 0.844 , 0.414 ]     # multilabel: independent
prediction = argmax = class 0 = "dog"

python — the whole zero-shot classifierimport numpy as np, torch

# ---- ONCE, at "deployment". No gradients anywhere in this file. ----
classes  = ["dog", "rooster", "rain", "violin", "chainsaw"]      # type anything
prompts  = [f"this is a sound of {c}" for c in classes]
with torch.no_grad():
    E_text = model.encode_text(prompts)      # (C, 1024), unit-norm, CACHE THIS

# ---- PER CLIP ----
with torch.no_grad():
    E_a  = model.encode_audio(log_mel(clip))     # (1, 1024), unit-norm
cos      = E_text @ E_a.T                       # (C, 1)  <- cosine similarities
logits   = model.logit_scale.exp() * cos
probs    = logits.softmax(dim=0)                 # multiclass / binary
scores   = torch.sigmoid(logits)                 # multilabel (FSD50K, AudioSet)
print(classes[probs.argmax()])

# adding a 6th class at 3am:
E_text = torch.cat([E_text, model.encode_text(["this is a sound of a smoke alarm"])])
# done. that is the entire deployment story.

python — the one-liner, for the whole test set at oncepreds = (model.encode_audio(X) @ model.encode_text(prompts).T).argmax(-1)
# (N,1024) @ (1024,C) -> (N,C) -> (N,)   ... the temperature cancels under argmax

That last comment is worth a beat. Under argmax the temperature is irrelevant — scaling every logit by the same positive constant cannot change which is largest. Temperature matters for the loss during training and for calibrated probabilities at inference. It does not matter for accuracy. Many people tune it hoping otherwise.

Chapter 6: 16 Tasks, 8 Domains

Every claim so far has been mechanical: this is the tensor, this is the loss, this is the inference procedure. Now the empirical question. A model trained on 128,010 captioned clips, with no classifier and no task-specific training, is pointed at sixteen benchmarks it has never seen. What happens?

The answer is not uniform, and the non-uniformity is the most instructive part. This chapter shows every number, including the ones that make CLAP look bad, because the shape of the failures is what tells you what the model actually learned.

How to read Table 1

Five rows, and confusing them is the fastest way to misread the paper:

Rows 3 and 4 are not comparable in fairness — one had labelled data for the task and one did not. That asymmetry is exactly what makes some of the numbers below remarkable.

Metrics differ by task, and the caption of Table 1 is precise about it: DCASE17 employs F1, FSD50K and AudioSet employ mAP, everything else uses accuracy. Higher is better everywhere. Random also differs by task, because it is set by the number of classes: 1/50 = 0.02 for ESC50, 1/2 = 0.5 for GTZAN music-versus-speech, and under 0.0018 for AudioSet's 527 multi-label classes.

The three metrics, so the table is readable

Comparing 0.826 on ESC50 with 0.3024 on FSD50K is meaningless unless you know they measure different things.

Three practical consequences. mAP numbers look low compared to accuracy numbers and are not worse for it — 0.30 mAP over 200 classes is a strong result. F1 depends on a threshold, so the number carries an unstated operating-point choice. And chance level differs wildly: 0.02 for ESC50, under 0.0018 for AudioSet, 0.5 for a binary task. Always locate the chance level before reacting to a score.

All sixteen, in full

Bold = CLAP is state of the art in its row's setting. Italic = CLAP is barely above chance. * The paper's random row lists 0.05 and 0.1 for the DCASE17/US8K pair; the values matching class counts are 1/17 ≈ 0.059 and 1/10 = 0.1. Dashes mean the paper reports no number in that cell.

The highs, and why they are genuinely surprising

ESC50: 82.6% zero-shot. Fifty environmental sound classes, five-second clips. The prior zero-shot best was AudioCLIP at 69.40%, so this is an absolute 12-point jump on a benchmark that had been competitively worked. And the paper flags the comparison that gets quoted everywhere: reported human accuracy on ESC50 is 81%. A model with no classifier, told the class names in English at inference time, beats the humans who defined the categories.

GTZAN Music vs Speech: 100% zero-shot. Perfect. And the supervised literature benchmark is 99.2%. Zero-shot beat fully supervised. This is a two-class task on 30-second clips, so it is the easiest cell in the table — but it also demonstrates that when the distinction is one the caption corpus talks about constantly ("a man speaks" versus "music plays"), the language prior is simply sufficient.

FSD50K: 30.24 mAP zero-shot, against Wav2CLIP's 3.02. An absolute 27-point mAP gain, the largest relative jump in the table. FSD50K is multi-label over 200 classes, and the paper notes CLAP's training data includes FSD50K clips — but paired with their free-text titles and descriptions, with the class labels deliberately ignored. So the model saw the audio and a human sentence about it, and then reconstructed a 200-way taxonomy it was never shown.

UrbanSound8K: 73.24% vs AudioCLIP's 65.31%. An 8-point absolute gain on ten urban sound classes.

The lows, stated plainly

The paper is honest here and so is this lesson: "CLAP (ZS) performed better than random on all downstream tasks and achieved good to slightly better than random on some music and speech-related tasks."

The supervised story: SoTA on five

Zero-shot is the headline, but the paper also asks a quieter question: is CLAP's audio representation good, independent of the zero-shot trick? Two setups answer it. Feature extraction freezes CLAP and trains 1 or 3 fully-connected layers on top. Finetune unfreezes the audio encoder along with those layers. CLAP (Best) is whichever won.

Result: state of the art on five datasets — GTZAN Music vs Speech (100%), GTZAN Genres (91.30%), Mridangam Stroke (97.94%), Mridangam Tonic (95.34%), and Vocal Sound (97.95%). Elsewhere it trails the literature by at most seven points, with the worst case being RAVDESS emotion at 64.36% against a 81.82% benchmark.

Table 6 puts CLAP head to head with the standard audio representation models. A few comparisons worth carrying:

Look at the Speech Commands row carefully, because it settles an important question. Frozen CLAP features score 0.3708 — worse than OpenL3 and far worse than Wav2Vec2's 0.8785. But finetuned CLAP reaches 0.9683, the best in the row. So the audio encoder is perfectly capable of keyword spotting once gradients are allowed to reshape it. What CLAP's frozen space lacks is not capacity; it is that the contrastive objective never had a reason to encode which word was spoken. Nothing in 128k sound captions ever said "the person said 'left'."

Domain by domain, with the mechanism

The headroom ordering is not a mystery once you ask, for each domain, would a person writing a caption about this clip mention the thing being classified?

Read the two columns together and the model stops looking mysterious. CLAP's zero-shot ability on a task is roughly the frequency with which that task's distinction appears in written descriptions of sound. It is not a general audio understanding system; it is a system that understands audio through the vocabulary humans use to write about it.

Notice the Mridangam pair, which is a clean natural experiment. Stroke classification captures 27% of the headroom; tonic classification captures 3.6%. Both are 5-fold tasks on the same 0.81-second clips from the same instrument. The difference is that a stroke is a percussive event with a distinct timbre — the kind of thing captions describe — while a tonic is a pitch, which requires naming a note. Same audio, same encoder, same prompt template; the gap is entirely in whether language talks about it.

What the ESC50 number does and does not mean

82.6% zero-shot on ESC50, above 81% human accuracy, is the line that got CLAP quoted. Two caveats keep it honest.

It is a comparison between different things. The human number comes from crowdworkers listening once and choosing among fifty categories, with human failure modes: attention, memory for the category list, unfamiliarity with some sounds. The model is not "better at hearing"; it is better at this particular forced-choice protocol.

The supervised ceiling is still higher. CLAP's own supervised finetune reaches 96.70% on ESC50, and the literature benchmark is 97.15%. Zero-shot bought you 82.6% for zero labelled examples. That is an extraordinary price-performance point, not an accuracy record.

Reading a benchmark table adversarially — five habits

Table 1 is well constructed, which makes it a good place to practise the habits that catch badly constructed ones.

1. Find the chance level before the score. 0.4746 on Beijing Opera sounds mediocre until you notice it is a 4-class task with chance at 0.25 — and then 0.4746 sounds much better. 0.4945 on Vocal Sound is a 6-class task with chance at 0.1667, which is better still. The raw numbers are nearly identical; the achievements are not.
2. Check that the comparison rows are comparable. "Benchmark (Best)" had labelled training data; "CLAP (ZS)" had none. Any row-to-row comparison across that line is measuring two different things, and the paper is careful to separate them — many papers are not.
3. Look for the missing cells. AudioSet has no CLAP (Best) entry. There is no zero-shot benchmark for eleven of the sixteen tasks. Absence is usually "nobody has done it", occasionally "we did and it looked bad" — and you cannot tell which from the table alone.
4. Recompute anything the prose claims. "A 5% acc increase" from prompts is 4.0 points in Table 3. "CLAP beat Wav2CLIP by an absolute 27% mAP" checks out exactly: 0.3024 − 0.0302 = 0.2722. One claim survives, one does not.
5. Ask what the average is averaging. Table 2's "Avg. ZS score" mixes accuracy, mAP, and F1 across chance levels from 0.0018 to 0.5. The comparison between its rows is valid; the absolute value is not interpretable. Chapter 7 returns to this.

Chapter 7: Which Tower Matters?

CLAP has two pretrained encoders and a training budget. You can update both, one, or neither. That is four experiments, and the paper ran all four. The result is the most quietly consequential finding in the paper, because it tells you what to build next.

Before you read the table, commit to a prediction. Which encoder do you expect to matter more? Most people say audio — the audio side is the one being taught something new, the text side already speaks English. Write your guess down.

Table 2, in full

Two of those rows are unsurprising. Both frozen is worst; both trainable is best; the paper calls this "expected because unfreezing both encoders allows them to learn the multimodal information from the pairs."

The middle two rows are the finding. Isolate the gains:

Unfreezing the text encoder is worth thirty-three times as much as unfreezing the audio encoder, averaged across the sixteen tasks. The paper's own reaction is candid: "Surprisingly, unfreezing the text encoder performed better than unfreezing the audio encoder. Our intuition was that unfreezing the audio encoder would enable learning beyond the SEC coming from the pretrained AudioSet information."

Why would the text side matter more? Three mechanisms

The paper does not explain the asymmetry beyond noting the parallel to CLIP. Here are three mechanisms that fit the evidence, flagged as analysis rather than as claims from the paper.

1. Asymmetric domain shift. CNN14 was pretrained on 2M AudioSet clips — the same kind of audio, labelled with the same kind of sound-event concepts. Its features arrive nearly on-task. BERT was pretrained on Wikipedia and BookCorpus, where nobody writes "an insect buzzing in the foreground as birds chirp in the background." Audio captioning is a distinctive, narrow, oddly-structured genre. The text encoder has much further to travel, so it gains more from being allowed to travel.

2. The alignment burden falls where the geometry is coarser. Both towers must be brought into one space, and something has to bend. If the audio geometry is already a good match for sound-event structure, the cheaper solution is to bend the text geometry toward it. Gradient descent will find whichever adaptation lowers the loss for less movement.

3. Forgetting risk is asymmetric. 128,010 pairs is a small dataset for a 80.8M-parameter encoder that learned from 2M clips. Unfreezing the audio tower risks overwriting the AudioSet knowledge with something narrower — and the frozen-audio rows suggest that knowledge is worth a great deal. The text tower has less to lose: BERT's general language ability is not what CLAP needs, only its ability to place caption-like sentences sensibly.

What "frozen" means mechanically

Freezing is not the same as removing. A frozen encoder still runs forward, still produces its 2048- or 768-dimensional vector, and still contributes to the loss. What changes is that requires_grad is off for its parameters, so the optimiser never updates them. Concretely:

python — the four regimes of Table 2def set_regime(model, train_audio, train_text):
    for p in model.audio_enc.parameters(): p.requires_grad = train_audio
    for p in model.text_enc.parameters():  p.requires_grad = train_text
    # the two projections and the temperature ALWAYS train — in all four rows.
    for p in [*model.La.parameters(), *model.Lt.parameters()]: p.requires_grad = True
    model.logit_scale.requires_grad = True

# row 1 of Table 2: 2.88M trainable of 193.68M  -> ESC50 55.55%
# row 2:          83.68M trainable              -> ESC50 64.15%
# row 3:         112.88M trainable              -> ESC50 76.31%
# row 4:         193.68M trainable              -> ESC50 82.60%

That comment block is the whole ablation, and it exposes something the prose glides over: the projections train in every row. So "both frozen" is not "no learning" — it is "learning restricted to a linear map between two fixed geometries." 55.55% on a 50-way task is what a rotation is worth.

Now the gradient story, which explains the asymmetry mechanically. The loss reaches the audio encoder only through L_a, and reaches the text encoder only through L_t. Both paths are equally short. What differs is what the gradient asks for at each end:

The LiT echo

The paper connects its finding to reference [27] — Zhai et al., LiT: Zero-Shot Transfer with Locked-image Text Tuning — noting "a similar insight was found for CLIP models in Computer Vision." LiT's headline is that locking the image tower and tuning only the text tower gives better zero-shot transfer than tuning both, precisely because a strong pretrained vision encoder should not be disturbed by a noisy image-text corpus.

Note the difference in conclusion, though, because it is easy to over-read. LiT found locked-image tuning best. CLAP found both-unfrozen best, with text-unfrozen the strongest single choice. Same direction, different endpoint — most plausibly because CLAP's 128k pairs are clean human captions rather than web alt-text, so overwriting the audio tower is less dangerous than overwriting a vision tower with internet noise.

The implication the authors draw, and why it is the most useful sentence in the paper

Unpack the engineering consequence. If the audio tower does not need to change much, then the audio tower is a swappable component. Take any strong pretrained audio encoder — PANNs, an Audio Spectrogram Transformer, HTSAT, a self-supervised speech model — bolt on a linear projection, train against captions with a language tower that is allowed to move, and you have converted a fixed-taxonomy classifier into an open-vocabulary one.

That is exactly what the field then did. Every CLAP successor in Chapter 9 is, structurally, this recipe with a different audio tower.

A caution about the average

One number in Table 2 deserves suspicion, and reading it carefully is good practice. "Avg. ZS score" is the mean of CLAP's zero-shot performance across all sixteen downstream tasks. But those tasks use different metrics (accuracy, mAP, F1) with different chance levels (0.0018 to 0.5). Averaging them is averaging apples, oranges, and a percentage of a percentage.

Concretely: GTZAN Music vs Speech contributes 1.000 to that mean while its chance level is 0.5, and AudioSet contributes 0.058 while its chance level is under 0.002. A tenth of the average is being supplied by a two-class task that any model solves. So the absolute values 0.2809 and 0.3265 do not mean "28% accurate" in any interpretable sense.

What survives the objection is the comparison. All four rows aggregate the same sixteen tasks the same way, so the ordering and the relative gaps are meaningful even though the level is not. The 33× asymmetry is a ratio of two differences computed under an identical aggregation, and it is robust to the aggregation being crude. The ESC50 column, which is a single clean accuracy on a single task, tells the same story at 2.4×.

What you would do with this in practice

The fifth row nobody ran

Table 2 has four rows because there are two binary switches. But the interesting design space has more dimensions, and naming the missing experiments sharpens what the four rows actually established.

Two of these five would have been nearly free — the nonlinear-projection run and the staged unfreeze reuse the same code and the same data. Their absence is a reminder that a four-page paper reports the experiments that fit, not the experiments that exist.

The practical recipe, in code

python — the "audio encoder of choice" recipe# Table 2 row 3: freeze audio, train text + both projections.
# Second-best zero-shot, ~42% of the trainable parameters, and your
# audio tower stays swappable next quarter.

audio_enc = load_any_pretrained_audio_encoder()      # PANNs, AST, BEATs, HTSAT...
for p in audio_enc.parameters(): p.requires_grad = False
audio_enc.eval()                                    # also freezes BatchNorm statistics —
                                                  # forgetting this silently trains the encoder

text_enc  = AutoModel.from_pretrained("bert-base-uncased")   # trainable
La, Lt    = nn.Linear(audio_dim, 1024), nn.Linear(768, 1024)

opt = torch.optim.Adam([
    {"params": text_enc.parameters(), "lr": 1e-5},   # gentle on the pretrained tower
    {"params": [*La.parameters(), *Lt.parameters(), logit_scale], "lr": 1e-3},
])
# with the audio tower frozen you can also PRECOMPUTE every E_a once and
# train on cached vectors — an order-of-magnitude cheaper per epoch.

That last comment is the underrated payoff. A frozen audio tower means the expensive forward pass happens once, offline, for the whole corpus. Training then reads 1024-float vectors off disk. On a single GPU that turns a multi-day run into an afternoon — and Table 2 says you give up 0.0156 of average zero-shot to get it.

Chapter 8: What the Paper Whispers

The main text of a four-page conference paper is a highlight reel. The interesting failures live in the appendix, in a footnote, or in a number that does not quite match its prose. CLAP has three appendix findings that matter more than most of Section 4, and a handful of smaller discrepancies worth cataloguing. This chapter reads the quiet parts out loud.

Whisper 1 — 1.7 million more pairs made the model worse

Appendix C.2, in full, is the most useful paragraph in the paper. AudioSet has 2M clips. The authors tried to turn them into pairs by "constructing the text description with the title and the class label(s)." Adding about 1.7M such pairs to the existing 128k produced this:

Fourteen times more data, and the flagship number dropped by fifteen points. In an era whose default assumption is that more pairs are always better, this is a result worth memorising.

The authors diagnose it precisely: "In AudioSet, often the YouTube titles and descriptions do not describe the acoustic content of the video segment under consideration but instead describe the video as a whole."

Picture the failure. A ten-second AudioSet segment contains a dog barking. Its parent video is titled "Ep. 42 — Our Trip to Portugal (Vlog)". The constructed caption describes a travel vlog. The clip contains a bark.

Now do the arithmetic on the mixture. After the addition, 1.7M of 1.828M pairs — 93% — come from the noisy source. The curated captions are outnumbered thirteen to one. The model is not learning "sound events plus some noise"; it is primarily learning "YouTube video titles, with an occasional real caption."

And the one improvement is just as diagnostic. Speech Commands went from 10% to 15%. The authors' guess: "Perhaps due to the large amount of audio containing speech in AudioSet." Even bad captions attached to speech-heavy audio taught the model something about speech — because the base corpus taught it essentially nothing. A signal from noise is still a signal when your baseline is zero.

Whisper 2 — the speech gap has five converging causes

Chapter 6 established the pattern: sound events near the human ceiling, speech near chance. The paper offers one cause. There are at least five, and they all point the same way.

Cause 1 dominates, and we know it dominates because of the Table 6 control from Chapter 6: finetuned CLAP reaches 96.83% on Speech Commands, the best number in that row. Causes 2 through 4 would still apply after finetuning. So they cannot be what is limiting the frozen model. The bottleneck really is that the contrastive objective was never given a reason to encode which word was spoken.

The paper's own forecast is reasonable and, in hindsight, correct: "We posit that as we increase training data and increase human speech-based captioning, CLAP's performance on speech datasets will increase." That is precisely the arc from CLAP to speech-aware audio-language models (Chapter 9).

Whisper 3 — the batch-768 "anomaly" probably is not one

Appendix C.1 reports a batch-size study: 4 to 24 GPUs, batch sizes 32 to 768, measuring average zero-shot performance across the downstream tasks. Larger batches helped — until 768, where performance dropped. The authors write: "This might be an anamoly in an increasing zero-shot performance trend. We leave the larger batch size investigation to future work."

Fair enough for a four-page paper. But two mechanisms predict exactly that drop, and both are computable from numbers the paper already gives.

Mechanism A: the learning rate was not rescaled, so the big-batch run got 4% of the updates. Section 3.2 fixes the learning rate at 10⁻³ and the schedule at 40 epochs. Epochs are fixed, so the number of optimiser steps falls linearly with batch size:

With a fixed learning rate, the batch-768 model took one twenty-fourth as many gradient steps as the batch-32 model. Standard practice would scale the learning rate with batch size (linearly, or by the square root) to compensate. Nothing in the paper says that was done. The "anomaly" may be a model that simply had not finished training.

Mechanism B: false negatives grow with N², and this corpus has duplicate audio by design. Recall Chapter 1: 128,010 pairs cover only 90,947 unique clips, because ClothoV2 supplies five captions per clip and MACS supplies several. So two entries in a batch can be the same audio with different captions — and the loss will insist that caption i matches clip i and not clip j, when clips i and j are byte-identical. That is an unsatisfiable constraint, and it actively penalises correct behaviour.

How often? Count the colliding pairs in the corpus — 5,929 ClothoV2 clips contribute C(5,2) = 10 collisions each, MACS contributes roughly 7.5 per clip over 3,930 clips — giving about 88,800 colliding pairs out of the C(128010, 2) ≈ 8.19 billion possible. The probability that a random pair of batch entries collides is about 1.08 × 10⁻⁵, and a batch of N contains C(N,2) pairs, so:

At batch 32 a collision essentially never happens. At batch 768 there are about three per batch, every batch, for all 40 epochs. Each one is a contradiction the optimiser cannot resolve, and it is aimed at exactly the captions the corpus most wants the model to treat as equivalent.

Whisper 4 — the overlap question, asked and answered

Any zero-shot claim invites the same challenge: how much of the evaluation did the model already see? Trace the corpora and the answer is more interesting than a simple yes or no.

Whisper 5 — what a four-page paper cannot tell you

Some gaps are not errors, just consequences of the format. Naming them is part of reading a paper honestly, and each one is a question you would have to answer yourself to reproduce the work.

None of these invalidate the paper. All of them are the difference between "I read the result" and "I could rebuild it." The last one is the only one with teeth, and even there the effect is bounded by Table 3's own spread — at most 1.4 points against the bare-label baseline.

Smaller whispers, collected

The counterfactual that would settle whisper 1

The AudioSet result is reported as a single before-and-after. What would turn it from an anecdote into a law? Four experiments, none of which needs new data.

The paper runs none of these, and one sentence explains why: the four-page format. But notice how much the single reported number still constrains the space. Whatever the mechanism, it is strong enough that a 14× increase in data lost 15 points — and no amount of "more data is better" intuition survives contact with that.

Reproduction checklist

If you set out to rebuild CLAP from the paper alone, here is what you know, what you must infer, and what you must guess.

Five stated, three inferable from internal consistency, two genuine guesses. That is a reasonably reproducible paper by 2022 standards — and the three inferable items are precisely the ones this lesson spent Chapters 3 and 4 deriving, because getting them wrong means getting a model that does not train at all.

Chapter 9: Legacy and Cheat Sheet

CLAP is a four-page ICASSP-style paper with a 128k-pair dataset and two off-the-shelf encoders. Its influence is out of all proportion to its size, for a reason worth naming: it did not introduce a technique, it relocated an interface. After CLAP, the natural way to ask an audio model a question is to type the question.

What CLAP made possible

What CLAP cannot do — the honest limits

CLAP against its neighbours, on one page

The row that carries the argument is "text supervision". Everything else in the CLAP column is borrowed, off the shelf, or smaller than the alternatives. Change only that row and the ESC50 number moves 12 points.

A short glossary, for the terms this lesson bolded

The cheat sheet — every equation, every symbol

The four equations, in order.

The numbers worth remembering

Where to go from here

Build it yourself — the weekend recipe

Everything in this lesson fits into one file. Here is the checklist, with the decision that matters at each step.

References worth reading next

1. Elizalde, B., Deshmukh, S., Al Ismail, M., Wang, H. "CLAP: Learning Audio Concepts from Natural Language Supervision," 2022 — arXiv:2206.04769. The paper this lesson is built on.
2. Radford, A. et al. "Learning Transferable Visual Models From Natural Language Supervision" (CLIP), 2021 — reference [6]. The recipe CLAP ports.
3. Kong, Q. et al. "PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition," 2020 — reference [23]. CNN14, the borrowed ear.
4. Devlin, J. et al. "BERT: Pre-training of Deep Bidirectional Transformers," 2019 — reference [24]. The donated geometry.
5. Zhai, X. et al. "LiT: Zero-Shot Transfer with Locked-image Text Tuning," 2021 — reference [27]. The vision-side echo of Chapter 7.
6. Wu, H.-H. et al. "Wav2CLIP," 2022 — reference [9]; Guzhov, A. et al. "AudioCLIP," 2022 — reference [10]. The label-supervised predecessors CLAP outperforms.
7. Drossos, K. et al. "Clotho," 2020 — [20]; Kim, C. D. et al. "AudioCaps," 2019 — [21]; Fonseca, E. et al. "FSD50K," 2022 — [19]; Martín-Morató, I. & Mesaros, A. "MACS," 2021 — [22]. The four corpora that became 128,010 pairs.
8. Turian, J. et al. "HEAR 2021: Holistic Evaluation of Audio Representations," 2022 — reference [12]. The source of many of Table 1's supervised benchmarks.