CLAP

A model learns to put a sound and a sentence about it at the same point. Then you can ask for any sound just by typing it. On ESC50, with no training on its 50 classes, CLAP names the right one 82.6% of the time.

Play a sound the fixed menu has never heard, then add its name as a caption and switch systems. Watch which one can answer. Then we build, piece by piece, how a sound and a sentence end up in the same place.

You need what a dot product is and what softmax does. We build the rest from zero.

Based on CLAP: Learning Audio Concepts from Natural Language Supervision by Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail and Huaming Wang (Microsoft, 2022). Every idea is retaught here in our own words and devices.

Name the sound

Ready
Tap to hear
Add a caption

Pick a sound. The fixed menu knows four classes: dog, rain, church bell and rooster.

Illustrative map: each sound and caption sits at an angle we chose, and the score is the cosine of the gap times 8. Real CLAP uses 1,024 dimensions and a learned scale near 143. The sounds are synthesized in your browser. The 82.6% is the paper's ESC50 zero-shot accuracy (Table 1); the paper reports 81% for humans.

Chapter 0

The Locked Menu

See why a trained sound classifier cannot say “I do not know”, and why adding one class means starting over

You have built a sound classifier. It learned 50 everyday sounds: dog bark, rain, chainsaw, church bell. It scores well on clips it has never heard, and it runs in 40 ms on a phone.

Then a product manager asks, “Can it flag smoke alarms?” Smoke alarm is not one of your 50 classes.

Here is the part that should worry you. The model cannot say “I do not know.” Play it an alarm and it still picks one of its 50 classes, and it sounds sure. The hero at the top of this page showed this: the fixed menu named a church bell.

Why softmax must pick

A classifier ends with one score for each class. These raw scores are called logits. A softmax turns them into probabilities: raise e to the power of each score, then divide by the total.

Take three classes and a clip of a smoke alarm. The logits are dog bark 1.2, rain 0.4 and church bell 0.9. These are toy numbers.

e1.2 = 3.320   e0.4 = 1.492   e0.9 = 2.460   sum = 7.272

p = (0.457, 0.205, 0.338) → “dog bark, 45.7% sure”

The clip holds no dog. The model could not say so. The three probabilities must add up to 1, so the answer is always “one of my classes is right.” That belief is built into the arithmetic, and it can be false.

The last layer is a fixed matrix

Look at the shapes. An audio encoder turns the clip into one vector h with D numbers, for example D = 2048. Then the classifier does one more step.

h
The clip as one vector of D = 2048 numbers. Everything the encoder learned about the sound lives here.
W
A matrix with C rows and D columns. Training learns one row for each class. C is fixed when training starts.
b
A bias: one extra number for each class.
z
The C logits. Row c of W takes a dot product with h to give the score for class c.
p
The C probabilities. They add up to 1 over the classes the matrix knows.

The encoder is rich. It knows texture, rhythm and echo. But the only way to ask it a question is through W, and W has exactly C rows. There is no row for smoke alarm. To add one you need labelled alarm recordings, a training run and a new release.

Try the usual fix: a threshold

The first idea is to ask for a minimum confidence. If the top probability is below a threshold θ, answer “unknown.” The device below uses the toy logits for four clips. Play each one, then move the threshold.

Find a threshold that works

Each bar is the fixed head’s top probability for one clip. The line is your threshold. Bars past the line are accepted as a known class. Tap a button to hear the clip.

0.50

Illustrative: the logits are toy numbers on a three-class head (dog bark, rain, church bell). Dog (2.4, 0.4, 0.9), quiet dog (1.9, 0.4, 0.9), wolf (2.1, 0.3, 1.0), smoke alarm (1.2, 0.4, 0.9). The probabilities are computed live. The sounds are synthesized in your browser.

At θ = 0.50 the smoke alarm is rejected. Good. But the wolf howl, which is also not in the taxonomy, scores 66.7% and gets through as a dog. To reject the wolf you must raise θ above 0.667. Then the quiet dog bark, a real dog at 62.9%, is rejected too.

No threshold gets all four right. The wolf and the quiet dog sit at the same place in the output, so one number cannot separate them. More classes do not help either. AudioSet has 527, and the 528th sound still has no row. A hand-made hierarchy has the same limit.

What the paper does instead

The paper opens with this complaint. Mainstream audio models learn from “one class label to many recordings, focusing on one task”, and that restricted supervision limits their flexibility. Self-supervised learning removes the labels from pretraining. But it still adds a fixed head at the end, and it learns nothing from language.

CLAP takes a middle path. It trains on audio paired with free sentences, and it removes the head. At test time you type the classes you care about. The model embeds the sound, embeds each sentence, and picks the closest sentence.

This is zero-shot classification: the model does the task with no training stage for that task. No fine-tuning, and not one labelled example. A new category is a string, not a project. The paper lists three contributions:

  1. The model. Two encoders trained on 128k audio and text pairs with contrastive learning.
  2. The ability. Zero-shot prediction, with no predefined set of categories.
  3. The evidence. Zero-shot results on 16 tasks across 8 domains, and supervised state of the art on 5.

The road

The hero’s controls map to chapters. The map and the bars are the shared space (Chapters 1 and 5). The captions are the text tower (Chapters 2 and 4). The sound is the audio tower (Chapters 3 and 4).

  1. Pull together, push apart: contrastive learning from zero.
  2. Captions as supervision: what a sentence teaches that a label cannot, and the 128,010 pairs.
  3. From pressure wave to tensor: how a clip becomes a 64 by 690 grid.
  4. Two towers, one space: every shape and parameter count.
  5. The symmetric loss: a 3 by 3 batch worked by hand, and the temperature puzzle.
  6. Zero-shot with prompts: classification becomes retrieval.
  7. The results, read honestly: 16 tasks, the highs and the lows.
  8. Ablations: which tower matters, and why more data hurt.
  9. Limits: order, speech, and the 8 kHz ceiling.
  10. What came after: LAION-CLAP, CLAP 2023, audio data pipelines and the CLAP score.
  11. Field guide: numbers, code and the whole model on one page.
A fixed-head classifier trained on 50 classes hears a smoke alarm. Why can it not report “none of the above”?

Chapter 1

Pull Together, Push Apart

Build contrastive learning from zero, and see why pulling matched pairs together is not enough

Picture a table with 1,000 recordings. Next to each one lies a card with a sentence that describes it. Your job is to draw a map. On the map, each recording must sit at the same spot as its own sentence.

A map like this is built from embeddings. An embedding is a list of numbers that places a thing on a map. CLAP uses 1,024 numbers for each clip and each sentence. Two things that mean the same should land close together.

How to measure “close”

The paper compares embeddings with a dot product. A dot product multiplies two vectors position by position and adds the results. It mixes two facts: the angle between the vectors, and their lengths.

a · b
The dot product of two embeddings.
∥a∥ ∥b∥
The two lengths multiplied. Length says nothing about meaning, so we want to remove it.
cos θ
The cosine similarity. It is 1 when the vectors point the same way, 0 when they are unrelated, and −1 when they point opposite ways.

Scale every vector to length 1. This is L2 normalisation. Then the dot product is exactly the cosine, and every point sits on a sphere. In the toy below the sphere is a circle, and a point is just an angle.

The first try, and how it fails

The simple plan is to make every clip’s embedding close to its own sentence’s embedding. Push the sum of the match cosines as high as it goes.

Run that plan and you get a perfect score and a useless model. Every clip and every sentence moves to the same point. All cosines are 1.0, matched or not. This is representational collapse. The objective only ever says “closer.” It never says “not that one.”

Train a tiny map

Three clips (teal dots) and their three captions (warm squares) sit on a circle. Dashed lines join true pairs. Press the train button and watch the pairs line up. Then drag the collapse slider and watch which score likes it.

0%

Illustrative: six free points on a circle, trained by gradient descent for 150 steps on the symmetric loss of Chapter 5 at scale 5. The collapse slider slides every point toward one spot, without training. The pull-only score is the mean of 1 minus the match cosine. Real CLAP does this in 1,024 dimensions with shared encoders.

Press train. The loss pulls each caption toward its own clip and pushes it away from the other clips. The three pairs spread around the circle, each caption next to its own clip.

Now drag the collapse slider. As the points crowd together, the pull-only score gets better and better. It reaches a perfect 0 when all six points sit on one spot. The pull-and-push loss gets worse, up to the 1.099 of pure guessing, and the captions lose their clips. A pull-only objective cannot tell these two maps apart. It prefers the useless one.

Negatives for free

The fix is to add a comparison. A match only counts if it beats the alternatives. So each sentence must be closer to its own clip than to the other clips.

Where do the alternatives come from? From the same batch, the group of N pairs the model reads at once. Pair i’s caption is the right match for clip i. It is almost surely a wrong match for every other clip in the batch.

So a batch of N pairs gives N right pairs and N² − N wrong pairs. Nobody labelled a single wrong pair. The paper says it in one sentence: the similarity matrix has “N correct pairs in the diagonal and N²−N incorrect pairs in the off-diagonal.”

Right pairs are positives and wrong pairs are negatives. Learning from both is contrastive learning. CLIP used it for images and text. CLAP carries the recipe over to sound.

Two numbers show why batch size matters. Each caption now has N − 1 competitors, so a larger batch asks a harder question. And a model that guesses at random scores a loss of ln N, so the bar to beat grows with N. For N = 768, the largest batch in the paper, that bar is ln 768 = 6.644.

Here is the whole method in one sentence. Encode the clip, encode the caption, and put both in one space. Then train so that each clip’s own caption is the nearest one in the batch. The next chapters fill in each piece.

You train with only “make each clip close to its own caption.” What does the model do?

Chapter 2

Captions Are Supervision

Compare a label with a caption for the same recording, and meet the 128,010 pairs CLAP learns from

Chapter 0 promised to replace the label with a sentence. That sounds like a small change to the annotation format. It is not. It changes what the loss can teach, and it changes what you can ask the model later.

Here is a real caption from the paper’s training data (AudioCaps): “Several sirens are wailing and a horn is honked twice.” Press play to hear a clip like it, then press Label to crush the caption into one class name.

Hear it, then crush it

The bar shows the clip over 2.5 seconds: sirens all the way through, and two horn honks. Under it, the supervision text. Switch between a label and a caption, and count what survives.

The clip is synthesized here to match the caption: two detuned siren sweeps and two horn honks. It is not the paper’s audio. The caption is the paper’s (Table 7, AudioCaps). The fact count is ours: sources, counts and manner that a listener would check.

The label “siren” keeps one fact. The caption keeps at least six: two sources, several sirens, wailing, honked, and twice. A one-hot label also hides the horn from the loss. The model is never asked to explain it.

How much more information is that?

A label picked from C classes carries at most log2 C bits. For the 50 classes of ESC50 that is 5.64 bits. For the 200 classes of FSD50K it is 7.64 bits. A 15-word caption carries on the order of 100 bits, a rough estimate of ours.

The size matters less than the shape. Take three one-hot targets: dog (1, 0, 0), wolf (0, 1, 0) and trombone (0, 0, 1). The squared distance between any two is 2. The loss cannot tell that a wolf is closer to a dog than to a trombone.

A text encoder already knows more. “A dog barking in a yard” and “a wolf howling in the distance” become two nearby vectors. So the text encoder lends its geometry to the audio encoder. The contrastive loss is the plumbing that moves the loan.

Where the 128,010 pairs came from

The authors built no new dataset. They joined four public ones. Two of them give several captions to each clip, so the same audio appears many times with different words.

SourcePairsUnique clipsHow the text was made
FSD50K36,79636,796Freesound title plus description. The class label was ignored.
ClothoV229,6465,929Five human captions for each clip, so five pairs for each clip.
AudioCaps44,29244,292One crowd-sourced caption for each AudioSet clip.
MACS17,2763,930Several annotators for each clip, so several pairs.
Total128,01090,947Table 4 of the paper.

Two details in the table matter later. First, 128,010 pairs cover only 90,947 clips, about 41% more pairs than clips. Five captions for one clip is free regularisation: the audio cannot latch onto one phrasing, so the shared meaning wins. Second, FSD50K ships clean class labels, and the authors threw them away. They bet that messy human prose beats a clean taxonomy.

python# how the 128,010 pairs are built (pseudo-code from the paper’s Appendix A)
pairs = []
for clip in fsd50k_train_and_val:                 # 36,796 clips
    pairs.append((clip.audio, clip.title + ". " + clip.description))
for clip in clotho_v2:                           # 5,929 clips, 5 captions each
    for caption in clip.captions:
        pairs.append((clip.audio, caption))             # 29,646 pairs
# AudioCaps: 44,292 pairs. MACS: 3,930 clips, 17,276 pairs.
len(pairs)                                          # 128,010
len({a for a, _ in pairs})                         # 90,947 unique clips

The scale is the surprise. CLIP, the vision model CLAP copies, used 400 million image and text pairs. CLAP used 128,010, about 0.032% as many. The paper’s conclusion writes “at least 0.001% smaller”, which is a loose figure. The real ratio is one part in about 3,100.

Why not just reuse CLIP’s text space?

Two earlier models did borrow CLIP: Wav2CLIP and AudioCLIP. They trained audio against AudioSet class labels. The text was a taxonomy wearing a sentence’s clothes. The paper says it was “yet to be explored” how natural language helps with new classes.

The zero-shot numbers show the gap. On ESC50, AudioCLIP gets 69.40% and CLAP gets 82.6%. On UrbanSound8K it is 65.31% against 73.24%. On the multi-label FSD50K, Wav2CLIP gets 3.02% mAP and CLAP gets 30.24%. The architecture family is the same. The text is the difference.

Why does the paper’s text supervision give a better zero-shot classifier than AudioSet class labels written as words?

Chapter 3

From Pressure Wave to Tensor

Follow a recording from air pressure to the 64 by 690 grid that the audio encoder reads, and hear what each choice removes

Chapter 2 gave us pairs: a clip and a sentence. Text encoders eat sentences. A clip is harder. A microphone hands over a list of air-pressure readings, and a convolutional network wants a two-dimensional array with local structure.

The paper gives the recipe in three lines. It uses a log Mel spectrogram at a 44.1 kHz sampling rate. The hop is 320, the window is 1024, and there are 64 Mel bins from 50 to 8000 Hz. Each clip is cut to a random 5 second segment, or padded if shorter. The paper writes the hop and window in “secs.” That is a typo. They are samples.

Step 1: why not use the raw samples?

Five seconds at 44,100 samples per second is 220,500 numbers. A network could read them directly. A convolutional encoder reads them badly, for three reasons.

Pitch hides at large scales. A 100 Hz hum repeats every 441 samples, which a small kernel cannot see. Phase is mostly noise: shift the wave by one sample and every number changes, but the sound does not. And 220,500 numbers a clip is a lot to train on for 40 epochs.

Step 2: cut into frames, then count frequencies

Cut time into overlapping frames. The window is how many samples go into one frame. The hop is how far you slide before the next frame. CLAP uses a window of 1024 samples and a hop of 320.

QuantitySamplesTime at 44.1 kHzWhy this value
Window102423.22 msA 100 Hz tone fits 2.3 cycles. The sound barely changes inside it.
Hop3207.26 msPlaces a click or a bark onset to about 7 ms.
Overlap70415.96 ms68.75% of each frame is shared, so no event falls between frames.
Frames in 5 s6901 + ⌊220,500 / 320⌋This is T, the time axis of the tensor.

Each frame goes through a Fourier transform, which splits it into frequencies. A real signal needs only 1024 / 2 + 1 = 513 bins. Each bin is 44,100 / 1024 = 43.07 Hz wide. We keep the magnitudes and drop the phase.

Step 3: Mel, a ruler built for ears

Those 513 bins are evenly spaced. But hearing is not. The step from 43 Hz to 86 Hz is a full octave. The step from 17,226 Hz to 17,269 Hz is inaudible. Even bins waste their numbers on distinctions nobody hears.

The Mel scale warps hertz so that equal steps sound like equal pitch steps. A Mel filter is a triangle that adds up the FFT bins under it.

f
A frequency in hertz.
m(f)
The same pitch on the Mel scale. Equal steps in m sound like equal steps in pitch.
2595
A constant chosen so the Mel scale matches listening tests.

Run CLAP’s numbers. The band 50 Hz to 8000 Hz spans 77.75 to 2840.02 mel. Sixty-four filters need 66 edges, so 65 steps of 42.50 mel each. Filter 0 runs from 50.0 to 108.7 Hz, which is 58.7 Hz wide. Filter 63 runs from 7368.0 to 8000.0 Hz, which is 632.0 Hz wide. That is 10.8 times wider. The Mel warp keeps fine detail where ears have it.

Step 4: the logarithm, and the tensor

A loud bark can carry a million times the energy of the room around it. A logarithm squeezes that range. It also turns volume into a plain offset: a sound four times more energetic adds the same constant (log 4) to every value. The network can ignore it.

The result is a grid, written Xa with F rows and T columns.

Xa ∈ ℝF×T = ℝ64×690 (F = 64 Mel bins, T = 690 frames)

That is 44,160 numbers, down from 220,500. Every one of them is tied to what ears hear. A batch of N clips is a tensor of shape (N, 64, 690), or (N, 1, 64, 690) once you add the channel axis that a 2-D convolution expects.

Now build it yourself. The device below makes a real log-Mel spectrogram of a sound synthesized in your browser, using a real FFT. Change the window, the hop and the Mel bins and watch the shape change.

Build the spectrogram

Top: the wave. Below: the log-Mel grid CLAP’s audio encoder would read. Tap a sound to hear it and see it. The paper’s settings are marked. Try a window of 256, then 4096, and look at the siren.

Window
Hop
64

Real computation, toy audio: a Hann window, an FFT, triangular Mel filters from 50 Hz to 8000 Hz, then 10 log10 of the power. The clip is 2.5 s, not the paper’s 5 s, so it has about half as many frames. The sounds are synthesized, not recordings. The speech-like clip is vowels plus a hiss, made with a simple vowel model.

Look at the read-out when the window is short. A narrow window gives crisp onsets but a coarse pitch: the siren becomes a smear. A long window gives a clear pitch but blurs a bark across frames. The product of window time and bin width stays near 1, so you cannot have both. This is the time-frequency trade-off. CLAP’s 23 ms is the usual compromise.

The ceiling that nobody flags

CLAP samples at 44.1 kHz, so frequencies up to 22.05 kHz exist. But the Mel filters stop at 8 kHz. That throws away 64% of the band. For barks, engines, rain and sirens this is fine, since their identity lives low. But the hiss of an “s” lives from 4 to 10 kHz. Switch on Keep up to 22 kHz with the speech-like clip and see the energy that CLAP never sees.

pythonimport torch, torchaudio.transforms as T
front_end = torch.nn.Sequential(
    T.MelSpectrogram(sample_rate=44100, n_fft=1024, win_length=1024,
                     hop_length=320, f_min=50, f_max=8000, n_mels=64),
    T.AmplitudeToDB())
X_a = front_end(waveform)        # waveform: (B, 220500)  ->  X_a: (B, 64, 690)

A learned front end could in principle find better filters. But with only 128,010 pairs, the strong prior wins. The Mel filters cost zero parameters and need zero data.

One more cost the paper does not price. Short clips are padded to 5 seconds. A 1 second Speech Commands clip is 80% silence, and the encoder averages over all 690 frames. The real evidence is diluted five times before the classifier sees it. This is our arithmetic, not a claim of the paper.

Random cropping is also a source of label noise. ClothoV2 clips run 15 to 30 seconds, and each caption describes the whole clip. A random 5 second window may hold the bow and none of the violins, so the caption is partly wrong for that crop. The crop changes every epoch, so over 40 epochs the noise averages toward a weak signal. Each single step is still noisier. This is our reading, not a claim of the paper.

CLAP samples at 44.1 kHz but its Mel filters stop at 8000 Hz. What is the direct consequence?

Chapter 4

Two Towers, One Space

Trace every tensor shape and parameter count from the grid and the caption to the shared 1,024-number space

A clip is now a grid of 64 by 690 numbers, and a caption is a string. Each must become one short list of numbers, so that the two can be compared. This chapter follows that step and writes down every shape and parameter count, because a diagram of boxes is not understanding.

fa
The audio encoder. It is CNN14, a convolutional network with 80.8 million parameters.
ft
The text encoder. It is BERT base uncased, with 110 million parameters.
X̂a
The audio representation, N by V with V = 2048. N is the batch size.
X̂t
The text representation, N by U with U = 768.

The audio tower is a borrowed ear

CNN14 comes from the PANNs project. It was pretrained on 2 million AudioSet clips with class labels. So CLAP does not learn to hear from 128k pairs. The ear already exists. The 128k pairs build a bridge from that ear to language.

That is why 128k pairs can be enough where CLIP needed 400 million. CLIP’s image tower started from nothing. The paper picked CNN14 to compare fairly with earlier work, since PANNs, Wav2CLIP and AudioCLIP all use it.

The shapes below follow the PANNs design, not this paper. Each of the first five blocks halves both axes. Then the model averages over frequency and pools over time.

StageShape for one clipWhat it holds
Log-Mel input(1, 64, 690)The grid of Chapter 3, with a channel axis.
Block 1(64, 32, 345)Small time and frequency edges.
Block 3(256, 8, 86)Textures about 60 ms long.
Block 5(1024, 2, 21)Event-sized patterns.
Block 6, pool, FC(2048,)One vector for the whole clip. This is X̂a.

The text tower hands over a [CLS] vector

BERT base uncased, from HuggingFace, reads the caption. The paper limits the text to 100 characters, not tokens. That is about 20 words. Two of the twelve captions the paper itself samples are longer: the bow caption has 106 characters and the leaves caption has 101. The tail is cut, and the tail is often where the second sound lives.

BERT puts a special [CLS] token at the front of every input. Self-attention lets it read every other token, so its final vector summarises the sentence. Its size is 768.

Both towers end in one vector. A clip with 690 frames and a caption with 20 words each collapse to a single point. The loss in Chapter 5 compares one audio vector with one text vector, so no structure can survive past this point.

Why two separate towers?

The other way is to let the text tokens and audio frames attend to each other inside one transformer. That cross-attention model can say more. CLAP does not use it, and the reason is cost.

Dual encoder (CLAP)Cross-attention
ESC50: 2,000 clips, 50 classes2,000 audio passes + 50 text passes, once100,000 joint passes
Searchable indexYes, embeddings do not depend on the queryNo, a score exists only for one pair
Training scores per batchN² scores from 2N passesN scores from N passes

The last row decides it. The free negatives of Chapter 1 need the towers to be independent. With joint scoring, a batch of 768 would need 589,824 passes instead of 1,536. The price is grounding: no word ever looks at a frame, so the model cannot say where in the clip the dog is.

Two projections join the towers

La
A linear map from 2048 to 1024 numbers. It holds 2,097,152 weights.
Lt
A linear map from 768 to 1024 numbers. It holds 786,432 weights.
Ea
The audio embedding in the joint space, N by d with d = 1024.
Et
The text embedding in the same space, also N by d.

Together the two maps hold 2,883,584 weights. That is 1.49% of the model’s 193.7 million parameters. These few weights do the joining.

Why linear? A linear map can rotate, scale and shear. It cannot fold or tear. Neighbours stay neighbours. That suits a job of aligning two existing geometries. A deeper network could fit the training pairs more tightly and still bend the space in ways that fail on class names it has never seen.

Note the asymmetry. Audio comes down from 2048 to 1024, so La must throw information away. Text goes up from 768 to 1024, so Lt cannot add any. The text embeddings live on a 768-dimensional sheet inside the 1024-dimensional space.

Trace the shapes

Choose a stage to light it up. Switch a tower to frozen to see how many parameters would still train.

Parameter counts are the paper's (CNN14 80.8M, BERT base 110M) plus the two projection matrices counted as weights only, without biases. Block shapes follow the PANNs design and are not from this paper.

Normalise, and why

The paper writes a bare dot product in the next equation, but every implementation first scales each embedding to length 1. Here is the reason. A dot product is length times length times cosine. The lengths mean nothing, so the model is free to inflate them.

The loss wants the diagonal large next to its row. The cheapest way is not better meaning. It is a longer vector for the easy examples. Length would start to encode confidence, not meaning. Normalising removes the shortcut. Every entry becomes a cosine between −1 and 1, and only a rotation can raise the diagonal.

pythonclass CLAP(nn.Module):
    def __init__(self, d=1024):
        super().__init__()
        self.audio_enc = cnn14_pretrained_on_audioset()        # 80.8M, outputs (N, 2048)
        self.text_enc  = AutoModel.from_pretrained("bert-base-uncased")  # 110M
        self.tok       = AutoTokenizer.from_pretrained("bert-base-uncased")
        self.La = nn.Linear(2048, d)                          # 2,097,152 weights
        self.Lt = nn.Linear(768,  d)                          #   786,432 weights
        self.logit_scale = nn.Parameter(torch.tensor(np.log(1/0.007)))
    def encode_audio(self, x):                                # x: (N, 64, 690)
        e = self.La(self.audio_enc(x))                          # (N, 1024)
        return e / e.norm(dim=-1, keepdim=True)                # unit length
    def encode_text(self, captions):                         # list of str
        b = self.tok([c[:100] for c in captions], padding=True, return_tensors="pt")
        e = self.Lt(self.text_enc(**b).last_hidden_state[:, 0])  # [CLS] -> (N, 1024)
        return e / e.norm(dim=-1, keepdim=True)
CLAP pools 690 audio frames into one 2048-number vector before comparing with text. What does that make hard?

Chapter 5

The Symmetric Loss

Build the similarity matrix and the loss on a 3 by 3 batch by hand, then solve the temperature puzzle

Take a batch of clips and their captions. Score every caption against every clip, and you get a square table of numbers. One formula turns that table into a single number that says how wrong the model is. If you can rebuild the 3 by 3 example below on paper, you can implement CLAP.

Et
Shape (N, d). Row i is the unit-length embedding of caption i.
EaT
Shape (d, N). Column j is the embedding of clip j.
τ
A learned scale on the cosines. In the code it multiplies; the next section settles the exact reading.
C
The N by N similarity matrix. Entry (i, j) scores caption i against clip j. The diagonal holds the N true pairs.
ℓtext
Softmax along each row, then −log of the diagonal entry. Row i asks: which clip does caption i describe?
ℓaudio
The same along each column. Column j asks: which caption describes clip j?
L
The symmetric loss. The labels are just the indices 0, 1, …, N − 1, since the diagonal is the answer.

Three moves make this loss. A softmax along a row turns similarities into a choice. The diagonal is the answer, so no one needs to label anything. And you do it both ways, because the two questions differ. The paper prints the log with the wrong sign, so read it as a negative log-likelihood.

A 3 by 3 batch, by hand

Shrink d from 1024 to 2 and N from 768 to 3. Nothing else changes. The two axes are impulsive energy (horizontal) and tonal energy (vertical). The embeddings are already unit length.

iClipEaCaptionEt
1a dog barking(0.966, 0.259)“a dog barks”(0.996, 0.087)
2knocking on a door(0.766, 0.643)“knocking on a wooden door”(0.574, 0.819)
3a violin note(−0.174, 0.985)“a violin plays a long note”(−0.342, 0.940)

The nine dot products give the raw cosines. Three of them in full:

Ĉ11 = 0.996 × 0.966 + 0.087 × 0.259 = 0.985

Ĉ12 = 0.996 × 0.766 + 0.087 × 0.643 = 0.819

Ĉ13 = 0.996 × (−0.174) + 0.087 × 0.985 = −0.088

Cell (1, 2) is the hard negative. The caption “a dog barks” sits at 0.819 from the knock, nearly as close as to the dog. Both are impulsive. Multiply by a scale of 4 and take softmaxes. Row 1 gives probabilities (0.654, 0.337, 0.009) and a loss term of −ln 0.654 = 0.424. Doing all rows and all columns gives:

ℓtext = 0.367   ℓaudio = 0.382   L = 0.375   (guessing: ln 3 = 1.099)

Use the worksheet to see every number. Tap any cell to read its row and column terms.

Work the loss by hand

Rows are captions, columns are clips. Pick how to normalise, move the scale, and tap a cell. The bars below show the loss against guessing.

4.0
Try

The six vectors and the scale of 4 are the hand-worked example, not model outputs. The paper’s real scale is 142.86, which saturates this toy; the slider goes to 20 and the last chip jumps to 142.86.

Four things the hand work shows. The same cell gives 0.554 in row 2 and 0.611 in column 2: same logit, different denominator. That is why both directions count. The hard negative at cell (1, 2) is penalised twice, once in row 1 and once in column 2. Its gradient is weighted by its softmax probability, so the knock takes a third of the push while the violin, at 0.009, takes almost none. And the easy violin pair adds only 0.086, so it goes quiet. This is why larger batches help: they supply new hard negatives.

The temperature puzzle

Section 3.2 says the temperature τ is learnable and starts at 0.007, and that the scaled logits are clipped at 100. Read the equation literally. Multiply cosines by 0.007, and every logit lies in [−0.007, 0.007]. A softmax over 768 of those is flat. The loss sits at ln 768 = 6.644 for ever, and nothing could ever reach a clip at 100.

The numbers close one way only. The multiplier is 1/0.007 = 142.86, as in CLIP. Then cosines map to [−142.86, 142.86], and a clip at 100 does fire. This is our reading, and the public implementations store a log-scale parameter set to log(1/0.007).

Why 142.86? At the start the towers are unaligned, so two embeddings are random unit vectors in 1024 dimensions. Their cosine has a standard deviation of 1/√1024 = 0.03125. Times 142.86 gives logits that spread by about 4.5. That is sharp enough to learn from and not saturated. The scale is tuned to the dimension.

The clip guards a different problem. Once the diagonal wins, the loss only falls as the scale grows. Left alone the scale runs away and exp() overflows. The clip is a numerical guardrail, not a modelling choice. At test time the scale does not matter: dividing every logit by the same positive number cannot change the argmax.

pythonimport torch, torch.nn.functional as F
logits  = model.logit_scale.exp().clamp(max=100) * E_t @ E_a.T    # (N, N), exp(log(1/0.007)) = 142.86
targets = torch.arange(len(logits), device=logits.device)   # [0, 1, ..., N-1]: the diagonal
loss = 0.5 * (F.cross_entropy(logits,   targets) +          # rows: text -> audio
              F.cross_entropy(logits.T, targets))            # columns: audio -> text

Three lines. The transpose is the whole content of “symmetric.” The arange is the whole content of “no labels needed.”

What you see in the logsLikely cause
Loss pinned at ln N from step oneThe scale is multiplied by 0.007 instead of 1/0.007.
Loss near zero but zero-shot at chanceA shortcut: a batch sorted by source, with a source artefact.
The loss turns invalid after a few hundred stepsThe learned scale grew without bound. Clamp it.
One of ℓtext, ℓaudio stallsThe same clip or caption appears twice in a batch.
In the hand-worked batch, cell (caption 1, clip 2) has cosine 0.819, close to the true pair’s 0.985. Why does that one cell dominate the gradient?

Chapter 6

Zero-Shot Is Retrieval in Disguise

Classify a clip by typing class names, and see why the text encoder works as a classifier-weight generator

Training is over. You have two encoders and two projections, and no classifier. Someone hands you 2,000 five-second clips and asks you to sort them into 50 categories. None of the categories was in your training objective.

Section 2.2 of the paper answers in three steps. First, embed the N clips and the C class names with the trained encoders and projections. Second, take the cosine between each clip and every class name, so each clip gets C scores. Third, turn the scores into probabilities: a softmax for one-label tasks, a sigmoid for multi-label ones.

The same equation, a different source for W

Go back to the prison of Chapter 0. A classifier computes z = W h, where W has one learned row for each class. Now write CLAP’s zero-shot step the same way.

Etext
Shape (C, d). The stack of the C class-name embeddings, one row for each prompted string.
Ea
Shape (d,). The embedding of the one clip we are classifying.
z
The C logits, one for each class name you typed.

It is the same equation. A matrix of shape (C, d) times a vector of shape (d,) gives C scores. The whole paper is the difference in where the matrix comes from. In a classifier, W is learned. In CLAP, the rows are the output of a function applied to strings.

So the text encoder is a classifier-weight generator. It maps a string to a row. The rows stopped being parameters, so the menu disappeared. A function is defined on strings nobody trained on.

One clip, four classes, with numbers

Use the 2-D toy space of Chapter 5. The test clip is a dog bark with Ea = (0.966, 0.259). Four class names, each wrapped in a prompt and embedded, give the table. The scale is 4, for readable arithmetic.

ClassEtext rowCosineLogitSoftmaxSigmoid
dog(0.985, 0.174)0.9963.9850.5850.982
rooster(0.707, 0.707)0.8663.4640.3480.970
rain(0.174, 0.985)0.4231.6920.0590.845
violin(−0.342, 0.940)−0.087−0.3480.0080.414

The prediction is dog, at 58.5%. Look at the runner-up: rooster, at 34.8%. Nothing told the model that dogs and roosters are both animal sounds. The language geometry did. A one-hot classifier’s second place carries no such information.

The sigmoid column is for multi-label tasks such as FSD50K, where one clip holds several events. Each class gets an independent yes-or-no score, so the scores need not add up to 1. The paper scores these tasks with mAP, mean average precision. It judges the ranking and needs no threshold.

Now try it with sounds. Pick a clip, switch captions on and off, and move the scale. The map is a toy, as in the hero.

Classify by typing

Tap a sound to hear it. Switch captions on to give CLAP a menu you choose. Turn off the true caption and watch what the softmax does.

Sound
Captions
12.0

Illustrative: positions on the circle are our choice and the score is the cosine times the scale. The model is not run. With every caption off the true class, the softmax still names a winner: zero-shot CLAP inherits the abstention problem of Chapter 0.

Two experiments. First, turn off the true caption. The probabilities redistribute over the survivors, because softmax still sums to 1. CLAP fixed the vocabulary and not the calibration. Second, add an unusual caption. It ranks sensibly although it exists only as a string.

The prompt problem

There is a mismatch hiding here. For 40 epochs the text encoder read full sentences. At test time you hand it the fragment “dog”. The paper says that using single words “affects how Zero-Shot learning transfers.” So the class name is wrapped in a prompt template, a sentence shaped like the training data. The default is “This is a sound of [class label].”

Rank the five prompts

The bars are Table 3 of the paper, ESC50 zero-shot accuracy. The dashed line is the bare class name. Tap a prompt to see it against the bare label.

All five accuracies are Table 3 of the paper. The zoomed axis starts at 76%; switch it off to see the bars from 0. The 81% band is the reported human accuracy on ESC50.

Read Table 3 against the story it should support. The naive story says any sentence beats a bare fragment. But two of the four templates lose to the bare label, and “i can hear” loses by 2.6 points. The prompt is a query into the caption distribution. “This is a sound of X” is written like a caption of a recording. “I can hear X” is a first-person report. Our guess, which the paper does not test, is that this genre barely appears in the training captions.

The paper says prompts give “a 5% acc increase.” The table spans 4.0 points from worst to best, and only 1.4 points against the bare label. Real and free, but small. Three domains also needed another prompt: “this person is feeling [class label]” for emotion, the bare keyword for Speech Commands, and “[number] persons speaking” for counting. In the next chapter, emotion and keywords turn out to be CLAP’s weakest tasks.

The same machine, run backwards

Nothing says the class list must be short. Loop over the other axis and the same model searches an archive. One sentence against N clips ranks the clips that sound like it. One clip against a pool of captions finds the nearest description. Both are the same two lines of code with the loops swapped.

python# ONCE, at deployment. No gradients anywhere.
classes = ["dog", "rooster", "rain", "violin", "chainsaw"]            # type anything
prompts = [f"this is a sound of {c}" for c in classes]
with torch.no_grad():
    E_text = model.encode_text(prompts)              # (C, 1024), unit length. CACHE THIS.
    E_a    = model.encode_audio(log_mel(clip))       # (1, 1024), unit length
logits = model.logit_scale.exp() * (E_text @ E_a.T)  # (C, 1)
probs  = logits.softmax(dim=0)                       # one label
scores = torch.sigmoid(logits)                       # many labels (FSD50K, AudioSet)
# add a 6th class at 3 a.m.: one BERT pass, no data, no redeploy
E_text = torch.cat([E_text, model.encode_text(["this is a sound of a smoke alarm"])])

The costs are small. Embedding the 50 ESC50 prompts is 50 passes of BERT, once. Storing them takes 50 × 1024 floats, about 200 KB. Classifying a clip is one CNN14 pass and one matrix-vector product. Under argmax the scale is irrelevant, since dividing every logit by one positive number cannot change the winner.

In CLAP’s zero-shot classification, what plays the role that the learned weight matrix W plays in an ordinary classifier?

Chapter 7

The Results, Read Honestly

Read all 16 tasks, the highs and the lows, against chance and against what captions talk about

Now the question that matters. A model trained on 128,010 captioned clips, with no classifier and no task training, faces 16 benchmarks it has never seen. The answer is not uniform, and the pattern in the failures tells you what the model learned.

How to read Table 1

The table has five kinds of row. Mixing them up is the fastest way to misread the paper.

RowWhat it isTrained on the task?
RandomChance, about 1 divided by the number of classes.No
Benchmark (ZS)Best earlier zero-shot result (Wav2CLIP, AudioCLIP).No
CLAP (ZS)CLAP with prompted class names.No
Benchmark (Best)Best supervised result in the literature.Yes
CLAP (Best)The better of CLAP’s two supervised setups.Yes

The metrics differ too. DCASE17 uses F1. FSD50K and AudioSet use mAP. The rest use accuracy. Higher is better in all of them. Chance also differs wildly: 0.02 for ESC50, 0.5 for a two-class task, under 0.0018 for AudioSet. Find the chance level before you react to a score.

Explore all 16 tasks

Each row is one task from Table 1. Switch what the bars show, and filter by domain. The tick marks chance.

Every score, chance level and benchmark is Table 1 of the paper. Headroom = (zero-shot − chance) / (1 − chance), our calculation. For FSD50K and AudioSet the paper gives chance only as an upper bound (under 0.005 and 0.0018), so those two headroom bars are approximate. Metric is accuracy except DCASE17 (F1), FSD50K and AudioSet (mAP).

The highs

ESC50: 82.6%. Fifty environmental classes, five-second clips. The best earlier zero-shot score was AudioCLIP’s 69.40%, so this is 13 points up (the paper writes 12) on a benchmark people had worked on. The paper also notes reported human accuracy of 81%.

GTZAN music versus speech: 100%. It beats the 99.2% supervised benchmark. It is a two-class task and the easiest cell, but it shows the point: captions say “a man speaks” and “music plays” all the time.

FSD50K: 30.24 mAP against Wav2CLIP’s 3.02, and UrbanSound8K: 73.24% against 65.31%. FSD50K is multi-label over 200 classes. The training set holds 36,796 FSD50K clips, but with their titles and descriptions and no class labels. The model rebuilt a 200-way taxonomy it was never shown.

The lows, stated plainly

The paper is honest: CLAP did better than random everywhere, and “good to slightly better than random on some music and speech-related tasks.”

TaskChanceCLAP (ZS)Lift
CREMA-D, emotion0.16670.1784+1.2 points
Mridangam tonic0.16670.1965+3.0 points
Speech Commands, keywords0.0830.1063+2.3 points
RAVDESS, emotion0.1250.1599+3.5 points

Raw scores mislead, because chance differs by task. Use the headroom measure in the device: the fraction of the gap from chance to perfect that zero-shot closed. It reads 100% for music versus speech, 82% for ESC50 and 70% for UrbanSound8K. At the bottom: Speech Commands at 2.5%, CREMA-D at 1.4%. That order is not random.

What the ordering says

Ask of each task: would a person writing a caption mention the thing being classified? For sound events, yes. For “a man speaks” versus “music plays”, constantly. For a genre label such as bebop, rarely. For an emotion, essentially never. For the spoken word “left”, never, since captions do not transcribe.

The paper says it in one sentence. Its training data “consist of audio captioning datasets, which mainly include the description of sound events, acoustic scenes, actions, and objects.” Speech is scarce, and captions “do not describe aspects of its content or context.” CLAP’s zero-shot skill tracks how often a task’s distinction appears in written descriptions of sound.

The Mridangam pair is a clean test. Strokes and tonics come from the same 0.81 second clips of the same instrument. A stroke is a percussive timbre that captions name, and zero-shot closes 27% of its gap. A tonic is a pitch that needs a trained vocabulary, and it closes 3.6%.

Here is the same idea, task by task. The headroom figures come from Table 1. The right-hand question is our judgment.

DomainWould a caption mention it?Headroom closed
Sound events (ESC50, UrbanSound8K, FSD50K)Yes. This is what audio captions are about.30% to 82%
Music versus speechYes. “A man speaks” and “music plays” are common phrases.100%
Vocal sounds (cough, sneeze, laugh)Yes. These are events with names.39%
Acoustic scene (TUT2017)Often. “In a park” appears in captions.25%
Instrument (Beijing Opera)Sometimes. “A drum” yes, a specific percussion name rarely.30%
Music genreRarely. Captions say “music plays”, not “bebop”.17%
Speaker countingOccasionally. “Two people talking” exists, a count from 0 to 10 does not.10%
Mridangam stroke and tonicAlmost never. It needs a trained musical vocabulary.27% and 3.6%
Emotion, keywordsEssentially never. Captions do not transcribe or name feelings.1% to 4%

Did the model already see the test audio?

Every zero-shot claim invites this question. Trace the corpora and the answer has two parts.

FSD50K is in-domain. The training set holds 36,796 FSD50K clips with their titles and descriptions, and the benchmark uses the same site’s sound events. We assume the paper tests on held-out clips. So the 30.24 mAP is zero-shot on the labels, but not on the recording style.

AudioSet is the stronger check. AudioCaps captions 44,292 AudioSet clips, and AudioSet is also a downstream task. If seeing the audio were enough, AudioSet would be CLAP’s best task. It is the weakest, at 0.058 mAP and about 5.6% headroom. Captions, not audio exposure, drive the score. The paper states no overlap for the other tasks, and ESC50 is one of them.

The supervised side, and one control

The paper also freezes CLAP and trains small layers on top (feature extraction), or unfreezes the audio encoder (fine-tuning). The better of the two, CLAP (Best), sets the state of the art on five datasets. They are music versus speech (100%), genres (91.30%), Mridangam stroke (97.94%) and tonic (95.34%), and Vocal Sound (97.95%). Elsewhere it trails the literature. The paper says by at most 7 points, but its own Table 1 shows wider gaps: 9.7 points on TUT2017 scenes (74.63% against 84.3%) and 17.5 points on RAVDESS emotion (64.36% against 81.82%).

Both supervised setups train for 30 epochs with Adam. The frozen-feature setup puts 1 or 3 fully connected layers on top at a learning rate of 0.001. The fine-tune setup unfreezes the audio encoder at 0.0001. The authors ran no grid search over hyperparameters because of compute limits.

Table 6 gives a clean control. On Speech Commands, frozen CLAP features score 0.3708. Fine-tuned CLAP scores 0.9683, the best in its row. So the audio tower can hear the words. What the frozen space lacks is a reason to encode them, since no training caption ever said what was spoken. The bottleneck is the supervision, not the capacity.

Two caveats

First, 82.6% against 81% compares different things. The paper cites the 81% without describing how it was measured, so the two numbers may not test the same thing. And CLAP’s own fine-tuned ESC50 score is 96.70%, with the literature at 97.15%. Zero-shot buys 82.6% for zero labelled examples. That is a price point, not an accuracy record.

Second, the paper reports no error bars. CLAP (Best) beats the benchmark on Mridangam tonic by 1.1 points, on a five-fold test over about 1,000 clips of 0.81 seconds. That margin is suggestive, not settled. The 27 point FSD50K gap needs no such care.

Third, four pages leave gaps. The paper never states the batch size of the headline model, yet Appendix C.1 says a batch of 768 did worse. It picks the best of five prompts on ESC50 and reports that same ESC50 score in Table 1, so 82.6% may be slightly optimistic (our caution). And the Table 1 “Best” row is a maximum over two supervised setups.

Frozen CLAP features score 0.3708 on Speech Commands. The fine-tuned model scores 0.9683. What does the pair show?

Chapter 8

Ablations: Which Part Matters

Freeze a tower, add noisy data, change the batch size, and see what the paper’s own experiments say

CLAP starts with two encoders that were already trained before it began, one for sound and one for text. During CLAP training you can update both, one or neither. That is four runs, and the paper ran all four (Table 2). Before you read it, commit to a guess. Which tower will matter more?

Most people say audio. The audio side is the one learning something new, and BERT already speaks English.

Freeze a tower

Pick one of the four regimes. Purple means gradients flow; grey means locked. The two bars are the paper’s results. The dashed line is the both-frozen row.

All eight numbers are Table 2 of the paper. The average is taken over the 16 downstream tasks, which mix accuracy, mAP and F1, so its level means little; the gaps between rows are what carry meaning. The projections and the temperature train in every row.

Both frozen is worst and both trainable is best, as expected. The middle two rows are the finding. Unfreezing only the audio encoder (80.8M parameters) adds +0.0009 to the average and 8.6 points on ESC50. Unfreezing only the text encoder (110M) adds +0.0300 and 20.8 points. On the 16-task average that is 33 times more.

The paper is candid: “Surprisingly, unfreezing the text encoder performed better than unfreezing the audio encoder.” Notice that on ESC50 alone the ratio is only 2.4. ESC50 is the task the AudioSet-trained CNN14 already fits, so a little audio change goes far there. On the other fifteen tasks, audio changes buy almost nothing and text changes help everywhere.

Why the text tower might matter more

The paper does not explain it. Here are three mechanisms that fit, flagged as our analysis, not the paper’s claims.

  1. Unequal distance to travel. CNN14 saw 2 million sound-event clips, so its features arrive almost on-task. BERT read Wikipedia and books, where nobody writes “an insect buzzing in the foreground as birds chirp.” The text tower has further to go.
  2. Bending the coarser geometry. Something has to bend to meet in the middle. Gradient descent picks the cheaper adaptation.
  3. Forgetting. 128,010 pairs is small for an encoder trained on 2 million clips. Unfreezing audio risks overwriting what it knows. The text tower has less to lose.

The paper links this to LiT, which found that locking the image tower and tuning the text tower works best for CLIP. The conclusion is useful: “it is possible to use an audio encoder of choice and turn it into a Zero-Shot classifier.” Any strong audio encoder plus a projection plus a moving text tower gives an open-vocabulary model. Later systems follow this recipe.

Look at the first row too. With both encoders frozen, only the 2.88M projection weights train. They still reach 55.55% on ESC50 where chance is 2%. A rotation between two pretrained geometries is worth that much.

pythondef set_regime(model, train_audio, train_text):
    for p in model.audio_enc.parameters(): p.requires_grad = train_audio
    for p in model.text_enc.parameters():  p.requires_grad = train_text
    # the projections and the temperature train in ALL four rows
    for p in [*model.La.parameters(), *model.Lt.parameters(), model.logit_scale]: p.requires_grad = True
model.audio_enc.eval()   # also freezes BatchNorm statistics; forgetting this silently trains the tower

Precomputing helps in practice (our suggestion, not the paper’s). With the audio tower frozen, run it once over the corpus and save the 2048-number vectors. Training then reads vectors from disk instead of running CNN14 every epoch. Table 2 says the price is 0.0156 of the average, the gap between rows 3 and 4.

Table 2 left some cheap experiments undone. A nonlinear projection with both encoders frozen would show whether the 55.55% is limited by the linear map or by the encoders. A randomly initialised audio tower would show how much of CLAP is AudioSet pretraining. Freezing audio for 20 epochs and then releasing it would test the forgetting guess. A four page paper reports the runs that fit.

More data hurt

Appendix C.2 is the most useful paragraph in the paper. AudioSet has 2 million clips but no captions. The authors built text from each video’s title and class labels, and added about 1.7 million pairs to the 128k. The result:

Task128,010 curated pairsPlus 1.7M AudioSet pairsChange
ESC5082.60%67.15%−15.45
UrbanSound8K73.24%70.93%−2.31
Speech Commands V210%15%+5

Adding 13 times as many pairs cut the headline score by 15 points. The paper’s reason: YouTube titles and descriptions “often do not describe the acoustic content of the video segment” but the video as a whole. A ten-second clip of a dog barking sits under the title “Ep. 42, Our Trip to Portugal (Vlog).”

Why is this worse for contrastive learning? In supervised learning a wrong label spoils one example. Here the diagonal is the label, so one wrong pair spoils its whole row and column. It pulls the bark toward the vlog title, pushes the title away from clips it may truly describe, and pushes the bark away from captions that do describe it. That is 2N − 1 relationships in one pair.

After the addition, 93% of the pairs are noisy and the curated ones are outnumbered 13 to 1. The one gain, Speech Commands, makes sense too. AudioSet holds a lot of speech, and even bad captions taught a model that had learned almost nothing about speech.

Dilute the data

Drag to add noisy AudioSet-style pairs to the 128,010 curated ones. Only the two ends are measured. The dashed line between them is a straight-line guess.

0

Measured (Appendix C.2): 82.6% at 0 noisy pairs, 67.15% at about 1.7M. Everything between is a straight-line interpolation of ours, not data. The real curve could bend either way.

The paper says finding good public pairs is hard, so “relying on large-scale noisy pairs is the only scalable approach.” Read that beside the 15-point drop. Noise is the only way to scale, and noise made it worse. The next generation answered with filters and generated captions. The bottleneck moved from collecting audio to writing about it.

The batch-size anomaly

Appendix C.1 varies the batch from 32 to 768 on 4 to 24 GPUs. Larger helped, until 768, where the score dropped. The authors call it “an anomaly” and leave it. Two mechanisms, from numbers the paper gives, would predict a drop. Both are our analysis.

Fewer updates. The learning rate is fixed at 10−3 and training runs 40 epochs. So steps fall as batch grows: about 160,000 steps at batch 32, 40,000 at 128, 10,000 at 512 and 6,700 at 768. That is 4% of the updates. Nothing says the learning rate was scaled to match.

Same-clip collisions. 128,010 pairs cover only 90,947 clips, so two entries in a batch can be the same audio with different captions. The loss insists they are negatives. We count about 88,800 colliding pairs out of 8.19 billion, a rate near 1.1 × 10−5 per random pair. A batch of N holds N(N − 1)/2 pairs, so expected collisions grow with N squared. They are about 0.005 at 32, 1.4 at 512 and 3.2 at 768. Deduplicating by audio when sampling would remove them.

Adding 1.7M noisy AudioSet pairs dropped ESC50 from 82.6% to 67.15%. Why is noise worse here than in ordinary supervised learning?

Chapter 9

What CLAP Cannot Do

Trace each limit back to the design choice that causes it, and hear one of them

CLAP is good at naming which sounds occur in a clip, and weak at most other questions you could ask about audio. This chapter lists what that costs. Each limit comes from a choice you have already met, so none of them is a surprise.

A bag of events

Chapter 4 pooled 690 frames into one vector. A pooled vector cannot hold “first this, then that.” Two clips with the same sounds in a different order can look almost the same to the model.

The device builds two clips from the same two parts. One is three dog barks then a siren sweep. The other is the siren sweep then the barks. Listen to both, then look at what a simple pooling step keeps.

Pool away the order

The grid is the log-Mel spectrogram of the clip you chose. The lines below are the same 64 Mel bands averaged over time. Switch the order and compare.

A simplification: real CNN14 pools after six convolution blocks, so it can keep a little more than a plain average. This device averages the log-Mel bands over time to show the direction of the effect. Both clips are synthesized from the same samples, only rearranged. The rain comparison uses a synthesized rain clip.

The two spectrograms look nothing alike: the dark band moves from one side to the other. The pooled lines nearly overlap. A clip of rain pooled the same way sits far from both. So the summary says which sounds are present and loses when.

One honest caveat. None of the paper’s 16 tasks asks when something happened or in what order. So this limit is inferred from the design, and the device above is a simplified demonstration. It is not a measurement of CLAP. The paper does not test it, and neither do we.

The count in a caption has the same problem. A caption can say “a horn is honked twice”. After pooling, the model holds only a trace of “horn”, and a clip with one honk can sit close to a clip with two. Treat counts and order as things CLAP was never asked to learn.

The speech gap has five causes

The paper offers one reason for the speech failures. There are at least five, and they all push the same way.

#CauseWhere from
1Captions describe events, not words. No caption transcribes speech or names an emotion.The paper, Section 4.2
2The Mel filters stop at 8 kHz and drop the hiss of “s” and “sh”.Section 3.2, Chapter 3
3A 1 second keyword clip is padded to 5 seconds, so 80% is silence.Section 3.2, our arithmetic
4Pooling to one vector loses the order of the sounds in a word.Chapter 4, the device above
5Three speech domains needed a custom prompt. That is a symptom, not a cause.Section 3.3

Cause 1 dominates, and Chapter 7’s control proves it. Fine-tuned CLAP reaches 96.83% on keywords, yet causes 2 to 4 still apply after fine-tuning. So they cannot be what limits the frozen model. The paper’s own forecast holds up: more speech data and speech-aware captions should help. That became the speech-aware audio-language models.

The honest list

LimitComes fromWhat later work tried
No order, no localisationPooling to one vector (Ch 4)Frame-level encoders and event-detection heads
Near chance on speech and emotionCaptions never describe speech (Ch 7)Speech-aware captions and speech encoders
Fixed 5 second windowRandom crop and padding (Ch 3)Fusion over long inputs, variable-length encoders
Cannot write textDual encoder by design (Ch 4)Captioning models and audio LLMs
Cannot abstainSoftmax over candidates (Ch 0, 6)Thresholds on the raw cosine, open-set methods
Depends on batch and caption qualityNegatives come from the batch (Ch 5, 8)Cleaner or generated captions
8 kHz ceilingThe Mel band edge (Ch 3)Full-band front ends

Bad input still gets an answer

Feed CLAP silence, a clip cut to 0.8 seconds, or an empty caption. Each one yields a valid unit vector, and so a valid ranking. A silent clip is a constant grid. CNN14 still emits 2048 numbers, and the ranking is confident nonsense. There is no null output. The abstention problem of Chapter 0 was inherited, not solved. CLAP fixed the vocabulary. It did not fix the calibration.

Text has a similar hole. Captions are cut at 100 characters, so about 17% of the paper’s own sample captions lose their tails. The tail often holds the second sound source.

Why do “a dog barks, then a siren wails” and “a siren wails, then a dog barks” get nearly the same CLAP embedding?

Chapter 10

What Came After

Follow the recipe into bigger models, audio data pipelines and the CLAP score, and filter a noisy corpus yourself

CLAP is a short paper with 128k pairs and two borrowed encoders. Its reach is far larger than its size, for one reason. It did not invent a technique. It moved an interface. After CLAP, the natural way to ask an audio model a question is to type the question.

Bigger versions of the same recipe

The 2022 paper is the source for everything above. The successors below come from their own papers, which you should check for exact figures.

CLAP inside an audio data pipeline

A pipeline cleans, tags and searches millions of clips. CLAP gives it a sentence-shaped hook into audio. Four jobs follow from the shared space.

  1. Search. Embed every clip once and store the vectors. Embed a query such as “glass breaking on tile”, then take the nearest clips.
  2. Filter pairs. Score each audio and caption pair. Drop pairs with a low cosine. This is the fix for the vlog-title problem of Chapter 8.
  3. Curate by sentence. Select a training set with “a siren wailing” instead of a label list that nobody wrote.
  4. Judge generated audio. The CLAP score is the cosine between the generated audio and the prompt. Text-to-audio systems report it, and some condition on CLAP embeddings.

Try job 2. Twelve pairs sit in a toy corpus. Some captions are right and some are video titles. Move the threshold and watch what is kept.

Filter a noisy corpus

Each row is a clip and its caption. The bar is the cosine between them. Pairs under your threshold are dropped. Tap a play key to hear a clip.

0.60

Illustrative: the cosines come from the same toy circle map as the hero, where each sound and caption has an angle we chose, not from a real model. Video-title captions sit at angles far from every sound. The wolf labelled as a dog scores high because the two are neighbours. Some clean pairs score low because the recording is noisy. A cosine filter drops obvious mismatches and keeps near-miss ones.

A threshold near 0.70 drops the vlog titles and keeps every clean pair. But the wolf labelled “a dog barks” scores 0.89, so it slips through. Raise the threshold to catch it and you lose a clean pair, such as the church bells recorded near traffic. No setting gets zero of both errors. Filtering by a model inherits that model’s blind spots. That is the bootstrapping loop later work uses: a model trained on clean pairs picks which noisy pairs to keep.

python# keep only audio-caption pairs that CLAP agrees with
E_a = model.encode_audio(batch_logmel)           # (N, 1024), unit length
E_t = model.encode_text(batch_captions)          # (N, 1024), unit length
score = (E_a * E_t).sum(-1)                    # (N,) the CLAP score of each pair
keep  = score > 0.3                             # choose the threshold on a labelled sample

# search: one sentence against an embedded archive
q = model.encode_text(["glass breaking on tile"])    # (1, 1024)
top20 = (archive_E_a @ q.T).squeeze().topk(20).indices

To set the threshold, hand-label a small sample, say 200 pairs, and mark each as right or wrong. Score them with the model. Then choose the cosine where precision and recall meet your needs. A strict filter keeps fewer pairs and cleaner ones. A loose filter keeps more pairs and some mistakes. The right choice depends on what a mistake costs your pipeline.

The same cosine works as a metric for generated sound. Ask a text-to-audio model for “rain on a tin roof”. Embed the prompt and the result, and take the cosine. Average it over a test set and you have a CLAP score. Compare two models on the same prompts and the higher score matched its prompts better.

Two cautions carry over from earlier chapters. The CLAP score is a bag-of-events score, so it cannot check that the horn honked twice or came after the siren (Chapter 9). And the score reflects what captions talk about, so it is weak on speech content (Chapter 7). Use it as a filter and a ranker, and keep a human on the borderline.

You filter a scraped audio corpus by CLAP score above 0.6. Which pairs get through most easily?

Chapter 11

The Field Guide

Keep the whole model on one page: numbers, equations, code, and a weekend recipe

CLAP puts a sound and a sentence about it at the same point. It does this with two pretrained encoders, two linear maps and one symmetric loss over a batch. Then classification becomes retrieval: embed the class names you type and pick the nearest. The recipe works at 128,010 pairs. Its limits are the limits of its captions.

The numbers worth keeping

NumberWhat it is
128,010Training pairs from FSD50K, ClothoV2, AudioCaps and MACS, over 90,947 unique clips.
82.6%ESC50 zero-shot, against AudioCLIP’s 69.40% and reported human accuracy of 81%.
73.24%, 30.24 mAPUrbanSound8K and FSD50K zero-shot. The FSD50K gain is 27 points of mAP over Wav2CLIP.
16 tasks, 8 domainsThe evaluation suite. Zero-shot beat chance on every task, and supervised CLAP set the state of the art on five.
64 × 690The log-Mel grid of a 5 second clip: 44.1 kHz, window 1024, hop 320, 64 bins from 50 to 8000 Hz.
2048, 768, 1024Audio embedding, text embedding, shared space. The projections hold 2,883,584 weights.
0.007, 142.86The initial temperature and the scale that multiplies the cosines. The scaled logits are clipped at 100.
0.2809 → 0.3265Average zero-shot with both encoders frozen, then both trainable. Text alone adds 33 times what audio alone adds.
55.55%ESC50 with both encoders frozen: the two projections alone.
82.6 → 67.15ESC50 after adding 1.7M noisy AudioSet pairs.

The equations in order

StepEquationShapes
Front endXa = logMel(wave)(64, 690)
(1) EncodeX̂a = fa(Xa), X̂t = ft(Xt)(N, 2048), (N, 768)
(2) ProjectEa = La(X̂a), Et = Lt(X̂t), then unit length(N, 1024) each
(3) CompareC = (1/τ) Et EaT, clipped at 100(N, N)
(4) LearnL = ½ (ℓtext + ℓaudio)scalar, baseline ln N
Zero-shotz = Etext Ea, then softmax or sigmoid(C, 1024) times (1024,)

The core as code

pythonimport torch, torch.nn as nn, torch.nn.functional as F
def clap_loss(E_t, E_a, logit_scale):          # both (N, d), unit length
    logits  = logit_scale.exp().clamp(max=100) * E_t @ E_a.T      # (N, N)
    targets = torch.arange(len(logits), device=logits.device)   # the diagonal
    return 0.5 * (F.cross_entropy(logits, targets) + F.cross_entropy(logits.T, targets))

@torch.no_grad()
def zero_shot(model, clip_logmel, class_names, template="this is a sound of {}"):
    E_text = model.encode_text([template.format(c) for c in class_names])   # (C, d)
    E_a    = model.encode_audio(clip_logmel[None])                              # (1, d)
    return (model.logit_scale.exp() * E_text @ E_a.T).softmax(0).squeeze()   # (C,)

Where it sits

CLIPWav2CLIP, AudioCLIPCLAPSelf-supervised audio
Modalitiesimage, textaudio, CLIP spaceaudio, textaudio
Text signalweb alt-textAudioSet class labelsfree human captionsnone
Pairs400Mlabel pairs128,010unlabelled hours
Open vocabularyyespartlyyesno, needs a fixed head

Build it in a weekend

StepDo thisThe choice that matters
1. PairsClothoV2 (5 captions a clip), or a few thousand of your own.Quality over quantity (Chapter 8).
2. Front endtorchaudio MelSpectrogram, then AmplitudeToDB.Match your band. Do not inherit an 8 kHz ceiling if you need speech.
3. Audio towerAny pretrained encoder. Freeze it.Costs little and saves compute (Chapter 8).
4. Text towerA sentence encoder. Unfreeze it.This is where the aligning happens.
5. ProjectionsTwo nn.Linear layers into 512 or 1024, then L2-normalise.Linear, not deeper.
6. ScaleA parameter set to log(1/0.007). Exponentiate. Clamp at 100.Tune to your d: random cosines spread as 1/√d.
7. BatchAs large as memory allows. Deduplicate by audio.Same-clip pairs are unsatisfiable negatives.
8. Sanity checkThe loss should fall below ln N in the first epoch.If it sits at ln N, the scale is the wrong way round.
9. EvaluateESC50 zero-shot with “this is a sound of [class]”.Five lines of code and the shared yardstick.

Keep going

The takeaway. CLAP turned the classifier’s weight matrix from trained parameters into a function of strings. The set of sounds you can recognise is now chosen when you ask, not when you annotate. It did this on 128,010 pairs, three orders of magnitude below CLIP. And it can only name what captions talk about.
Which sentence best explains why CLAP mattered more than its size suggests?

Now press Present or Teach and explain CLAP out loud from memory. Say why a fixed head cannot abstain and what pulling and pushing does. Give the shape of the grid and of the two vectors. Show how the diagonal of a 3 by 3 matrix becomes a loss, and why the text tower mattered 33 times more. Then go back to the sound at the top, play the smoke alarm, and explain which caption wins and why.

CLAP: Learning Audio Concepts from Natural Language Supervision
Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, Huaming Wang (Microsoft, 2022)
Read the paper · Back to Veanors

References: Radford et al., CLIP (2021). Kong et al., PANNs, the source of CNN14 (2020). Devlin et al., BERT (2019). Zhai et al., LiT (2021). Wu et al., Wav2CLIP (2022). Guzhov et al., AudioCLIP (2022). The four corpora: Drossos et al., Clotho (2020); Kim et al., AudioCaps (2019); Fonseca et al., FSD50K (2022); Martín-Morató and Mesaros, MACS (2021). Wu et al., LAION-CLAP (2023). Elizalde et al., CLAP 2023.