AudioFM learns to hear by playing a guessing game. It hides part of a recording, then picks the hidden patch out of a line-up of patches from other recordings. Added to whole-clip matching, that one game makes its frozen features the best or joint best on 10 of 12 tasks, from bird calls to heart murmurs.
Learn how one encoder can hold both the whole scene of a recording and its tiny moments, without a single label.
Play the student. A patch of the sound picture is hidden: look at what is still visible, then pick the patch that belongs in the hole. Then we build, piece by piece, the model that plays this game 6,144 recordings at a time.
You need what a vector and a dot product are and a rough idea of what a neural network does. We build the rest from zero.
Look at the outlined hole and the sound around it, then tap the patch you think belongs there.
The recordings are synthetic and drawn as small 4 × 4 patches so they are easy to compare. The masks (60% of time columns or 40% of frequency bands), the line-up of other recordings at the same spot, and the temperature of 0.2 are the paper's (Section 3.1, Section 4.1, Appendix A.1). The toy student fills the hole from its visible neighbours; the real student is a 12-layer Transformer trained on 1.68 million clips.
Chapter 0
See why one recording asks for both the big picture and the tiny details, and why that is hard to learn without labels
You record ten seconds in a busy café. Rain taps on the window. People talk at the next table. Then, at about second six, somebody drops a spoon.
Now ask three questions about that one recording. Is this a café, a train station or a street? Did something drop, and when? Is the coffee machine's pump making the high whine that means it is about to fail? The first question is about the whole ten seconds. The second is about a fifth of a second. The third is about one thin stripe of pitch.
Other fields ask the same three kinds of question. A bird survey asks which species sang in an hour of forest sound. A doctor listens to a heartbeat and asks whether there is a faint whoosh between two beats: a murmur. A sleep clinic asks whether a patient stopped breathing for ten seconds. A call centre asks whether a voice sounds angry. Each answer lives at a different scale of the same sound.
A synthetic café recording, drawn as a sound picture: time runs left to right, pitch runs bottom to top, brighter means louder. Pick a question and see which part of the picture holds its answer.
An illustrative recording. The point it makes is the paper's own (Section 1): overall context informs scene recognition, while subtle temporal and spectral differences separate animal calls or patterns in body sounds.
A model that answers all three questions needs a description of the sound that keeps all three scales. In machine learning such a description is called a representation: a list of numbers (a vector) that a model computes from the input. A good representation puts similar sounds near each other and keeps the facts a later task needs. A representation of the whole clip is one vector. A representation of each moment is one vector per short slice of time.
You could. You would need labels for each question: someone must listen to thousands of clips and write "café", or mark the exact second the spoon fell, or have a cardiologist grade each heartbeat. Labels like these are slow and expensive. Medical and animal labels also need experts.
Self-supervised learning (SSL) avoids that cost. The model invents its own task from unlabelled recordings, for example "hide part of the sound and guess it back". It learns a representation while it solves that task. Then you throw the task away and keep the representation. A model trained like this, then reused for many tasks, is called a foundation model. The paper's goal is an audio foundation model: one encoder whose features work for scenes, events, animals, hearts, lungs and voices.
Audio SSL has two main families. The first makes two altered copies of the same clip and trains the model so that both copies get the same clip vector. This is recording-level contrast. It is good at the scene question. But it only grades one pooled vector per clip, so nothing checks what each moment's vector says.
The second family hides parts of the sound and asks the model to fill them in. This is masked prediction. It grades every hidden piece, so it is good at moments and detail. But "fill in the blank" can be solved by copying nearby sound, and it does not ask whether the whole clip hangs together.
AudioFM, from Google Research and Google DeepMind, keeps the first family's clip-level contrast and adds a new kind of masked task. The model must not only guess the hidden piece. It must pick the true hidden piece out of a line-up of pieces from other recordings at the same spot. That is the game in the hero. The paper calls it masked latent discrimination, or MLD.
The results back the idea. On 12 tasks across four domains, with the encoder frozen and only a linear layer trained on top, AudioFM is best or joint best on 10. It matches the best published score on AudioSet-2M fine-tuning (50.2 mAP) with about 13% fewer training clips. It also leads on all three measures of finding sound events in time. Here is the road, chapter by chapter. Each part of the hero is explained on the way.
Chapter 1
Turn a pressure wiggle into a log-Mel spectrogram, cut it into 512 patches, and follow their shapes into a Transformer
A microphone measures air pressure many times a second. AudioFM resamples every recording to 16,000 measurements per second, from one channel (mono). Its input window is 10.24 seconds long, so one clip is 16,000 × 10.24 = 163,840 numbers in a row. That row is the waveform.
The waveform is hard to read. A dropped spoon and a spoken vowel are both just wiggles. What our ears actually notice is which pitches are loud at each moment. So the first step turns the row into a picture of pitch over time.
The same synthetic clip at four stages. Step through them, then drag the time cursor. In the last stage, tap any patch to see what it covers.
The 16 kHz rate, 128 Mel bands, 1,024 frames and 16 × 16 patches are the paper's (Section 4.1). The patch frequency ranges assume a Mel filter bank from 0 to 8,000 Hz, so they are approximate. The clip itself is synthetic.
Cut the waveform into short overlapping slices, each about 25 ms long, starting a new slice every 10 ms. For each slice, ask: how much of a pure tone at 100 Hz is in here? At 101 Hz? At every frequency? To answer, multiply the slice point by point with a pure tone and add up the products. If the slice contains that tone, the products line up and the sum is large. If not, positive and negative products cancel and the sum is near zero. Doing this for every frequency at once is the Fourier transform.
The 25 ms slice length is a trade. A longer slice can tell 100 Hz from 101 Hz, but blurs when things happened. A shorter slice is sharp in time but blurry in pitch. You cannot have both; 25 ms is the usual compromise for speech and everyday sound.
For each slice, the transform measures how much energy sits at each frequency. One slice's result is a spectrum. Stack the spectra side by side and you get a spectrogram: time across, frequency up, brightness for energy. At one slice every 10 ms, 10.24 seconds gives 1,024 slices, which the paper calls frames.
Our ears are fussy about low pitches and relaxed about high ones. The step from 100 Hz to 200 Hz sounds huge. The step from 7,100 Hz to 7,200 Hz is barely noticeable. The Mel scale stretches the low end and squeezes the high end to match:
100 Hz → 150 · 1,000 Hz → 1,000 · 4,000 Hz → 2,146 · 8,000 Hz → 2,840 Mel
So the octave from 4,000 to 8,000 Hz gets 694 Mel, while the first 1,000 Hz alone get 1,000. AudioFM uses 128 Mel bands, evenly spaced in Mel. Finally it takes the logarithm of each band's energy, because loudness also works in ratios: a sound ten times stronger sounds a fixed step louder, not ten times louder. The result is a log-Mel spectrogram: a grid of 128 bands by 1,024 frames, 131,072 numbers. Before training, every value is shifted and scaled by one fixed mean and spread (−4.268 and 4.569 for AudioSet, Table 4) so the numbers sit near zero.
Now the trick borrowed from image models. Cut the grid into non-overlapping squares of 16 bands by 16 frames. Each square is a patch. The frequency axis gives 128 / 16 = 8 patches, and the time axis gives 1,024 / 16 = 64. That is a grid of 8 × 64 = 512 patches.
16 frames × 10 ms = 160 ms per patch column · 64 columns × 160 ms = 10.24 s
Remember that 160 ms. It is the paper's unit of time for everything that follows. A dropped spoon lasts one or two columns. A spoken syllable lasts one to three. A held note can fill twenty.
We name each patch by its place: column t (time, 1 to 64) and row f (frequency band, 1 to 8). The paper writes the set of all places as P = {1, …, T} × {1, …, F} with T = 64 and F = 8. A place (t, f) means the same slice of time and pitch in every recording. That fact becomes important in Chapter 5.
Each patch is 16 × 16 = 256 numbers. A learned linear layer maps those 256 numbers to a vector of D = 768 numbers. "Linear layer" sounds grand, but it is a table of weights, 256 rows by 768 columns. Each of the 768 outputs is a weighted sum of the 256 inputs, with its own column of weights. Training tunes the weights so that the sums become useful. The 768-number result is a token. A learned position embedding, one per place (t, f), is added to it, so the model knows where each token came from. One more learned token, the class token or CLS, joins the front of the list.
The tokens go into a Transformer: a stack of 12 identical layers. In each layer every token looks at every other token and updates itself with what it found. This is attention. Each token scores every other token with a dot product, a measure of how well two vectors point the same way. It turns the scores into weights that sum to 1, then adds in a weighted mix of the others. A patch of a dog's bark can pull in information from the bark's other patches, wherever they are in the grid. This exact design is the Vision Transformer, Base size (ViT-B): 12 layers, 768 numbers per token, 12 attention heads. Out comes one 768-number vector per patch, plus the class token. Those output vectors are the representation.
| Stage | Shape | What it is |
|---|---|---|
| Waveform | 163,840 | pressure, 16,000 per second for 10.24 s |
| Log-Mel | 128 × 1,024 | bands by 10 ms frames, normalised |
| Patches | 512 × 256 | 8 bands × 64 columns of 16 × 16 squares |
| Tokens | 513 × 768 | linear map + position embedding, plus the class token |
| Encoder output | 513 × 768 | after 12 Transformer layers: one vector per patch |
Here is the same thing as code. It is a sketch with the paper's sizes, not the authors' code.
pythonimport torch, torchaudio wave = torchaudio.functional.resample(wave, sr, 16000)[:, :163840] # mono, 10.24 s fbank = torchaudio.compliance.kaldi.fbank(wave, num_mel_bins=128, frame_shift=10, frame_length=25) # (1024, 128) x = ((fbank - (-4.268)) / 4.569).T # (128, 1024): bands x frames patches = x.unfold(0, 16, 16).unfold(1, 16, 16) # (8, 64, 16, 16) patches = patches.reshape(512, 256) # one row per patch tokens = embed(patches) + pos_embed # (512, 768) tokens = torch.cat([cls_token, tokens]) # (513, 768) -> 12 ViT layers
Chapter 2
Build the clip-level contrast loss from a dot product, a softmax and a logarithm, and see why it needs other clips
We have no labels. So where does a learning signal come from? Here is the oldest trick in self-supervised learning. Take one clip and make two altered copies of it, called views. Whatever the clip "is" (rain in a café, a dog in a park) did not change. So the model should give both views the same vector.
AudioFM makes its two views in two steps. First it applies random changes to each copy of the waveform, following the recipe of AudioMosaic, an earlier model by other authors. Then it hides a different part of each view's spectrogram (Chapter 3). The model must see through both changes and recognise the same recording.
The obvious loss is "make the two vectors close". Measure the distance between them and push it to zero. Try it and something silly happens. The model learns to output the same vector for every clip it ever sees. Then every pair is identical, the distance is zero, and the loss is perfect. The vector says nothing. This failure is called collapse.
The fix is to also say what the vector must not match. A clip's view should be close to its own partner view and far from the views of every other clip in the batch. Then the constant answer scores terribly, because it is also close to all the wrong clips. This is contrastive learning: learn by contrasting the right match with wrong ones. The wrong ones are called negatives.
First we need a number for "how alike are two vectors?". Take two vectors and multiply them entry by entry, then add up the products. This is the dot product. If both vectors are scaled to length 1 first (ℓ2 normalisation: divide each entry by the vector's length), the dot product is the cosine of the angle between them. It is 1 when they point the same way, 0 when they are at right angles, and −1 when they point opposite ways.
a = (0.6, 0.8), b = (0.8, 0.6): a · b = 0.6 × 0.8 + 0.8 × 0.6 = 0.96 (both have length 1, so the angle is about 16°)
Normalising matters. Without it, the model could raise a similarity by making a vector longer, which is cheating. With it, all vectors live on the surface of a ball, and only their directions count. AudioFM projects each pooled clip vector down to 128 numbers and normalises it. The paper calls the result z.
Now the anchor view has one score for each candidate: its partner and every negative. We want a probability for "candidate k is my partner". Probabilities must be positive and add up to 1, and a higher score should mean a higher probability. The simplest recipe that does all three: raise e to the power of each score, then divide by the total. This is the softmax.
One more knob. Before taking e to the power, divide every score by a small number τ, the temperature. Dividing by 0.1 multiplies every gap between scores by ten. That makes the softmax much more decisive. Here are three scores, a partner at 0.9 and two negatives at 0.3 and 0.1:
| τ | e0.9/τ (partner) | e0.3/τ | e0.1/τ | p(partner) | loss = −log p |
|---|---|---|---|---|---|
| 1.0 | 2.46 | 1.35 | 1.11 | 0.50 | 0.692 |
| 0.1 | 8,103 | 20.1 | 2.72 | 0.997 | 0.003 |
At τ = 1 the model is barely sure, even though the partner clearly scores highest. At τ = 0.1, the paper's value for the global loss, the same scores give near certainty.
We want the partner's probability p to be 1. A convenient penalty is −log p. It is 0 when p = 1, and it climbs without limit as p falls toward 0. That makes confident mistakes very expensive. This penalty is the cross-entropy loss, the same one used to train every classifier. Here the "classes" are the candidates in the batch, and the right class is the partner view.
Put the three pieces together and you have the global loss AudioFM uses, called NT-Xent (normalised temperature-scaled cross-entropy):
Every one of the 2B views takes a turn as the anchor, and the losses are averaged. That is why it is called symmetric. With the paper's batch of B = 6,144 clips there are 12,288 views, so each anchor's sum has 12,287 terms: one partner and 12,286 negatives.
Six clips, two views each, shown as directions on a circle (real vectors have 128 numbers; two are enough to see the idea). Tap a dot to make it the anchor. Press Train to run the loss. Then turn negatives off and train again.
A toy: two-number vectors and exact gradients of the NT-Xent loss above, six clips instead of 6,144. With negatives off, the loss is only "move the partners together", which shows collapse.
Train with negatives on and the pairs snap together while the six clips spread out around the circle. Turn negatives off and every dot slides to the same spot. The loss falls to zero while the vectors become useless. That is collapse, live.
One last detail. The pooled vector passes through a small extra network, the projection head, before the loss sees it. After training the head is thrown away and only the encoder is kept. The head absorbs the loss's demands (for example, "ignore loudness"), so the encoder's own features can stay richer than the loss strictly needs. This trick comes from SimCLR in vision, and every model in this paper uses some version of it.
Chapter 3
Build the two complementary masks, count what each view keeps, and hear the difference
The two views are mostly made different by hiding. Hiding does two jobs at once. It makes the views genuinely different, so agreeing is not trivial. It also makes training cheaper, because the student network only processes the patches it can see.
The question is what to hide. The cheapest choice is random patches here and there. But then almost every hidden patch has visible neighbours on all four sides, and a network can fill the gap by blending them, like smoothing over a scratch on a photo. It learns to blend, not to understand.
AudioFM, following AudioMosaic, hides in stripes, and the two views use different stripes:
One synthetic clip, masked two ways. Drag the two ratios. The read-out counts what each view keeps and how much they share, and gives the paper's score at that ratio where it measured one. Listen plays a view with its hidden parts silent.
The default ratios (60% of time columns, 40% of bands) and the measured scores at rt = 0.45, 0.60, 0.75 and rf = 0.25, 0.40, 0.75 are the paper's (Figure 6, AudioSet-20K linear probing; values other than the default 35.0 are read off the plot). With 8 bands, 40% is 3.2, so the slider steps through whole bands; how the paper rounds is not stated.
Let's count with the defaults. Sixty percent of 64 columns is 38.4; call it 38 hidden columns and 26 visible. View 1 keeps 26 × 8 = 208 patches and hides 304. Forty percent of 8 bands is 3.2; call it 3 hidden bands and 5 visible. View 2 keeps 5 × 64 = 320 patches and hides 192.
Now the interesting number: how many patches can both views see? A patch is visible in both only if its column survived view 1 and its band survived view 2:
(26 / 64) × (5 / 8) × 512 = 26 × 5 = 130 patches (about a quarter of the grid)
So the two views share only a quarter of their evidence. View 1 knows every pitch at a few moments. View 2 knows a few pitches at every moment. For their pooled vectors to agree, the encoder cannot just match the patches they share. It must work out what the whole recording is from two very different glimpses. That is the sense in which the masks are complementary.
The authors varied each ratio and measured how good the frozen features were on AudioSet-20K (Appendix A.2). Hide fewer columns, rt = 0.45, and the score drops from 35.0 to about 32.4 mAP. Hide more, 0.75, and it holds at about 35.0. For bands, hiding 25% gives about 34.0, and hiding 75% falls to about 32.5.
Why the asymmetry? Time is long: even with 75% of columns gone, 16 columns of complete spectrum remain, enough to tell what the scene is. Bands are few: hiding 6 of 8 bands can remove a sound's whole energy range. A bark that lives in bands 3 to 5 may vanish from view 2 entirely. Then the two views no longer show the same events, and "make them agree" asks for something impossible.
Chapter 4
Prove that a clip-level loss gives every patch the same instruction, and see what the masked-prediction family does about it
The global loss of Chapter 2 sees one vector per view: the pooled vector, the average of all the visible patch outputs. An average has a blind spot that you can test in a second. Move the bark from second 2 to second 8. Shuffle the columns in any order you like. The average does not change.
A park recording with one dog bark. The middle row shows a toy vector for each time column (its eight band energies). The bars at the bottom show the average over all columns, the pooled vector. Drag the bark, or shuffle every column.
A toy: each column's vector is its band energies, and the background is perfectly steady so the average is exactly unchanged. A trained encoder's patch vectors also depend on context, but the pooled average is still blind to order in the same way.
Blindness to order is only half the problem. The deeper half is what the loss teaches each patch. Write the pooled vector as an average of the N visible patch outputs h1, …, hN:
The loss depends on the patches only through h̄. Training asks each output "which way should you move to lower the loss?". That direction is the gradient. By the chain rule, the gradient for patch n is the gradient for h̄ times how much h̄ changes when hn changes, which is 1/N:
∂L / ∂hn = (1/N) · ∂L / ∂h̄ the same for every n
Every visible patch receives the identical instruction, whether it holds the bark, the rain or silence. The loss says "nudge the average this way" and splits the nudge evenly. Nothing in the global loss rewards the bark's patch for describing the bark. Any set of patch vectors with the right average scores exactly the same.
The paper measures the cost directly. Training with the global loss alone, then adding AudioFM's local loss, raises frame-level event detection on AudioSet Strong by 1.1 points of frame mAP and 2.1 points of PSDS2 (Figure 4b). Clip-level tagging on AudioSet-20K gains 0.6 mAP. The biggest gains are on the questions that need when.
Masked prediction attacks the gap head on: it grades every hidden patch. Three versions came before AudioFM, and they differ in what they ask the model to predict:
| Target | Example | Weak spot |
|---|---|---|
| The hidden spectrogram itself | AudioMAE | spends effort on exact noise texture that no task cares about |
| A discrete code for the patch | BEATs | needs a separate tokenizer, trained in rounds |
| A teacher network's vector for the patch | EAT, SSLAM, M2D, BAT | a regression loss is happy with a blurry average guess |
That last weak spot deserves a closer look, because it is what AudioFM's line-up fixes. Suppose the hole could hold a dog bark or a door slam, each equally likely given the visible context. A loss that measures squared distance to the target is lowest, on average, when the guess sits exactly between the two. So the model learns to predict a blend of bark and slam, a vector that matches neither. Averaged over many holes, the predictions drift toward bland middles.
A line-up does not reward the middle. If the true patch must out-score decoys from other recordings, a vague guess loses to any decoy that happens to sit near the middle. The prediction has to commit to something specific to this recording. Chapter 5 builds that line-up.
Chapter 5
Build masked latent discrimination step by step, work its loss by hand, and see why the decoys come from other recordings
Here is the plan in plain words. For each hidden place in a view, the student makes a guess: a vector that says "this is what I think was here". A teacher, which saw the complete spectrogram, provides the true answer at that place, also as a vector. The student must then pick its own answer out of a line-up. The line-up is the teacher's answers at the same place in every other recording of the batch.
That is the game you played in the hero, with one difference: you compared small pictures, and the model compares 128-number vectors. Let's build it in four steps.
The student encoder processes only the visible patches of a view. Its outputs go into a second, smaller network, the predictor, which puts them back on the 8 × 64 grid and fills every hidden place with a guess. Chapter 6 opens the predictor up. For now, what matters is its output. A small network called the local projection head maps each guess to 128 numbers and normalises it to length 1. The paper names the student's guess for recording i, view v, place (t, f):
pi,t,fv · 128 numbers, length 1 · one for every hidden place
The teacher is a second copy of the encoder that sees the same view without any hiding: all 512 patches. Its output at each place goes through its own copy of the local projection head and is normalised. That gives the target qi,t,fv, also 128 numbers of length 1. No gradient flows back into the teacher. It only supplies answers.
For one hidden place (t, f) of recording i, the candidates are the teacher targets at that same place (t, f) in every recording of the batch, j = 1 to B. Exactly one of them, j = i, is the right answer. The score of candidate j is the dot product pi,t,f · qj,t,f.
From here it is Chapter 2 again: softmax over the scores, with a temperature, and minus the log of the right answer's probability. This is Equation 1 of the paper:
Work it by hand with a line-up of four. Say the guess scores 0.80 against its own target and 0.55, 0.30 and 0.10 against three decoys. Divide by τp = 0.2 and raise e to each power:
e4.00 = 54.60 · e2.75 = 15.64 · e1.50 = 4.48 · e0.50 = 1.65 → total 76.37
p = 54.60 / 76.37 = 0.715 loss = −log 0.715 = 0.336
The 0.55 decoy is doing real damage: on its own it takes 20% of the probability. To lower the loss, the student must make its guess more specific, so that the decoy's score falls. That is exactly the pressure the regression loss of Chapter 4 lacked. In the paper the line-up is not four but 6,144 recordings long.
A toy batch of recordings, each time-masked. Tap any hidden place on the top recording to choose (t, f). The grid below has one row per recording's guess and one column per target at that same place. Every row is one line-up, and its right answer sits on the diagonal. Then switch where the decoys come from.
A toy. Each place's vector is its 4 × 4 patch of energies plus a short summary of its whole recording (how loud, and how changeable, each band is), centred and normalised, the way a Transformer's patch outputs carry context; the toy student guesses a hidden patch by blending the nearest visible columns; and the whole toy batch shares one mask, where the paper draws a mask per recording. The line-up construction (same place, other recordings), Equation 1 and τp = 0.2 are the paper's. Recording 3 is a steady engine hum on purpose: switch to "same recording" to see why.
This is the design choice that separates AudioFM from earlier local contrast methods, so it is worth slowing down. Older methods (SSAST, MAE-AST in audio; SPIRAL and ConMIM elsewhere) took their decoys from other places in the same recording. That sounds natural: "is this the patch from second 3, or from second 7?". Try it on the hum in the device.
A steady engine hum sounds the same at second 3 and at second 7. Their patches are nearly identical. If second 7 is a decoy, the loss demands that the guess for second 3 score low against second 7. The model is punished for noticing that two moments sound the same, which is true. A decoy that is secretly a right answer is called a false negative. The paper names the problem directly: its candidate construction avoids requiring acoustically similar regions within a recording to be distinguished solely because they occupy different positions.
Decoys from other recordings at the same place fix both halves of the problem:
The scale is large. With the default masks there are about 304 hidden places in view 1 and 192 in view 2. Every one of the 6,144 recordings plays the game at every hidden place, so one training step plays about 3 million line-ups, each 6,144 candidates long. In code, each place's line-ups are one matrix multiply: all B guesses against all B targets, a B × B grid of scores whose diagonal holds the right answers. That is the grid in the device.
Chapter 6
Build the answer key from a slow copy of the student, and open the predictor that turns visible patches into guesses
The line-up needs an answer key: a target vector for every place of every recording. Where can good targets come from, with no labels? They must be meaningful, available for free, and they should improve as training goes on. A fixed, frozen network would not improve. The raw spectrogram is free but full of detail nobody needs (Chapter 4).
The answer used by AudioFM, and by BYOL, DINO, data2vec and many others, is surprising at first: use the student itself, but a slower, steadier version of it. This copy is the teacher.
The teacher has the same architecture as the student, with its own weights θ̄. After every training step, each teacher weight moves a small fraction of the way toward the matching student weight:
Unroll the rule and you see what it means. The teacher is a weighted average of all past students: the newest gets weight (1 − m), the one before m(1 − m), then m2(1 − m), and so on. Old students fade by a factor m per step. This is an exponential moving average (EMA). A handy measure is the half-life: after how many steps does a student's influence halve? Solve mn = 1/2:
n = ln(0.5) / ln(m) · m = 0.990 → 69 steps · m = 0.9995 → 1,386 steps
Early in training the student changes fast and is mostly wrong, so a teacher that follows it within about 70 steps keeps the answers current. Late in training the student is good and its changes are small, noisy refinements. A teacher averaging over about 1,400 steps smooths out that noise and gives steady targets.
Two weights of a toy student wander as training steps go by: purple. The green teacher follows by the rule above. Drag the momentum and press Run.
A toy random walk with a slow drift, 3,000 steps. The momentum values 0.990 and 0.9995 are the paper's start and end (Appendix A.1); the others are there to compare.
The teacher sees the full view. For each student view, the teacher gets the same augmented spectrogram with nothing hidden: all 512 patches. So its vector for a place was computed with full knowledge of the whole clip. When the student, from partial evidence, matches that vector, it has inferred what full knowledge would say.
No gradient flows into the teacher. This is called stop-gradient. Without it, the loss could be lowered by moving the targets toward the guesses instead of the other way round, and both would drift toward a shared trivial answer. With it, the targets are fixed for this step, and only the student moves.
Now the missing piece of Chapter 5: how visible patch outputs become guesses at hidden places. The predictor is a small Transformer that works on the whole grid. Follow one view through it:
| Step | Shape | What happens, and why |
|---|---|---|
| Student encoder | 209 × 768 | the 208 visible patches of view 1 plus the class token |
| Restore the grid | 512 × 768 | drop the class token, put each visible output back at its (t, f), fill the 304 hidden places with one shared learned mask token |
| Narrow | 512 × 384 | a linear map to Dpred = 384: a deliberate bottleneck |
| Say where, say which view | 512 × 384 | add fixed sine-cosine positions (192 numbers for time, 192 for frequency) and a learned embedding that says "time-masked" or "frequency-masked" |
| 4 Transformer blocks | 512 × 384 | full attention across time and frequency, 6 heads: each hidden place gathers evidence from every visible patch |
| Widen | 512 × 768 | a linear map back to 768 |
| Local projection head | 512 × 128 | a two-layer network 768 → 512 → 128 with LayerNorm, then length 1. Only the 304 hidden places enter the loss |
Three of these choices carry real reasons. The bottleneck (384 instead of 768) and the small depth (4 blocks instead of 12) keep the predictor weak on purpose. A strong predictor could do the inferring by itself and leave the encoder lazy. A weak one forces the encoder's outputs to already contain what the guesses need.
The view embedding exists because the two views pose different puzzles. A missing column must be inferred from the moments around it. A missing band must be inferred from the pitches around it. Telling the predictor which puzzle it faces lets one shared network solve both. The fixed positions tell every hidden place exactly where it sits, so its guess is about that place.
After pretraining, the predictor, both projection heads and the teacher are all thrown away. Only the student encoder is kept: 86 million weights out of the 94 million trained.
Chapter 7
Give partial credit to the moments just before and after, and block the shortcut that would make it cheating
Most sounds last longer than 160 ms. A vowel, a hum or a bird's trill at column 20 is often still there at column 21. So if the student's guess for (20, f) happens to look like the teacher's answer at (21, f), that is not a mistake. It is a sensible near-miss. Chapter 5's loss gives it no credit at all.
AudioFM adds that credit. The same guess pi,t,f also plays a line-up against the targets at (t−1, f) and (t+1, f): one column earlier and one column later, at the same pitch. Each of those line-ups uses its own decoys, from other recordings at that neighbouring place. The three losses are mixed with fixed weights that keep the guess's own place in charge:
weights for offsets −1, 0, +1 = 0.1, 0.8, 0.1 (radius K = 1, a window of W = 3 columns)
These weights are a bell curve sampled at three points and scaled to add up to 1 (Appendix A.1). Their spread works out to about half a column, roughly 80 ms. So "neighbour" here means the very next moment, not the general area.
There is a trap. Suppose column 20 is hidden but column 21 is visible. Then the student saw column 21's patch directly. To score on the (21, f) line-up, the predictor does not need to infer anything: it can copy the visible patch's features into the guess. The loss would reward copying, which is the shortcut Chapter 3's stripes were designed to remove.
So AudioFM only counts a neighbour if it is inside the grid and also hidden in the student's view (Equation 2). The guess's own place always counts, because it is hidden by definition:
Then the surviving weights are rescaled to sum to 1, keeping their proportions, and the neighbourhood loss is their weighted sum (Equation 3):
A worked case. Column 20 is hidden. Column 19 is visible, so offset −1 is dropped. Column 21 is hidden, so +1 stays. The surviving weights 0.8 and 0.1 add to 0.9, so they become 0.8 / 0.9 and 0.1 / 0.9:
ℓ = 0.889 · ℓ(own place) + 0.111 · ℓ(one column later) (the paper's 8/9 and 1/9)
If neither neighbour is hidden, the own place gets weight 1 and the loss is plain Equation 1. The two views behave very differently here, and the device shows why. In the time-masked view, a neighbour counts only if its whole column was also hidden. In the frequency-masked view, a hidden band is hidden at every moment, so both neighbours always count, except at the first and last column.
Pick a view and a radius, then tap any hidden (hatched) place. Its window lights up: green neighbours count, red ones were visible to the student and are dropped. The bars show the weights before and after rescaling, and the paper's measured gains for that radius.
The weights (K = 1: 0.1, 0.8, 0.1; K = 2: 0.03, 0.20, 0.54, 0.20, 0.03), the mask-aware rule and the gains over global-only training are the paper's (Appendix A.1, Figure 4b). Only the K = 1 gains are printed in the figure; the K = 0 and K = 2 gains are read off its bars and are approximate.
The paper compared three radii, each against a model trained with the global loss only (Figure 4b). Center-only, K = 0, already helps. K = 1 helps most on every measure: +0.6 mAP on AudioSet-20K tagging, +1.1 frame mAP, +0.8 PSDS1 and +2.1 PSDS2 on event detection. K = 2 is worse than K = 0 and on some measures worse than no local loss at all.
The K = 2 weights explain why. They put only 54% on the own place and spread 46% over four neighbours up to 320 ms away. Two columns away, a sound has often changed. Now the guess is pulled toward several different targets at once, which is the blur problem of Chapter 4 returning through the back door. One column of tolerance helps; two columns smear the moment.
Chapter 8
Follow a batch through both branches with every shape, add the two losses, and read the whole thing as code
Every piece is now on the table. Let's run one training step from start to finish, with the paper's numbers. A step takes a batch of B = 6,144 unlabelled clips from AudioSet. For each clip:
Every box is one stage of the step. Tap a box to see its shape, what it does and whether it is trained. Press Play to watch a clip flow through in order. Then pick a local-loss weight to see the paper's ablation.
Shapes use the default masks with the rounding of Chapter 3 (208 and 320 visible patches). The λ gains are the paper's (Figure 4a: AudioSet-20K probing mAP over global-only training, with K = 1).
Why add the losses with weight 1? Here is a neat way to see that 1 is a natural starting point. At the very start of training, the vectors are close to random, so every candidate scores about the same. Then the softmax gives each candidate equal probability, and the loss is minus the log of 1 over the number of candidates:
Lglobal ≈ ln 12,287 = 9.42 · LMLD ≈ ln 6,144 = 8.72
The two losses start at almost the same size, so neither drowns the other. The paper then tested the weight (Figure 4a). Every positive weight beats training without the local loss: λ = 0.5 adds 0.2 mAP, λ = 1.0 adds 0.6, and λ = 2.0 adds 0.3. Too little local loss and the patches are under-taught. Too much and the local game starts to crowd out the clip-level structure that tagging tasks rely on.
Pretraining ran on a Cloud TPU v4-128 slice (64 chips) for 125,000 steps at a batch of 6,144 clips. That is 6,144 × 125,000 = 768 million clip draws, about 457 passes over the 1.68 million clips the authors could still download from AudioSet. The optimiser is AdamW with a peak learning rate of 6 × 10−4. The rate warms up over 2,500 steps, then falls along a half cosine to 10−6. Weight decay is 0.05, and gradients are clipped at 1.0 (Table 4).
Here is the local loss as code, written for clarity rather than speed. The real thing batches the loops into tensor operations.
pythonimport torch, torch.nn.functional as nnf def mld_loss(p, q, hidden, w=(0.1, 0.8, 0.1), tau=0.2): """p, q: (B, T, F, 128) student guesses and teacher targets, each length 1. hidden: (B, T, F) True where the place is hidden in this clip's student view.""" B, T, F, _ = p.shape K = len(w) // 2 right = torch.arange(B) # row i's right answer is column i total, n = 0.0, 0 for t in range(T): for f in range(F): anchors = hidden[:, t, f] # clips that hid (t, f) if not anchors.any(): continue ls, ws = [], [] for k, d in enumerate(range(-K, K + 1)): s = t + d if s < 0 or s >= T: continue # off the grid logits = p[:, t, f] @ q[:, s, f].T / tau # (B, B) line-ups at place (s, f) ce = nnf.cross_entropy(logits, right, reduction="none") ok = hidden[:, s, f].float() # mask-aware: the neighbour must be hidden too ls.append(ce * ok); ws.append(w[k] * ok) W = torch.stack(ws) L = (torch.stack(ls) * W).sum(0) / W.sum(0).clamp_min(1e-8) # rescaled weights total = total + L[anchors].sum(); n += int(anchors.sum()) return total / n # one step (sketch) loss = nt_xent(z1, z2, tau=0.1) + 1.0 * 0.5 * (mld_loss(p1, q1, hid1) + mld_loss(p2, q2, hid2)) loss.backward(); opt.step(); ema_update(teacher, student, m)
Chapter 9
Learn the fair way to score a representation, read the paper's twelve tasks, and explain a strange collapse
After pretraining we have an encoder that turns any clip into vectors. How good are those vectors? There are two common tests, and they measure different things.
Fine-tuning adds a small output layer for a task and then trains everything, the encoder included, on that task's labels. It shows how good a starting point the encoder is. But scores also depend on the fine-tuning recipe: the data augmentation, the learning rates, how long you train. Two papers with different recipes are hard to compare.
Linear probing freezes the encoder completely and trains only one linear layer on top: a weighted sum of the vector's numbers for each class. A linear layer cannot build new features. It can only read what is already there. So a probe asks a sharper question: how easily can a task's answer be read off these vectors? The paper ran every encoder through the same probing pipeline: the same data splits, labels, metric and probe recipe, each averaged over three random seeds.
The paper probes two ways. The bare probe reads the raw features. The flexible probe first applies a LayerNorm. For each clip's vector, it subtracts the mean of the 768 numbers and divides by their spread. Then it applies a learned scale and shift to each number. That puts every feature on a similar scale before the linear layer sees it.
That sounds like a small detail. Look at what it does to two strong baselines on AudioSet-20K:
EAT: bare 1.1 mAP → LayerNorm 18.7 · SSLAM: bare 0.9 → 18.2 · AudioFM: bare 31.2 → 35.0
A score near 1 mAP on 527 classes is close to random guessing. These are good encoders: with fine-tuning, both reach over 39 mAP. Their raw features simply do not work with a fixed, simple probe recipe. The paper reports this without explaining it. The most likely reason is scale: if a few feature numbers are hundreds of times larger than the rest, one learning rate cannot suit them all. Steps that are right for the big numbers are far too small for the informative small ones. Try it.
200 toy clips of two classes. The useful feature is small; a useless one is huge. Both probes get the same recipe: the same learning rate and the same number of steps. Press Train, then drag the size of the useless feature.
A toy logistic-regression probe, plain gradient descent with a fixed step size and 300 steps. It shows one mechanism that can sink a bare probe; the paper does not say what goes wrong inside EAT and SSLAM.
So a bare probe rewards features that are well-scaled out of the box, which is exactly what a downstream user wants from a ready-made encoder. AudioFM's raw features hold up. Under bare probing it is best or joint best on 10 of the 12 tasks. Explore the full table:
Each group of bars is one task; each bar is one encoder with its weights frozen. Pick a domain and a probe.
All numbers are the paper's Table 3: means over three seeds, in percent. A-MAE is AudioMAE; AMos is AudioMosaic.
The table uses four measures. Each answers a slightly different question, so it helps to build them from zero.
Two medical results stand out. On sleep apnea subtypes (PSG-Audio) AudioFM's bare features beat the next best by 3.2 points of macro-F1, and on heart murmurs (CirCor) by 4.7. Neither task was anywhere in the pretraining data, which was YouTube audio.
With full fine-tuning (Table 2) AudioFM also leads, but the margins are small, as they usually are at the top of these benchmarks:
| Model | AS-20K | AS-2M | FSD50K | ESC-50 | SPC-2 | SPC-1 |
|---|---|---|---|---|---|---|
| BEATsiter3 | 38.0 | 48.0 / 46.7 | 57.6 | 95.6 | 98.3 | 97.7 |
| AudioMAE | 35.5 | 47.3 / 46.5 | 57.3 | 94.1 | 98.3 | 96.9 |
| EAT | 40.5 | 48.6 / 48.8 | 67.5 | 95.9 | 98.3 | – |
| SSLAM | 39.3 | 50.2 / 50.0 | 66.6 | 96.2 | 98.1 | 98.8 |
| AudioMosaic | 41.8 | 50.2 / 49.2 | 70.5 | 97.5 | 98.4 | 99.0 |
| AudioFM | 42.3 | 50.2 | 71.4 | 98.2 | 98.5 | 99.1 |
The AS-2M column shows each model's published score, then its score re-measured on the paper's own evaluation clips (16,103 that could still be downloaded). On that shared subset AudioFM's 50.2 is 1.0 above AudioMosaic and 0.2 above SSLAM. It got there with 1.68 million pretraining clips against the 1.9 to 2.0 million most others used, about 13% fewer. The fine-tuning gaps are a few tenths. The frozen-feature gaps of the probing table are where the method really shows.
Chapter 10
Test the encoder on events in time, on recordings it was never trained for, and as the model and data grow
Chapter 4 promised that the line-up would make each moment's vector carry its own moment. Clip-level tasks cannot check that promise. So the paper runs three harder tests.
Sound event detection asks not just "is there a dog?" but "from when to when?". The paper uses AudioSet Strong, whose labels mark the start and end of every event. A linear probe reads each time column's vector (one per 160 ms) and predicts which classes are active in it. The encoder stays frozen. Three scores grade it:
AudioFM leads on all three: 28.5 frame mAP, 24.9 PSDS1 and 45.9 PSDS2, against the runner-up AudioMosaic's 27.3, 24.0 and 44.5 (Figure 3). AudioMosaic is the natural comparison, because it is AudioFM without the local line-up.
Pick a test. Every number is from the paper.
Figure 3 (event detection; only AudioMosaic and AudioFM are labelled with numbers, so only they are drawn), Figure 5a–c (transfer; values without a printed label are read off the bars and marked approximate), Figure 4c–d (scaling, AudioSet-20K fine-tuning).
A new kind of home. RealDESED is a set of real domestic recordings: doors, dishes, keyboards, a vacuum cleaner. The paper takes each encoder with its AudioSet-trained classifier, maps 12 of RealDESED's classes onto AudioSet's labels, and tests with no training at all on RealDESED. Long recordings are cut into 10.24 s windows, and each class takes its highest score over the windows. AudioFM reaches 79.9 mAP; AudioMosaic 79.0, EAT 78.9, SSLAM 78.8, AudioMAE 76.2.
New voices. A probe that recognises emotion is trained on one dataset of actors, CREMA-D, and tested unchanged on another, RAVDESS, then on a third, TESS. AudioFM is first on CREMA-D to RAVDESS with 51.9 macro-F1. On CREMA-D to TESS it is second, at 35.4, well behind AudioMAE. Within TESS, a probe trained on a 26-year-old speaker and tested on a 64-year-old one scores 67.1, and 55.8 the other way round; AudioFM is first on both. Three firsts and one second, out of four.
A picture of sleep. The paper also embeds sleep-clinic recordings with the frozen encoder and squashes the 768-number vectors to two dimensions with t-SNE, a method that keeps nearby points nearby. Normal breathing and apnea fall into visibly separate regions, and the four apnea subtypes sit in a clearer order than with AudioMosaic. That is a picture, not a score, and the paper presents it as a reason for further study.
Does the method keep paying as you scale? Fine-tuned on AudioSet-20K, the encoder goes 39.8 (Small) → 42.3 (Base) → 43.1 mAP (Large). Training Base on 1%, 10% and 100% of the data gives 31.9 → 37.9 → 42.3 mAP. Each tenfold increase in data adds about 5 points: 6.0, then 4.4. The curve is still rising at the right edge. Both directions keep improving, with the usual shrinking returns.
The authors list the limits plainly. Pretraining used AudioSet only, and a limited range of model sizes. Every recording was squashed to one channel, so the encoder knows nothing about where a sound comes from. Robustness to very distorted channels or a change of microphone was not tested. And the medical results, while striking, come from two datasets.
Chapter 11
Put the whole method on one page, place it in the field, and take the core with you
AudioFM trains one Vision Transformer on unlabelled audio with two games at once. The global game makes two differently masked views of a clip agree, against every other clip in the batch, which teaches the scene. The local game makes the encoder's guess for each hidden patch pick out its own full-view teacher target from a line-up of other recordings at the same place, with a little credit for hidden neighbours one column away. The global loss alone gives every patch the same instruction. The line-up gives each patch its own. The result is a frozen encoder that is best or joint best on 10 of 12 tasks and leads on finding events in time.
| What | Value | Where |
|---|---|---|
| Input | 128 Mel bands × 1,024 frames, 10.24 s at 16 kHz | Section 4.1 |
| Patches | 8 × 64 = 512, each column 160 ms | Section 4.1, A.1 |
| Masks | 60% of time columns; 40% of frequency bands | Section 4.1 |
| Temperatures | τg = 0.1 global, τp = 0.2 local | A.1 |
| Neighbours | K = 1, weights 0.1 / 0.8 / 0.1, hidden neighbours only | Section 3.1, A.1 |
| Loss weight | λlocal = 1.0 | A.1, Figure 4a |
| Teacher momentum | 0.990 → 0.9995 | A.1 |
| Training | 1.68M clips, batch 6,144, 125,000 steps, TPU v4-128 | Section 4.1, Table 4 |
| Frozen probing | best or joint best on 10 of 12 tasks (bare) | Table 3 |
| Fine-tuning | 42.3 AS-20K, 50.2 AS-2M, 71.4 FSD50K | Table 2 |
| Event detection | 28.5 frame mAP, 24.9 PSDS1, 45.9 PSDS2 | Figure 3 |
pythondef train_step(clips, student, predictor, g_head, l_head, teacher, t_head, m): X1, X2 = log_mel(augment(clips)), log_mel(augment(clips)) # (B, 128, 1024) each v1, hid1 = time_mask(patchify(X1), ratio=0.6) # visible tokens, (B, 64, 8) hidden map v2, hid2 = freq_mask(patchify(X2), ratio=0.4) h1, h2 = student(v1), student(v2) # visible outputs + class token z1, z2 = normalize(g_head(pool(h1))), normalize(g_head(pool(h2))) # (B, 128) L_global = nt_xent(z1, z2, tau=0.1) p1 = normalize(l_head(predictor(h1, hid1, view=1))) # (B, 64, 8, 128) p2 = normalize(l_head(predictor(h2, hid2, view=2))) with torch.no_grad(): # stop-gradient q1 = normalize(t_head(teacher(patchify(X1)))) # full view: all 512 places q2 = normalize(t_head(teacher(patchify(X2)))) L_mld = 0.5 * (mld_loss(p1, q1, hid1) + mld_loss(p2, q2, hid2)) loss = L_global + 1.0 * L_mld loss.backward(); optimizer.step(); optimizer.zero_grad() ema_(teacher, student, m); ema_(t_head, l_head, m) # m: 0.990 -> 0.9995
| Method | Clip-level game | Patch-level target | Patch-level decoys |
|---|---|---|---|
| AudioMAE | none | the spectrogram (regression) | none |
| BEATs | none | discrete codes from a tokenizer | the other codes |
| SSAST, MAE-AST | none | raw spectrogram patches | other places, same clip |
| EAT, SSLAM | utterance-level regression | teacher features (regression) | none |
| AudioMosaic | contrast, time vs frequency masks | none | none |
| AudioFM | contrast, time vs frequency masks | full-view teacher features | other clips, same place, + hidden neighbours |
The same ideas run through other parts of the site. The global game is contrastive learning, the family of SimCLR and CLAP. The EMA teacher comes from BYOL and DINO. Predicting a teacher's features for hidden parts is the idea of JEPA. For the wider landscape of audio features, see audio representations and self-supervised speech. For a bioacoustics encoder that wins with supervised labels instead, read Perch 2.0 next to this one: the two papers disagree, usefully, about whether labels or self-supervision is the better bet.
Now press Present or Teach and explain, out loud and from memory, why the decoys come from other recordings at the same place and not from other places in the same recording. If you can, you own this paper. Then go back to the line-up, switch the decoys to "same recording", and find a hole the hum makes impossible to fill.