AudioFM

AudioFM learns to hear by playing a guessing game. It hides part of a recording, then picks the hidden patch out of a line-up of patches from other recordings. Added to whole-clip matching, that one game makes its frozen features the best or joint best on 10 of 12 tasks, from bird calls to heart murmurs.

Learn how one encoder can hold both the whole scene of a recording and its tiny moments, without a single label.

Play the student. A patch of the sound picture is hidden: look at what is still visible, then pick the patch that belongs in the hole. Then we build, piece by piece, the model that plays this game 6,144 recordings at a time.

You need what a vector and a dot product are and a rough idea of what a neural network does. We build the rest from zero.

Find the missing patch

Ready

Look at the outlined hole and the sound around it, then tap the patch you think belongs there.

The recordings are synthetic and drawn as small 4 × 4 patches so they are easy to compare. The masks (60% of time columns or 40% of frequency bands), the line-up of other recordings at the same spot, and the temperature of 0.2 are the paper's (Section 3.1, Section 4.1, Appendix A.1). The toy student fills the hole from its visible neighbours; the real student is a 12-layer Transformer trained on 1.68 million clips.

Chapter 0

Hear the scene, catch the moment

See why one recording asks for both the big picture and the tiny details, and why that is hard to learn without labels

You record ten seconds in a busy café. Rain taps on the window. People talk at the next table. Then, at about second six, somebody drops a spoon.

Now ask three questions about that one recording. Is this a café, a train station or a street? Did something drop, and when? Is the coffee machine's pump making the high whine that means it is about to fail? The first question is about the whole ten seconds. The second is about a fifth of a second. The third is about one thin stripe of pitch.

Other fields ask the same three kinds of question. A bird survey asks which species sang in an hour of forest sound. A doctor listens to a heartbeat and asks whether there is a faint whoosh between two beats: a murmur. A sleep clinic asks whether a patient stopped breathing for ten seconds. A call centre asks whether a voice sounds angry. Each answer lives at a different scale of the same sound.

One clip, three questions

A synthetic café recording, drawn as a sound picture: time runs left to right, pitch runs bottom to top, brighter means louder. Pick a question and see which part of the picture holds its answer.

An illustrative recording. The point it makes is the paper's own (Section 1): overall context informs scene recognition, while subtle temporal and spectral differences separate animal calls or patterns in body sounds.

A model that answers all three questions needs a description of the sound that keeps all three scales. In machine learning such a description is called a representation: a list of numbers (a vector) that a model computes from the input. A good representation puts similar sounds near each other and keeps the facts a later task needs. A representation of the whole clip is one vector. A representation of each moment is one vector per short slice of time.

Why not just train one model per question?

You could. You would need labels for each question: someone must listen to thousands of clips and write "café", or mark the exact second the spoon fell, or have a cardiologist grade each heartbeat. Labels like these are slow and expensive. Medical and animal labels also need experts.

Self-supervised learning (SSL) avoids that cost. The model invents its own task from unlabelled recordings, for example "hide part of the sound and guess it back". It learns a representation while it solves that task. Then you throw the task away and keep the representation. A model trained like this, then reused for many tasks, is called a foundation model. The paper's goal is an audio foundation model: one encoder whose features work for scenes, events, animals, hearts, lungs and voices.

Two families, two blind spots

Audio SSL has two main families. The first makes two altered copies of the same clip and trains the model so that both copies get the same clip vector. This is recording-level contrast. It is good at the scene question. But it only grades one pooled vector per clip, so nothing checks what each moment's vector says.

The second family hides parts of the sound and asks the model to fill them in. This is masked prediction. It grades every hidden piece, so it is good at moments and detail. But "fill in the blank" can be solved by copying nearby sound, and it does not ask whether the whole clip hangs together.

AudioFM, from Google Research and Google DeepMind, keeps the first family's clip-level contrast and adds a new kind of masked task. The model must not only guess the hidden piece. It must pick the true hidden piece out of a line-up of pieces from other recordings at the same spot. That is the game in the hero. The paper calls it masked latent discrimination, or MLD.

The idea in one line: grade the clip, and grade every hidden patch against other clips. Clip-level contrast keeps the scene. A line-up for each hidden patch forces every moment's vector to be specific enough to recognise. Together they keep both scales in one encoder.

The results back the idea. On 12 tasks across four domains, with the encoder frozen and only a linear layer trained on top, AudioFM is best or joint best on 10. It matches the best published score on AudioSet-2M fine-tuning (50.2 mAP) with about 13% fewer training clips. It also leads on all three measures of finding sound events in time. Here is the road, chapter by chapter. Each part of the hero is explained on the way.

  1. Sound as a picture: from a pressure wiggle to a grid of 512 patches, each 160 ms long.
  2. Agree with yourself: the clip-level contrast loss, built from a dot product and a temperature.
  3. Hide time, hide pitch: the two complementary masks behind the hero's first switch.
  4. What pooling forgets: why a clip-level loss alone cannot say when.
  5. Pick your patch out of a line-up: masked latent discrimination, the heart of the paper, and the hero's second switch.
  6. A teacher that sees everything: where the answer key comes from, and why it moves slowly.
  7. Lean on your neighbours: partial credit for nearby moments, and the rule that stops cheating.
  8. One training step, end to end: every tensor shape, the joint loss and the code.
  9. Freeze it and probe: how to score a representation fairly, and a strange collapse.
  10. Moments, new places, more data: event detection, transfer and scaling.
  11. Connections: the cheat sheet, the limits, and where this sits in the field.
Why does a representation trained only to make two copies of a clip agree tend to miss the "when did the spoon drop?" question?

Chapter 1

Sound as a picture

Turn a pressure wiggle into a log-Mel spectrogram, cut it into 512 patches, and follow their shapes into a Transformer

A microphone measures air pressure many times a second. AudioFM resamples every recording to 16,000 measurements per second, from one channel (mono). Its input window is 10.24 seconds long, so one clip is 16,000 × 10.24 = 163,840 numbers in a row. That row is the waveform.

The waveform is hard to read. A dropped spoon and a spoken vowel are both just wiggles. What our ears actually notice is which pitches are loud at each moment. So the first step turns the row into a picture of pitch over time.

From wiggle to picture

The same synthetic clip at four stages. Step through them, then drag the time cursor. In the last stage, tap any patch to see what it covers.

3.84 s

The 16 kHz rate, 128 Mel bands, 1,024 frames and 16 × 16 patches are the paper's (Section 4.1). The patch frequency ranges assume a Mel filter bank from 0 to 8,000 Hz, so they are approximate. The clip itself is synthetic.

Step 1: slice time, measure pitch

Cut the waveform into short overlapping slices, each about 25 ms long, starting a new slice every 10 ms. For each slice, ask: how much of a pure tone at 100 Hz is in here? At 101 Hz? At every frequency? To answer, multiply the slice point by point with a pure tone and add up the products. If the slice contains that tone, the products line up and the sum is large. If not, positive and negative products cancel and the sum is near zero. Doing this for every frequency at once is the Fourier transform.

The 25 ms slice length is a trade. A longer slice can tell 100 Hz from 101 Hz, but blurs when things happened. A shorter slice is sharp in time but blurry in pitch. You cannot have both; 25 ms is the usual compromise for speech and everyday sound.

For each slice, the transform measures how much energy sits at each frequency. One slice's result is a spectrum. Stack the spectra side by side and you get a spectrogram: time across, frequency up, brightness for energy. At one slice every 10 ms, 10.24 seconds gives 1,024 slices, which the paper calls frames.

Step 2: bend the pitch axis like an ear

Our ears are fussy about low pitches and relaxed about high ones. The step from 100 Hz to 200 Hz sounds huge. The step from 7,100 Hz to 7,200 Hz is barely noticeable. The Mel scale stretches the low end and squeezes the high end to match:

m
The pitch in Mel units: equal steps of m sound like equal steps of pitch.
f
The frequency in hertz (vibrations per second).

100 Hz → 150  ·  1,000 Hz → 1,000  ·  4,000 Hz → 2,146  ·  8,000 Hz → 2,840 Mel

So the octave from 4,000 to 8,000 Hz gets 694 Mel, while the first 1,000 Hz alone get 1,000. AudioFM uses 128 Mel bands, evenly spaced in Mel. Finally it takes the logarithm of each band's energy, because loudness also works in ratios: a sound ten times stronger sounds a fixed step louder, not ten times louder. The result is a log-Mel spectrogram: a grid of 128 bands by 1,024 frames, 131,072 numbers. Before training, every value is shifted and scaled by one fixed mean and spread (−4.268 and 4.569 for AudioSet, Table 4) so the numbers sit near zero.

Step 3: cut the picture into patches

Now the trick borrowed from image models. Cut the grid into non-overlapping squares of 16 bands by 16 frames. Each square is a patch. The frequency axis gives 128 / 16 = 8 patches, and the time axis gives 1,024 / 16 = 64. That is a grid of 8 × 64 = 512 patches.

16 frames × 10 ms = 160 ms per patch column  ·  64 columns × 160 ms = 10.24 s

Remember that 160 ms. It is the paper's unit of time for everything that follows. A dropped spoon lasts one or two columns. A spoken syllable lasts one to three. A held note can fill twenty.

We name each patch by its place: column t (time, 1 to 64) and row f (frequency band, 1 to 8). The paper writes the set of all places as P = {1, …, T} × {1, …, F} with T = 64 and F = 8. A place (t, f) means the same slice of time and pitch in every recording. That fact becomes important in Chapter 5.

Step 4: from patches to tokens

Each patch is 16 × 16 = 256 numbers. A learned linear layer maps those 256 numbers to a vector of D = 768 numbers. "Linear layer" sounds grand, but it is a table of weights, 256 rows by 768 columns. Each of the 768 outputs is a weighted sum of the 256 inputs, with its own column of weights. Training tunes the weights so that the sums become useful. The 768-number result is a token. A learned position embedding, one per place (t, f), is added to it, so the model knows where each token came from. One more learned token, the class token or CLS, joins the front of the list.

The tokens go into a Transformer: a stack of 12 identical layers. In each layer every token looks at every other token and updates itself with what it found. This is attention. Each token scores every other token with a dot product, a measure of how well two vectors point the same way. It turns the scores into weights that sum to 1, then adds in a weighted mix of the others. A patch of a dog's bark can pull in information from the bark's other patches, wherever they are in the grid. This exact design is the Vision Transformer, Base size (ViT-B): 12 layers, 768 numbers per token, 12 attention heads. Out comes one 768-number vector per patch, plus the class token. Those output vectors are the representation.

StageShapeWhat it is
Waveform163,840pressure, 16,000 per second for 10.24 s
Log-Mel128 × 1,024bands by 10 ms frames, normalised
Patches512 × 2568 bands × 64 columns of 16 × 16 squares
Tokens513 × 768linear map + position embedding, plus the class token
Encoder output513 × 768after 12 Transformer layers: one vector per patch

Here is the same thing as code. It is a sketch with the paper's sizes, not the authors' code.

pythonimport torch, torchaudio

wave = torchaudio.functional.resample(wave, sr, 16000)[:, :163840]  # mono, 10.24 s
fbank = torchaudio.compliance.kaldi.fbank(wave, num_mel_bins=128,
                                          frame_shift=10, frame_length=25)  # (1024, 128)
x = ((fbank - (-4.268)) / 4.569).T                         # (128, 1024): bands x frames
patches = x.unfold(0, 16, 16).unfold(1, 16, 16)            # (8, 64, 16, 16)
patches = patches.reshape(512, 256)                         # one row per patch
tokens = embed(patches) + pos_embed                          # (512, 768)
tokens = torch.cat([cls_token, tokens])                      # (513, 768) -> 12 ViT layers
A 10.24-second clip becomes a log-Mel spectrogram of 128 bands by 1,024 frames, cut into 16 × 16 patches. How many patches are there, and how long is one patch column?

Chapter 2

Agree with yourself

Build the clip-level contrast loss from a dot product, a softmax and a logarithm, and see why it needs other clips

We have no labels. So where does a learning signal come from? Here is the oldest trick in self-supervised learning. Take one clip and make two altered copies of it, called views. Whatever the clip "is" (rain in a café, a dog in a park) did not change. So the model should give both views the same vector.

AudioFM makes its two views in two steps. First it applies random changes to each copy of the waveform, following the recipe of AudioMosaic, an earlier model by other authors. Then it hides a different part of each view's spectrogram (Chapter 3). The model must see through both changes and recognise the same recording.

The first attempt fails: collapse

The obvious loss is "make the two vectors close". Measure the distance between them and push it to zero. Try it and something silly happens. The model learns to output the same vector for every clip it ever sees. Then every pair is identical, the distance is zero, and the loss is perfect. The vector says nothing. This failure is called collapse.

The fix is to also say what the vector must not match. A clip's view should be close to its own partner view and far from the views of every other clip in the batch. Then the constant answer scores terribly, because it is also close to all the wrong clips. This is contrastive learning: learn by contrasting the right match with wrong ones. The wrong ones are called negatives.

Piece 1: similarity is a dot product of unit vectors

First we need a number for "how alike are two vectors?". Take two vectors and multiply them entry by entry, then add up the products. This is the dot product. If both vectors are scaled to length 1 first (ℓ2 normalisation: divide each entry by the vector's length), the dot product is the cosine of the angle between them. It is 1 when they point the same way, 0 when they are at right angles, and −1 when they point opposite ways.

a = (0.6, 0.8), b = (0.8, 0.6):   a · b = 0.6 × 0.8 + 0.8 × 0.6 = 0.96 (both have length 1, so the angle is about 16°)

Normalising matters. Without it, the model could raise a similarity by making a vector longer, which is cheating. With it, all vectors live on the surface of a ball, and only their directions count. AudioFM projects each pooled clip vector down to 128 numbers and normalises it. The paper calls the result z.

Piece 2: turn scores into "which one is my partner?"

Now the anchor view has one score for each candidate: its partner and every negative. We want a probability for "candidate k is my partner". Probabilities must be positive and add up to 1, and a higher score should mean a higher probability. The simplest recipe that does all three: raise e to the power of each score, then divide by the total. This is the softmax.

One more knob. Before taking e to the power, divide every score by a small number τ, the temperature. Dividing by 0.1 multiplies every gap between scores by ten. That makes the softmax much more decisive. Here are three scores, a partner at 0.9 and two negatives at 0.3 and 0.1:

τe0.9/τ (partner)e0.3/τe0.1/τp(partner)loss = −log p
1.02.461.351.110.500.692
0.18,10320.12.720.9970.003

At τ = 1 the model is barely sure, even though the partner clearly scores highest. At τ = 0.1, the paper's value for the global loss, the same scores give near certainty.

Piece 3: the loss is minus the log of that probability

We want the partner's probability p to be 1. A convenient penalty is −log p. It is 0 when p = 1, and it climbs without limit as p falls toward 0. That makes confident mistakes very expensive. This penalty is the cross-entropy loss, the same one used to train every classifier. Here the "classes" are the candidates in the batch, and the right class is the partner view.

Put the three pieces together and you have the global loss AudioFM uses, called NT-Xent (normalised temperature-scaled cross-entropy):

ℓ
The loss for one anchor: minus the log of the probability it gives its own partner.
ziv
The anchor: clip i's normalised 128-number vector from view v (v is 1 or 2).
zi3−v
The partner: the same clip's other view (if v = 1, then 3 − v = 2).
τg
The global temperature, 0.1.
Σk ≠ anchor
The sum over every view in the batch except the anchor itself: the partner plus both views of every other clip.

Every one of the 2B views takes a turn as the anchor, and the losses are averaged. That is why it is called symmetric. With the paper's batch of B = 6,144 clips there are 12,288 views, so each anchor's sum has 12,287 terms: one partner and 12,286 negatives.

Pull partners, push strangers

Six clips, two views each, shown as directions on a circle (real vectors have 128 numbers; two are enough to see the idea). Tap a dot to make it the anchor. Press Train to run the loss. Then turn negatives off and train again.

0.10

A toy: two-number vectors and exact gradients of the NT-Xent loss above, six clips instead of 6,144. With negatives off, the loss is only "move the partners together", which shows collapse.

Train with negatives on and the pairs snap together while the six clips spread out around the circle. Turn negatives off and every dot slides to the same spot. The loss falls to zero while the vectors become useless. That is collapse, live.

One last detail. The pooled vector passes through a small extra network, the projection head, before the loss sees it. After training the head is thrown away and only the encoder is kept. The head absorbs the loss's demands (for example, "ignore loudness"), so the encoder's own features can stay richer than the loss strictly needs. This trick comes from SimCLR in vision, and every model in this paper uses some version of it.

What stops the contrast loss from collapsing to "give every clip the same vector"?

Chapter 3

Hide time, hide pitch

Build the two complementary masks, count what each view keeps, and hear the difference

The two views are mostly made different by hiding. Hiding does two jobs at once. It makes the views genuinely different, so agreeing is not trivial. It also makes training cheaper, because the student network only processes the patches it can see.

The question is what to hide. The cheapest choice is random patches here and there. But then almost every hidden patch has visible neighbours on all four sides, and a network can fill the gap by blending them, like smoothing over a scratch on a photo. It learns to blend, not to understand.

AudioFM, following AudioMosaic, hides in stripes, and the two views use different stripes:

Two ways to hide

One synthetic clip, masked two ways. Drag the two ratios. The read-out counts what each view keeps and how much they share, and gives the paper's score at that ratio where it measured one. Listen plays a view with its hidden parts silent.

0.60
3 of 8

The default ratios (60% of time columns, 40% of bands) and the measured scores at rt = 0.45, 0.60, 0.75 and rf = 0.25, 0.40, 0.75 are the paper's (Figure 6, AudioSet-20K linear probing; values other than the default 35.0 are read off the plot). With 8 bands, 40% is 3.2, so the slider steps through whole bands; how the paper rounds is not stated.

Counting what each view keeps

Let's count with the defaults. Sixty percent of 64 columns is 38.4; call it 38 hidden columns and 26 visible. View 1 keeps 26 × 8 = 208 patches and hides 304. Forty percent of 8 bands is 3.2; call it 3 hidden bands and 5 visible. View 2 keeps 5 × 64 = 320 patches and hides 192.

Now the interesting number: how many patches can both views see? A patch is visible in both only if its column survived view 1 and its band survived view 2:

(26 / 64) × (5 / 8) × 512 = 26 × 5 = 130 patches (about a quarter of the grid)

So the two views share only a quarter of their evidence. View 1 knows every pitch at a few moments. View 2 knows a few pitches at every moment. For their pooled vectors to agree, the encoder cannot just match the patches they share. It must work out what the whole recording is from two very different glimpses. That is the sense in which the masks are complementary.

How much to hide: the paper's sweep

The authors varied each ratio and measured how good the frozen features were on AudioSet-20K (Appendix A.2). Hide fewer columns, rt = 0.45, and the score drops from 35.0 to about 32.4 mAP. Hide more, 0.75, and it holds at about 35.0. For bands, hiding 25% gives about 34.0, and hiding 75% falls to about 32.5.

Why the asymmetry? Time is long: even with 75% of columns gone, 16 columns of complete spectrum remain, enough to tell what the scene is. Bands are few: hiding 6 of 8 bands can remove a sound's whole energy range. A bark that lives in bands 3 to 5 may vanish from view 2 entirely. Then the two views no longer show the same events, and "make them agree" asks for something impossible.

Why does AudioFM hide whole columns or whole bands instead of scattered single patches?

Chapter 4

What pooling forgets

Prove that a clip-level loss gives every patch the same instruction, and see what the masked-prediction family does about it

The global loss of Chapter 2 sees one vector per view: the pooled vector, the average of all the visible patch outputs. An average has a blind spot that you can test in a second. Move the bark from second 2 to second 8. Shuffle the columns in any order you like. The average does not change.

Move the bark, watch the average

A park recording with one dog bark. The middle row shows a toy vector for each time column (its eight band energies). The bars at the bottom show the average over all columns, the pooled vector. Drag the bark, or shuffle every column.

2.08 s

A toy: each column's vector is its band energies, and the background is perfectly steady so the average is exactly unchanged. A trained encoder's patch vectors also depend on context, but the pooled average is still blind to order in the same way.

The gradient every patch receives

Blindness to order is only half the problem. The deeper half is what the loss teaches each patch. Write the pooled vector as an average of the N visible patch outputs h1, …, hN:

h̄
The pooled vector of one view: the input to the projection head and then the global loss.
N
The number of visible patches in that view (208 or 320 with the default masks).
hn
The encoder's 768-number output for visible patch n.

The loss depends on the patches only through h̄. Training asks each output "which way should you move to lower the loss?". That direction is the gradient. By the chain rule, the gradient for patch n is the gradient for h̄ times how much h̄ changes when hn changes, which is 1/N:

∂L / ∂hn = (1/N) · ∂L / ∂h̄    the same for every n

Every visible patch receives the identical instruction, whether it holds the bark, the rain or silence. The loss says "nudge the average this way" and splits the nudge evenly. Nothing in the global loss rewards the bark's patch for describing the bark. Any set of patch vectors with the right average scores exactly the same.

The gap, in one sentence. A clip-level loss constrains the average of the patch vectors and leaves the patch vectors themselves free. Good patch vectors can still emerge, because the encoder computes them all with shared weights, but the loss never asks for them.

The paper measures the cost directly. Training with the global loss alone, then adding AudioFM's local loss, raises frame-level event detection on AudioSet Strong by 1.1 points of frame mAP and 2.1 points of PSDS2 (Figure 4b). Clip-level tagging on AudioSet-20K gains 0.6 mAP. The biggest gains are on the questions that need when.

The other family, and its weak spot

Masked prediction attacks the gap head on: it grades every hidden patch. Three versions came before AudioFM, and they differ in what they ask the model to predict:

TargetExampleWeak spot
The hidden spectrogram itselfAudioMAEspends effort on exact noise texture that no task cares about
A discrete code for the patchBEATsneeds a separate tokenizer, trained in rounds
A teacher network's vector for the patchEAT, SSLAM, M2D, BATa regression loss is happy with a blurry average guess

That last weak spot deserves a closer look, because it is what AudioFM's line-up fixes. Suppose the hole could hold a dog bark or a door slam, each equally likely given the visible context. A loss that measures squared distance to the target is lowest, on average, when the guess sits exactly between the two. So the model learns to predict a blend of bark and slam, a vector that matches neither. Averaged over many holes, the predictions drift toward bland middles.

A line-up does not reward the middle. If the true patch must out-score decoys from other recordings, a vague guess loses to any decoy that happens to sit near the middle. The prediction has to commit to something specific to this recording. Chapter 5 builds that line-up.

With only the clip-level loss, what gradient does the patch holding a dog bark receive, compared with a patch of background rain?

Chapter 5

Pick your patch out of a line-up

Build masked latent discrimination step by step, work its loss by hand, and see why the decoys come from other recordings

Here is the plan in plain words. For each hidden place in a view, the student makes a guess: a vector that says "this is what I think was here". A teacher, which saw the complete spectrogram, provides the true answer at that place, also as a vector. The student must then pick its own answer out of a line-up. The line-up is the teacher's answers at the same place in every other recording of the batch.

That is the game you played in the hero, with one difference: you compared small pictures, and the model compares 128-number vectors. Let's build it in four steps.

Step 1: the student's guess

The student encoder processes only the visible patches of a view. Its outputs go into a second, smaller network, the predictor, which puts them back on the 8 × 64 grid and fills every hidden place with a guess. Chapter 6 opens the predictor up. For now, what matters is its output. A small network called the local projection head maps each guess to 128 numbers and normalises it to length 1. The paper names the student's guess for recording i, view v, place (t, f):

pi,t,fv  ·  128 numbers, length 1  ·  one for every hidden place

Step 2: the answer key

The teacher is a second copy of the encoder that sees the same view without any hiding: all 512 patches. Its output at each place goes through its own copy of the local projection head and is normalised. That gives the target qi,t,fv, also 128 numbers of length 1. No gradient flows back into the teacher. It only supplies answers.

Step 3: the line-up

For one hidden place (t, f) of recording i, the candidates are the teacher targets at that same place (t, f) in every recording of the batch, j = 1 to B. Exactly one of them, j = i, is the right answer. The score of candidate j is the dot product pi,t,f · qj,t,f.

Step 4: the loss

From here it is Chapter 2 again: softmax over the scores, with a temperature, and minus the log of the right answer's probability. This is Equation 1 of the paper:

ℓiv
The loss for one hidden place (t, f) of recording i in view v.
pi,t,f
The student's guess at that place: 128 numbers, length 1.
qi,t,f
The teacher's answer at that place, from the full view of the same recording. The right candidate.
qj,t,f
The teacher's answer at the same place in recording j. For j ≠ i these are the decoys.
τp
The local temperature, 0.2: a little softer than the global 0.1.
Σj=1B
A sum over every recording in the batch, the right one included. Only place (t, f) appears; no other place of any recording is a candidate.

Work it by hand with a line-up of four. Say the guess scores 0.80 against its own target and 0.55, 0.30 and 0.10 against three decoys. Divide by τp = 0.2 and raise e to each power:

e4.00 = 54.60  ·  e2.75 = 15.64  ·  e1.50 = 4.48  ·  e0.50 = 1.65  →  total 76.37

p = 54.60 / 76.37 = 0.715    loss = −log 0.715 = 0.336

The 0.55 decoy is doing real damage: on its own it takes 20% of the probability. To lower the loss, the student must make its guess more specific, so that the decoy's score falls. That is exactly the pressure the regression loss of Chapter 4 lacked. In the paper the line-up is not four but 6,144 recordings long.

Run the line-up for a whole batch

A toy batch of recordings, each time-masked. Tap any hidden place on the top recording to choose (t, f). The grid below has one row per recording's guess and one column per target at that same place. Every row is one line-up, and its right answer sits on the diagonal. Then switch where the decoys come from.

6
0.20

A toy. Each place's vector is its 4 × 4 patch of energies plus a short summary of its whole recording (how loud, and how changeable, each band is), centred and normalised, the way a Transformer's patch outputs carry context; the toy student guesses a hidden patch by blending the nearest visible columns; and the whole toy batch shares one mask, where the paper draws a mask per recording. The line-up construction (same place, other recordings), Equation 1 and τp = 0.2 are the paper's. Recording 3 is a steady engine hum on purpose: switch to "same recording" to see why.

Why decoys from other recordings at the same place?

This is the design choice that separates AudioFM from earlier local contrast methods, so it is worth slowing down. Older methods (SSAST, MAE-AST in audio; SPIRAL and ConMIM elsewhere) took their decoys from other places in the same recording. That sounds natural: "is this the patch from second 3, or from second 7?". Try it on the hum in the device.

A steady engine hum sounds the same at second 3 and at second 7. Their patches are nearly identical. If second 7 is a decoy, the loss demands that the guess for second 3 score low against second 7. The model is punished for noticing that two moments sound the same, which is true. A decoy that is secretly a right answer is called a false negative. The paper names the problem directly: its candidate construction avoids requiring acoustically similar regions within a recording to be distinguished solely because they occupy different positions.

Decoys from other recordings at the same place fix both halves of the problem:

Why the line-up gives each patch its own job. In Chapter 4 every patch received the same gradient. Here each hidden place has its own line-up, with its own right answer, so each place's guess gets its own instruction. Because the guesses are built from the encoder's patch outputs, those outputs must now carry what is specific to their own moment and pitch.

The scale is large. With the default masks there are about 304 hidden places in view 1 and 192 in view 2. Every one of the 6,144 recordings plays the game at every hidden place, so one training step plays about 3 million line-ups, each 6,144 candidates long. In code, each place's line-ups are one matrix multiply: all B guesses against all B targets, a B × B grid of scores whose diagonal holds the right answers. That is the grid in the device.

Why does AudioFM take its decoys from other recordings at the same place, rather than from other places in the same recording?

Chapter 6

A teacher that sees everything

Build the answer key from a slow copy of the student, and open the predictor that turns visible patches into guesses

The line-up needs an answer key: a target vector for every place of every recording. Where can good targets come from, with no labels? They must be meaningful, available for free, and they should improve as training goes on. A fixed, frozen network would not improve. The raw spectrogram is free but full of detail nobody needs (Chapter 4).

The answer used by AudioFM, and by BYOL, DINO, data2vec and many others, is surprising at first: use the student itself, but a slower, steadier version of it. This copy is the teacher.

The exponential moving average

The teacher has the same architecture as the student, with its own weights θ̄. After every training step, each teacher weight moves a small fraction of the way toward the matching student weight:

θ̄
The teacher's weights (encoder and local projection head).
θ
The student's weights right after this step's update.
m
The momentum: how much of its old self the teacher keeps each step. AudioFM raises it from 0.990 to 0.9995 during training.

Unroll the rule and you see what it means. The teacher is a weighted average of all past students: the newest gets weight (1 − m), the one before m(1 − m), then m2(1 − m), and so on. Old students fade by a factor m per step. This is an exponential moving average (EMA). A handy measure is the half-life: after how many steps does a student's influence halve? Solve mn = 1/2:

n = ln(0.5) / ln(m)  ·  m = 0.990 → 69 steps  ·  m = 0.9995 → 1,386 steps

Early in training the student changes fast and is mostly wrong, so a teacher that follows it within about 70 steps keeps the answers current. Late in training the student is good and its changes are small, noisy refinements. A teacher averaging over about 1,400 steps smooths out that noise and gives steady targets.

Follow the student, slowly

Two weights of a toy student wander as training steps go by: purple. The green teacher follows by the rule above. Drag the momentum and press Run.

0.990

A toy random walk with a slow drift, 3,000 steps. The momentum values 0.990 and 0.9995 are the paper's start and end (Appendix A.1); the others are there to compare.

Two rules that keep the teacher honest

The teacher sees the full view. For each student view, the teacher gets the same augmented spectrogram with nothing hidden: all 512 patches. So its vector for a place was computed with full knowledge of the whole clip. When the student, from partial evidence, matches that vector, it has inferred what full knowledge would say.

No gradient flows into the teacher. This is called stop-gradient. Without it, the loss could be lowered by moving the targets toward the guesses instead of the other way round, and both would drift toward a shared trivial answer. With it, the targets are fixed for this step, and only the student moves.

Inside the predictor

Now the missing piece of Chapter 5: how visible patch outputs become guesses at hidden places. The predictor is a small Transformer that works on the whole grid. Follow one view through it:

StepShapeWhat happens, and why
Student encoder209 × 768the 208 visible patches of view 1 plus the class token
Restore the grid512 × 768drop the class token, put each visible output back at its (t, f), fill the 304 hidden places with one shared learned mask token
Narrow512 × 384a linear map to Dpred = 384: a deliberate bottleneck
Say where, say which view512 × 384add fixed sine-cosine positions (192 numbers for time, 192 for frequency) and a learned embedding that says "time-masked" or "frequency-masked"
4 Transformer blocks512 × 384full attention across time and frequency, 6 heads: each hidden place gathers evidence from every visible patch
Widen512 × 768a linear map back to 768
Local projection head512 × 128a two-layer network 768 → 512 → 128 with LayerNorm, then length 1. Only the 304 hidden places enter the loss

Three of these choices carry real reasons. The bottleneck (384 instead of 768) and the small depth (4 blocks instead of 12) keep the predictor weak on purpose. A strong predictor could do the inferring by itself and leave the encoder lazy. A weak one forces the encoder's outputs to already contain what the guesses need.

The view embedding exists because the two views pose different puzzles. A missing column must be inferred from the moments around it. A missing band must be inferred from the pitches around it. Telling the predictor which puzzle it faces lets one shared network solve both. The fixed positions tell every hidden place exactly where it sits, so its guess is about that place.

After pretraining, the predictor, both projection heads and the teacher are all thrown away. Only the student encoder is kept: 86 million weights out of the 94 million trained.

The teacher's momentum rises from 0.990 to 0.9995 during training. What does that change in practice?

Chapter 7

Lean on your neighbours

Give partial credit to the moments just before and after, and block the shortcut that would make it cheating

Most sounds last longer than 160 ms. A vowel, a hum or a bird's trill at column 20 is often still there at column 21. So if the student's guess for (20, f) happens to look like the teacher's answer at (21, f), that is not a mistake. It is a sensible near-miss. Chapter 5's loss gives it no credit at all.

AudioFM adds that credit. The same guess pi,t,f also plays a line-up against the targets at (t−1, f) and (t+1, f): one column earlier and one column later, at the same pitch. Each of those line-ups uses its own decoys, from other recordings at that neighbouring place. The three losses are mixed with fixed weights that keep the guess's own place in charge:

weights for offsets −1, 0, +1  =  0.1, 0.8, 0.1 (radius K = 1, a window of W = 3 columns)

These weights are a bell curve sampled at three points and scaled to add up to 1 (Appendix A.1). Their spread works out to about half a column, roughly 80 ms. So "neighbour" here means the very next moment, not the general area.

The shortcut, and the rule that blocks it

There is a trap. Suppose column 20 is hidden but column 21 is visible. Then the student saw column 21's patch directly. To score on the (21, f) line-up, the predictor does not need to infer anything: it can copy the visible patch's features into the guess. The loss would reward copying, which is the shortcut Chapter 3's stripes were designed to remove.

So AudioFM only counts a neighbour if it is inside the grid and also hidden in the student's view (Equation 2). The guess's own place always counts, because it is hidden by definition:

Vi,t,f
The valid offsets for this hidden place: the ones that will count.
δ
An offset in columns: −1 is one column earlier (160 ms), +1 one column later.
Miv
The set of hidden places of recording i in view v. A place off the edge of the grid is not in it.

Then the surviving weights are rescaled to sum to 1, keeping their proportions, and the neighbourhood loss is their weighted sum (Equation 3):

ℓi,t,f
The final loss for one hidden place.
Σδ ∈ V
A sum over the valid offsets only.
w̃δ
The weight for offset δ, rescaled over the valid offsets so the weights add up to 1.
ℓ(p ; (t+δ, f))
Chapter 5's line-up loss, but with the right answer and the decoys all taken at place (t + δ, f).

A worked case. Column 20 is hidden. Column 19 is visible, so offset −1 is dropped. Column 21 is hidden, so +1 stays. The surviving weights 0.8 and 0.1 add to 0.9, so they become 0.8 / 0.9 and 0.1 / 0.9:

ℓ = 0.889 · ℓ(own place) + 0.111 · ℓ(one column later) (the paper's 8/9 and 1/9)

If neither neighbour is hidden, the own place gets weight 1 and the loss is plain Equation 1. The two views behave very differently here, and the device shows why. In the time-masked view, a neighbour counts only if its whole column was also hidden. In the frequency-masked view, a hidden band is hidden at every moment, so both neighbours always count, except at the first and last column.

Which neighbours count?

Pick a view and a radius, then tap any hidden (hatched) place. Its window lights up: green neighbours count, red ones were visible to the student and are dropped. The bars show the weights before and after rescaling, and the paper's measured gains for that radius.

The weights (K = 1: 0.1, 0.8, 0.1; K = 2: 0.03, 0.20, 0.54, 0.20, 0.03), the mask-aware rule and the gains over global-only training are the paper's (Appendix A.1, Figure 4b). Only the K = 1 gains are printed in the figure; the K = 0 and K = 2 gains are read off its bars and are approximate.

How wide should the window be?

The paper compared three radii, each against a model trained with the global loss only (Figure 4b). Center-only, K = 0, already helps. K = 1 helps most on every measure: +0.6 mAP on AudioSet-20K tagging, +1.1 frame mAP, +0.8 PSDS1 and +2.1 PSDS2 on event detection. K = 2 is worse than K = 0 and on some measures worse than no local loss at all.

The K = 2 weights explain why. They put only 54% on the own place and spread 46% over four neighbours up to 320 ms away. Two columns away, a sound has often changed. Now the guess is pulled toward several different targets at once, which is the blur problem of Chapter 4 returning through the back door. One column of tolerance helps; two columns smear the moment.

In the time-masked view, column 20 is hidden, column 19 is visible and column 21 is hidden. With K = 1, what weights does the loss use?

Chapter 8

One training step, end to end

Follow a batch through both branches with every shape, add the two losses, and read the whole thing as code

Every piece is now on the table. Let's run one training step from start to finish, with the paper's numbers. A step takes a batch of B = 6,144 unlabelled clips from AudioSet. For each clip:

  1. Two views. Apply random waveform changes twice, independently, and compute two log-Mel spectrograms X1 and X2, each 128 × 1,024.
  2. Patches. Cut each into 512 patches of 16 × 16 and turn them into 768-number tokens.
  3. Hide. View 1 hides 60% of its time columns (about 304 patches). View 2 hides 40% of its bands (about 192 patches). The hidden sets are drawn separately for each clip.
  4. Student. The student encoder runs on the visible tokens of each view, plus the class token.
  5. Global branch. Pool each view's outputs, project to 128 numbers, normalise: z1 and z2. Run NT-Xent with τg = 0.1 over all 12,288 views of the batch.
  6. Local branch. The predictor fills the hidden places; the local head gives a normalised guess p at each.
  7. Teacher. The EMA teacher runs on the full X1 and X2, 512 patches each, and its local head gives a target q at every place. No gradient.
  8. Line-ups. For every hidden place, run the line-up of Equation 1 against its own place and its valid neighbours (Equations 2 and 3), with τp = 0.2. Average over all hidden places of all clips in each view, then average the two views equally. That is LMLD.
  9. Add. L = Lglobal + λlocal · LMLD, with λlocal = 1.0.
  10. Update. AdamW takes one step on the student encoder, the predictor and both student heads. Then the teacher takes its EMA step toward the student.

Trace one clip through a step

Every box is one stage of the step. Tap a box to see its shape, what it does and whether it is trained. Press Play to watch a clip flow through in order. Then pick a local-loss weight to see the paper's ablation.

Shapes use the default masks with the rounding of Chapter 3 (208 and 320 visible patches). The λ gains are the paper's (Figure 4a: AudioSet-20K probing mAP over global-only training, with K = 1).

Balancing the two losses

Why add the losses with weight 1? Here is a neat way to see that 1 is a natural starting point. At the very start of training, the vectors are close to random, so every candidate scores about the same. Then the softmax gives each candidate equal probability, and the loss is minus the log of 1 over the number of candidates:

Lglobal ≈ ln 12,287 = 9.42  ·  LMLD ≈ ln 6,144 = 8.72

The two losses start at almost the same size, so neither drowns the other. The paper then tested the weight (Figure 4a). Every positive weight beats training without the local loss: λ = 0.5 adds 0.2 mAP, λ = 1.0 adds 0.6, and λ = 2.0 adds 0.3. Too little local loss and the patches are under-taught. Too much and the local game starts to crowd out the clip-level structure that tagging tasks rely on.

The budget

Pretraining ran on a Cloud TPU v4-128 slice (64 chips) for 125,000 steps at a batch of 6,144 clips. That is 6,144 × 125,000 = 768 million clip draws, about 457 passes over the 1.68 million clips the authors could still download from AudioSet. The optimiser is AdamW with a peak learning rate of 6 × 10−4. The rate warms up over 2,500 steps, then falls along a half cosine to 10−6. Weight decay is 0.05, and gradients are clipped at 1.0 (Table 4).

Here is the local loss as code, written for clarity rather than speed. The real thing batches the loops into tensor operations.

pythonimport torch, torch.nn.functional as nnf

def mld_loss(p, q, hidden, w=(0.1, 0.8, 0.1), tau=0.2):
    """p, q: (B, T, F, 128) student guesses and teacher targets, each length 1.
    hidden: (B, T, F) True where the place is hidden in this clip's student view."""
    B, T, F, _ = p.shape
    K = len(w) // 2
    right = torch.arange(B)                       # row i's right answer is column i
    total, n = 0.0, 0
    for t in range(T):
        for f in range(F):
            anchors = hidden[:, t, f]               # clips that hid (t, f)
            if not anchors.any(): continue
            ls, ws = [], []
            for k, d in enumerate(range(-K, K + 1)):
                s = t + d
                if s < 0 or s >= T: continue       # off the grid
                logits = p[:, t, f] @ q[:, s, f].T / tau  # (B, B) line-ups at place (s, f)
                ce = nnf.cross_entropy(logits, right, reduction="none")
                ok = hidden[:, s, f].float()        # mask-aware: the neighbour must be hidden too
                ls.append(ce * ok); ws.append(w[k] * ok)
            W = torch.stack(ws)
            L = (torch.stack(ls) * W).sum(0) / W.sum(0).clamp_min(1e-8)  # rescaled weights
            total = total + L[anchors].sum(); n += int(anchors.sum())
    return total / n

# one step (sketch)
loss = nt_xent(z1, z2, tau=0.1) + 1.0 * 0.5 * (mld_loss(p1, q1, hid1) + mld_loss(p2, q2, hid2))
loss.backward(); opt.step(); ema_update(teacher, student, m)
At the very start of training, roughly how large is each loss, and why does that make λ = 1 a natural choice?

Chapter 9

Freeze it and probe

Learn the fair way to score a representation, read the paper's twelve tasks, and explain a strange collapse

After pretraining we have an encoder that turns any clip into vectors. How good are those vectors? There are two common tests, and they measure different things.

Fine-tuning adds a small output layer for a task and then trains everything, the encoder included, on that task's labels. It shows how good a starting point the encoder is. But scores also depend on the fine-tuning recipe: the data augmentation, the learning rates, how long you train. Two papers with different recipes are hard to compare.

Linear probing freezes the encoder completely and trains only one linear layer on top: a weighted sum of the vector's numbers for each class. A linear layer cannot build new features. It can only read what is already there. So a probe asks a sharper question: how easily can a task's answer be read off these vectors? The paper ran every encoder through the same probing pipeline: the same data splits, labels, metric and probe recipe, each averaged over three random seeds.

Bare and LayerNorm probes

The paper probes two ways. The bare probe reads the raw features. The flexible probe first applies a LayerNorm. For each clip's vector, it subtracts the mean of the 768 numbers and divides by their spread. Then it applies a learned scale and shift to each number. That puts every feature on a similar scale before the linear layer sees it.

That sounds like a small detail. Look at what it does to two strong baselines on AudioSet-20K:

EAT: bare 1.1 mAP → LayerNorm 18.7  ·  SSLAM: bare 0.9 → 18.2  ·  AudioFM: bare 31.2 → 35.0

A score near 1 mAP on 527 classes is close to random guessing. These are good encoders: with fine-tuning, both reach over 39 mAP. Their raw features simply do not work with a fixed, simple probe recipe. The paper reports this without explaining it. The most likely reason is scale: if a few feature numbers are hundreds of times larger than the rest, one learning rate cannot suit them all. Steps that are right for the big numbers are far too small for the informative small ones. Try it.

Train a probe here

200 toy clips of two classes. The useful feature is small; a useless one is huge. Both probes get the same recipe: the same learning rate and the same number of steps. Press Train, then drag the size of the useless feature.

×316

A toy logistic-regression probe, plain gradient descent with a fixed step size and 300 steps. It shows one mechanism that can sink a bare probe; the paper does not say what goes wrong inside EAT and SSLAM.

So a bare probe rewards features that are well-scaled out of the box, which is exactly what a downstream user wants from a ready-made encoder. AudioFM's raw features hold up. Under bare probing it is best or joint best on 10 of the 12 tasks. Explore the full table:

Twelve tasks, six frozen encoders

Each group of bars is one task; each bar is one encoder with its weights frozen. Pick a domain and a probe.

All numbers are the paper's Table 3: means over three seeds, in percent. A-MAE is AudioMAE; AMos is AudioMosaic.

Reading the scores

The table uses four measures. Each answers a slightly different question, so it helps to build them from zero.

Two medical results stand out. On sleep apnea subtypes (PSG-Audio) AudioFM's bare features beat the next best by 3.2 points of macro-F1, and on heart murmurs (CirCor) by 4.7. Neither task was anywhere in the pretraining data, which was YouTube audio.

Fine-tuning: close at the top

With full fine-tuning (Table 2) AudioFM also leads, but the margins are small, as they usually are at the top of these benchmarks:

ModelAS-20KAS-2MFSD50KESC-50SPC-2SPC-1
BEATsiter338.048.0 / 46.757.695.698.397.7
AudioMAE35.547.3 / 46.557.394.198.396.9
EAT40.548.6 / 48.867.595.998.3–
SSLAM39.350.2 / 50.066.696.298.198.8
AudioMosaic41.850.2 / 49.270.597.598.499.0
AudioFM42.350.271.498.298.599.1

The AS-2M column shows each model's published score, then its score re-measured on the paper's own evaluation clips (16,103 that could still be downloaded). On that shared subset AudioFM's 50.2 is 1.0 above AudioMosaic and 0.2 above SSLAM. It got there with 1.68 million pretraining clips against the 1.9 to 2.0 million most others used, about 13% fewer. The fine-tuning gaps are a few tenths. The frozen-feature gaps of the probing table are where the method really shows.

EAT scores 1.1 mAP with a bare probe on AudioSet-20K but 18.7 with a LayerNorm probe. What does that tell you?

Chapter 10

Moments, new places, more data

Test the encoder on events in time, on recordings it was never trained for, and as the model and data grow

Chapter 4 promised that the line-up would make each moment's vector carry its own moment. Clip-level tasks cannot check that promise. So the paper runs three harder tests.

Test 1: find events in time

Sound event detection asks not just "is there a dog?" but "from when to when?". The paper uses AudioSet Strong, whose labels mark the start and end of every event. A linear probe reads each time column's vector (one per 160 ms) and predicts which classes are active in it. The encoder stays frozen. Three scores grade it:

AudioFM leads on all three: 28.5 frame mAP, 24.9 PSDS1 and 45.9 PSDS2, against the runner-up AudioMosaic's 27.3, 24.0 and 44.5 (Figure 3). AudioMosaic is the natural comparison, because it is AudioFM without the local line-up.

Three harder tests

Pick a test. Every number is from the paper.

Figure 3 (event detection; only AudioMosaic and AudioFM are labelled with numbers, so only they are drawn), Figure 5a–c (transfer; values without a printed label are read off the bars and marked approximate), Figure 4c–d (scaling, AudioSet-20K fine-tuning).

Test 2: recordings from somewhere else

A new kind of home. RealDESED is a set of real domestic recordings: doors, dishes, keyboards, a vacuum cleaner. The paper takes each encoder with its AudioSet-trained classifier, maps 12 of RealDESED's classes onto AudioSet's labels, and tests with no training at all on RealDESED. Long recordings are cut into 10.24 s windows, and each class takes its highest score over the windows. AudioFM reaches 79.9 mAP; AudioMosaic 79.0, EAT 78.9, SSLAM 78.8, AudioMAE 76.2.

New voices. A probe that recognises emotion is trained on one dataset of actors, CREMA-D, and tested unchanged on another, RAVDESS, then on a third, TESS. AudioFM is first on CREMA-D to RAVDESS with 51.9 macro-F1. On CREMA-D to TESS it is second, at 35.4, well behind AudioMAE. Within TESS, a probe trained on a 26-year-old speaker and tested on a 64-year-old one scores 67.1, and 55.8 the other way round; AudioFM is first on both. Three firsts and one second, out of four.

A picture of sleep. The paper also embeds sleep-clinic recordings with the frozen encoder and squashes the 768-number vectors to two dimensions with t-SNE, a method that keeps nearby points nearby. Normal breathing and apnea fall into visibly separate regions, and the four apnea subtypes sit in a clearer order than with AudioMosaic. That is a picture, not a score, and the paper presents it as a reason for further study.

Test 3: bigger model, more data

Does the method keep paying as you scale? Fine-tuned on AudioSet-20K, the encoder goes 39.8 (Small) → 42.3 (Base) → 43.1 mAP (Large). Training Base on 1%, 10% and 100% of the data gives 31.9 → 37.9 → 42.3 mAP. Each tenfold increase in data adds about 5 points: 6.0, then 4.4. The curve is still rising at the right edge. Both directions keep improving, with the usual shrinking returns.

What the paper does not cover

The authors list the limits plainly. Pretraining used AudioSet only, and a limited range of model sizes. Every recording was squashed to one channel, so the encoder knows nothing about where a sound comes from. Robustness to very distorted channels or a change of microphone was not tested. And the medical results, while striking, come from two datasets.

Why is AudioMosaic the most telling comparison for AudioFM's event-detection gains?

Chapter 11

Connections

Put the whole method on one page, place it in the field, and take the core with you

AudioFM trains one Vision Transformer on unlabelled audio with two games at once. The global game makes two differently masked views of a clip agree, against every other clip in the batch, which teaches the scene. The local game makes the encoder's guess for each hidden patch pick out its own full-view teacher target from a line-up of other recordings at the same place, with a little credit for hidden neighbours one column away. The global loss alone gives every patch the same instruction. The line-up gives each patch its own. The result is a frozen encoder that is best or joint best on 10 of 12 tasks and leads on finding events in time.

The numbers to remember

WhatValueWhere
Input128 Mel bands × 1,024 frames, 10.24 s at 16 kHzSection 4.1
Patches8 × 64 = 512, each column 160 msSection 4.1, A.1
Masks60% of time columns; 40% of frequency bandsSection 4.1
Temperaturesτg = 0.1 global, τp = 0.2 localA.1
NeighboursK = 1, weights 0.1 / 0.8 / 0.1, hidden neighbours onlySection 3.1, A.1
Loss weightλlocal = 1.0A.1, Figure 4a
Teacher momentum0.990 → 0.9995A.1
Training1.68M clips, batch 6,144, 125,000 steps, TPU v4-128Section 4.1, Table 4
Frozen probingbest or joint best on 10 of 12 tasks (bare)Table 3
Fine-tuning42.3 AS-20K, 50.2 AS-2M, 71.4 FSD50KTable 2
Event detection28.5 frame mAP, 24.9 PSDS1, 45.9 PSDS2Figure 3

The cheat sheet

The whole step, in code

pythondef train_step(clips, student, predictor, g_head, l_head, teacher, t_head, m):
    X1, X2 = log_mel(augment(clips)), log_mel(augment(clips))      # (B, 128, 1024) each
    v1, hid1 = time_mask(patchify(X1), ratio=0.6)                  # visible tokens, (B, 64, 8) hidden map
    v2, hid2 = freq_mask(patchify(X2), ratio=0.4)
    h1, h2 = student(v1), student(v2)                               # visible outputs + class token

    z1, z2 = normalize(g_head(pool(h1))), normalize(g_head(pool(h2)))  # (B, 128)
    L_global = nt_xent(z1, z2, tau=0.1)

    p1 = normalize(l_head(predictor(h1, hid1, view=1)))            # (B, 64, 8, 128)
    p2 = normalize(l_head(predictor(h2, hid2, view=2)))
    with torch.no_grad():                                           # stop-gradient
        q1 = normalize(t_head(teacher(patchify(X1))))               # full view: all 512 places
        q2 = normalize(t_head(teacher(patchify(X2))))
    L_mld = 0.5 * (mld_loss(p1, q1, hid1) + mld_loss(p2, q2, hid2))

    loss = L_global + 1.0 * L_mld
    loss.backward(); optimizer.step(); optimizer.zero_grad()
    ema_(teacher, student, m); ema_(t_head, l_head, m)             # m: 0.990 -> 0.9995

Where it sits in the field

MethodClip-level gamePatch-level targetPatch-level decoys
AudioMAEnonethe spectrogram (regression)none
BEATsnonediscrete codes from a tokenizerthe other codes
SSAST, MAE-ASTnoneraw spectrogram patchesother places, same clip
EAT, SSLAMutterance-level regressionteacher features (regression)none
AudioMosaiccontrast, time vs frequency masksnonenone
AudioFMcontrast, time vs frequency masksfull-view teacher featuresother clips, same place, + hidden neighbours

The same ideas run through other parts of the site. The global game is contrastive learning, the family of SimCLR and CLAP. The EMA teacher comes from BYOL and DINO. Predicting a teacher's features for hidden parts is the idea of JEPA. For the wider landscape of audio features, see audio representations and self-supervised speech. For a bioacoustics encoder that wins with supervised labels instead, read Perch 2.0 next to this one: the two papers disagree, usefully, about whether labels or self-supervision is the better bet.

The takeaway. A loss that grades only the whole clip leaves every patch free, and a loss that regresses each patch rewards a blur. AudioFM grades every hidden patch by asking it to recognise itself in a line-up of other recordings at the same place. That one change makes each moment's vector specific, keeps the scene intact, and needs no labels.

Now press Present or Teach and explain, out loud and from memory, why the decoys come from other recordings at the same place and not from other places in the same recording. If you can, you own this paper. Then go back to the line-up, switch the decoys to "same recording", and find a hole the hum makes impossible to fill.

Based on "AudioFM: Towards General Purpose Audio Representation Learning" by Yamin Li, Ruidong Zhang, Anran Wang, Eduardo Fonseca, Dan Ellis and colleagues (Google Research, Google DeepMind, Vanderbilt University and the University of Washington, 2026)
Read the paper · Back to Veanors