Attention Pooling Heads

A frozen audio model turns a 10-second clip into 100 frame vectors. Your detector needs one vector. Average the frames, and a 0.3-second glass break is 3% of the mix. An attention-pooling head learns where in the clip to look, and averages there.

Learn why mean pooling drowns short events, how a learned weighted average fixes it, why its gradient teaches it to focus, and how to build, train and test one on cached embeddings in about 25 lines of PyTorch.

Set the event length and press Train all three. Three heads learn from 800 synthetic clips, in your browser. Switch the view to see where each head looks. Make the event shorter and watch mean pooling fall toward a coin flip.

You need vectors, the dot product, and the idea of a trained classifier. We build the rest from zero.

Find the glass break in the clip

Ready
0.3 s

Synthetic data, illustrative numbers. Each clip is 100 frames (10 s at 10 frames a second) of 16-dimensional embeddings: noise, plus an offset shared by the whole clip. Positive clips add one fixed "glass" direction to the event's frames. The heads train here with Adam and are scored on 400 new clips. The plot on the right starts with four event lengths; each run you start adds or replaces a point.

Chapter 0

The needle in the clip

See why a detector needs one vector per clip, and measure how badly plain averaging hides a short sound

You are building a glass-break detector for a home camera. You have a few hundred labelled 10-second clips. Each label says only "glass broke somewhere in this clip" or "it did not". Nobody marked the second.

You also have a strong audio model that you will not train. A frozen encoder is a pretrained network whose weights you keep fixed. Many audio encoders cut a clip into short frames (here, one every 100 ms) and output one vector per frame. Ten seconds gives 100 frame vectors. The HTSAT audio tower inside LAION-CLAP, for example, produces 768-dimensional features before its projection layer LAION-CLAP · §3.3.

A detector, though, is a classifier. A classifier takes one vector of a fixed size. Clips come in different lengths: 3 seconds, 10 seconds, a minute. So something must squeeze "any number of frame vectors" into "one vector". That squeeze is pooling, and the small network that pools and then classifies is a pooling head.

The obvious squeeze, and its flaw

The simplest pool is the average. Mean pooling adds the T frame vectors and divides by T. It works for any length. It does not care where the event happens. The CLAP paper notes this is the usual way to handle long audio: average the per-frame or per-chunk embeddings LAION-CLAP · §3.4.

Now count. A glass break lasts about 0.3 seconds: 3 frames out of 100. The other 97 frames are room noise, a TV, a fridge. The average is 97% "not glass". The event's signal is diluted by its share of the clip. We call this the dilution problem.

How much does averaging dilute?

The curves show the pooled "glass score" for many clips without glass (blue) and with glass (warm). Where they overlap, no threshold can separate them. Shorten the event, or lengthen the clip.

0.3 s
10 s

An idealised model with illustrative numbers: 10 frames a second, frame noise of spread 1 along the glass direction, and an event that adds 2 to each of its frames. "Perfect focus" is the best any pooling could do if it knew where the event was.

Dilution in one formula

Look only along the "glass direction" in embedding space. A background frame scores about 0, with noise of spread σ. An event frame scores m more. With E event frames in T frames, mean pooling gives a pooled score of (E/T)·m, and averaging T noisy frames shrinks the noise to σ/√T. The separation, in units of noise, is the detectability d′:

d′
Detectability: the gap between the two classes' average pooled score, divided by the noise. With balanced classes, the best accuracy is Φ(d′/2), where Φ is the normal curve's cumulative area.
E
Number of frames that contain the event.
T
Number of frames in the clip.
m
How far an event frame sits from a background frame along the glass direction.
σ
Spread of one frame's noise along that direction.

Worked: E = 3, T = 100, m = 2, σ = 1 → d′mean = 6/10 = 0.6 → accuracy Φ(0.3) = 61.8%. Perfect focus: d′ = 2√3 = 3.46 → Φ(1.73) = 95.8%

Divide the two: focusing beats averaging by a factor of √(T/E). For a 3-frame event in 100 frames that is √33 ≈ 5.8 times more separation. Double the clip length, and mean pooling gets √2 worse while focus stays the same.

The lesson in one line. A pooling head is a learned weighted average over frames: it keeps mean pooling's "any length in, one vector out", but learns from labels alone which frames deserve the weight.
  1. Frames in, one vector out: the shapes, the frozen encoder, caching, padding and masks.
  2. A learned weighted average: scores, softmax, and the two common score functions.
  3. Why it works: derive the gradient and see credit flow to the event.
  4. Between mean and max: temperature, entropy, and the best place between the limits.
  5. Build it in PyTorch: by hand, from scratch, then a library line. How much data it needs.
  6. More than one question: multi-query pooling, from Set Transformer to Q-Former.
  7. Attentive statistics: pool the spread as well as the mean.
  8. Gated attention: why Ilse et al. multiply in a sigmoid.
  9. Did it work?: baselines, shuffle and deletion tests, and why weights are not proof.
  10. Where else it works, and when not to use it: six fields, and the cases it fails.
  11. Cheat sheet: the block on one screen, and where to go next.
A 0.5-second event sits in a 20-second clip at 10 frames a second. How much more separation does perfect focus give than mean pooling?

Chapter 1

Frames in, one vector out

Follow the tensor shapes from waveform to probability, cache the frozen features, and see what padding does to a pool without a mask

Before any maths, fix the data flow. Every later chapter changes only one box in it.

StepShapeWhat it is
WaveformB × 160,000B clips of 10 s at 16 kHz.
Frozen encoderB × T × DT frames (100 for 10 s at 10 frames a second), D numbers each (768 for an HTSAT-style tower). Never trained.
PoolB × DOne vector per clip. This is the only new idea in this lesson.
Linear layerB × COne logit per class: C = 1 for "glass or not".
SigmoidB × CProbability of each class, between 0 and 1.

Why keep the encoder frozen?

Three reasons. First, data: a few hundred labels cannot safely retrain millions of weights; they would overfit. Second, the encoder already separates sounds well, because it saw far more audio than you have; LAION-CLAP trained on LAION-Audio-630K, 633,526 audio-text pairs LAION-CLAP · abstract. Third, speed: if the encoder never changes, you run it once and store its output.

That stored output is the embedding cache. Training then reads small tensors from disk instead of running a big network. Each epoch takes seconds, so you can try many heads.

Worked: 10,000 clips × 100 frames × 768 numbers × 2 bytes (float16) = 1.54 GB on disk

Different lengths: pad, then mask

A batch is one tensor, so clips of 3, 6 and 10 seconds are padded with zero frames to the longest, 100 frames. A mask is a B × T table of true or false: true for a real frame, false for padding. Every pool must use it. A masked mean divides by the real count. A masked attention pool gives padding a score of minus infinity, so its softmax weight is exactly 0.

ht
The frame embedding at time t: a vector of D numbers.
mt
The mask: 1 for a real frame, 0 for padding.
z
The pooled clip vector, D numbers.

Worked: a 3-second clip padded to 100 frames. Without the mask, the mean is (sum of 30 frames)/100: the clip's signal shrinks to 30%. With it, (sum of 30)/30: full strength.

Pad, mask, pool

A batch of three clips, padded to 100 frames. Each has a glass break. Bars under each row are the pooling weights. Turn the mask off and see weight leak into the padding.

Toy frames as in the hero, with a fixed attention query that already points at the glass direction. Shapes: input 3 × 100 × 16 plus a 3 × 100 mask; output 3 × 16.

A batch holds a 4-second and a 10-second clip, padded to 100 frames. What does a correct attention pool do with the 60 padding frames of the short clip?

Chapter 2

A learned weighted average

Write every pool as weights over frames, then let a small network choose the weights from each frame's content

Here is the trick that unifies everything. Write any pool as a weighted average: z = Σ at ht, with weights at ≥ 0 that add up to 1. Mean pooling sets every at = 1/T. Max pooling, in its "pick the top frame" form, puts weight 1 on one frame and 0 elsewhere. Neither looks at what the frames contain when it sets the weights.

Attention pooling computes the weights from the frames. Each frame gets a score st: one number, "how much does this frame look like what I need?" A softmax turns the scores into weights.

st
The frame's score, computed by a small network (below).
at
The frame's weight. The softmax makes every weight positive and the weights sum to 1.
vt
The frame's value: what gets averaged. Often vt = ht, or a learned projection Wvht.
z
The pooled clip vector that the classifier sees.

Worked: scores (2.0, 0.1, −0.3, 0.2) → es = (7.39, 1.11, 0.74, 1.22), sum 10.46 → a = (0.707, 0.106, 0.071, 0.117). With 1-D values v = (3, 0, 1, 0): z = 0.707×3 + 0.071×1 = 2.19

Two ways to score a frame

A learned query. Keep one learned vector q, the query. Project each frame to a key kt = Wkht. Score by the dot product, scaled by the key size d: st = q·kt / √d. The query is a template of "what I am looking for". This is the form inside CLIP's attention pool, Set Transformer's PMA, Perceiver and Q-Former (Chapter 6).

A small MLP (additive scoring). Ilse, Tomczak and Welling score each instance with st = w⊤ tanh(V ht), where V is an L × D matrix and w has L numbers Ilse 2018 · §2.3, eq. 8. It is the additive scoring style that Bahdanau, Cho and Bengio used for translation in 2014. The tanh lets the score bend; the query form is a straight ramp.

Point the query

Twelve frames of one clip, drawn in a flat 2-D embedding space. Nine are background (blue); three are the glass break (warm). Turn the query and change its length. The purple ring is the pooled vector z.

25°
1.50

Real frames have hundreds of dimensions; two are enough to see the geometry. The additive form here uses one tanh unit along the query direction, scaled by the query length.

Two things to notice. Length acts as sharpness: a long query makes the scores far apart, and the softmax becomes almost one-hot. Direction acts as the question: point it away from the event and the head attends to the wrong frames, with the same confidence.

Scores (1, 1, 1, 1) go through a softmax. What pool do you get?

Chapter 3 Why it works

Why it learns to focus

Derive the gradient of the loss with respect to each frame's score, and watch credit flow to the event from clip-level labels alone

Nobody tells the head which frames hold the glass. The label is per clip. So how do the scores learn to rise on the right frames? The answer is in one derivative. We derive it in three steps.

Step 1: the softmax derivative. For weights aj = esj/Σes, raising score st raises at and lowers all others: ∂aj/∂st = aj(δjt − at), where δjt is 1 if j = t and 0 otherwise.

Step 2: the pooled vector. Put that into z = Σj ajhj: ∂z/∂st = atht − atΣjajhj = at(ht − z). Raising one score pulls z toward that frame, in proportion to its weight.

Step 3: the loss. Let g = ∂L/∂z, the direction in which z should move to cut the loss. For a logistic detector with logit u·z + b and binary cross-entropy, g = (p − y)u. The chain rule gives:

at
The frame's current weight. Credit flows in proportion to attention already paid.
ht − z
How this frame differs from the current pooled vector.
g
∂L/∂z = (p − y)u: the direction the loss wants z to move. Its dot product with ht − z is the frame's relevance.
∂L/∂st
Negative means "raise this score". Gradient descent subtracts it.

Worked: 4 frames, all weights 0.25. Detector scores u·h = (3, 0, 1, 0), so u·z = 1 and p = σ(1) = 0.731 for a positive clip (y = 1). Then g·ht = (p−1)u·ht: ∂L/∂s1 = 0.25 × (−0.269) × (3 − 1) = −0.134 (raise it); frames 2 and 4: +0.067 (lower them); frame 3 sits at the average: 0.

Read the two factors. The frame that pushes the decision the right way, more than the average does, gains score. Frames that drag the average down lose score. And the gain is scaled by at: frames the head already watches learn fastest. We checked this formula against PyTorch autograd on random inputs; they agree to 6 decimal places.

Follow the credit

One positive clip of 8 frames; frames 3 and 4 are the glass. Purple bars are the weights. Arrows show minus the gradient for each score: up means "raise". Press Step, or Run. Then try a bad start.

Here the scores are free numbers and the detector u is fixed, so you see only the attention part learning. Learning rate 2. Illustrative values.

What the bad start teaches

Locked on frame 7, the event frames have weights near 0.002. Their credit is scaled by those tiny weights, so they rise slowly. This is the "rich get richer" side of the formula. It is why heads start with small random score weights (so the softmax starts near uniform) and why a too-high learning rate early can lock the head onto the wrong frame.

A frame has weight at = 0.5 and relevance g·(ht − z) = 0. What does one gradient step do to its score?

Chapter 4

Between mean and max

Add a temperature to the softmax, recover mean and max pooling as its two limits, and find where detection is best

Divide every score by a temperature τ before the softmax. A large τ squashes the scores together; a small τ stretches them apart. The query length in Chapter 2 did the same job.

τ
Temperature. τ → ∞ gives equal weights: mean pooling. τ → 0 gives all weight to the top score: max pooling.
H
Entropy of the weights, in nats: 0 when one frame has everything, ln T when all are equal.
Neff
The effective number of frames the head averages: from 1 (max) to T (mean).

Worked: scores (2, 0, 0, 0). τ = 1 → a = (0.711, 0.096, 0.096, 0.096), H = 0.92, Neff = 2.5. τ = 0.5 → a1 = 0.948, Neff = 1.3 (near max). τ = 4 → a1 = 0.355, Neff = 3.9 (near mean, which is 4).

So mean and max are not rivals of attention. They are its two ends. The real question is where between them the best detector sits. The device answers it on 400 synthetic clips, with the query fixed to the true glass direction, so only the temperature changes.

Find the best temperature

Top: accuracy against temperature (log scale), for a 0.3-second event in 10-second clips. Bottom: the weights on one positive clip at your temperature. Slide from max to mean.

1.00
0.3 s

Computed live: 400 clips, scores st = (glass direction)·ht, pooled score Σatst, best threshold. Same toy as the hero (event strength 3, noise 1, clip offset 0.4). Illustrative numbers.

The peak sits between the ends. Max pooling trusts one frame, so one noisy background frame can fool it. Mean pooling trusts all frames, so the event drowns. A middle temperature averages the few frames that matter. As the event gets longer, the best τ moves up, toward the mean; as it gets shorter, down, toward the max.

Also look at the gradient. From Chapter 3, credit is proportional to at. Mean pooling gives every frame credit 1/T. Max pooling gives one frame all of it and the rest nothing, so a wrong pick gets no correction from the others. Attention spreads credit over Neff frames.

A head's weights on a 100-frame clip have Neff = 1.1. Which pool is it behaving like, and what is the risk?

Chapter 5 How

Build it in PyTorch

Go from the hand calculation to a 25-line module, train it on cached embeddings, find the library one-liner, and measure how much data it needs

You have done the arithmetic by hand twice: scores to weights (Chapter 2) and weights to gradients (Chapter 3). Now the same thing as code. Here is a gated attention-pooling head with a mask, as a PyTorch nn.Module. Every line has its shape.

import torch
import torch.nn as nn

class AttentionPool(nn.Module):
    """Gated attention pooling (Ilse et al. 2018) plus a linear detector."""
    def __init__(self, d_in=768, d_attn=128, n_classes=1, gated=True):
        super().__init__()
        self.V = nn.Linear(d_in, d_attn)                 # tanh branch
        self.U = nn.Linear(d_in, d_attn) if gated else None  # sigmoid gate
        self.w = nn.Linear(d_attn, 1)                    # one score per frame
        self.cls = nn.Linear(d_in, n_classes)            # detector on the pooled vector

    def forward(self, h, mask=None):           # h: (B, T, D); mask: (B, T), True = real frame
        a = torch.tanh(self.V(h))              # (B, T, d_attn)
        if self.U is not None:
            a = a * torch.sigmoid(self.U(h))   # gate, element by element
        s = self.w(a).squeeze(-1)              # (B, T): one score per frame
        if mask is not None:
            s = s.masked_fill(~mask, float('-inf'))  # padding gets weight 0
        alpha = torch.softmax(s, dim=1)        # (B, T); each row sums to 1
        z = torch.einsum('bt,btd->bd', alpha, h)     # (B, D): the pooled clip vector
        return self.cls(z), alpha              # logits (B, n_classes), weights (B, T)

Why each choice. d_attn = 128 is small because the scorer only answers one question per frame; Ilse et al. tried 64, 128 and 256 Ilse 2018 · §4.1. The values are the raw frames, so z lives in the encoder's own space and a linear detector on it is a fair comparison with a linear probe. The weights come back with the logits, because you will plot them in Chapter 9.

Cache the frozen features once

A stand-in encoder here makes the block runnable anywhere; swap in your real one. The only contract is: waveform in, B × T × D frame embeddings out.

import torch
import torch.nn as nn
torch.manual_seed(0)
D, T, SR = 768, 100, 16000                     # 10 s clips at 16 kHz, 100 ms frames
# Stand-in for a frozen audio encoder that returns frame embeddings (B, T, D).
# Swap in your real one (for example the audio tower of a CLAP model).
encoder = nn.Conv1d(1, D, kernel_size=1600, stride=1600)
encoder.requires_grad_(False).eval()

wavs = torch.randn(64, SR * 10)                # 64 clips of raw audio (B, samples)
labels = torch.randint(0, 2, (64,)).float()    # 1 = glass break somewhere in the clip

feats = []
with torch.no_grad():                          # the encoder runs once, never again
    for i in range(0, len(wavs), 16):
        x = wavs[i:i + 16].unsqueeze(1)        # (16, 1, samples)
        h = encoder(x).transpose(1, 2)         # (16, T, D) frame embeddings
        feats.append(h.half())                 # float16 halves the disk
feats = torch.cat(feats)                       # (64, 100, 768)
torch.save({'h': feats, 'y': labels}, 'cache.pt')

Train only the head

import torch
import torch.nn.functional as F
# AttentionPool is the class above
data = torch.load('cache.pt')
h, y = data['h'].float(), data['y']            # (N, T, D), (N,)
head = AttentionPool(d_in=h.shape[-1])
opt = torch.optim.AdamW(head.parameters(), lr=1e-3, weight_decay=1e-2)
for epoch in range(20):
    perm = torch.randperm(len(h))
    for i in range(0, len(h), 32):
        idx = perm[i:i + 32]
        logits, alpha = head(h[idx])           # no encoder in the loop: fast
        loss = F.binary_cross_entropy_with_logits(logits.squeeze(-1), y[idx])
        opt.zero_grad(); loss.backward(); opt.step()

The training details that matter. Use a validation split and stop early: a head this small overfits a few hundred clips in minutes. Use weight decay. If positives are rare, set pos_weight in the loss or balance the batches. Save the weights alpha for the analysis.

The library one-liners

import torch
import torch.nn as nn
from timm.layers import AttentionPoolLatent
B, T, D = 4, 100, 768
h = torch.randn(B, T, D)
mask = torch.ones(B, T, dtype=torch.bool)
# PyTorch: one learned query, 8 heads
q = nn.Parameter(torch.randn(1, 1, D) * 0.02)
mha = nn.MultiheadAttention(D, num_heads=8, batch_first=True)
z, w = mha(q.expand(B, -1, -1), h, h, key_padding_mask=~mask)   # z: (4, 1, 768), w: (4, 1, 100)
# timm: a learned latent query plus an MLP, in one module
pool = AttentionPoolLatent(in_features=D, num_heads=8)
print(pool(h).shape)                           # torch.Size([4, 768])

Note that key_padding_mask uses the opposite convention: True means "ignore this frame". We ran all four blocks (PyTorch 2.10, timm 1.0.20) and the shapes printed are the real output.

How much data does it need?

An attention head has more to learn than a linear probe: where to look, as well as what to say. So with very few clips it can lose. The device trains all three heads at six training-set sizes, for three kinds of label.

Learning curves

Pick what the label depends on, then press Run. Each point is one head trained from scratch on that many clips and scored on 400 new ones.

Synthetic clips as in the hero; 300 Adam steps per head with weight decay; each point is the average of two runs, so expect a few points of noise. Illustrative numbers.

Real results tell the same story. On BREAST CANCER, with only 58 weakly labelled images, Ilse et al. report accuracy 0.741 for embedding mean pooling and 0.745 for attention: a tie, and mean pooling had the higher AUC (0.796 against 0.775). On COLON CANCER, with 100 images, attention pulled ahead: 0.904 against 0.860 Ilse 2018 · Tables 2, 3. And on AudioSet's released frozen embeddings, attention heads lifted mAP from Google's 0.314 baseline to 0.337 (single-level attention) and 0.369 (feature-level attention) PANNs · Table IV.

You cached 2,000 clips of frame embeddings. Your label is "this room recording is a café", true across the whole clip. You have 40 labelled clips. What head do you try first?

Chapter 6

More than one question

See what one softmax cannot do, and how extra learned queries fix it, from Set Transformer's seeds to CLIP's pool, Perceiver's latents and Q-Former's queries

One softmax has a budget of exactly 1. If a clip holds two things you care about, a dog bark and a cough, one query must split its weight between them. The pooled vector then holds half of each. Worse, it looks the same as a clip with only one of them, at full strength.

So a single query cannot answer "are both sounds here?" Try it below. The fix is to ask several questions at once. Use K learned queries q1 … qK, each with its own softmax and its own pooled vector:

Q
K learned query vectors, a K × d matrix. Each row asks one question.
K⊤, V
Keys and values of the T frames (T × d and T × D). The softmax runs over frames, separately for each query.
Z
K pooled vectors (K × D). Concatenate them, or keep them as a short set of tokens.

Worked: one query, a bark and a cough both present and equally strong → about 0.5 weight on each → each sound reaches z at half strength. Two queries → q1 takes the bark, q2 the cough → both at full strength, in separate slots.

One query or two?

Label: 1 only if a bark (warm) and a cough (pink) are both in the clip. Pick a head and press Train. The curves are each query's weights on a clip that has both.

Trained here: 60-frame clips of 12-dimensional embeddings, 800 training clips (a quarter each: both, bark only, cough only, neither), 1,200 Adam steps, scored on 800 new clips. Illustrative numbers.

The same block, under four names

WhereQueriesWhat the paper says
Set Transformer PMA (Lee et al., 2019)k learned "seed" vectorsPooling by Multihead Attention: PMAk(Z) = MAB(S, rFF(Z)). One seed (k = 1) in most cases; four seeds when each output must describe one cluster Set Transformer · §3.2, §5.3.
CLIP ResNet image encoder (Radford et al., 2021)1, computed from the imageGlobal average pooling replaced by "a single layer of transformer-style multi-head QKV attention where the query is conditioned on the global average-pooled representation" CLIP · §2.4.
Perceiver (Jaegle et al., 2021)N latents (512 on ImageNet)A learned latent array cross-attends to a large input, such as the 50,176 pixels of a 224 × 224 image, then thinks in the small latent space Perceiver · §3.1.
Q-Former in BLIP-2 (Li et al., 2023)32 learned queries32 queries of 768 numbers each; the output Z (32 × 768) is much smaller than frozen ViT-L/14 features (257 × 1024) BLIP-2 · §3.1.

Read the table as one idea at four scales. With one query, you have the detector of this lesson. With a few, you keep several summaries. With hundreds, the pooled set becomes a compact sequence that another network reads: Perceiver thinks in it, and BLIP-2 feeds it to a frozen language model. The cost stays linear in T, because each query reads each frame once.

Why can a one-query attention head not tell "bark and cough" from "bark only"?

Chapter 7

Attentive statistics

Pool how much the frames vary, not only their average, weighted by attention, as speaker-verification models do

Some labels are not "is a sound present?" but "how does the sound behave over time?". A healthy engine hums; a failing one rattles. Both can have the same average frame. The difference is in the spread: how far frames swing from their mean. Mean pooling throws spread away.

Speaker verification met this first. A voice is not one frame; it is how the frames vary. Statistics pooling concatenates the mean and the standard deviation over frames. Attentive statistics pooling weights both by attention, so silence and noise count less Okabe 2018 · §3:

μ̃
Weighted mean of the frames: the attention pool you already know.
σ̃
Weighted standard deviation, per dimension. ⊙ is element-by-element multiplication.
z
Both, stacked: 2D numbers per clip.

Worked (one dimension, equal weights): frames (1, 3, 1, 3) → μ̃ = 2, σ̃ = √(5 − 4) = 1. Frames (2, 2, 2, 2) → μ̃ = 2, σ̃ = 0. Same mean, and only the spread tells them apart.

Hum or rattle?

Each dot is one clip: steady engines (blue) and rattling ones (warm), placed by their pooled mean and spread. Clips also hold silent gaps of random length. Pick the pooled features and add more silence.

60%

Trained here: logistic regression on the pooled features of 400 clips, scored on 400 more. Attention weights come from a frame's energy (silent frames score low). Illustrative numbers.

Silence hurts plain statistics twice. It drags the mean toward the silent value, and it changes the spread by an amount that depends on how much silence the clip has. Attention gives silent frames little weight, so both statistics describe the engine, not the gaps.

On speaker verification, Okabe, Koshinaka and Shinoda report that attentive statistics pooling cut equal error rates by 7.5% on NIST SRE 2012 and 8.1% on VoxCeleb, relative to plain statistics pooling Okabe 2018 · abstract. ECAPA-TDNN made the attention channel-dependent: each feature channel gets its own weights over frames, and the scorer also sees global statistics of the whole recording ECAPA-TDNN · §3.1.

Two classes have identical average frames but different frame-to-frame variation. Which pooled feature can separate them?

Chapter 8

Gated attention

See what one tanh unit cannot select, and how multiplying in a sigmoid gate carves out the target band

Ilse et al. noticed a limit in the tanh scorer. tanh(x) is nearly a straight line for x between −1 and 1. So one tanh unit scores frames by a ramp along one direction. It cannot say "this much, but not more". Their fix multiplies in a gate: a sigmoid of a second projection Ilse 2018 · §2.3, eq. 9.

V
L × D matrix of the tanh branch: "how strongly does the frame show the pattern?"
U
L × D matrix of the gate. sigm(x) = 1/(1 + e−x), between 0 and 1: "should this unit count for this frame?"
st
The score that goes into the softmax, as before.

Worked, one unit: tanh branch 0.9 (strong pattern), gate sigm(−3) = 0.05 (too loud) → 0.9 × 0.05 = 0.045. Same pattern at normal loudness, gate sigm(2) = 0.88 → 0.79. The gate is an AND: pattern present and loudness in range.

Picture the frames by two features. "Impact" is how sharp the onset is; "loudness" is how loud. Glass breaks are sharp and medium-loud. Door slams are sharp and very loud. Spoon clinks are sharp and quiet. A single ramp ranks frames along one direction, so it cannot rank the middle band above both sides.

Carve out the band

The shading is the score each scorer gives every point of the plane (purple = high). Press Fit: each scorer, with one hidden unit, learns to put the softmax weight on the glass frames (warm). Compare.

Fitted here by gradient ascent on the log of the softmax weight given to glass frames, in bags that hold all four kinds of frame. One hidden unit each, to show what one unit can express. Illustrative data.

Is the gate needed in practice? The honest answer is "a little, sometimes". With several hidden units, plain tanh can build a band too, from two ramps. In Ilse et al.'s tables, gated attention beat plain attention on BREAST CANCER (accuracy 0.755 against 0.745) and tied on COLON CANCER (0.898 against 0.904, inside the error bars) Ilse 2018 · Tables 2, 3. It costs one extra matrix. Most pathology code keeps it as a default.

What does the sigmoid gate add to the tanh scorer?

Chapter 9 Did it work?

Did it work?

Compare with the right baselines, run the shuffle and deletion tests, and catch a head that attends to the wrong sound

A good test score is not enough. You want to know that the head works for the right reason. Here is the checklist, in the order to run it.

  1. Baselines. Train mean pooling + a linear layer (a linear probe on mean features) and max pooling on the same cache. If attention does not beat them on held-out data, keep the simpler one.
  2. The right metric. Detection is usually imbalanced. Report average precision (AP) or the AUC per class, and the false alarms per hour at your operating threshold, not only accuracy.
  3. Shuffle test. Permute the frames of each test clip. The prediction must not change at all (Chapter 10 explains why). If it changes, position is leaking in, for example through unmasked padding.
  4. Deletion test. Remove the top-k weighted frames. The positive score should fall a lot. Remove k random frames: it should barely move. If both moves are small, the weights are not doing the work.
  5. Spot-check localisation. Hand-mark the event time in 50 positive clips and see whether the weight peak lands inside.
  6. Confound probes. Build test sets that break co-occurrences in your data. The device shows why.
  7. Label shuffle. Train on shuffled labels. Accuracy must fall to chance; if not, you have a leak between train and test.

The head that heard the scream

In the training clips, a loud scream (pink) always follows the glass (warm). Train, then read the three test scores. Use Shuffle and Delete on the shown clip. Turn the confound off and train again.

Trained here: 600 clips of 60 frames, 12 dimensions. Test sets of 400 clips each: like training; glass with no scream anywhere; and screams in random clips. Illustrative numbers.

The trap is clear. The head scored almost perfectly on data like its training set. It attended to the scream, because the scream was louder and always present in positives. On clips that break the link, it falls to a coin flip. Its attention map looked confident the whole time.

Attention weights are a clue, not a proof

Attention weights give you free, weak localisation: from clip labels alone, a heatmap of where the head looked. Ilse et al. used it to highlight regions of interest in histopathology slides Ilse 2018 · §4.3. Use it. But do not read it as an explanation of the decision.

Jain and Wallace tested this in NLP models. They found that attention weights are often uncorrelated with gradient-based measures of feature importance, and that very different attention patterns can give the same predictions Jain & Wallace 2019 · abstract. Wiegreffe and Pinter replied that the answer depends on what you mean by explanation, and proposed tests that account for the whole model Wiegreffe & Pinter 2019 · abstract. For pooling heads there is a simple reason to be careful: the classifier reads z, and many weightings give similar z.

You delete the 3 top-weighted frames of each positive test clip and the detector's scores barely change. What do you conclude?

Chapter 10 What elseWhat not

Where else it works, and when not to use it

Move the same block to slides, voices, videos, documents, wearables and point clouds, then meet the tasks it cannot do

Strip the audio away and the block is general. You have a bag (any set of items) of instances (the items), each already embedded by a frozen encoder. You have one label per bag. Attention pooling learns which instances matter. In machine learning this setting is called multiple instance learning (MIL).

Where else it works

FieldInstances → bagWhat changes, what stays
Pathology slidesImage patches → one slideBags are huge (thousands of patches). Ilse et al. cut each 896 × 768 BREAST CANCER image into 32 × 32 patches, 672 per bag Ilse 2018 · §4.3. CLAM uses attention pooling with slide-level labels only, and turns the weights into heatmaps CLAM · 2021. Same head; the encoder is a patch CNN.
Speaker verificationSpeech frames → one utterancePool mean and spread (Chapter 7). The output is an embedding compared by cosine, not a class.
Video actionsFrame features → one videoAttention Clusters used many attention units over frame features with no recurrence and reported 79.4% top-1 on Kinetics for one model Attention Clusters · abstract. Order is ignored, which is fine for "is someone cooking?".
DocumentsSentence embeddings → one document"Does this support ticket contain a legal threat?" One sentence decides. The weights point to it for the reviewer.
WearablesSensor windows → one session"Did the wearer fall in this hour?" A few seconds matter out of 3,600. Exactly the dilution problem of Chapter 0.
Point cloudsPoints → one objectPointNet uses max pooling as its symmetric function. In Set Transformer's tests, attention pooling (rFF + PMA) beat average pooling at 100 points (0.8076 against 0.7951) but lost at 5,000 (0.8628 against 0.8933) Set Transformer · Table 4.

What stays the same in every row: frozen instance embeddings, one weighted average, one linear layer, and the analysis of Chapter 9. What changes: the encoder, the bag size, and whether you pool spread as well.

When not to use it: order

Attention pooling is permutation invariant: shuffle the frames and z does not change, because a sum does not care about order. Zaheer et al. showed that any permutation-invariant function of a set can be written as a function of a sum of per-item features Deep Sets · Theorem 2. That is a strength for "is glass here?". It is fatal for "did the door open before the alarm?".

Before or after?

Every clip has a door (warm) and an alarm (pink). Label 1 if the door comes first. Train the head. Then add a position feature to each frame and train again.

Trained here: 600 clips of 60 frames, 12 dimensions, 500 Adam steps, scored on 400 new clips. The position feature is one extra number per frame, running from −2 at the start to +2 at the end. Illustrative numbers.

The decision table

SituationAvoid attention pooling becauseUse instead
The label depends on order or timingPooling is permutation invariantPosition features at least; better a sequence head (a small transformer or GRU over frames), see positional encoding
Every frame is labelled, or you need onset and offset timesClip-level weights are weak, biased localisationA frame-wise head: a classifier on every frame, then smoothing (see event detection heads)
Tiny data, evidence spread over the clipExtra freedom fits noise (Chapter 5)Mean pooling + a linear probe
The label is a count ("three barks")The softmax normalises the amount awaySum pooling, or frame-wise detection then counting
Very long bags (thousands of frames)Weights spread thin or collapse onto one frame; memory grows with TPool in chunks, then pool the chunks (hierarchical); or a few latent queries, Perceiver-style
One short loud event, little dataNothing wrong, but max pooling is as good and simplerTry max pooling first as the baseline
Why does an attention-pooling head with no position information score exactly chance on "door before alarm"?

Chapter 11

Cheat sheet

The whole block on one screen, a chooser for your next problem, honest limits, and where to go next

Answer three questions about your problem. The device picks the head and the first baseline to beat.

Which head for my problem?

Choose an answer in each row. The pipeline below updates.

A rule of thumb from this lesson, not a law. Always compare with the baseline on held-out data.

The block on one screen

PartIn one line
Shape signature(B, T, D) frames + (B, T) mask → (B, D) pooled → (B, C) logits; also returns (B, T) weights.
Mathsa = softmax(s / τ) over frames, st = w·tanh(Vht) or q·kt/√d; z = Σ atvt.
Why it works∂L/∂st = at·g·(ht − z): credit goes to watched frames that move z the right way.
Limitsτ → ∞ is mean pooling; τ → 0 is max pooling; the best is usually between.
Knobsd_attn (64 to 256), gated or not, number of queries K, mean only or mean + std, value projection, weight decay, early stopping.
AlwaysMask padding with minus infinity; cache the frozen features; start the softmax near uniform.
Did it work?Beat mean + linear and max; shuffle test (no change); deletion test (top-k falls, random does not); confound probes.
Failure modesConfounds, padding leaks, order tasks, count tasks, tiny data, very long bags, weights read as explanations.
AlternativesMean + linear probe; max or linear-softmax pooling; frame-wise heads; sequence heads; fine-tuning the encoder.

Honest limits

Where this sits, and where to go next

This lesson is part of the Builder's Toolkit series on reusable heads for frozen backbones. Go on to:

The takeaway. Pooling turns a bag of frozen embeddings into one vector. Mean pooling dilutes short evidence by √(T/E). Attention pooling is a learned weighted average whose gradient, at·g·(ht − z), teaches it to focus from bag labels alone. Use several queries for several things, mean plus spread for texture, positions for order, and always test it against the simple pools.
You build a fall detector from 1-hour wearable sessions with session-level labels and 5,000 sessions. Which design and which check come first?

Now press Present or Teach and explain, out loud and from memory, why averaging drowns a short sound and how the gradient teaches the head to focus. Then go back to the hero and find the event length where mean pooling stops working.

Attention Pooling Heads. Sources: M. Ilse, J. M. Tomczak, M. Welling, "Attention-based Deep Multiple Instance Learning" (ICML 2018, arXiv); D. Bahdanau, K. Cho, Y. Bengio, "Neural Machine Translation by Jointly Learning to Align and Translate" (2014, arXiv); K. Okabe, T. Koshinaka, K. Shinoda, "Attentive Statistics Pooling for Deep Speaker Embedding" (Interspeech 2018, arXiv); B. Desplanques, J. Thienpondt, K. Demuynck, "ECAPA-TDNN" (Interspeech 2020, arXiv); J. Lee et al., "Set Transformer" (ICML 2019, arXiv); A. Radford et al., "Learning Transferable Visual Models From Natural Language Supervision" (CLIP, ICML 2021, arXiv); A. Jaegle et al., "Perceiver" (ICML 2021, arXiv); J. Li, D. Li, S. Savarese, S. Hoi, "BLIP-2" (2023, arXiv); Y. Wu et al., "Large-scale Contrastive Language-Audio Pretraining" (LAION-CLAP, ICASSP 2023, arXiv); Q. Kong et al., "PANNs" (2020, arXiv); Y. Wang, J. Li, F. Metze, "A Comparison of Five Multiple Instance Learning Pooling Functions for Sound Event Detection with Weak Labeling" (ICASSP 2019, arXiv); S. Jain, B. C. Wallace, "Attention is not Explanation" (NAACL 2019, arXiv); S. Wiegreffe, Y. Pinter, "Attention is not not Explanation" (EMNLP 2019, arXiv); M. Y. Lu et al., "Data-efficient and weakly supervised computational pathology on whole-slide images" (CLAM, Nature Biomedical Engineering 2021, arXiv); X. Long et al., "Attention Clusters" (CVPR 2018, arXiv); M. Zaheer et al., "Deep Sets" (NeurIPS 2017, arXiv); C. R. Qi et al., "PointNet" (CVPR 2017, arXiv).
Back to Gleams