A frozen audio model turns a 10-second clip into 100 frame vectors. Your detector needs one vector. Average the frames, and a 0.3-second glass break is 3% of the mix. An attention-pooling head learns where in the clip to look, and averages there.
Learn why mean pooling drowns short events, how a learned weighted average fixes it, why its gradient teaches it to focus, and how to build, train and test one on cached embeddings in about 25 lines of PyTorch.
Set the event length and press Train all three. Three heads learn from 800 synthetic clips, in your browser. Switch the view to see where each head looks. Make the event shorter and watch mean pooling fall toward a coin flip.
You need vectors, the dot product, and the idea of a trained classifier. We build the rest from zero.
Synthetic data, illustrative numbers. Each clip is 100 frames (10 s at 10 frames a second) of 16-dimensional embeddings: noise, plus an offset shared by the whole clip. Positive clips add one fixed "glass" direction to the event's frames. The heads train here with Adam and are scored on 400 new clips. The plot on the right starts with four event lengths; each run you start adds or replaces a point.
Chapter 0
See why a detector needs one vector per clip, and measure how badly plain averaging hides a short sound
You are building a glass-break detector for a home camera. You have a few hundred labelled 10-second clips. Each label says only "glass broke somewhere in this clip" or "it did not". Nobody marked the second.
You also have a strong audio model that you will not train. A frozen encoder is a pretrained network whose weights you keep fixed. Many audio encoders cut a clip into short frames (here, one every 100 ms) and output one vector per frame. Ten seconds gives 100 frame vectors. The HTSAT audio tower inside LAION-CLAP, for example, produces 768-dimensional features before its projection layer LAION-CLAP · §3.3.
A detector, though, is a classifier. A classifier takes one vector of a fixed size. Clips come in different lengths: 3 seconds, 10 seconds, a minute. So something must squeeze "any number of frame vectors" into "one vector". That squeeze is pooling, and the small network that pools and then classifies is a pooling head.
The simplest pool is the average. Mean pooling adds the T frame vectors and divides by T. It works for any length. It does not care where the event happens. The CLAP paper notes this is the usual way to handle long audio: average the per-frame or per-chunk embeddings LAION-CLAP · §3.4.
Now count. A glass break lasts about 0.3 seconds: 3 frames out of 100. The other 97 frames are room noise, a TV, a fridge. The average is 97% "not glass". The event's signal is diluted by its share of the clip. We call this the dilution problem.
The curves show the pooled "glass score" for many clips without glass (blue) and with glass (warm). Where they overlap, no threshold can separate them. Shorten the event, or lengthen the clip.
An idealised model with illustrative numbers: 10 frames a second, frame noise of spread 1 along the glass direction, and an event that adds 2 to each of its frames. "Perfect focus" is the best any pooling could do if it knew where the event was.
Look only along the "glass direction" in embedding space. A background frame scores about 0, with noise of spread σ. An event frame scores m more. With E event frames in T frames, mean pooling gives a pooled score of (E/T)·m, and averaging T noisy frames shrinks the noise to σ/√T. The separation, in units of noise, is the detectability d′:
Worked: E = 3, T = 100, m = 2, σ = 1 → d′mean = 6/10 = 0.6 → accuracy Φ(0.3) = 61.8%. Perfect focus: d′ = 2√3 = 3.46 → Φ(1.73) = 95.8%
Divide the two: focusing beats averaging by a factor of √(T/E). For a 3-frame event in 100 frames that is √33 ≈ 5.8 times more separation. Double the clip length, and mean pooling gets √2 worse while focus stays the same.
Chapter 1
Follow the tensor shapes from waveform to probability, cache the frozen features, and see what padding does to a pool without a mask
Before any maths, fix the data flow. Every later chapter changes only one box in it.
| Step | Shape | What it is |
|---|---|---|
| Waveform | B × 160,000 | B clips of 10 s at 16 kHz. |
| Frozen encoder | B × T × D | T frames (100 for 10 s at 10 frames a second), D numbers each (768 for an HTSAT-style tower). Never trained. |
| Pool | B × D | One vector per clip. This is the only new idea in this lesson. |
| Linear layer | B × C | One logit per class: C = 1 for "glass or not". |
| Sigmoid | B × C | Probability of each class, between 0 and 1. |
Three reasons. First, data: a few hundred labels cannot safely retrain millions of weights; they would overfit. Second, the encoder already separates sounds well, because it saw far more audio than you have; LAION-CLAP trained on LAION-Audio-630K, 633,526 audio-text pairs LAION-CLAP · abstract. Third, speed: if the encoder never changes, you run it once and store its output.
That stored output is the embedding cache. Training then reads small tensors from disk instead of running a big network. Each epoch takes seconds, so you can try many heads.
Worked: 10,000 clips × 100 frames × 768 numbers × 2 bytes (float16) = 1.54 GB on disk
A batch is one tensor, so clips of 3, 6 and 10 seconds are padded with zero frames to the longest, 100 frames. A mask is a B × T table of true or false: true for a real frame, false for padding. Every pool must use it. A masked mean divides by the real count. A masked attention pool gives padding a score of minus infinity, so its softmax weight is exactly 0.
Worked: a 3-second clip padded to 100 frames. Without the mask, the mean is (sum of 30 frames)/100: the clip's signal shrinks to 30%. With it, (sum of 30)/30: full strength.
A batch of three clips, padded to 100 frames. Each has a glass break. Bars under each row are the pooling weights. Turn the mask off and see weight leak into the padding.
Toy frames as in the hero, with a fixed attention query that already points at the glass direction. Shapes: input 3 × 100 × 16 plus a 3 × 100 mask; output 3 × 16.
Chapter 2
Write every pool as weights over frames, then let a small network choose the weights from each frame's content
Here is the trick that unifies everything. Write any pool as a weighted average: z = Σ at ht, with weights at ≥ 0 that add up to 1. Mean pooling sets every at = 1/T. Max pooling, in its "pick the top frame" form, puts weight 1 on one frame and 0 elsewhere. Neither looks at what the frames contain when it sets the weights.
Attention pooling computes the weights from the frames. Each frame gets a score st: one number, "how much does this frame look like what I need?" A softmax turns the scores into weights.
Worked: scores (2.0, 0.1, −0.3, 0.2) → es = (7.39, 1.11, 0.74, 1.22), sum 10.46 → a = (0.707, 0.106, 0.071, 0.117). With 1-D values v = (3, 0, 1, 0): z = 0.707×3 + 0.071×1 = 2.19
A learned query. Keep one learned vector q, the query. Project each frame to a key kt = Wkht. Score by the dot product, scaled by the key size d: st = q·kt / √d. The query is a template of "what I am looking for". This is the form inside CLIP's attention pool, Set Transformer's PMA, Perceiver and Q-Former (Chapter 6).
A small MLP (additive scoring). Ilse, Tomczak and Welling score each instance with st = w⊤ tanh(V ht), where V is an L × D matrix and w has L numbers Ilse 2018 · §2.3, eq. 8. It is the additive scoring style that Bahdanau, Cho and Bengio used for translation in 2014. The tanh lets the score bend; the query form is a straight ramp.
Twelve frames of one clip, drawn in a flat 2-D embedding space. Nine are background (blue); three are the glass break (warm). Turn the query and change its length. The purple ring is the pooled vector z.
Real frames have hundreds of dimensions; two are enough to see the geometry. The additive form here uses one tanh unit along the query direction, scaled by the query length.
Two things to notice. Length acts as sharpness: a long query makes the scores far apart, and the softmax becomes almost one-hot. Direction acts as the question: point it away from the event and the head attends to the wrong frames, with the same confidence.
Chapter 3 Why it works
Derive the gradient of the loss with respect to each frame's score, and watch credit flow to the event from clip-level labels alone
Nobody tells the head which frames hold the glass. The label is per clip. So how do the scores learn to rise on the right frames? The answer is in one derivative. We derive it in three steps.
Step 1: the softmax derivative. For weights aj = esj/Σes, raising score st raises at and lowers all others: ∂aj/∂st = aj(δjt − at), where δjt is 1 if j = t and 0 otherwise.
Step 2: the pooled vector. Put that into z = Σj ajhj: ∂z/∂st = atht − atΣjajhj = at(ht − z). Raising one score pulls z toward that frame, in proportion to its weight.
Step 3: the loss. Let g = ∂L/∂z, the direction in which z should move to cut the loss. For a logistic detector with logit u·z + b and binary cross-entropy, g = (p − y)u. The chain rule gives:
Worked: 4 frames, all weights 0.25. Detector scores u·h = (3, 0, 1, 0), so u·z = 1 and p = σ(1) = 0.731 for a positive clip (y = 1). Then g·ht = (p−1)u·ht: ∂L/∂s1 = 0.25 × (−0.269) × (3 − 1) = −0.134 (raise it); frames 2 and 4: +0.067 (lower them); frame 3 sits at the average: 0.
Read the two factors. The frame that pushes the decision the right way, more than the average does, gains score. Frames that drag the average down lose score. And the gain is scaled by at: frames the head already watches learn fastest. We checked this formula against PyTorch autograd on random inputs; they agree to 6 decimal places.
One positive clip of 8 frames; frames 3 and 4 are the glass. Purple bars are the weights. Arrows show minus the gradient for each score: up means "raise". Press Step, or Run. Then try a bad start.
Here the scores are free numbers and the detector u is fixed, so you see only the attention part learning. Learning rate 2. Illustrative values.
Locked on frame 7, the event frames have weights near 0.002. Their credit is scaled by those tiny weights, so they rise slowly. This is the "rich get richer" side of the formula. It is why heads start with small random score weights (so the softmax starts near uniform) and why a too-high learning rate early can lock the head onto the wrong frame.
Chapter 4
Add a temperature to the softmax, recover mean and max pooling as its two limits, and find where detection is best
Divide every score by a temperature τ before the softmax. A large τ squashes the scores together; a small τ stretches them apart. The query length in Chapter 2 did the same job.
Worked: scores (2, 0, 0, 0). τ = 1 → a = (0.711, 0.096, 0.096, 0.096), H = 0.92, Neff = 2.5. τ = 0.5 → a1 = 0.948, Neff = 1.3 (near max). τ = 4 → a1 = 0.355, Neff = 3.9 (near mean, which is 4).
So mean and max are not rivals of attention. They are its two ends. The real question is where between them the best detector sits. The device answers it on 400 synthetic clips, with the query fixed to the true glass direction, so only the temperature changes.
Top: accuracy against temperature (log scale), for a 0.3-second event in 10-second clips. Bottom: the weights on one positive clip at your temperature. Slide from max to mean.
Computed live: 400 clips, scores st = (glass direction)·ht, pooled score Σatst, best threshold. Same toy as the hero (event strength 3, noise 1, clip offset 0.4). Illustrative numbers.
The peak sits between the ends. Max pooling trusts one frame, so one noisy background frame can fool it. Mean pooling trusts all frames, so the event drowns. A middle temperature averages the few frames that matter. As the event gets longer, the best τ moves up, toward the mean; as it gets shorter, down, toward the max.
Also look at the gradient. From Chapter 3, credit is proportional to at. Mean pooling gives every frame credit 1/T. Max pooling gives one frame all of it and the rest nothing, so a wrong pick gets no correction from the others. Attention spreads credit over Neff frames.
Chapter 5 How
Go from the hand calculation to a 25-line module, train it on cached embeddings, find the library one-liner, and measure how much data it needs
You have done the arithmetic by hand twice: scores to weights (Chapter 2) and weights to gradients (Chapter 3). Now the same thing as code. Here is a gated attention-pooling head with a mask, as a PyTorch nn.Module. Every line has its shape.
import torch import torch.nn as nn class AttentionPool(nn.Module): """Gated attention pooling (Ilse et al. 2018) plus a linear detector.""" def __init__(self, d_in=768, d_attn=128, n_classes=1, gated=True): super().__init__() self.V = nn.Linear(d_in, d_attn) # tanh branch self.U = nn.Linear(d_in, d_attn) if gated else None # sigmoid gate self.w = nn.Linear(d_attn, 1) # one score per frame self.cls = nn.Linear(d_in, n_classes) # detector on the pooled vector def forward(self, h, mask=None): # h: (B, T, D); mask: (B, T), True = real frame a = torch.tanh(self.V(h)) # (B, T, d_attn) if self.U is not None: a = a * torch.sigmoid(self.U(h)) # gate, element by element s = self.w(a).squeeze(-1) # (B, T): one score per frame if mask is not None: s = s.masked_fill(~mask, float('-inf')) # padding gets weight 0 alpha = torch.softmax(s, dim=1) # (B, T); each row sums to 1 z = torch.einsum('bt,btd->bd', alpha, h) # (B, D): the pooled clip vector return self.cls(z), alpha # logits (B, n_classes), weights (B, T)
Why each choice. d_attn = 128 is small because the scorer only answers one question per frame; Ilse et al. tried 64, 128 and 256 Ilse 2018 · §4.1. The values are the raw frames, so z lives in the encoder's own space and a linear detector on it is a fair comparison with a linear probe. The weights come back with the logits, because you will plot them in Chapter 9.
A stand-in encoder here makes the block runnable anywhere; swap in your real one. The only contract is: waveform in, B × T × D frame embeddings out.
import torch import torch.nn as nn torch.manual_seed(0) D, T, SR = 768, 100, 16000 # 10 s clips at 16 kHz, 100 ms frames # Stand-in for a frozen audio encoder that returns frame embeddings (B, T, D). # Swap in your real one (for example the audio tower of a CLAP model). encoder = nn.Conv1d(1, D, kernel_size=1600, stride=1600) encoder.requires_grad_(False).eval() wavs = torch.randn(64, SR * 10) # 64 clips of raw audio (B, samples) labels = torch.randint(0, 2, (64,)).float() # 1 = glass break somewhere in the clip feats = [] with torch.no_grad(): # the encoder runs once, never again for i in range(0, len(wavs), 16): x = wavs[i:i + 16].unsqueeze(1) # (16, 1, samples) h = encoder(x).transpose(1, 2) # (16, T, D) frame embeddings feats.append(h.half()) # float16 halves the disk feats = torch.cat(feats) # (64, 100, 768) torch.save({'h': feats, 'y': labels}, 'cache.pt')
import torch import torch.nn.functional as F # AttentionPool is the class above data = torch.load('cache.pt') h, y = data['h'].float(), data['y'] # (N, T, D), (N,) head = AttentionPool(d_in=h.shape[-1]) opt = torch.optim.AdamW(head.parameters(), lr=1e-3, weight_decay=1e-2) for epoch in range(20): perm = torch.randperm(len(h)) for i in range(0, len(h), 32): idx = perm[i:i + 32] logits, alpha = head(h[idx]) # no encoder in the loop: fast loss = F.binary_cross_entropy_with_logits(logits.squeeze(-1), y[idx]) opt.zero_grad(); loss.backward(); opt.step()
The training details that matter. Use a validation split and stop early: a head this small overfits a few hundred clips in minutes. Use weight decay. If positives are rare, set pos_weight in the loss or balance the batches. Save the weights alpha for the analysis.
import torch import torch.nn as nn from timm.layers import AttentionPoolLatent B, T, D = 4, 100, 768 h = torch.randn(B, T, D) mask = torch.ones(B, T, dtype=torch.bool) # PyTorch: one learned query, 8 heads q = nn.Parameter(torch.randn(1, 1, D) * 0.02) mha = nn.MultiheadAttention(D, num_heads=8, batch_first=True) z, w = mha(q.expand(B, -1, -1), h, h, key_padding_mask=~mask) # z: (4, 1, 768), w: (4, 1, 100) # timm: a learned latent query plus an MLP, in one module pool = AttentionPoolLatent(in_features=D, num_heads=8) print(pool(h).shape) # torch.Size([4, 768])
Note that key_padding_mask uses the opposite convention: True means "ignore this frame". We ran all four blocks (PyTorch 2.10, timm 1.0.20) and the shapes printed are the real output.
An attention head has more to learn than a linear probe: where to look, as well as what to say. So with very few clips it can lose. The device trains all three heads at six training-set sizes, for three kinds of label.
Pick what the label depends on, then press Run. Each point is one head trained from scratch on that many clips and scored on 400 new ones.
Synthetic clips as in the hero; 300 Adam steps per head with weight decay; each point is the average of two runs, so expect a few points of noise. Illustrative numbers.
Real results tell the same story. On BREAST CANCER, with only 58 weakly labelled images, Ilse et al. report accuracy 0.741 for embedding mean pooling and 0.745 for attention: a tie, and mean pooling had the higher AUC (0.796 against 0.775). On COLON CANCER, with 100 images, attention pulled ahead: 0.904 against 0.860 Ilse 2018 · Tables 2, 3. And on AudioSet's released frozen embeddings, attention heads lifted mAP from Google's 0.314 baseline to 0.337 (single-level attention) and 0.369 (feature-level attention) PANNs · Table IV.
Chapter 6
See what one softmax cannot do, and how extra learned queries fix it, from Set Transformer's seeds to CLIP's pool, Perceiver's latents and Q-Former's queries
One softmax has a budget of exactly 1. If a clip holds two things you care about, a dog bark and a cough, one query must split its weight between them. The pooled vector then holds half of each. Worse, it looks the same as a clip with only one of them, at full strength.
So a single query cannot answer "are both sounds here?" Try it below. The fix is to ask several questions at once. Use K learned queries q1 … qK, each with its own softmax and its own pooled vector:
Worked: one query, a bark and a cough both present and equally strong → about 0.5 weight on each → each sound reaches z at half strength. Two queries → q1 takes the bark, q2 the cough → both at full strength, in separate slots.
Label: 1 only if a bark (warm) and a cough (pink) are both in the clip. Pick a head and press Train. The curves are each query's weights on a clip that has both.
Trained here: 60-frame clips of 12-dimensional embeddings, 800 training clips (a quarter each: both, bark only, cough only, neither), 1,200 Adam steps, scored on 800 new clips. Illustrative numbers.
| Where | Queries | What the paper says |
|---|---|---|
| Set Transformer PMA (Lee et al., 2019) | k learned "seed" vectors | Pooling by Multihead Attention: PMAk(Z) = MAB(S, rFF(Z)). One seed (k = 1) in most cases; four seeds when each output must describe one cluster Set Transformer · §3.2, §5.3. |
| CLIP ResNet image encoder (Radford et al., 2021) | 1, computed from the image | Global average pooling replaced by "a single layer of transformer-style multi-head QKV attention where the query is conditioned on the global average-pooled representation" CLIP · §2.4. |
| Perceiver (Jaegle et al., 2021) | N latents (512 on ImageNet) | A learned latent array cross-attends to a large input, such as the 50,176 pixels of a 224 × 224 image, then thinks in the small latent space Perceiver · §3.1. |
| Q-Former in BLIP-2 (Li et al., 2023) | 32 learned queries | 32 queries of 768 numbers each; the output Z (32 × 768) is much smaller than frozen ViT-L/14 features (257 × 1024) BLIP-2 · §3.1. |
Read the table as one idea at four scales. With one query, you have the detector of this lesson. With a few, you keep several summaries. With hundreds, the pooled set becomes a compact sequence that another network reads: Perceiver thinks in it, and BLIP-2 feeds it to a frozen language model. The cost stays linear in T, because each query reads each frame once.
Chapter 7
Pool how much the frames vary, not only their average, weighted by attention, as speaker-verification models do
Some labels are not "is a sound present?" but "how does the sound behave over time?". A healthy engine hums; a failing one rattles. Both can have the same average frame. The difference is in the spread: how far frames swing from their mean. Mean pooling throws spread away.
Speaker verification met this first. A voice is not one frame; it is how the frames vary. Statistics pooling concatenates the mean and the standard deviation over frames. Attentive statistics pooling weights both by attention, so silence and noise count less Okabe 2018 · §3:
Worked (one dimension, equal weights): frames (1, 3, 1, 3) → μ̃ = 2, σ̃ = √(5 − 4) = 1. Frames (2, 2, 2, 2) → μ̃ = 2, σ̃ = 0. Same mean, and only the spread tells them apart.
Each dot is one clip: steady engines (blue) and rattling ones (warm), placed by their pooled mean and spread. Clips also hold silent gaps of random length. Pick the pooled features and add more silence.
Trained here: logistic regression on the pooled features of 400 clips, scored on 400 more. Attention weights come from a frame's energy (silent frames score low). Illustrative numbers.
Silence hurts plain statistics twice. It drags the mean toward the silent value, and it changes the spread by an amount that depends on how much silence the clip has. Attention gives silent frames little weight, so both statistics describe the engine, not the gaps.
On speaker verification, Okabe, Koshinaka and Shinoda report that attentive statistics pooling cut equal error rates by 7.5% on NIST SRE 2012 and 8.1% on VoxCeleb, relative to plain statistics pooling Okabe 2018 · abstract. ECAPA-TDNN made the attention channel-dependent: each feature channel gets its own weights over frames, and the scorer also sees global statistics of the whole recording ECAPA-TDNN · §3.1.
Chapter 8
See what one tanh unit cannot select, and how multiplying in a sigmoid gate carves out the target band
Ilse et al. noticed a limit in the tanh scorer. tanh(x) is nearly a straight line for x between −1 and 1. So one tanh unit scores frames by a ramp along one direction. It cannot say "this much, but not more". Their fix multiplies in a gate: a sigmoid of a second projection Ilse 2018 · §2.3, eq. 9.
Worked, one unit: tanh branch 0.9 (strong pattern), gate sigm(−3) = 0.05 (too loud) → 0.9 × 0.05 = 0.045. Same pattern at normal loudness, gate sigm(2) = 0.88 → 0.79. The gate is an AND: pattern present and loudness in range.
Picture the frames by two features. "Impact" is how sharp the onset is; "loudness" is how loud. Glass breaks are sharp and medium-loud. Door slams are sharp and very loud. Spoon clinks are sharp and quiet. A single ramp ranks frames along one direction, so it cannot rank the middle band above both sides.
The shading is the score each scorer gives every point of the plane (purple = high). Press Fit: each scorer, with one hidden unit, learns to put the softmax weight on the glass frames (warm). Compare.
Fitted here by gradient ascent on the log of the softmax weight given to glass frames, in bags that hold all four kinds of frame. One hidden unit each, to show what one unit can express. Illustrative data.
Is the gate needed in practice? The honest answer is "a little, sometimes". With several hidden units, plain tanh can build a band too, from two ramps. In Ilse et al.'s tables, gated attention beat plain attention on BREAST CANCER (accuracy 0.755 against 0.745) and tied on COLON CANCER (0.898 against 0.904, inside the error bars) Ilse 2018 · Tables 2, 3. It costs one extra matrix. Most pathology code keeps it as a default.
Chapter 9 Did it work?
Compare with the right baselines, run the shuffle and deletion tests, and catch a head that attends to the wrong sound
A good test score is not enough. You want to know that the head works for the right reason. Here is the checklist, in the order to run it.
In the training clips, a loud scream (pink) always follows the glass (warm). Train, then read the three test scores. Use Shuffle and Delete on the shown clip. Turn the confound off and train again.
Trained here: 600 clips of 60 frames, 12 dimensions. Test sets of 400 clips each: like training; glass with no scream anywhere; and screams in random clips. Illustrative numbers.
The trap is clear. The head scored almost perfectly on data like its training set. It attended to the scream, because the scream was louder and always present in positives. On clips that break the link, it falls to a coin flip. Its attention map looked confident the whole time.
Attention weights give you free, weak localisation: from clip labels alone, a heatmap of where the head looked. Ilse et al. used it to highlight regions of interest in histopathology slides Ilse 2018 · §4.3. Use it. But do not read it as an explanation of the decision.
Jain and Wallace tested this in NLP models. They found that attention weights are often uncorrelated with gradient-based measures of feature importance, and that very different attention patterns can give the same predictions Jain & Wallace 2019 · abstract. Wiegreffe and Pinter replied that the answer depends on what you mean by explanation, and proposed tests that account for the whole model Wiegreffe & Pinter 2019 · abstract. For pooling heads there is a simple reason to be careful: the classifier reads z, and many weightings give similar z.
Chapter 10 What elseWhat not
Move the same block to slides, voices, videos, documents, wearables and point clouds, then meet the tasks it cannot do
Strip the audio away and the block is general. You have a bag (any set of items) of instances (the items), each already embedded by a frozen encoder. You have one label per bag. Attention pooling learns which instances matter. In machine learning this setting is called multiple instance learning (MIL).
| Field | Instances → bag | What changes, what stays |
|---|---|---|
| Pathology slides | Image patches → one slide | Bags are huge (thousands of patches). Ilse et al. cut each 896 × 768 BREAST CANCER image into 32 × 32 patches, 672 per bag Ilse 2018 · §4.3. CLAM uses attention pooling with slide-level labels only, and turns the weights into heatmaps CLAM · 2021. Same head; the encoder is a patch CNN. |
| Speaker verification | Speech frames → one utterance | Pool mean and spread (Chapter 7). The output is an embedding compared by cosine, not a class. |
| Video actions | Frame features → one video | Attention Clusters used many attention units over frame features with no recurrence and reported 79.4% top-1 on Kinetics for one model Attention Clusters · abstract. Order is ignored, which is fine for "is someone cooking?". |
| Documents | Sentence embeddings → one document | "Does this support ticket contain a legal threat?" One sentence decides. The weights point to it for the reviewer. |
| Wearables | Sensor windows → one session | "Did the wearer fall in this hour?" A few seconds matter out of 3,600. Exactly the dilution problem of Chapter 0. |
| Point clouds | Points → one object | PointNet uses max pooling as its symmetric function. In Set Transformer's tests, attention pooling (rFF + PMA) beat average pooling at 100 points (0.8076 against 0.7951) but lost at 5,000 (0.8628 against 0.8933) Set Transformer · Table 4. |
What stays the same in every row: frozen instance embeddings, one weighted average, one linear layer, and the analysis of Chapter 9. What changes: the encoder, the bag size, and whether you pool spread as well.
Attention pooling is permutation invariant: shuffle the frames and z does not change, because a sum does not care about order. Zaheer et al. showed that any permutation-invariant function of a set can be written as a function of a sum of per-item features Deep Sets · Theorem 2. That is a strength for "is glass here?". It is fatal for "did the door open before the alarm?".
Every clip has a door (warm) and an alarm (pink). Label 1 if the door comes first. Train the head. Then add a position feature to each frame and train again.
Trained here: 600 clips of 60 frames, 12 dimensions, 500 Adam steps, scored on 400 new clips. The position feature is one extra number per frame, running from −2 at the start to +2 at the end. Illustrative numbers.
| Situation | Avoid attention pooling because | Use instead |
|---|---|---|
| The label depends on order or timing | Pooling is permutation invariant | Position features at least; better a sequence head (a small transformer or GRU over frames), see positional encoding |
| Every frame is labelled, or you need onset and offset times | Clip-level weights are weak, biased localisation | A frame-wise head: a classifier on every frame, then smoothing (see event detection heads) |
| Tiny data, evidence spread over the clip | Extra freedom fits noise (Chapter 5) | Mean pooling + a linear probe |
| The label is a count ("three barks") | The softmax normalises the amount away | Sum pooling, or frame-wise detection then counting |
| Very long bags (thousands of frames) | Weights spread thin or collapse onto one frame; memory grows with T | Pool in chunks, then pool the chunks (hierarchical); or a few latent queries, Perceiver-style |
| One short loud event, little data | Nothing wrong, but max pooling is as good and simpler | Try max pooling first as the baseline |
Chapter 11
The whole block on one screen, a chooser for your next problem, honest limits, and where to go next
Answer three questions about your problem. The device picks the head and the first baseline to beat.
Choose an answer in each row. The pipeline below updates.
A rule of thumb from this lesson, not a law. Always compare with the baseline on held-out data.
| Part | In one line |
|---|---|
| Shape signature | (B, T, D) frames + (B, T) mask → (B, D) pooled → (B, C) logits; also returns (B, T) weights. |
| Maths | a = softmax(s / τ) over frames, st = w·tanh(Vht) or q·kt/√d; z = Σ atvt. |
| Why it works | ∂L/∂st = at·g·(ht − z): credit goes to watched frames that move z the right way. |
| Limits | τ → ∞ is mean pooling; τ → 0 is max pooling; the best is usually between. |
| Knobs | d_attn (64 to 256), gated or not, number of queries K, mean only or mean + std, value projection, weight decay, early stopping. |
| Always | Mask padding with minus infinity; cache the frozen features; start the softmax near uniform. |
| Did it work? | Beat mean + linear and max; shuffle test (no change); deletion test (top-k falls, random does not); confound probes. |
| Failure modes | Confounds, padding leaks, order tasks, count tasks, tiny data, very long bags, weights read as explanations. |
| Alternatives | Mean + linear probe; max or linear-softmax pooling; frame-wise heads; sequence heads; fine-tuning the encoder. |
This lesson is part of the Builder's Toolkit series on reusable heads for frozen backbones. Go on to:
Now press Present or Teach and explain, out loud and from memory, why averaging drowns a short sound and how the gradient teaches the head to focus. Then go back to the hero and find the event length where mean pooling stops working.