Weak Labels and Multiple-Instance Learning

Your labels say whether a dog barked in a clip, never when. Train a frame classifier on those clip labels anyway, through a pooling function, and it learns when. Which pooling you pick decides how well.

Learn the multiple-instance assumption, derive what six pooling functions teach each frame, build the head in PyTorch, and learn how to check that localization really emerged.

Each clip has 64 frames. Some clips hide a 3-frame event; the labels only say "event: yes" or "no". Pick a pooling and press Train. A real classifier trains in this page. Watch which frames light up, and where the run lands on the chart of clip accuracy against frame F1.

You need logistic regression and the idea of a gradient. We build the rest from zero.

Train on whether, test on when

Ready

A real model trained here: a logistic-regression frame classifier on 6 features per frame, 160 training clips with clip labels only, Adam, 360 steps. Frame F1 is scored on 120 test clips whose event frames are known. The data is synthetic and the numbers are illustrative; the pattern (max misses frames, mean and attention add false alarms, linear softmax sits between) is the one Wang, Li and Metze report Wang 2019 · Table 2.

Chapter 0

When did the dog bark?

Meet the gap between the labels you have and the answers you want, and why that gap is the normal case

You have ten thousand 10-second sound clips. Each one carries a few tags, such as "dog bark" or "siren". Your job is a detector that says when the dog barks: the start and end of each bark, to a tenth of a second.

Nobody marked the seconds. The tags say a bark is somewhere in the clip, and nothing more. Can a model learn "when" from labels that only say "whether"? This lesson says yes, under one assumption, and shows exactly how.

Strong labels and weak labels

A strong label says what happened and where: "dog bark from 3.2 s to 3.9 s". A weak label says only what happened somewhere in a larger unit: "this clip contains a dog bark". The same split appears in many fields.

FieldWeak label (what you have)Strong label (what you want)
SoundClip-level tags: "this clip has a siren"Onset and offset times of every siren
PathologySlide-level diagnosis: "this slide is malignant"An outline around every cancer cell
PhotosImage tags: "this photo has a car"A box around every car

Weak labels are what you usually have, for a simple reason: they are cheap. A person can tag a clip after one listen. Marking every onset means scrubbing back and forth, frame by frame. Weak labels also come for free from metadata: a video title, a pathology report, a photo caption.

The biggest public sound dataset is built this way. Google's Audio Set is weakly labelled: 10-second YouTube clips, each tagged with the sounds it contains Wang 2019 · §1. The DCASE 2017 challenge task on sound events trained on 51,172 weakly labelled clips from it, but tested on clips with strong labels Wang 2019 · §3.1. That is the setting of this lesson: train on whether, judge on when.

What does the label tell you?

Pick a field. The example is split into its small parts. Turn on Strong labels to see the information a weak label leaves out.

The examples are drawn for illustration. The annotation times are rough illustrative assumptions, not measurements.

How much information is missing

Count the answers each kind of label gives. Suppose an example has n small parts and there are C classes:

answersweak
Yes-or-no answers a weak label gives for one example: one per class.
answersstrong
Yes-or-no answers a strong label gives: one per class for every part.
n
The number of small parts: frames in a clip, tiles in a slide, regions in a photo.
C
The number of classes.

Worked numbers from the DCASE 2017 setup: 17 classes and 100 frames per clip after the network Wang 2019 · §3.1, Fig. 2. A weak label gives 17 answers per clip. A strong label gives 100 × 17 = 1,700. The model must recover about 99% of the answers on its own.

  1. Bags and instances: the multiple-instance assumption and its variants.
  2. The pooling function: six ways to turn frame scores into one clip score.
  3. What max, mean and noisy-OR teach: the gradient each frame receives.
  4. What softmax, linear softmax and attention teach: the compromises.
  5. Localization for free: reading the frame scores as a detector.
  6. Build it in PyTorch: shapes, a switchable module, the training loop.
  7. What the papers found: Wang, Li and Metze 2019, and Ilse et al. 2018.
  8. Where else it works: molecules, slides, video, text, satellites.
  9. Did it work?: the checks that catch a model that tags well but localizes badly.
  10. When not to use it: counts, order, interactions, cheap strong labels.
  11. Connections: the cheat sheet and where to go next.
A clip of 100 frames, 17 classes. How many yes-or-no answers does a weak label give, against a strong label?

Chapter 1

Bags and instances

Name the parts, state the one assumption that links a bag's label to its contents, and try the variants

Multiple-instance learning gives this problem a vocabulary. A bag is the unit that carries the label: a clip, a slide, a photo. An instance is one small part of a bag: a frame, a tile, a region. We get labels for bags. We never see labels for instances.

Multiple-instance learning (MIL) is learning from labelled bags of unlabelled instances. The name comes from a drug-design problem. Dietterich, Lathrop and Lozano-Pérez (1997) wanted to predict which molecules bind to a target. A molecule can fold into many shapes, called conformations. It works if at least one of its shapes binds Ilse 2018 · §4.1. The molecule is the bag. Each shape is an instance.

The standard assumption

Sound follows the same logic. A clip is labelled "dog bark" if at least one of its frames contains a bark. Wang, Li and Metze call this the standard multiple instance (SMI) assumption: "the bag label is positive if and only if the bag contains at least one positive instance" Wang 2019 · §1. A positive instance inside a positive bag is called a witness.

Y
The bag label: 1 if the clip contains the event, 0 if not. This is the weak label we are given.
yi
The label of instance i: 1 if frame i contains the event. Hidden from us.
max over i
For labels that are 0 or 1, the largest one is 1 exactly when any one is 1. That is the logical OR.

Worked example: a clip of 8 frames with frame labels (0, 0, 0, 1, 1, 0, 0, 0). The max is 1, so the clip is positive. Flip both 1s to 0 and the max is 0. One witness is enough; zero witnesses means negative.

The assumption has a useful one-way rule. Every instance in a negative bag is negative. So negative bags are fully labelled for free. All the uncertainty sits inside positive bags.

Make a bag, read its label

Click frames to mark them as containing the event. Then switch the rule that turns frame labels into a clip label.

A logic toy: no learning here. The three rules are the standard assumption and two common relaxations of it.

The variants

The standard assumption is not the only one. A threshold rule says a bag is positive if it holds at least k positive instances: "a busy road" needs several cars, not one. A collective rule says the bag label depends on a share of instances: "mostly rain". Foulds and Frank (2010) review these assumptions in detail.

The rule matters because it decides which pooling functions can express the label. Max pooling matches "at least one" exactly. It cannot express "at least three". Chapter 10 trains on these rules and watches max pooling fail.

Note the two different jobs MIL can do. Bag classification asks: is this clip positive? Instance classification asks: which frames are? Sound event detection wants both: tagging, then localization Wang 2019 · abstract. A model can be good at the first and bad at the second, as Chapter 9 shows.

Under the standard MIL assumption, what do you know for certain about a frame in a clip labelled negative?

Chapter 2

The pooling function

Follow one clip through the network, then compare six ways to turn many frame scores into one clip score

Here is the trick in one sentence. Build a model that scores every frame, squash the frame scores into one clip score, and compare that with the clip label. The squashing step is the pooling function. It is the bridge between the instances we cannot label and the bag we can.

Follow one clip through. The audio becomes a spectrogram. A network turns it into T frame embeddings. A small classifier gives each frame a probability yi between 0 and 1. The pooling function turns (y1, …, yT) into one clip probability y. The loss compares y with the clip label t Wang 2019 · Fig. 1.

Six pooling functions

max
The clip is as likely as its most likely frame. Faithful to "at least one".
mean
Every frame counts equally; n is the number of frames.
noisy-OR
The chance that at least one frame fires, if frames fire independently.
softmax (log-sum-exp)
A smooth max over the frame logits zi (where yi = σ(zi)). The sharpness r slides it from mean (small r) to max (large r). σ is the sigmoid.
linear softmax
A weighted average where each frame's weight is its own probability: confident frames count more.
attention
A weighted average with weights wi > 0 from a separate, learned layer.

Worked example: four frames with y = (0.9, 0.6, 0.2, 0.1).

Five of the six sit between the mean and the max. Noisy-OR is the odd one: it is above the max. That is a warning sign we return to in Chapter 3.

One bag, six clip scores

Drag the orange bars to set each frame's probability. In Attention mode, drag the pink bars to set the weights. The small ticks under each frame show how much it counts in the chosen pool.

1.0
0

The extra background frames all sit at probability 0.1, so you can see how each pool reacts to a long clip with a short event.

Try the background slider with the event frames fixed. Add 50 quiet frames and the mean collapses toward 0.1, while the max does not move. Noisy-OR climbs toward 1, because fifty chances of 0.1 add up. Only the max ignores clip length entirely.

A clip has frame probabilities (0.9, 0.6, 0.2, 0.1). Which pooling gives 0.678?

Chapter 3

Why it works: what max, mean and noisy-OR teach

Derive the gradient each frame receives from a clip label, and see which frames learn and which way they move

The frame classifier learns only from gradients. So the question "what does this pooling teach?" has an exact answer: compute the gradient of the loss with respect to each frame's score. Wang, Li and Metze do exactly this for five pooling functions Wang 2019 · §2.2. We follow them.

The chain rule, once for all poolings

The loss is binary cross-entropy between the clip probability y and the clip label t (0 or 1): L = −t log y − (1 − t) log(1 − y) Wang 2019 · Eq. 1. Split its gradient into three pieces:

∂L/∂zi
How the loss changes with frame i's logit. Gradient descent moves zi the opposite way.
∂L/∂y
The clip-level error. Negative for a positive clip (raise y), positive for a negative clip (lower y). The same for every pooling Wang 2019 · Eq. 3.
∂y/∂yi
How much frame i moves the clip score. This is the only part the pooling decides.
yi(1 − yi)
The sigmoid's slope at frame i. Small when the frame is already confident.

So each pooling is defined, for learning purposes, by its middle term. Let us derive it for three poolings, with the clip y = (0.9, 0.6, 0.2, 0.1) from Chapter 2.

Max: only the top frame learns

y = max yi, so ∂y/∂yi is 1 for the largest frame and 0 for all others Wang 2019 · Table 1. For our clip: (1, 0, 0, 0). The sign is right: a positive clip raises the top frame, a negative clip lowers it. But the frame at 0.6, which might be the second half of the same bark, gets nothing. Wang et al.: only one frame receives an error signal, which "may cause many frame-level false negatives" Wang 2019 · §2.2.

Mean: every frame is pushed the same

y = (1/n) Σ yi, so ∂y/∂yi = 1/n for every frame: (0.25, 0.25, 0.25, 0.25). For a negative clip this is correct: every frame should go down. For a positive clip it is wrong: every frame goes up, including the silent ones. The result is many false positive frames Wang 2019 · §2.2.

Noisy-OR: right in theory, saturated in practice

y = 1 − Π(1 − yj). Differentiate one factor: ∂y/∂yi = Πj≠i(1 − yj) = (1 − y)/(1 − yi). For our clip, 1 − y = 0.0288, so the terms are (0.288, 0.072, 0.036, 0.032). Larger frames get more, which looks good. The trouble is length. With 64 frames at only 0.05 each, y = 1 − 0.9564 = 0.96. A silent clip looks positive. Wang et al. found earlier that noisy-OR "does not work for localization" despite its neat probability story Wang 2019 · §1.

Who gets pushed, and which way?

The green ticks mark the real event frames, which the model cannot see. Blue arrows show the push gradient descent gives each frame. Press Take 10 steps to apply it and watch the frames move.

Each frame's logit is treated as a free number and updated by plain gradient descent on this one clip's loss, so you see the pooling's learning rule with nothing else in the way. Frame 10 is a loud background sound, not the event.

With mean pooling, what happens to a silent frame inside a clip labelled positive?

Chapter 4

Why it works: the compromises

Derive the gradients of softmax, linear softmax and attention pooling, and find the one that pushes frames apart

Max teaches too few frames. Mean teaches too many, in the wrong direction. The remaining poolings try to sit between. Each gives every frame some signal, but more to the frames that look like the event. Their gradients show how well each one manages.

Softmax (log-sum-exp): a dial from mean to max

This pool works on logits. Let Z = (1/r) log((1/n) Σ er zj) and y = σ(Z). The derivative of a log-sum-exp is a softmax:

∂Z/∂zi
Frame i's share of the softmax over the frames, at sharpness r. The shares add to 1 and are all positive.

Worked, r = 1, logits (2.20, 0.41, −1.39, −2.20): the shares are (0.829, 0.138, 0.023, 0.010). The clip score is 0.731, so for a positive clip y − t = −0.269. The frames get pushes of (0.223, 0.037, 0.006, 0.003) upward. Large r sends all of it to the top frame, like max. Small r spreads it evenly, like mean. Every share is positive, so in a positive clip every frame still goes up a little.

Linear softmax: the push changes sign

y = Σ yj2 / Σ yj. Use the quotient rule: ∂y/∂yi = (2yi Σyj − Σyj2) / (Σyj)2. Divide top and bottom by Σyj:

∂y/∂yi
Frame i's effect on the clip score. Unlike every pool so far, it can be negative Wang 2019 · Table 1.
yi > y/2
The dividing line. Frames above half the clip score raise it; frames below it lower it.

Worked, y = (0.9, 0.6, 0.2, 0.1), clip score 0.678, sum 1.8: the terms are (0.623, 0.290, −0.154, −0.266). In a positive clip, the two large frames are pushed up and the two small ones pushed down. The frames are driven to 0 and 1: sharp, well-localized detections. In a negative clip, every frame is pulled toward y/2, and since y keeps falling, all frames head to zero Wang 2019 · §2.2. This is the behaviour the SMI assumption asks for.

Attention: two things learn, and one can cheat

y = Σ wi yi / Σ wi, with weights from their own layer. Two gradients now matter Wang 2019 · Table 1:

∂y/∂yi
Always positive: frames move with the clip label, scaled by their weight.
∂y/∂wi
Positive for frames above the clip score. So the weights can also move to change y.

In a positive clip, weights grow on high frames. Good. In a negative clip, the network can lower y two ways: lower the frames, or move the weights onto the quiet frames. Worked: a negative clip y = (0.6, 0.1, 0.1, 0.1), equal weights, clip score 0.225. Shift the weights to (0.05, 0.32, 0.32, 0.32) and the clip score drops to 0.125, while the 0.6 frame stays at 0.6. That frame becomes a false positive. Wang et al. saw exactly this: the attention focused on low-probability regions while frames between 4 and 6 s fired freely Wang 2019 · §3.1, Fig. 3.

The compromises, step by step

Same clip as before. Blue arrows are frame gradients; in Attention mode, pink arrows show how the weights will move. Try Attention with a negative label, then take steps and watch frame 10.

1.0

As in Chapter 3, each frame's logit (and each attention logit) is a free number updated by gradient descent on this one clip. The behaviour you see is the pooling's learning rule, derived above.

A fifth pool in Wang et al. is the exponential softmax: weights eyi on the probabilities. Its gradient, (1 − y + yi) eyi / Σ eyj, is always positive. So it shares mean pooling's false positives, but more mildly, because small frames get smaller pushes Wang 2019 · §2.2.

In linear softmax pooling, a positive clip has score y = 0.678. What happens to a frame at 0.2?

Chapter 5

Localization for free

Read a clip-trained model's frame scores as a detector, pick a threshold, and score it against strong labels

Here is the payoff. After training on clip labels, the frame classifier still exists. Feed it a new clip and it gives a probability for every frame. Threshold those probabilities and you get onsets and offsets. The model was never shown a single onset, yet it localizes. That is localization from weak labels.

Wang et al. do exactly this. They threshold clip scores with class-specific thresholds tuned for tagging. Then they apply the same thresholds to frame scores, averaged over 1-second segments Wang 2019 · §3.1. One threshold, two jobs: that only works if frame and clip scores agree.

From frame scores to detections

This model was trained in your browser with linear softmax pooling, from clip labels only. Move the threshold. Turn on smoothing. Step through test clips.

0.50

The same synthetic data as the hero: 160 training clips with clip labels, 120 test clips with known event frames. Numbers are illustrative. Smoothing is a 3-frame median filter, a common post-processing step in sound event detection.

Scoring localization

Count frames (or segments) three ways. A true positive (TP) is predicted and real. A false positive (FP) is predicted but not real. A false negative (FN) is real but missed.

P, precision
Of the frames you flagged, the share that were real.
R, recall
Of the real event frames, the share you flagged.
F1
The harmonic mean of the two. It is high only if both are.

Worked with the paper's numbers. Linear softmax on DCASE 2017, localization: TP = 1,832, FN = 2,528, FP = 2,187 Wang 2019 · Table 2. P = 1,832 / 4,019 = 45.6%. R = 1,832 / 4,360 = 42.0%. F1 = 43.7%. All three match the table.

Sound event detection also reports an error rate. It counts substitutions S (right time, wrong class), deletions D (missed) and insertions I (false alarms) against the N reference events:

ER
Error rate. Lower is better. It can exceed 100%, because insertions are not capped.
S
Substitutions: a segment with a reference event and a detection, but the wrong class.
D
Deletions: reference events that were missed.
I
Insertions: detections with no reference event.
N
The number of reference events; here TP + FN = 4,360.

Linear softmax: (1,040 + 1,488 + 1,147) / 4,360 = 84.3%.   Mean: (1,385 + 861 + 2,373) / 4,360 = 105.9%

Both match Table 2. Mean pooling's 2,373 insertions push its error rate above 100%: it would do better, on this metric, by predicting nothing.

Mean pooling scored the best tagging F1 in Wang et al.'s Table 2. Why was its localization error rate over 100%?

Chapter 6

Build it in PyTorch

Trace the tensor shapes, write a head with switchable pooling, train it on clip labels, and meet the library one-liners

The block is small. It sits on top of any backbone that gives one embedding per instance. Here is the data flow, with B clips in a batch, T frames, D features and C classes:

h
Frame embeddings from the backbone: a CNN, a CRNN, or a frozen audio foundation model.
z
Frame logits: the instance classifier's output, one per frame and class. Keep it: this is your detector.
y
Clip probabilities after pooling over the T axis.
t
The weak labels: one 0 or 1 per clip and class. Multi-label, so each class is its own yes-or-no problem.

Real shapes from TALNet: 400 input frames of 64 filterbank bins become 100 frames of 1,024 features at 10 Hz. A sigmoid layer gives 100 × 527 frame probabilities, pooled to 1 × 527 clip probabilities Wang 2019 · Fig. 2. The head is a 1,024 × 527 matrix plus 527 biases: 540,175 numbers. With attention, a second layer of the same size makes the weights.

Shapes and sizes

Set the sizes. The diagram traces one batch through the head. The default is TALNet's head on Audio Set.

100
1,024
527

Batch size B is left symbolic; it does not change the parameter count. The gated attention block uses a hidden size of 128, one of the sizes Ilse et al. tried (64, 128 and 256) Ilse 2018 · §4.

Form 1: by hand

You already did this. Chapter 2 pooled (0.9, 0.6, 0.2, 0.1) six ways. Chapters 3 and 4 computed each frame's gradient. Every line below is those same numbers, batched.

Form 2: from scratch in PyTorch

First the pooling, with every option behind one switch. It takes frame logits so that softmax pooling can work on them directly.

import math
import torch
import torch.nn as nn
import torch.nn.functional as F

class MILPool(nn.Module):
    """Frame logits (B, T, C) -> clip probabilities (B, C)."""
    def __init__(self, kind="linsoft", dim=None, n_classes=None, r=5.0):
        super().__init__()
        self.kind, self.r = kind, r
        if kind == "attention":                       # a weight per frame and class
            self.att = nn.Linear(dim, n_classes)

    def forward(self, z, h=None):                    # z: (B, T, C), h: (B, T, D)
        p = torch.sigmoid(z)                          # frame probabilities
        if self.kind == "max":
            return p.amax(dim=1)                      # (B, C)
        if self.kind == "mean":
            return p.mean(dim=1)
        if self.kind == "noisy_or":                   # 1 - prod(1 - p), in log space
            return 1 - torch.exp(torch.log1p(-p.clamp(max=1 - 1e-6)).sum(dim=1))
        if self.kind == "lse":                        # soft max over logits
            T = z.shape[1]
            Z = (torch.logsumexp(self.r * z, dim=1) - math.log(T)) / self.r
            return torch.sigmoid(Z)
        if self.kind == "linsoft":                    # sum p^2 / sum p
            return (p * p).sum(dim=1) / p.sum(dim=1).clamp_min(1e-7)
        if self.kind == "attention":                  # weights from their own layer
            w = torch.softmax(self.att(h), dim=1)     # (B, T, C), sums to 1 over T
            return (w * p).sum(dim=1)
        raise ValueError(self.kind)

Three lines need a reason. Noisy-OR is computed as a sum of logs, because a product of 100 numbers below 1 underflows. Linear softmax clamps the denominator, because a silent clip can have every p near 0. Attention uses a softmax over T, which is the same as exp-activated weights divided by their sum, Wang's form Wang 2019 · §3.1.

Now the head and the training loop. The backbone is frozen; only the head learns. The toy batch plants a 3-frame event on feature 0 of positive clips.

class WeakSED(nn.Module):
    """A frame head on frozen embeddings, trained from clip labels only."""
    def __init__(self, dim, n_classes, pool="linsoft"):
        super().__init__()
        self.head = nn.Linear(dim, n_classes)         # the instance classifier
        self.pool = MILPool(pool, dim, n_classes)

    def forward(self, h):                             # h: (B, T, D)
        z = self.head(h)                              # (B, T, C) frame logits
        return self.pool(z, h), torch.sigmoid(z)      # (B, C) clip, (B, T, C) frames

def toy_batch(B=32, T=64, D=16):
    h = torch.randn(B, T, D)                          # stand-in backbone output
    Y = (torch.rand(B, 1) < 0.5).float()              # weak clip labels
    frames = torch.zeros(B, T, 1)                     # hidden strong labels
    for b in range(B):
        if Y[b, 0]:
            t0 = torch.randint(0, T - 3, (1,)).item()
            h[b, t0:t0 + 3, 0] += 4.0                 # the event lives on feature 0
            frames[b, t0:t0 + 3, 0] = 1
    return h, Y, frames

torch.manual_seed(0)
model = WeakSED(dim=16, n_classes=1, pool="linsoft")
opt = torch.optim.Adam(model.parameters(), lr=3e-2)
for step in range(500):
    h, Y, _ = toy_batch()
    clip, _ = model(h)                                # (B, 1)
    loss = F.binary_cross_entropy(clip.clamp(1e-6, 1 - 1e-6), Y)
    opt.zero_grad(); loss.backward(); opt.step()

h, Y, frames = toy_batch(B=256)                       # a test set WITH frame labels
with torch.no_grad():
    clip, fr = model(h)
acc = ((clip > 0.5).float() == Y).float().mean()
f1 = 2 * ((fr > 0.5) & (frames > 0)).sum() / ((fr > 0.5).sum() + frames.sum())
print(f"clip acc {acc:.2f}  frame F1 {f1:.2f}")      # about 0.98 and 0.87

Ilse et al.'s attention is different in one important way: it pools embeddings, not scores. The bag embedding is a weighted average of instance embeddings, and one classifier reads it Ilse 2018 · §2.4. The weights use a tanh branch times a sigmoid gate, because tanh alone is almost linear near zero Ilse 2018 · §2.4.

class GatedAttentionMIL(nn.Module):
    """Ilse et al. 2018: pool embeddings, not scores."""
    def __init__(self, dim, hidden=128, n_classes=1):
        super().__init__()
        self.V = nn.Linear(dim, hidden)               # tanh branch
        self.U = nn.Linear(dim, hidden)               # sigmoid gate
        self.w = nn.Linear(hidden, 1)                 # one score per instance
        self.cls = nn.Linear(dim, n_classes)

    def forward(self, h):                             # h: (B, T, D)
        s = self.w(torch.tanh(self.V(h)) * torch.sigmoid(self.U(h)))  # (B, T, 1)
        a = torch.softmax(s, dim=1)                   # attention over instances
        bag = (a * h).sum(dim=1)                      # (B, D) bag embedding
        return self.cls(bag), a.squeeze(-1)           # logits (B, C), weights (B, T)

Notice what this version gives up. There is no frame classifier, so there are no frame scores to threshold. The attention weights are the only localization signal, and they say "this instance mattered", not "this instance is positive". For whole-slide pathology that is often enough. For onset times it is not.

Form 3: the library one-liners

Most of the pools are one call on the frame tensor p of shape (B, T, C): p.amax(1) for max, p.mean(1) for mean, torch.logsumexp(r * z, 1) for the softmax pool, (p * p).sum(1) / p.sum(1) for linear softmax. There is no standard library layer for attention MIL; the 15 lines above are the usual implementation.

Training details that matter

TALNet's head maps 100 × 1,024 frame features to 527 classes. How many weights and biases does the frame classifier have?

Chapter 7

What the papers found

Read Wang, Li and Metze's comparison of five pools on DCASE 2017 and Audio Set, then Ilse et al.'s attention results

Wang, Li and Metze (ICASSP 2019) trained the same network five times, changing only the pooling. The data was DCASE 2017 Task 4: 17 vehicle and warning sounds, 10-second clips from Audio Set Wang 2019 · §3.1. Training used weak labels on 50,030 clips. Testing used the 488 strongly labelled clips of the public test set.

Five pools, one network

Pick a metric. Each bar is one pooling function, everything else the same.

All numbers from Wang, Li and Metze 2019: Table 2 (DCASE 2017 Task 4 test set, the CRNN of Fig. 2 left) and Table 3 (TALNet trained on Audio Set, then applied to DCASE 2017 with no fine-tuning).

What the numbers say

Their conclusion has three parts. Linear softmax lets the gradient flow to many frames, balances misses against false alarms, and keeps clip and frame predictions consistent Wang 2019 · §4. They also say it is "by no means the ultimate optimal pooling function", and that the findings may apply to any MIL problem.

Ilse et al.: attention for bag classification

Ilse, Tomczak and Welling (ICML 2018) asked a different question: classify the bag well, and show which instances mattered. Their gated attention beat max and mean pooling on two pathology datasets. On colon cancer slides, attention reached 0.904 accuracy, against 0.842 for instance-level max and 0.772 for instance-level mean Ilse 2018 · Table 3. On breast cancer slides, gated attention reached 0.755, against 0.614 for instance-level max Ilse 2018 · Table 2.

The two papers do not contradict each other. Ilse et al. pool embeddings and judge the bag. Wang et al. pool probabilities and judge the frames. Attention is excellent at the first job. On the second, its freedom to move weights becomes a source of false alarms.

Rmax
Localization recall of max pooling: the share of real event segments it found Wang 2019 · Table 2. Every other pool found over 42%.
In Table 2, why did max pooling have the lowest localization recall?

Chapter 8

Where else it works

Move the same block to molecules, slides, video, text and satellite images, and find the one number that decides the pooling

Nothing in the block is about sound. It needs three things: bags with labels, instances you can embed, and a reason to believe "at least one instance makes the bag positive". Many fields have all three.

FieldBag → instanceWhat changes, what stays
Drug activity, the original MIL problemMolecule → its conformations (3D shapes). Musk1 has 92 bags and 476 instances with 166 features Ilse 2018 · Table 4.Tiny bags (about 5 shapes each); hand-made shape features instead of a deep backbone. Dietterich, Lathrop and Lozano-Pérez (1997) solved it with axis-parallel rectangles in feature space.
Computational pathologyWhole-slide image → tissue tiles. Campanella, Silva and Fuchs (2018) trained on 12,160 slides with slide-level diagnoses.Huge bags (thousands of tiles) and a tiny witness rate. A frozen image backbone embeds tiles; attention pooling (Ilse) gives both the diagnosis and a heatmap Ilse 2018 · Fig. 5.
Video anomaly detectionSurveillance video → 32 temporal segments. Sultani, Chen and Shah (2018) used 1,900 videos, 128 hours, with 13 anomaly types.Labels say only "this video has an anomaly". Instead of BCE they rank: the top segment of an anomalous video must score above the top segment of a normal one: max pooling inside a hinge loss.
Document classificationReview → sentences. A star rating labels the review; the model learns which sentences carry the sentiment (Kotzias et al., 2015).Instances interact more (negation across sentences). High witness rate: many sentences carry the opinion, so mean-like pools do well.
Satellite and photo tagsImage → regions or feature-map cells. A tag like "ship" or "car" labels the image; max pooling over a CNN's feature map points at the object (Oquab et al., 2015, "Is object localization for free?").Instances are overlapping cells of one feature map, so they are not independent. Output is a coarse location, not a box.

The same five parts stay in every row: an instance encoder, an instance scorer, a pooling function, a bag loss, and the bag labels. What changes is the encoder, the bag size, and one number that decides the pooling.

The witness rate

The witness rate is the share of instances in a positive bag that are positive. A 3-frame bark in a 64-frame clip: 3/64 = 4.7%. A tumour covering 1 tile in 1,000: 0.1%. A positive review where half the sentences praise the product: 50%. The witness rate and the bag size together decide which pool can separate the bags.

gapmean
How much higher a positive bag's mean score is than a negative bag's, on average.
ρ
The witness rate. μ+ and μ− are the average scores of positive and negative instances.
P(false alarmmax)
The chance that a negative bag's max crosses the threshold, if each negative instance does so independently.
q
The chance that one negative instance scores above the threshold. n is the bag size.

Worked numbers. Mean pooling, ρ = 4.7%, μ+ − μ− = 0.6: the gap is only 0.028, smaller than the noise of a 64-frame average. Max pooling with q = 1%: n = 64 gives 1 − 0.9964 = 47% false alarms; n = 1,000 gives almost 100%. Mean dies when witnesses are rare. Max dies when bags are huge and negatives are noisy. Learned weightings exist to survive both.

Which pool survives this field?

Pick a field to load a typical bag size and witness rate, then move the sliders. Each row is one pool: positive bags in purple above, negative bags in grey below. The number is how often a random positive bag outscores a random negative one.

64
4.7%

A Monte Carlo run in your browser: 200 positive and 200 negative bags. Instance scores are drawn from a fixed, imperfect classifier: positives near 0.75, negatives near 0.2, and 1 negative in 100 a look-alike near 0.6. The field presets for bag size and witness rate are illustrative, except Musk1's about 5 instances per bag and the 32 video segments.

Whole-slide images have thousands of tiles and a very small share of tumour tiles. Which pooling problem is worst there?

Chapter 9

Did it work?

Learn the checks that catch a model that tags well but localizes the wrong thing, and run them on a model trained in the page

A weakly supervised model can fool you. Its clip accuracy can be excellent while its frame scores point at the wrong sound. The clip label never says which frames should fire, so any frame pattern that predicts the label is fine by the loss. You have to check the frames yourself.

The checks

  1. Label a few clips strongly. You do not need many. DCASE 2017 trained on about 50,000 weak clips and tested on 488 strong ones Wang 2019 · §3.1. A few hundred strong clips turn guesses into frame F1 and error rate.
  2. Report clip and frame metrics side by side. A gap between them is a symptom (Chapter 5).
  3. Shuffle-label test. Train on randomly permuted clip labels. Test clip accuracy must fall to chance. If it does not, information is leaking from somewhere else.
  4. Look at the top frames. Listen to, or view, the 20 highest-scoring frames per class. Ilse et al. show attention maps on MNIST bags: in a positive bag, every 9 gets a high weight and other digits almost none Ilse 2018 · Fig. 4.
  5. Ablate the pooling. Train max, mean and linear softmax on the same backbone. If they disagree about which frames fire, find out why.
  6. Check for confounders. Find a sound that co-occurs with the class in training. A model can learn that instead.

The confounder trap

Pick a scenario and a pool, then press Train. The strip shows one test clip: green is the real event, red marks a different sound. Compare the two bars.

The hero's model and data, with one change per scenario. Confounder: a louder, different sound appears in 95% of positive training clips and 5% of negative ones. Look-alike: an everyday background sound points almost the same way as the event in feature space. Shuffled: clip labels are randomly permuted. Numbers are illustrative.

Run the confounder scenario with linear softmax. Clip accuracy stays high, while frame F1 can collapse toward zero. The model found the easiest predictor of the label: the louder sound that travels with it. Nothing in the clip labels can tell it apart. Only strong test labels, or listening to the top frames, reveal it.

accshuffled
The test clip accuracy you should see after training on shuffled labels.
π
The share of positive clips. Always guessing the larger class scores max(π, 1 − π).

Worked: half the toy's clips are positive, so π = 0.5 and the shuffled model should score near 50%. A shuffled model at 70% would mean the test set shares something with training that is not the event, such as duplicated clips.

A weakly trained model reaches 97% clip accuracy, but its frame F1 against a few strong labels is near zero. What is the likely cause?

Chapter 10

When not to use it

Break the MIL assumption on purpose with counts and order, and learn which block to use instead

Every pool in this lesson treats a bag as a set. The order of instances is invisible, and so is any interaction between them. That is a strength when the label really is "at least one". It is fatal when the label depends on something else.

f
Instance scorer followed by any pool in this lesson: the bag score.
π
Any reordering of the instances. The score cannot change.

Worked: a clip with a door slam at frame 10 and a scream at frame 40, labelled "slam then scream: yes". Another clip has the scream at 10 and the slam at 40, labelled "no". They hold the same set of frames, so every pool gives them the same score. At most one can be right. Over many such clips, accuracy sits at 50%.

Break the assumption

Choose the rule that labels each training clip, and a pool. Press Train. The bars collect the clip accuracy of each pool you have tried for this rule.

Same model and features as the hero. "At least three": clips hold 0 to 5 events. "Event before sound B": every clip holds one event and one other sound, and the label says which came first. Numbers are illustrative.

Two results deserve a sentence each. Max cannot count: one event and five events give the same max, so "at least three" is beyond it. Mean and linear softmax can partly count, because more events raise their score, but now a frame score no longer means "event here". Order defeats every pool, as the equation promised.

SituationUse MIL pooling?Use instead
Bag label means "at least one instance", and you want to know whichYes. Linear softmax for localization; attention for bag accuracy.—
Bag label depends on a count ("three or more coughs")No: max cannot express it.Sum the frame probabilities and train against the count, or a counting head.
Bag label depends on order or timing ("glass breaks after the brakes")No: pools are order-blind.A sequence model over frames (RNN, temporal convolution, transformer) before the head.
Instances interact (a sentence's meaning flips with the next one)Only after context is mixed in.Contextualize instances first (self-attention), then pool.
Strong labels are cheap, or you already have themNo need.Train a frame classifier directly on frame labels; it will beat any weak method.
You only need the bag label and the witness rate is highOptional.Mean-pool the embeddings and train a plain classifier (a linear probe).
Positive instances look like things in negative bagsIt will fail quietly.Better features first (fine-tune the backbone), or some strong labels.
Why can no pool in this lesson learn "the event comes before sound B"?

Chapter 11

Connections

One screen to keep, an honest list of limits, a pooling picker, and where this block sits among the others

The cheat sheet

PartIn one line
The blockInstance scorer + pooling over instances, trained with bag labels only.
Shape signature[B×T×D] → Linear → [B×T×C] → pool over T → [B×C]; keep the [B×T×C] as the detector.
The assumptionA bag is positive if and only if at least one instance is: Y = max yi.
The maths∂L/∂zi = ∂L/∂y · ∂y/∂yi · yi(1 − yi); the pool chooses only ∂y/∂yi.
The poolsMax: one frame learns. Mean: all frames pushed alike. Noisy-OR: saturates. Softmax: a dial. Linear softmax: (2yi − y)/Σy, pushes apart. Attention: learned weights, can hide false alarms.
KnobsThe pool; softmax sharpness r; per-class thresholds; median smoothing; frozen or fine-tuned backbone; batch balancing.
Did it work?A few strong test labels; clip and frame metrics side by side; shuffled-label test; listen to the top frames; ablate the pool.
Failure modesConfounders, look-alike negatives, labels by count or order, interacting instances, tiny witness rates in huge bags.
AlternativesStrong labels; mean-pooled linear probe; counting head; sequence model before the pool.

Pick a pool for your problem

Answer three questions. The tree lights the path to a recommendation, based on the evidence in this lesson.

A rule of thumb drawn from Chapters 3 to 10, not a law. Always confirm with the checks in Chapter 9.

Honest limits

Where it sits

MIL began in 1997 with molecules and hand-made features. Deep learning turned it into a head on a backbone, and the pooling function into the main design choice. Weakly labelled sound event detection made DCASE 2017 Task 4 a testbed, and later DCASE editions kept weak labels for domestic sounds while adding unlabelled and synthetic clips. In pathology, attention MIL on frozen tile embeddings is now a standard recipe for slide-level labels.

Go next

Also on the site: AudioFM for audio foundation models whose frozen features feed heads like this, including their sound event detection results; PANNs for a large Audio Set tagger; CLAP for text-audio embeddings; transfer learning; loss functions for binary cross-entropy; metric design; understanding attention; and the PyTorch training loop.

The takeaway. A label that only says "whether" can still teach "when", if one instance decides the bag. Score the instances, pool them, and choose the pool by the gradient it sends each instance. Then check the instances, because the bag score will not tell you they are wrong.

Now press Present or Teach and explain, from memory, why max pooling misses the tails of an event and mean pooling fills a clip with false alarms. Then go back to the hero and train every pool.

You must find tumour regions in slides labelled only "malignant" or "benign", and tumour tiles are rare. Which starting point fits this lesson's evidence?
Weak Labels and Multiple-Instance Learning. Main source: Y. Wang, J. Li, F. Metze, "A Comparison of Five Multiple Instance Learning Pooling Functions for Sound Event Detection with Weak Labeling" (ICASSP 2019, arXiv:1810.09050). Also: M. Ilse, J. Tomczak, M. Welling, "Attention-based Deep Multiple Instance Learning" (ICML 2018, arXiv:1802.04712); T. Dietterich, R. Lathrop, T. Lozano-Pérez, "Solving the multiple instance problem with axis-parallel rectangles" (Artificial Intelligence 89, 1997); J. Foulds, E. Frank, "A review of multi-instance learning assumptions" (Knowledge Engineering Review, 2010); G. Campanella, V. Silva, T. Fuchs, "Terabyte-scale Deep Multiple Instance Learning for Classification and Localization in Pathology" (2018, arXiv); W. Sultani, C. Chen, M. Shah, "Real-world Anomaly Detection in Surveillance Videos" (CVPR 2018, arXiv); D. Kotzias, M. Denil, N. de Freitas, P. Smyth, "From Group to Individual Labels using Deep Features" (KDD 2015); M. Oquab, L. Bottou, I. Laptev, J. Sivic, "Is object localization for free?" (CVPR 2015).
Back to Gleams