Event Detection Heads

A model can score every tenth of a second: "how likely is a dog bark right now?" But you asked for a list: which events happened, from when to when. The road from frame scores to clean onsets and offsets is part of the model, and it decides most of your score.

Learn how a small head on a frozen encoder scores each frame, why it needs to see the frames around it, how thresholds, median filters, hysteresis and minimum durations turn scores into events, and how to grade the result honestly.

Below is a 40-second stream of frame scores. Green bars are the true events; blue bars are what your settings detect. Turn the four knobs and watch fragments merge, false alarms vanish and event F1 climb.

You need a rough idea of a classifier and a probability. If "frozen encoder plus small head" is new, Backbone + Head is the hub of this series. We build the rest from zero.

Frame scores in, events out

Ready
0.50
0
1 frame
off

A synthetic stream: 400 frames of 100 ms, 9 true events, some quiet, some with short dips, plus short false spikes. The scores are illustrative, made by this page. Scoring uses the DCASE event-based rule: onsets within 200 ms, offsets within 200 ms or 20% of the event's length. Marks: I insertion, D deletion, F fragmented, B boundary error, M merged.

Chapter 0

What happened, and when?

Meet the task in five different streams, and see that every one of them asks for the same output

You install a microphone in a kitchen for a week. Your client asks one question: "Every time the smoke alarm went off, when did it start and when did it stop?" A yes or no per hour is useless. A score per second is not an answer either. They want a list.

That list is the output of event detection: find each occurrence of each class in a stream, with its onset (start time) and offset (end time). It is a different job from classification, which gives one label per clip. Classification answers "what"; detection answers "what, and when".

The same question appears far from kitchens. Pick a stream below.

Five streams, one question

Each stream is a different signal. The green bars are the events a person marked. The readout lists them the way a detector must.

Synthetic signals drawn for illustration: a sound envelope, an ECG trace with one early wide beat, wrist acceleration with keystroke taps, motion energy from a cooking video, and request latency from a web server.

Every stream above gives the same kind of answer. Write one event as a triple:

e
One event: one occurrence of one class.
c
Its class, such as "smoke alarm" or "early beat".
ton
The onset, in seconds from the start of the stream.
toff
The offset. Its length is toff − ton.

Worked: an alarm from 2.50 s to 4.00 s is e = (alarm, 2.50, 4.00), 1.50 s long. Two dog barks inside it add (dog, 2.80, 2.95) and (dog, 3.10, 3.22). Events of different classes may overlap in time. That one fact will shape the head in Chapter 2.

The usual recipe has three parts. A frozen encoder (a pretrained model whose weights we do not change) turns the stream into one vector per short time step. A small trained head turns each vector into class scores. Then post-processing turns scores into the event list. This lesson is about the last two parts, and it treats post-processing as part of the model, because it is.

  1. Frames: chop time into steps, and the shape B × T × D.
  2. The framewise head: a linear map per frame, and why sigmoid, not softmax.
  3. Context: why one frame is not enough, and the receptive field.
  4. Which temporal layer? Convolutions, recurrent layers and attention, and what each costs.
  5. Threshold and median filter: what a window of k removes, derived.
  6. Hysteresis, minimum duration, gaps: three more rules that remove chatter.
  7. Frames to events, and scoring them: collars, segments and overlaps.
  8. Did it work? Threshold curves, PSDS, per-class thresholds and error types.
  9. Build it in PyTorch: the module, the post-processing, frame and weak labels.
  10. Where else, and when not: six fields, the limits, and CTC.
  11. Connections: the cheat sheet and where to go next.
What does an event detector output that a clip classifier does not?

Chapter 1

Frames: the shape of a stream

Chop time into short steps, follow the tensor shapes from raw signal to frame embeddings, and see what the step size costs at the boundaries

A detector cannot reason about "time" in general. It works on frames: short, equally spaced slices of the stream. The distance between the starts of two frames is the hop. Each frame covers a window of the signal, which is often longer than the hop, so neighbouring frames overlap.

T
The number of frames in the clip.
N
The number of raw samples in the clip.
W
The window: how many samples one frame looks at.
H
The hop: how many samples the next frame moves on. Frame i starts at i · H.

Worked: 10 s of audio at 16,000 samples a second is N = 160,000. A 25 ms window is W = 400 and a 10 ms hop is H = 160. T = ⌊159,600 / 160⌋ + 1 = 997 + 1 = 998 frames.

Encoders often pool frames further. A transformer that cuts a spectrogram into patches 16 frames wide gives one output vector per 160 ms column; the AudioFM lesson shows such a frozen encoder probed column by column. The DCASE 2023 sound event detection baseline pools its 16 ms hop four times, so each output frame is 64 ms DESED baseline config. Whatever the encoder, the head sees the same shape.

Raw clipsB × N samples
FramesB × T × F features (for example 64 mel bands)
Frozen encoderB × T × D vectors
HeadB × T × C scores
Post-processinga list of (class, onset, offset) per clip

Here B is the batch size (clips at once), D the embedding width (768 for a ViT-Base encoder) and C the number of classes. Only the last two steps are ours to train and tune.

The hop sets a hard limit on timing. A detected event can only start or stop at a frame edge. So a true onset that falls between two edges is reported late or early. Slide the hop below to see by how much.

The hop is a ruler

The true event runs from 1.23 s to 2.71 s. A frame counts as "on" if its centre is inside the event. The blue bar is the best a perfect head could report.

100 ms

Exact arithmetic. With centre-based labels, a boundary can be wrong by at most half a hop.

With a perfect head, each boundary error is at most H / 2. At a 64 ms hop that is 32 ms. That is far inside the 200 ms tolerance most sound benchmarks allow (Chapter 7). At a 500 ms hop it is 250 ms, and a perfect head would already fail them.

Your encoder outputs one vector per 320 ms. With a perfect head and centre-based labels, what is the largest onset error?

Chapter 2 Why it works

The framewise head

Apply one small classifier to every frame, see why each class needs its own sigmoid, and train both kinds of head in the page

The simplest head is a framewise linear head: the same linear map, applied to every frame on its own. It is a clip classifier's head, used T times. Each class c has a weight vector wc and a bias bc:

pt,c
The probability that class c is active at frame t.
σ
The sigmoid, 1 / (1 + e−z). It squashes any score z into 0 to 1, for each class on its own.
wc, bc
The trained weights for class c: D numbers and one bias. The whole head has D × C + C weights.
ht
The frozen encoder's vector for frame t: D numbers.

Worked: ht = (1, −0.5, 2), wc = (0.8, 0.4, −0.3), bc = 0.1. z = 0.8 − 0.2 − 0.6 + 0.1 = 0.1, so p = 1 / (1 + e−0.1) = 0.525. With D = 768 and C = 10, the head has 7,690 weights.

Training grades every frame against its frame label yt,c (1 if class c is active at frame t) with binary cross-entropy, averaged over all frames and classes:

L
The loss. Its gradient for the score zt,c is simply pt,c − yt,c.

Worked: with p = 0.525 and y = 1, this frame's loss is −ln 0.525 = 0.644, and its gradient is 0.525 − 1 = −0.475: push this score up.

Why a sigmoid per class, not one softmax

A softmax makes the class probabilities of one frame add up to 1. That suits "which one of these?" But a dog can bark while someone speaks. When two classes are both active, a softmax must split its single unit of probability between them. Each gets about 0.5, and a threshold of 0.5 misses one or both. A sigmoid per class lets both reach 0.9. Sound with overlapping classes is called polyphonic. Train both heads below on the same data.

Two classes at once

Speech (pink) and a dog (purple) sometimes sound together; those frames are bracketed. Both heads are linear and trained here for 400 steps. Switch heads and watch the bracketed frames.

Synthetic 4-number frame vectors: each active class adds its own direction, plus noise. The softmax head has three outputs (background, speech, dog) and is trained with the target split 50/50 when both are active, the best a softmax can do. Test frames are new draws. Illustrative numbers.

Why do sound event detectors use one sigmoid per class rather than a softmax across classes?

Chapter 3 Why it works

Context: the receptive field

See why one frame is too little evidence, train a head that looks at its neighbours, and derive how far a stack of layers can see

One frame is a short, noisy look. A single 100 ms slice of a dog bark can look like a door knock. But events last. If the frames on both sides also look like a bark, this one probably is too. Using the neighbours is like asking several witnesses instead of one.

The simplest way to use neighbours is a temporal convolution: the score at frame t is a weighted sum of the k frames centred on t. The number of input frames that can affect one output is its receptive field. Train heads with wider and wider windows below.

How many neighbours?

Each frame vector is noisy. The head is logistic regression on a window of k frames, trained in your browser on 400 frames and tested on 2,000 new ones. Slide k.

1 frame

Synthetic 3-number frame vectors, noise larger than the signal on any one frame. Events last 5 to 32 frames. "Raw events" counts on-runs at a 0.5 threshold with no post-processing. Illustrative, but trained and tested for real.

The error falls fast at first: three frames already beat one by a lot. Then it flattens and rises again. Two things push it up. Near a boundary, a wide window mixes frames from both sides, so edges blur. And a wider window has more weights to fit from the same 400 frames. The best window is about as wide as the shortest events you care about.

Receptive field arithmetic

One wide kernel is costly. Instead, stack small kernels. A dilated convolution skips frames: with dilation d, a kernel of size k reads frames t − d, t and t + d (for k = 3). Each layer widens the view. The receptive field of a stack of L layers is:

R
How many input frames one output frame depends on.
Σ over layers
Each layer adds its own reach to the reach of the layers below.
kl
Kernel size of layer l: how many taps it reads.
dl
Dilation of layer l: the spacing between its taps.
s1·…·sl−1
The product of the strides below layer l. With stride 1 everywhere (usual for heads, to keep T) it is 1.

Worked: three layers, k = 3, dilations 1, 2, 4, stride 1. R = 1 + 2 × 1 + 2 × 2 + 2 × 4 = 1 + 2 + 4 + 8 = 15 frames. At 64 ms per frame that is 960 ms of context, with only 3 × 3 = 9 taps.

Why add (k − 1) · d? A kernel of k taps spaced d apart spans (k − 1) · d + 1 frames. Stacking layers adds their spans, minus the shared centre frame each time. That gives the formula.

Four conv layers, kernel 3, dilations 1, 2, 4 and 8, stride 1. What is the receptive field?

Chapter 4 Why it works

Which temporal layer?

Compare a conv stack, a causal conv, a bidirectional GRU and a transformer layer by what they see, what they cost and whether they can run live

Three families of layers give a head context. Each makes a different trade.

For live use there is a fourth option. A causal convolution reads only the past and the present. It never waits for the future, so it can run on a live stream.

What each layer sees, and costs

The top dot is one output frame. Lines show which input frames it depends on. Frames to its right are in the future. Change the hidden width H and clip length T.

256
250 frames

Exact counts for D = 768 encoder width and C = 10 classes, ignoring normalisation layers. GRU weights follow PyTorch's layout: 3H(D + H) + 6H per direction. The transformer row is one layer with a D-to-H input projection, 4 attention matrices, a 4H feed-forward block and two LayerNorms. Multiply-adds are per frame.

Here are the counts behind the device, for D = 768, H = 256, C = 10.

k · Din · H + H
Each of H output channels has a k-by-Din kernel and a bias.
3H(D + H) + 6H
A GRU has three gates. Each reads the input (D) and the memory (H), with two bias vectors.

Worked, conv stack: 3 · 768 · 256 + 256 = 590,080, then two layers of 3 · 256 · 256 + 256 = 196,864, then 256 · 10 + 10 = 2,570. Total 986,378 weights. BiGRU: 2 × (3 · 256 · 1,024 + 1,536) + 5,130 = 1,581,066.

The conv stack's view is centred, so it needs 7 future frames: at 64 ms frames that is 448 ms of delay before it can score "now". A BiGRU or transformer needs the whole clip first. The causal conv needs none, but loses the frames after an onset, which are often the clearest evidence.

You must flag glass breaking on a live doorbell camera within 300 ms. Which head fits?

Chapter 5 Why it works

Threshold and median filter

Turn scores into on and off, then derive exactly which short runs a median window of k removes and which it keeps

The first post-processing step is a threshold: a frame is "on" for class c if pt,c ≥ θ. That gives a row of 0s and 1s per class. Real rows flicker. A one-frame spike becomes a whole false event. A one-frame dip splits a true event in two.

A median filter of width k replaces each frame with the median of the k frames centred on it. On 0s and 1s the median is a majority vote. Click frames below to build your own row, then widen the window.

A majority vote in a sliding window

Top row: your thresholded frames (tap a cell to flip it). Bottom row: after the median filter. Runs are labelled with their length.

3

Exact. Edges are handled by repeating the first and last frame, as many libraries do.

What a window of k removes, derived

Let k = 2m + 1 (k is odd). Take a run of L ones with long runs of zeros on both sides.

k
The window width, in frames.
m
Half the window, not counting the centre: m = (k − 1) / 2.
L
The length of a run of ones (a blip) or of zeros (a gap) between long runs.

Worked: the DCASE 2023 baseline uses a median window of 7 output frames DESED baseline config. So m = 3. With 64 ms frames it deletes blips up to 3 × 64 = 192 ms and fills gaps up to 192 ms, without moving the boundaries of anything longer.

The rule is symmetric: gaps are runs of zeros, so the same window fills them. That is why one filter fixes both false spikes and fragmented events. The price: true events of m frames or fewer vanish too. Choose m below the shortest event you must keep.

A median window of 9 frames is applied to thresholded frames. Which runs are removed?

Chapter 6 Why it works

Hysteresis, minimum duration and gaps

Use two thresholds so a hovering score stops chattering, then drop events that are too short and merge those split by short gaps

A median filter fixes short flicker. It fails when a score hovers near the threshold for a long time: 0.52, 0.47, 0.55, 0.49. Every crossing starts or ends an event. This is chatter.

A thermostat solves the same problem. It turns the heat on at 19 degrees and off at 21, not on and off at 20. Two thresholds, with a dead band between them, are called hysteresis. For detection: switch on when the score rises to the high threshold, and stay on until it falls below the low one.

st
The on/off state at frame t. It depends on the score and on the state just before.
θhi
The high threshold: evidence needed to start an event.
θlo
The low threshold: evidence needed to continue one. θlo < θhi; the difference is the hysteresis gap.

Worked, θhi = 0.6, θlo = 0.35, scores 0.1, 0.2, 0.7, 0.4, 0.8, 0.9, 0.3, 0.6, 0.8, 0.2, 0.1, 0.9, 0.1. States: 0 0 1 1 1 1 0 1 1 0 0 1 0. The 0.4 kept the event alive; with one threshold at 0.6 it would have split. Chapter 9 runs this exact row.

Two more rules finish the job. A minimum duration drops any event shorter than a set number of frames: a 100 ms "alarm" is not an alarm. Gap merging joins two events of the same class if the gap between them is shorter than a set length: a dog that barks, pauses 200 ms and barks again may count as one event, depending on how your labels were made.

Stop the chatter

Twelve seconds of scores with three true events: a strong one with a dip, a quiet one that hovers, and one split by a short silence. There is also one false spike.

0.55
0.30
off
off

120 frames of 100 ms, hand-built to show each failure once. Order of operations: thresholds, then gap merging, then minimum duration.

Order matters. Merge gaps before dropping short events, or you may drop two halves of a real event that merging would have joined. Many pipelines run: threshold (or hysteresis), median filter, merge gaps, drop short. The hero uses that order.

With hysteresis, θhi = 0.7 and θlo = 0.4. The detector is on, and the next score is 0.55. What happens?

Chapter 7

From frames to events, and scoring them

Read onsets and offsets off the frames, then grade the same detection three ways: collars, one-second segments and overlap

After post-processing, each class has a clean row of 0s and 1s. Each run of ones becomes one event. If a run starts at frame s and ends just before frame e, then onset = s · H and offset = e · H, in seconds. This is run-length decoding. Now: is the event list right?

Event-based scoring with collars

The event-based metric matches whole events. A detected event counts as a hit, a true positive (TP), if its class matches and its boundaries fall within a tolerance called a collar Mesaros et al. 2016. In the DCASE sound event detection challenges, the onset collar is 200 ms and the offset collar is the larger of 200 ms and 20% of the true event's length DCASE 2019 Task 4 baseline:

t̂ − t
Detected time minus true time.
tc
The collar: 0.2 s in DCASE.
ρ
The share of the true length allowed at the offset: 0.2 in DCASE Task 4. Offsets are harder to mark than onsets.

Unmatched detections are false positives (FP); unmatched true events are false negatives (FN). The F1 score balances them:

Worked: a true event from 2.0 s to 5.0 s (3.0 s long). The onset must land in 1.8 to 2.2 s. The offset tolerance is max(0.2, 0.2 × 3.0) = 0.6 s, so 4.4 to 5.6 s. A detection from 2.15 to 4.5 s: TP, F1 = 1. A detection from 2.3 to 5.0 s: onset 0.3 s late, so it is an FP and the truth is an FN. F1 = 0 / (0 + 1 + 1) = 0.

That last case is the double penalty: one slightly late event costs both a false alarm and a miss. Splitting one event in two is worse still. Try it below, and compare with two other ways to score.

One detection, three judges

The truth runs from 2.0 s to 5.0 s. Teal bands are the collars. Move the detection's onset and offset, or split it into pieces.

+0.10 s
−0.30 s

Exact. Collar rule: 200 ms onset, max(200 ms, 20% of length) offset. Segments: 1 s, as in DCASE's segment-based metric. Overlap rule: the PSDS criteria of Chapter 8 with both tolerances at 0.5.

Two other judges

Segment-based scoring cuts time into fixed segments (1 s in DCASE) and asks, per segment and class, "active or not?" Mesaros et al. 2016. It ignores how many events there are. A split event loses nothing. It suits classes whose boundaries are fuzzy by nature, like rain or traffic.

Overlap-based scoring, from the PSDS framework Bilen et al. 2020, replaces collars with two tests. The detection tolerance criterion (DTC) keeps a detection only if at least a share ρDTC of its length overlaps true events of its class; the rest are false positives. The ground truth intersection criterion (GTC) counts a true event as found if kept detections cover at least a share ρGTC of it. A fragmented detection still finds the event.

This matters in practice. Bilen et al. re-scored three DCASE 2019 systems. The challenge baseline scored 23.70% collar-based F1 but 61.38% with DTC/GTC at 0.5; the authors traced many of its false positives to split detections Bilen et al. 2020 · Table 1. The ranking of the three systems changed.

A true 3 s event is detected as two pieces that together cover 90% of it. What does collar-based scoring report?

Chapter 8 Did it work?

Did it work? Thresholds, PSDS and error types

Sweep the threshold per class, score across all thresholds at once with PSDS, and break the errors down by kind

An F1 score belongs to one operating point: one threshold and one set of post-processing settings. Change the threshold and the F1 changes. So a single F1 mixes two questions. Are the scores good? And was the threshold well chosen? Bilen et al. make this point about SED metrics Bilen et al. 2020 · §1.

Three plots answer the questions separately. Switch between them below.

Three classes, one threshold?

Twenty minutes of scores for three classes. The dog head is well calibrated, the alarm head scores everything high, and the speech head is timid. Slide the shared threshold, or let each class pick its own.

0.50

Synthetic scores, 12,000 frames of 100 ms per class, post-processed with a median window of 5 and a 3-frame minimum. F1 uses the collar rule. The PSD-ROC uses DTC = GTC = 0.5, no cross-trigger cost and emax = 100 false positives per hour, the settings of Bilen et al.'s Figure 3a. Per-class thresholds are tuned on these same 20 minutes, which flatters them; tune on a validation set in practice.

Per-class thresholds

Heads are rarely calibrated alike. A class with plenty of training data often scores high everywhere; a rare class scores low. One shared threshold is then too high for one class and too low for another. Picking the best threshold per class, on a validation set (held-out data, not the test set), often adds more F1 than a bigger head does. Imbalance and Calibration Heads goes further.

PSDS: score every threshold at once

The polyphonic sound detection score (PSDS) removes the threshold from the question Bilen et al. 2020 · §3. Sweep the threshold. At each one, measure each class's TP ratio (the share of true events found, by DTC/GTC) and its false positive rate per hour of audio. Plot one against the other: that is the class's PSD-ROC curve. Average the curves across classes, then take the area up to a maximum false-positive rate emax:

1 / emax
Normalises the area so a perfect system scores 1.
r(e)
The effective TP ratio when false positives are capped at e per hour.
μTP, σTP
Mean and standard deviation of the TP ratio across classes.
αST
How much to penalise uneven classes. 0 ignores it; 1 subtracts one standard deviation.

Worked: one class's curve is a staircase: TP ratio 0.4 up to 20 FPs per hour, 0.6 up to 60, and 0.7 up to 100. Area = 0.4 × 20 + 0.6 × 40 + 0.7 × 40 = 8 + 24 + 28 = 60. PSDS = 60 / 100 = 0.60. With two classes at 0.5 and 0.7 everywhere, μ = 0.6 and σ = 0.1, so αST = 1 gives r = 0.5.

The full definition also has a cross-trigger term: a false positive that lands on another class's true event (a "dog" fired by a cat), weighted by αCT Bilen et al. 2020 · eq. 6. The DCASE 2023 baseline reports two settings: scenario 1 with DTC = GTC = 0.7, αCT = 0, αST = 1, strict on timing; and scenario 2 with DTC = GTC = 0.1, a cross-trigger tolerance of 0.3, αCT = 0.5, αST = 1, loose on timing but strict on confusions. Both use emax = 100 per hour DESED baseline code.

The error types

Counts tell you which knob to turn. Match each true event to the detections that overlap it:

For many classes, also plot confusion over time: a timeline with each class's truth and detections stacked, so you can see a "speech" detection sitting on a "TV" event. Cross-triggers of this kind point to classes the encoder cannot tell apart, which no threshold fixes.

Two systems have the same F1 at a threshold of 0.5. What does PSDS add?

Chapter 9 Build it

Build it in PyTorch

Trace the shapes through the whole detector, write the frozen encoder and conv head, the post-processing in numpy, and train with frame or weak labels

By hand first. Chapter 2 scored one frame: z = 0.1, p = 0.525, loss 0.644, gradient −0.475. Chapter 6 ran the post-processing by hand on 13 scores. Now step through the shapes of the full detector.

Follow the shapes

B = 16 clips, T = 250 frames, F = 64 input features, D = 768, H = 256, C = 10 classes. Press Next. Switch the labels to see where the loss attaches.

Shapes are exact for the code below with these sizes.

From scratch. The head is a dilated conv stack with receptive field 15, then a 1 × 1 convolution, which is a linear map applied to every frame. The wrapper freezes the encoder and keeps it in eval mode.

import torch
import torch.nn as nn

class TemporalConvHead(nn.Module):
    """(B, T, D) frame embeddings -> (B, T, C) frame logits."""
    def __init__(self, d, c, hidden=256, k=3, dilations=(1, 2, 4), p=0.1):
        super().__init__()
        layers, ch = [], d
        for dil in dilations:                    # receptive field 1 + (k-1)*sum(dil) = 15
            layers += [nn.Conv1d(ch, hidden, k, padding=dil * (k - 1) // 2, dilation=dil),
                       nn.BatchNorm1d(hidden), nn.GELU(), nn.Dropout(p)]
            ch = hidden
        self.body = nn.Sequential(*layers)
        self.out = nn.Conv1d(hidden, c, kernel_size=1)   # one linear map per frame

    def forward(self, h):                        # h: (B, T, D)
        x = h.transpose(1, 2)                    # (B, D, T): Conv1d wants channels first
        x = self.body(x)                         # (B, H, T): padding keeps T
        return self.out(x).transpose(1, 2)       # (B, T, C) logits

class EventDetector(nn.Module):
    """Frozen encoder + trainable temporal head."""
    def __init__(self, encoder, d, c):
        super().__init__()
        self.encoder = encoder.requires_grad_(False).eval()
        self.head = TemporalConvHead(d, c)

    def train(self, mode=True):                  # model.train() must not wake the encoder
        super().train(mode)
        self.encoder.eval()
        return self

    def forward(self, x):                        # x: (B, T, F) input frames
        with torch.no_grad():
            h = self.encoder(x)                  # (B, T, D) frozen embeddings
        return self.head(h)                      # (B, T, C); sigmoid only at inference

Why each choice. The padding dil * (k - 1) // 2 keeps T, so output frame t lines up with input frame t. BatchNorm and dropout regularise a head that may have more weights than you have labelled frames. The head returns logits, not probabilities, because BCEWithLogitsLoss applies the sigmoid inside the loss in a numerically safe way.

Training, in the same file. The stand-in encoder lets it run with no downloads; swap in a real one. The labels are random, so this is a smoke test: a falling loss here means the head is memorising 16 clips, which is the shuffled-label check of Chapter 8.

torch.manual_seed(0)
B, T, F, D, C = 16, 250, 40, 64, 3
encoder = nn.Sequential(nn.Linear(F, D), nn.GELU(), nn.Linear(D, D))  # stand-in
model = EventDetector(encoder, D, C)
x = torch.randn(B, T, F)                         # 16 clips of 250 frames
y = (torch.rand(B, T, C) < 0.1).float()          # frame labels: 1 = class active
bce = nn.BCEWithLogitsLoss()
opt = torch.optim.AdamW(model.head.parameters(), lr=1e-3, weight_decay=1e-2)
model.train()
for step in range(200):
    logits = model(x)                            # (B, T, C)
    loss = bce(logits, y)                        # strong labels: grade every frame
    # weak labels (one 0/1 per clip and class): pool over time first
    # loss = bce(logits.amax(dim=1), y.amax(dim=1))   # (B, C) vs (B, C)
    opt.zero_grad()
    loss.backward()
    opt.step()
model.eval()
probs = torch.sigmoid(model(x))                  # (B, T, C) in [0, 1]
print(probs.shape, f"loss {loss.item():.3f}")

Frame labels or weak labels? Strong labels mark every onset and offset, so every frame gets a grade. They are slow to make. Weak labels only say which classes occur somewhere in a clip. To train on them, pool the frame scores over time (max, mean or attention) into one clip score and grade that. The pooling decides which frames get the gradient: max pooling sends it only to the top frame, so the head learns "where" as a side effect. That is multiple instance learning; see Weak Labels and MIL and Attention-Pooling Heads. DCASE Task 4 trains on a mix of strong, weak and unlabelled clips for this reason.

Post-processing, in numpy, per class:

import numpy as np

def hysteresis(p, hi, lo):
    """Switch on at p >= hi, stay on while p >= lo."""
    on, out = False, np.zeros(len(p), dtype=bool)
    for t, v in enumerate(p):
        on = v >= lo if on else v >= hi
        out[t] = on
    return out

def median_binary(b, k):
    """Median of a 0/1 window = majority vote. k must be odd."""
    if k <= 1:
        return b
    m = k // 2
    padded = np.pad(b.astype(int), m, mode="edge")
    win = np.lib.stride_tricks.sliding_window_view(padded, k)  # (T, k)
    return win.sum(axis=1) > m

def to_events(b, hop, min_frames=0, max_gap=0):
    """0/1 frames -> [(onset_s, offset_s)]: merge short gaps, drop short events."""
    d = np.diff(np.concatenate([[0], b.astype(int), [0]]))
    starts, ends = np.flatnonzero(d == 1), np.flatnonzero(d == -1)  # [start, end)
    ev = []
    for s, e in zip(starts, ends):
        if ev and s - ev[-1][1] <= max_gap:
            ev[-1][1] = e                        # close a short gap
        else:
            ev.append([s, e])
    return [(round(float(s * hop), 3), round(float(e * hop), 3))
            for s, e in ev if e - s >= min_frames]

p = np.array([.1, .2, .7, .4, .8, .9, .3, .6, .8, .2, .1, .9, .1])
b = median_binary(hysteresis(p, hi=.6, lo=.35), k=3)
print(b.astype(int))                  # [0 0 1 1 1 1 1 1 1 0 0 0 0]
print(to_events(b, hop=0.1, min_frames=2))  # [(0.2, 0.9)]

Trace it against Chapter 6. Hysteresis gave 0 0 1 1 1 1 0 1 1 0 0 1 0. The median of 3 fills the one-frame gap at frame 6 and removes the one-frame blip at frame 11. One event remains: 0.2 s to 0.9 s.

The library versions. scipy filters along time in one line; sed_eval and psds_eval score the result with the same settings DCASE uses DESED baseline code.

from scipy.ndimage import median_filter
smooth = median_filter(probs_np, size=(7, 1))     # (T, C): window 7 along time only

import sed_eval
metric = sed_eval.sound_event.EventBasedMetrics(
    event_label_list=classes, t_collar=0.200, percentage_of_length=0.2)

from psds_eval import PSDSEval
psds = PSDSEval(ground_truth=gt_df, metadata=dur_df,
                dtc_threshold=0.7, gtc_threshold=0.7, cttc_threshold=0.3)
# add one operating point per threshold, then:
# psds.psds(alpha_ct=0, alpha_st=1, max_efpr=100)
You only have clip-level labels ("a dog barks somewhere in this clip"). How do you train the framewise head?

Chapter 10 Where else When not

Where else it works, and when not to use it

Move the same block to six fields, see what changes and what stays, and learn when a pooled head, segment metrics, a causal head or CTC is the better choice

The block moves across fields with three swaps: an encoder pretrained near the new data, a hop that fits the events' timescale, and post-processing rules that encode how long events last. The head and the metrics stay the same.

FieldStream and eventsWhat changes
Audio SEDMicrophone audio; alarms, barks, dishes, speechPolyphonic, so sigmoid per class; DCASE collars and PSDS.
ECG beats and arrhythmiaHeart electrical signal; each beat, and runs of abnormal beatsBeats are near-instants: detect peaks, match within a small window. Arrhythmia episodes are intervals, with a minimum duration.
Wearable activityWrist or phone IMU; walking, typing, single keystrokesActivities are long, so segment metrics fit; keystrokes are short taps that need a fine hop.
Video action localisationFrames from a frozen video encoder; "pour", "stir"Boundaries are fuzzy, so scoring uses overlap (temporal IoU) at several thresholds, not tight collars.
Speech activity (VAD)Audio; one class, "someone is speaking"Must run live: causal head; a "hangover" keeps speech on through short pauses, which is hysteresis plus gap merging.
Industrial and log anomaliesSensor or metrics streams; fault and incident intervalsFew or no labels: score frames by anomaly, then the same threshold, hysteresis and minimum duration.

Frame head, or something else?

Answer four questions. The card shows the block to start with and the one rule that decided it.

Rules of thumb for a first build, not laws. Confirm with the checks of Chapter 8.

When not to use it

Avoid it whenBecauseUse instead
Only clip-level presence mattersOnsets you never use cost labels and tuningA pooled clip head (attention pooling)
Events of one class overlap heavily (a crowd of voices)One 0/1 row per class cannot say "two at once"; runs mergeCount or density regression per frame, or source separation first
Boundaries are fuzzy by nature (rain, traffic)Collars punish disagreement that is not an errorThe same head, scored segment-based or with loose DTC/GTC
You must answer within a few frames, liveBidirectional context, wide medians and gap merging all wait for the futureA causal head, causal smoothing, and a stated latency budget
Labels give the order of tokens, not their timesThere are no frame targets to gradeCTC, below

CTC: when you know the order, not the times

Speech transcripts say "the cat sat" but not when each word starts. Connectionist temporal classification (CTC; Graves et al., 2006) trains a framewise head on such unaligned label sequences. The head outputs, per frame, a distribution over the tokens plus a blank. The loss sums the probability of every frame-level path that collapses to the target sequence (merge repeats, then drop blanks), using dynamic programming. CTC fits when the targets are a sequence of discrete tokens that do not overlap, such as words, phonemes, characters in handwriting, or ordered steps in a procedure. Its outputs tend to be peaky, a single frame per token, so it gives order and rough position but not reliable onsets and offsets. When you need durations, or classes overlap, use frame labels and the head of this lesson.

You must report, for a 24-hour recording, only whether each hour contained any glass breaking. Which head should you start with?

Chapter 11

Connections

Get starting settings for your field, keep the one-screen cheat sheet, know the limits, and follow the series onward

Everything in this lesson fits on one card. Pick your field to get a first set of settings. Then use the cheat sheet to adjust them.

Your first settings

Choose a field. The card gives a hop, a head, post-processing defaults and the metric to report.

Starting points to tune on validation data, not final answers. The sound defaults echo the DCASE 2023 baseline's 64 ms frames and median of 7.

The cheat sheet

PartIn one line
The blockFrozen encoder → temporal head → sigmoid per class → post-processing → event list.
Shape signatureB×N → B×T×D → B×T×C → [(class, onset, offset)]; onset = start frame × hop.
Maths in one linept,c = σ(wc·ht + bc) on context; R = 1 + Σ(k − 1)d; a median of 2m + 1 removes runs ≤ m.
KnobsHop; receptive field; per-class threshold; hysteresis gap; median window; merge gap; minimum duration.
Failure modesInsertions, deletions, fragmentation, merges, boundary errors; softmax on overlapping classes; tuning on the test set.
ChecksF1 vs threshold per class; PSDS; error-type counts; confusion over time; shuffled labels; a linear framewise probe as baseline.
AlternativesPooled head if only "if"; segment metrics if fuzzy; causal head if live; CTC if only the order is labelled.

Honest limits

Where this sits, and where to go next

This lesson is part of The Builder's Toolkit. Its hub is Backbone + Head. The neighbours:

Already on the site:

The takeaway. A detector is a framewise head plus a set of rules about time. The head scores each frame using its neighbours. The rules (threshold, median, hysteresis, gap merging, minimum duration) turn scores into events, and they move event F1 more than most changes to the head. Tune them per class on validation data, and grade with the metric that matches your use.
Your detector finds every alarm but splits most of them into 3 to 5 pieces. Which change helps most?

Now press Present or Teach and explain, out loud and from memory, how a row of frame scores becomes a list of events, and what each post-processing rule removes. Then go back to the hero, start from the raw threshold, and fix the errors one knob at a time.

Event Detection Heads, part of The Builder's Toolkit. Sources: Ç. Bilen, G. Ferroni, F. Tuveri, J. Azcarreta, S. Krstulović, "A Framework for the Robust Evaluation of Sound Event Detection" (ICASSP 2020, arXiv); A. Mesaros, T. Heittola, T. Virtanen, "Metrics for Polyphonic Sound Event Detection" (Applied Sciences 6(6):162, 2016, doi); the DCASE Task 4 baseline code and configuration (DESED_task; N. Turpault et al., DCASE Workshop 2019); A. Graves, S. Fernández, F. Gomez, J. Schmidhuber, "Connectionist Temporal Classification" (ICML 2006); W. Luo, Y. Li, R. Urtasun, R. Zemel, "Understanding the Effective Receptive Field in Deep Convolutional Neural Networks" (NeurIPS 2016, arXiv). All toy streams and their numbers are synthetic and illustrative.
Back to Gleams