A model can score every tenth of a second: "how likely is a dog bark right now?" But you asked for a list: which events happened, from when to when. The road from frame scores to clean onsets and offsets is part of the model, and it decides most of your score.
Learn how a small head on a frozen encoder scores each frame, why it needs to see the frames around it, how thresholds, median filters, hysteresis and minimum durations turn scores into events, and how to grade the result honestly.
Below is a 40-second stream of frame scores. Green bars are the true events; blue bars are what your settings detect. Turn the four knobs and watch fragments merge, false alarms vanish and event F1 climb.
You need a rough idea of a classifier and a probability. If "frozen encoder plus small head" is new, Backbone + Head is the hub of this series. We build the rest from zero.
A synthetic stream: 400 frames of 100 ms, 9 true events, some quiet, some with short dips, plus short false spikes. The scores are illustrative, made by this page. Scoring uses the DCASE event-based rule: onsets within 200 ms, offsets within 200 ms or 20% of the event's length. Marks: I insertion, D deletion, F fragmented, B boundary error, M merged.
Chapter 0
Meet the task in five different streams, and see that every one of them asks for the same output
You install a microphone in a kitchen for a week. Your client asks one question: "Every time the smoke alarm went off, when did it start and when did it stop?" A yes or no per hour is useless. A score per second is not an answer either. They want a list.
That list is the output of event detection: find each occurrence of each class in a stream, with its onset (start time) and offset (end time). It is a different job from classification, which gives one label per clip. Classification answers "what"; detection answers "what, and when".
The same question appears far from kitchens. Pick a stream below.
Each stream is a different signal. The green bars are the events a person marked. The readout lists them the way a detector must.
Synthetic signals drawn for illustration: a sound envelope, an ECG trace with one early wide beat, wrist acceleration with keystroke taps, motion energy from a cooking video, and request latency from a web server.
Every stream above gives the same kind of answer. Write one event as a triple:
Worked: an alarm from 2.50 s to 4.00 s is e = (alarm, 2.50, 4.00), 1.50 s long. Two dog barks inside it add (dog, 2.80, 2.95) and (dog, 3.10, 3.22). Events of different classes may overlap in time. That one fact will shape the head in Chapter 2.
The usual recipe has three parts. A frozen encoder (a pretrained model whose weights we do not change) turns the stream into one vector per short time step. A small trained head turns each vector into class scores. Then post-processing turns scores into the event list. This lesson is about the last two parts, and it treats post-processing as part of the model, because it is.
Chapter 1
Chop time into short steps, follow the tensor shapes from raw signal to frame embeddings, and see what the step size costs at the boundaries
A detector cannot reason about "time" in general. It works on frames: short, equally spaced slices of the stream. The distance between the starts of two frames is the hop. Each frame covers a window of the signal, which is often longer than the hop, so neighbouring frames overlap.
Worked: 10 s of audio at 16,000 samples a second is N = 160,000. A 25 ms window is W = 400 and a 10 ms hop is H = 160. T = ⌊159,600 / 160⌋ + 1 = 997 + 1 = 998 frames.
Encoders often pool frames further. A transformer that cuts a spectrogram into patches 16 frames wide gives one output vector per 160 ms column; the AudioFM lesson shows such a frozen encoder probed column by column. The DCASE 2023 sound event detection baseline pools its 16 ms hop four times, so each output frame is 64 ms DESED baseline config. Whatever the encoder, the head sees the same shape.
Here B is the batch size (clips at once), D the embedding width (768 for a ViT-Base encoder) and C the number of classes. Only the last two steps are ours to train and tune.
The hop sets a hard limit on timing. A detected event can only start or stop at a frame edge. So a true onset that falls between two edges is reported late or early. Slide the hop below to see by how much.
The true event runs from 1.23 s to 2.71 s. A frame counts as "on" if its centre is inside the event. The blue bar is the best a perfect head could report.
Exact arithmetic. With centre-based labels, a boundary can be wrong by at most half a hop.
With a perfect head, each boundary error is at most H / 2. At a 64 ms hop that is 32 ms. That is far inside the 200 ms tolerance most sound benchmarks allow (Chapter 7). At a 500 ms hop it is 250 ms, and a perfect head would already fail them.
Chapter 2 Why it works
Apply one small classifier to every frame, see why each class needs its own sigmoid, and train both kinds of head in the page
The simplest head is a framewise linear head: the same linear map, applied to every frame on its own. It is a clip classifier's head, used T times. Each class c has a weight vector wc and a bias bc:
Worked: ht = (1, −0.5, 2), wc = (0.8, 0.4, −0.3), bc = 0.1. z = 0.8 − 0.2 − 0.6 + 0.1 = 0.1, so p = 1 / (1 + e−0.1) = 0.525. With D = 768 and C = 10, the head has 7,690 weights.
Training grades every frame against its frame label yt,c (1 if class c is active at frame t) with binary cross-entropy, averaged over all frames and classes:
Worked: with p = 0.525 and y = 1, this frame's loss is −ln 0.525 = 0.644, and its gradient is 0.525 − 1 = −0.475: push this score up.
A softmax makes the class probabilities of one frame add up to 1. That suits "which one of these?" But a dog can bark while someone speaks. When two classes are both active, a softmax must split its single unit of probability between them. Each gets about 0.5, and a threshold of 0.5 misses one or both. A sigmoid per class lets both reach 0.9. Sound with overlapping classes is called polyphonic. Train both heads below on the same data.
Speech (pink) and a dog (purple) sometimes sound together; those frames are bracketed. Both heads are linear and trained here for 400 steps. Switch heads and watch the bracketed frames.
Synthetic 4-number frame vectors: each active class adds its own direction, plus noise. The softmax head has three outputs (background, speech, dog) and is trained with the target split 50/50 when both are active, the best a softmax can do. Test frames are new draws. Illustrative numbers.
Chapter 3 Why it works
See why one frame is too little evidence, train a head that looks at its neighbours, and derive how far a stack of layers can see
One frame is a short, noisy look. A single 100 ms slice of a dog bark can look like a door knock. But events last. If the frames on both sides also look like a bark, this one probably is too. Using the neighbours is like asking several witnesses instead of one.
The simplest way to use neighbours is a temporal convolution: the score at frame t is a weighted sum of the k frames centred on t. The number of input frames that can affect one output is its receptive field. Train heads with wider and wider windows below.
Each frame vector is noisy. The head is logistic regression on a window of k frames, trained in your browser on 400 frames and tested on 2,000 new ones. Slide k.
Synthetic 3-number frame vectors, noise larger than the signal on any one frame. Events last 5 to 32 frames. "Raw events" counts on-runs at a 0.5 threshold with no post-processing. Illustrative, but trained and tested for real.
The error falls fast at first: three frames already beat one by a lot. Then it flattens and rises again. Two things push it up. Near a boundary, a wide window mixes frames from both sides, so edges blur. And a wider window has more weights to fit from the same 400 frames. The best window is about as wide as the shortest events you care about.
One wide kernel is costly. Instead, stack small kernels. A dilated convolution skips frames: with dilation d, a kernel of size k reads frames t − d, t and t + d (for k = 3). Each layer widens the view. The receptive field of a stack of L layers is:
Worked: three layers, k = 3, dilations 1, 2, 4, stride 1. R = 1 + 2 × 1 + 2 × 2 + 2 × 4 = 1 + 2 + 4 + 8 = 15 frames. At 64 ms per frame that is 960 ms of context, with only 3 × 3 = 9 taps.
Why add (k − 1) · d? A kernel of k taps spaced d apart spans (k − 1) · d + 1 frames. Stacking layers adds their spans, minus the shared centre frame each time. That gives the formula.
Chapter 4 Why it works
Compare a conv stack, a causal conv, a bidirectional GRU and a transformer layer by what they see, what they cost and whether they can run live
Three families of layers give a head context. Each makes a different trade.
For live use there is a fourth option. A causal convolution reads only the past and the present. It never waits for the future, so it can run on a live stream.
The top dot is one output frame. Lines show which input frames it depends on. Frames to its right are in the future. Change the hidden width H and clip length T.
Exact counts for D = 768 encoder width and C = 10 classes, ignoring normalisation layers. GRU weights follow PyTorch's layout: 3H(D + H) + 6H per direction. The transformer row is one layer with a D-to-H input projection, 4 attention matrices, a 4H feed-forward block and two LayerNorms. Multiply-adds are per frame.
Here are the counts behind the device, for D = 768, H = 256, C = 10.
Worked, conv stack: 3 · 768 · 256 + 256 = 590,080, then two layers of 3 · 256 · 256 + 256 = 196,864, then 256 · 10 + 10 = 2,570. Total 986,378 weights. BiGRU: 2 × (3 · 256 · 1,024 + 1,536) + 5,130 = 1,581,066.
The conv stack's view is centred, so it needs 7 future frames: at 64 ms frames that is 448 ms of delay before it can score "now". A BiGRU or transformer needs the whole clip first. The causal conv needs none, but loses the frames after an onset, which are often the clearest evidence.
Chapter 5 Why it works
Turn scores into on and off, then derive exactly which short runs a median window of k removes and which it keeps
The first post-processing step is a threshold: a frame is "on" for class c if pt,c ≥ θ. That gives a row of 0s and 1s per class. Real rows flicker. A one-frame spike becomes a whole false event. A one-frame dip splits a true event in two.
A median filter of width k replaces each frame with the median of the k frames centred on it. On 0s and 1s the median is a majority vote. Click frames below to build your own row, then widen the window.
Top row: your thresholded frames (tap a cell to flip it). Bottom row: after the median filter. Runs are labelled with their length.
Exact. Edges are handled by repeating the first and last frame, as many libraries do.
Let k = 2m + 1 (k is odd). Take a run of L ones with long runs of zeros on both sides.
Worked: the DCASE 2023 baseline uses a median window of 7 output frames DESED baseline config. So m = 3. With 64 ms frames it deletes blips up to 3 × 64 = 192 ms and fills gaps up to 192 ms, without moving the boundaries of anything longer.
The rule is symmetric: gaps are runs of zeros, so the same window fills them. That is why one filter fixes both false spikes and fragmented events. The price: true events of m frames or fewer vanish too. Choose m below the shortest event you must keep.
Chapter 6 Why it works
Use two thresholds so a hovering score stops chattering, then drop events that are too short and merge those split by short gaps
A median filter fixes short flicker. It fails when a score hovers near the threshold for a long time: 0.52, 0.47, 0.55, 0.49. Every crossing starts or ends an event. This is chatter.
A thermostat solves the same problem. It turns the heat on at 19 degrees and off at 21, not on and off at 20. Two thresholds, with a dead band between them, are called hysteresis. For detection: switch on when the score rises to the high threshold, and stay on until it falls below the low one.
Worked, θhi = 0.6, θlo = 0.35, scores 0.1, 0.2, 0.7, 0.4, 0.8, 0.9, 0.3, 0.6, 0.8, 0.2, 0.1, 0.9, 0.1. States: 0 0 1 1 1 1 0 1 1 0 0 1 0. The 0.4 kept the event alive; with one threshold at 0.6 it would have split. Chapter 9 runs this exact row.
Two more rules finish the job. A minimum duration drops any event shorter than a set number of frames: a 100 ms "alarm" is not an alarm. Gap merging joins two events of the same class if the gap between them is shorter than a set length: a dog that barks, pauses 200 ms and barks again may count as one event, depending on how your labels were made.
Twelve seconds of scores with three true events: a strong one with a dip, a quiet one that hovers, and one split by a short silence. There is also one false spike.
120 frames of 100 ms, hand-built to show each failure once. Order of operations: thresholds, then gap merging, then minimum duration.
Order matters. Merge gaps before dropping short events, or you may drop two halves of a real event that merging would have joined. Many pipelines run: threshold (or hysteresis), median filter, merge gaps, drop short. The hero uses that order.
Chapter 7
Read onsets and offsets off the frames, then grade the same detection three ways: collars, one-second segments and overlap
After post-processing, each class has a clean row of 0s and 1s. Each run of ones becomes one event. If a run starts at frame s and ends just before frame e, then onset = s · H and offset = e · H, in seconds. This is run-length decoding. Now: is the event list right?
The event-based metric matches whole events. A detected event counts as a hit, a true positive (TP), if its class matches and its boundaries fall within a tolerance called a collar Mesaros et al. 2016. In the DCASE sound event detection challenges, the onset collar is 200 ms and the offset collar is the larger of 200 ms and 20% of the true event's length DCASE 2019 Task 4 baseline:
Unmatched detections are false positives (FP); unmatched true events are false negatives (FN). The F1 score balances them:
Worked: a true event from 2.0 s to 5.0 s (3.0 s long). The onset must land in 1.8 to 2.2 s. The offset tolerance is max(0.2, 0.2 × 3.0) = 0.6 s, so 4.4 to 5.6 s. A detection from 2.15 to 4.5 s: TP, F1 = 1. A detection from 2.3 to 5.0 s: onset 0.3 s late, so it is an FP and the truth is an FN. F1 = 0 / (0 + 1 + 1) = 0.
That last case is the double penalty: one slightly late event costs both a false alarm and a miss. Splitting one event in two is worse still. Try it below, and compare with two other ways to score.
The truth runs from 2.0 s to 5.0 s. Teal bands are the collars. Move the detection's onset and offset, or split it into pieces.
Exact. Collar rule: 200 ms onset, max(200 ms, 20% of length) offset. Segments: 1 s, as in DCASE's segment-based metric. Overlap rule: the PSDS criteria of Chapter 8 with both tolerances at 0.5.
Segment-based scoring cuts time into fixed segments (1 s in DCASE) and asks, per segment and class, "active or not?" Mesaros et al. 2016. It ignores how many events there are. A split event loses nothing. It suits classes whose boundaries are fuzzy by nature, like rain or traffic.
Overlap-based scoring, from the PSDS framework Bilen et al. 2020, replaces collars with two tests. The detection tolerance criterion (DTC) keeps a detection only if at least a share ρDTC of its length overlaps true events of its class; the rest are false positives. The ground truth intersection criterion (GTC) counts a true event as found if kept detections cover at least a share ρGTC of it. A fragmented detection still finds the event.
This matters in practice. Bilen et al. re-scored three DCASE 2019 systems. The challenge baseline scored 23.70% collar-based F1 but 61.38% with DTC/GTC at 0.5; the authors traced many of its false positives to split detections Bilen et al. 2020 · Table 1. The ranking of the three systems changed.
Chapter 8 Did it work?
Sweep the threshold per class, score across all thresholds at once with PSDS, and break the errors down by kind
An F1 score belongs to one operating point: one threshold and one set of post-processing settings. Change the threshold and the F1 changes. So a single F1 mixes two questions. Are the scores good? And was the threshold well chosen? Bilen et al. make this point about SED metrics Bilen et al. 2020 · §1.
Three plots answer the questions separately. Switch between them below.
Twenty minutes of scores for three classes. The dog head is well calibrated, the alarm head scores everything high, and the speech head is timid. Slide the shared threshold, or let each class pick its own.
Synthetic scores, 12,000 frames of 100 ms per class, post-processed with a median window of 5 and a 3-frame minimum. F1 uses the collar rule. The PSD-ROC uses DTC = GTC = 0.5, no cross-trigger cost and emax = 100 false positives per hour, the settings of Bilen et al.'s Figure 3a. Per-class thresholds are tuned on these same 20 minutes, which flatters them; tune on a validation set in practice.
Heads are rarely calibrated alike. A class with plenty of training data often scores high everywhere; a rare class scores low. One shared threshold is then too high for one class and too low for another. Picking the best threshold per class, on a validation set (held-out data, not the test set), often adds more F1 than a bigger head does. Imbalance and Calibration Heads goes further.
The polyphonic sound detection score (PSDS) removes the threshold from the question Bilen et al. 2020 · §3. Sweep the threshold. At each one, measure each class's TP ratio (the share of true events found, by DTC/GTC) and its false positive rate per hour of audio. Plot one against the other: that is the class's PSD-ROC curve. Average the curves across classes, then take the area up to a maximum false-positive rate emax:
Worked: one class's curve is a staircase: TP ratio 0.4 up to 20 FPs per hour, 0.6 up to 60, and 0.7 up to 100. Area = 0.4 × 20 + 0.6 × 40 + 0.7 × 40 = 8 + 24 + 28 = 60. PSDS = 60 / 100 = 0.60. With two classes at 0.5 and 0.7 everywhere, μ = 0.6 and σ = 0.1, so αST = 1 gives r = 0.5.
The full definition also has a cross-trigger term: a false positive that lands on another class's true event (a "dog" fired by a cat), weighted by αCT Bilen et al. 2020 · eq. 6. The DCASE 2023 baseline reports two settings: scenario 1 with DTC = GTC = 0.7, αCT = 0, αST = 1, strict on timing; and scenario 2 with DTC = GTC = 0.1, a cross-trigger tolerance of 0.3, αCT = 0.5, αST = 1, loose on timing but strict on confusions. Both use emax = 100 per hour DESED baseline code.
Counts tell you which knob to turn. Match each true event to the detections that overlap it:
For many classes, also plot confusion over time: a timeline with each class's truth and detections stacked, so you can see a "speech" detection sitting on a "TV" event. Cross-triggers of this kind point to classes the encoder cannot tell apart, which no threshold fixes.
Chapter 9 Build it
Trace the shapes through the whole detector, write the frozen encoder and conv head, the post-processing in numpy, and train with frame or weak labels
By hand first. Chapter 2 scored one frame: z = 0.1, p = 0.525, loss 0.644, gradient −0.475. Chapter 6 ran the post-processing by hand on 13 scores. Now step through the shapes of the full detector.
B = 16 clips, T = 250 frames, F = 64 input features, D = 768, H = 256, C = 10 classes. Press Next. Switch the labels to see where the loss attaches.
Shapes are exact for the code below with these sizes.
From scratch. The head is a dilated conv stack with receptive field 15, then a 1 × 1 convolution, which is a linear map applied to every frame. The wrapper freezes the encoder and keeps it in eval mode.
import torch import torch.nn as nn class TemporalConvHead(nn.Module): """(B, T, D) frame embeddings -> (B, T, C) frame logits.""" def __init__(self, d, c, hidden=256, k=3, dilations=(1, 2, 4), p=0.1): super().__init__() layers, ch = [], d for dil in dilations: # receptive field 1 + (k-1)*sum(dil) = 15 layers += [nn.Conv1d(ch, hidden, k, padding=dil * (k - 1) // 2, dilation=dil), nn.BatchNorm1d(hidden), nn.GELU(), nn.Dropout(p)] ch = hidden self.body = nn.Sequential(*layers) self.out = nn.Conv1d(hidden, c, kernel_size=1) # one linear map per frame def forward(self, h): # h: (B, T, D) x = h.transpose(1, 2) # (B, D, T): Conv1d wants channels first x = self.body(x) # (B, H, T): padding keeps T return self.out(x).transpose(1, 2) # (B, T, C) logits class EventDetector(nn.Module): """Frozen encoder + trainable temporal head.""" def __init__(self, encoder, d, c): super().__init__() self.encoder = encoder.requires_grad_(False).eval() self.head = TemporalConvHead(d, c) def train(self, mode=True): # model.train() must not wake the encoder super().train(mode) self.encoder.eval() return self def forward(self, x): # x: (B, T, F) input frames with torch.no_grad(): h = self.encoder(x) # (B, T, D) frozen embeddings return self.head(h) # (B, T, C); sigmoid only at inference
Why each choice. The padding dil * (k - 1) // 2 keeps T, so output frame t lines up with input frame t. BatchNorm and dropout regularise a head that may have more weights than you have labelled frames. The head returns logits, not probabilities, because BCEWithLogitsLoss applies the sigmoid inside the loss in a numerically safe way.
Training, in the same file. The stand-in encoder lets it run with no downloads; swap in a real one. The labels are random, so this is a smoke test: a falling loss here means the head is memorising 16 clips, which is the shuffled-label check of Chapter 8.
torch.manual_seed(0) B, T, F, D, C = 16, 250, 40, 64, 3 encoder = nn.Sequential(nn.Linear(F, D), nn.GELU(), nn.Linear(D, D)) # stand-in model = EventDetector(encoder, D, C) x = torch.randn(B, T, F) # 16 clips of 250 frames y = (torch.rand(B, T, C) < 0.1).float() # frame labels: 1 = class active bce = nn.BCEWithLogitsLoss() opt = torch.optim.AdamW(model.head.parameters(), lr=1e-3, weight_decay=1e-2) model.train() for step in range(200): logits = model(x) # (B, T, C) loss = bce(logits, y) # strong labels: grade every frame # weak labels (one 0/1 per clip and class): pool over time first # loss = bce(logits.amax(dim=1), y.amax(dim=1)) # (B, C) vs (B, C) opt.zero_grad() loss.backward() opt.step() model.eval() probs = torch.sigmoid(model(x)) # (B, T, C) in [0, 1] print(probs.shape, f"loss {loss.item():.3f}")
Frame labels or weak labels? Strong labels mark every onset and offset, so every frame gets a grade. They are slow to make. Weak labels only say which classes occur somewhere in a clip. To train on them, pool the frame scores over time (max, mean or attention) into one clip score and grade that. The pooling decides which frames get the gradient: max pooling sends it only to the top frame, so the head learns "where" as a side effect. That is multiple instance learning; see Weak Labels and MIL and Attention-Pooling Heads. DCASE Task 4 trains on a mix of strong, weak and unlabelled clips for this reason.
Post-processing, in numpy, per class:
import numpy as np def hysteresis(p, hi, lo): """Switch on at p >= hi, stay on while p >= lo.""" on, out = False, np.zeros(len(p), dtype=bool) for t, v in enumerate(p): on = v >= lo if on else v >= hi out[t] = on return out def median_binary(b, k): """Median of a 0/1 window = majority vote. k must be odd.""" if k <= 1: return b m = k // 2 padded = np.pad(b.astype(int), m, mode="edge") win = np.lib.stride_tricks.sliding_window_view(padded, k) # (T, k) return win.sum(axis=1) > m def to_events(b, hop, min_frames=0, max_gap=0): """0/1 frames -> [(onset_s, offset_s)]: merge short gaps, drop short events.""" d = np.diff(np.concatenate([[0], b.astype(int), [0]])) starts, ends = np.flatnonzero(d == 1), np.flatnonzero(d == -1) # [start, end) ev = [] for s, e in zip(starts, ends): if ev and s - ev[-1][1] <= max_gap: ev[-1][1] = e # close a short gap else: ev.append([s, e]) return [(round(float(s * hop), 3), round(float(e * hop), 3)) for s, e in ev if e - s >= min_frames] p = np.array([.1, .2, .7, .4, .8, .9, .3, .6, .8, .2, .1, .9, .1]) b = median_binary(hysteresis(p, hi=.6, lo=.35), k=3) print(b.astype(int)) # [0 0 1 1 1 1 1 1 1 0 0 0 0] print(to_events(b, hop=0.1, min_frames=2)) # [(0.2, 0.9)]
Trace it against Chapter 6. Hysteresis gave 0 0 1 1 1 1 0 1 1 0 0 1 0. The median of 3 fills the one-frame gap at frame 6 and removes the one-frame blip at frame 11. One event remains: 0.2 s to 0.9 s.
The library versions. scipy filters along time in one line; sed_eval and psds_eval score the result with the same settings DCASE uses DESED baseline code.
from scipy.ndimage import median_filter smooth = median_filter(probs_np, size=(7, 1)) # (T, C): window 7 along time only import sed_eval metric = sed_eval.sound_event.EventBasedMetrics( event_label_list=classes, t_collar=0.200, percentage_of_length=0.2) from psds_eval import PSDSEval psds = PSDSEval(ground_truth=gt_df, metadata=dur_df, dtc_threshold=0.7, gtc_threshold=0.7, cttc_threshold=0.3) # add one operating point per threshold, then: # psds.psds(alpha_ct=0, alpha_st=1, max_efpr=100)
Chapter 10 Where else When not
Move the same block to six fields, see what changes and what stays, and learn when a pooled head, segment metrics, a causal head or CTC is the better choice
The block moves across fields with three swaps: an encoder pretrained near the new data, a hop that fits the events' timescale, and post-processing rules that encode how long events last. The head and the metrics stay the same.
| Field | Stream and events | What changes |
|---|---|---|
| Audio SED | Microphone audio; alarms, barks, dishes, speech | Polyphonic, so sigmoid per class; DCASE collars and PSDS. |
| ECG beats and arrhythmia | Heart electrical signal; each beat, and runs of abnormal beats | Beats are near-instants: detect peaks, match within a small window. Arrhythmia episodes are intervals, with a minimum duration. |
| Wearable activity | Wrist or phone IMU; walking, typing, single keystrokes | Activities are long, so segment metrics fit; keystrokes are short taps that need a fine hop. |
| Video action localisation | Frames from a frozen video encoder; "pour", "stir" | Boundaries are fuzzy, so scoring uses overlap (temporal IoU) at several thresholds, not tight collars. |
| Speech activity (VAD) | Audio; one class, "someone is speaking" | Must run live: causal head; a "hangover" keeps speech on through short pauses, which is hysteresis plus gap merging. |
| Industrial and log anomalies | Sensor or metrics streams; fault and incident intervals | Few or no labels: score frames by anomaly, then the same threshold, hysteresis and minimum duration. |
Answer four questions. The card shows the block to start with and the one rule that decided it.
Rules of thumb for a first build, not laws. Confirm with the checks of Chapter 8.
| Avoid it when | Because | Use instead |
|---|---|---|
| Only clip-level presence matters | Onsets you never use cost labels and tuning | A pooled clip head (attention pooling) |
| Events of one class overlap heavily (a crowd of voices) | One 0/1 row per class cannot say "two at once"; runs merge | Count or density regression per frame, or source separation first |
| Boundaries are fuzzy by nature (rain, traffic) | Collars punish disagreement that is not an error | The same head, scored segment-based or with loose DTC/GTC |
| You must answer within a few frames, live | Bidirectional context, wide medians and gap merging all wait for the future | A causal head, causal smoothing, and a stated latency budget |
| Labels give the order of tokens, not their times | There are no frame targets to grade | CTC, below |
Speech transcripts say "the cat sat" but not when each word starts. Connectionist temporal classification (CTC; Graves et al., 2006) trains a framewise head on such unaligned label sequences. The head outputs, per frame, a distribution over the tokens plus a blank. The loss sums the probability of every frame-level path that collapses to the target sequence (merge repeats, then drop blanks), using dynamic programming. CTC fits when the targets are a sequence of discrete tokens that do not overlap, such as words, phonemes, characters in handwriting, or ordered steps in a procedure. Its outputs tend to be peaky, a single frame per token, so it gives order and rough position but not reliable onsets and offsets. When you need durations, or classes overlap, use frame labels and the head of this lesson.
Chapter 11
Get starting settings for your field, keep the one-screen cheat sheet, know the limits, and follow the series onward
Everything in this lesson fits on one card. Pick your field to get a first set of settings. Then use the cheat sheet to adjust them.
Choose a field. The card gives a hop, a head, post-processing defaults and the metric to report.
Starting points to tune on validation data, not final answers. The sound defaults echo the DCASE 2023 baseline's 64 ms frames and median of 7.
| Part | In one line |
|---|---|
| The block | Frozen encoder → temporal head → sigmoid per class → post-processing → event list. |
| Shape signature | B×N → B×T×D → B×T×C → [(class, onset, offset)]; onset = start frame × hop. |
| Maths in one line | pt,c = σ(wc·ht + bc) on context; R = 1 + Σ(k − 1)d; a median of 2m + 1 removes runs ≤ m. |
| Knobs | Hop; receptive field; per-class threshold; hysteresis gap; median window; merge gap; minimum duration. |
| Failure modes | Insertions, deletions, fragmentation, merges, boundary errors; softmax on overlapping classes; tuning on the test set. |
| Checks | F1 vs threshold per class; PSDS; error-type counts; confusion over time; shuffled labels; a linear framewise probe as baseline. |
| Alternatives | Pooled head if only "if"; segment metrics if fuzzy; causal head if live; CTC if only the order is labelled. |
This lesson is part of The Builder's Toolkit. Its hub is Backbone + Head. The neighbours:
Already on the site:
Now press Present or Teach and explain, out loud and from memory, how a row of frame scores becomes a list of events, and what each post-processing rule removes. Then go back to the hero, start from the raw threshold, and fix the errors one knob at a time.