A new problem lands on your desk, from a field you have never worked in. Nine steps take you from that problem to a model you can trust. Most failures come from the boring steps, not the clever ones: a random split can make a 69% model look like a 95% model.
Frame the task, audit the data, pick a backbone, run baselines, choose a head, decide what to unfreeze, handle imbalance, evaluate honestly, and ship. Each step has a reason, a few lines of PyTorch, and a check.
Pick one of four problems. At each of the nine steps, choose an option. The toy reports what that choice does to the score you would claim and the score the field would see. The run ends with a scorecard.
You need the backbone-and-head idea and a little PyTorch. Every step links to the series lesson that goes deep on it.
The two bars come from a small simulated pipeline: scores are two bell curves whose separation and threshold your choices change, scored as F1 on the rare class. The "toy run" lines are live computations in your browser: a kNN on grouped data for the split, a trained attention head for pooling, a trained adapter for unfreezing, and a shuffle-label test. All of these numbers are illustrative. Numbers marked with a source chip in the text are real.
Chapter 0
Four requests from four fields arrive in one week. They look nothing alike. One recipe solves all four
It is Monday. Four requests are waiting.
Audio, medicine, remote sensing and robotics. Different sensors, different labels, different costs of a mistake. Yet you can attack all four with the same nine steps. Each step asks one question. The answer changes with the field. The question does not.
This lesson is the last part of The Builder's Toolkit. The earlier lessons each built one block: a probe, a pooling head, a weak-label trick, an adapter. This lesson is the recipe: the order in which you use the blocks, and the checks between them.
Pick a problem. The picture shows one example and what its label marks. Turn on "show the groups" to see the hidden structure that breaks most first attempts.
Drawn signals are synthetic sketches. The ECG sampling rate (360 Hz) and the 48 subjects come from the MIT-BIH description in Mousavi et al. (§II). EuroSAT's 64 × 64 tiles, 13 bands and 27,000 images come from Helber et al. (Abstract, §III).
A model is one decision. A project is about forty. Most of them happen before any training: what the label means, how to split the data, which number to report. A wrong early decision does not crash anything. It quietly inflates the score, and the truth arrives later, from real users.
A recipe protects you from that. Each step has three parts in this lesson:
Chapter 1 Step 1
Write down the input, the output, how fine the label is, and the one number that matches what a mistake costs
Before any code, write four lines. What goes in? What comes out? How fine is the label? And which number tells you a model is good enough to use?
The label granularity is how much the label tells you. There are three common levels:
Two rules follow. First, your head cannot be finer than your labels without a trick. With "whether" labels, you need multiple-instance learning to get "when". Second, ask for the granularity the user needs, not the finest one you can. The sound team needs "is there glass breaking in this minute?", not exact onsets.
Now the number. Suppose 1 clip in 100 holds the sound. A detector that always says "no" is right 99% of the time. Its accuracy, the share of correct answers, is 99%. It is also useless.
Two other numbers see the rare class. Recall is the share of real positives you catch. Precision is the share of your alarms that are real. The right trade-off between them depends on cost. A missed heart problem costs far more than a false alarm a nurse can dismiss. So write down both costs, and let them set the decision rule:
Worked: a miss costs 20, a false alarm costs 1 → say yes when p ≥ 1 / 21 = 0.048. Equal costs → p ≥ 0.5.
Why this rule? Saying "yes" costs (1 − p) · cFP on average. Saying "no" costs p · cFN. Say yes when the first is smaller. Rearrange and you get the rule. A costly miss pushes the threshold down, so you raise more alarms.
Make positives rare and watch accuracy stay high while recall collapses. Then raise the cost of a miss and see the best threshold move left.
Scores of negatives and positives are bell curves 2 spreads apart, and p is the exact posterior at the chosen positive rate. Cost is per 1,000 examples. Numbers are illustrative.
Write a one-line spec and show it to the person who asked: "Input: 10-second clip. Output: yes or no. Metric: recall at no more than one false alarm per hour." If they disagree with the metric, you just saved a month. Then compute the score of the "always no" model in that metric. That is the floor every later step must beat.
Chapter 2 Step 2
Count it, check its balance and its labels, and find its groups. Then split by group, or every later number is a lie
Open the data before you open a notebook for training. Four questions:
Beats from one patient look alike. Their shape depends on that patient's body and electrode placement. Clips cut from one long recording share its room and microphone. Tiles from one satellite scene share its haze and season.
Now split the data at random. Beats from the same patient land in both training and test. A flexible model can learn "this looks like patient 207, and patient 207's beats are mostly abnormal". It scores well on test, because the test is full of patients it has met. On a new patient, that shortcut is gone.
The fix is a group split: all examples from one group go to the same side. In ECG research this is called the inter-patient split, against the leaky intra-patient split. Mousavi and colleagues list both for earlier methods on the MIT-BIH arrhythmia data Mousavi 2018 · Tables II, III:
| Method | Random split (intra-patient) | Patient split (inter-patient) | S-beat sensitivity, random → patient |
|---|---|---|---|
| Ye et al. 2010 | 96.5% | 75.2% | 72.4% → 3.2% |
| Yu and Chou 2008 | 95.4% | 75.2% | 73.8% → 1.8% |
| Song et al. 2005 | 98.7% | 76.3% | 86.4% → 27.0% |
Same methods, same database. Only the split changed, and accuracy fell by about 20 points. Worse, the rare S beats went from "mostly caught" to "almost never caught". The inter-patient split uses the record lists DS1 for training and DS2 for testing, from 44 records Mousavi 2018 · §II.
20 patients, 30 beats each. Each patient has a signature and mostly one class. A kNN is trained live with 5-fold cross-validation. Raise "how alike one patient's beats are" and watch the random split's score climb while the honest one does not.
Six feature dimensions; the patient signature sits in five of them, the class signal in one. Each patient is 95% one class, half of them normal-heavy and half abnormal-heavy. kNN with k = 5. Numbers are illustrative.
Grouped data also shrinks your sample size. Ten thousand beats from ten patients do not carry ten thousand beats of evidence. Statisticians use the design effect to count the independent examples:
Worked: 101,290 beats from 48 subjects Mousavi 2018 · §IV → m ≈ 2,110. With an illustrative ρ = 0.3: 1 + 2,109 × 0.3 = 633.7 → neff ≈ 160.
So the evidence is closer to that of 160 beats than 101,290. That is why per-patient variation dominates, and why more patients beat more beats per patient.
In code, a group split is one line. Pass the group ids, and scikit-learn keeps each group on one side:
from sklearn.model_selection import GroupKFold # y: (N,) labels; groups: (N,) patient, recording, site or session id for tr, te in GroupKFold(n_splits=5).split(np.zeros(len(y)), y, groups): assert not set(groups[tr]) & set(groups[te]) # no group on both sides
Ask the data owner: "What could make two examples secretly alike?" List every candidate group: patient, device, day, site, session, annotator. Split by the coarsest one that matches how the model will be used. A model for new hospitals needs a split by hospital, not just by patient.
Chapter 3 Step 3
Choose the pretrained encoder by modality and by symmetry, run it once, and store its output
The backbone is the big pretrained model that turns raw signal into vectors (see Backbone + Head). Two questions pick it.
Modality first. Use a backbone pretrained on the same kind of signal. Audio models for audio, image models for images. A backbone that has seen millions of examples of your signal type has already learned its edges, onsets and textures.
Symmetry second. Ask what should not change the answer. A heartbeat is still a heartbeat if it starts 50 ms later: the task is shift-invariant in time. A forest tile is still forest if you rotate it: the task is rotation-invariant. A good backbone has these symmetries built in or learned. A 1-D convolution shares its weights across time, so it treats every moment alike. Inductive Bias and Symmetry covers this in depth.
| Case | Backbone | Symmetry it brings | Shape out |
|---|---|---|---|
| Sound | CLAP-style audio encoder, frozen | Shift in time (convolutions) | B × T × D frames |
| ECG | Small 1-D CNN, trained on your beats | Shift in time | B × D per beat window |
| Satellite | DINOv2-style ViT, frozen or lightly adapted | Learned from augmented crops | B × N × D patch tokens |
| Gripper | Tiny causal 1-D CNN per sensor | Shift in time, and causal | B × T × D per sensor |
Some facts behind those picks. CLAP's audio encoder has 80.8 million weights and an embedding size of 2048 CLAP · §3.2. Its authors use it frozen, "as a feature extractor followed by training a classifier of 1 or 3 fully-connected layers" §3.3. DINOv2's ViT-B/14 cuts a 224 × 224 image into 16 × 16 patches plus one summary token: 257 tokens of 768 numbers. For ECG and touch there is no backbone as strong and as standard as these, so a small network trained on your own data is often the honest choice.
A frozen backbone gives the same output for the same input. So run it once, save the output, and train every head on the saved copy. This is feature caching. The cost is disk space:
Worked, EuroSAT's 27,000 tiles with a ViT-B/14: pooled, 27,000 × 768 × 2 = 41.5 MB. All 257 tokens: 257 × 41.5 MB = 10.7 GB.
Pick a case and slide the dataset size. The top row is the shape at each step. The bars compare caching one pooled vector per example with caching every frame or token.
CLAP's D = 2048 (CLAP §3.2) and DINOv2 ViT-B/14's 257 × 768 are real. The frame counts for sound, the ECG window and the gripper sizes are illustrative. fp16 storage.
This is the first block of the project. The others follow in later chapters, and together they run as one file. Stand-in data and a stand-in encoder let it run with no downloads. Swap in your own.
import numpy as np import torch import torch.nn as nn torch.manual_seed(0) N, T, D_IN, D = 600, 50, 16, 64 # clips, frames, raw size, feature size G = 30 # recordings: the groups # Stand-in data. Replace with your clips, labels and recording ids. raw = torch.randn(N, T, D_IN) # (N, T, D_IN) raw frames groups = torch.randint(0, G, (N,)) # (N,) which recording each clip came from y = (torch.rand(N) < 0.1).float() # (N,) rare positives, about 10% raw[y == 1, 20:25] += 1.0 # a short "event" inside each positive # Stand-in frozen backbone. Swap in a real encoder (CLAP, DINOv2, ...). backbone = nn.Sequential(nn.Linear(D_IN, D), nn.GELU(), nn.Linear(D, D)) backbone.requires_grad_(False).eval() with torch.no_grad(): # run the backbone ONCE feats = torch.cat([backbone(x) for x in raw.split(64)]) # (N, T, D) torch.save({"feats": feats.half(), "y": y, "groups": groups}, "cache.pt")
from torch.utils.data import Dataset, DataLoader class CachedFeatures(Dataset): """Serves cached frames; the backbone never runs again.""" def __init__(self, feats, y, idx): self.x = feats[idx].float() # (n, T, D) self.y = y[idx] # (n,) def __len__(self): return len(self.y) def __getitem__(self, i): return self.x[i], self.y[i] # (T, D), ()
Note what is saved: every frame, not a pooled vector. Pooling now would throw away the timing that the attention head in step 5 needs. Save the group ids too. Step 8 needs them.
Run the backbone twice on the same 10 inputs and compare. The outputs must match exactly. If they do not, dropout or BatchNorm is still in training mode, and your cache is noisy. Also re-check that cached and live features match after any change to preprocessing.
Chapter 4 Step 4
Before anything clever, score the majority class, a kNN and a linear probe. They tell you what "good" means for this problem
A number alone means nothing. Is 89% accuracy good? On MIT-BIH's patient split, it is worse than doing nothing. The test set DS2 holds 44,196 normal beats, 1,836 S beats and 3,216 V beats Mousavi 2018 · §IV. A model that always says "normal" scores 44,196 / 49,248 = 89.7%. The methods in the table of Chapter 2 scored about 75% on that split. They were below the do-nothing line.
So before any real model, run three baselines, each one smarter than the last:
Each bar is trained live on cached toy features. Slide the labels and the feature quality. Look for the point where the bigger model stops beating the simple one by more than its error bar.
10-dimensional synthetic features, 25% positives, test set of 400. The MLP has 32 hidden units. Error bars are ±2 standard errors. Numbers are illustrative.
Two models that differ by one point may be the same model on a different test set. The standard error of an accuracy p measured on n test examples is:
Worked: p = 0.897, n = 49,248 beats → SE = 0.0014, ±0.3 points. With neff = 160 instead → SE = 0.024, ±4.8 points.
The second number is the honest one for a model that must work on new patients. It explains why ECG papers that differ by one point may not differ at all.
from sklearn.dummy import DummyClassifier from sklearn.neighbors import KNeighborsClassifier from sklearn.linear_model import LogisticRegression from sklearn.model_selection import cross_val_score, GroupKFold X = feats.float().mean(dim=1).numpy() # (N, D) mean-pooled cache cv = GroupKFold(n_splits=5) for name, m in [("majority", DummyClassifier()), ("kNN", KNeighborsClassifier(5)), ("linear", LogisticRegression(max_iter=1000))]: s = cross_val_score(m, X, y.numpy(), groups=groups.numpy(), cv=cv, scoring="average_precision") print(name, s.mean().round(3), "+/-", s.std().round(3))
Write the three baseline scores at the top of your results table, in the metric from step 1, with the grouped split from step 2. Every later row must beat the best of them by more than two standard errors. If none does, the features are the problem, not the head.
Chapter 5 Step 5
The label's shape and where the signal sits pick the head from the toolkit. Then build it in PyTorch
The head is the small trained part on top of the backbone. You do not invent it. You read it off two facts: the shape of your label, and where the signal sits in the input.
| Your label | Where the signal is | Head | Shape |
|---|---|---|---|
| Whether, one per example | Spread over the whole input | Mean pool + linear (pooling) | B×T×D → B×C |
| Whether, one per example | In a short part | Attention pooling, trained as MIL | B×T×D → B×C, plus B×T weights |
| When, per time step | Onsets and offsets | Frame head + smoothing | B×T×D → B×T×C |
| Where, per patch | Local | Dense head on patch tokens | B×N×D → B×N×C |
| Same or different | Identity | Projection head | B×D → B×K, unit length |
| Several labels at once | Shared | Multi-task heads | B×D → several outputs |
| Any, from two sensors | Split across sensors | Fusion block then a head | (B×D1, B×D2) → B×C |
The sound detector has "whether" labels and a short event. That points to attention pooling. Each frame gets a score from a learned query. The scores become weights, and the weights average the frames:
Worked, 4 frames, scores (0, 0, 0, 3): e3 = 20.09, sum = 23.09 → a = (0.043, 0.043, 0.043, 0.870). Mean pooling gives each frame 0.25.
Mean pooling dilutes a short event by the clip length. If the event fills 3 frames out of 48, its signal is cut to one sixteenth. Attention can put most of the weight on those 3 frames. Train both below and see.
200 training clips of 48 frames. Positives hide an event somewhere. Shorten the event and watch mean pooling fall behind. The strip shows the trained attention weights on one test clip.
6-dimensional frames of unit noise; the event adds 2.5 to one dimension and 1.5 to another. Both heads are trained by gradient descent in your browser (mean: logistic regression on pooled vectors; attention: query, weights and bias trained jointly). Test set of 400 clips. Numbers are illustrative.
The second block of the project. It follows the formula line by line, with a LayerNorm first so all frames are on one scale.
class AttnPoolHead(nn.Module): """(B, T, D) frames -> (B,) logit, plus (B, T) attention weights.""" def __init__(self, d): super().__init__() self.norm = nn.LayerNorm(d) self.score = nn.Linear(d, 1) # the learned query q self.out = nn.Linear(d, 1) # one logit: yes or no def forward(self, h): # h: (B, T, D) h = self.norm(h) a = self.score(h).squeeze(-1).softmax(dim=1) # (B, T), rows sum to 1 z = torch.einsum("bt,btd->bd", a, h) # (B, D) weighted average return self.out(z).squeeze(-1), a # (B,), (B, T)
Compare the head against the linear probe from step 4, on the same grouped folds. Then look at the attention weights on ten positive test clips. If the peaks sit on the event, the head learned the right thing. If they sit on silence or on clip edges, it found a shortcut.
Chapter 6 Step 6
The number of labels and the distance between the backbone's world and yours decide whether to probe, adapt or fine-tune
So far the backbone has stayed frozen. Should it? Two facts decide.
How many labels? Every weight you unfreeze can fit noise. A few hundred labels can safely train a few thousand weights, not eighty million.
How big is the domain gap? The domain gap is how far your data is from the backbone's training data. CLAP heard everyday sounds, so a glass-break detector is near. DINOv2 saw photos taken from the ground. A satellite tile seen from above, with no horizon and odd colours, is far. The EuroSAT authors note that ImageNet-pretrained networks had "not been trained on images other than RGB" EuroSAT · §IV-B, yet the tiles have 13 spectral bands.
| Near domain | Far domain | |
|---|---|---|
| Few labels | Freeze; probe or small head | Adapters or LoRA; or a nearer backbone |
| Many labels | Unfreeze the top blocks | Full fine-tuning, small backbone learning rate |
LoRA is the cheap middle path. It keeps each big weight matrix W frozen and learns a low-rank change: W + BA, where B and A are thin. Adapters and Fine-Tuning Depth derives it. The weight count is what makes it safe with few labels:
Worked, ViT-B (d = 768), rank 8 on the query and value matrices of all 12 blocks: 8 × (768 + 768) = 12,288 per matrix × 24 matrices = 294,912 weights, against about 85 million in the backbone.
A frozen map keeps 4 of 24 input directions. The domain gap turns the class signal out of those 4. Three models train live. Find where the probe wins, and where adapting wins.
Inputs have 24 unit-noise dimensions. The frozen map keeps the first 4. The class signal points along a direction tilted by the gap angle away from them. Probe: head only (5 weights). Adapter: a rank-1 change to the map plus the head (33). Full: the whole 4 × 24 map plus the head (101). Each bar averages 3 training draws, tested on 1,000 points. Numbers are illustrative.
For the sound case, nothing unfreezes: the cache from step 3 is the backbone. Here is the third block, the training loop. The optimizer sees only the head. The loss weighs positives up, which is step 7.
def train_head(feats, y, idx, epochs=30, lr=1e-3): ds = CachedFeatures(feats, y, idx) head = AttnPoolHead(ds.x.shape[-1]) pos = ds.y.sum().clamp(min=1) w = (len(ds) - pos) / pos # negatives per positive loss_fn = nn.BCEWithLogitsLoss(pos_weight=w) # step 7: imbalance opt = torch.optim.AdamW(head.parameters(), lr=lr, weight_decay=1e-2) for _ in range(epochs): for h, t in DataLoader(ds, batch_size=32, shuffle=True): logit, _ = head(h) # (B,) loss = loss_fn(logit, t) opt.zero_grad() loss.backward() opt.step() return head.eval()
For the satellite case, you would instead wrap the backbone with LoRA and give the optimizer two groups: the head at 1e-3 and the LoRA weights at about 1e-4. The adapters lesson has that code.
Unfreezing must beat the frozen probe by more than the error bar, on the grouped split. If it does not, keep the frozen model: it is cheaper, faster and less likely to forget. Also plot training and test loss. A test loss that rises while training loss falls means you unfroze too much for your labels.
Chapter 7 Step 7
Rare positives need weighting during training and a correction afterwards, or the model's probabilities lie
Rare classes are the rule in applied work. In MIT-BIH's training records, there are 45,796 normal beats and only 941 S beats Mousavi 2018 · §IV. That is 49 normal beats for each S beat. A model trained on this as-is learns that "normal" is almost always right.
Three common fixes change what the loss sees:
pos_weight in the step 6 code.Every one of these fixes trains the model as if positives were common. So its output probabilities describe a balanced world. In the real world, positives are rare. A model trained at 50% positives says 0.8 for cases that are really positive far less often. You must shift its logit back:
Worked: balanced model says 0.80 → ℓbal = ln(0.8 / 0.2) = 1.386. Real rate 2%: ln(0.02 / 0.98) = −3.892 → ℓreal = −2.506 → p = 0.075.
A "0.8" from the balanced model means a 7.5% chance in the field. Act on 0.8 as if it were 80%, and you will be wrong most of the time. The full story, with temperature scaling and reliability plots, is in Imbalance and Calibration.
The diagonal is perfect calibration: when the model says 0.3, 30% are positive. Make positives rarer, then switch on the correction.
Score bell curves 2 spreads apart. The balanced model's probability is the exact posterior at a 50% rate. Expected calibration error (ECE) is computed over 10 equal-width bins. Numbers are illustrative.
Plot a reliability diagram on grouped validation data: predicted probability on one axis, observed frequency on the other. Points on the diagonal mean the probabilities are honest. Then pick the threshold from the step 1 cost rule using the corrected probabilities.
Chapter 8 Did it work?
Grouped cross-validation, a learning curve, errors by slice, and the shuffle-label test that catches leaks you did not think of
Step 2 stopped the leaks you could name. Step 8 hunts the ones you could not. Four tools, in order.
Grouped cross-validation. One test split is one roll of the dice. Five grouped folds give five scores. Report their mean and spread. Every fold must hold out whole groups.
A learning curve. Train on 10%, 20%, and up to 80% of the training groups, and plot the score. If it still rises at the end, more labels will pay. The EuroSAT paper gives such a curve for a ResNet-50: 75.06% with 10% of the data for training, 88.53% at 20%, 94.45% at 50% and 96.43% at 80% EuroSAT · Table II. The steep start says labels mattered most at first. The flat end says more of the same labels would add little.
Errors by slice. Split the test errors by group, device, class or season. An average can hide one slice where the model fails. The EuroSAT confusion matrix shows the model mixing up the crop classes, and highway with river EuroSAT · §IV-A.
The shuffle-label test. Shuffle the labels, then run the whole pipeline again, from preprocessing to score. The labels now carry no information. The test score must fall to chance. If it stays above chance, something in your pipeline is reading the labels: a leak.
Worked: c = 0.5, n = 80 → 2 × √(0.25 / 80) = 0.112 → band 0.39 to 0.61. A shuffled score of 0.68 is outside it: hunt for the leak.
80 examples with 200 features, 3 of them useful. A kNN is scored by 5-fold cross-validation, live. Turn on a leak, then press "Shuffle labels". A clean pipeline drops to the chance band. A leaky one does not.
25% positives; the 3 useful features shift by 1.4 spreads for positives. Feature selection keeps the 10 features with the largest class-mean gap. Oversampling makes synthetic positives on lines between random pairs of positives, as SMOTE does. Each bar averages 5 random datasets. Balanced accuracy; chance is 0.5. Numbers are illustrative.
The fourth block: grouped cross-validation, then the same thing with shuffled labels. Because the whole pipeline sits inside the loop, the shuffle test covers every step.
from sklearn.model_selection import GroupKFold from sklearn.metrics import average_precision_score def grouped_cv(feats, y, groups, k=5): """Average precision on recordings the head never saw.""" aps = [] for tr, te in GroupKFold(n_splits=k).split(np.zeros(len(y)), y, groups): head = train_head(feats, y, tr) with torch.no_grad(): logit, _ = head(feats[te].float()) # (n_test,) aps.append(average_precision_score(y[te].numpy(), logit.numpy())) return float(np.mean(aps)), float(np.std(aps)) c = torch.load("cache.pt") feats, y, groups = c["feats"], c["y"], c["groups"] print("grouped AP %.3f +/- %.3f" % grouped_cv(feats, y, groups)) y_shuf = y[torch.randperm(len(y))] # the shuffle-label test print("shuffled AP %.3f +/- %.3f" % grouped_cv(feats, y_shuf, groups)) print("chance (the positive rate) %.3f" % y.mean().item())
On the stand-in data, this printed a grouped AP of 0.996, a shuffled AP of 0.151, and a chance level of 0.088. The shuffled score sits near chance. With only about 50 positives, average precision is noisy, so a small gap is expected.
Write a one-page evaluation: grouped CV mean and spread, the three baselines, the learning curve, the worst three slices, and the shuffled-label score with its chance band. If any line surprises you, stop and find out why before you ship.
Chapter 9 Step 9
Pick the threshold on validation data, check the time budget, and watch the inputs change after launch
A shipped model is a score, a threshold, a clock and a monitor. Each one can fail on its own.
Pick it on grouped validation data with the cost rule from step 1. Never pick it on the test set: that tunes the model to the test, and the test score is no longer honest. Write the threshold into the model's config, next to the backbone version and the cache date.
A detector is late by more than its compute time. If it looks at a window centred on now, it must wait for the future half of that window. Then it waits up to one hop for the next window to start. The worst-case delay is:
Worked: centred 1 s window, 0.5 s hop, 10 ms model → 500 + 500 + 10 = 1,010 ms. Causal window, 5 ms hop, 1 ms model → 0 + 5 + 1 = 6 ms.
Real systems show why this matters. Mohammad and colleagues detected contact on a parallel robot with a model-based observer, then fused one low-cost IMU with the joint encoders. Detection time fell from 6–47 ms to 3–39 ms, a cut of 17% to 50% across their four contact tests Mohammad 2025 · Table III. Their method is physics, not learning, but the budget lesson is the same: for touch, milliseconds are the metric.
Left: the time budget. Choose a deadline, then set the window, hop and model time. Right: after launch, the sensor drifts. Slide the drift and watch false alarms climb at a fixed threshold.
Drift moves the negative score distribution toward the threshold by the slider amount, in spreads. False alarms assume 3,600 windows per hour and a threshold set for 1 per hour before drift. Population stability index (PSI) over 10 bins. Numbers are illustrative.
After launch, inputs change. Microphones age, a hospital buys a new ECG machine, seasons change the satellite colours. A model with a fixed threshold then raises more or fewer alarms, with no error message. So log two cheap numbers every day: the alarm rate, and the distribution of the model's scores. Compare both with the validation week. A shift means it is time to re-label a sample and re-check.
The fifth block picks the threshold on held-out recordings, times the head, and logs the alarm rate.
def pick_threshold(logits, y, c_fn=20.0, c_fp=1.0): """The cut with the lowest cost on VALIDATION data.""" best_t, best_cost = 0.0, float("inf") for t in torch.unique(logits): pred = logits >= t cost = (c_fn * (~pred & (y == 1)).sum() + c_fp * (pred & (y == 0)).sum()).item() if cost < best_cost: best_t, best_cost = t.item(), cost return best_t val = torch.from_numpy(np.isin(groups.numpy(), [0, 1, 2, 3, 4, 5])) # 6 held-out recordings head = train_head(feats, y, torch.where(~val)[0]) with torch.no_grad(): logit_val, attn = head(feats[val].float()) # (n_val,), (n_val, T) t_star = pick_threshold(logit_val, y[val])
import time x1 = feats[:1].float() # one clip: (1, T, D) t0 = time.perf_counter() with torch.no_grad(): for _ in range(100): head(x1) print("head latency %.3f ms" % ((time.perf_counter() - t0) * 10)) alarm_rate = (logit_val >= t_star).float().mean().item() print("alarm rate on validation %.3f (log it; watch it drift)" % alarm_rate)
On the stand-in data the head took about 0.05 ms per clip on a laptop CPU, and raised alarms on 8.7% of validation clips. The backbone's time is not in that number. Time the whole path, from raw signal to decision, on the real device.
Chapter 10 What else What not
Walk each problem through all nine steps, see what changes between fields and what stays, and learn when the defaults fail
Here are the four problems from Chapter 0, each decided step by step. Slide through the steps to compare the four side by side.
Slide through the steps. Each card is the decision for one case. The line under it is the reason.
Decisions are this lesson's recommendations. Real numbers in the cards are from the cited sources; others are labelled as examples.
Across the four, the backbone, the head and the metric all changed. The questions did not. In every case the group split, the trivial baselines and the shuffle test stayed. These three are the cheapest steps and the ones most often skipped.
| Default | Fails when | Do instead |
|---|---|---|
| Frozen pretrained backbone | No backbone near your signal (touch, odd sensors) | Small model from scratch, or self-supervised pretraining on your unlabelled data |
| Cache the features | Augmentation matters, or the backbone trains | Cache a few augmented copies, or run the backbone live |
| Group split by source | There are very few groups (3 patients) | Leave-one-group-out CV, and report the spread honestly |
| Neural head | The data is a table of measured values | Gradient-boosted trees |
| One threshold | Costs differ across users or sites | A threshold per site, or ship calibrated probabilities |
Chapter 11
Build a recipe card for your own problem, keep the one-screen cheat sheet, know the limits, and see where the series goes
Describe your own problem below. The card gives a first decision for each of the nine steps.
Set five facts about your problem. The card updates each step's first decision.
First decisions, not final answers. Each step has its own chapter above.
| Step | Decide | Check |
|---|---|---|
| 1 Frame | Input, output, whether / when / where, a metric that matches the cost; say yes when p ≥ cFP / (cFP + cFN) | The "always no" score in your metric; the asker agrees |
| 2 Audit | Size, balance, label noise, groups; split by group; neff = n / (1 + (m − 1)ρ) | No group on both sides; read 50 labels |
| 3 Backbone | By modality, symmetry and deadline; cache frames: N·T·D·b bytes | Same input, same features, twice |
| 4 Baselines | Majority, kNN, linear probe on the cache | Later models beat them by > 2 SE |
| 5 Head | From label shape: pool, attention + MIL, frame, dense, projection, multi-task, fusion | Beats the probe; attention lands on the event |
| 6 Unfreeze | Few + near: freeze. Far: LoRA, r(din + dout) weights. Many + far: full, small LR | Beats the frozen probe; test loss does not rise |
| 7 Imbalance | Weights or balanced batches; oversample inside the training fold only; correct logits by the log-odds of the real rate | Reliability diagram on the diagonal |
| 8 Evaluate | Grouped CV, learning curve, slices, shuffle test | Shuffled score inside c ± 2√(c(1 − c)/n) |
| 9 Ship | Threshold on validation; delay = Wfuture + hop + tmodel; log alarm rate and score drift | Delay under the deadline on the real device |
This is the last lesson of The Builder's Toolkit. Each step points back to a block:
Now press Present or Teach and walk one of the four cases through all nine steps, out loud and from memory. Then go back to the hero, run the ECG case with a random split, and see how far the claim drifts from the truth.