The Cross-Domain Recipe

A new problem lands on your desk, from a field you have never worked in. Nine steps take you from that problem to a model you can trust. Most failures come from the boring steps, not the clever ones: a random split can make a 69% model look like a 95% model.

Frame the task, audit the data, pick a backbone, run baselines, choose a head, decide what to unfreeze, handle imbalance, evaluate honestly, and ship. Each step has a reason, a few lines of PyTorch, and a check.

Pick one of four problems. At each of the nine steps, choose an option. The toy reports what that choice does to the score you would claim and the score the field would see. The run ends with a scorecard.

You need the backbone-and-head idea and a little PyTorch. Every step links to the series lesson that goes deep on it.

The recipe runner

Step 1 of 9

The two bars come from a small simulated pipeline: scores are two bell curves whose separation and threshold your choices change, scored as F1 on the rare class. The "toy run" lines are live computations in your browser: a kNN on grouped data for the split, a trained attention head for pooling, a trained adapter for unfreezing, and a shuffle-label test. All of these numbers are illustrative. Numbers marked with a source chip in the text are real.

Chapter 0

Four problems on one desk

Four requests from four fields arrive in one week. They look nothing alike. One recipe solves all four

It is Monday. Four requests are waiting.

Audio, medicine, remote sensing and robotics. Different sensors, different labels, different costs of a mistake. Yet you can attack all four with the same nine steps. Each step asks one question. The answer changes with the field. The question does not.

This lesson is the last part of The Builder's Toolkit. The earlier lessons each built one block: a probe, a pooling head, a weak-label trick, an adapter. This lesson is the recipe: the order in which you use the blocks, and the checks between them.

Four problems, seen as data

Pick a problem. The picture shows one example and what its label marks. Turn on "show the groups" to see the hidden structure that breaks most first attempts.

Drawn signals are synthetic sketches. The ECG sampling rate (360 Hz) and the 48 subjects come from the MIT-BIH description in Mousavi et al. (§II). EuroSAT's 64 × 64 tiles, 13 bands and 27,000 images come from Helber et al. (Abstract, §III).

Why a recipe, and not just a model

A model is one decision. A project is about forty. Most of them happen before any training: what the label means, how to split the data, which number to report. A wrong early decision does not crash anything. It quietly inflates the score, and the truth arrives later, from real users.

A recipe protects you from that. Each step has three parts in this lesson:

The road map

  1. Frame the task: whether, when or where, and a metric that matches the cost. Step 1
  2. Audit the data: size, balance, noise, and the group split that stops leaks. Step 2
  3. Pick a backbone and cache: by modality and symmetry. Step 3
  4. Trivial baselines first: majority, kNN and a linear probe. Step 4
  5. Choose the head: from the label's shape. Step 5
  6. Decide what to unfreeze: from data size and domain gap. Step 6
  7. Imbalance and calibration: rare positives and honest probabilities. Step 7
  8. Evaluate honestly: grouped CV, the shuffle-label test, slices. Step 8
  9. Ship: thresholds, latency and drift. Step 9
  10. Four cases, start to finish: and when the recipe's defaults fail. What else / what not
  11. Connections: your recipe card, the cheat sheet, and honest limits.

The Builder's Toolkit

  1. Backbone + Head: the pattern behind almost every applied model.
  2. Probing Representations: linear and kNN probes. Used in step 4.
  3. Attention-Pooling Heads: learn which frames matter. Used in step 5.
  4. Weak Labels and MIL: one label per clip. Used in steps 1 and 5.
  5. Event-Detection Heads: per-frame onsets and offsets. Used in steps 1, 5 and 9.
  6. Adapters and Fine-Tuning Depth: what to unfreeze. Used in step 6.
  7. Projection Heads and Metric Learning: heads for search and few-shot. Used in step 5.
  8. Multi-Task Heads: one trunk, many answers. Used in step 5.
  9. Fusion Blocks: join two sensors. Used in the gripper case.
  10. Inductive Bias and Symmetry: pick blocks that match the data. Used in step 3.
  11. Imbalance and Calibration: rare classes and honest scores. Used in step 7.
  12. The Cross-Domain Recipe (this lesson): the whole method, end to end.
According to this chapter, where do most failed applied projects go wrong?

Chapter 1 Step 1

Frame the task

Write down the input, the output, how fine the label is, and the one number that matches what a mistake costs

Before any code, write four lines. What goes in? What comes out? How fine is the label? And which number tells you a model is good enough to use?

1aWhether, when or where

The label granularity is how much the label tells you. There are three common levels:

Two rules follow. First, your head cannot be finer than your labels without a trick. With "whether" labels, you need multiple-instance learning to get "when". Second, ask for the granularity the user needs, not the finest one you can. The sound team needs "is there glass breaking in this minute?", not exact onsets.

1bA metric that matches the cost

Now the number. Suppose 1 clip in 100 holds the sound. A detector that always says "no" is right 99% of the time. Its accuracy, the share of correct answers, is 99%. It is also useless.

Two other numbers see the rare class. Recall is the share of real positives you catch. Precision is the share of your alarms that are real. The right trade-off between them depends on cost. A missed heart problem costs far more than a false alarm a nurse can dismiss. So write down both costs, and let them set the decision rule:

p
The model's probability that this example is positive. It must be calibrated: of the cases where p = 0.2, about 20% are really positive (step 7).
cFP
The cost of a false positive: an alarm with nothing there.
cFN
The cost of a false negative: a real event you missed.

Worked: a miss costs 20, a false alarm costs 1 → say yes when p ≥ 1 / 21 = 0.048. Equal costs → p ≥ 0.5.

Why this rule? Saying "yes" costs (1 − p) · cFP on average. Saying "no" costs p · cFN. Say yes when the first is smaller. Rearrange and you get the rule. A costly miss pushes the threshold down, so you raise more alarms.

Accuracy hides the rare class

Make positives rare and watch accuracy stay high while recall collapses. Then raise the cost of a miss and see the best threshold move left.

10
20
0.50

Scores of negatives and positives are bell curves 2 spreads apart, and p is the exact posterior at the chosen positive rate. Cost is per 1,000 examples. Numbers are illustrative.

Check it

Write a one-line spec and show it to the person who asked: "Input: 10-second clip. Output: yes or no. Metric: recall at no more than one false alarm per hour." If they disagree with the metric, you just saved a month. Then compute the score of the "always no" model in that metric. That is the floor every later step must beat.

A missed fault costs 9 times as much as a false alarm. With a calibrated model, when should you say "fault"?

Chapter 2 Step 2

Audit the data

Count it, check its balance and its labels, and find its groups. Then split by group, or every later number is a lie

Open the data before you open a notebook for training. Four questions:

2aWhy groups matter

Beats from one patient look alike. Their shape depends on that patient's body and electrode placement. Clips cut from one long recording share its room and microphone. Tiles from one satellite scene share its haze and season.

Now split the data at random. Beats from the same patient land in both training and test. A flexible model can learn "this looks like patient 207, and patient 207's beats are mostly abnormal". It scores well on test, because the test is full of patients it has met. On a new patient, that shortcut is gone.

The fix is a group split: all examples from one group go to the same side. In ECG research this is called the inter-patient split, against the leaky intra-patient split. Mousavi and colleagues list both for earlier methods on the MIT-BIH arrhythmia data Mousavi 2018 · Tables II, III:

MethodRandom split (intra-patient)Patient split (inter-patient)S-beat sensitivity, random → patient
Ye et al. 201096.5%75.2%72.4% → 3.2%
Yu and Chou 200895.4%75.2%73.8% → 1.8%
Song et al. 200598.7%76.3%86.4% → 27.0%

Same methods, same database. Only the split changed, and accuracy fell by about 20 points. Worse, the rare S beats went from "mostly caught" to "almost never caught". The inter-patient split uses the record lists DS1 for training and DS2 for testing, from 44 records Mousavi 2018 · §II.

The leak, measured

20 patients, 30 beats each. Each patient has a signature and mostly one class. A kNN is trained live with 5-fold cross-validation. Raise "how alike one patient's beats are" and watch the random split's score climb while the honest one does not.

2.5
1.5

Six feature dimensions; the patient signature sits in five of them, the class signal in one. Each patient is 95% one class, half of them normal-heavy and half abnormal-heavy. kNN with k = 5. Numbers are illustrative.

2bHow many examples do you really have?

Grouped data also shrinks your sample size. Ten thousand beats from ten patients do not carry ten thousand beats of evidence. Statisticians use the design effect to count the independent examples:

n
The number of examples you have.
m
The average number of examples per group.
ρ
The intra-class correlation: how alike two examples from the same group are, from 0 (independent) to 1 (copies).
neff
How many independent examples carry the same evidence.

Worked: 101,290 beats from 48 subjects Mousavi 2018 · §IV → m ≈ 2,110. With an illustrative ρ = 0.3: 1 + 2,109 × 0.3 = 633.7 → neff ≈ 160.

So the evidence is closer to that of 160 beats than 101,290. That is why per-patient variation dominates, and why more patients beat more beats per patient.

Build it

In code, a group split is one line. Pass the group ids, and scikit-learn keeps each group on one side:

from sklearn.model_selection import GroupKFold
# y: (N,) labels; groups: (N,) patient, recording, site or session id
for tr, te in GroupKFold(n_splits=5).split(np.zeros(len(y)), y, groups):
    assert not set(groups[tr]) & set(groups[te])   # no group on both sides

Check it

Ask the data owner: "What could make two examples secretly alike?" List every candidate group: patient, device, day, site, session, annotator. Split by the coarsest one that matches how the model will be used. A model for new hospitals needs a split by hospital, not just by patient.

You have 2,000 clips cut from 40 long recordings. How should you split?

Chapter 3 Step 3

Pick a backbone, then cache it

Choose the pretrained encoder by modality and by symmetry, run it once, and store its output

The backbone is the big pretrained model that turns raw signal into vectors (see Backbone + Head). Two questions pick it.

Modality first. Use a backbone pretrained on the same kind of signal. Audio models for audio, image models for images. A backbone that has seen millions of examples of your signal type has already learned its edges, onsets and textures.

Symmetry second. Ask what should not change the answer. A heartbeat is still a heartbeat if it starts 50 ms later: the task is shift-invariant in time. A forest tile is still forest if you rotate it: the task is rotation-invariant. A good backbone has these symmetries built in or learned. A 1-D convolution shares its weights across time, so it treats every moment alike. Inductive Bias and Symmetry covers this in depth.

CaseBackboneSymmetry it bringsShape out
SoundCLAP-style audio encoder, frozenShift in time (convolutions)B × T × D frames
ECGSmall 1-D CNN, trained on your beatsShift in timeB × D per beat window
SatelliteDINOv2-style ViT, frozen or lightly adaptedLearned from augmented cropsB × N × D patch tokens
GripperTiny causal 1-D CNN per sensorShift in time, and causalB × T × D per sensor

Some facts behind those picks. CLAP's audio encoder has 80.8 million weights and an embedding size of 2048 CLAP · §3.2. Its authors use it frozen, "as a feature extractor followed by training a classifier of 1 or 3 fully-connected layers" §3.3. DINOv2's ViT-B/14 cuts a 224 × 224 image into 16 × 16 patches plus one summary token: 257 tokens of 768 numbers. For ECG and touch there is no backbone as strong and as standard as these, so a small network trained on your own data is often the honest choice.

3aCache the features

A frozen backbone gives the same output for the same input. So run it once, save the output, and train every head on the saved copy. This is feature caching. The cost is disk space:

N
The number of examples.
T
Vectors kept per example: frames, tokens or 1 if you pool before saving.
D
The feature size.
b
Bytes per number: 2 for half precision (fp16), 4 for full.

Worked, EuroSAT's 27,000 tiles with a ViT-B/14: pooled, 27,000 × 768 × 2 = 41.5 MB. All 257 tokens: 257 × 41.5 MB = 10.7 GB.

What to cache

Pick a case and slide the dataset size. The top row is the shape at each step. The bars compare caching one pooled vector per example with caching every frame or token.

2,000

CLAP's D = 2048 (CLAP §3.2) and DINOv2 ViT-B/14's 257 × 768 are real. The frame counts for sound, the ECG window and the gripper sizes are illustrative. fp16 storage.

Build it

This is the first block of the project. The others follow in later chapters, and together they run as one file. Stand-in data and a stand-in encoder let it run with no downloads. Swap in your own.

import numpy as np
import torch
import torch.nn as nn

torch.manual_seed(0)
N, T, D_IN, D = 600, 50, 16, 64        # clips, frames, raw size, feature size
G = 30                                  # recordings: the groups

# Stand-in data. Replace with your clips, labels and recording ids.
raw = torch.randn(N, T, D_IN)           # (N, T, D_IN) raw frames
groups = torch.randint(0, G, (N,))      # (N,) which recording each clip came from
y = (torch.rand(N) < 0.1).float()       # (N,) rare positives, about 10%
raw[y == 1, 20:25] += 1.0               # a short "event" inside each positive

# Stand-in frozen backbone. Swap in a real encoder (CLAP, DINOv2, ...).
backbone = nn.Sequential(nn.Linear(D_IN, D), nn.GELU(), nn.Linear(D, D))
backbone.requires_grad_(False).eval()

with torch.no_grad():                   # run the backbone ONCE
    feats = torch.cat([backbone(x) for x in raw.split(64)])   # (N, T, D)
torch.save({"feats": feats.half(), "y": y, "groups": groups}, "cache.pt")
from torch.utils.data import Dataset, DataLoader

class CachedFeatures(Dataset):
    """Serves cached frames; the backbone never runs again."""
    def __init__(self, feats, y, idx):
        self.x = feats[idx].float()     # (n, T, D)
        self.y = y[idx]                 # (n,)
    def __len__(self):
        return len(self.y)
    def __getitem__(self, i):
        return self.x[i], self.y[i]     # (T, D), ()

Note what is saved: every frame, not a pooled vector. Pooling now would throw away the timing that the attention head in step 5 needs. Save the group ids too. Step 8 needs them.

Check it

Run the backbone twice on the same 10 inputs and compare. The outputs must match exactly. If they do not, dropout or BatchNorm is still in training mode, and your cache is noisy. Also re-check that cached and live features match after any change to preprocessing.

Why does the project cache all T frames per clip instead of one mean-pooled vector?

Chapter 4 Step 4

Trivial baselines first

Before anything clever, score the majority class, a kNN and a linear probe. They tell you what "good" means for this problem

A number alone means nothing. Is 89% accuracy good? On MIT-BIH's patient split, it is worse than doing nothing. The test set DS2 holds 44,196 normal beats, 1,836 S beats and 3,216 V beats Mousavi 2018 · §IV. A model that always says "normal" scores 44,196 / 49,248 = 89.7%. The methods in the table of Chapter 2 scored about 75% on that split. They were below the do-nothing line.

So before any real model, run three baselines, each one smarter than the last:

The baseline ladder

Each bar is trained live on cached toy features. Slide the labels and the feature quality. Look for the point where the bigger model stops beating the simple one by more than its error bar.

60
1.50

10-dimensional synthetic features, 25% positives, test set of 400. The MLP has 32 hidden units. Error bars are ±2 standard errors. Numbers are illustrative.

4aHow big is a real difference?

Two models that differ by one point may be the same model on a different test set. The standard error of an accuracy p measured on n test examples is:

p
The measured accuracy, as a fraction.
n
The number of independent test examples. With groups, use neff from step 2, not the raw count.
SE
How much p would wobble across test sets. A 95% interval is about p ± 2 SE.

Worked: p = 0.897, n = 49,248 beats → SE = 0.0014, ±0.3 points. With neff = 160 instead → SE = 0.024, ±4.8 points.

The second number is the honest one for a model that must work on new patients. It explains why ECG papers that differ by one point may not differ at all.

Build it

from sklearn.dummy import DummyClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score, GroupKFold

X = feats.float().mean(dim=1).numpy()       # (N, D) mean-pooled cache
cv = GroupKFold(n_splits=5)
for name, m in [("majority", DummyClassifier()),
                ("kNN", KNeighborsClassifier(5)),
                ("linear", LogisticRegression(max_iter=1000))]:
    s = cross_val_score(m, X, y.numpy(), groups=groups.numpy(), cv=cv,
                        scoring="average_precision")
    print(name, s.mean().round(3), "+/-", s.std().round(3))

Check it

Write the three baseline scores at the top of your results table, in the metric from step 1, with the grouped split from step 2. Every later row must beat the best of them by more than two standard errors. If none does, the features are the problem, not the head.

On MIT-BIH's patient split, a model scores 88% accuracy. What do you conclude?

Chapter 5 Step 5

Choose the head by the label's shape

The label's shape and where the signal sits pick the head from the toolkit. Then build it in PyTorch

The head is the small trained part on top of the backbone. You do not invent it. You read it off two facts: the shape of your label, and where the signal sits in the input.

Your labelWhere the signal isHeadShape
Whether, one per exampleSpread over the whole inputMean pool + linear (pooling)B×T×D → B×C
Whether, one per exampleIn a short partAttention pooling, trained as MILB×T×D → B×C, plus B×T weights
When, per time stepOnsets and offsetsFrame head + smoothingB×T×D → B×T×C
Where, per patchLocalDense head on patch tokensB×N×D → B×N×C
Same or differentIdentityProjection headB×D → B×K, unit length
Several labels at onceSharedMulti-task headsB×D → several outputs
Any, from two sensorsSplit across sensorsFusion block then a head(B×D1, B×D2) → B×C

The sound detector has "whether" labels and a short event. That points to attention pooling. Each frame gets a score from a learned query. The scores become weights, and the weights average the frames:

ht
The cached feature vector of frame t, of size D.
q
A learned query vector of size D: "what an interesting frame looks like".
at
The weight of frame t. All weights are positive and sum to 1.
z
One pooled vector for the clip. A linear layer turns it into the logit.

Worked, 4 frames, scores (0, 0, 0, 3): e3 = 20.09, sum = 23.09 → a = (0.043, 0.043, 0.043, 0.870). Mean pooling gives each frame 0.25.

Mean pooling dilutes a short event by the clip length. If the event fills 3 frames out of 48, its signal is cut to one sixteenth. Attention can put most of the weight on those 3 frames. Train both below and see.

Mean or attention pooling?

200 training clips of 48 frames. Positives hide an event somewhere. Shorten the event and watch mean pooling fall behind. The strip shows the trained attention weights on one test clip.

3

6-dimensional frames of unit noise; the event adds 2.5 to one dimension and 1.5 to another. Both heads are trained by gradient descent in your browser (mean: logistic regression on pooled vectors; attention: query, weights and bias trained jointly). Test set of 400 clips. Numbers are illustrative.

Build it

The second block of the project. It follows the formula line by line, with a LayerNorm first so all frames are on one scale.

class AttnPoolHead(nn.Module):
    """(B, T, D) frames -> (B,) logit, plus (B, T) attention weights."""
    def __init__(self, d):
        super().__init__()
        self.norm = nn.LayerNorm(d)
        self.score = nn.Linear(d, 1)                  # the learned query q
        self.out = nn.Linear(d, 1)                    # one logit: yes or no
    def forward(self, h):                             # h: (B, T, D)
        h = self.norm(h)
        a = self.score(h).squeeze(-1).softmax(dim=1)  # (B, T), rows sum to 1
        z = torch.einsum("bt,btd->bd", a, h)          # (B, D) weighted average
        return self.out(z).squeeze(-1), a             # (B,), (B, T)

Check it

Compare the head against the linear probe from step 4, on the same grouped folds. Then look at the attention weights on ten positive test clips. If the peaks sit on the event, the head learned the right thing. If they sit on silence or on clip edges, it found a shortcut.

Clips are labelled "has a cough or not", and a cough fills about 2% of the clip. Which head fits?

Chapter 6 Step 6

Decide what to unfreeze

The number of labels and the distance between the backbone's world and yours decide whether to probe, adapt or fine-tune

So far the backbone has stayed frozen. Should it? Two facts decide.

How many labels? Every weight you unfreeze can fit noise. A few hundred labels can safely train a few thousand weights, not eighty million.

How big is the domain gap? The domain gap is how far your data is from the backbone's training data. CLAP heard everyday sounds, so a glass-break detector is near. DINOv2 saw photos taken from the ground. A satellite tile seen from above, with no horizon and odd colours, is far. The EuroSAT authors note that ImageNet-pretrained networks had "not been trained on images other than RGB" EuroSAT · §IV-B, yet the tiles have 13 spectral bands.

Near domainFar domain
Few labelsFreeze; probe or small headAdapters or LoRA; or a nearer backbone
Many labelsUnfreeze the top blocksFull fine-tuning, small backbone learning rate

LoRA is the cheap middle path. It keeps each big weight matrix W frozen and learns a low-rank change: W + BA, where B and A are thin. Adapters and Fine-Tuning Depth derives it. The weight count is what makes it safe with few labels:

r
The rank of the change: often 4 to 16.
din, dout
The input and output sizes of the frozen matrix W.
weights
The trained weights LoRA adds to that one matrix.

Worked, ViT-B (d = 768), rank 8 on the query and value matrices of all 12 blocks: 8 × (768 + 768) = 12,288 per matrix × 24 matrices = 294,912 weights, against about 85 million in the backbone.

Probe, adapt or fine-tune?

A frozen map keeps 4 of 24 input directions. The domain gap turns the class signal out of those 4. Three models train live. Find where the probe wins, and where adapting wins.

60°
16

Inputs have 24 unit-noise dimensions. The frozen map keeps the first 4. The class signal points along a direction tilted by the gap angle away from them. Probe: head only (5 weights). Adapter: a rank-1 change to the map plus the head (33). Full: the whole 4 × 24 map plus the head (101). Each bar averages 3 training draws, tested on 1,000 points. Numbers are illustrative.

Build it

For the sound case, nothing unfreezes: the cache from step 3 is the backbone. Here is the third block, the training loop. The optimizer sees only the head. The loss weighs positives up, which is step 7.

def train_head(feats, y, idx, epochs=30, lr=1e-3):
    ds = CachedFeatures(feats, y, idx)
    head = AttnPoolHead(ds.x.shape[-1])
    pos = ds.y.sum().clamp(min=1)
    w = (len(ds) - pos) / pos                         # negatives per positive
    loss_fn = nn.BCEWithLogitsLoss(pos_weight=w)      # step 7: imbalance
    opt = torch.optim.AdamW(head.parameters(), lr=lr, weight_decay=1e-2)
    for _ in range(epochs):
        for h, t in DataLoader(ds, batch_size=32, shuffle=True):
            logit, _ = head(h)                        # (B,)
            loss = loss_fn(logit, t)
            opt.zero_grad()
            loss.backward()
            opt.step()
    return head.eval()

For the satellite case, you would instead wrap the backbone with LoRA and give the optimizer two groups: the head at 1e-3 and the LoRA weights at about 1e-4. The adapters lesson has that code.

Check it

Unfreezing must beat the frozen probe by more than the error bar, on the grouped split. If it does not, keep the frozen model: it is cheaper, faster and less likely to forget. Also plot training and test loss. A test loss that rises while training loss falls means you unfroze too much for your labels.

You have 300 labelled satellite tiles, and the backbone was pretrained on ground-level photos. What is the best first unfreezing choice?

Chapter 7 Step 7

Handle imbalance, then calibrate

Rare positives need weighting during training and a correction afterwards, or the model's probabilities lie

Rare classes are the rule in applied work. In MIT-BIH's training records, there are 45,796 normal beats and only 941 S beats Mousavi 2018 · §IV. That is 49 normal beats for each S beat. A model trained on this as-is learns that "normal" is almost always right.

Three common fixes change what the loss sees:

7aThe price: probabilities that lie

Every one of these fixes trains the model as if positives were common. So its output probabilities describe a balanced world. In the real world, positives are rare. A model trained at 50% positives says 0.8 for cases that are really positive far less often. You must shift its logit back:

ℓbal
The model's logit (log-odds) as trained on the balanced data.
π
The real positive rate where the model will run.
πtrain
The positive rate the loss saw: 0.5 after full balancing, so its log-odds term is 0.
ℓreal
The corrected logit. The probability is 1 / (1 + e−ℓ).

Worked: balanced model says 0.80 → ℓbal = ln(0.8 / 0.2) = 1.386. Real rate 2%: ln(0.02 / 0.98) = −3.892 → ℓreal = −2.506 → p = 0.075.

A "0.8" from the balanced model means a 7.5% chance in the field. Act on 0.8 as if it were 80%, and you will be wrong most of the time. The full story, with temperature scaling and reliability plots, is in Imbalance and Calibration.

The balanced model's promise

The diagonal is perfect calibration: when the model says 0.3, 30% are positive. Make positives rarer, then switch on the correction.

2%

Score bell curves 2 spreads apart. The balanced model's probability is the exact posterior at a 50% rate. Expected calibration error (ECE) is computed over 10 equal-width bins. Numbers are illustrative.

Check it

Plot a reliability diagram on grouped validation data: predicted probability on one axis, observed frequency on the other. Points on the diagonal mean the probabilities are honest. Then pick the threshold from the step 1 cost rule using the corrected probabilities.

A model trained on balanced data outputs 0.8. Positives are 2% of real cases. Roughly what is the real chance?

Chapter 8 Did it work?

Evaluate honestly

Grouped cross-validation, a learning curve, errors by slice, and the shuffle-label test that catches leaks you did not think of

Step 2 stopped the leaks you could name. Step 8 hunts the ones you could not. Four tools, in order.

Grouped cross-validation. One test split is one roll of the dice. Five grouped folds give five scores. Report their mean and spread. Every fold must hold out whole groups.

A learning curve. Train on 10%, 20%, and up to 80% of the training groups, and plot the score. If it still rises at the end, more labels will pay. The EuroSAT paper gives such a curve for a ResNet-50: 75.06% with 10% of the data for training, 88.53% at 20%, 94.45% at 50% and 96.43% at 80% EuroSAT · Table II. The steep start says labels mattered most at first. The flat end says more of the same labels would add little.

Errors by slice. Split the test errors by group, device, class or season. An average can hide one slice where the model fails. The EuroSAT confusion matrix shows the model mixing up the crop classes, and highway with river EuroSAT · §IV-A.

The shuffle-label test. Shuffle the labels, then run the whole pipeline again, from preprocessing to score. The labels now carry no information. The test score must fall to chance. If it stays above chance, something in your pipeline is reading the labels: a leak.

c
The chance score: 0.5 for balanced accuracy, the positive rate for average precision.
n
The number of independent test examples.
band
Where a shuffled-label score should land about 95% of the time. Above it means a leak.

Worked: c = 0.5, n = 80 → 2 × √(0.25 / 80) = 0.112 → band 0.39 to 0.61. A shuffled score of 0.68 is outside it: hunt for the leak.

The shuffle-label test

80 examples with 200 features, 3 of them useful. A kNN is scored by 5-fold cross-validation, live. Turn on a leak, then press "Shuffle labels". A clean pipeline drops to the chance band. A leaky one does not.

25% positives; the 3 useful features shift by 1.4 spreads for positives. Feature selection keeps the 10 features with the largest class-mean gap. Oversampling makes synthetic positives on lines between random pairs of positives, as SMOTE does. Each bar averages 5 random datasets. Balanced accuracy; chance is 0.5. Numbers are illustrative.

Build it

The fourth block: grouped cross-validation, then the same thing with shuffled labels. Because the whole pipeline sits inside the loop, the shuffle test covers every step.

from sklearn.model_selection import GroupKFold
from sklearn.metrics import average_precision_score

def grouped_cv(feats, y, groups, k=5):
    """Average precision on recordings the head never saw."""
    aps = []
    for tr, te in GroupKFold(n_splits=k).split(np.zeros(len(y)), y, groups):
        head = train_head(feats, y, tr)
        with torch.no_grad():
            logit, _ = head(feats[te].float())        # (n_test,)
        aps.append(average_precision_score(y[te].numpy(), logit.numpy()))
    return float(np.mean(aps)), float(np.std(aps))

c = torch.load("cache.pt")
feats, y, groups = c["feats"], c["y"], c["groups"]
print("grouped AP  %.3f +/- %.3f" % grouped_cv(feats, y, groups))
y_shuf = y[torch.randperm(len(y))]                    # the shuffle-label test
print("shuffled AP %.3f +/- %.3f" % grouped_cv(feats, y_shuf, groups))
print("chance (the positive rate) %.3f" % y.mean().item())

On the stand-in data, this printed a grouped AP of 0.996, a shuffled AP of 0.151, and a chance level of 0.088. The shuffled score sits near chance. With only about 50 positives, average precision is noisy, so a small gap is expected.

Check it

Write a one-page evaluation: grouped CV mean and spread, the three baselines, the learning curve, the worst three slices, and the shuffled-label score with its chance band. If any line surprises you, stop and find out why before you ship.

With shuffled labels, your pipeline scores 0.71 balanced accuracy on 200 test examples. What does it mean?

Chapter 9 Step 9

Ship: thresholds, latency, drift

Pick the threshold on validation data, check the time budget, and watch the inputs change after launch

A shipped model is a score, a threshold, a clock and a monitor. Each one can fail on its own.

9aThe threshold

Pick it on grouped validation data with the cost rule from step 1. Never pick it on the test set: that tunes the model to the test, and the test score is no longer honest. Write the threshold into the model's config, next to the backbone version and the cache date.

9bThe clock

A detector is late by more than its compute time. If it looks at a window centred on now, it must wait for the future half of that window. Then it waits up to one hop for the next window to start. The worst-case delay is:

Wfuture
How much of the window lies after the moment you report on: 0 for a causal window that only looks back, W / 2 for a centred one.
hop
The time between two windows.
tmodel
Feature and head compute time per window, plus data transfer.

Worked: centred 1 s window, 0.5 s hop, 10 ms model → 500 + 500 + 10 = 1,010 ms. Causal window, 5 ms hop, 1 ms model → 0 + 5 + 1 = 6 ms.

Real systems show why this matters. Mohammad and colleagues detected contact on a parallel robot with a model-based observer, then fused one low-cost IMU with the joint encoders. Detection time fell from 6–47 ms to 3–39 ms, a cut of 17% to 50% across their four contact tests Mohammad 2025 · Table III. Their method is physics, not learning, but the budget lesson is the same: for touch, milliseconds are the metric.

The clock and the drift

Left: the time budget. Choose a deadline, then set the window, hop and model time. Right: after launch, the sensor drifts. Slide the drift and watch false alarms climb at a fixed threshold.

200
50
8
0.00

Drift moves the negative score distribution toward the threshold by the slider amount, in spreads. False alarms assume 3,600 windows per hour and a threshold set for 1 per hour before drift. Population stability index (PSI) over 10 bins. Numbers are illustrative.

9cThe monitor

After launch, inputs change. Microphones age, a hospital buys a new ECG machine, seasons change the satellite colours. A model with a fixed threshold then raises more or fewer alarms, with no error message. So log two cheap numbers every day: the alarm rate, and the distribution of the model's scores. Compare both with the validation week. A shift means it is time to re-label a sample and re-check.

Build it

The fifth block picks the threshold on held-out recordings, times the head, and logs the alarm rate.

def pick_threshold(logits, y, c_fn=20.0, c_fp=1.0):
    """The cut with the lowest cost on VALIDATION data."""
    best_t, best_cost = 0.0, float("inf")
    for t in torch.unique(logits):
        pred = logits >= t
        cost = (c_fn * (~pred & (y == 1)).sum()
                + c_fp * (pred & (y == 0)).sum()).item()
        if cost < best_cost:
            best_t, best_cost = t.item(), cost
    return best_t

val = torch.from_numpy(np.isin(groups.numpy(), [0, 1, 2, 3, 4, 5]))   # 6 held-out recordings
head = train_head(feats, y, torch.where(~val)[0])
with torch.no_grad():
    logit_val, attn = head(feats[val].float())       # (n_val,), (n_val, T)
t_star = pick_threshold(logit_val, y[val])
import time
x1 = feats[:1].float()                                # one clip: (1, T, D)
t0 = time.perf_counter()
with torch.no_grad():
    for _ in range(100):
        head(x1)
print("head latency %.3f ms" % ((time.perf_counter() - t0) * 10))
alarm_rate = (logit_val >= t_star).float().mean().item()
print("alarm rate on validation %.3f  (log it; watch it drift)" % alarm_rate)

On the stand-in data the head took about 0.05 ms per clip on a laptop CPU, and raised alarms on 8.7% of validation clips. The backbone's time is not in that number. Time the whole path, from raw signal to decision, on the real device.

A contact detector uses a centred 100 ms window, a 20 ms hop and 5 ms of compute. What is its worst-case delay?

Chapter 10 What else What not

Four cases, start to finish

Walk each problem through all nine steps, see what changes between fields and what stays, and learn when the defaults fail

Here are the four problems from Chapter 0, each decided step by step. Slide through the steps to compare the four side by side.

One step, four fields

Slide through the steps. Each card is the decision for one case. The line under it is the reason.

2 Audit

Decisions are this lesson's recommendations. Real numbers in the cards are from the cited sources; others are labelled as examples.

Case (a): a sound detector on frozen CLAP-style features

Case (b): arrhythmia beats in ECG

Case (c): satellite tiles to land use

Case (d): gripper contact from touch and joints

What changes, and what stays

Across the four, the backbone, the head and the metric all changed. The questions did not. In every case the group split, the trivial baselines and the shuffle test stayed. These three are the cheapest steps and the ones most often skipped.

When the recipe's defaults fail

DefaultFails whenDo instead
Frozen pretrained backboneNo backbone near your signal (touch, odd sensors)Small model from scratch, or self-supervised pretraining on your unlabelled data
Cache the featuresAugmentation matters, or the backbone trainsCache a few augmented copies, or run the backbone live
Group split by sourceThere are very few groups (3 patients)Leave-one-group-out CV, and report the spread honestly
Neural headThe data is a table of measured valuesGradient-boosted trees
One thresholdCosts differ across users or sitesA threshold per site, or ship calibrated probabilities
Which steps stayed the same across all four cases?

Chapter 11

Connections

Build a recipe card for your own problem, keep the one-screen cheat sheet, know the limits, and see where the series goes

Describe your own problem below. The card gives a first decision for each of the nine steps.

Your recipe card

Set five facts about your problem. The card updates each step's first decision.

First decisions, not final answers. Each step has its own chapter above.

The cheat sheet

StepDecideCheck
1 FrameInput, output, whether / when / where, a metric that matches the cost; say yes when p ≥ cFP / (cFP + cFN)The "always no" score in your metric; the asker agrees
2 AuditSize, balance, label noise, groups; split by group; neff = n / (1 + (m − 1)ρ)No group on both sides; read 50 labels
3 BackboneBy modality, symmetry and deadline; cache frames: N·T·D·b bytesSame input, same features, twice
4 BaselinesMajority, kNN, linear probe on the cacheLater models beat them by > 2 SE
5 HeadFrom label shape: pool, attention + MIL, frame, dense, projection, multi-task, fusionBeats the probe; attention lands on the event
6 UnfreezeFew + near: freeze. Far: LoRA, r(din + dout) weights. Many + far: full, small LRBeats the frozen probe; test loss does not rise
7 ImbalanceWeights or balanced batches; oversample inside the training fold only; correct logits by the log-odds of the real rateReliability diagram on the diagonal
8 EvaluateGrouped CV, learning curve, slices, shuffle testShuffled score inside c ± 2√(c(1 − c)/n)
9 ShipThreshold on validation; delay = Wfuture + hop + tmodel; log alarm rate and score driftDelay under the deadline on the real device

Honest limits

Where this sits, and where to go next

This is the last lesson of The Builder's Toolkit. Each step points back to a block:

The takeaway. Frame the task by its cost, split by the groups that make examples alike, beat three trivial baselines, choose the head from the label's shape, unfreeze only what your labels can pay for, correct for rarity, and prove it all with grouped CV and a shuffled-label run. The model changes with the field. The recipe does not.
A new problem: classify cough sounds as sick or healthy, 600 labelled coughs from 40 people. What is the first experiment?

Now press Present or Teach and walk one of the four cases through all nine steps, out loud and from memory. Then go back to the hero, run the ECG case with a random split, and see how far the claim drifts from the truth.

The Cross-Domain Recipe, the capstone of The Builder's Toolkit. Sources: S. Mousavi, F. Afghah, U. R. Acharya, "Inter- and Intra-Patient ECG Heartbeat Classification for Arrhythmia Detection: A Sequence to Sequence Deep Learning Approach" (2018, arXiv); P. Helber, B. Bischke, A. Dengel, D. Borth, "EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification" (IEEE JSTARS 2019, arXiv); B. Elizalde, S. Deshmukh, M. Al Ismail, H. Wang, "CLAP: Learning Audio Concepts from Natural Language Supervision" (2022, arXiv); A. Mohammad, J. Piosik, D. Lehmann, T. Seel, M. Schappler, "Fast Contact Detection via Fusion of Joint and Inertial Sensors for Parallel Robots in Human-Robot Collaboration" (IEEE RA-L 2025, arXiv); M. Oquab et al., "DINOv2: Learning Robust Visual Features without Supervision" (TMLR 2024, arXiv); E. Hu et al., "LoRA: Low-Rank Adaptation of Large Language Models" (ICLR 2022, arXiv). The cost threshold, design effect, standard error and prior-shift correction are standard results.
Back to Gleams