Backbone + Head

Almost every applied model has two parts. A big backbone, pretrained on lots of data, turns raw signal into vectors. A small head, trained on your few labels, turns those vectors into your answer. With good features, a simple head wins when labels are few.

Learn what a representation is, why pretrained features transfer, how to wire the parts with the right shapes, what to freeze, how to build it in PyTorch, and how to prove it worked.

Pick a backbone and a head. Slide the number of labels. The curves are real: each point is a head trained in your browser, right now.

You need vectors, a dot product and the idea of training by gradient descent. We build everything else from zero.

Good features, few labels

Ready
4

Synthetic 12-dimensional features for 3 classes; only two dimensions are drawn. "Random" hides each class in two opposite blobs (not linearly separable); "weak" overlaps; "strong" separates. Each curve point averages 3 training draws, tested on 300 held-out points. Linear: softmax regression. MLP: 32 hidden ReLU units. kNN: 5 neighbours. The dashed red curve is random features with an MLP head, for contrast. All numbers are illustrative.

Chapter 0

One sound, four fields

Start from one real build: a frozen audio model plus a tiny head that finds one sound. Then see the same pattern in vision, text and sensor data

Say you want a detector for one specific sound. Maybe it is your doorbell, a smoke alarm, or one bird's call. You have about two hundred labelled clips. That is far too few to train an audio model from scratch.

So you borrow one. CLAP is an audio model that was pretrained on 128k pairs of audio and text CLAP · §3.2. You run your clips through its audio encoder and keep its weights fixed. We say the encoder is frozen: no weight changes during your training. Out comes a short list of vectors per clip, one for each slice of time.

Then you train only a small piece on top. Here it is an attention-pooling head: it learns which time slices matter, averages them with those weights, and makes one yes-or-no score. It has a few thousand weights. It trains in seconds. And it works.

The big frozen part is the backbone. The small trained part is the head. The same split hides inside almost every applied model you will meet. Pick a field below and look.

Same pattern, four fields

Pick a field. Each box is one step; the line under it is the shape of the data. Turn on "show what trains" to see how little of the model learns from your labels.

CLAP's audio encoder (CNN14, embedding size 2048), BERT-base (110 million weights) and the 44.1 kHz rate are from the CLAP paper (§3.2). The number of time frames T, the class counts and the sensor format are illustrative. B is the batch size.

What changes and what stays

Across the four fields, the backbone changes: an audio encoder, a vision transformer, a text transformer, a signal encoder. The head changes a little. But the plan stays the same. Raw signal goes in. The backbone makes vectors. A short step squeezes them to one vector, or keeps one per position. A small head turns that into scores. A loss compares the scores with your labels, and only the head learns from it.

The CLAP authors evaluate their own model in exactly this way. They use it "as a feature extractor followed by training a classifier of 1 or 3 fully-connected layers" CLAP · §3.2. The pattern is not a hack. It is how the field measures what a pretrained model knows.

The road map of this lesson

Each part of the lesson answers one of five questions. Why does it work? How do you build it? Where else does it work? When should you not use it? And did it work?

  1. What a representation is: a learned map where useful differences become simple. Why
  2. Features that transfer: separability, and what the evidence says. Why
  3. Small heads need few labels: bias, variance and a worked count. Why
  4. The anatomy, with shapes: backbone, neck, head, loss. How
  5. A field guide to heads: and an attention-pooling head you train live. How
  6. Freeze or tune?: the 2 × 2 of data size and domain gap. How
  7. Build it in PyTorch: by hand, from scratch, then in one line. How
  8. Cache the features: the biggest practical speed-up. How
  9. Did it work?: probes, kNN, error bars and shuffled labels. Did it work
  10. Where else, and when not: six fields and a decision table. What else / what not
  11. Connections: the cheat sheet, honest limits, and the rest of the series.

The Builder's Toolkit

This lesson is the hub of a twelve-part series. Each later lesson takes one block from here and goes deep.

  1. Backbone + Head (this lesson): the pattern, and the judgement to use it.
  2. Probing Representations: linear and kNN probes, and what each layer knows.
  3. Attention-Pooling Heads: learn which frames or patches matter.
  4. Weak Labels and MIL: train with one label per clip when you do not know when the event happened.
  5. Event-Detection Heads: per-frame heads that say when a sound starts and stops.
  6. Adapters and Fine-Tuning Depth: LoRA, adapters, and how deep to tune.
  7. Projection Heads and Metric Learning: heads that shape distance for search and few-shot.
  8. Multi-Task Heads: one backbone, many heads, and how their losses compete.
  9. Fusion Blocks: join two backbones, such as audio with video.
  10. Inductive Bias and Symmetry: build what you know into the block.
  11. Imbalance and Calibration: rare classes, thresholds and honest scores.
  12. The Cross-Domain Recipe: the whole method, applied to a new problem in a new field.
In the sound detector, which part learns from your two hundred labelled clips?

Chapter 1 Why it works

What a representation is

A learned map from raw signal to vectors, built so that the differences you care about become simple

Look at the raw numbers of a sound clip. A one-second clip at 44.1 kHz is 44,100 numbers. A doorbell and a dog bark differ in all of them, in tangled ways. No single number says "doorbell". If you draw two clips as points in that huge space, the closeness of the points says almost nothing about how alike the sounds are.

A representation is a different set of coordinates for the same input. A network f maps the raw input x to a vector z. A good map puts clips of the same kind near each other and different kinds far apart. Then a simple rule on z, such as "which side of a line?", can tell them apart.

x
The raw input: samples of a waveform, pixels of an image, token ids of a sentence.
f
The backbone: a fixed, pretrained function. Its output is the representation.
z
The feature vector, or embedding: D numbers, for example D = 768.
W, b
The linear head: a C × D weight matrix and C biases, one row per class.
s
The C class scores, called logits. The largest one wins.

Worked, D = 2, C = 2: z = (1.0, −0.5), W = [[2, 0], [−1, 1]], b = (0, 0.5) → s1 = 2.0, s2 = −1 − 0.5 + 0.5 = −1.0 → class 1 wins

Each row of W is a direction in feature space. A score is a dot product: how far z points along that direction. The head can only draw straight boundaries. So all the hard work of bending the space must already be done by f.

Bend the space, then draw a line

Three classes sit on three rings. A linear head is trained live on whatever coordinates you give it. Slide from the raw coordinates to the learned features and watch its accuracy.

0.00

The "learned features" here are hand-made: distance from the centre and angle. A real backbone learns its map from data. 180 points; the head is softmax regression trained for 150 steps. Numbers are illustrative.

The manifold picture

Why can a map like this exist at all? Real signals are not spread evenly through their huge raw space. Natural images, speech and engine noise fill only thin, curved sheets inside it. Mathematicians call such a sheet a manifold. Moving along it changes the content a little: the pitch of a bark, the angle of a cat.

A good backbone flattens these sheets and pulls apart sheets of different kinds. After that, the classes are near-straight clumps. That is the moment a linear head starts to work. In the device, the rings are curved sheets; the learned map unrolls them into flat bands.

Why does the same linear head fail on the raw ring coordinates but succeed on the learned features?

Chapter 2 Why it works

Features that transfer

Measure how separable the classes are, see how that sets the best a linear head can do, and read what the evidence says about pretrained features

Features learned for one task help another task because different tasks share structure. Every photo is made of edges, textures and parts. Every sound is made of onsets, harmonics and noise bands. A backbone that learned these pieces from millions of examples has already done most of the work for your task.

We can make "good features" precise. Picture two classes in feature space as two round clouds. Let the distance between their centres, measured in units of the clouds' spread, be d. This is the separation. If d is large, the clouds barely touch. The best possible straight boundary then makes few mistakes:

d
The separation: distance between the two class centres divided by the spread (standard deviation) of each cloud.
Φ
The bell-curve CDF: the chance that a standard normal number is below its input. Φ(1) = 0.841, Φ(2) = 0.977.
acc*
The best accuracy any rule can reach for two equal round Gaussian classes. Here the best rule is a straight line halfway between the centres.

Worked: d = 2 → Φ(1) = 84.1%.   d = 4 → Φ(2) = 97.7%. Doubling the separation cuts the error from 15.9% to 2.3%.

Why d / 2? The boundary sits halfway, d / 2 spreads from each centre. A point is wrong only if its noise pushes it more than d / 2 spreads toward the other side. That has chance 1 − Φ(d / 2). A better backbone raises d. Nothing a head does afterwards can beat Φ(d / 2).

Separation sets the ceiling

Raise the separation and watch the ceiling rise. Then add useless dimensions: the ceiling stays, but a head trained on few labels falls below it. More labels bring it back.

2.0
0
4

Two Gaussian classes with unit spread in every dimension. The probe is logistic regression trained on fresh samples 16 times; its accuracy is computed exactly from its boundary, not estimated. Numbers are illustrative.

What the evidence says

Three studies shape how practitioners think about this.

General first, specific last. Yosinski and colleagues split ImageNet into two halves and moved trained layers from one half's network to the other's Yosinski 2014 · §4. Early layers learn edge and colour detectors that transfer almost freely. Later layers are more specific. With the copied layers frozen up to the top, accuracy fell by 8% when the two halves were similar. It fell by 25% when one half held only man-made objects and the other only natural ones §4.3. Even features from a distant task beat random weights.

Better backbones give better features. Kornblith, Shlens and Le tested 16 networks on 12 datasets Kornblith 2019 · Abstract. A network's ImageNet accuracy predicted its transfer accuracy with a linear head on frozen features very closely (correlation r = 0.99), and with full fine-tuning too (r = 0.96).

Frozen features can be enough. DINOv2's largest model reaches 86.5% on ImageNet with only a linear head on frozen features, and 83.5% with a kNN rule DINOv2 · Table 4. Fine-tuning the whole backbone adds 2.0 points, to 88.5% Table 5. A linear probe on CLIP's features beat the strongest model in the authors' test suite, Noisy Student EfficientNet-L2, on 21 of 27 datasets CLIP · §3.2.

Two classes are 3 spreads apart (d = 3). What is the best accuracy any straight boundary can reach?

Chapter 3 Why it works

Small heads need few labels

Split the error into bias and variance, count the weights each choice trains, and see why "simple head on good features" beats "big model on little data"

A trained model makes two kinds of error. Bias is the error that stays even with endless data: the model is too simple for the truth. Variance is the error from fitting the noise in your particular few examples: the model is too free for the data.

Good features attack bias: the truth becomes a simple function of z. A small head attacks variance: it has few weights to fit noise with. Together they need few labels. A big model on raw data has low bias but huge variance. It needs many labels.

For the simplest head there is an exact formula. Fit a linear model with p weights by least squares on n examples, with noise of variance σ2. If the features are Gaussian, the expected extra test error is:

excess
How much worse the fitted model is than the true model, on new data. This is the variance part.
σ2
The label noise: what even the true model cannot predict.
p
The number of weights the head fits.
n
The number of labelled examples. The formula needs n > p + 1.

Worked, σ2 = 1, n = 100: p = 10 → 10/89 = 0.11.   p = 50 → 50/49 = 1.02.   p = 90 → 90/9 = 10.

The error grows slowly while p is small next to n, then explodes as p nears n. If the truth needs 5 features and your head has fewer, bias appears instead. Try it below.

Bias, variance and head size

The truth uses 5 features. Your head uses the first p of 120. Below 5 it is too simple (bias). Far above 5 it fits noise (variance). Add labels and the valley widens.

5
100

Features are independent standard normals; the truth is the sum of the first 5 features plus noise of variance 1. The dot averages 8 least-squares fits; error is computed exactly from the fitted weights. Numbers are illustrative.

Count the weights

Now apply this to real choices. Say D = 768 features and C = 2 classes.

What you trainWeights trainedHow
Linear head1,538768 × 2 + 2
MLP head, 512 hidden394,754768 × 512 + 512 + 512 × 2 + 2
The whole ViT-Base backboneabout 86 millionViT paper, Table 1

With two hundred clips, the linear head has about 8 examples per weight. The full backbone has about 430,000 weights per example. The formula is exact only for least squares, and deep networks are kinder than it says. But the direction is right, and it is the reason behind the whole pattern.

With n = 100 examples and noise variance 1, a least-squares head with p = 50 weights has expected excess error about:

Chapter 4 Build it

The anatomy, with shapes

Follow one batch through backbone, neck, head and loss, and read the tensor shape at every step for image, audio, text and time series

Every backbone-and-head model has four parts. Learn their names once and you can read any model card.

Most bugs in this pattern are shape bugs. So we track shapes. We write them like B × T × D: batch size, then sequence length, then feature size. A vision transformer, for example, cuts an image into square patches of P × P pixels. Each patch becomes one token:

H, W
Image height and width in pixels.
P
Patch size: 16 for ViT-B/16, 14 for DINOv2 models.
+ 1
One extra CLS token: a learned slot that gathers a summary of the whole image.
N
The number of tokens the backbone outputs, each a D-vector.

Worked: 224 / 16 = 14 → 14 × 14 + 1 = 197 tokens of D = 768. With P = 14: 16 × 16 + 1 = 257.

Follow the shapes

Pick a kind of data. Step through the pipeline. Each block is drawn to the size of one example's tensor, so you can see where the data shrinks.

input

Block area is drawn on a log scale of the number of values per example. Audio frame counts and the sensor format are illustrative; ViT and BERT sizes are standard.

Why each step exists

The neck exists because heads want one vector per example, and backbones give many. An image gives 197 tokens. A 10-second clip gives dozens of frames. A sentence gives one vector per token. To classify the whole thing, you must squeeze the sequence. Mean pooling averages all vectors. Taking the CLS token uses the backbone's own summary. Attention pooling learns the weights (Chapter 5).

For dense tasks you skip the squeeze. Segmentation keeps one vector per patch. Event detection keeps one per frame. The head then runs on every position: B × T × D becomes B × T × C.

A DINOv2 ViT-B/14 sees a 224 × 224 image. What shape leaves the backbone for a batch of 8?

Chapter 5 Build it

A field guide to heads

Meet the seven common kinds of head, learn when each one fits, and train an attention-pooling head live to see it find a short event

A head is any small trainable module after the backbone. Here are the seven you will meet most. They differ in what shape they take in and what they put out.

HeadShape in → outUse it when
LinearB×D → B×CFeatures are good and labels are few. Always your first baseline.
MLPB×D → B×H → B×CClasses curve in feature space and you have thousands of labels.
Pooling (mean, max, attention)B×T×D → B×DOne label for a whole clip, image or document. Attention when the evidence is short.
Sequence (per-frame, per-token)B×T×D → B×T×CYou need a label at each time step: event onsets, named entities.
Projection / metricB×D → B×K, unit lengthYou compare examples by distance: search, few-shot, face matching.
Dense / decoderB×N×D → B×C×H×WEvery pixel needs a label: segmentation, depth.
Multi-taskone B×D → several outputsSeveral labels share one input; one backbone pass serves all heads.

Back to the sound detector. A 10-second clip holds a doorbell for maybe half a second. Mean pooling averages the bell with nine and a half seconds of other sound, so the bell's signal shrinks. An attention-pooling head learns a query vector q. It scores each frame against q, turns the scores into weights that sum to 1, and averages with those weights:

ht
The backbone's feature vector for frame t, one of T.
q
A learned query: "what does the event look like?" It is the only new part besides the output layer.
at
The attention weight of frame t: escore divided by the sum of escore over all frames.
z
The pooled clip vector. A linear layer then turns it into the score.

Worked, 3 frames with scores (0, 0, 2): e0 = 1, e2 = 7.39, sum 9.39 → a = (0.11, 0.11, 0.79). Mean pooling would give each frame 0.33.

Train a pooling head

Half the clips hold a 2-to-3-frame event, hidden among 40 frames of background. Pick a pooling rule and press Train. The bars show the attention weights on one test clip; the shaded band is where the event really is.

Synthetic frame features: 40 frames of 6 numbers each, background noise plus a slow drift. The event adds a fixed direction spread over four of the six numbers, so no single number gives it away. 160 training clips, 200 test clips, 300 steps of Adam. Numbers are illustrative.

The attention head wins because its weights can follow the event, wherever it is. Max pooling looks at each feature number on its own, so an event spread across several numbers hides in the noise. The weights at also tell you when the event happened, for free. The Attention-Pooling Heads lesson takes this block apart.

Why does mean pooling struggle to detect a half-second sound in a 10-second clip?

Chapter 6 Build it

Freeze or tune?

Decide how much of the backbone to train from two facts: how many labels you have, and how far your data is from what the backbone saw

Freezing is one end of a scale. At the other end, you train every weight of the backbone. In between are two useful middle steps.

Two facts pick the step. The first is how many labels you have. The second is the domain gap: how different your data is from the backbone's pretraining data. Bird calls are near to CLAP's training sounds. Ultrasound scans are far from ImageNet photos.

The freeze-or-tune chart

Drag the dot, or use the sliders, to place your project. The chart names the usual strategy and counts the weights it trains on a ViT-Base backbone.

200
near

ViT-Base: 12 blocks, D = 768, MLP width 3072; weight counts are computed from those sizes, for C = 10 classes. The region borders are a rule of thumb, not a law.

Why do adapters work? A weight update during fine-tuning tends to be simple. LoRA bets that the change to each d × d matrix W has low rank r. So it learns the change as the product of two thin matrices:

W
The frozen pretrained d × d matrix.
B
A trainable d × r matrix, started at zero so that W′ = W at the start.
A
A trainable r × d matrix. Rank r is small, often 4 to 16.
W′
The adapted matrix, used in the forward pass.

Worked, d = 768, r = 8: B A has 2 × 768 × 8 = 12,288 weights; W has 768² = 589,824. LoRA trains 2.1% of that matrix.

What the evidence says about the axes

Data size matters most. With 47 to 800 total examples, a linear head on frozen features matched or beat full fine-tuning across Kornblith's datasets Kornblith 2019 · §4.7. With more data, fine-tuning pulls ahead. Domain gap matters next. On Stanford Cars, fine-tuning from ImageNet reached 93.3% and training from scratch reached 92.7%: the fine-grained car models are poorly covered by ImageNet's few car classes §4.5. Yet fine-tuning still reached its accuracy in far fewer steps: 17 times fewer on average §4.6.

When you do tune, give the backbone a smaller learning rate than the head. The head starts from random and must move far. The backbone starts from good weights and should move a little. The CLAP paper used a learning rate of 10−3 for heads on frozen features, and 10−4 when it unfroze the encoder CLAP · §3.2.

LoRA with rank r = 4 adapts one 1024 × 1024 matrix. How many weights does it train for that matrix?

Chapter 7 Build it

Build it in PyTorch

Do one training step by hand, write the frozen backbone and head from scratch, then meet the one-line library versions

By hand first. Take the Chapter 1 head: z = (1.0, −0.5), scores s = (2.0, −1.0), and the true class is 1. The softmax turns scores into probabilities. The cross-entropy loss is minus the log of the true class's probability:

p
Class probabilities: esc divided by the sum over all classes.
L
The loss for this example; y is the index of the true class.
∂L/∂s
The gradient of the loss with respect to the scores. Then ∂L/∂W = (p − onehot(y)) zT.

Worked: e2 = 7.389, e−1 = 0.368 → p = (0.953, 0.047), L = −ln 0.953 = 0.049, ∂L/∂s = (−0.047, 0.047). Row 1 of W moves along +z; row 2 moves along −z.

The gradient stops at the head. Nothing flows into f if the backbone is frozen. Getting that right in PyTorch takes four switches, and each one does something different. Flip them below.

Four switches that freeze a backbone

The safe setting is all four on. Turn them off one at a time and read what breaks.

Backbone: ViT-Base, about 86 million weights (ViT paper, Table 1), modelled as 85.8 million in 12 blocks. Head: 768 × 10 + 10 weights. Memory counts fp32 gradients (4 bytes) and Adam's two moment buffers (8 bytes) per trained weight; activations are not counted.

From scratch. Here is a frozen wrapper and an attention-pooling head. The wrapper overrides train(), because calling model.train() on the whole model would otherwise switch the backbone back to training mode.

import torch
import torch.nn as nn

class FrozenBackbone(nn.Module):
    """Wraps any pretrained encoder so it never trains."""
    def __init__(self, encoder):
        super().__init__()
        self.encoder = encoder
        self.encoder.requires_grad_(False)   # no weight gets a gradient
        self.encoder.eval()                  # dropout off, BatchNorm uses stored stats

    def train(self, mode=True):              # model.train() must not wake it up
        super().train(mode)
        self.encoder.eval()
        return self

    @torch.no_grad()                         # keep no graph: less memory, faster
    def forward(self, x):
        return self.encoder(x)

class AttentionPoolHead(nn.Module):
    """(B, T, D) frames -> (B, C) logits, plus (B, T) weights."""
    def __init__(self, d, c):
        super().__init__()
        self.norm = nn.LayerNorm(d)
        self.score = nn.Linear(d, 1)          # s_t = q . h_t + bias
        self.out = nn.Linear(d, c)

    def forward(self, h):                    # h: (B, T, D)
        h = self.norm(h)
        a = self.score(h).softmax(dim=1)     # (B, T, 1), sums to 1 over T
        z = (a * h).sum(dim=1)               # (B, D) weighted average
        return self.out(z), a.squeeze(-1)    # (B, C), (B, T)

And the training loop, in the same file. The stand-in encoder lets the code run with no downloads; swap in a real one later. The labels are random on purpose: this is the smoke test from Chapter 9, so expect chance accuracy.

from torch.utils.data import DataLoader, TensorDataset

torch.manual_seed(0)
B, T, D, C = 32, 50, 64, 2
# Stand-in pretrained encoder: (N, T, 16) raw frames -> (N, T, D)
encoder = nn.Sequential(nn.Linear(16, D), nn.GELU(), nn.Linear(D, D))
backbone = FrozenBackbone(encoder)
raw = torch.randn(512, T, 16)                    # 512 clips
labels = torch.randint(0, C, (512,))             # random: a smoke test

feats = torch.cat([backbone(x) for x in raw.split(B)])   # (512, T, D), run ONCE
head = AttentionPoolHead(D, C)
opt = torch.optim.AdamW(head.parameters(), lr=1e-3, weight_decay=1e-2)
loader = DataLoader(TensorDataset(feats, labels), batch_size=B, shuffle=True)
for epoch in range(30):
    for h, y in loader:                          # h: (B, T, D), y: (B,)
        logits, _ = head(h)
        loss = nn.functional.cross_entropy(logits, y)
        opt.zero_grad()
        loss.backward()
        opt.step()
print(f"last batch loss {loss.item():.3f}")

The library one-liners. A linear probe on cached, pooled features is one line of scikit-learn. A timm model with a new head, and everything else frozen, is three.

from sklearn.linear_model import LogisticRegression
X = feats.mean(dim=1).numpy()                     # (512, D) mean-pooled
probe = LogisticRegression(max_iter=1000).fit(X, labels.numpy())

import timm
model = timm.create_model("vit_base_patch16_224", pretrained=True, num_classes=5)
for name, p in model.named_parameters():
    p.requires_grad = name.startswith("head.")     # only the new head trains
You called requires_grad_(False) on a CNN backbone but forgot eval(). What goes wrong?

Chapter 8 Build it

Cache the features

Run the frozen backbone once, store its output, and train heads on the stored features many times faster

A frozen backbone gives the same output for the same input every time. So why compute it every epoch? Run it once over the whole dataset. Save the features to disk. Then train the head on the saved features. This is feature caching, and it is the biggest practical speed-up in this pattern.

Let tb be the backbone's time per example, and th the head's. Training for E epochs over N examples, and trying S head settings, costs:

tb
Backbone time per example. For a big model, many milliseconds.
th
Head time per example, often hundreds of times smaller.
E
Training epochs.
S
Head settings you try: learning rates, head types, seeds. Each one reuses the cache.

Worked: S = 1, E = 30, tb = 20 ms, th = 0.05 ms → 30 × 20.05 / (20 + 1.5) = 601.5 / 21.5 = 28× faster.

The cost of not caching

Move the sliders. The bars compare wall-clock time; the line below gives the cache's size on disk. Try a sweep of 10 head settings.

10,000
30
1
20

Head time 0.05 ms per example. Cache size at 2 bytes per number (fp16), with D = 768 and T = 197 tokens. Timings are illustrative; measure your own.

The choice of what to cache matters. Pooled vectors are tiny: 10,000 × 768 numbers at 2 bytes is about 15 MB. But then you cannot train an attention-pooling head, because the frames are gone. Caching every token keeps that option and costs 197 times more. A common middle path: cache all frames for audio clips, which have dozens of frames, and pooled vectors for images.

You try 10 head settings for 30 epochs each, with tb = 20 ms and th = 0.05 ms. Roughly how much faster is caching?

Chapter 9 Did it work?

Did it work?

Compare every head with a linear probe and a kNN baseline, put error bars on accuracy, draw a learning curve, and run the shuffled-label test

An accuracy number alone means little. 90% is great for ten balanced classes and poor if one class is 90% of the data. You need baselines: simple methods your head must beat to earn its complexity. Run these four checks on every project.

  1. Majority class. Always guess the most common label. This is the floor.
  2. kNN probe. Label a test example by its k nearest training examples in feature space. It has no trained weights, so it measures the features alone.
  3. Linear probe. A linear head on frozen features. If your fancy head barely beats it, keep the probe.
  4. Shuffled labels. Shuffle the training labels and train again. Test accuracy must fall to chance. If it does not, information is leaking from test to train.

Run the checks

Pick the feature quality and the number of labels, then press Run. Hollow bars are training accuracy; solid bars are test accuracy. Watch the shuffled-label run: high training accuracy, chance on test.

16

The same synthetic features as the hero, 3 classes, 300 test points. Error bars are ±1.96 standard errors. Numbers are illustrative.

Is a gain real? Error bars

A test set is a sample, so its accuracy has noise. If the true accuracy is p and the test set has n examples, the standard error is:

p
The accuracy, as a fraction.
n
The number of test examples.
SE
The standard error. About 95% of repeats land within ±1.96 SE.

Worked: p = 0.90, n = 300 → √(0.09 / 300) = 0.0173 → ±1.96 × 1.73 = ±3.4 points. A 1-point gain on 300 examples is noise.

Two more plots finish the analysis. A learning curve plots test accuracy against the number of labels, as in the hero. If it is still rising at your data size, more labels will pay; if it is flat, change the features or the head. And per-class errors, as a confusion matrix, show which classes your head mixes up. For attention heads, also plot the weights at on a few clips: they should sit on the event.

You shuffle the training labels and retrain. Test accuracy is 71% on a balanced 3-class task. What does this tell you?

Chapter 10 Where else When not

Where else it works, and when not to use it

Transfer the block to six fields, see what changes and what stays, and learn the cases where a simpler or different tool wins

The pattern moves across fields with three swaps: a backbone pretrained near the new data, a neck that fits the output's shape, and a loss that fits the labels. Everything else stays: freeze first, probe first, cache, check.

FieldBackboneNeck and headWhat changes
AudioCLAP or another audio encoder, frozenAttention pool → linearOne clip label; short events need learned pooling.
VisionDINOv2 or CLIP image encoderCLS token → linear; or patches → dense headSegmentation keeps per-patch vectors.
TextA BERT-style or embedding modelMasked mean pool → logistic regressionMask the padding; tagging uses per-token heads.
Medical time seriesAn encoder pretrained on unlabelled recordingsPool over time → multi-label sigmoid headSeveral findings per record: sigmoid, not softmax.
RoboticsA frozen image encoder on the wrist cameraPooled features + joint angles → MLP policy headThe head outputs actions; the loss is imitation or reward.
Search and few-shotAny of the aboveProjection head, unit lengthThe loss shapes distances, not class scores.

Use it, or use something else?

Answer four questions about your problem. The path through the tree lights up and ends at a recommendation.

A rule of thumb for a first build. Always confirm with the checks of Chapter 9.

When not to use it

Avoid it whenBecauseUse instead
Your data is a table of columnsThere is no big pretrained backbone that knows your columnsGradient-boosted trees, with good feature engineering
No backbone was trained on anything like your dataFrozen features miss what matters (Yosinski's 25% drop)Self-supervised pretraining on your own unlabelled data, then a head
You have millions of labelsThe frozen ceiling Φ(d/2) now costs youFull fine-tuning, or training from scratch
The answer hides in details the backbone threw awayA backbone keeps what its pretraining needed: fine phase, tiny text, exact countsEarlier-layer features, higher input resolution, or a task-specific model
Latency or memory is tight on a small deviceA 300-million-weight backbone is too slow at the edgeDistil the backbone and head into a small student model
Which project is the worst fit for a frozen backbone plus a small head?

Chapter 11

Connections

Get a first recipe for your own problem, keep the one-screen cheat sheet, know the limits, and follow the series onward

Everything in this lesson fits on one card. Pick your data and your label count below to get a first recipe. Then use the cheat sheet to adjust it.

Your first recipe

Choose a kind of data and a label count. The card gives the shape signature and the first three runs to try, in order.

Starting points, not final answers. Learning rates are common defaults for AdamW.

The cheat sheet

PartIn one line
The blockPretrained backbone (frozen or tuned) → optional neck → small task head → loss.
Shape signaturex → B×T×D → (pool) B×D → B×C. Dense tasks skip the pool: B×T×C.
Maths in one lineGood features raise separation d, so a few weights reach near Φ(d/2); excess error grows like p / (n − p).
KnobsWhich backbone and layer; freeze depth (none, top blocks, LoRA, all); pooling; head size; head LR 1e-3, backbone LR 10× smaller.
Failure modesFar domain; detail lost in pretraining; BatchNorm drift without eval(); padding in the mean; stale cache; leaks.
ChecksMajority, kNN and linear probe baselines; error bars; learning curve; shuffled labels; per-class errors; attention maps.
AlternativesTrees for tables; self-supervised pretraining on your data; full fine-tuning with many labels; distillation for small devices.

Honest limits

Where this sits, and where to go next

This is the hub of The Builder's Toolkit. The rest of the series takes each block apart:

Foundations used here, already on the site:

The takeaway. A pretrained backbone bends raw signal into a space where your classes are far apart. A small head then needs few labels, because it has few weights to fit noise with. Freeze first, probe first, cache the features, and prove the result with baselines, error bars and shuffled labels.
You have 400 labelled X-ray images and a backbone pretrained only on internet photos. What is the best first run?

Now press Present or Teach and explain, out loud and from memory, why a simple head on good features needs so few labels. Then go back to the hero, pick random features with an MLP head, and see how many labels it takes to catch up.

Backbone + Head, the hub of The Builder's Toolkit. Sources: J. Yosinski, J. Clune, Y. Bengio, H. Lipson, "How transferable are features in deep neural networks?" (NeurIPS 2014, arXiv); S. Kornblith, J. Shlens, Q. V. Le, "Do Better ImageNet Models Transfer Better?" (CVPR 2019, arXiv); A. Radford et al., "Learning Transferable Visual Models From Natural Language Supervision" (CLIP, ICML 2021, arXiv); M. Oquab et al., "DINOv2: Learning Robust Visual Features without Supervision" (TMLR 2024, arXiv); B. Elizalde, S. Deshmukh, M. Al Ismail, H. Wang, "CLAP: Learning Audio Concepts from Natural Language Supervision" (2022, arXiv); A. Dosovitskiy et al., "An Image is Worth 16x16 Words" (ViT, ICLR 2021, arXiv); E. Hu et al., "LoRA" (ICLR 2022, arXiv); C. Zhang et al., "Understanding deep learning requires rethinking generalization" (ICLR 2017, arXiv). The least-squares excess-error formula is the standard result for Gaussian features.
Back to Gleams