Almost every applied model has two parts. A big backbone, pretrained on lots of data, turns raw signal into vectors. A small head, trained on your few labels, turns those vectors into your answer. With good features, a simple head wins when labels are few.
Learn what a representation is, why pretrained features transfer, how to wire the parts with the right shapes, what to freeze, how to build it in PyTorch, and how to prove it worked.
Pick a backbone and a head. Slide the number of labels. The curves are real: each point is a head trained in your browser, right now.
You need vectors, a dot product and the idea of training by gradient descent. We build everything else from zero.
Synthetic 12-dimensional features for 3 classes; only two dimensions are drawn. "Random" hides each class in two opposite blobs (not linearly separable); "weak" overlaps; "strong" separates. Each curve point averages 3 training draws, tested on 300 held-out points. Linear: softmax regression. MLP: 32 hidden ReLU units. kNN: 5 neighbours. The dashed red curve is random features with an MLP head, for contrast. All numbers are illustrative.
Chapter 0
Start from one real build: a frozen audio model plus a tiny head that finds one sound. Then see the same pattern in vision, text and sensor data
Say you want a detector for one specific sound. Maybe it is your doorbell, a smoke alarm, or one bird's call. You have about two hundred labelled clips. That is far too few to train an audio model from scratch.
So you borrow one. CLAP is an audio model that was pretrained on 128k pairs of audio and text CLAP · §3.2. You run your clips through its audio encoder and keep its weights fixed. We say the encoder is frozen: no weight changes during your training. Out comes a short list of vectors per clip, one for each slice of time.
Then you train only a small piece on top. Here it is an attention-pooling head: it learns which time slices matter, averages them with those weights, and makes one yes-or-no score. It has a few thousand weights. It trains in seconds. And it works.
The big frozen part is the backbone. The small trained part is the head. The same split hides inside almost every applied model you will meet. Pick a field below and look.
Pick a field. Each box is one step; the line under it is the shape of the data. Turn on "show what trains" to see how little of the model learns from your labels.
CLAP's audio encoder (CNN14, embedding size 2048), BERT-base (110 million weights) and the 44.1 kHz rate are from the CLAP paper (§3.2). The number of time frames T, the class counts and the sensor format are illustrative. B is the batch size.
Across the four fields, the backbone changes: an audio encoder, a vision transformer, a text transformer, a signal encoder. The head changes a little. But the plan stays the same. Raw signal goes in. The backbone makes vectors. A short step squeezes them to one vector, or keeps one per position. A small head turns that into scores. A loss compares the scores with your labels, and only the head learns from it.
The CLAP authors evaluate their own model in exactly this way. They use it "as a feature extractor followed by training a classifier of 1 or 3 fully-connected layers" CLAP · §3.2. The pattern is not a hack. It is how the field measures what a pretrained model knows.
Each part of the lesson answers one of five questions. Why does it work? How do you build it? Where else does it work? When should you not use it? And did it work?
This lesson is the hub of a twelve-part series. Each later lesson takes one block from here and goes deep.
Chapter 1 Why it works
A learned map from raw signal to vectors, built so that the differences you care about become simple
Look at the raw numbers of a sound clip. A one-second clip at 44.1 kHz is 44,100 numbers. A doorbell and a dog bark differ in all of them, in tangled ways. No single number says "doorbell". If you draw two clips as points in that huge space, the closeness of the points says almost nothing about how alike the sounds are.
A representation is a different set of coordinates for the same input. A network f maps the raw input x to a vector z. A good map puts clips of the same kind near each other and different kinds far apart. Then a simple rule on z, such as "which side of a line?", can tell them apart.
Worked, D = 2, C = 2: z = (1.0, −0.5), W = [[2, 0], [−1, 1]], b = (0, 0.5) → s1 = 2.0, s2 = −1 − 0.5 + 0.5 = −1.0 → class 1 wins
Each row of W is a direction in feature space. A score is a dot product: how far z points along that direction. The head can only draw straight boundaries. So all the hard work of bending the space must already be done by f.
Three classes sit on three rings. A linear head is trained live on whatever coordinates you give it. Slide from the raw coordinates to the learned features and watch its accuracy.
The "learned features" here are hand-made: distance from the centre and angle. A real backbone learns its map from data. 180 points; the head is softmax regression trained for 150 steps. Numbers are illustrative.
Why can a map like this exist at all? Real signals are not spread evenly through their huge raw space. Natural images, speech and engine noise fill only thin, curved sheets inside it. Mathematicians call such a sheet a manifold. Moving along it changes the content a little: the pitch of a bark, the angle of a cat.
A good backbone flattens these sheets and pulls apart sheets of different kinds. After that, the classes are near-straight clumps. That is the moment a linear head starts to work. In the device, the rings are curved sheets; the learned map unrolls them into flat bands.
Chapter 2 Why it works
Measure how separable the classes are, see how that sets the best a linear head can do, and read what the evidence says about pretrained features
Features learned for one task help another task because different tasks share structure. Every photo is made of edges, textures and parts. Every sound is made of onsets, harmonics and noise bands. A backbone that learned these pieces from millions of examples has already done most of the work for your task.
We can make "good features" precise. Picture two classes in feature space as two round clouds. Let the distance between their centres, measured in units of the clouds' spread, be d. This is the separation. If d is large, the clouds barely touch. The best possible straight boundary then makes few mistakes:
Worked: d = 2 → Φ(1) = 84.1%. d = 4 → Φ(2) = 97.7%. Doubling the separation cuts the error from 15.9% to 2.3%.
Why d / 2? The boundary sits halfway, d / 2 spreads from each centre. A point is wrong only if its noise pushes it more than d / 2 spreads toward the other side. That has chance 1 − Φ(d / 2). A better backbone raises d. Nothing a head does afterwards can beat Φ(d / 2).
Raise the separation and watch the ceiling rise. Then add useless dimensions: the ceiling stays, but a head trained on few labels falls below it. More labels bring it back.
Two Gaussian classes with unit spread in every dimension. The probe is logistic regression trained on fresh samples 16 times; its accuracy is computed exactly from its boundary, not estimated. Numbers are illustrative.
Three studies shape how practitioners think about this.
General first, specific last. Yosinski and colleagues split ImageNet into two halves and moved trained layers from one half's network to the other's Yosinski 2014 · §4. Early layers learn edge and colour detectors that transfer almost freely. Later layers are more specific. With the copied layers frozen up to the top, accuracy fell by 8% when the two halves were similar. It fell by 25% when one half held only man-made objects and the other only natural ones §4.3. Even features from a distant task beat random weights.
Better backbones give better features. Kornblith, Shlens and Le tested 16 networks on 12 datasets Kornblith 2019 · Abstract. A network's ImageNet accuracy predicted its transfer accuracy with a linear head on frozen features very closely (correlation r = 0.99), and with full fine-tuning too (r = 0.96).
Frozen features can be enough. DINOv2's largest model reaches 86.5% on ImageNet with only a linear head on frozen features, and 83.5% with a kNN rule DINOv2 · Table 4. Fine-tuning the whole backbone adds 2.0 points, to 88.5% Table 5. A linear probe on CLIP's features beat the strongest model in the authors' test suite, Noisy Student EfficientNet-L2, on 21 of 27 datasets CLIP · §3.2.
Chapter 3 Why it works
Split the error into bias and variance, count the weights each choice trains, and see why "simple head on good features" beats "big model on little data"
A trained model makes two kinds of error. Bias is the error that stays even with endless data: the model is too simple for the truth. Variance is the error from fitting the noise in your particular few examples: the model is too free for the data.
Good features attack bias: the truth becomes a simple function of z. A small head attacks variance: it has few weights to fit noise with. Together they need few labels. A big model on raw data has low bias but huge variance. It needs many labels.
For the simplest head there is an exact formula. Fit a linear model with p weights by least squares on n examples, with noise of variance σ2. If the features are Gaussian, the expected extra test error is:
Worked, σ2 = 1, n = 100: p = 10 → 10/89 = 0.11. p = 50 → 50/49 = 1.02. p = 90 → 90/9 = 10.
The error grows slowly while p is small next to n, then explodes as p nears n. If the truth needs 5 features and your head has fewer, bias appears instead. Try it below.
The truth uses 5 features. Your head uses the first p of 120. Below 5 it is too simple (bias). Far above 5 it fits noise (variance). Add labels and the valley widens.
Features are independent standard normals; the truth is the sum of the first 5 features plus noise of variance 1. The dot averages 8 least-squares fits; error is computed exactly from the fitted weights. Numbers are illustrative.
Now apply this to real choices. Say D = 768 features and C = 2 classes.
| What you train | Weights trained | How |
|---|---|---|
| Linear head | 1,538 | 768 × 2 + 2 |
| MLP head, 512 hidden | 394,754 | 768 × 512 + 512 + 512 × 2 + 2 |
| The whole ViT-Base backbone | about 86 million | ViT paper, Table 1 |
With two hundred clips, the linear head has about 8 examples per weight. The full backbone has about 430,000 weights per example. The formula is exact only for least squares, and deep networks are kinder than it says. But the direction is right, and it is the reason behind the whole pattern.
Chapter 4 Build it
Follow one batch through backbone, neck, head and loss, and read the tensor shape at every step for image, audio, text and time series
Every backbone-and-head model has four parts. Learn their names once and you can read any model card.
Most bugs in this pattern are shape bugs. So we track shapes. We write them like B × T × D: batch size, then sequence length, then feature size. A vision transformer, for example, cuts an image into square patches of P × P pixels. Each patch becomes one token:
Worked: 224 / 16 = 14 → 14 × 14 + 1 = 197 tokens of D = 768. With P = 14: 16 × 16 + 1 = 257.
Pick a kind of data. Step through the pipeline. Each block is drawn to the size of one example's tensor, so you can see where the data shrinks.
Block area is drawn on a log scale of the number of values per example. Audio frame counts and the sensor format are illustrative; ViT and BERT sizes are standard.
The neck exists because heads want one vector per example, and backbones give many. An image gives 197 tokens. A 10-second clip gives dozens of frames. A sentence gives one vector per token. To classify the whole thing, you must squeeze the sequence. Mean pooling averages all vectors. Taking the CLS token uses the backbone's own summary. Attention pooling learns the weights (Chapter 5).
For dense tasks you skip the squeeze. Segmentation keeps one vector per patch. Event detection keeps one per frame. The head then runs on every position: B × T × D becomes B × T × C.
Chapter 5 Build it
Meet the seven common kinds of head, learn when each one fits, and train an attention-pooling head live to see it find a short event
A head is any small trainable module after the backbone. Here are the seven you will meet most. They differ in what shape they take in and what they put out.
| Head | Shape in → out | Use it when |
|---|---|---|
| Linear | B×D → B×C | Features are good and labels are few. Always your first baseline. |
| MLP | B×D → B×H → B×C | Classes curve in feature space and you have thousands of labels. |
| Pooling (mean, max, attention) | B×T×D → B×D | One label for a whole clip, image or document. Attention when the evidence is short. |
| Sequence (per-frame, per-token) | B×T×D → B×T×C | You need a label at each time step: event onsets, named entities. |
| Projection / metric | B×D → B×K, unit length | You compare examples by distance: search, few-shot, face matching. |
| Dense / decoder | B×N×D → B×C×H×W | Every pixel needs a label: segmentation, depth. |
| Multi-task | one B×D → several outputs | Several labels share one input; one backbone pass serves all heads. |
Back to the sound detector. A 10-second clip holds a doorbell for maybe half a second. Mean pooling averages the bell with nine and a half seconds of other sound, so the bell's signal shrinks. An attention-pooling head learns a query vector q. It scores each frame against q, turns the scores into weights that sum to 1, and averages with those weights:
Worked, 3 frames with scores (0, 0, 2): e0 = 1, e2 = 7.39, sum 9.39 → a = (0.11, 0.11, 0.79). Mean pooling would give each frame 0.33.
Half the clips hold a 2-to-3-frame event, hidden among 40 frames of background. Pick a pooling rule and press Train. The bars show the attention weights on one test clip; the shaded band is where the event really is.
Synthetic frame features: 40 frames of 6 numbers each, background noise plus a slow drift. The event adds a fixed direction spread over four of the six numbers, so no single number gives it away. 160 training clips, 200 test clips, 300 steps of Adam. Numbers are illustrative.
The attention head wins because its weights can follow the event, wherever it is. Max pooling looks at each feature number on its own, so an event spread across several numbers hides in the noise. The weights at also tell you when the event happened, for free. The Attention-Pooling Heads lesson takes this block apart.
Chapter 6 Build it
Decide how much of the backbone to train from two facts: how many labels you have, and how far your data is from what the backbone saw
Freezing is one end of a scale. At the other end, you train every weight of the backbone. In between are two useful middle steps.
Two facts pick the step. The first is how many labels you have. The second is the domain gap: how different your data is from the backbone's pretraining data. Bird calls are near to CLAP's training sounds. Ultrasound scans are far from ImageNet photos.
Drag the dot, or use the sliders, to place your project. The chart names the usual strategy and counts the weights it trains on a ViT-Base backbone.
ViT-Base: 12 blocks, D = 768, MLP width 3072; weight counts are computed from those sizes, for C = 10 classes. The region borders are a rule of thumb, not a law.
Why do adapters work? A weight update during fine-tuning tends to be simple. LoRA bets that the change to each d × d matrix W has low rank r. So it learns the change as the product of two thin matrices:
Worked, d = 768, r = 8: B A has 2 × 768 × 8 = 12,288 weights; W has 768² = 589,824. LoRA trains 2.1% of that matrix.
Data size matters most. With 47 to 800 total examples, a linear head on frozen features matched or beat full fine-tuning across Kornblith's datasets Kornblith 2019 · §4.7. With more data, fine-tuning pulls ahead. Domain gap matters next. On Stanford Cars, fine-tuning from ImageNet reached 93.3% and training from scratch reached 92.7%: the fine-grained car models are poorly covered by ImageNet's few car classes §4.5. Yet fine-tuning still reached its accuracy in far fewer steps: 17 times fewer on average §4.6.
When you do tune, give the backbone a smaller learning rate than the head. The head starts from random and must move far. The backbone starts from good weights and should move a little. The CLAP paper used a learning rate of 10−3 for heads on frozen features, and 10−4 when it unfroze the encoder CLAP · §3.2.
Chapter 7 Build it
Do one training step by hand, write the frozen backbone and head from scratch, then meet the one-line library versions
By hand first. Take the Chapter 1 head: z = (1.0, −0.5), scores s = (2.0, −1.0), and the true class is 1. The softmax turns scores into probabilities. The cross-entropy loss is minus the log of the true class's probability:
Worked: e2 = 7.389, e−1 = 0.368 → p = (0.953, 0.047), L = −ln 0.953 = 0.049, ∂L/∂s = (−0.047, 0.047). Row 1 of W moves along +z; row 2 moves along −z.
The gradient stops at the head. Nothing flows into f if the backbone is frozen. Getting that right in PyTorch takes four switches, and each one does something different. Flip them below.
The safe setting is all four on. Turn them off one at a time and read what breaks.
Backbone: ViT-Base, about 86 million weights (ViT paper, Table 1), modelled as 85.8 million in 12 blocks. Head: 768 × 10 + 10 weights. Memory counts fp32 gradients (4 bytes) and Adam's two moment buffers (8 bytes) per trained weight; activations are not counted.
From scratch. Here is a frozen wrapper and an attention-pooling head. The wrapper overrides train(), because calling model.train() on the whole model would otherwise switch the backbone back to training mode.
import torch
import torch.nn as nn
class FrozenBackbone(nn.Module):
"""Wraps any pretrained encoder so it never trains."""
def __init__(self, encoder):
super().__init__()
self.encoder = encoder
self.encoder.requires_grad_(False) # no weight gets a gradient
self.encoder.eval() # dropout off, BatchNorm uses stored stats
def train(self, mode=True): # model.train() must not wake it up
super().train(mode)
self.encoder.eval()
return self
@torch.no_grad() # keep no graph: less memory, faster
def forward(self, x):
return self.encoder(x)
class AttentionPoolHead(nn.Module):
"""(B, T, D) frames -> (B, C) logits, plus (B, T) weights."""
def __init__(self, d, c):
super().__init__()
self.norm = nn.LayerNorm(d)
self.score = nn.Linear(d, 1) # s_t = q . h_t + bias
self.out = nn.Linear(d, c)
def forward(self, h): # h: (B, T, D)
h = self.norm(h)
a = self.score(h).softmax(dim=1) # (B, T, 1), sums to 1 over T
z = (a * h).sum(dim=1) # (B, D) weighted average
return self.out(z), a.squeeze(-1) # (B, C), (B, T)
And the training loop, in the same file. The stand-in encoder lets the code run with no downloads; swap in a real one later. The labels are random on purpose: this is the smoke test from Chapter 9, so expect chance accuracy.
from torch.utils.data import DataLoader, TensorDataset torch.manual_seed(0) B, T, D, C = 32, 50, 64, 2 # Stand-in pretrained encoder: (N, T, 16) raw frames -> (N, T, D) encoder = nn.Sequential(nn.Linear(16, D), nn.GELU(), nn.Linear(D, D)) backbone = FrozenBackbone(encoder) raw = torch.randn(512, T, 16) # 512 clips labels = torch.randint(0, C, (512,)) # random: a smoke test feats = torch.cat([backbone(x) for x in raw.split(B)]) # (512, T, D), run ONCE head = AttentionPoolHead(D, C) opt = torch.optim.AdamW(head.parameters(), lr=1e-3, weight_decay=1e-2) loader = DataLoader(TensorDataset(feats, labels), batch_size=B, shuffle=True) for epoch in range(30): for h, y in loader: # h: (B, T, D), y: (B,) logits, _ = head(h) loss = nn.functional.cross_entropy(logits, y) opt.zero_grad() loss.backward() opt.step() print(f"last batch loss {loss.item():.3f}")
The library one-liners. A linear probe on cached, pooled features is one line of scikit-learn. A timm model with a new head, and everything else frozen, is three.
from sklearn.linear_model import LogisticRegression X = feats.mean(dim=1).numpy() # (512, D) mean-pooled probe = LogisticRegression(max_iter=1000).fit(X, labels.numpy()) import timm model = timm.create_model("vit_base_patch16_224", pretrained=True, num_classes=5) for name, p in model.named_parameters(): p.requires_grad = name.startswith("head.") # only the new head trains
Chapter 8 Build it
Run the frozen backbone once, store its output, and train heads on the stored features many times faster
A frozen backbone gives the same output for the same input every time. So why compute it every epoch? Run it once over the whole dataset. Save the features to disk. Then train the head on the saved features. This is feature caching, and it is the biggest practical speed-up in this pattern.
Let tb be the backbone's time per example, and th the head's. Training for E epochs over N examples, and trying S head settings, costs:
Worked: S = 1, E = 30, tb = 20 ms, th = 0.05 ms → 30 × 20.05 / (20 + 1.5) = 601.5 / 21.5 = 28× faster.
Move the sliders. The bars compare wall-clock time; the line below gives the cache's size on disk. Try a sweep of 10 head settings.
Head time 0.05 ms per example. Cache size at 2 bytes per number (fp16), with D = 768 and T = 197 tokens. Timings are illustrative; measure your own.
The choice of what to cache matters. Pooled vectors are tiny: 10,000 × 768 numbers at 2 bytes is about 15 MB. But then you cannot train an attention-pooling head, because the frames are gone. Caching every token keeps that option and costs 197 times more. A common middle path: cache all frames for audio clips, which have dozens of frames, and pooled vectors for images.
Chapter 9 Did it work?
Compare every head with a linear probe and a kNN baseline, put error bars on accuracy, draw a learning curve, and run the shuffled-label test
An accuracy number alone means little. 90% is great for ten balanced classes and poor if one class is 90% of the data. You need baselines: simple methods your head must beat to earn its complexity. Run these four checks on every project.
Pick the feature quality and the number of labels, then press Run. Hollow bars are training accuracy; solid bars are test accuracy. Watch the shuffled-label run: high training accuracy, chance on test.
The same synthetic features as the hero, 3 classes, 300 test points. Error bars are ±1.96 standard errors. Numbers are illustrative.
A test set is a sample, so its accuracy has noise. If the true accuracy is p and the test set has n examples, the standard error is:
Worked: p = 0.90, n = 300 → √(0.09 / 300) = 0.0173 → ±1.96 × 1.73 = ±3.4 points. A 1-point gain on 300 examples is noise.
Two more plots finish the analysis. A learning curve plots test accuracy against the number of labels, as in the hero. If it is still rising at your data size, more labels will pay; if it is flat, change the features or the head. And per-class errors, as a confusion matrix, show which classes your head mixes up. For attention heads, also plot the weights at on a few clips: they should sit on the event.
Chapter 10 Where else When not
Transfer the block to six fields, see what changes and what stays, and learn the cases where a simpler or different tool wins
The pattern moves across fields with three swaps: a backbone pretrained near the new data, a neck that fits the output's shape, and a loss that fits the labels. Everything else stays: freeze first, probe first, cache, check.
| Field | Backbone | Neck and head | What changes |
|---|---|---|---|
| Audio | CLAP or another audio encoder, frozen | Attention pool → linear | One clip label; short events need learned pooling. |
| Vision | DINOv2 or CLIP image encoder | CLS token → linear; or patches → dense head | Segmentation keeps per-patch vectors. |
| Text | A BERT-style or embedding model | Masked mean pool → logistic regression | Mask the padding; tagging uses per-token heads. |
| Medical time series | An encoder pretrained on unlabelled recordings | Pool over time → multi-label sigmoid head | Several findings per record: sigmoid, not softmax. |
| Robotics | A frozen image encoder on the wrist camera | Pooled features + joint angles → MLP policy head | The head outputs actions; the loss is imitation or reward. |
| Search and few-shot | Any of the above | Projection head, unit length | The loss shapes distances, not class scores. |
Answer four questions about your problem. The path through the tree lights up and ends at a recommendation.
A rule of thumb for a first build. Always confirm with the checks of Chapter 9.
| Avoid it when | Because | Use instead |
|---|---|---|
| Your data is a table of columns | There is no big pretrained backbone that knows your columns | Gradient-boosted trees, with good feature engineering |
| No backbone was trained on anything like your data | Frozen features miss what matters (Yosinski's 25% drop) | Self-supervised pretraining on your own unlabelled data, then a head |
| You have millions of labels | The frozen ceiling Φ(d/2) now costs you | Full fine-tuning, or training from scratch |
| The answer hides in details the backbone threw away | A backbone keeps what its pretraining needed: fine phase, tiny text, exact counts | Earlier-layer features, higher input resolution, or a task-specific model |
| Latency or memory is tight on a small device | A 300-million-weight backbone is too slow at the edge | Distil the backbone and head into a small student model |
Chapter 11
Get a first recipe for your own problem, keep the one-screen cheat sheet, know the limits, and follow the series onward
Everything in this lesson fits on one card. Pick your data and your label count below to get a first recipe. Then use the cheat sheet to adjust it.
Choose a kind of data and a label count. The card gives the shape signature and the first three runs to try, in order.
Starting points, not final answers. Learning rates are common defaults for AdamW.
| Part | In one line |
|---|---|
| The block | Pretrained backbone (frozen or tuned) → optional neck → small task head → loss. |
| Shape signature | x → B×T×D → (pool) B×D → B×C. Dense tasks skip the pool: B×T×C. |
| Maths in one line | Good features raise separation d, so a few weights reach near Φ(d/2); excess error grows like p / (n − p). |
| Knobs | Which backbone and layer; freeze depth (none, top blocks, LoRA, all); pooling; head size; head LR 1e-3, backbone LR 10× smaller. |
| Failure modes | Far domain; detail lost in pretraining; BatchNorm drift without eval(); padding in the mean; stale cache; leaks. |
| Checks | Majority, kNN and linear probe baselines; error bars; learning curve; shuffled labels; per-class errors; attention maps. |
| Alternatives | Trees for tables; self-supervised pretraining on your data; full fine-tuning with many labels; distillation for small devices. |
This is the hub of The Builder's Toolkit. The rest of the series takes each block apart:
Foundations used here, already on the site:
Now press Present or Teach and explain, out loud and from memory, why a simple head on good features needs so few labels. Then go back to the hero, pick random features with an MLP head, and see how many labels it takes to catch up.