A classification head can only answer with the names it was trained on. A projection head answers with a place on a sphere, where distance means similarity. Train it on four classes, and a fifth class it has never seen can still land in a cluster of its own.
Learn why distance generalises to new classes, how cosine similarity and temperature work, how contrastive, triplet, InfoNCE and ArcFace losses shape the sphere, why SimCLR throws its projection head away, and how to prove a metric space works.
Pick a loss. Slide the temperature and the margin: the head retrains in your browser each time. Then show the new class and read how often five examples of it are enough to recognise the rest.
You need vectors, the dot product, softmax and the idea of a gradient. Read Backbone + Head first if the word "head" is new.
A toy you can trust: 8-number "backbone features" for 5 classes, 40 training and 40 test examples each. Only classes 1 to 4 are used in training. The head is linear (8 → 2) followed by L2 normalisation, so every point lies on the circle; it trains for 500 steps of Adam with batches of 32. Accuracy: 20 random episodes, 5 examples per class as the support set, every other test point classified by its nearest class prototype. The InfoNCE option uses same-class positives (the supervised form). All numbers are illustrative.
Chapter 0
Meet the problem a classification head cannot solve, and see how a head that outputs a place instead of a name solves it
Say you build a door camera for an office of forty people. You collect photos of each person and train a network with forty outputs, one per person. It works well. Then on Monday a new person joins.
The network has no output for her. It must pick one of the forty names it knows. Worse, it often picks one with high confidence. To add her, you must collect her photos and retrain. You must do this again for every new hire.
Now change one thing. Train the network to output a point in space instead of a name. Train it so that photos of the same person land close together and photos of different people land far apart. On Monday, you take one photo of the new person and store her point. When the camera sees her, it finds the nearest stored point. Nothing retrains.
A head that outputs class scores is a classification head. A head that outputs a point, where distance means similarity, is a metric head. When the point is made by a small network on top of a backbone, it is called a projection head. Training a space so that distance is meaningful is metric learning.
Four known people sit in a 2-D feature space. Drag the hollow cluster: it is a new person. Switch heads and read what each one says about her. Then enrol one photo.
Toy features in 2-D. The classification head is softmax over scores 4 × (mean · x) − 2 × |mean|², which is the nearest-mean rule with a temperature. The metric head accepts a match only when the distance is below 0.9. Numbers are illustrative.
A space where distance means similarity does four jobs that a fixed list of names cannot.
All four use the same rule: compare distances. The labels you trained on are not the labels you will be asked about. That is why the space must generalise: it must place classes it never saw.
Each part answers one of five questions. Why does it work? How do you build it? Where else does it work? When should you not use it? And did it work?
Chapter 1 Why it works
Follow the shapes from backbone to unit vector, define cosine similarity, and see why every point is pushed onto a sphere
A projection head is small. It takes the backbone's feature vector h, with D numbers, and maps it to a shorter vector with K numbers. Then it divides that vector by its own length. The result z has length exactly 1. We call this L2 normalisation: the L2 norm is the ordinary length, the square root of the sum of squares.
Worked, K = 2: u = (3, 4) → ‖u‖ = √(9 + 16) = 5 → z = (0.6, 0.8), and 0.6² + 0.8² = 1.
For a batch, the shapes are: B × D in, B × K out, and every row has length 1. SimCLR uses a ResNet-50 with D = 2048 and a 2-layer MLP head to K = 128 SimCLR · §2.3. FaceNet uses K = 128 FaceNet · §1. ArcFace uses K = 512 ArcFace · §4.1.
Once two embeddings have length 1, their dot product is the cosine of the angle between them. This is cosine similarity: 1 for the same direction, 0 for a right angle, −1 for opposite. And the distance between them follows from the cosine alone:
Worked: cos θ = 0.8 → squared distance 2 − 1.6 = 0.4. cos θ = 0 → 2. cos θ = −1 → 4, the largest possible.
So on the sphere, "nearest by distance" and "most similar by cosine" pick the same neighbour. You can use either. Try it below, then turn normalisation off.
Drag the tips of a and b. Then slide "stretch a": with normalisation off, the dot product grows without the direction changing. With it on, only the angle counts.
Exact arithmetic on the vectors you set.
Without normalisation, a loss that wants a large dot product has an easy cheat. It can make vectors longer instead of better aligned. Training then grows lengths and ignores directions. Normalisation removes the cheat. It also bounds every similarity between −1 and 1, so a single number, the temperature, can set how sharp the comparisons are (Chapter 4).
SimCLR tested this. Without L2 normalisation, its contrastive task was easier to solve, but the representation was worse under a linear probe: 57.2% ImageNet top-1 at the best temperature, against 64.4% with normalisation SimCLR · Table 5.
Chapter 2 Why it works
Write the first metric losses, derive the triplet gradient by hand, and see why most triplets teach nothing
How do you tell a network "these two should be close"? The oldest answer uses pairs. A positive pair is two items of the same class. A negative pair is two items of different classes. The contrastive loss pulls positive pairs together and pushes negative pairs apart, but only until they are a margin m apart:
Worked, m = 1: a positive pair at d = 0.3 costs 0.09. A negative pair at d = 0.4 costs (0.6)² = 0.36. A negative pair at d = 1.2 costs 0.
The pair loss has a flaw. It asks for absolute distances, but faces under different light may never get very close. The triplet loss asks only for an order. Take an anchor a, a positive p of the same class, and a negative n of another class. The positive must be closer than the negative, by a margin α:
The gradient, by hand. When the loss is active, L = ‖a − p‖² − ‖a − n‖² + α. The derivative of ‖a − p‖² with respect to a is 2(a − p). So:
Worked on the unit circle, α = 0.2: a at 0°, p at 40°, n at 45°. ‖a − p‖² = 2 − 2 cos 40° = 0.468. ‖a − n‖² = 2 − 2 cos 45° = 0.586. L = 0.468 − 0.586 + 0.2 = 0.082, active. Move n to 70°: ‖a − n‖² = 1.316, L = 0.
Drag the three points around the circle. Warm arrows show the step each point takes. Then switch to a whole batch: most triplets are already satisfied and give no gradient.
The batch view uses 4 classes × 8 points on the circle, each class spread by about 20°: 4 × 8 × 7 × 24 = 5,376 triplets. Numbers are illustrative.
Sort the triplets by where the negative sits. An easy negative is already farther than the positive plus α; its loss and gradient are zero. A semi-hard negative is farther than the positive, but inside the margin. A hard negative is closer than the positive. As training goes on, almost every random triplet becomes easy. The loss goes quiet while the space is still poor.
So you mine: you pick the triplets that still teach. FaceNet found that always taking the hardest negatives "can result in a collapsed model (i.e. f(x) = 0)" early in training, and used semi-hard negatives inside large batches of about 1,800 examples FaceNet · §3.2. Hard negatives are often mislabelled or truly ambiguous, so they pull the space in bad directions.
Chapter 3 Why it works
Turn "be closer than the negatives" into a classification over all negatives at once, derive the loss from softmax, and see how it weights hard negatives for free
A triplet compares one positive with one negative. Why not compare it with all of them at once? Picture a party game. You hold one half of a torn photo. In the room are many other halves. Your task: point to your own other half. That is a classification problem with as many classes as there are candidates.
Softmax solves classification problems. So score each candidate k by its cosine similarity to the anchor, sk = z · zk, divide by a temperature τ, and apply softmax. The loss is minus the log of the probability given to the true partner:
Worked: s+ = 0.8; negatives 0.5, 0.1, −0.3; τ = 0.5. Scores /τ = 1.6, 1.0, 0.2, −0.6. ex = 4.95, 2.72, 1.22, 0.55; sum 9.44. p+ = 0.525 → ℓ = −ln 0.525 = 0.645.
The gradient weights negatives by hardness. This is the same softmax cross-entropy as any classifier, so its gradient has the same form. With pk the softmax probability of candidate k:
Worked, same numbers: p = (0.525, 0.288, 0.129, 0.058). The hardest negative (0.5) gets 0.288 / 0.475 = 61% of the push on negatives. At τ = 0.1 it gets 98%.
So InfoNCE mines hard negatives by itself, softly, with no sampling rule. This is the main reason it beat the alternatives in SimCLR's comparison. With the same normalisation and tuned settings, ImageNet linear-probe accuracy was 50.9% for the margin triplet loss, 57.5% for the triplet loss with semi-hard mining, and 63.9% for NT-Xent SimCLR · Table 4.
The anchor (teal) sits at the top of the circle. Drag its positive, or a negative. The bars show each candidate's softmax probability: the positive's bar is what the loss wants to fill; each negative's bar is how hard it gets pushed away.
Eight negatives on a 2-D circle, so cosine = cos(angle). Exact softmax arithmetic. Positions are illustrative.
The loss asks the positive to beat every negative. With more negatives, the task is harder, and the chance that some negative is truly close goes up. That is the hard negative the gradient needs. SimCLR draws its negatives from the same batch, so it trains with very large batches: 4096 by default, and a batch of 8192 gives 16,382 negatives per positive pair SimCLR · §2.2, §2.3. Other methods keep a queue of past embeddings instead.
Chapter 4 Why it works
See temperature as a focus knob, work its effect with numbers, and read what SimCLR measured when it turned the knob
Cosine similarities live between −1 and 1. That is a narrow range for softmax. e1 is only 7.4 times e−1, so a plain softmax over cosines can never be confident. Dividing by τ stretches the range. With τ = 0.1, the scores run from −10 to 10.
Think of τ as a focus knob. A large τ spreads attention over all negatives, near and far. A small τ focuses it on the few nearest ones. Look at the ratio of the push on two negatives with similarities s1 and s2:
Worked, s1 = 0.5, s2 = 0.1: τ = 1 → e0.4 = 1.5×. τ = 0.5 → e0.8 = 2.2×. τ = 0.1 → e4 = 55×. τ = 0.05 → e8 = 2,981×.
Too large a τ, and near and far negatives get almost the same push. The loss stops caring about the hard cases. Too small a τ, and only the single nearest negative counts. If that negative is a near-duplicate or a mislabel, the loss chases noise. And the gradient grows like 1 / τ, so training gets unstable. The best τ sits in between.
255 negatives sit at evenly spread angles. The curve is the share of the push each one gets, against its cosine with the anchor. Slide τ: the push slides from everyone to only the nearest few. The bars on the right are SimCLR's real results.
The curve: exact softmax over 255 negatives on a 2-D circle, illustrative. The bars: SimCLR Table 5, ResNet-50, 100 epochs, batch 4096, linear evaluation on ImageNet; the "no L2 norm" bar is the best unnormalised run (τ = 10).
SimCLR trained the same model at four temperatures with L2 normalisation. Linear-probe accuracy on ImageNet was 59.7% at τ = 0.05, 64.4% at τ = 0.1, 60.7% at τ = 0.5 and 58.0% at τ = 1 SimCLR · Table 5. The best value sat in the middle. Khosla and colleagues used τ = 0.1 for supervised contrastive learning and note that the gradient scales inversely with τ and that "extremely low temperatures are harder to train due to numerical instability" SupCon · §4.5.
You can see the same thing in the hero. Pick InfoNCE and slide τ to 0.03. The head trains worse on the known classes, and the new class falls apart.
Chapter 5 Why it works
Use your labels to define positives, then turn a softmax classifier into a metric learner by adding a margin to the angle
SimCLR has no labels. Its only positive is another view of the same image. If you do have labels, every other item of the same class is a positive too. The supervised contrastive loss, SupCon, averages the InfoNCE term over all of them SupCon · Eq. 2:
Khosla and colleagues reached 78.7% ImageNet top-1 with a ResNet-50 trained with SupCon, against 78.2% for cross-entropy in their setup SupCon · Table 2.
There is another way to use labels. Keep a softmax classifier, but make it work on the sphere. Normalise the embedding x and every class weight wj. Then each logit is wj · x = cos θj, the cosine of the angle to class j. Multiply by a scale s, which plays the role of 1 / τ. This is the normalised softmax.
It already clusters classes by angle. But a test face only needs to be a little closer to its own centre than to another to be classified correctly. For matching new people, we want classes tight, with gaps between them. ArcFace adds a margin m to the angle of the true class during training only ArcFace · §3.1:
Derive the handicap. The angle-sum rule gives cos(θ + m) = cos θ cos m − sin θ sin m. So the code needs only cos θ from the dot product; sin θ = √(1 − cos² θ).
Worked, m = 0.5: θy = 30° → cos 30° = 0.866, but cos(30° + 28.6°) = 0.866 × 0.878 − 0.5 × 0.479 = 0.520. Another class at θ = 60° scores cos 60° = 0.5. With s = 10: without the margin, py = 0.975 and the loss is 0.025. With it, py = 0.551 and the loss is 0.597. The loss keeps pulling.
Why the margin tightens classes. Take two class centres an angle Δ apart and a training point between them, at angle φ from centre 1. Without a margin, it counts as class 1 while φ < Δ / 2: the boundary is halfway. With the margin, it must satisfy φ + m < Δ − φ, that is φ < (Δ − m) / 2. Each class's training zone shrinks by m / 2 on each side. A gap of width m opens between neighbours. Classes get pulled into tight cones.
Two class centres sit on the circle. Drag the teal training point, which belongs to the blue class. Raise the margin: the blue zone shrinks and a gap opens between the classes.
Two classes on a 2-D circle, 100° apart. Exact ArcFace logits and softmax for the point you place. Illustrative geometry.
On face verification, the margin pays off. With the same ResNet-50 trained on CASIA, verification accuracy on LFW, CFP-FP and AgeDB was 99.08%, 94.39% and 92.33% with plain softmax, and 99.53%, 95.56% and 95.15% with ArcFace at m = 0.5 ArcFace · Table 2. A triplet loss with margin 0.35 reached 98.98%, 91.90% and 89.98% in the same table.
Chapter 6 Why it works
Read SimCLR's surprising result, that the layer before the projection head transfers better than the head's output, and train a toy that shows why
Here is a puzzle. SimCLR trains a projection head on top of the backbone, and the loss is computed on the head's output z. Then, for every downstream task, it throws the head away. It uses h, the backbone's output, instead SimCLR · §2.1. Why train something you then discard?
Because the head helps h become better, and h is better than z. Chen and colleagues measured linear-probe accuracy for three heads SimCLR · §4.2, Fig. 8:
Their explanation: the contrastive loss trains z to be the same for two augmented views of one image. So z must be blind to whatever the augmentations change, such as colour or orientation. But those facts can matter for a downstream task. With a head in place, the loss can make z blind while h keeps the information. Without a head, h itself must go blind.
They tested this directly. They trained small MLPs to predict, from h or from z, which transformation had been applied to an image SimCLR · Table 3:
| What to predict | Random guess | From h | From z = g(h) |
|---|---|---|---|
| Colour vs greyscale | 80 | 99.3 | 97.4 |
| Rotation (4 ways) | 25 | 67.6 | 25.6 |
| Original vs corrupted | 50 | 99.5 | 59.6 |
| Original vs Sobel filtered | 50 | 96.6 | 56.3 |
From z, rotation is predicted at 25.6%, barely above the 25% guess. From h it is 67.6%. The head discards what the loss does not need. Try it yourself:
Each item has a shape class (3 kinds) and a colour (filled or hollow). The augmentation repaints the colour at random. Press Train: the head learns to ignore colour. Then read what a linear probe can still find, before and after the head.
h = (x, y, 1.2 × colour), with 3 classes in the (x, y) plane and colour ±1 at random. Head: linear 3 → 2, then L2 normalise, trained with NT-Xent (τ = 0.2, 24 items per batch, two views each). Probes: logistic regression on 60 training items, tested on 60 others. Chance for colour is about 50%. Numbers are illustrative.
If you pretrain with a contrastive loss and then transfer, take the features before the projection head. If you deploy the metric space itself, for search or verification, use the head's output z: it was shaped for exactly that comparison. And when you add a projection head on top of a frozen backbone, as in Backbone + Head, the head's output keeps only what your loss rewards. Make sure the loss rewards what you need.
Chapter 7 Build it
Recognise a new class from five examples by averaging them, see why that is a linear classifier in disguise, and search a gallery by sorting distances
You have a good space. A new bird species arrives with five labelled calls. How do you recognise more calls of it? The simplest rule that works: embed the five calls, average them, and call the average the class's prototype. A new call belongs to the class whose prototype is nearest. Snell, Swersky and Zemel turned this into Prototypical Networks ProtoNets · §2.2:
Worked, 2-D: shots (0.9, 0.1), (0.7, 0.3), (0.8, 0.2) → c = (0.8, 0.2). A query at (0.75, 0.25) has d = 0.05² + 0.05² = 0.005 to c.
A linear classifier in disguise. Expand the squared distance: −‖z − ck‖² = 2 ck · z − ‖ck‖² − ‖z‖². The last term is the same for every class, so it cancels in the softmax. What is left is a linear layer with weights 2ck and bias −‖ck‖² ProtoNets · §2.4. Few-shot learning with prototypes is building a linear head from data, with no gradient steps.
The paper trained the embedding in episodes: each step samples N classes and K shots per class, as at test time. On miniImageNet 5-way, it reached 49.42% with 1 shot and 68.20% with 5 shots; on Omniglot 5-way, 98.8% and 99.7% ProtoNets · Tables 1, 2. It also found squared Euclidean distance "greatly outperforms" cosine in its unnormalised space §1. On a unit sphere the two rank neighbours the same way (Chapter 1).
Each episode draws 5 new classes, K support shots each (big dots), and queries (small dots). Stars are prototypes. Change K and the spread: 200 episodes rerun at once and compare prototypes with 1-nearest-neighbour. Draw another episode to see a different one.
Synthetic 2-D embeddings: class centres placed at random, each point the centre plus Gaussian noise of the chosen spread. 15 queries per class. Numbers are illustrative.
Retrieval needs no classes at all. Embed every item in a gallery once and store the vectors. For a query, embed it and sort the gallery by cosine similarity. The top k results are the answer. With unit vectors, all cosines for a query are one matrix product: Q (1 × K) times GT (K × M) gives M scores. For millions of items, an approximate nearest-neighbour index finds the top k without scoring every item; see vector databases.
Chapter 8 Build it
Trace the shapes through NT-Xent and ArcFace, then write the projection head and both losses from scratch, and finish with the library version
By hand first. Take a batch of N = 2 images, two views each, so 2N = 4 embeddings: views 0 and 1 come from image A, views 2 and 3 from image B. Their 4 × 4 cosine matrix S = Z ZT has 1s on the diagonal. We set the diagonal to −∞, because an item is not its own candidate. Then row 0's target is column 1, row 1's is column 0, row 2's is column 3, and row 3's is column 2. NT-Xent is ordinary cross-entropy on the rows of S / τ with those targets.
Pick a loss and step through one training step. For NT-Xent, the similarity matrix shows the masked diagonal (grey) and each row's positive (green).
Shapes for a ResNet-50 backbone (D = 2048), a projection to K = 128 for NT-Xent, and K = 512 with C class weights for ArcFace, as in the two papers. The matrix cells use random unit vectors, for illustration.
From scratch. The projection head and NT-Xent. Every line is commented with its shape.
import torch import torch.nn as nn import torch.nn.functional as F class ProjectionHead(nn.Module): """(B, D) features -> (B, K) unit vectors.""" def __init__(self, d=2048, hidden=512, k=128): super().__init__() self.net = nn.Sequential(nn.Linear(d, hidden), nn.ReLU(), nn.Linear(hidden, k)) def forward(self, h): return F.normalize(self.net(h), dim=1) # every row has length 1 def nt_xent(z1, z2, tau=0.1): """z1, z2: (N, K) unit vectors, two views of the same N items.""" n = z1.shape[0] z = torch.cat([z1, z2]) # (2N, K) sim = z @ z.T / tau # (2N, 2N) cosines / tau sim.fill_diagonal_(float("-inf")) # an item is not its own candidate target = torch.cat([torch.arange(n, 2 * n), # row i's partner is i + N ... torch.arange(0, n)]) # ... and row i + N's is i return F.cross_entropy(sim, target) # mean over all 2N anchors torch.manual_seed(0) head = ProjectionHead() h1, h2 = torch.randn(8, 2048), torch.randn(8, 2048) # stand-in backbone features loss = nt_xent(head(h1), head(h2)) loss.backward() print(f"NT-Xent {loss.item():.3f}, chance level ln(2N-1) = {torch.log(torch.tensor(15.)):.3f}")
With random inputs, the loss sits near ln(2N − 1), the value for guessing among 15 candidates. That is a useful first sanity check. Now ArcFace, with the class weights as a trainable matrix and the angle-sum rule from Chapter 5:
import math class ArcFace(nn.Module): """(B, K) embeddings + (B,) labels -> scalar loss. Weights: (C, K).""" def __init__(self, k, n_classes, s=64.0, m=0.5): super().__init__() self.W = nn.Parameter(torch.randn(n_classes, k) * 0.01) self.s, self.m = s, m def forward(self, x, y): cos = F.normalize(x, dim=1) @ F.normalize(self.W, dim=1).T # (B, C) cos = cos.clamp(-1 + 1e-7, 1 - 1e-7) theta = torch.acos(cos.gather(1, y[:, None])) # (B, 1) angle to true class target = torch.where(theta + self.m < math.pi, torch.cos(theta + self.m), # the handicapped score cos.gather(1, y[:, None]) - self.m * math.sin(self.m)) logits = cos.scatter(1, y[:, None], target) * self.s # (B, C) return F.cross_entropy(logits, y) arc = ArcFace(k=128, n_classes=10) emb = head(torch.randn(16, 2048)) # (16, 128) labels = torch.randint(0, 10, (16,)) loss = arc(emb, labels) loss.backward() # trains head AND arc.W print(f"ArcFace {loss.item():.2f}")
Two details matter. First, the optimizer must include arc.parameters(), or the class weights never move. Second, the torch.where guard handles θ + m > π, where cos(θ + m) would start to rise again. At test time you keep only head.
The library version. The pytorch-metric-learning package has all of these losses and the miners. Note that its ArcFace margin is in degrees, 28.6 by default, which is 0.5 radians, and that the loss owns the class weights:
# pip install pytorch-metric-learning from pytorch_metric_learning import losses, miners emb = head(torch.randn(32, 2048)) # (32, 128) labels = torch.randint(0, 4, (32,)) supcon = losses.SupConLoss(temperature=0.1) # Chapter 5, labels as positives miner = miners.TripletMarginMiner(margin=0.2, type_of_triplets="semihard") triplet = losses.TripletMarginLoss(margin=0.2) arcface = losses.ArcFaceLoss(num_classes=4, embedding_size=128, margin=28.6, scale=64) print(supcon(emb, labels).item(), triplet(emb, labels, miner(emb, labels)).item(), arcface(emb, labels).item()) opt = torch.optim.AdamW(list(head.parameters()) + list(arcface.parameters()), lr=1e-3)
Chapter 9 Did it work?
Measure a metric space with recall@k, mAP and the verification error rate, always on classes held out from training, and learn to spot collapse, hubs and shortcuts
The first rule: test on classes the head never saw. A metric space exists to place new classes. If the test classes were also training classes, a classification head would do as well, and you learn nothing. Split by class, not by example.
Then pick the metric that matches the job.
Worked: ranked list ✓ ✗ ✓ ✗ ✗, R = 2. Recall@1 = 1. P@1 = 1, P@3 = 2/3 → AP = (1 + 0.667) / 2 = 0.833.
Green: cosine scores of pairs from the same class. Grey: pairs from different classes. Slide the threshold. Then pick a failure mode and watch the distributions, the ROC curve and the EER change.
Simulated embeddings: 40 held-out classes × 6 items in 16-D, each item its class direction plus noise. Collapse squeezes all points toward one shared direction; hubs give items a shared offset of very uneven size, so a few sit near everyone; the shortcut keeps only a weak class signal, as when the cue the head learned is missing on test data. All 28,680 pair scores are computed. Numbers are illustrative.
Chapter 10 Where else When not
Carry the same head to six fields, see what changes and what stays, and learn when a plain classifier or another tool is the better choice
The block moves between fields with three swaps: a backbone suited to the data, a definition of "positive pair", and a loss that fits the labels you have. The head itself, Linear or MLP then L2 normalise, stays the same.
| Field | Positive pair | Loss | What changes |
|---|---|---|---|
| Face and speaker verification | Two photos or recordings of one person | ArcFace or another angular margin | Millions of identities; the EER at a fixed false-accept rate is the metric. |
| Product and image retrieval | Two photos of one product | SupCon, or triplet with semi-hard mining | The catalogue changes daily; new products are enrolled, not trained. |
| Sound similarity and audio search | A clip and its text caption (CLAP), or two crops of one clip | Symmetric InfoNCE across the two modalities | Two encoders, two projection heads into one shared space. |
| Few-shot bioacoustics | Two calls of one species | Prototypes on a pretrained bird or audio embedding | Five calls of a rare species define its prototype. |
| Text semantic search | A question and the passage that answers it | InfoNCE with in-batch negatives | Hard negatives come from a keyword search that returned the wrong passage. |
| Robot place recognition | Two camera frames of one place, at different times | Triplet or contrastive, positives found by GPS | Lighting, season and viewpoint must not move the point; loop closure uses recall@k. |
The audio row is the CLAP recipe: an audio encoder and a text encoder, each followed by its own projection, trained so a clip and its caption land close. See the CLAP paper lesson and CLIP's contrastive training. For bird calls with few examples, see Perch 2.
Answer four questions about your problem. The path lights up and ends in a recommendation.
A rule of thumb for a first build. Always confirm with held-out classes, as in Chapter 9.
| Avoid it when | Because | Use instead |
|---|---|---|
| The class list is fixed and every class has many labels | You never need to place a new class, and softmax uses the labels directly | A plain linear or MLP classification head with cross-entropy |
| Batches are tiny and there are no labels | In-batch contrastive losses starve for negatives | A memory queue of negatives, a non-contrastive method, or a pretrained embedding with prototypes |
| You need calibrated probabilities | A cosine of 0.7 is not a 70% chance; the scale depends on τ and the data | A classifier with calibration, or fit a calibration map on held-out pairs |
| "Similar" changes with the user or the query | One space encodes one notion of similarity | A cross-encoder or a reranker that sees both items together |
| The positives are defined by an augmentation that destroys your signal | The head learns to ignore whatever the augmentation changes (Chapter 6) | Change the augmentations, or use labelled positives |
Chapter 11
Get a first recipe for your own field, keep the one-screen cheat sheet, know the limits, and follow the series onward
Everything here fits on one card. Pick your field below for a first recipe, then use the cheat sheet to adjust it.
Choose a field. The card gives the shape signature, the loss and its settings, and the check that proves it worked.
Starting points, not final answers. Settings follow the papers cited in this lesson where they exist; tune them on held-out classes.
| Part | In one line |
|---|---|
| The block | Backbone → projection head (Linear or MLP) → L2 normalise → metric loss. At test time: compare by cosine. |
| Shape signature | B × D → B × K, every row length 1; a batch gives a B × B (or 2N × 2N) similarity matrix. |
| Maths in one line | ‖z1 − z2‖² = 2 − 2 cos θ; InfoNCE pushes each negative by pk / τ; ArcFace trains on s · cos(θy + m). |
| Knobs | Output size K (128 to 512); temperature τ (about 0.05 to 0.2); margin (triplet 0.2, ArcFace 0.5 rad, s = 64); batch size; what counts as a positive. |
| Failure modes | Collapse; hubs; shortcut features; quiet loss from easy triplets; τ too small or too large; the head discarding what you need. |
| Checks | Held-out classes only; recall@k, mAP, EER; frozen-feature and random-head baselines; spread of embeddings; hub counts; cross-session tests. |
| Alternatives | Softmax head for fixed classes; calibrated classifier for probabilities; cross-encoder for query-specific similarity. |
This lesson is part of The Builder's Toolkit, a series on reusable blocks:
Foundations on the site:
Now press Present or Teach and explain, out loud and from memory, why a space can recognise a class it never saw. Then go back to the hero, pick InfoNCE, slide τ down to 0.03, and watch the new class fall apart.