Heads That Shape a Space

A classification head can only answer with the names it was trained on. A projection head answers with a place on a sphere, where distance means similarity. Train it on four classes, and a fifth class it has never seen can still land in a cluster of its own.

Learn why distance generalises to new classes, how cosine similarity and temperature work, how contrastive, triplet, InfoNCE and ArcFace losses shape the sphere, why SimCLR throws its projection head away, and how to prove a metric space works.

Pick a loss. Slide the temperature and the margin: the head retrains in your browser each time. Then show the new class and read how often five examples of it are enough to recognise the rest.

You need vectors, the dot product, softmax and the idea of a gradient. Read Backbone + Head first if the word "head" is new.

Four known classes, one stranger

Ready
0.10
0.50

A toy you can trust: 8-number "backbone features" for 5 classes, 40 training and 40 test examples each. Only classes 1 to 4 are used in training. The head is linear (8 → 2) followed by L2 normalisation, so every point lies on the circle; it trains for 500 steps of Adam with batches of 32. Accuracy: 20 random episodes, 5 examples per class as the support set, every other test point classified by its nearest class prototype. The InfoNCE option uses same-class positives (the supervised form). All numbers are illustrative.

Chapter 0

The face it has never seen

Meet the problem a classification head cannot solve, and see how a head that outputs a place instead of a name solves it

Say you build a door camera for an office of forty people. You collect photos of each person and train a network with forty outputs, one per person. It works well. Then on Monday a new person joins.

The network has no output for her. It must pick one of the forty names it knows. Worse, it often picks one with high confidence. To add her, you must collect her photos and retrain. You must do this again for every new hire.

Now change one thing. Train the network to output a point in space instead of a name. Train it so that photos of the same person land close together and photos of different people land far apart. On Monday, you take one photo of the new person and store her point. When the camera sees her, it finds the nearest stored point. Nothing retrains.

A head that outputs class scores is a classification head. A head that outputs a point, where distance means similarity, is a metric head. When the point is made by a small network on top of a backbone, it is called a projection head. Training a space so that distance is meaningful is metric learning.

Name it, or place it?

Four known people sit in a 2-D feature space. Drag the hollow cluster: it is a new person. Switch heads and read what each one says about her. Then enrol one photo.

Toy features in 2-D. The classification head is softmax over scores 4 × (mean · x) − 2 × |mean|², which is the nearest-mean rule with a temperature. The metric head accepts a match only when the distance is below 0.9. Numbers are illustrative.

One space, four jobs

A space where distance means similarity does four jobs that a fixed list of names cannot.

All four use the same rule: compare distances. The labels you trained on are not the labels you will be asked about. That is why the space must generalise: it must place classes it never saw.

The road map of this lesson

Each part answers one of five questions. Why does it work? How do you build it? Where else does it work? When should you not use it? And did it work?

  1. The projection head and the sphere: shapes, cosine similarity, and why we normalise. Why
  2. Pairs and triplets: the first metric losses, their gradient, and hard negatives. Why
  3. InfoNCE: find your partner among many, derived from softmax. Why
  4. Temperature: one number that sets how hard the loss looks. Why
  5. Labels and angles: supervised contrastive and ArcFace's margin. Why
  6. Before or after the head?: why SimCLR throws its head away. Why
  7. Prototypes and retrieval: few-shot with a mean, search with a sort. How
  8. Build it in PyTorch: by hand, from scratch, and with a library. How
  9. Did it work?: recall@k, mAP, EER, and the three failure modes. Did it work
  10. Where else, and when not: six fields and a decision table. What else / what not
  11. Connections: the cheat sheet, honest limits, and where to go next.
A new person joins the office. Why can the metric head recognise her without retraining?

Chapter 1 Why it works

The projection head and the sphere

Follow the shapes from backbone to unit vector, define cosine similarity, and see why every point is pushed onto a sphere

A projection head is small. It takes the backbone's feature vector h, with D numbers, and maps it to a shorter vector with K numbers. Then it divides that vector by its own length. The result z has length exactly 1. We call this L2 normalisation: the L2 norm is the ordinary length, the square root of the sum of squares.

h
The backbone's output for one input: D numbers, for example D = 2048 for a ResNet-50.
g
The projection head: one linear layer, or a small MLP such as Linear(D, 512), ReLU, Linear(512, K).
u
The raw projection: K numbers, for example K = 128.
z
The embedding: u scaled to length 1. All embeddings lie on the unit sphere in K dimensions.

Worked, K = 2: u = (3, 4) → ‖u‖ = √(9 + 16) = 5 → z = (0.6, 0.8), and 0.6² + 0.8² = 1.

For a batch, the shapes are: B × D in, B × K out, and every row has length 1. SimCLR uses a ResNet-50 with D = 2048 and a 2-layer MLP head to K = 128 SimCLR · §2.3. FaceNet uses K = 128 FaceNet · §1. ArcFace uses K = 512 ArcFace · §4.1.

Once two embeddings have length 1, their dot product is the cosine of the angle between them. This is cosine similarity: 1 for the same direction, 0 for a right angle, −1 for opposite. And the distance between them follows from the cosine alone:

cos θ
z1 · z2, the cosine similarity. It works because ‖z1‖ = ‖z2‖ = 1.
‖z1 − z2‖²
The squared straight-line distance. Expand it: 1 − 2 z1·z2 + 1.

Worked: cos θ = 0.8 → squared distance 2 − 1.6 = 0.4. cos θ = 0 → 2. cos θ = −1 → 4, the largest possible.

So on the sphere, "nearest by distance" and "most similar by cosine" pick the same neighbour. You can use either. Try it below, then turn normalisation off.

Why normalise?

Drag the tips of a and b. Then slide "stretch a": with normalisation off, the dot product grows without the direction changing. With it on, only the angle counts.

1.00×

Exact arithmetic on the vectors you set.

Why normalise at all?

Without normalisation, a loss that wants a large dot product has an easy cheat. It can make vectors longer instead of better aligned. Training then grows lengths and ignores directions. Normalisation removes the cheat. It also bounds every similarity between −1 and 1, so a single number, the temperature, can set how sharp the comparisons are (Chapter 4).

SimCLR tested this. Without L2 normalisation, its contrastive task was easier to solve, but the representation was worse under a linear probe: 57.2% ImageNet top-1 at the best temperature, against 64.4% with normalisation SimCLR · Table 5.

Two unit vectors have cosine similarity 0.5. What is their squared distance?

Chapter 2 Why it works

Pairs and triplets

Write the first metric losses, derive the triplet gradient by hand, and see why most triplets teach nothing

How do you tell a network "these two should be close"? The oldest answer uses pairs. A positive pair is two items of the same class. A negative pair is two items of different classes. The contrastive loss pulls positive pairs together and pushes negative pairs apart, but only until they are a margin m apart:

d
The distance between the two embeddings.
y
1 for a positive pair, 0 for a negative pair.
m
The margin. A negative pair farther apart than m costs nothing.
L
The loss for one pair.

Worked, m = 1: a positive pair at d = 0.3 costs 0.09. A negative pair at d = 0.4 costs (0.6)² = 0.36. A negative pair at d = 1.2 costs 0.

The pair loss has a flaw. It asks for absolute distances, but faces under different light may never get very close. The triplet loss asks only for an order. Take an anchor a, a positive p of the same class, and a negative n of another class. The positive must be closer than the negative, by a margin α:

‖a − p‖²
The squared distance from the anchor to its positive.
‖a − n‖²
The squared distance from the anchor to the negative.
α
The margin. FaceNet sets α = 0.2 on unit-length 128-D embeddings FaceNet · §3.3.
L
Zero when the negative is already farther by at least α.

The gradient, by hand. When the loss is active, L = ‖a − p‖² − ‖a − n‖² + α. The derivative of ‖a − p‖² with respect to a is 2(a − p). So:

∂L/∂a
2(a − p) − 2(a − n). A step against it moves a toward p and away from n.
∂L/∂p
A step against it moves p toward a.
∂L/∂n
A step against it moves n away from a. When L = 0, all three gradients are zero.

Worked on the unit circle, α = 0.2: a at 0°, p at 40°, n at 45°. ‖a − p‖² = 2 − 2 cos 40° = 0.468. ‖a − n‖² = 2 − 2 cos 45° = 0.586. L = 0.468 − 0.586 + 0.2 = 0.082, active. Move n to 70°: ‖a − n‖² = 1.316, L = 0.

The triplet and its pull

Drag the three points around the circle. Warm arrows show the step each point takes. Then switch to a whole batch: most triplets are already satisfied and give no gradient.

0.20

The batch view uses 4 classes × 8 points on the circle, each class spread by about 20°: 4 × 8 × 7 × 24 = 5,376 triplets. Numbers are illustrative.

Most triplets teach nothing: mining

Sort the triplets by where the negative sits. An easy negative is already farther than the positive plus α; its loss and gradient are zero. A semi-hard negative is farther than the positive, but inside the margin. A hard negative is closer than the positive. As training goes on, almost every random triplet becomes easy. The loss goes quiet while the space is still poor.

So you mine: you pick the triplets that still teach. FaceNet found that always taking the hardest negatives "can result in a collapsed model (i.e. f(x) = 0)" early in training, and used semi-hard negatives inside large batches of about 1,800 examples FaceNet · §3.2. Hard negatives are often mislabelled or truly ambiguous, so they pull the space in bad directions.

A triplet has ‖a − p‖² = 0.5, ‖a − n‖² = 0.9 and α = 0.2. What gradient does it give?

Chapter 3 Why it works

InfoNCE: find your partner

Turn "be closer than the negatives" into a classification over all negatives at once, derive the loss from softmax, and see how it weights hard negatives for free

A triplet compares one positive with one negative. Why not compare it with all of them at once? Picture a party game. You hold one half of a torn photo. In the room are many other halves. Your task: point to your own other half. That is a classification problem with as many classes as there are candidates.

Softmax solves classification problems. So score each candidate k by its cosine similarity to the anchor, sk = z · zk, divide by a temperature τ, and apply softmax. The loss is minus the log of the probability given to the true partner:

s+
The cosine similarity between the anchor and its positive (the other view of the same item).
k
Runs over the positive and all negatives. In SimCLR, a batch of N items gives 2N views, so each anchor has 2N − 2 negatives.
τ
The temperature. Small τ makes the softmax sharp.
ℓ
The loss for one anchor. Chen and colleagues call this NT-Xent, the normalised temperature-scaled cross-entropy SimCLR · Eq. 1. The general form is called InfoNCE.

Worked: s+ = 0.8; negatives 0.5, 0.1, −0.3; τ = 0.5. Scores /τ = 1.6, 1.0, 0.2, −0.6. ex = 4.95, 2.72, 1.22, 0.55; sum 9.44. p+ = 0.525 → ℓ = −ln 0.525 = 0.645.

The gradient weights negatives by hardness. This is the same softmax cross-entropy as any classifier, so its gradient has the same form. With pk the softmax probability of candidate k:

pk
esk/τ divided by the sum. A negative that looks like the positive gets a large pk.
∂ℓ/∂sk
How hard the loss pushes on that similarity. Each negative is pushed in proportion to its own softmax probability.

Worked, same numbers: p = (0.525, 0.288, 0.129, 0.058). The hardest negative (0.5) gets 0.288 / 0.475 = 61% of the push on negatives. At τ = 0.1 it gets 98%.

So InfoNCE mines hard negatives by itself, softly, with no sampling rule. This is the main reason it beat the alternatives in SimCLR's comparison. With the same normalisation and tuned settings, ImageNet linear-probe accuracy was 50.9% for the margin triplet loss, 57.5% for the triplet loss with semi-hard mining, and 63.9% for NT-Xent SimCLR · Table 4.

Who gets the push?

The anchor (teal) sits at the top of the circle. Drag its positive, or a negative. The bars show each candidate's softmax probability: the positive's bar is what the loss wants to fill; each negative's bar is how hard it gets pushed away.

0.50

Eight negatives on a 2-D circle, so cosine = cos(angle). Exact softmax arithmetic. Positions are illustrative.

Why "more negatives" matters

The loss asks the positive to beat every negative. With more negatives, the task is harder, and the chance that some negative is truly close goes up. That is the hard negative the gradient needs. SimCLR draws its negatives from the same batch, so it trains with very large batches: 4096 by default, and a batch of 8192 gives 16,382 negatives per positive pair SimCLR · §2.2, §2.3. Other methods keep a queue of past embeddings instead.

In InfoNCE, which negative receives the strongest push away from the anchor?

Chapter 4 Why it works

Temperature: how hard the loss looks

See temperature as a focus knob, work its effect with numbers, and read what SimCLR measured when it turned the knob

Cosine similarities live between −1 and 1. That is a narrow range for softmax. e1 is only 7.4 times e−1, so a plain softmax over cosines can never be confident. Dividing by τ stretches the range. With τ = 0.1, the scores run from −10 to 10.

Think of τ as a focus knob. A large τ spreads attention over all negatives, near and far. A small τ focuses it on the few nearest ones. Look at the ratio of the push on two negatives with similarities s1 and s2:

s1 − s2
How much more similar negative 1 is to the anchor than negative 2.
τ
The temperature.
p1 / p2
How many times harder negative 1 is pushed than negative 2.

Worked, s1 = 0.5, s2 = 0.1: τ = 1 → e0.4 = 1.5×. τ = 0.5 → e0.8 = 2.2×. τ = 0.1 → e4 = 55×. τ = 0.05 → e8 = 2,981×.

Too large a τ, and near and far negatives get almost the same push. The loss stops caring about the hard cases. Too small a τ, and only the single nearest negative counts. If that negative is a near-duplicate or a mislabel, the loss chases noise. And the gradient grows like 1 / τ, so training gets unstable. The best τ sits in between.

The focus knob

255 negatives sit at evenly spread angles. The curve is the share of the push each one gets, against its cosine with the anchor. Slide τ: the push slides from everyone to only the nearest few. The bars on the right are SimCLR's real results.

0.10

The curve: exact softmax over 255 negatives on a 2-D circle, illustrative. The bars: SimCLR Table 5, ResNet-50, 100 epochs, batch 4096, linear evaluation on ImageNet; the "no L2 norm" bar is the best unnormalised run (τ = 10).

What the evidence says

SimCLR trained the same model at four temperatures with L2 normalisation. Linear-probe accuracy on ImageNet was 59.7% at τ = 0.05, 64.4% at τ = 0.1, 60.7% at τ = 0.5 and 58.0% at τ = 1 SimCLR · Table 5. The best value sat in the middle. Khosla and colleagues used τ = 0.1 for supervised contrastive learning and note that the gradient scales inversely with τ and that "extremely low temperatures are harder to train due to numerical instability" SupCon · §4.5.

You can see the same thing in the hero. Pick InfoNCE and slide τ to 0.03. The head trains worse on the known classes, and the new class falls apart.

Two negatives have cosines 0.6 and 0.2 with the anchor. At τ = 0.1, how much harder is the first one pushed?

Chapter 5 Why it works

Labels and angles: SupCon and ArcFace

Use your labels to define positives, then turn a softmax classifier into a metric learner by adding a margin to the angle

SimCLR has no labels. Its only positive is another view of the same image. If you do have labels, every other item of the same class is a positive too. The supervised contrastive loss, SupCon, averages the InfoNCE term over all of them SupCon · Eq. 2:

P(i)
All other items in the batch with the same label as anchor i.
a ≠ i
Every other item in the batch, positive or negative.
ℓi
The loss for anchor i. With one positive, it is InfoNCE again.

Khosla and colleagues reached 78.7% ImageNet top-1 with a ResNet-50 trained with SupCon, against 78.2% for cross-entropy in their setup SupCon · Table 2.

ArcFace: a classifier with a margin on the angle

There is another way to use labels. Keep a softmax classifier, but make it work on the sphere. Normalise the embedding x and every class weight wj. Then each logit is wj · x = cos θj, the cosine of the angle to class j. Multiply by a scale s, which plays the role of 1 / τ. This is the normalised softmax.

It already clusters classes by angle. But a test face only needs to be a little closer to its own centre than to another to be classified correctly. For matching new people, we want classes tight, with gaps between them. ArcFace adds a margin m to the angle of the true class during training only ArcFace · §3.1:

θy
The angle between the embedding and its own class's weight vector: arccos(wy · x).
m
The additive angular margin, in radians. ArcFace uses m = 0.5, about 28.6°.
s
The scale. ArcFace uses s = 64.
logity
The true class's score, made worse on purpose. The model must beat the other classes even with this handicap.

Derive the handicap. The angle-sum rule gives cos(θ + m) = cos θ cos m − sin θ sin m. So the code needs only cos θ from the dot product; sin θ = √(1 − cos² θ).

Worked, m = 0.5: θy = 30° → cos 30° = 0.866, but cos(30° + 28.6°) = 0.866 × 0.878 − 0.5 × 0.479 = 0.520. Another class at θ = 60° scores cos 60° = 0.5. With s = 10: without the margin, py = 0.975 and the loss is 0.025. With it, py = 0.551 and the loss is 0.597. The loss keeps pulling.

Why the margin tightens classes. Take two class centres an angle Δ apart and a training point between them, at angle φ from centre 1. Without a margin, it counts as class 1 while φ < Δ / 2: the boundary is halfway. With the margin, it must satisfy φ + m < Δ − φ, that is φ < (Δ − m) / 2. Each class's training zone shrinks by m / 2 on each side. A gap of width m opens between neighbours. Classes get pulled into tight cones.

The margin opens a gap

Two class centres sit on the circle. Drag the teal training point, which belongs to the blue class. Raise the margin: the blue zone shrinks and a gap opens between the classes.

0.50
16

Two classes on a 2-D circle, 100° apart. Exact ArcFace logits and softmax for the point you place. Illustrative geometry.

On face verification, the margin pays off. With the same ResNet-50 trained on CASIA, verification accuracy on LFW, CFP-FP and AgeDB was 99.08%, 94.39% and 92.33% with plain softmax, and 99.53%, 95.56% and 95.15% with ArcFace at m = 0.5 ArcFace · Table 2. A triplet loss with margin 0.35 reached 98.98%, 91.90% and 89.98% in the same table.

Two class centres are 100° apart. With ArcFace margin m = 20°, how far from its own centre can a training point be and still count as correct?

Chapter 6 Why it works

Before or after the head?

Read SimCLR's surprising result, that the layer before the projection head transfers better than the head's output, and train a toy that shows why

Here is a puzzle. SimCLR trains a projection head on top of the backbone, and the loss is computed on the head's output z. Then, for every downstream task, it throws the head away. It uses h, the backbone's output, instead SimCLR · §2.1. Why train something you then discard?

Because the head helps h become better, and h is better than z. Chen and colleagues measured linear-probe accuracy for three heads SimCLR · §4.2, Fig. 8:

Their explanation: the contrastive loss trains z to be the same for two augmented views of one image. So z must be blind to whatever the augmentations change, such as colour or orientation. But those facts can matter for a downstream task. With a head in place, the loss can make z blind while h keeps the information. Without a head, h itself must go blind.

They tested this directly. They trained small MLPs to predict, from h or from z, which transformation had been applied to an image SimCLR · Table 3:

What to predictRandom guessFrom hFrom z = g(h)
Colour vs greyscale8099.397.4
Rotation (4 ways)2567.625.6
Original vs corrupted5099.559.6
Original vs Sobel filtered5096.656.3

From z, rotation is predicted at 25.6%, barely above the 25% guess. From h it is 67.6%. The head discards what the loss does not need. Try it yourself:

The head forgets the colour

Each item has a shape class (3 kinds) and a colour (filled or hollow). The augmentation repaints the colour at random. Press Train: the head learns to ignore colour. Then read what a linear probe can still find, before and after the head.

h = (x, y, 1.2 × colour), with 3 classes in the (x, y) plane and colour ±1 at random. Head: linear 3 → 2, then L2 normalise, trained with NT-Xent (τ = 0.2, 24 items per batch, two views each). Probes: logistic regression on 60 training items, tested on 60 others. Chance for colour is about 50%. Numbers are illustrative.

What this means for you

If you pretrain with a contrastive loss and then transfer, take the features before the projection head. If you deploy the metric space itself, for search or verification, use the head's output z: it was shaped for exactly that comparison. And when you add a projection head on top of a frozen backbone, as in Backbone + Head, the head's output keeps only what your loss rewards. Make sure the loss rewards what you need.

Why does SimCLR's z = g(h) predict image rotation at only 25.6%, near the 25% guess?

Chapter 7 Build it

Prototypes and retrieval

Recognise a new class from five examples by averaging them, see why that is a linear classifier in disguise, and search a gallery by sorting distances

You have a good space. A new bird species arrives with five labelled calls. How do you recognise more calls of it? The simplest rule that works: embed the five calls, average them, and call the average the class's prototype. A new call belongs to the class whose prototype is nearest. Snell, Swersky and Zemel turned this into Prototypical Networks ProtoNets · §2.2:

Sk
The support set of class k: its few labelled examples, the "shots".
ck
The prototype: the mean embedding of the support set.
d
A distance. The paper uses the squared Euclidean distance.

Worked, 2-D: shots (0.9, 0.1), (0.7, 0.3), (0.8, 0.2) → c = (0.8, 0.2). A query at (0.75, 0.25) has d = 0.05² + 0.05² = 0.005 to c.

A linear classifier in disguise. Expand the squared distance: −‖z − ck‖² = 2 ck · z − ‖ck‖² − ‖z‖². The last term is the same for every class, so it cancels in the softmax. What is left is a linear layer with weights 2ck and bias −‖ck‖² ProtoNets · §2.4. Few-shot learning with prototypes is building a linear head from data, with no gradient steps.

The paper trained the embedding in episodes: each step samples N classes and K shots per class, as at test time. On miniImageNet 5-way, it reached 49.42% with 1 shot and 68.20% with 5 shots; on Omniglot 5-way, 98.8% and 99.7% ProtoNets · Tables 1, 2. It also found squared Euclidean distance "greatly outperforms" cosine in its unnormalised space §1. On a unit sphere the two rank neighbours the same way (Chapter 1).

How many shots?

Each episode draws 5 new classes, K support shots each (big dots), and queries (small dots). Stars are prototypes. Change K and the spread: 200 episodes rerun at once and compare prototypes with 1-nearest-neighbour. Draw another episode to see a different one.

5
0.34

Synthetic 2-D embeddings: class centres placed at random, each point the centre plus Gaussian noise of the chosen spread. 15 queries per class. Numbers are illustrative.

Retrieval: search is a sort

Retrieval needs no classes at all. Embed every item in a gallery once and store the vectors. For a query, embed it and sort the gallery by cosine similarity. The top k results are the answer. With unit vectors, all cosines for a query are one matrix product: Q (1 × K) times GT (K × M) gives M scores. For millions of items, an approximate nearest-neighbour index finds the top k without scoring every item; see vector databases.

Why is classification by the nearest prototype (squared Euclidean) a linear classifier?

Chapter 8 Build it

Build it in PyTorch

Trace the shapes through NT-Xent and ArcFace, then write the projection head and both losses from scratch, and finish with the library version

By hand first. Take a batch of N = 2 images, two views each, so 2N = 4 embeddings: views 0 and 1 come from image A, views 2 and 3 from image B. Their 4 × 4 cosine matrix S = Z ZT has 1s on the diagonal. We set the diagonal to −∞, because an item is not its own candidate. Then row 0's target is column 1, row 1's is column 0, row 2's is column 3, and row 3's is column 2. NT-Xent is ordinary cross-entropy on the rows of S / τ with those targets.

Follow the shapes

Pick a loss and step through one training step. For NT-Xent, the similarity matrix shows the masked diagonal (grey) and each row's positive (green).

images
4

Shapes for a ResNet-50 backbone (D = 2048), a projection to K = 128 for NT-Xent, and K = 512 with C class weights for ArcFace, as in the two papers. The matrix cells use random unit vectors, for illustration.

From scratch. The projection head and NT-Xent. Every line is commented with its shape.

import torch
import torch.nn as nn
import torch.nn.functional as F

class ProjectionHead(nn.Module):
    """(B, D) features -> (B, K) unit vectors."""
    def __init__(self, d=2048, hidden=512, k=128):
        super().__init__()
        self.net = nn.Sequential(nn.Linear(d, hidden), nn.ReLU(), nn.Linear(hidden, k))

    def forward(self, h):
        return F.normalize(self.net(h), dim=1)  # every row has length 1

def nt_xent(z1, z2, tau=0.1):
    """z1, z2: (N, K) unit vectors, two views of the same N items."""
    n = z1.shape[0]
    z = torch.cat([z1, z2])                          # (2N, K)
    sim = z @ z.T / tau                              # (2N, 2N) cosines / tau
    sim.fill_diagonal_(float("-inf"))              # an item is not its own candidate
    target = torch.cat([torch.arange(n, 2 * n),   # row i's partner is i + N ...
                        torch.arange(0, n)])        # ... and row i + N's is i
    return F.cross_entropy(sim, target)             # mean over all 2N anchors

torch.manual_seed(0)
head = ProjectionHead()
h1, h2 = torch.randn(8, 2048), torch.randn(8, 2048)    # stand-in backbone features
loss = nt_xent(head(h1), head(h2))
loss.backward()
print(f"NT-Xent {loss.item():.3f}, chance level ln(2N-1) = {torch.log(torch.tensor(15.)):.3f}")

With random inputs, the loss sits near ln(2N − 1), the value for guessing among 15 candidates. That is a useful first sanity check. Now ArcFace, with the class weights as a trainable matrix and the angle-sum rule from Chapter 5:

import math

class ArcFace(nn.Module):
    """(B, K) embeddings + (B,) labels -> scalar loss. Weights: (C, K)."""
    def __init__(self, k, n_classes, s=64.0, m=0.5):
        super().__init__()
        self.W = nn.Parameter(torch.randn(n_classes, k) * 0.01)
        self.s, self.m = s, m

    def forward(self, x, y):
        cos = F.normalize(x, dim=1) @ F.normalize(self.W, dim=1).T   # (B, C)
        cos = cos.clamp(-1 + 1e-7, 1 - 1e-7)
        theta = torch.acos(cos.gather(1, y[:, None]))           # (B, 1) angle to true class
        target = torch.where(theta + self.m < math.pi,
                             torch.cos(theta + self.m),            # the handicapped score
                             cos.gather(1, y[:, None]) - self.m * math.sin(self.m))
        logits = cos.scatter(1, y[:, None], target) * self.s  # (B, C)
        return F.cross_entropy(logits, y)

arc = ArcFace(k=128, n_classes=10)
emb = head(torch.randn(16, 2048))                         # (16, 128)
labels = torch.randint(0, 10, (16,))
loss = arc(emb, labels)
loss.backward()                                              # trains head AND arc.W
print(f"ArcFace {loss.item():.2f}")

Two details matter. First, the optimizer must include arc.parameters(), or the class weights never move. Second, the torch.where guard handles θ + m > π, where cos(θ + m) would start to rise again. At test time you keep only head.

The library version. The pytorch-metric-learning package has all of these losses and the miners. Note that its ArcFace margin is in degrees, 28.6 by default, which is 0.5 radians, and that the loss owns the class weights:

# pip install pytorch-metric-learning
from pytorch_metric_learning import losses, miners

emb = head(torch.randn(32, 2048))                         # (32, 128)
labels = torch.randint(0, 4, (32,))

supcon = losses.SupConLoss(temperature=0.1)                # Chapter 5, labels as positives
miner = miners.TripletMarginMiner(margin=0.2, type_of_triplets="semihard")
triplet = losses.TripletMarginLoss(margin=0.2)
arcface = losses.ArcFaceLoss(num_classes=4, embedding_size=128, margin=28.6, scale=64)

print(supcon(emb, labels).item(), triplet(emb, labels, miner(emb, labels)).item(),
      arcface(emb, labels).item())
opt = torch.optim.AdamW(list(head.parameters()) + list(arcface.parameters()), lr=1e-3)
In the nt_xent code, why is the diagonal of sim set to minus infinity?

Chapter 9 Did it work?

Did it work?

Measure a metric space with recall@k, mAP and the verification error rate, always on classes held out from training, and learn to spot collapse, hubs and shortcuts

The first rule: test on classes the head never saw. A metric space exists to place new classes. If the test classes were also training classes, a classification head would do as well, and you learn nothing. Split by class, not by example.

Then pick the metric that matches the job.

R
The number of correct (relevant) items for this query.
P@k
Precision at rank k: the fraction of the top k that is correct.
AP
Average precision for one query. mAP is its mean over queries.

Worked: ranked list ✓ ✗ ✓ ✗ ✗, R = 2. Recall@1 = 1. P@1 = 1, P@3 = 2/3 → AP = (1 + 0.667) / 2 = 0.833.

The verification check

Green: cosine scores of pairs from the same class. Grey: pairs from different classes. Slide the threshold. Then pick a failure mode and watch the distributions, the ROC curve and the EER change.

0.45

Simulated embeddings: 40 held-out classes × 6 items in 16-D, each item its class direction plus noise. Collapse squeezes all points toward one shared direction; hubs give items a shared offset of very uneven size, so a few sit near everyone; the shortcut keeps only a weak class signal, as when the cue the head learned is missing on test data. All 28,680 pair scores are computed. Numbers are illustrative.

The three failure modes

Your face model gets recall@1 of 99% when tested on the 500 people it was trained on. What is wrong with this test?

Chapter 10 Where else When not

Where else it works, and when not to use it

Carry the same head to six fields, see what changes and what stays, and learn when a plain classifier or another tool is the better choice

The block moves between fields with three swaps: a backbone suited to the data, a definition of "positive pair", and a loss that fits the labels you have. The head itself, Linear or MLP then L2 normalise, stays the same.

FieldPositive pairLossWhat changes
Face and speaker verificationTwo photos or recordings of one personArcFace or another angular marginMillions of identities; the EER at a fixed false-accept rate is the metric.
Product and image retrievalTwo photos of one productSupCon, or triplet with semi-hard miningThe catalogue changes daily; new products are enrolled, not trained.
Sound similarity and audio searchA clip and its text caption (CLAP), or two crops of one clipSymmetric InfoNCE across the two modalitiesTwo encoders, two projection heads into one shared space.
Few-shot bioacousticsTwo calls of one speciesPrototypes on a pretrained bird or audio embeddingFive calls of a rare species define its prototype.
Text semantic searchA question and the passage that answers itInfoNCE with in-batch negativesHard negatives come from a keyword search that returned the wrong passage.
Robot place recognitionTwo camera frames of one place, at different timesTriplet or contrastive, positives found by GPSLighting, season and viewpoint must not move the point; loop closure uses recall@k.

The audio row is the CLAP recipe: an audio encoder and a text encoder, each followed by its own projection, trained so a clip and its caption land close. See the CLAP paper lesson and CLIP's contrastive training. For bird calls with few examples, see Perch 2.

Metric head, or something else?

Answer four questions about your problem. The path lights up and ends in a recommendation.

A rule of thumb for a first build. Always confirm with held-out classes, as in Chapter 9.

When not to use it

Avoid it whenBecauseUse instead
The class list is fixed and every class has many labelsYou never need to place a new class, and softmax uses the labels directlyA plain linear or MLP classification head with cross-entropy
Batches are tiny and there are no labelsIn-batch contrastive losses starve for negativesA memory queue of negatives, a non-contrastive method, or a pretrained embedding with prototypes
You need calibrated probabilitiesA cosine of 0.7 is not a 70% chance; the scale depends on τ and the dataA classifier with calibration, or fit a calibration map on held-out pairs
"Similar" changes with the user or the queryOne space encodes one notion of similarityA cross-encoder or a reranker that sees both items together
The positives are defined by an augmentation that destroys your signalThe head learns to ignore whatever the augmentation changes (Chapter 6)Change the augmentations, or use labelled positives
Which project is the worst fit for a projection head with a metric loss?

Chapter 11

Connections

Get a first recipe for your own field, keep the one-screen cheat sheet, know the limits, and follow the series onward

Everything here fits on one card. Pick your field below for a first recipe, then use the cheat sheet to adjust it.

Your first recipe

Choose a field. The card gives the shape signature, the loss and its settings, and the check that proves it worked.

Starting points, not final answers. Settings follow the papers cited in this lesson where they exist; tune them on held-out classes.

The cheat sheet

PartIn one line
The blockBackbone → projection head (Linear or MLP) → L2 normalise → metric loss. At test time: compare by cosine.
Shape signatureB × D → B × K, every row length 1; a batch gives a B × B (or 2N × 2N) similarity matrix.
Maths in one line‖z1 − z2‖² = 2 − 2 cos θ; InfoNCE pushes each negative by pk / τ; ArcFace trains on s · cos(θy + m).
KnobsOutput size K (128 to 512); temperature τ (about 0.05 to 0.2); margin (triplet 0.2, ArcFace 0.5 rad, s = 64); batch size; what counts as a positive.
Failure modesCollapse; hubs; shortcut features; quiet loss from easy triplets; τ too small or too large; the head discarding what you need.
ChecksHeld-out classes only; recall@k, mAP, EER; frozen-feature and random-head baselines; spread of embeddings; hub counts; cross-session tests.
AlternativesSoftmax head for fixed classes; calibrated classifier for probabilities; cross-encoder for query-specific similarity.

Honest limits

Where this sits, and where to go next

This lesson is part of The Builder's Toolkit, a series on reusable blocks:

Foundations on the site:

The takeaway. A classification head answers with a fixed list of names. A projection head answers with a point on a sphere, where cosine means similarity, so new classes can be placed, matched and searched without retraining. The loss decides what the space keeps: temperature sets its focus, a margin sets its gaps, and the layer before the head keeps what the head throws away.
You pretrained an encoder with NT-Xent and now want a linear probe for a new image task. Which features do you use?

Now press Present or Teach and explain, out loud and from memory, why a space can recognise a class it never saw. Then go back to the hero, pick InfoNCE, slide τ down to 0.03, and watch the new class fall apart.

Projection Heads and Metric Learning, part of The Builder's Toolkit. Sources: T. Chen, S. Kornblith, M. Norouzi, G. Hinton, "A Simple Framework for Contrastive Learning of Visual Representations" (SimCLR, ICML 2020, arXiv); J. Snell, K. Swersky, R. Zemel, "Prototypical Networks for Few-shot Learning" (NeurIPS 2017, arXiv); J. Deng, J. Guo, N. Xue, S. Zafeiriou, "ArcFace: Additive Angular Margin Loss for Deep Face Recognition" (CVPR 2019; numbers from the extended arXiv version with J. Yang and I. Kotsia, arXiv); P. Khosla et al., "Supervised Contrastive Learning" (NeurIPS 2020, arXiv); F. Schroff, D. Kalenichenko, J. Philbin, "FaceNet: A Unified Embedding for Face Recognition and Clustering" (CVPR 2015, arXiv); R. Hadsell, S. Chopra, Y. LeCun, "Dimensionality Reduction by Learning an Invariant Mapping" (CVPR 2006, the contrastive pair loss); A. van den Oord, Y. Li, O. Vinyals, "Representation Learning with Contrastive Predictive Coding" (2018, arXiv); T. Wang, P. Isola, "Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere" (ICML 2020, arXiv); K. Musgrave, S. Belongie, S.-N. Lim, "A Metric Learning Reality Check" (ECCV 2020, arXiv).
Back to Gleams