FaceNet to ArcFace: the metric-learning canon

Chapter 1: The Triplet, Derived

We want a function f that maps an image to a vector, such that distance means identity. Let us try to write the loss for that requirement and watch it fail twice before it works. The two failures are not a detour — each one explains a term in the final formula, and if you skip them the margin looks like a magic number.

Attempt 1: just pull the same person together

The obvious first loss. Take two photos of the same person, an anchor a and a positive p, and minimise the distance between their embeddings:

Now find the minimiser. Set f(x) = c for every image x and any constant vector c. Then every distance is zero, L₁ = 0 everywhere, and the loss is globally minimised. The network has learned to output the same vector for every photograph in the world. This is called collapse, and it is not a rare failure mode you tune away — it is the exact optimum of the objective you wrote.

Attempt 2: pull together and push apart

Add a third image: a negative n, a photo of a different person. Now ask for the positive to be closer than the negative:

Collapse is dead: a constant function makes both terms zero and L₂ = 0, which is no longer the minimum, because the loss can go negative without bound by pushing negatives to infinity. Which is exactly the new problem. L₂ keeps rewarding you forever. A triplet that is already resolved — the positive is right on top of the anchor and the negative is on the other side of the space — still produces a large negative loss and a nonzero gradient. So the optimiser keeps spending capacity on triplets that were finished long ago, and the pairs that are actually confusable get drowned out.

There is also a scale problem. If the network can make all its outputs bigger, it can improve L₂ without improving any decision, simply by multiplying the whole embedding by 10 — every distance grows, and the difference of squared distances grows fastest for the already-easy triplets. The loss is not invariant to something we do not care about.

Attempt 3: a hinge with a margin, on a sphere

Both problems have the same fix in two parts. First, constrain the embedding to the unit hypersphere:

This is one line of code (divide by the norm) and it buys three things at once. Scale cheating becomes impossible. Squared distance becomes bounded: expand the square and use the unit norms,

so every squared distance lives in [0, 4] and is a strictly decreasing function of the cosine. And third, a single numeric margin now means the same thing everywhere in the space, which is Requirement 3 from Chapter 0.

Second, replace the unbounded difference with a hinge: demand that the negative be farther than the positive by at least a margin, and stop caring the instant that is true.

That is the triplet loss, and it is FaceNet's entire objective. In the paper's own notation the constraint being enforced is

with α = 0.2 in all of FaceNet's experiments. The hinge is what makes the loss satisfiable: once a triplet obeys the constraint, its loss is exactly zero and its gradient is exactly zero, and the optimiser moves on. That property is the entire reason the method works, and — as Chapter 2 shows — it is also the reason the method is so painful to train.

One more reading. If the positive is perfect — θ_ap = 0, so d_ap² = 0 — then the constraint says d_an² ≥ 0.2, so cos θ_an ≤ 0.9, so θ_an ≥ 25.84°. FaceNet is asking for about twenty-six degrees of angular clearance around each identity in the best case. Chapter 4's ArcFace will ask for 28.65° from every sample unconditionally. The two methods are closer than their formulas look.

Why three images and not two?

There is an older, simpler loss that also avoids collapse, and understanding why FaceNet did not use it is understanding what "metric" means here. The contrastive loss (Chopra, Hadsell and LeCun, 2005–2006) takes a pair and a binary label y (1 if same person):

Same-person pairs are pulled to distance zero; different-person pairs are pushed to at least m. It works, and it has one structural flaw: both targets are absolute. Every identity in the world is required to collapse to a single point, and every other identity is required to sit at least m away from it, with m a global constant.

Consider two identities in a real dataset. Identity A appears in 4 studio portraits: same lighting, same pose, same year. Identity B appears in 300 photographs spanning fifteen years, a beard, three continents and two decades of camera technology. The contrastive loss demands both collapse to a point. For B that is either impossible or, if the network manages it, achieved by discarding the very information that makes the embedding useful — it must learn to ignore everything that changed, which is almost everything.

The triplet loss asks for something strictly weaker and much more sensible: relative ordering with a margin. Identity B is allowed to occupy a large region of the sphere, as long as every other identity stays further from each of B's photos than B's other photos are. Formally the constraint is per-anchor, not per-identity, and never mentions an absolute distance.

Put numbers on it. Suppose after training, identity A's photos are all within 0.05 squared distance of each other and identity B's are spread over 0.30.

Read the last two rows: the triplet loss gives the two identities different effective margins, derived from their own data, from one global constant α. FaceNet's paper puts it as enforcing a margin "between each pair of faces from one person to all other faces", and notes this allows the faces of one identity to live on a manifold rather than a point.

A worked triplet, by hand

Two dimensions, so you can check every number with a pencil. Take unit vectors (each has norm exactly 1 — verify: 0.6² + 0.8² = 0.36 + 0.64 = 1):

Anchor-positive distance first, straight from the definition:

Cross-check with the cosine identity: a·p = 0.60(0.80) + 0.80(0.60) = 0.48 + 0.48 = 0.96, so d² = 2 − 2(0.96) = 0.08. The two routes agree, as they must. The angle is arccos(0.96) = 16.26°.

Now the three negatives, each with α = 0.2:

Three negatives, three completely different training signals from the same anchor and positive. n₁ contributes a forward pass, a backward pass and exactly nothing else. That single fact is the whole of Chapter 2.

The gradient, and one step by hand

When the hinge is open (loss > 0), the max disappears and the loss is a plain quadratic. Differentiate each of the three terms:

Read them physically. Descending on p means stepping along −2(p−a), which is straight toward the anchor. Descending on n means stepping along −2(a−n) = 2(n−a), straight away from the anchor. And the anchor itself moves along 2(p−n): toward the positive and away from the negative in one motion. Attraction and repulsion fall out of the algebra; nobody had to write them separately.

Take the semi-hard case (a, p, n₂) with learning rate η = 0.05 and step all three:

Recompute the loss with the new geometry. The anchor-positive angle shrank from 16.26° to 49.51 − 38.49 = 11.02°, and the anchor-negative angle grew from 20.61° to 75.75 − 49.51 = 26.24°:

One step, one triplet, and the loss fell by 80%. Two more steps of this size and the hinge closes; from then on this triplet is free to compute and worth nothing. Nothing in the optimiser knows that in advance.

Play with the hinge

Set the margin to zero and watch what happens: the loss becomes active only when the negative is strictly closer than the positive, so the model stops as soon as the ordering is right, with the two classes touching. That is the closed-set optimum from Chapter 0, and it is exactly the configuration that fails on strangers. The margin is the price of admission to open-set.

The realisation — what this is in code

The loss itself is four lines, and the shapes matter more than the arithmetic.

python — triplet loss, explicitimport torch, torch.nn.functional as F

# emb: (B, 128) raw encoder output — NOT yet on the sphere
emb = F.normalize(encoder(images), p=2, dim=1)   # (B, 128), every row has norm 1

a, p, n = emb[a_idx], emb[p_idx], emb[n_idx]         # (T, 128) each, T triplets

d_ap = (a - p).pow(2).sum(1)                       # (T,)  squared, no sqrt
d_an = (a - n).pow(2).sum(1)                       # (T,)
loss = F.relu(d_ap - d_an + 0.2).mean()             # the hinge, alpha = 0.2

# Diagnostic you must log from day one:
active = (d_ap - d_an + 0.2 > 0).float().mean()   # fraction of triplets doing any work

That last line is the most important diagnostic in triplet training, and it is the one people forget. If active is 0.01, then 99% of your batch produced a gradient of exactly zero and you paid full price for it. If active is 1.0 after a few epochs, something is wrong in the other direction — the model is not learning at all, or the mining is picking pathological negatives.

Chapter 2: The Mining Problem

Chapter 1 gave you a loss that is four lines long and provably does the right thing. This chapter is about the fact that the loss is the easy part, and that FaceNet's real contribution — the part that took a team at Google and a cluster to get right — is deciding which triplets to compute it on.

First, the size of the problem

FaceNet trained on 100 to 200 million face thumbnails covering about 8 million identities. Take the middle: N = 200M images, C = 8M identities, so roughly 200M/8M = 25 images per identity. Count the triplets.

For a given identity with k images, the number of ordered anchor-positive pairs is k(k−1). Each of those can pair with any image of any other identity, which is about N − k ≈ N negatives. So:

Nearly 10¹⁸ triplets. At FaceNet's batch size of roughly 1,800 exemplars, and even if every batch somehow produced a million usable triplets, you would need 10¹² batches to see them all once. The universe is about 4×10¹⁷ seconds old. Enumeration is not a strategy, and no amount of engineering makes it one.

Second, the fatal fact: almost every triplet is useless

Sample a triplet uniformly at random. What is the chance it is active — that its hinge is open and its gradient is nonzero?

Put a simple model on it. Suppose that after a little training the anchor-positive squared distances concentrate around d_ap² ≈ 0.25 (about 29°) and random anchor-negative squared distances concentrate around d_an² ≈ 1.6 (about 76°), with a spread of about 0.25 either way. The triplet is active when

We need a random negative to land at squared distance below 0.45 when it is typically at 1.6, which is about 4.6 standard deviations away. For a roughly Gaussian tail that is on the order of 2 in a million. Even being extremely generous and calling it one in a thousand, a batch of 1,800 exemplars sampled at random gives you a couple of useful triplets and 1,798 forward passes of pure heat.

Third: why the obvious fix destroys the model

If you want active triplets, take the most active ones: for each anchor-positive pair, search the dataset for the negative that is closest to the anchor. Every triplet now violates, every gradient is large, no compute is wasted. FaceNet tried it. The paper reports that selecting the hardest negatives "can in practice lead to bad local minima early on in training, specifically it can result in a collapsed model (i.e. f(x) = 0)."

The word "collapse" is worth deriving rather than accepting, because the derivation is three lines and it makes the failure permanent knowledge.

Suppose the network outputs the same vector for every image: f(x) = c for all x. Then for every triplet a = p = n = c, and from Chapter 1:

Every gradient is exactly zero, and the loss value is exactly α for every triplet. So total collapse is an exact stationary point of the triplet loss, sitting at loss α = 0.2, which looks like a plateau rather than a catastrophe on a loss curve. That is why it is a trap: the loss does not spike, it flatlines at a plausible-looking number, and if you are only watching the loss you will let it run overnight.

Now the mechanism that walks you into the trap. Real face datasets contain label noise: mislabelled images, two different people under one identity, occlusions, near-duplicates across identities. The globally hardest negative for an anchor is very often one of these — a photo labelled as a different person that is in fact the same person, or a photo so degraded that its embedding is meaningless. Hard mining seeks out precisely the examples where the label is most likely to be wrong. The gradient then commands the model to separate two images that genuinely should be together. Repeated a few thousand times with maximum gradient magnitude, the cheapest way to satisfy a pile of contradictory constraints is to satisfy none of them and go flat.

Semi-hard mining, derived

We want the hardest triplets that are still trustworthy. Draw the number line for d_an² with d_ap² and d_ap² + α marked on it, and there are exactly three regions:

FaceNet's rule is: within the current minibatch, for each anchor-positive pair, pick a negative satisfying

The bound on loss is the point. In Region 2 the loss is at most α = 0.2, so no single triplet can dominate a batch, so a mislabelled pair cannot swing the update. You get the largest gradient that is still bounded, which is the practical definition of a safe hard-example strategy.

Batch construction: the 1,800-exemplar story

Semi-hard selection requires knowing the distances, which requires embeddings, which requires forward passes. Doing that over the whole dataset every step is impossible. FaceNet's answer — the design that everyone copied — is online mining inside a large, deliberately structured minibatch.

The paper's setup: minibatches of around 1,800 exemplars, constructed so that roughly 40 faces are sampled per identity, with randomly sampled negative faces filling the rest. Do the arithmetic on what one such batch buys.

Two more details from the paper that are easy to skim past and expensive to rediscover:

Use all anchor-positive pairs, not the hardest one. Symmetry suggests mining hard positives too — for each anchor, the farthest same-identity image. FaceNet reports that using all anchor-positive pairs in the minibatch was more stable and converged slightly faster at the start of training. Hardest-positive mining has the same label-noise exposure as hardest-negative mining, pointed the other way: the farthest "positive" is disproportionately likely to be someone else's photo filed under this identity.

Offline versus online. You can also mine offline: every n steps, embed a subset of the data with the most recent checkpoint and compute the argmin/argmax. FaceNet describes both and uses the online version, because offline mining goes stale — the checkpoint that computed the distances is not the model taking the step, and the staler it gets, the more the "hard" negatives are hard for a model that no longer exists.

The deeper reason random sampling fails: distances concentrate

The estimate above ("about one in a thousand") came from a plausible model. There is a sharper, geometric reason that is worth knowing because it is a fact about high-dimensional space rather than about faces.

Take two independent, uniformly random unit vectors in d dimensions. Their inner product has mean 0 and standard deviation 1/√d. So the squared distance d² = 2 − 2cos θ has mean 2 and standard deviation 2/√d. Put in FaceNet's d = 128:

Almost every random pair sits at squared distance 2 ± 0.35. In 512 dimensions the spread shrinks further, to 2/22.6 = 0.088. This is concentration of measure: in high dimensions, random points are all about the same distance apart, and the useful structure lives in a vanishingly thin sliver of the distribution.

Now the consequence for mining. A useful negative must land below d_ap² + α ≈ 0.45. Against a distribution centred at 2.0 with standard deviation 0.177, that is (2.0 − 0.45)/0.177 = 8.8 standard deviations below the mean. Even accounting for the fact that real embeddings are far from uniform — identities cluster, and similar-looking people are genuinely close — you are fishing in the extreme tail. Uniform sampling is the wrong instrument for finding tail events; that is what importance sampling is for, and mining is importance sampling by another name.

Batch-hard: the variant that actually shipped everywhere else

Two years later, Hermans, Beyer and Leibe's "In Defense of the Triplet Loss for Person Re-Identification" simplified the recipe into the form most code uses today. Build a batch of P identities × K images each (typical: P = 18, K = 4, so B = 72). For each image in the batch as anchor, take the hardest positive and the hardest negative within the batch only:

The trick is that "hardest in a batch of 72" is a moderate difficulty, not a pathological one — a bounded form of hard mining, where the bound comes from the batch being small. Same principle as semi-hard, different implementation. They also propose the soft-margin variant, replacing max(0, x) with ln(1 + e^x), which removes the hyperparameter α entirely and never has exactly zero gradient.

The realisation — mining in code

All of it is one distance matrix and two masks. Read the shapes.

python — online semi-hard mining inside a batchemb = F.normalize(encoder(x), dim=1)          # (B, 128), B = 1800
D   = torch.cdist(emb, emb).pow(2)               # (B, B) squared distances — one matmul

same = labels[:, None] == labels[None, :]        # (B, B) bool: same identity?
pos  = same & ~torch.eye(B, dtype=torch.bool)   # exclude self-pairs
neg  = ~same

# For every (anchor, positive) pair we need negatives in the band
#     d_ap  <  d_an  <  d_ap + alpha
d_ap   = D.unsqueeze(2)                          # (B, B, 1) — anchor a, positive p
d_an   = D.unsqueeze(1)                          # (B, 1, B) — anchor a, negative n
band   = (d_an > d_ap) & (d_an < d_ap + 0.2)      # (B, B, B) semi-hard mask
valid  = pos.unsqueeze(2) & neg.unsqueeze(1) & band

losses = F.relu(d_ap - d_an + 0.2)[valid]        # only the goldilocks band
loss   = losses.mean() if losses.numel() else D.sum() * 0.0

# Fallback matters: if no semi-hard triplet exists in the batch, the
# standard recipe falls back to the hardest AVAILABLE negative rather
# than emitting zero — otherwise late training silently stops.

Note the memory line nobody mentions: the mask band is B³ booleans. At B = 1800 that is 5.8 × 10⁹ entries — 5.8 GB — so production implementations never materialise it. They loop over anchors, or they select one negative per anchor-positive pair with a masked argmin, keeping everything at O(B²). Writing the naive version once, seeing it die, and then fixing it is a good afternoon.

Chapter 3: Onto the Sphere

Here is the move that dissolves Chapter 2. It arrives in three steps, and each one is a single line of code with a real derivation behind it.

Step 1: replace sampled negatives with learned proxies

The expensive thing about a triplet is that the negative is a data point, so finding a good one means searching the data. Suppose instead each identity owns one learnable vector — call it a proxy — that stands in for all of that identity's images. Then a triplet becomes (x, proxy of my class, proxy of some other class), and the search space shrinks from "every image in the dataset" to "the C proxies", which you can simply enumerate.

This is ProxyNCA (Movshovitz-Attias et al., 2017). Its loss is neighbourhood component analysis with proxies substituted for data points:

where π_j is the proxy for class j and both x and the proxies are normalised. The paper reports roughly three times faster convergence than sampling-based metric learning while improving Recall@1 on the standard retrieval benchmarks. The reason is exactly the counting argument from Chapter 2: every sample now compares against all classes on every step, so there is no sampler, no staleness, and no possibility of spending a batch on triplets that were resolved last week.

That reframing is what turns Chapters 4 and 5 from a bag of tricks into a single idea: if the classifier head is a metric learner, then fix the metric it is learning. Two things are wrong with it out of the box.

The counting argument is worth doing, because it is the whole reason the field moved.

Read the third column downward. Each row buys negatives more cheaply than the last, and the last row buys all of them for the price of one matrix multiply. That is why the sampler disappears: not because someone found a better sampling strategy, but because the thing being sampled stopped being necessary.

What the proxies become, and how they get there

A proxy is initialised randomly, so at first it means nothing. Watch what the gradient does to it. For softmax cross-entropy with logits z_j = W_j · x, the gradient on class j's proxy from a single sample x with true class y is

Read the two cases. For the correct class, p_y − 1 is negative, so descending moves W_y along +x: the proxy is pulled toward the sample. For every wrong class, p_j is positive, so W_j is pushed along −x, away from the sample — and the push is proportional to how much that class currently believes it owns this sample. Sum over an epoch and W_y becomes a weighted average of its own class's features, weighted toward the ones the model finds hardest. The proxy converges to something like the class mean, and it was never told to.

Two useful consequences follow immediately. First, a class with few images gets few pulls, so its proxy stays close to its random initialisation and its logit stays weak — the long-tail problem, visible in the weight norms. Second, because the proxy is a running summary of the class rather than a specific image, the loss is far less sensitive to any single mislabelled photo than hardest-negative mining was: one bad image contributes one term to an average instead of selecting itself as the entire gradient. That robustness, more than the speed, is why margin-softmax replaced triplets.

Step 2: kill the bias, and see the two confounds in the norms

The logit for class j is

Three factors and an offset, and only one of them is about identity. Take them one at a time.

The bias b_j. It shifts a class's score regardless of where the feature points. A class with a large bias wins arguments it should lose. At deployment you throw W and b away entirely and compare features to features, so every unit of capacity spent on b is capacity that does nothing for the thing you ship — and worse, it lets the network satisfy training without organising the geometry. Every method from SphereFace onward sets b = 0. One line, no downside.

The weight norm ‖W_j‖. It absorbs class frequency. If identity A has 500 training images and identity B has 12, gradient descent grows ‖W_A‖ much larger, because a bigger norm reduces loss on every one of A's 500 images. The classifier then wins by prior, not by direction. Concretely, suppose ‖W_A‖ = 12 and ‖W_B‖ = 4, and a test feature (unit norm) sits at 70° from W_A and 40° from W_B:

The classifier says A. The geometry says B, by a wide margin — 40° versus 70°. After normalising the weights the logits become cos 70° = 0.342 and cos 40° = 0.766 and the answer flips to B, which is the answer a nearest-neighbour search on the embedding would have given. Since nearest-neighbour search is what you deploy, the unnormalised classifier was optimising the wrong decision rule.

The feature norm ‖x‖. It absorbs image quality. A sharp, well-lit, frontal face produces a large-norm feature; a blurred profile at 30 pixels produces a small one. Since the logit scales with ‖x‖, the easiest way for the network to reduce total loss is to inflate the norms of samples it already gets right — the gradient is larger there, and the improvement is free. So a plain softmax spends its capacity making easy samples easier, which is the opposite of what an open-set system needs.

Step 3: normalise both, and you are on a hypersphere

Set b = 0, and rescale so that ‖W_j‖ = 1 and ‖x‖ = 1. The logit reduces to a single quantity:

Every feature and every proxy now lives on the unit hypersphere S^d−1. All information is angular. And here is the geometric fact that Chapter 5 turns on: on a unit sphere, the geodesic distance between two points — the shortest path along the surface — is exactly the angle θ between them, in radians. Arc length equals radius times angle, and the radius is 1. So "angle" is not a proxy for distance on the sphere; it is the distance.

This is NormFace (Wang et al., 2017), and it immediately breaks in an instructive way.

Why normalising alone stops training dead

Cosines live in [−1, 1]. The softmax over such logits can never be confident. Take the best case imaginable: the feature sits exactly on its own proxy (cos θ_y = 1) and is orthogonal to all C−1 others (cos = 0). Then

using C = 85,742, the identity count of MS1MV2, the dataset ArcFace trains on. The cross-entropy is −ln(3.170×10⁻⁵) = 10.36. That is the floor: a perfect model, with every sample exactly on its proxy, cannot get the loss below 10.36. Compare that with the loss of a randomly-initialised model, which is ln(85,742) = 11.36. Perfection buys you one nat out of eleven. The gradient signal distinguishing "perfect" from "random" is a rounding error, and training does essentially nothing.

The fix is a single scalar. Multiply the logits by a scale s:

Rerun the same best case with s = 64. The correct logit is 64, the others are 0, and

with a loss of 1.4 × 10⁻²³. Now the loss can actually reach zero, so gradients mean something all the way down. The scale s is not a tuning knob you found empirically — it is the repair for a defect that normalisation introduced, and Chapter 5 derives how small it is allowed to be.

The last proxy: a function instead of a table

One more step is available, and it is the one that connects this entire lesson to modern multimodal models. The proxies are a lookup table: C rows, one per class, learned. What if the proxy for a class were computed instead of stored?

That is exactly what CLIP and CLAP do. Write their loss for one image i in a batch of N, where E_img and E_txt are L2-normalised:

Put that next to Chapter 3's normalised softmax and the correspondence is exact, term for term.

So the arc of this lesson has one more step in it than the title suggests: sampled negatives → stored proxies → generated proxies. Each step removes a bookkeeping burden and buys a new kind of generalisation. Chapter 9 closes the loop; our CLAP veanor derives the generated-proxy version end to end in the audio domain.

The scoreboard so far

That last row is the remaining hole, and it is the hole Chapters 4 and 5 fill. Normalised softmax asks each feature to be closer to its own proxy than to any other proxy. Satisfying it by one degree scores as well as satisfying it by thirty. For an open-set system, the thirty degrees is the entire product.

python — normalised softmax (NormFace), the base every margin builds onclass CosineHead(nn.Module):
    def __init__(self, d, n_classes, s=64.0):
        super().__init__()
        self.W = nn.Parameter(torch.randn(n_classes, d) * 0.01)   # the proxies — NO bias
        self.s = s

    def forward(self, x):                    # x: (B, d) raw encoder features
        x = F.normalize(x, dim=1)            # kill the quality confound
        W = F.normalize(self.W, dim=1)       # kill the frequency confound
        return self.s * x @ W.T           # (B, C) — every entry is s·cos(theta)

Three lines of change from a normal linear head, and the model that comes out is a metric learner with no sampler. Everything from here is one more term inside that final expression.

Chapter 4: Two Margins

We have a normalised softmax whose logits are s·cos θ_j. It trains stably, needs no sampler, and produces embeddings that are separable. Chapter 0's Requirement 3 asked for more than separable. This chapter is where we ask for it, in two different ways, and hand-compute the difference.

What the objective currently demands, exactly

For sample x with true class y and nearest competitor j, plain normalised softmax is happy when

The decision boundary between two classes is the angular bisector, and "correct" means being on your own side of it by any amount whatsoever. Nothing in the loss says how far.

Now remember what happens at deployment. The proxies W are thrown away. A stranger arrives, gets embedded, and must be distinguishable from a different stranger. Neither of them was ever optimised. They will land somewhere in the neighbourhood of the training identities that look like them. If the training identities were packed together with zero clearance, there is nowhere for the strangers to go.

The three ways to insert a margin

The target logit is s·cos θ_y. There are exactly three places to put a number m into cos θ_y, and the literature tried all three in order.

SphereFace came first and is the reason the other two exist. Multiplying the angle is elegant — the required gap grows with the angle — but cos(m θ) is only monotonically decreasing for θ ∈ [0, π/m], so beyond 45° (with m = 4) the "loss" starts rewarding larger angles. SphereFace patches this with a piecewise surrogate ψ(θ) = (−1)^kcos(m θ) − 2k and then, because even the patched version will not train from scratch, anneals it: start at ordinary softmax and gradually increase the margin's influence with a schedule on a parameter λ. Two hyperparameters and a schedule to make one margin behave. CosFace and ArcFace are both, in large part, "the same idea without the schedule".

CosFace, derived

Start from the requirement and work backwards. We want a classifier that is only satisfied when the correct class beats the runner-up by a fixed amount of cosine:

The trick is that you do not have to change the loss to demand this — you change the logit and keep ordinary cross-entropy. Subtract m from the true class's cosine before the softmax, and cross-entropy will not be happy until the shifted logit wins. The full loss, which the CosFace paper calls the Large Margin Cosine Loss:

Note carefully what m does not touch: the other classes' logits are untouched. The margin is a handicap applied to the class that is supposed to win. During training the model is graded as if its correct answer were worse than it is; at inference you drop the handicap entirely and simply use the embedding.

ArcFace, derived

Same construction, different quantity. ArcFace observes that the thing you actually care about — the separation on the hypersphere, the geodesic distance from Chapter 3 — is the angle, not its cosine. So add the margin to the angle before taking the cosine:

and, because cos is decreasing on [0, π], adding m to θ_y lowers the target logit exactly as subtracting m from the cosine did. The satisfied condition is

Read that right-hand form slowly, because it is the entire argument of the next chapter. ArcFace's constraint, after the cosines cancel, is a statement about angles with no cosines in it at all: your angle to your own class, plus a constant, must be less than your angle to any other class. A constant angular gap. Which, on a unit sphere, is a constant arc length — a constant distance.

Expanding with the angle-sum identity shows what it costs to compute:

so you never call arccos. With m = 0.5 rad, cos m = 0.877583 and sin m = 0.479426 are constants; the whole margin costs one square root and two multiplies per sample. That efficiency is not a footnote — it is why ArcFace was adopted so fast. It is three lines added to a standard classification head, with no sampler, no schedule and no measurable slowdown.

One sample, three losses, all by hand

This is the arithmetic to carry with you. A single training sample, three classes, s = 64.

Plain normalised softmax. Logits: 64(0.5) = 32.000, 64(0.258819) = 16.564, 64(−0.342020) = −21.889. Subtract the max (32) for stability and exponentiate: e⁰ = 1, e^−15.436 = 1.979×10⁻⁷, e^−53.889 ≈ 4×10⁻²⁴. So

The model is fifteen degrees from a different identity — genuinely tight for face verification — and the loss is two ten-millionths. This sample is, as far as training is concerned, finished. Nothing further will be learned from it. Compare that with the mining problem of Chapter 2 and notice it is the same disease: an objective that stops caring long before the geometry is good.

CosFace, m = 0.35. Only the target logit changes: 64(0.500000 − 0.35) = 64(0.15) = 9.600. The others are unchanged, so the largest logit is now the impostor's 16.564. Shift by that max: e^{9.600−16.564} = e^−6.964 = 9.449×10⁻⁴; e⁰ = 1; e^−38.454 ≈ 2×10⁻¹⁷.

ArcFace, m = 0.5 rad = 28.65°. The target angle becomes 60° + 28.65° = 88.65°, whose cosine is 0.023562 (it is sin 1.35°). Target logit: 64(0.023562) = 1.508. Shift by the impostor's 16.564: e^{1.508−16.564} = e^−15.056 = 2.891×10⁻⁷.

What each loss is asking the sample to do

The loss value is a scold. The useful question is: where does the sample have to move before the scolding stops? Solve each condition for θ_y, with the impostor fixed at 75°.

CosFace needs cos θ_y − 0.35 > 0.258819, so cos θ_y > 0.608819, so θ_y < arccos(0.608819) = 52.50°. The sample must travel 7.5°.

ArcFace needs θ_y + 28.65° < 75°, so θ_y < 46.35°. The sample must travel 13.65°.

Same nominal "margin idea", nearly double the demand in this instance. And that ratio is not fixed — it depends on where the sample sits, which is the entire subject of the next chapter.

The same three losses across the whole range

One sample is an anecdote. Sweep θ_y with the impostor fixed at 75° and s = 64, and the character of each loss shows up.

Two readings. Down a column: every loss grows as the sample gets worse, as it must. Across a row: at 40° the margined losses are ten orders of magnitude larger than the plain one, and at 80° only five to seven times larger. The margin's relative effect is largest exactly where the plain loss has given up — on samples that are correct but unremarkable, which is the overwhelming majority of a training set and precisely the population whose geometry determines open-set performance.

And look at the first row, where ArcFace's loss is four orders of magnitude smaller than CosFace's. That is not a mistake: near its own class centre, ArcFace's cosine-space penalty is only 0.122 against CosFace's flat 0.35, so it leaves easy samples alone and saves its pressure for the middle of the range. Chapter 5 derives exactly where that crossover happens and why it is the right shape.

One formula that contains all three

ArcFace's paper ends this argument neatly by writing a combined margin with three knobs:

That last row is the honest summary of a three-year argument: these are three points in one small family, and the differences between well-tuned members are smaller than the differences between training sets. Which is worth remembering the next time a leaderboard moves by 0.05%.

The realisation — ArcFace in code, including the part that bites

python — ArcFace head, reference formclass ArcFace(nn.Module):
    def __init__(self, d, n_classes, s=64.0, m=0.5):
        super().__init__()
        self.W = nn.Parameter(torch.randn(n_classes, d))
        nn.init.xavier_normal_(self.W)
        self.s, self.m = s, m
        self.cos_m, self.sin_m = math.cos(m), math.sin(m)   # 0.877583, 0.479426
        self.th = math.cos(math.pi - m)                     # -0.877583
        self.mm = math.sin(math.pi - m) * m                 #  0.239713

    def forward(self, x, labels):
        cos = F.linear(F.normalize(x), F.normalize(self.W))    # (B, C) cosines
        sin = torch.sqrt((1.0 - cos.pow(2)).clamp(0, 1))          # clamp: float error
        phi = cos * self.cos_m - sin * self.sin_m              # cos(theta + m)

        # Guard: cos(theta+m) is only decreasing while theta <= pi - m.
        # Past that, fall back to a LINEAR penalty so the target logit
        # keeps decreasing in theta instead of turning around.
        phi = torch.where(cos > self.th, phi, cos - self.mm)

        onehot = F.one_hot(labels, cos.size(1)).float()
        logits = self.s * (onehot * phi + (1 - onehot) * cos)   # margin ONLY on target
        return F.cross_entropy(logits, labels)

Everything after phi = cos * cos_m - sin * sin_m exists because of a problem the formula does not advertise. cos(θ + m) decreases in θ only while θ + m ≤ π. Past θ = π − m = 180° − 28.65° = 151.35°, the cosine turns around and starts increasing, so a sample that is even further from its class would get a higher target logit and a lower loss. The optimiser would be rewarded for making things worse. The two constants patch this: past the threshold cos θ = cos(π − m) = −0.877583, use the linear fallback cos θ − m·sin(π − m) = cos θ − 0.239713, which keeps decreasing forever.

Chapter 5: Why Angular Wins, and How s and m Kill You

Two margins, one sphere. This chapter answers why ArcFace's version is the principled one, and then spends the rest of its length on the two numbers that decide whether your training run converges or produces a very expensive constant function.

The geodesic argument

Chapter 3 established that on the unit hypersphere the geodesic distance between two points is the angle between them in radians. So the natural notion of "how much room is there around this identity" is measured in radians, and a margin that is constant in radians is a margin that is constant in distance on the manifold the model actually lives on.

ArcFace's satisfied condition was θ_y + m < θ_j: a fixed arc length, everywhere, for every sample. CosFace's was cos θ_y − m > cos θ_j, which is fixed in cosine units. Since cosine is a nonlinear function of angle, a fixed cosine gap is a varying angular gap. Compute how much it varies.

For CosFace, a sample at angle θ is treated as though it were at the angle θ′ = arccos(cos θ − m). The effective angular push is θ′ − θ:

CosFace's demand varies by a factor of 1.67 across the range, and — look at the direction — it demands most from samples that are already well inside their class, and least from the samples sitting near the boundary. Those boundary samples are the ones that will be confused with a stranger at deployment. The margin is weakest exactly where the product fails.

The honest comparison

The temptation is to conclude that ArcFace is strictly better. The numbers do not support anything that strong, and it is worth seeing why.

ArcFace's own paper concedes the closeness in the most useful way possible: it defines a combined margin, cos(m₁θ + m₂) − m₃, which contains SphereFace (m₁), ArcFace (m₂) and CosFace (m₃) as special cases, and reports that combinations of the three work about as well as any single one. The three papers are three points in one family, not three theories.

The s knob: a floor and a cliff

Chapter 3 showed that s exists because the loss has a floor without it. Now derive how small s can be.

Suppose you want a well-classified sample to reach probability P_W on its correct class. Best case again: cos θ_y = 1 and the other C−1 classes are orthogonal. Then

CosFace derives essentially this bound (with a (C−1)/C factor from a slightly more careful setup). Put ArcFace's numbers in: C = 85,742 and a modest target of P_W = 0.9.

So s must be at least about 14 just to make a perfect sample confident. Real samples are not perfect — they sit at 20° or 40°, not 0° — so you need headroom, which is why the papers use 64 and speaker-verification systems, with far fewer classes, use 30.

Check the s = 8 row by hand: e⁸ = 2980.96, so p = 2980.96/(2980.96 + 85,741) = 0.03360, and −ln(0.03360) = 3.393. A model that has solved the problem perfectly still reports a loss of 3.4, and every diagnostic you have will tell you it is broken.

The cliff at the other end is about gradients, not floors. Differentiating cross-entropy through the scaled logit gives ∂L/∂cos θ_y = s(p_y − 1), so every gradient carries a factor of s. At s = 64 a confidently wrong sample delivers a gradient 64 times what an unscaled loss would; with a mislabelled image, that is 64 times the wrong thing. In practice: keep s in [16, 64], and if you must raise it, lower the learning rate proportionally.

The m knob: three distinct ways to die

Failure 1: the initialisation blow-up. At initialisation the proxies are random, and in high dimensions two random vectors are nearly orthogonal — so θ_y ≈ 90° for every sample and every class. Plain softmax then gives all logits ≈ 0, so p = 1/C and the loss is ln(85,742) = 11.36, which is exactly what you see in step 1 of any classifier. Now add ArcFace's margin:

The margin adds thirty nats of loss at step zero, before the model has done anything wrong, and every one of those nats is multiplied by s in the gradient. Many runs diverge in the first few hundred steps. The standard remedies: warm up with m = 0 for an epoch and then switch on the margin; or ramp m linearly over the first few thousand steps (SphereFace's λ annealing is the ancestor of this); or use the easy-margin variant, which applies the margin only when cos θ > 0, so the worst case at initialisation is neutral rather than catastrophic.

Failure 2: the non-monotone tail. Chapter 4's guard clause. Without it, samples past θ = π − m are rewarded for getting worse, and a subpopulation of your data can migrate to the far side of the sphere and stay there, invisible on the training loss.

Failure 3: an unsatisfiable constraint. Push m high enough and the model literally cannot arrange 85,742 classes with that much clearance in the space available, so the loss plateaus high, accuracy falls, and eventually training destabilises. CosFace sweeps m from 0 up to 0.45 and finds performance improving to about 0.35 and then degrading, with the large settings unstable. ArcFace sweeps m and reports 0.5 as the best, with 0.4 and 0.6 close behind and larger values worse. These are not universal constants: they are the right answers for ~85K identities in 512 dimensions with a ResNet-100. Change the class count, the dimension or the data quality and the optimum moves.

Where the gradient actually points, derived

One more derivation, because it settles the ArcFace-versus-CosFace question more convincingly than any benchmark. Differentiate the loss with respect to the angle — the quantity we actually care about — using the chain rule through the target logit.

For cross-entropy, ∂L/∂z_y = p_y − 1. For ArcFace, z_y = s cos(θ_y + m), so ∂z_y/∂θ_y = −s sin(θ_y + m). Multiply:

It is positive, so descent decreases θ_y: the feature rotates toward its own class. Three factors multiply, and each has a clean meaning:

Now run the same derivation for CosFace, where z_y = s(cos θ_y − m) and ∂z_y/∂θ_y = −s sin θ_y:

Check it numerically on the Chapter 4 sample (θ_y = 60°, impostor 75°, s = 64). ArcFace: p_y = 2.89×10⁻⁷, so the gradient is 64 × 0.9999997 × sin(88.65°) = 64 × 0.99972 = 63.98 per radian. CosFace: p_y = 9.44×10⁻⁴, so 64 × 0.99906 × sin(60°) = 64 × 0.8652 = 55.37. About 16% more angular force on exactly the sample that needs it, from the same s and comparable m.

Reading a training curve: symptom to cause

A tuning procedure you can actually follow

Step 5 deserves emphasis. Adding a margin raises the training loss on purpose — that is what a handicap does. If you tune m by watching the loss go down, you will choose m = 0 every time. This is the most common way engineers "disprove" ArcFace on their own data.

Chapter 6: TAR at FAR, and Why Accuracy Lies

You have an embedding. Somebody asks how good it is. If your answer is a single accuracy number, you have not answered, and this chapter is about why — with every number computed by hand on a set small enough to check on paper.

The vocabulary, pinned down

A verification system compares two templates and produces a similarity score. There are exactly two kinds of comparison and exactly two kinds of error.

The two error rates trade off through one scalar: the threshold. Raise it and impostors are rejected along with some genuine users; lower it and everybody gets in. There is no setting that reduces both. Reporting one without the other is meaningless, which is why the field's standard unit of measurement is TAR at a fixed FAR — "how many real users do we let in, given that we accept a stranger only once in ten thousand tries".

A ROC computed entirely by hand

Twenty pairs. Ten genuine, ten impostor. Cosine scores, sorted:

For a threshold t, accept a pair when score ≥ t. Then FAR(t) = (impostors ≥ t)/10 and TAR(t) = (genuines ≥ t)/10. Sweep t downward and count.

Read the answers straight off the table.

TAR at FAR = 0 is 0.80, at threshold 0.55: we can accept 8 of 10 genuine pairs while admitting zero impostors, because the best impostor scores 0.52 and the eighth-best genuine scores 0.55. The whole system's quality lives in that three-hundredths of a gap.

EER: find where FAR = FRR. At t = 0.45, FAR = 0.2 and FRR = 0.2 exactly. The equal error rate is 20%, at threshold 0.45. Note it corresponds to a threshold that is worse for both metrics than 0.55 — EER is a summary of the curve, not a recommended operating point.

Accuracy, if you insist: at t = 0.45 we accept 8 genuine (8 correct) and reject 8 impostors (8 correct), so 16/20 = 80%. That single number hides that we let in two strangers, which for a door lock is the only fact anybody cares about.

Why LFW cannot tell ArcFace from CosFace

LFW's standard protocol has 6,000 pairs: 3,000 genuine and 3,000 impostor. Two consequences follow immediately.

Resolution. The smallest nonzero FAR you can measure is 1/3,000 = 3.3×10⁻⁴. Any question about behaviour at FAR = 10⁻⁶ — the operating point of an actual access-control system — is unanswerable on LFW in principle, not for want of a better model.

Noise. At 99.83% accuracy the model makes 0.0017 × 6,000 ≈ 10 errors in total. A rival at 99.80% makes 12. The difference is two pairs. Rerun with a different fold split, or fix two mislabelled pairs in the benchmark (LFW is known to contain a handful), and the ranking flips. Differences in the third decimal place on LFW are noise, and papers that rank methods by them are ranking noise.

This is why the field moved to IJB-B and IJB-C. IJB-C's 1:1 protocol contains on the order of 19,600 genuine and 15.6 million impostor comparisons, which supports measurement down to FAR = 10⁻⁶ and separates methods that are indistinguishable on LFW.

How many pairs do you need? A Poisson argument

Suppose the true FAR at your threshold is 10⁻⁶ and you test with exactly 10⁶ impostor pairs. The expected number of false accepts is 10⁶ × 10⁻⁶ = 1. The observed count is Poisson with mean 1: you will see 0 about 37% of the time, 1 about 37%, 2 about 18%, 3 or more about 8%. Your FAR estimate is therefore 0, 10⁻⁶, 2×10⁻⁶ or worse, essentially at random.

The relative standard error of a count k is 1/√k. To pin FAR down to ±10% you need about k = 100 false accepts, so:

And impostor pairs are cheap only if identities are plentiful: with G identities and one template each, you get G(G−1)/2 impostor pairs, so 10⁸ pairs needs about 14,150 identities. This is the arithmetic behind every "we need a bigger benchmark" paper of the last decade.

Identification is harder than verification, and the arithmetic says how much

Now the 1:N task. A probe is compared against a gallery of N templates, and the system returns a ranked list. Two measurements:

Closed-set: the CMC curve. Rank-k identification rate is the fraction of probes whose true identity appears in the top k. Rank-1 is the headline; the curve to rank-20 tells you whether the failures are near-misses (the curve shoots up) or catastrophic (it stays flat). A flat CMC curve means the errors are not "the model was unsure between two people" but "the correct person is nowhere near the top", which usually indicates a data problem rather than a model problem.

Open-set: false positives compound with N. This is the part that gets systems deployed into disaster. If each comparison has an independent false-match probability FMR, then the probability that at least one of the N gallery entries falsely matches is

Put in a gallery of one million and an excellent per-comparison FMR of 10⁻⁶:

Nearly two thirds of searches against a one-million gallery return at least one wrong person — using a per-comparison error rate that is genuinely excellent. To get that down to a 1% chance of any false match you need

a hundred times stricter than the FMR you started with — and, by the Poisson argument above, a rate you would need 10¹⁰ impostor comparisons to even measure.

The demographic differential, computed before we discuss it

That last slider deserves its own arithmetic, because Chapter 8's argument depends on it and it is usually asserted rather than derived.

Model impostor scores for two groups as Gaussians with the same spread σ = 0.05 but slightly different means: group B at 0, group A at 0.05 — one standard deviation higher, a shift so small you would never see it on a histogram. Set a single global threshold at t = 0.25. The false match rate for each group is the Gaussian tail above t:

A one-sigma shift in the mean impostor score — the kind of difference produced by, say, a training set with fewer examples of a group, or a face detector calibrated on different skin tones — becomes a hundredfold difference in false match rate at a shared threshold. Not because the tail is unfair, but because it is exponential. This is the mechanism behind the demographic differentials that NIST measured across the industry, and it is why "our model has the same accuracy for everyone" is not the claim that matters.

A CMC curve, also by hand

Ten probes searched against a gallery of five identities. For each probe, record the rank at which the true identity appears:

The cumulative match characteristic at rank k is the fraction of probes whose true identity appears at rank k or better. Count them:

CMC is non-decreasing by construction, and its shape is the diagnostic. A curve that jumps from 0.70 to 0.95 by rank 3 says the model has the right neighbourhood and is losing on fine discrimination — more capacity or a bigger margin will help. A curve that crawls says the failures are not near-misses at all: the correct identity is nowhere near the top, which points at the front end (a bad crop, a wrong pose) rather than the embedding.

Templates: many photos, one vector

Real systems rarely have one image per person. Enrolment gives you several; a video gives you hundreds. The standard move is to aggregate into a template: average the embeddings, then renormalise.

The arithmetic of why this works is worth doing. Model each embedding as the identity's true direction plus isotropic noise with per-component standard deviation σ. Averaging k independent samples divides the noise standard deviation by √k while leaving the signal unchanged, so the angular error to the true direction shrinks by the same factor:

Two details decide whether you get that √k. Average before you normalise, not after: the pre-normalisation norm carries quality information (Chapter 3), so summing raw features quietly downweights the blurry frames — a free quality-weighted average. And the frames must be independent: 100 consecutive frames of a video are not 100 samples, they are approximately one sample with 100 copies of the same noise, so the √k never materialises. This is also why "we evaluated on video and got 99.9%" claims deserve suspicion.

The realisation — evaluation in code

python — TAR at a target FAR, the function you will write a hundred timesimport numpy as np

def tar_at_far(gen_scores, imp_scores, target_far=1e-4):
    # The threshold is set by the IMPOSTORS only — never by the genuine pairs.
    imp = np.sort(imp_scores)[::-1]            # descending
    k   = int(np.floor(target_far * len(imp)))     # how many false accepts we allow
    if k == 0:
        raise ValueError(
            f"Only {len(imp)} impostor pairs — cannot resolve FAR={target_far}. "
            "You need at least 1/FAR pairs to see one false accept, and ~100/FAR to measure it.")
    thr = imp[k - 1]                              # the k-th highest impostor score
    tar = (np.asarray(gen_scores) >= thr).mean()
    return tar, thr, k                          # ALWAYS return k — it is your error bar

# Report per demographic group at the SHARED threshold, not per-group thresholds:
for g, idx in groups.items():
    fmr_g = (imp_scores[idx] >= thr).mean()
    print(g, "FMR", fmr_g, "n_pairs", len(idx))     # n_pairs is the honesty column

Two details in that snippet are the difference between an evaluation and a decoration. The threshold is derived from impostor scores alone, so it never peeks at the genuine set. And k — the number of false accepts your threshold actually admits — is returned, because k = 3 means your TAR figure has an error bar of ±58% and nobody should quote its third digit.

Chapter 7: The Same Recipe, Everywhere Else

Nothing in Chapters 1 through 5 mentioned faces. Go back and check: the derivations used "identity", "positive", "negative", "proxy" and "angle". Faces were the motivating example and the benchmark, not an assumption. So the machinery transfers, and it transferred fast — usually within a year of each paper, and usually with the same hyperparameters.

The transfer table

Speaker verification: the cleanest transplant in machine learning

Speaker verification has exactly the structure of face verification: enrol a speaker from a few seconds of audio, then accept or reject future utterances. The field went through the same sequence with a delay of a year or two — i-vectors and PLDA (hand-designed), then x-vectors with a softmax classification bottleneck (the "take an intermediate layer" trick FaceNet criticised), then triplet losses, then AAM-softmax.

The migration is unusually literal: AAM-softmax is ArcFace's formula, with the same two hyperparameters, differently tuned. The typical speaker setting is s = 30 and m = 0.2 rather than s = 64 and m = 0.5, and Chapter 5 predicts both changes. Speaker corpora have thousands of speakers rather than 85,000 identities, so the floor formula demands a much smaller s: with C = 6,000 and P_W = 0.9, s ≥ ln(5,999 × 9) = ln(53,991) = 10.9, and 30 gives ample headroom. And utterance embeddings have larger intra-class variance than face crops, so a smaller margin is what the data can actually satisfy.

The evaluation transferred too, including its problems: VoxCeleb reports EER and minimum detection cost, systems are compared at the third decimal place, and the same "is this difference real" question applies.

Visual place recognition, and the loop back to this site's veanors

Place recognition asks whether two photographs show the same physical location, with the same open-set structure — a robot will visit places absent from its training set. NetVLAD's contribution was a differentiable aggregation layer plus a weakly supervised triplet loss, which handles the fact that GPS gives you "these two photos were taken near each other" but not "these two photos show the same thing". Its positives are therefore potential positives, and the loss takes the best one — a mining problem with an extra layer of uncertainty on top of Chapter 2's.

Then the same reformulation happened as in faces: CosPlace and EigenPlaces recast VPR as classification with a margin over discretised geographic cells, precisely to escape the sampler. If Chapters 2 and 3 felt familiar while reading those papers, that is why.

A port, worked all the way through: speaker verification

Do not take "it transfers" on faith. Here is the whole port, decision by decision, for a system that verifies a speaker from a few seconds of audio.

Why s = 30 and not 64. Apply Chapter 5's floor formula with a speaker corpus of about 6,000 identities and a target confidence of 0.9:

Roughly a third of the 13.56 that 85,742 face identities demanded, because the floor grows only with the logarithm of the class count. Three times the floor gives about 33, and the field settled on 30. The face value of 64 is about 4.7× its floor; 30 is 2.8× its own. Both are in the same regime, which is the point: the number is derived, not inherited.

Why m = 0.2 and not 0.5. The margin is a claim about how tightly a class can be squeezed, and that is a property of the data. A three-second utterance varies with the sentence spoken, the room, the microphone, the speaker's health and their mood; an aligned face crop has already had pose, scale and rotation removed by the front end. Ask for 28.65° of clearance from utterances and a large fraction of the training set becomes unsatisfiable — the loss plateaus high and the embedding gets worse, which is Chapter 5's third failure mode arriving in a different domain.

Person re-identification and dense retrieval — the mining story repeats

Two more instances, both of which rediscovered Chapter 2 rather than Chapter 5.

Person re-ID matches a pedestrian across non-overlapping cameras. Identities number in the low thousands, images are noisy crops from surveillance video, and the standard recipe is batch-hard triplet with P = 18 identities × K = 4 images, often summed with a margin-softmax head. Why triplets survived here when faces abandoned them: with only a few thousand identities the proxy table is tiny and the classification signal is weak, while the batch-hard bound keeps mining honest at B = 72. Small C is the regime where triplets remain competitive.

Dense text retrieval embeds queries and documents into a shared space and searches by cosine — the same interface again. It began with in-batch negatives (the batch is the sampler), and then discovered that in-batch negatives become too easy as the model improves, which is precisely Chapter 2's self-defeating loop. The field's answer was to mine hard negatives from the corpus with the current model and refresh them periodically — ANCE, RocketQA and their descendants — which is FaceNet's offline mining, with the staleness problem and all. Read those papers with Chapter 2 in hand and there are no surprises left in them.

The unifying view: alignment and uniformity

Every loss in this lesson can be read as two forces, and Wang and Isola's 2020 analysis of contrastive representation learning gives them names.

Collapse, from Chapter 1, is perfect alignment with zero uniformity. A model that spreads everything uniformly with no alignment is a random projection. Every method in the canon is a different way of holding the two in balance, and our alignment and uniformity veanor works through the formal version.

Visual place recognition, worked

VPR is the port where the "what is an identity?" question is genuinely hard, which makes it the most instructive one.

A place is not a discrete object. Two photographs taken five metres apart show the same place; two taken five metres apart around a corner do not. GPS gives you proximity, not visual overlap. NetVLAD's answer was weak supervision: treat every geographically-near image as a potential positive, and let the loss pick the best one:

The min over candidate positives is the new part — a mining problem inside the positive set, on top of Chapter 2's mining problem in the negative set. It works, and it inherits every difficulty this lesson catalogued, doubled.

Then the field did what faces did. CosPlace and EigenPlaces discretise the map into cells — a few metres of position crossed with a heading bin — call each cell a class, and train a margin-softmax over tens of thousands of such classes. The sampler disappears; training becomes a classification run; the deployed artifact is still a descriptor compared by cosine. One extra wrinkle is worth noting because it has no analogue in faces: adjacent cells are genuinely similar, so pushing them apart with a hard margin is fighting the data. The fix is to partition the cells into groups whose members are far apart, and train on one group at a time, so no batch ever contains two neighbouring cells. That is a domain-specific repair to a domain-agnostic recipe, and it is the shape most good ports take.

The skeleton, once, for any domain

python — the whole recipe, domain-agnostic# 1. An encoder that turns your thing into a vector. Faces: a ResNet on a
#    112x112 crop. Speech: a TDNN with attentive pooling. Places: a CNN with
#    a VLAD or GeM aggregation. Text: a transformer with a pooled token.
emb = F.normalize(encoder(x), dim=1)              # (B, d) on the unit sphere

# 2. A margin head over whatever you decided a class is.
loss = arcface_head(emb, class_ids)                # scalar

# 3. Deployment: delete the head. Store vectors. Compare with a dot product.
gallery = F.normalize(encoder(enrol_batch), dim=1)   # (N, d) — the database
scores  = F.normalize(encoder(probe), dim=1) @ gallery.T
# scores >= threshold  ->  verification
# scores.argmax()      ->  identification
# graph over scores    ->  clustering

# 4. Evaluate with TAR@FAR on identities the encoder has never seen,
#    reported per subgroup at a SHARED threshold, with the pair counts.

Four steps, and only step 1 is domain knowledge. That is what it means to say the canon transferred: everything below the encoder became a solved, shared substrate, and the research moved to the front end and to the data.

What actually changes when you port this

Chapter 8: The Honest Chapter

Every other technology in this lesson's family — speaker embeddings, place recognition, product retrieval — is a tool. Face recognition is a tool that is also a political object, and a lesson that teaches you to build one without teaching you when not to has not finished the job. This chapter is not a disclaimer bolted on the end. It is the part where the arithmetic of Chapters 0 and 6 gets pointed at people.

Start from the properties we deliberately engineered

Go back through the lesson and list what we worked to achieve. Then read the list again as a threat model, because it is the same list.

The bias findings, stated precisely

"Face recognition is biased" is true and too vague to act on. The measured findings are specific, and the specificity is what makes them useful.

NIST FRVT Part 3: Demographic Effects (NISTIR 8280, December 2019) evaluated 189 algorithms from 99 developers on large operational datasets. Its findings, in the report's own framing:

• In one-to-one verification, false positive differentials of a factor of 10 to 100 were common across demographic groups — with the highest false match rates, for many US-developed algorithms, on West and East African and East Asian faces relative to Eastern European faces.
• False positives were generally higher for women than men, and elevated for the elderly and for children.
• In one-to-many search against a US mugshot database, some algorithms showed elevated false positive rates for Black women specifically.
• The differential is not universal. Algorithms developed in China did not show the elevated false match rate on East Asian faces. Same architectures, same losses — different training data. That single comparison is the most informative result in the report, because it locates the cause.

Gender Shades (Buolamwini and Gebru, 2018) audited commercial gender classification from face images and found error rates ranging from 0.8% on lighter-skinned men to 34.7% on darker-skinned women. Be precise about this one: gender classification is a different task from identity verification, and quoting it as a face-recognition error rate is a common overreach. It belongs here because it demonstrated the same root cause — benchmark and training sets whose demographic composition nobody had audited — and because it is what made the industry start measuring.

Why the loss functions in this lesson cannot fix it

Chapter 6 already did this derivation, and it is worth restating as the mechanism rather than an accusation.

The margin loss enforces its constraint per class, identically for all classes. It is scrupulously fair in that narrow sense. But the resulting embedding's score distributions need not be identical across groups: if a group is under-represented, or if the face detector and landmark aligner upstream were tuned on a different distribution, that group's impostor scores can sit slightly higher. And Chapter 6 computed what "slightly" does:

at a shared threshold. The exponential tail is an amplifier: distribution differences too small to see become error-rate differences of two orders of magnitude. Three consequences follow, and all three are actionable:

The failure chain is not the model

A 1:N search returns a ranked list of similarities. Follow what happens to that list in a real process.

At each step a probabilistic ranking is laundered into a categorical assertion, and by step 4 nothing in the paperwork records that the original object had a confidence attached to it. There are publicly documented wrongful arrests in the United States following face-recognition leads — among them Robert Williams and Porcha Woodruff in Detroit and Nijeer Parks in New Jersey — all of Black Americans, in cases where the technical match was one input among several failures.

The datasets, and what happened to them

The models in this lesson were trained on data that in several cases no longer exists, for reasons worth knowing.

Note the asymmetry that matters: the datasets were retracted; the models trained on them were not. Weights trained on MS-Celeb-1M are still downloaded thousands of times a week. "We removed the dataset" is not a remedy when the artifact that encodes it has already been distributed.

The template is not anonymous, and it cannot be reissued

A common defence of storing face embeddings is that they are "just numbers" — irreversible, unlike a photograph. Two facts make that defence weaker than it sounds, and both are technical rather than political.

Embeddings can be inverted. Given a target vector and access to a face model, you can optimise an image (or train a decoder) so that its embedding matches. The reconstructions are not photographs of the person, but they are recognisably that person — enough for a human or another matcher. This is exactly the same attack surface as text-embedding inversion, and our embedding security lesson works through the general case. The practical consequence: a leaked template database should be treated as a leaked photo database, not as a leaked hash.

A face is not a password. Chapter 0's convenience — no enrolment cost, works on strangers, stable across years — is the same property that makes a breach permanent. A leaked password is rotated in seconds. A leaked template cannot be, because the credential is your face, you cannot change it, and it is the same credential at every other system that ever enrols you. Biometrics have no revocation story, and every deployment inherits that.

Where the line is drawn in law

Briefly, and with the caveat that this moves fast — check the current text before relying on any of it.

• EU AI Act (in force from 2024, prohibitions applying from 2025): bans untargeted scraping of facial images from the internet or CCTV to build recognition databases, and bans real-time remote biometric identification in publicly accessible spaces for law enforcement, with narrow, judicially authorised exceptions.
• Illinois BIPA: requires informed written consent before collecting a biometric identifier, and carries a private right of action — the reason for the 2021 Meta settlement and the 2022 Clearview AI settlement restricting sales to most private entities in the US.
• Several US cities and states restrict government use; multiple jurisdictions have moved in both directions since. There is no stable global answer, which is itself a design constraint: a product that is legal in one market may be prohibited in another with the same code.

What good practice looks like when the use is legitimate

Phone unlock is the honest counter-example, and it is instructive precisely because it is built to eliminate almost every property in the threat-model table.

None of that is about the loss function. All of it is architecture and policy, decided before any model was trained.

A checklist for the decision you may actually face

You will more likely be asked to build one of these than to decide whether one should exist. Here is the shortest honest list of conditions under which the answer should be no.

Chapter 9: The Canon, and a Cheat Sheet

Ten years, one idea, and a long argument about where to put a constant. Here is the whole lineage in one table, then the numbers, then what to read next, then a recipe you can run this weekend.

The lineage

The same system, two implementations

If you have to choose one today, choose the right-hand column. But the comparison is more useful than the verdict, because it shows exactly which parts of a system a loss function decides.

Note what did not change: the embedding, L2 normalisation, cosine similarity at inference, the threshold, and every line of the evaluation code. Ten years of progress happened entirely inside the training objective, behind an interface that FaceNet fixed in 2015 and nobody has had a reason to move since.

The cheat sheet

Where to go from here

Build it yourself — the weekend recipe

The bugs that eat a weekend

Collected from the chapters, in the order you are likely to hit them.

References worth reading next

1. Schroff, F., Kalenichenko, D., Philbin, J. "FaceNet: A Unified Embedding for Face Recognition and Clustering," CVPR 2015 — arXiv:1503.03832. Chapters 1 and 2.
2. Deng, J., Guo, J., Xue, N., Zafeiriou, S. "ArcFace: Additive Angular Margin Loss for Deep Face Recognition," CVPR 2019 — arXiv:1801.07698. Chapters 4 and 5.
3. Wang, H., Wang, Y., Zhou, Z., Ji, X., Gong, D., Zhou, J., Li, Z., Liu, W. "CosFace: Large Margin Cosine Loss for Deep Face Recognition," CVPR 2018 — arXiv:1801.09414. The additive cosine margin and the bound on s.
4. Liu, W., Wen, Y., Yu, Z., Li, M., Raj, B., Song, L. "SphereFace: Deep Hypersphere Embedding for Face Recognition," CVPR 2017 — arXiv:1704.08063. The multiplicative ancestor, and why annealing was needed.
5. Wang, F., Xiang, X., Cheng, J., Yuille, A. "NormFace: L2 Hypersphere Embedding for Face Verification," ACM MM 2017 — arXiv:1704.06369. Chapter 3's normalisation and the scale.
6. Movshovitz-Attias, Y., Toshev, A., Leung, T., Ioffe, S., Singh, S. "No Fuss Distance Metric Learning using Proxies," ICCV 2017 — arXiv:1703.07464. The proxy idea that makes Chapter 3 inevitable.
7. Hermans, A., Beyer, L., Leibe, B. "In Defense of the Triplet Loss for Person Re-Identification," 2017 — arXiv:1703.07737. Batch-hard mining and the soft margin.
8. Wu, C.-Y., Manmatha, R., Smola, A., Krähenbühl, P. "Sampling Matters in Deep Embedding Learning," ICCV 2017 — arXiv:1706.07567. The most careful treatment of Chapter 2's problem.
9. Wang, T., Isola, P. "Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere," ICML 2020 — arXiv:2005.10242. Chapter 7's unifying frame.
10. Meng, Q. et al. "MagFace: A Universal Representation for Face Recognition and Quality Assessment," CVPR 2021 — arXiv:2103.06627; Kim, M., Jain, A., Liu, X. "AdaFace: Quality Adaptive Margin," CVPR 2022 — arXiv:2204.00964. Feature norm, used on purpose.
11. An, X. et al. "Partial FC: Training 10 Million Identities on a Single Machine," 2020 — arXiv:2010.05222. What to do when the C×d proxy table stops fitting.
12. Grother, P., Ngan, M., Hanaoka, K. "Face Recognition Vendor Test Part 3: Demographic Effects," NISTIR 8280, NIST, 2019. Chapter 8's numbers, at their source.
13. Buolamwini, J., Gebru, T. "Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification," FAT* 2018. The audit that started the measurement.
14. Maze, B. et al. "IARPA Janus Benchmark-C: Face Dataset and Protocol," ICB 2018. The benchmark that can resolve what LFW cannot.