SimCSE: Sentence Embeddings from Dropout

Chapter 1: InfoNCE, Symbol by Symbol

Before we can talk about where positive pairs come from, we need to know what we are going to do with them. This chapter builds the contrastive objective from nothing, one repair at a time, until we arrive at the exact equation SimCSE optimises. Every symbol will be named, the gradient will be derived, and we will run a small batch through by hand.

Attempt 1: just pull positives together

We want an encoder f such that a sentence x and its paraphrase x⁺ get nearby vectors. The most direct way to say that is to write it as a loss and minimise it:

Where N is the number of pairs in the batch, f is the encoder (BERT plus whatever pooling), and the double bars are the Euclidean norm. Lower means the pair is closer. Minimise it and the pairs get closer. Done?

No. There is a solution with loss exactly zero that requires no understanding of language at all:

Map everything to the origin. Every pair distance is zero. The loss is zero. It is the global minimum, it is trivially reachable by gradient descent, and it is completely useless. This is representational collapse — the failure mode where a model satisfies an objective by discarding all information rather than by organising it.

Watch collapse happen, in numbers

"Gradient descent finds the constant function" deserves a demonstration rather than an assertion, and for a linear encoder we can do it exactly.

Let the encoder be a matrix: f(x) = A x. Write δ_i = x_i − x_i⁺ for the difference between a pair. Then

Differentiate: ∂L/∂A = 2AS. So one gradient step with learning rate η gives

The update multiplies A by a fixed contraction. Suppose S = diag(0.5, 0.2) and η = 0.5, so I − 2ηS = diag(0.5, 0.8). After ten steps every column of A has been scaled by 0.5¹⁰ = 0.00098 in the first direction and 0.8¹⁰ = 0.107 in the second. After a hundred steps, 7.9 × 10⁻³¹ and 2.0 × 10⁻¹⁰.

The encoder is being annihilated, exponentially, in every direction where the data has any variation at all. And the loss is dropping the whole time — monotonically, smoothly, exactly as a healthy training curve looks. This is why collapse is so dangerous in practice: the loss curve gives you no warning whatsoever.

Attempt 2: add something to push against

The repair is to make the objective relative. It should not be enough for the positive pair to be close; it must be closer than the alternatives. Now the constant function fails, because if everything is at the origin, the positive is not closer than anything — it ties with all of them.

There are two classical ways to write "closer than the alternatives". The first is a triplet margin loss:

Read it as: the positive must be closer than the negative by at least a margin m; if it already is, the loss is zero and nothing happens. This works — it is what SBERT and most of the 2015–2019 metric-learning literature used — and it has two properties that InfoNCE fixes.

The "exactly zero" row is the one that decides it. With a margin of 0.2 and a batch where 90% of triplets are already separated by more than 0.2, ninety percent of your compute produces no gradient at all. Teams spend enormous effort on mining pipelines to fix this. InfoNCE makes the mining a property of the loss instead of a property of the data loader, which is why it took over.

The second way, which is what SimCSE uses, is to turn the whole thing into a multiple-choice test.

Attempt 3: contrastive learning as an N-way classification

Here is the reframing, and it is the single most useful mental model in this lesson.

Take a batch of N pairs: (x₁, x₁⁺), (x₂, x₂⁺), …, (x_N, x_N⁺). Now scramble the second elements and hand the model a quiz:

That is an ordinary N-way classification problem, and we already know how to train those: produce a score for each candidate, softmax the scores into a distribution, take the cross-entropy against the correct index. The labels are free — they are just 0, 1, 2, ..., N-1, the identity permutation. No human wrote them.

This is the entire idea. The in-batch negatives trick means every other example in the batch serves as a negative for every example, so a batch of size N yields N positive constraints and N(N−1) negative constraints at the cost of N forward passes. Negatives are free.

The score function: why cosine, and what it costs

We need a number saying how compatible two embeddings are. The obvious choice is the dot product h_i^Th_j. It is the wrong choice, for a reason worth seeing concretely.

Suppose the encoder produces h₁ = (3, 4) for a sentence and h₁⁺ = (6, 8) for its paraphrase. Those point in exactly the same direction — the second is the first doubled. Semantically the encoder has done a perfect job. The dot product is 3(6) + 4(8) = 18 + 32 = 50. Now take a genuinely unrelated sentence whose embedding happens to be large: h₂⁺ = (20, 1). The dot product with h₁ is 3(20) + 4(1) = 64. The unrelated sentence wins, purely because it is long.

Worse, the model can exploit this. Under a raw dot product, one way to reduce the loss is to grow ‖h‖ for confident examples, which inflates every score in that row. The optimiser will happily spend capacity on a magnitude channel that has nothing to do with meaning.

The fix is cosine similarity: divide out the lengths, keeping only direction.

Check the example: ‖(3,4)‖ = 5, ‖(6,8)‖ = 10, so the cosine is 50/(5×10) = 1.000, correctly reporting a perfect match. For (20,1): ‖(20,1)‖ = √401 = 20.025, so the cosine is 64/(5 × 20.025) = 64/100.12 = 0.639. Order restored.

Equivalently: L2-normalise every embedding onto the unit sphere first, then the plain dot product is the cosine. That is how it is implemented, and it is the reason you will see F.normalize in every contrastive codebase.

The objective, assembled

Put the pieces together. For example i in a batch of N:

This is the InfoNCE loss (Noise-Contrastive Estimation with an information-theoretic reading, from van den Oord et al. 2018), and it is exactly the loss SimCSE minimises. Every symbol:

Two details people get wrong. First, the correct pair appears in the denominator as well as the numerator — the sum is over all N, not the N−1 negatives. That is what makes the ratio a genuine probability and bounds the loss below by 0. Second, the loss is written per row: row i asks "which key belongs to query i". The unsupervised SimCSE objective as published is one-directional, exactly like this. (CLIP-style symmetric losses also add the transpose direction; Chapter 5 says when that matters.)

What the floor and the ceiling mean

Two reference values make the loss readable at a glance.

The chance level is ln N. If the model has no idea — all N scores equal — then every probability is 1/N, and ℓ_i = −log(1/N) = log N. For N = 64 that is ln 64 = 4.159. If your contrastive loss is sitting at 4.16 with a batch of 64, your model is guessing, and the most common cause by far is a mis-set temperature.

The floor is 0, reached when p_ii = 1, i.e. the positive score is infinitely far above all others. With cosine scores bounded in [−1, 1] and a finite temperature, this is unreachable: the best possible row has sim = 1 on the diagonal and −1 everywhere else, giving

At τ = 0.05 and N = 64: e⁻⁴⁰ = 4.25 × 10⁻¹⁸, times 63, gives 2.7 × 10⁻¹⁶. So the practical floor is zero to sixteen decimal places. At τ = 1 the same expression gives log(1 + 63 × 0.1353) = log(9.525) = 2.254 — the loss literally cannot go below 2.25 no matter how perfect the encoder is. That single calculation is the most compelling argument for a small temperature there is.

Worked example: one batch of three, by hand

Numbers make this concrete. Suppose a batch of N = 3 produces the following cosine similarity matrix, where row i is query i and column j is key j:

Take row 1 at τ = 1 first, so the arithmetic is transparent. Logits are just the cosines: 0.72, 0.54, 0.19.

Sum: 2.0544 + 1.7160 + 1.2092 = 4.9796. Probabilities:

Loss for row 1: ℓ₁ = −ln(0.4126) = 0.8853.

Compare to chance, ln 3 = 1.0986. The model is only 0.21 nats better than guessing, even though the correct answer has the highest cosine by a comfortable 0.18 margin. That is the flatness problem: with τ = 1 a decisive geometric win translates into a feeble probabilistic one.

Now the same row at τ = 0.05, multiplying every cosine by 20. Logits: 14.4, 10.8, 3.8. Subtract the max (a standard numerical-stability step that leaves the softmax unchanged) to get 0, −3.6, −10.6:

Sum: 1.0273488. So p₁₁ = 1/1.0273488 = 0.97338, and ℓ₁ = ln(1.0273488) = 0.02698.

Same geometry. Same encoder. Thirty-three times less loss, purely from the divisor. Chapter 5 unpacks what that does to the gradients — because "less loss" is not automatically "better training", and the trade-off is sharp.

The gradient, derived

To understand any loss you must know what it does to the parameters. Write s_ij = sim(h_i, h_j⁺) and the logit z_ij = s_ij/τ. Then

Differentiate with respect to a single logit z_ik. The first term contributes −1 when k = i and 0 otherwise. The second term is the log-sum-exp, whose derivative is the softmax:

and therefore, chaining through the temperature,

This little expression is worth sitting with, because it tells you the entire training dynamics.

Two consequences that people pay for when they forget them. First, the total force is balanced: ∑_k ∂ℓ_i/∂z_ik = ∑_k p_ik − 1 = 0. The attraction on the positive exactly equals the sum of repulsions on the negatives. Second, InfoNCE performs automatic hard-negative mining: you never have to select difficult negatives, because the softmax weights them by exactly how threatening they currently are.

Check that on the worked numbers. At τ = 1, row 1's probabilities were (0.4126, 0.3446, 0.2428). The attraction weight is 1 − 0.4126 = 0.5874, and the two repulsions are 0.3446 and 0.2428 — the harder negative gets 1.42× the push of the easier one. At τ = 0.05, the probabilities were (0.97338, 0.026596, 0.0000241); the ratio between the two repulsions is now 0.026596/0.0000241 = 1103. Sharpening the softmax does not just shrink the loss; it re-allocates virtually all of the repulsive force onto the single hardest negative. Hold that thought for Chapter 5.

The realisation: shapes and code

Here is the whole objective as it actually executes, with every tensor shape named. This is the code SimCSE runs; everything else in the paper is about what you feed it.

python — the contrastive core, end to endimport torch, torch.nn.functional as F

# z1, z2 are the two views of the SAME N sentences.
# z1: (N, 768)   the "query" view h_i
# z2: (N, 768)   the "key"   view h_i^+   (row i is row i's partner)

z1 = F.normalize(z1, dim=-1)          # (N, 768), every row now has norm 1
z2 = F.normalize(z2, dim=-1)          # (N, 768)

sim = z1 @ z2.T                          # (N, N)  sim[i,j] = cos(h_i, h_j^+)
logits = sim / 0.05                     # (N, N)  temperature = 0.05 -> multiply by 20

labels = torch.arange(z1.shape[0], device=z1.device)   # (N,) = [0,1,2,...,N-1]
loss = F.cross_entropy(logits, labels)                # scalar

Six lines. The one that carries the whole idea is the labels line: the supervision is arange(N). No human produced it; it is a consequence of how the batch was assembled. Everything creative in this paper happens upstream of these six lines, in the question of how z1 and z2 came to be two views of the same sentence.

A caution about what "loss went down" proves

Because the labels come from the batch construction rather than from the world, a falling contrastive loss proves only that the model can tell batch members apart. It is entirely possible to drive the loss to 0.001 while producing embeddings that are useless for STS — if the model discovers a shortcut feature (sentence length, first token, a punctuation quirk) that identifies each batch member without touching meaning.

And SimCSE is the method where that worry bites hardest, because its two views are the same token sequence. Sentence length, punctuation, rare tokens, casing — every surface feature is preserved exactly across the positive pair. A model that learned nothing but "match the sentence with the same number of tokens" would drive this loss down beautifully. This is the strongest objection to the whole method, and you should be holding it against the paper right now.

The answer is an ablation we will meet in Chapter 2, and it is clean. If surface identity were what the model exploits, then the variant with dropout switched off — where the two views are not merely surface-identical but bit-identical, making every shortcut maximally available — ought to do at least as well. It does not. It scores 71.1 on STS-B dev against 82.5 with dropout at p = 0.1. The shortcut is fully available in both settings; only one of them learns anything worth having. Whatever dropout supplies, it is not a shortcut, and Chapter 3 will name it precisely.

Chapter 2: Dropout Is the Augmentation

Chapter 1 built the machine and left a hole in it: h_i⁺. The loss needs a second view of every sentence, and Chapter 0 established that editing tokens to produce one is a losing game. This chapter fills the hole.

What dropout actually is

Dropout is a regularisation technique from 2014: during training, each activation in a layer is independently zeroed with probability p. Nothing more exotic than that. Let u be a layer's output vector and m a random binary mask with m_k ~ Bernoulli(1 − p). Then the naive version is

This has a problem you can see with one line of arithmetic. The expected value of the output has shrunk: E[h_k] = E[m_k] · u_k = (1 − p) u_k. At p = 0.1, every activation is 10% smaller in expectation than it would be without dropout. So a network trained with dropout and evaluated without it sees systematically inflated inputs at every layer, and the inflation compounds through depth.

The universal fix is inverted dropout: scale the survivors up during training so the expectation is preserved, and do nothing at all at test time.

Check it at p = 0.1: survivors are multiplied by 1/0.9 = 1.111. Nine tenths of the units survive and are 11.1% larger; one tenth is zero. The mean is unchanged; the variance is not. That injected variance is the regularisation — and, for our purposes, it is the augmentation.

Where the masks actually live inside BERT

"BERT uses dropout" is too vague to reason about. Here is the real inventory for BERT-base, all at p = 0.1 by default in the released config:

Count the Bernoulli draws in one forward pass of a 32-token sentence through the 12 layers:

Three quarters of a million random bits are sampled every time a sentence goes through the model. The probability that two forward passes draw the same mask is, for practical purposes, zero. So the following statement — which sounds like a bug report — is simply true:

The unsupervised SimCSE objective

Write h_i^z = f_θ(x_i, z) for the encoder's output on sentence x_i under dropout mask z. Sample two independent masks, z_i and z_i′. Then the objective is Chapter 1's InfoNCE with the two views defined this way:

That is the entire unsupervised method. The only difference from a loss you could have written in 2018 is the superscripts: the positive pair is one sentence under two masks, and the negatives are the other sentences in the batch under their own masks.

The realisation: how you actually get two masks

Here is the part that is genuinely charming. SimCSE does not run the model twice. It duplicates each sentence inside the batch and runs one forward pass, because dropout masks are sampled per element — two copies of the same row get independent masks automatically.

python — the four lines that make the positive pair# batch: N sentences, tokenised to length T
input_ids = batch["input_ids"]                    # (N, T)

# 1. duplicate: each sentence appears twice, adjacent
input_ids = input_ids.unsqueeze(1).repeat(1, 2, 1)   # (N, 2, T)
input_ids = input_ids.view(-1, input_ids.size(-1))     # (2N, T)  <- flatten

# 2. ONE forward pass in train mode. Dropout is sampled per row,
#    so rows 2i and 2i+1 (the same tokens!) get DIFFERENT masks.
out = encoder(input_ids).last_hidden_state[:, 0]     # (2N, 768)  [CLS] token
out = mlp(out)                                        # (2N, 768)  pooler: Linear + Tanh

# 3. split back into the two views
out = out.view(-1, 2, out.size(-1))                  # (N, 2, 768)
z1, z2 = out[:, 0], out[:, 1]                       # (N, 768) each

# 4. Chapter 1's six lines, unchanged.
loss = F.cross_entropy(F.normalize(z1,dim=-1) @ F.normalize(z2,dim=-1).T / 0.05,
                       torch.arange(z1.size(0), device=z1.device))

Notice what is absent: there is no augmentation function, no synonym dictionary, no translation model, no data preprocessing step. The augmentation is a side effect of model.train(). The compute cost is one extra forward pass per sentence — the batch is 2N rows instead of N — and nothing else.

See the twins

Play with it until three things are obvious. At p = 0 the two views are identical and the cosine is exactly 1.000, forever, no matter how many times you resample. At p = 0.5 the views are wildly different and the cosine wanders down toward 0.5 — the "positive pair" is now barely a pair at all. At p = 0.1 the cosine sits high but never at 1: the twins are close, and reliably not identical. That narrow band is where the method lives.

How far apart are the twins? A closed form

The sim shows the twin cosine hovering near 0.9 at p = 0.1 and near 0.5 at p = 0.5. Those numbers are not accidents. For our simplified single-site dropout there is an exact answer, and it is startlingly clean.

Write the two views component-wise, with m^A and m^B independent Bernoulli(1−p) masks:

The numerator. A unit contributes to the dot product only if it survived both masks, which happens with probability (1−p)² by independence. The 1/(1−p) factors contribute 1/(1−p)². They cancel exactly:

The denominator. Each norm involves only one mask, so only one factor of (1−p) survives against the squared rescaling:

For large d these quantities concentrate tightly around their means, so ‖h^A‖ ‖h^B‖ ≈ ‖u‖²/(1−p), and

The expected cosine between a sentence and its dropout twin is one minus the dropout rate. Check it against a Monte-Carlo simulation over 4,000 random vectors:

One honest correction before you over-apply it. Real BERT applies dropout at 37 sites inside the network, and each perturbation is partially absorbed by the LayerNorms and residual connections that follow it — the model was trained to be robust to exactly this noise. So the observed twin cosine in a real encoder is higher than 1−p, not equal to it. What survives is the monotone relationship and the shape of the curve, which is all the argument needs.

The ablation that settles it

An idea this cheap deserves scepticism, so the paper ran the comparison directly: replace dropout with each of the token-level augmentations everyone had been proposing, keep everything else identical, and read the STS-B development score.

Two rows deserve individual attention.

"Delete exactly one word" scores 75.9. This is as gentle as a discrete augmentation can possibly be — a single token, out of maybe twenty. It still costs 6.6 points. The reason is that "one token" is not a small perturbation in a code with almost no redundancy. Delete not, and the sentence means the opposite. Delete the, and nothing happens. The augmenter cannot tell those cases apart, so a fraction of your positive pairs are actively teaching the model that a sentence and its negation are the same thing. Every one of those is a wrong label, and Chapter 5 will show that at τ = 0.05 wrong labels are punished with enormous force.

MLM refill scores 62.2, the worst of all. This is the most surprising row, because it is the most sophisticated method — use a language model to keep the sentence fluent. But fluency is not fidelity. Mask 15% of "the concert was cancelled because of the storm" and a well-trained MLM will cheerfully produce "the concert was cancelled because of the singer". Grammatical, plausible, and a different fact. The better your refiller, the more confidently wrong the augmentation becomes.

Other things the paper tried, and why they lost

Two more families of alternative are worth recording, because they are the ideas most people propose next.

Use a neighbouring sentence as the positive. If the sentence after this one is on the same topic, it is a kind of positive. This is the idea behind several 2020 methods. As reported in the paper's second ablation table, taking the next sentence as the positive scores 67.1, and taking one of the next three scores 67.4 — both far below 82.5. The reason is a mismatch between "related" and "equivalent". STS asks whether two sentences mean the same thing; discourse adjacency only says they are about the same thing. Training on the second and evaluating on the first blurs precisely the distinction being measured.

Use two different encoders. Contrastive frameworks often use a query encoder and a separate key encoder (MoCo's momentum encoder, for instance). The paper tried it: with a shared encoder, dropout-SimCSE scores 82.5; with two independently parameterised encoders, 80.1. Sharing wins. And it wins for a structural reason: with two encoders, the model can satisfy the positive term by making the two encoders agree rather than by making the representation good, which is a cheaper and less useful solution. A single encoder removes that escape route.

The data: one million random Wikipedia sentences

The unsupervised model trains on 10⁶ sentences sampled from English Wikipedia. Not filtered for quality, not paired, not deduplicated by topic — just a million sentences. That deserves a moment, because it is where the "no labels" claim cashes out.

Fifteen thousand steps on unlabelled text, and the resulting model beats a supervised system trained on a million human annotations. The asymmetry between the effort and the result is the reason this paper has 4,000+ citations.

The first hint of why it works

Before the next chapter builds the proper lens, here is the ablation that will drive it — the dropout-rate sweep, on STS-B dev:

Read the two ends. At p = 0 the two views are identical, and the score falls 11 points. At p = 0.5 the two views are barely related, and the score falls 11 points. There is a sweet spot, and BERT's default happens to sit almost exactly on it — a coincidence worth noticing, since nobody chose 0.1 with this in mind.

But the column that should stop you is the last one. Fixed 0.1 means dropout is on, at the standard rate, but the same mask is reused for both views. The two representations are again identical, exactly as at p = 0 — and yet the score is 43.6 rather than 71.1. Same "identical twins" situation, twenty-seven points worse. Chapter 3 is the machinery that explains this, and Chapter 8 comes back for the full reading.

Chapter 3: Alignment and Uniformity

We now have a method that works and no account of why. "Dropout is a good augmentation" is a restatement, not an explanation. This chapter installs the lens the paper borrows from Wang & Isola (2020), and it is the most portable idea in the whole lesson: two numbers that between them diagnose almost every failure of a representation space.

The two things a good embedding space must do

Think about what you are actually asking for. Two requirements, and they pull in opposite directions.

One: things that mean the same thing must be close. Otherwise similarity search misses obvious matches. Call this alignment.

Two: the space must be used. If every sentence maps to nearly the same point, then requirement one is trivially satisfied and the embedding is worthless — that is the collapse from Chapter 1. The embeddings should spread out and cover the sphere. Call this uniformity.

Alignment alone is maximised by the constant function. Uniformity alone is maximised by a random hash, which spreads everything perfectly and puts paraphrases in unrelated places. Any useful embedding is a compromise, and the value of Wang & Isola's framework is that it lets you measure where on that compromise you are, with two scalars, on a held-out set.

Alignment, defined

Assume every embedding has been L2-normalised, so it lives on the unit sphere. Let p_pos be the distribution of semantically equivalent pairs. Then

The expected squared distance between the embeddings of positive pairs. Lower is better. Zero would mean every paraphrase pair maps to exactly the same point.

Because the vectors are unit length, this quantity has a much friendlier form. Expand the square:

So alignment is just a rescaled cosine: ℓ_align = 2(1 − E[cos]). A positive-pair cosine of 0.99 gives ℓ_align = 0.02; a cosine of 0.80 gives 0.40. The range is [0, 4], with 4 meaning every pair is antipodal.

Uniformity, defined

Uniformity is subtler, because "spread out on a sphere" needs a functional form. Wang & Isola use the Gaussian potential — the same object physicists use for repulsive energy:

Take two independent samples, measure their squared distance, and evaluate a decaying exponential of it. Pairs that are close contribute nearly 1; pairs that are far contribute nearly 0. Average, then take the log. Lower is better here too.

Three questions people always have, answered.

Why the exponential? Because it is a soft nearest-neighbour count. e^−2d2 asks "is there another point right here?" and the average answers "on average, how crowded is a random point's neighbourhood?" A collapsed representation is maximally crowded.

Why the log? Purely for scale. The raw average lives in (0, 1]; the log maps it to (−∞, 0], which makes differences between well-spread configurations legible instead of all being squashed near zero.

Why is the maximum 0? If every point maps to the same place, every distance is 0, every exponential is 1, the average is 1, and log 1 = 0. So ℓ_uniform = 0 is total collapse, and every real model produces a negative number. The more negative, the more spread.

Worked example: computing both by hand

Four sentences, embedded on a unit circle so the arithmetic is visible. We will compute uniformity over all six distinct pairs, and alignment on one positive pair.

Configuration A — the cone (what pretrained BERT looks like): the four points sit at 0°, 10°, 20°, 30°. Every pair is close. The six pairwise angles are 10°, 20°, 30°, 10°, 20°, 10°.

Squared distances, using ‖a−b‖² = 2(1 − cosθ):

Average over the six pairs (three at 10°, two at 20°, one at 30°):

Barely below zero. This configuration is nearly collapsed.

Configuration B — spread: the same four points at 0°, 90°, 180°, 270°. Four of the six pairs are 90° apart (d² = 2) and two are antipodal (d² = 4).

From −0.19 to −4.40. The same four points, rearranged, and the uniformity number moves by 4.2 nats. This measure is extremely sensitive to exactly the pathology we care about.

Now alignment. Suppose sentence 1, at 0°, has a paraphrase that the model places at 5°. Then

Excellent alignment. But suppose that in the process of spreading the points out to reach configuration B, the model also pushed that paraphrase to 25° away:

Twenty-five times worse. That is the trade-off, quantified: the spreading that fixed uniformity destroyed alignment. Everything in this chapter is about which methods pay that price and which do not.

Reading InfoNCE as an alignment/uniformity machine

Here is the connection that makes the lens more than a diagnostic. Split Chapter 1's loss into its two pieces using log(a/b) = log a − log b:

The first term wants s_ii (the positive-pair cosine) as large as possible. That is alignment, up to the constant factors in ℓ_align = 2(1 − s_ii).

The second term is a log-sum-exp of all similarities, which is dominated by the largest ones; minimising it pushes down whichever pairs are currently closest. That is a soft-max flavoured cousin of uniformity, whose log-mean-exp of −2d² = −4(1−cos) has the same structure with a different constant. Wang & Isola made this rigorous: as the number of negatives goes to infinity, the contrastive loss converges to a weighted sum of exactly these two objectives.

So the contrastive objective is not related to alignment and uniformity. It is them, added together, with τ setting the exchange rate.

The derivation that explains the whole dropout-rate table

Now we can prove — not describe, prove — why removing dropout wrecks the model.

Suppose the two views are produced by the same deterministic function: h_i⁺ = h_i exactly, for every i, for every setting of the parameters. Then the positive similarity is

The alignment term is a constant. Its gradient is identically zero. It contributes nothing to any update, ever. The loss reduces to the uniformity term alone:

And "everything" includes sentence pairs that genuinely mean the same thing. The model has no term telling it that some pairs should stay together, so it separates them along with all the others. Held-out alignment collapses. That is the mechanism, and it is exact: the missing gradient is not weak, it is zero.

Dropout restores it. With independent masks, h_i^z ≠ h_i^z′, so s_ii < 1 and depends on θ. The model must now actively arrange its parameters so that a sentence's representation is stable under its own internal noise. That stability requirement is a smoothness constraint, and smoothness generalises: a model that cannot distinguish a sentence from its noisy self also cannot violently separate two sentences that differ only slightly in meaning. Alignment on held-out semantic pairs survives.

The map of the field, in two coordinates

Run the three dropout settings and watch the trail take three different shapes.

This is the paper's Figure 3, reproduced from scratch in a toy small enough to read. The real version tracks the two measures every ten steps of actual BERT training and shows the same three shapes.

Where the real models sit

Applying the lens to the actual methods produces a map that explains why each one succeeds or fails, and why the post-hoc fixes plateau.

(The two numeric anchors are approximate coordinates from the paper's Figure 2; read the pattern rather than the digits. The STS column is exact.)

The story the table tells is clean. Raw BERT's problem was never alignment — it was uniformity. The post-hoc methods correctly diagnosed that and fixed it, but they operate on frozen embeddings and can only redistribute what is already there, so improving uniformity necessarily costs alignment. SimCSE fixes uniformity by training, which lets it simultaneously hold alignment in place with the positive term. Two coordinates, one method that moves the right one.

Diagnosing your own model with two numbers

Because both quantities are cheap to compute — one pass over a held-out set and a pairwise matrix — they make a genuine diagnostic instrument. Here is the decision table.

python — the two numbers, on a held-out setimport torch, torch.nn.functional as F

def align_uniform(model, pairs, singles):
    # pairs:   list of (sent_a, sent_b) that HUMANS called equivalent
    # singles: a flat list of sentences from the same distribution
    A = F.normalize(model.encode([p[0] for p in pairs]), dim=-1)   # (P, d)
    B = F.normalize(model.encode([p[1] for p in pairs]), dim=-1)   # (P, d)
    align = (A - B).norm(dim=-1).pow(2).mean()                        # scalar, lower better

    S = F.normalize(model.encode(singles), dim=-1)                 # (M, d)
    d2 = torch.cdist(S, S).pow(2)                                  # (M, M)
    iu = torch.triu_indices(len(S), len(S), offset=1)              # distinct pairs only
    unif = torch.log(torch.exp(-2 * d2[iu[0], iu[1]]).mean())        # scalar, lower better
    return align.item(), unif.item()

Log these two next to your loss on every evaluation. A contrastive loss that falls while alignment climbs is the single most common silent failure in this whole family of methods, and no other instrument catches it.

Chapter 4: Supervised SimCSE — Entailment In, Contradiction Against

Everything so far assumed you have no labels. Now suppose you do. The interesting question is not "can labels help" — obviously they can — but "does the same objective absorb them, or do you need a different method?" SimCSE's answer is that you change one thing: where the positive comes from. And then you get a bonus that turns out to be worth more than the positive itself.

What NLI data is

Natural Language Inference is the task of deciding whether one sentence follows from another. Annotators are given a premise and asked to write three hypotheses:

Two large corpora exist: SNLI (570,152 pairs, premises taken from image captions) and MNLI (392,702 pairs, spanning ten genres from fiction to government reports). SimCSE takes the union and keeps the entailment pairs — 314,315 of them — as its supervised training set.

The move: an entailment pair is a ready-made positive

Set (x_i, x_i⁺) = (premise, entailment hypothesis) and run Chapter 1's loss unchanged. That is the entire supervised method, at first pass.

It is worth being uncomfortable about this for a moment, because entailment is not equivalence. Entailment is asymmetric and lossy: "There are animals outdoors" follows from the premise but is far more general. If you embed them at the same point, you are asserting something the logic does not license.

The paper's defence is empirical, and it is the right kind of defence: this pair source beats every alternative that is symmetric.

(The ordering is the finding; the middle decimals are as reported in the paper's Table 4.) Note first that three of these labelled sources are no better than free dropout twins. Collecting paraphrase data is not automatically worth the money.

Why does entailment win? Two reasons, and both are about difficulty.

The surface forms are maximally different. A dropout twin shares every token with its partner. A back-translated paraphrase shares most of them. An entailment hypothesis was written from scratch by a human who read the premise and produced a genuinely different sentence — different words, different length, often different syntactic frame. Matching it requires the model to work in meaning space, because there is nothing else to work with.

The annotation is checked. ParaNMT pairs are machine-generated and inherit MT artefacts; QQP duplicates were crowd-labelled with known noise; Flickr30k captions of the same photo often describe different aspects of it ("a man in a red shirt" / "someone climbing a rock face"). NLI entailment was written under an explicit logical instruction and validated by multiple annotators.

The asymmetry, taken seriously

The empirical defence is not a licence to stop thinking, so let us look at what the asymmetry actually costs. Entailment is a one-way relation:

Train on these as positives and you are asserting similarity where the data only licenses implication. The visible consequence is a real and documented one: models trained this way place a specific sentence very close to its own generalisation, so retrieval over a mixed corpus tends to surface short generic sentences for specific queries. "A person is preparing food" becomes a strong match for a great many cooking queries, because it sits near all of them.

Two things keep the damage contained, and both are worth knowing because they generalise.

The loss only ever sees relative comparisons. InfoNCE never asks "is this pair similar in absolute terms?" It asks "is this pair more similar than those other N−1 pairs?" A premise and its generalisation genuinely are more related than a premise and a random Wikipedia sentence, so the constraint being imposed is true even though the equality it implies is not.

The errors are unbiased in direction. Because NLI annotators wrote hypotheses at many levels of generality, the corpus contains generalisations, specialisations and rephrasings in roughly comparable measure. What the model absorbs is not "specific implies general" but the union of the ways two sentences can be semantically linked — averaged over 314,315 human judgements. The systematic component largely cancels; the shared semantic component survives. This is the same "five noisy targets around a truth beat one clean target of the wrong kind" argument that made multiple captions per clip valuable in the caption-supervision literature.

The bonus: contradiction pairs are free hard negatives

Now the part that matters more than the positive. Each NLI premise comes with a contradiction hypothesis, and it has a remarkable property: it is topically identical, lexically overlapping, and semantically opposite.

An in-batch negative is a random sentence. Separating it from the premise is nearly free — different topic, different vocabulary, different everything. The contradiction shares dogs/pets, the outdoor-scene frame, the animal-activity frame, and differs on exactly the thing that matters. Learning to separate those two is where a similarity model earns its keep.

A hard negative is a negative example that the current model finds confusable with the positive. In Chapter 1 we saw that InfoNCE mines them automatically, weighting each negative by p_ik. But the softmax can only weight negatives that are present in the batch, and a random batch of Wikipedia sentences contains almost none that are genuinely hard. NLI contradiction hypotheses inject hard negatives directly, one per example, for free, because a human already wrote them.

The objective with hard negatives

Each training example is now a triple (x_i, x_i⁺, x_i⁻) = (premise, entailment, contradiction). The loss extends by adding the hard negatives to the denominator:

Read the denominator carefully, because the detail is easy to miss: the sum over j runs to N in both terms. Row i's denominator contains not only its own contradiction x_i⁻, but every row's contradiction. The multiple-choice test now has 2N candidates instead of N, and exactly one of them is right.

Worked example: what one hard negative is worth

This is the calculation that justifies the whole design. Take a batch of two supervised examples and the following cosines for row 1, at τ = 0.05 (multiplier 20):

Factor e¹⁵ out of the denominator so the arithmetic stays on human scale. The denominator becomes e¹⁵(1 + e⁻⁹ + e^−2.6 + e⁻¹⁰):

Now delete the two hard negatives and recompute with in-batch negatives only:

The same encoder, the same premise, the same positive. Adding one human-written contradiction multiplies this row's loss — and therefore its gradient — by nearly six hundred.

Look at where the repulsive force goes, using Chapter 1's gradient formula (the weight on negative k is p_ik):

Ninety-nine point eight percent of the repulsion in this row is being spent on the one negative that is worth learning from. Random negatives are not merely less useful — once a hard negative is present, they are numerically invisible.

Things the authors tried that did not help

A good ablation section tells you where the idea stops working, and this one does.

Weighting the hard negatives. You can add a coefficient α on the hard-negative term, e^α·sim/τ — turning their influence up or down. The paper swept α and found α = 1 (no weighting at all) to be the best or tied-best setting. The softmax's own weighting was already correct; a second knob on top of it just distorts the mining that Chapter 1 showed happens automatically.

Adding the neutral hypotheses as extra negatives. Every NLI premise also has a neutral hypothesis, and it is sitting right there. Using it as another hard negative does not help. The reason is a labelling subtlety: "neutral" means the premise does not decide, which is entirely compatible with the two sentences being very similar. Training the model to push neutrals away teaches it something false.

Adding ANLI. ANLI is an adversarially-collected NLI dataset, built specifically so that models fail on it. Adding it does not improve STS. Adversarially hard inference examples are not the same thing as informative similarity examples; ANLI's difficulty comes from requiring multi-step reasoning, which a bag-of-meaning sentence vector is never going to represent anyway.

The realisation: shapes for the supervised loop

python — supervised SimCSE, one step# Each example is a TRIPLE. features has 3 sentences per row.
# input_ids: (N, 3, T)  ->  [premise, entailment, contradiction]
ids = input_ids.view(-1, T)                   # (3N, T)

out = mlp(encoder(ids).last_hidden_state[:, 0])  # (3N, 768)
out = out.view(-1, 3, 768)                      # (N, 3, 768)
z1, z2, z3 = out[:,0], out[:,1], out[:,2]      # anchor / positive / hard neg

z1 = F.normalize(z1, dim=-1)
z2 = F.normalize(z2, dim=-1)
z3 = F.normalize(z3, dim=-1)

sim_pos  = z1 @ z2.T / 0.05                    # (N, N)  diagonal = the answers
sim_hard = z1 @ z3.T / 0.05                    # (N, N)  EVERY row's contradiction
logits   = torch.cat([sim_pos, sim_hard], dim=1)  # (N, 2N)

labels = torch.arange(N, device=z1.device)      # (N,) — answer still on the diagonal
loss = F.cross_entropy(logits, labels)          # scalar

Two lines carry the whole chapter. torch.cat([sim_pos, sim_hard], dim=1) widens the multiple-choice test from N options to 2N, and the labels do not change — the correct answer is still at index i, because the hard negatives were appended after the positive block. Append them before it and every label is wrong by N; the loss will sit near ln 2N and you will spend an hour blaming the learning rate.

The supervised recipe otherwise differs from the unsupervised one in exactly three numbers: batch size 512 instead of 64, learning rate 5e-5 instead of 3e-5, and 3 epochs over 314k triples instead of 1 epoch over 1M sentences. And unlike the unsupervised setting, the MLP pooling head is kept at test time.

Chapter 5: Temperature and the Shape of the Batch

τ = 0.05 looks like a detail. It is not. It is the single hyperparameter most likely to be responsible for a contrastive model that will not train, and the mechanism by which it breaks things is worth understanding in full, because the same τ appears in CLIP, MoCo, SimCLR, E5, BGE and every embedding model shipped since.

What temperature does, from zero

The softmax over scaled scores is

Consider what happens at the two extremes, because the limits tell you what the knob is for.

τ → ∞. Every score is divided by an enormous number, so every logit approaches 0, so every exponential approaches 1, so p_ij → 1/N. The distribution is uniform: the model expresses no preference at all, and the loss is pinned at ln N regardless of what the encoder does.

τ → 0. Differences between scores are multiplied by a huge factor, so the largest score's exponential dwarfs all others: p → a one-hot vector on the argmax. The softmax becomes a hard maximum, which is where the name comes from.

So τ interpolates between "no opinion" and "absolute certainty". The name is borrowed from statistical physics, where the Boltzmann distribution p ∝ e^−E/kT has exactly this form: high temperature means a system spread across many states, low temperature means it settles into the ground state.

Why cosine similarity forces τ to be small

Chapter 1 gave the argument; here is the number that makes it undeniable. Cosine similarities live in [−1, 1]. At τ = 1, the very best possible row — positive at cosine 1, all N−1 negatives at cosine −1 — produces

A perfect encoder cannot get the loss below 2.254, while a random one sits at ln 64 = 4.159. The entire dynamic range available to learning is 1.9 nats, and the gradient is correspondingly feeble everywhere in it.

At τ = 0.05 the same expression gives log(1 + 63 e⁻⁴⁰) = log(1 + 2.7 × 10⁻¹⁶) ≈ 2.7 × 10⁻¹⁶. The floor is zero for all practical purposes, and the full range from 4.159 down to 0 is available.

Worked example: the same row at four temperatures

Take the row-1 cosines from Chapter 1: positive 0.72, negatives 0.54 and 0.19. Nothing about the encoder changes across the four columns below. Only the divisor changes.

Walk the τ = 0.20 row by hand so you trust the rest. Multiply the cosines by 5: 3.60, 2.70, 0.95. Subtract the max: 0, −0.90, −2.65. Exponentiate: 1, 0.40657, 0.07065. Sum: 1.47722. So p₁₁ = 1/1.47722 = 0.67695 and ℓ₁ = ln(1.47722) = 0.39016. The positive-pair gradient magnitude is (1 − 0.67695)/0.20 = 0.32305/0.20 = 1.6153.

Now read the fifth column, which is the one that matters, and notice that it is not monotonic.

Where the peak is, in closed form

That maximum can be located exactly, which turns a vague "tune it" into a rule of thumb worth carrying around.

Take the simplest case: one positive at similarity s and one negative at s − Δ, so Δ is the cosine gap the encoder currently achieves. The softmax over two candidates is a logistic function:

So the positive-pair gradient magnitude is

Substitute x = Δ/τ (so τ = Δ/x) and the Δ factors out:

Maximise the bracket. Differentiating x/(1+e^x) and setting the numerator to zero gives (1 + e^x) − x e^x = 0, that is

Check: e^1.2785 = 3.5917, and 3.5917 × 0.2785 = 1.0003. So

Set the temperature to about three quarters of the cosine gap your model currently produces. Verify it on our numbers: with Δ = 0.72 − 0.54 = 0.18, the formula predicts τ* = 0.1408, and a numerical sweep over the two-candidate row finds the peak at 0.1410. Adding the third candidate shifts it to 0.158, because a second negative adds probability mass off the diagonal.

The final column is the other half of the story: as τ falls, the ratio of attention paid to the hard negative versus the easy one explodes from 1.42 to 10¹⁵. Small τ does not make the model push harder overall; it makes the model push almost exclusively on whatever is currently closest. Contrastive learning at low temperature is hard-negative mining with extra steps.

The dark side: what a small τ does to a wrong label

Press that last button. The scenario is one every practitioner meets: a negative in the batch is actually a paraphrase of the anchor, so the model correctly scores it above the designated positive. The loss now punishes the model for being right.

Suppose the positive scores 0.50 and this false negative scores 0.70, with a third candidate at 0.20. At τ = 1:

At τ = 0.05 the logits are 10, 14, 4. Factor out e¹⁴:

The penalty for the same mislabelling is 3.7 times larger. And since this row's loss now dominates the batch, its (incorrect) gradient dominates the update.

Batch composition, quantified

Let q be the probability that two randomly drawn sentences from your corpus are semantically equivalent. In a batch of N, the expected number of false negatives is N(N−1)q.

Read row three. On a support-ticket corpus with a batch of 64, roughly forty of your sixty-three "negatives" are things the model should be pulling closer. The loss is fighting itself, and lowering τ makes the fight fiercer. This single table explains why so many teams reproduce SimCSE perfectly on Wikipedia and then watch it fail on their own data.

The mitigations, in order of how often they are the right answer:

On batch size, the paper is worth quoting for what it does not say. Unsupervised SimCSE uses a batch of 64 — tiny by contrastive standards, where vision models routinely use 4,096 — and the appendix reports that the model is not very sensitive to batch size provided the learning rate is adjusted with it. That is a meaningful negative result. In image contrastive learning, batch size is a headline hyperparameter; here the negatives arrive already-diverse from a million-sentence pool, so more of them add little.

Symmetric or not?

One structural difference from CLIP is worth naming since people port code between them. CLIP's loss is symmetric: it computes cross-entropy along the rows and the columns of the similarity matrix and averages. SimCSE's unsupervised loss, as published, is one-directional — rows only.

The asymmetry is defensible here and not there. In CLIP the two towers are different networks over different modalities, so "which caption matches this image" and "which image matches this caption" are genuinely different questions with different failure modes. In SimCSE both views come from the same encoder applied to the same sentence, so the similarity matrix is nearly symmetric already and the second direction adds little beyond compute. Adding it is harmless; expecting it to matter is not.

Chapter 6: The Narrow Cone

Chapter 3 gave us a name for BERT's problem — bad uniformity — and a number for it. This chapter explains where that pathology comes from, why two post-hoc fixes almost solve it, and derives, from the contrastive loss itself, the reason training fixes it better than any transform can.

The measurement that started it

Take two sentences with nothing in common. "The 1957 Chevrolet Bel Air was produced in Flint, Michigan." and "She practises the cello every morning before school." Embed both with BERT and take the cosine.

You will get something in the neighbourhood of 0.6 to 0.8. Not 0.0, which is what "unrelated" ought to look like on a sphere. Repeat with a thousand random pairs and the average barely moves. Ethayarajh (2019) measured this systematically across BERT, ELMo and GPT-2 and found that in the upper layers the expected cosine similarity between randomly chosen words and sentences is high — often above 0.5, sometimes far above.

The name for this is anisotropy. A distribution is isotropic if it looks the same in every direction — its second moment is a multiple of the identity, so variance is spread evenly across all d axes. It is anisotropic when the mass concentrates in a small subspace: a narrow cone rather than a ball.

That toy is the whole of BERT-whitening in two lines, and it also tells you why raw BERT looked like it had "good alignment" in Chapter 3's table: when everything is at cosine 0.8 from everything, paraphrases are close — but so is every other pair, which is precisely what uniformity measures.

Where the cone comes from

Anisotropy is not a bug in BERT's code. It is a predictable consequence of how language models are trained, and the mechanism was identified by Gao et al. (2019) under the name representation degeneration.

A language model ends in a softmax over the vocabulary: for hidden state h and output embedding w_v of token v, the score is h^Tw_v. Now think about a rare token — one that appears as the target almost never. What gradient does its output embedding receive?

Almost every update in which it participates is a negative one: it appears in the softmax denominator, so the loss pushes h^Tw_rare down, for whatever hidden states happen to be in the batch. Over millions of steps, the accumulated push drives w_rare toward the direction that minimises its dot product with the typical hidden state — roughly, toward the negative of the average hidden direction. Every rare token is pushed toward the same place, because they all see the same average. The rare tokens pile into a shared cone.

Meanwhile hidden states are pushed to have large dot products with the output embeddings of frequent tokens, which are themselves few and concentrated. Both halves of the model converge on a preferred direction, and the whole representation drifts off-centre.

The post-hoc fixes, and their ceiling

If the problem is a common component, subtract it. Two 2020–21 methods do essentially that, without touching the encoder.

BERT-whitening (Su et al. 2021). Collect embeddings over a corpus, compute the mean μ and covariance Σ, eigendecompose Σ = UΛU^T, and transform

Subtracting μ removes the shared offset; multiplying by UΛ^−1/2 rotates onto the principal axes and rescales each so that every direction has unit variance. The transformed covariance is exactly the identity. Perfect isotropy, achieved with a matrix multiply and no gradient steps. Result: 66.28.

BERT-flow (Li et al. 2020). Same goal, more machinery: train an invertible normalising flow that maps the BERT embedding distribution onto a standard Gaussian, then use the mapped vectors. Result: 66.55.

Both take raw BERT from 56.70 to about 66.4, a ten-point gain for essentially no cost. And then both stop, ten points short of SimCSE. Chapter 3 told us why in the abstract — they trade alignment for uniformity. Here is the concrete version:

The derivation: how the loss flattens the spectrum

This is the paper's theoretical contribution and it is short enough to do in full. Assemble the m embeddings of your corpus as rows of a matrix W ∈ R^m×d, all rows L2-normalised.

Step 1: isolate the repulsive term. Chapter 3 split the loss as −s_ii/τ + log∑_je^s_ij/τ. Write the second term as an expectation over the data:

Step 2: apply Jensen's inequality. For any random variable Z, log E[e^Z] ≥ E[Z], because the exponential is convex. Setting Z = h_i^Th_j/τ with j uniform:

Averaging over i as well:

where Sum(·) means the sum of all entries of the matrix. So the repulsive term, times τm², is an upper bound on Sum(WW^T). Driving the term down drives the bound down.

Step 3: recognise what Sum(WW^T) is. Expand the double sum:

It is the squared norm of the mean embedding (times m²). Minimising it is centring the embeddings. The contrastive loss derives, all by itself, the first half of what whitening does by hand.

Step 4: the trace is pinned. Because every row is a unit vector, the diagonal of WW^T is all ones, so

The eigenvalues must always sum to m, no matter what the encoder does. This is the constraint that makes the whole argument work: you cannot shrink the spectrum, only redistribute it.

Step 5: Merikoski's bound. For a matrix all of whose entries are non-negative, the sum of the entries is an upper bound on the largest eigenvalue: Sum(WW^T) ≥ λ₁. So minimising Sum squeezes λ₁ downward — while ∑λ_k stays fixed at m. Mass taken from the top eigenvalue must reappear in the smaller ones. The spectrum flattens.

Worked example: three configurations, by hand

Four unit vectors in two dimensions. We compute Sum(WW^T), the eigenvalues of W^TW (which are the non-zero eigenvalues of WW^T, and whose square roots are the singular values of W), and the uniformity from Chapter 3.

Configuration A — the cone, at 0°, 10°, 20°, 30°. Rows: (1, 0), (0.98481, 0.17365), (0.93969, 0.34202), (0.86603, 0.50000).

Sum of rows: x-component 1 + 0.98481 + 0.93969 + 0.86603 = 3.79053; y-component 0 + 0.17365 + 0.34202 + 0.50000 = 1.01567. So

Now W^TW, a 2×2 matrix of column inner products:

Trace = 3.60287 + 0.39713 = 4.000, exactly m as promised. Determinant = 3.60287 × 0.39713 − 0.92542² = 1.43075 − 0.85640 = 0.57436. Eigenvalues of a 2×2 with trace T and determinant D are (T ± √(T² − 4D))/2:

Condition number λ₁/λ₂ = 25.8. Singular values √λ = 1.962 and 0.386. One direction carries 96% of the energy; the space is effectively one-dimensional. And check Merikoski: 15.400 ≥ 3.851. It holds, because every cosine here is positive.

Configuration B — a quarter turn, at 0°, 30°, 60°, 90°. Here ∑x² = ∑y² = 2.000 and ∑xy = 0.86603, so λ = 2 ± 0.86603:

Configuration C — fully spread, at 0°, 90°, 180°, 270°. The rows sum to the zero vector, so Sum(WW^T) = 0. And W^TW = 2I, giving λ₁ = λ₂ = 2 and ratio 1.00 — a perfectly flat spectrum.

Every column moves together. Sum(WW^T) falls, λ₁ falls while the trace stays at 4, the condition number collapses toward 1, and uniformity improves by more than four nats. The theoretical quantity the loss provably minimises, the spectral quantity we care about, and the empirical measure from Chapter 3 are three views of one thing.

Two things to notice while you play. First, "+ centring" alone — just subtracting the mean, no rescaling — recovers most of the improvement, which is the same lesson as the (1, ±0.3) example: the dominant pathology is a shared offset. Second, "+ whitening" produces a perfectly flat spectrum, flatter than the contrastive preset, and yet we know it scores ten points worse on STS. Flatness is not the goal. Flatness with the right structure preserved is the goal, and only a trained objective with an alignment term can tell the difference.

Measure it on your own model, in nine lines

Anisotropy is not a theoretical concern; it is a property of whatever encoder you happen to be holding, and it takes under a minute to check.

python — is your embedding space a cone?import numpy as np

E = model.encode(sentences)                       # (m, d), m ≈ 2000 is plenty
E = E / np.linalg.norm(E, axis=1, keepdims=True)   # unit rows — required

G = E @ E.T                                        # (m, m) all pairwise cosines
iu = np.triu_indices(len(E), 1)
print("mean random-pair cosine :", G[iu].mean())      # > 0.4 = a cone. Should be ≈ 0
print("||mean embedding||      :", np.linalg.norm(E.mean(0)))
#   Sum(WW^T) = m^2 * ||mean||^2  — the exact quantity Chapter 6 minimises

s = np.linalg.svd(E, compute_uv=False)              # (d,) singular values, descending
print("spectrum decay          :", (s[:6] / s[0]).round(3))
q = s**2 / (s**2).sum()                            # eigenvalue shares, sums to 1
print("effective rank          :", np.exp(-(q * np.log(q + 1e-12)).sum()))

The last line is worth explaining because it is the most useful single number. Treating the normalised eigenvalue shares q_k as a probability distribution, its spectral entropy H = −∑ q_k log q_k measures how many directions the mass is spread over, and e^H — the effective rank — converts that back into a dimension count.

Sanity-check it on our three toy configurations. For configuration C, q = (0.5, 0.5), so H = ln 2 and the effective rank is exactly 2 — both dimensions fully used. For configuration A, q = (3.8508/4, 0.1492/4) = (0.9627, 0.0373), giving H = −(0.9627 ln 0.9627 + 0.0373 ln 0.0373) = −(−0.0366 − 0.1227) = 0.1593 and an effective rank of e^0.1593 = 1.17. Four vectors in two dimensions, using 1.17 of them.

Run this on raw BERT with d = 768 and the effective rank comes out in the low tens rather than the hundreds. That number, more than any plot, is what "the space is a cone" means operationally: you are paying to store 768 floats and getting the discriminative power of a couple of dozen.

What the paper measured

The paper plots the singular value distribution of the sentence embedding matrix for BERT, BERT-whitening, and SimCSE, and the picture matches the derivation exactly: BERT's spectrum decays steeply, SimCSE's is markedly flatter. It also reports the alignment/uniformity coordinates we saw in Chapter 3, showing SimCSE moving down the uniformity axis while holding its horizontal position.

And there is a satisfying consistency check available. Chapter 3 defined uniformity through a pairwise Gaussian potential; this chapter derived flattening through a matrix spectrum. They are the same phenomenon in different coordinates — the pairwise view and the second-moment view of "is the mass spread out". Our three-configuration table demonstrates the equivalence numerically, and it is the reason you can diagnose anisotropy either way: measure the average cosine of random pairs (cheap, one number) or eigendecompose the embedding matrix (informative, tells you how many directions are being used).

Chapter 7: The Toy, By Hand

Six chapters of machinery. Now we run the whole thing on paper: three sentences, two dropout masks each, every cosine, every exponential, and a final loss number you can check against real PyTorch. If you can reproduce this page with a pencil, you understand SimCSE.

The setup

Three sentences in the batch:

The four units are not arbitrary. Read them as: unit 1 ≈ "human agent", unit 2 ≈ "music", unit 3 ≈ "finance/abstract", unit 4 ≈ "temporal reference". Sentences 1 and 2 have similar profiles because they mean nearly the same thing; sentence 3 does not. This is what a good encoder is supposed to produce, and our toy encoder already has it — we are studying what the loss does, given a representation.

Dropout at p = 0.5, so each mask keeps two of the four units and multiplies survivors by 1/(1 − 0.5) = 2. (Real SimCSE uses 0.1; we use 0.5 so the masks are visible.) The six masks — two per sentence, drawn independently:

Check one cell before continuing. Sentence 1, view A, keeps units 1 and 2 of (4, 3, 2, 1) and doubles them: (2×4, 2×3, 0, 0) = (8, 6, 0, 0). Units 3 and 4 were dropped, so they are exactly zero.

Norms, which we will need for every cosine:

Step 1: the 3 × 3 similarity matrix

We need s_ij = cos(h_i^A, h_j^B) for all nine combinations. Three in full, the rest in the table.

The positive pair, s₁₁. h₁^A = (8, 6, 0, 0), h₁^B = (8, 0, 4, 0). Only unit 1 survived in both views:

The near-neighbour, s₁₂. h₁^A = (8, 6, 0, 0) against h₂^B = (0, 8, 0, 4). The overlap is unit 2 only:

The far one, s₁₃. h₁^A = (8, 6, 0, 0) against h₃^B = (2, 0, 8, 0). Overlap on unit 1:

Continuing the same way for rows 2 and 3 (row 3's second entry involves the only negative product, 8 × 0 + 0 × 8 + 8 × 0 + (−6)(4) = −24, over 89.4427):

(If these look familiar: Chapter 1's three-sentence worked example used exactly this matrix, rounded to two decimals. It was a preview of this page.)

Three sanity checks. The diagonal is the largest entry in every row — the model already prefers the right answer, so this batch is "easy" and we expect a small loss. The 1–2 cross terms (0.5367) are much larger than the 1–3 cross terms (0.194), which is correct: the guitar and the musical instrument really are related. And s₃₂ is negative, which is only possible because u₃ has negative components — the finance sentence actively points away from the music direction.

Step 2: scale by 1/τ

τ = 0.05, so every entry is multiplied by 20:

Step 3: softmax and per-row loss

Row 1. Subtract the row maximum (14.3108) from every entry — this changes nothing about the softmax and keeps the exponentials on human scale:

Row 2. Maximum is again 14.3108, on the diagonal. Differences: −3.5777, 0, −11.4004.

Row 3. Maximum 15.5223. Differences: −8.3669, −20.8889, 0.

Step 4: the batch loss

That is the number. Chance level for a batch of three is ln 3 = 1.0986, so this batch is being solved almost perfectly — which is what we expected from a similarity matrix whose diagonal already dominates.

Row 3 contributes essentially nothing (0.00023). The finance sentence is trivially distinguishable from the two music sentences, so it is done: no gradient will flow from it. All the learning in this batch happens in rows 1 and 2, where the two music sentences confuse each other. This is InfoNCE's automatic curriculum from Chapter 1, visible in three numbers.

The same batch at τ = 1

Rerun step 2 with no scaling and everything else identical:

At τ = 1 the batch loss is 0.821 against a chance level of 1.099 — the model is barely a quarter of a nat better than random, despite the diagonal winning every row. Row 3's probability of being right is 0.4975: a coin flip, on a pair the geometry separates by 0.42 in cosine. Chapter 5's argument, now on our own numbers.

What happens with identical masks

Rerun the whole thing with z′ = z for every sentence — the paper's "Fixed 0.1" pathology. Now h^B = h^A, so the diagonal is exactly 1 and the off-diagonals are computed within the same two-unit subspace:

Row 1 at τ = 0.05: logits 20, 19.2, −3.578; differences 0, −0.8, −23.578; exponentials 1, 0.449329, 4 × 10⁻¹¹; sum 1.449329; so p₁₁ = 0.689975 and ℓ₁ = ln(1.449329) = 0.371101. By symmetry ℓ₂ is the same and ℓ₃ ≈ 0, giving L = 0.24740.

The loss is thirteen times larger than the dropout version. A naive reading would call that a harder, better training signal. It is the opposite, and here is the precise statement of why:

The loss wants very badly to raise s₁₁, and it structurally cannot. Every gradient that actually reaches the parameters comes from the repulsive terms, pushing sentences 1 and 2 apart — two sentences that mean nearly the same thing. Chapter 3's argument, on our own numbers, with the two factors of the chain rule laid side by side.

Verify it yourself

python — reproduces every number on this pageimport torch, torch.nn.functional as F

u = torch.tensor([[4., 3, 2, 1], [3., 4, 1, 2], [1., -2, 4, -3]])   # (3, 4)

# the six masks from the table, as 0/1 rows; inverted dropout scale = 1/(1-0.5) = 2
zA = torch.tensor([[1.,1,0,0], [1.,1,0,0], [0.,0,1,1]])
zB = torch.tensor([[1.,0,1,0], [0.,1,0,1], [1.,0,1,0]])

hA, hB = 2 * u * zA, 2 * u * zB                 # (3, 4) each
sim = F.normalize(hA, dim=-1) @ F.normalize(hB, dim=-1).T   # (3, 3)

print(sim)                     # matches the table to 6 dp
print(F.cross_entropy(sim / 0.05, torch.arange(3)))   # 0.018462
print(F.cross_entropy(sim / 1.00, torch.arange(3)))   # 0.820716

# the pathology: same mask for both views
hB2 = 2 * u * zA
sim2 = F.normalize(hA, dim=-1) @ F.normalize(hB2, dim=-1).T
print(F.cross_entropy(sim2 / 0.05, torch.arange(3)))  # 0.247400 — and zero useful gradient

Chapter 8: What the Numbers Say

A method is only as good as the evaluation that measured it, and this paper's evaluation section contains a methodological correction that is arguably as valuable as the model. Let us do the numbers properly: what was measured, how, what it establishes, and what it does not.

First, an evaluation bug the paper fixed

Each STS task is not one dataset but several sub-datasets. STS12, for instance, contains five topical subsets. When you report "STS12 Spearman", you have three defensible options, and they give different numbers:

Papers before SimCSE were not consistent about which they used, and some reported mean while comparing against numbers computed with all. That is not a rounding-level discrepancy: mean is a strictly easier setting, because it forgives a model whose similarity scale drifts between topics.

SimCSE's appendix documents this, adopts all throughout, and re-evaluates every baseline under the same protocol. The 56.70 for averaged BERT and the 74.89 for SBERT in this lesson are the paper's re-computed figures, not the ones from the original publications.

The headline table

Four readings worth extracting.

The unsupervised model beats a supervised one. 76.25 over 74.89. This is the number that made the paper famous, and the honest framing is: dropout twins are a better positive-pair source than the paraphrase supervision SBERT was built on, once you pair them with in-batch negatives and a sensible temperature.

The post-hoc methods plateau together. BERT-flow 66.55, BERT-whitening 66.28, IS-BERT 66.58 — three quite different techniques landing within 0.3 of each other. When several independent approaches converge on the same ceiling, the ceiling is a property of the setup rather than of any one method. Here the setup is frozen embeddings: you can only redistribute the information BERT already put in the vector, and Chapter 6 explained why that trades alignment for uniformity.

Supervision and the contrastive objective stack. 76.25 unsupervised, 81.57 supervised. If NLI labels merely duplicated what dropout provides, the two would not add. They are doing different work: dropout supplies local stability, entailment pairs supply semantic identity across different surface forms, and contradiction pairs supply hard negatives. Three different jobs.

Scale still helps, but modestly. RoBERTa-large gets 83.76 versus BERT-base's 81.57 — 2.2 points for roughly three times the parameters. Compare that with the 24.9 points the objective bought. In sentence embedding, the loss function has historically mattered far more than model size.

The dropout ablation, read properly

We have met this table three times. Here is the full reading, with each row explained by the machinery we have built.

The curve is an inverted U with a hard floor at both ends, and the two ends fail for opposite reasons — too little perturbation kills the gradient, too much kills the meaning. That is the signature of a genuine sweet spot rather than a monotone "more is better" knob.

The other ablations

Pooling. Options are the [CLS] token with BERT's MLP pooler, [CLS] with the MLP used only during training, and mean-pooling over tokens. The paper's finding is asymmetric and slightly odd: for the unsupervised model, keeping the MLP during training and discarding it at test time is best; for the supervised model, keeping it throughout is best. The differences are around a point. The practical advice is to treat pooling as a hyperparameter to check, not a principle — and to note that this is a real reproducibility trap, because "which pooler at test time" is exactly the kind of detail that gets lost between a paper and a re-implementation.

An MLM auxiliary objective. Adding λ · (masked language modelling loss) to the contrastive loss, with a small λ, improves transfer tasks and slightly hurts STS. The reason is legible: MLM preserves token-level information that a purely contrastive objective is free to discard, and downstream classification probes benefit from that information while sentence-level similarity does not. It is a clean demonstration that "a better embedding" is task-relative.

Transfer tasks. Evaluated on the SentEval classification suite (MR, CR, SUBJ, MPQA, SST-2, TREC, MRPC), SimCSE is roughly on par with previous methods — not the blowout it achieves on STS. This is honest reporting and it makes sense: those tasks train a logistic-regression probe on top of the frozen embedding, and a probe can undo an arbitrary linear transformation. The anisotropy that wrecks cosine similarity is largely invisible to a trained probe, so fixing it buys little there.

What the numbers do not establish

A lesson that only reports the wins is advertising. Four genuine limitations, in rough order of how likely they are to bite you.

1. STS is symmetric; retrieval is not. Every result here concerns pairs of sentences of similar length and register. Real retrieval matches a short question against a long passage — an asymmetric task with a completely different similarity structure. The BEIR benchmark (2021) measured exactly this and found that SimCSE-style models trained on STS-flavoured objectives frequently underperform BM25, a keyword-matching algorithm from the 1990s, on out-of-domain retrieval. Strong STS is not evidence of strong retrieval, and treating it as such has cost real teams real quarters.

2. Checkpoint selection uses labels. The unsupervised recipe evaluates on STS-B dev every 250 steps and keeps the best checkpoint. With a single training run this is mild; across a hyperparameter sweep it is not negligible. "Unsupervised" here means "no labels in the loss", and the number of labelled pairs touched by model selection is small but not zero.

3. English, general domain, short sentences. Max sequence length 32 tokens. Wikipedia prose. If your text is 500-token legal clauses, none of the reported numbers transfer, and Chapter 5's false-negative table says the failure will be sharp rather than gradual.

4. Sentence embeddings are a bag of meaning. A single vector cannot represent compositional structure with much fidelity. SimCSE improves the geometry; it does not give the representation a syntax. Negation, quantifier scope, and argument order remain systematically underserved — "the dog bit the man" and "the man bit the dog" will be close under any of these models.

The cost, and why that matters

Fifteen thousand steps of ordinary fine-tuning, on free text, moving a well-studied benchmark by twenty points. That ratio — not the absolute score — is the paper's real claim on your attention.

If you are reproducing this and it will not work

Chapter 9: The Recipe Everyone Inherited

SimCSE's lasting contribution is not 76.25. It is a template. Every text embedding model shipped since is a variation on four decisions this paper fixed, and knowing which of the four a new model changed is usually enough to understand it in a paragraph.

The template

Rows 1 and 2 did not move at all. Rows 3 and 4 are where five years of progress happened, and the direction of travel is consistent: the objective was right; the data was the bottleneck.

What the descendants changed

Read the middle column. In every row it says the same thing.

The honest limits, carried forward

Two of Chapter 8's limitations shaped the field's next five years, so they are worth restating as forward-looking facts rather than caveats.

Symmetric similarity is not retrieval. BEIR made this unmissable in 2021, and the entire prefix-and-instruction line of work (E5's query:/passage:, Instructor, BGE's instructions) exists to give one model two different notions of similarity. If you take one design lesson from this lesson into your own system, make it this: decide whether your task is symmetric or asymmetric before you pick an objective, because the same InfoNCE loss trained on the wrong pair type produces a model that benchmarks well and retrieves badly.

The bottleneck moved from method to data. After SimCSE, the objective stopped being where the gains were. MTEB (2022) turned embedding evaluation into a 50-task suite precisely because single-task STS had been saturated by a method anyone could implement in an afternoon. The frontier moved to pair mining, hard-negative mining, instruction conditioning, and multilingual coverage — all data questions.

Cheat sheet

The symbols.

The four equations.

The numbers worth remembering.

Build it yourself — the weekend recipe

References

1. Gao, T., Yao, X., Chen, D. "SimCSE: Simple Contrastive Learning of Sentence Embeddings," EMNLP 2021 — arXiv:2104.08821. The paper this lesson is built on.
2. Wang, T., Isola, P. "Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere," ICML 2020 — arXiv:2005.10242. Chapter 3's entire lens.
3. van den Oord, A., Li, Y., Vinyals, O. "Representation Learning with Contrastive Predictive Coding," 2018 — arXiv:1807.03748. Where InfoNCE and the log N bound come from.
4. Reimers, N., Gurevych, I. "Sentence-BERT," EMNLP 2019 — arXiv:1908.10084. The supervised baseline SimCSE overtakes without labels.
5. Ethayarajh, K. "How Contextual are Contextualized Word Representations?," EMNLP 2019 — arXiv:1909.00512. The anisotropy measurement.
6. Gao, J. et al. "Representation Degeneration Problem in Training Natural Language Generation Models," ICLR 2019 — arXiv:1907.12009. Why the cone forms.
7. Li, B. et al. "On the Sentence Embeddings from Pre-trained Language Models" (BERT-flow), EMNLP 2020 — arXiv:2011.05864.
8. Su, J. et al. "Whitening Sentence Representations for Better Semantics and Faster Retrieval," 2021 — arXiv:2103.15316.
9. Chen, T. et al. "SimCLR: A Simple Framework for Contrastive Learning of Visual Representations," 2020 — arXiv:2002.05709; He, K. et al. "MoCo," 2019 — arXiv:1911.05722. The vision lineage whose augmentations text could not copy.
10. Bowman, S. et al. "SNLI," 2015 — arXiv:1508.05326; Williams, A. et al. "MultiNLI," 2017 — arXiv:1704.05426. The 314k triples.
11. Thakur, N. et al. "BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models," 2021 — arXiv:2104.08663. Why strong STS is not strong retrieval.
12. Muennighoff, N. et al. "MTEB: Massive Text Embedding Benchmark," 2022 — arXiv:2210.07316. What replaced STS as the yardstick.
13. Wang, L. et al. "Text Embeddings by Weakly-Supervised Contrastive Pre-training" (E5), 2022 — arXiv:2212.03533; Li, Z. et al. "GTE," 2023 — arXiv:2308.03281; Xiao, S. et al. "C-Pack / BGE," 2023 — arXiv:2309.07597. The direct descendants.
14. Su, H. et al. "One Embedder, Any Task" (Instructor), 2022 — arXiv:2212.09741; Kusupati, A. et al. "Matryoshka Representation Learning," 2022 — arXiv:2205.13147; Gao, L. et al. "Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup" (GradCache), 2021 — arXiv:2101.06983.
15. Srivastava, N. et al. "Dropout: A Simple Way to Prevent Neural Networks from Overfitting," JMLR 2014. The regulariser that turned out to be an augmentation. Merikoski, J. K. "On the trace and the sum of elements of a matrix," Linear Algebra and its Applications, 1984 — the bound in Chapter 6.

Where to go from here