Sentence-BERT: Siamese Networks for Sentence Embeddings

Chapter 1: BERT's Bad Vectors

Chapter 0 left a puzzle on the table. BERT is the best sentence-pair model in the world, and its own sentence vectors are worse than averaging word vectors from 2014. Both statements are true simultaneously. Resolving that contradiction is the whole intellectual content of this paper, so we are going to take it slowly.

What "a BERT sentence vector" even means

BERT does not output a sentence vector. It outputs a sequence of vectors — one per token, each 768-dimensional for the base model. Feed it a 9-token sentence and you get back a tensor of shape (9, 768). To get one vector you have to pool: collapse the token axis. Two obvious ways were in universal use.

The [CLS] token is a special symbol BERT prepends to every input. During pretraining, the vector above it is fed to a classifier for the next-sentence-prediction task — a binary question, "did these two segments actually appear next to each other in the corpus?" So [CLS] is the one position that was explicitly trained to summarise the whole input. Every tutorial diagram in 2019 labelled it "the sentence representation." It seemed obvious.

The number that should stop you

Here is the paper's Table 1 in full: Spearman rank correlation between each method's cosine similarities and human similarity ratings, multiplied by 100, over the seven standard Semantic Textual Similarity datasets. None of these models were trained on STS data — this is pure out-of-the-box quality.

The paper's own summary is blunt: using the output of BERT directly "leads to rather poor performances… the resulting sentence embeddings are worse than averaging GloVe embeddings."

Follow one MLM gradient, with numbers

The claim "the loss never sees a pair" is easy to assert. Compute it instead, on a vocabulary of four words so every number fits on the page. Suppose the model must fill the blank in "the cat sat on the ___", the true word is mat, and the hidden vector at that masked position is H = (0.5, −0.2, 0.3). The output matrix W_vocab has one row per word:

Now the gradient into the hidden vector. With δ = p − y = (−0.6173, 0.3229, 0.1274, 0.1669):

Look hard at what that vector is made of: rows of the vocabulary matrix, weighted by how much probability mass each word currently holds. Descent moves H toward the "mat" row and away from the others. Every quantity involved — H, W_vocab, y — belongs to one sentence. There is no second sentence anywhere in the computation, so there is no term that could possibly reward or punish the distance between two sentence representations.

Compare the same calculation for GloVe. Its loss is J = ∑_ij f(X_ij)(w_i · w̃_j + b_i + b̃_j − log X_ij)² — and there, right in the middle, is w_i · w̃_j: a dot product between two vectors, fit to a target. The geometry is directly supervised at the word level. That single structural difference is worth 6.51 STS points to a method with no neural network at inference time.

Notice how much worse the [CLS] row is. On STS-B it scores 16.50 — a correlation so low that shuffling the model's answers would barely change it. The position everyone pointed at as "the sentence vector" is the worst of all the options tested. That is not a small empirical detail; it is a sign that our mental model of what BERT was doing was wrong.

Justification 1: follow the gradient of the MLM loss

Be concrete about what masked language modelling optimises. You take a sentence, replace 15% of tokens with [MASK], and predict the originals. For a masked position i with true token t:

Every gradient the encoder receives arrives through H_i, the vector at a masked position, and it says exactly one thing: "make this vector line up better with the row of W_vocab for the word that belongs here." The loss is a function of one sentence. It is never a function of two.

So ask: what would make a token vector good under this loss? Being predictive of the missing word. That rewards representations that carry local syntax, collocation, and word identity strongly — the very things that let you guess a blank. It does not reward, or punish, anything about how the sentence as a whole relates to a different sentence. A geometry in which "the app crashes" sits near "the application force-closes" is neither encouraged nor discouraged. It is simply outside the loss's field of view.

Justification 2: the [CLS] slot was trained for a task nobody kept

Why is [CLS] so much worse than the mean? Its pretraining signal came from next-sentence-prediction, which asks a binary question about adjacency. A model can score well on NSP by detecting topic continuity — do these segments share a subject? — which the literature later showed is a weak, partly-trivial task. RoBERTa dropped NSP entirely a few months after BERT, with no loss in downstream quality.

So the [CLS] vector is optimised to answer a shallow topical question, and it is optimised for consumption by a trained classifier head sitting on top of it, not for direct geometric comparison. A vector can be a perfect input to a learned linear map and still have a useless metric structure: the classifier can undo any rotation, rescaling, or coordinate-wise weighting you like. Cosine similarity cannot. It takes the coordinates literally.

This table is the crux. Fine-tuning is forgiving; cosine is not. BERT was built to be fine-tuned, so its representations were never held to the standard cosine imposes. Sentence-BERT's whole contribution is to hold them to it, during training, so that at inference nothing has to.

An aside that saves you a week: which layer?

A natural first reaction to "BERT's vectors are bad" is "you took them from the wrong layer." It is a good instinct and it has been tested exhaustively. The short version: pooling from the second-to-last layer is usually marginally better than the last, averaging the last four layers is usually marginally better still, and all of these variants land within a few points of each other — nowhere near closing the 20-point gap to a fine-tuned model.

The reason is the one we have already established. Every layer of BERT was shaped by the same objective, and that objective never looked at a pair of sentences. Changing which un-supervised layer you read from is choosing between shades of the same absence. The paper does not chase this, and neither should you: it is the highest-effort, lowest-return knob in the whole area.

Justification 3: anisotropy — the crowded cone

There is a third, purely geometric reason, and it is worth deriving because it explains a bug you will hit yourself. Contextual embedding spaces are anisotropic: instead of spreading out in all directions, the vectors occupy a narrow cone. Ethayarajh (2019) measured this — in BERT's upper layers, two randomly chosen words from random sentences have an average cosine similarity far above zero, often 0.4 to 0.6, when a well-spread space would give roughly 0.

Here is why a common component wrecks cosine, worked in numbers. Take two genuinely unrelated 4-dimensional "meanings" a and b, plus a large shared component c that every vector has because the space is coned:

Their meanings are orthogonal: a · b = 1(0) + 0(1) + (−1)(0) + 0(−1) = 0, so cos(a, b) = 0. Perfect. Now what the model actually emits is u = c + a and v = c + b:

Dot product: 4(3) + 3(4) + 2(3) + 3(2) = 12 + 12 + 6 + 6 = 36. Norms: ‖u‖ = √(16+9+4+9) = √38 = 6.164, and ‖v‖ = √(9+16+9+4) = √38 = 6.164. So:

Two sentences with orthogonal meanings score 0.947. The shared offset c has ‖c‖² = 36 and dominates the dot product completely; the meaning-carrying parts contribute exactly 0 of the 36. Every pair in your corpus will score between roughly 0.9 and 1.0, and the ranking among them is decided by the tiny residual — which is exactly the regime where noise wins.

Run the centring on our example to see it. The corpus mean here is (u+v)/2 = (3.5, 3.5, 2.5, 2.5). Subtract:

Now they are maximally dissimilar — an overcorrection caused by having only two points to estimate the mean from, but the direction of the fix is unmistakable. Removing the common component restored the signal that the raw cosine had buried.

What the cosine of two random vectors should be

To know that 0.947 is pathological you need a baseline, and it is derivable. Take two vectors whose coordinates are independent draws from any zero-mean distribution in d dimensions. Their dot product is a sum of d independent zero-mean terms, so its expectation is 0 and its standard deviation grows like √d. Each norm grows like √d as well. So:

Put d = 768 in: 1/√768 = 0.036. In a healthy, isotropic 768-dimensional space, two unrelated sentences should score about 0.00 with a typical deviation of 0.04, so essentially everything unrelated lands in [−0.11, +0.11]. That is the yardstick.

This table is also the practical reason a similarity threshold cannot be copied between models. A cosine of 0.72 is a strong match in a SimCSE space and an unremarkable one in raw BERT. Always measure your own model's floor before choosing a number, a discipline Chapter 8 turns into a procedure.

What people tried instead, in 2019

The first three rows share a shape: they treat the problem as a post-processing problem. The insight of the paper is that it is a training problem, and that the training required is astonishingly cheap once you notice.

Why averaging GloVe does better than averaging BERT

This is the detail that makes the table feel unfair, so let us be precise. GloVe vectors are trained on a co-occurrence objective: the dot product of two word vectors is fit to the log of how often those words appear together. The loss is a function of a dot product between two vectors. So GloVe's geometry was directly supervised, at the word level, and the space is far more isotropic as a result.

Averaging is then a surprisingly good sentence operation for such a space: it is a low-variance estimator of "what this sentence is about", it inherits the word-level geometry, and it degrades gracefully. It throws away word order entirely — "dog bites man" and "man bites dog" get identical vectors — which caps it at 61.32, but 61.32 of honest signal beats 54.81 of coned noise.

Read the second column top to bottom and the ranking in the last column stops looking mysterious. Every method that ever had a pair-level signal in its loss beats every method that did not, regardless of how good the underlying encoder is. Encoder quality is worth something — SBERT beats InferSent by 10 points using the same NLI data, because BERT is a better encoder than a BiLSTM — but only after the pair-level pressure is applied.

Two more diagnostics worth knowing

The mean random cosine is the first thing to measure, and two others tell you about failure modes it misses.

Hubness. In high-dimensional spaces, some points become the nearest neighbour of an unreasonable number of other points — they are hubs. Count, for each vector, how many other vectors have it in their top-10. In a healthy space, that count is roughly 10 for everyone. In a coned space, a handful of vectors near the cone's axis will appear in hundreds of top-10 lists while others appear in none.

The user-visible symptom is unmistakable once you know it: the same three documents come back for everything. Teams usually diagnose this as "those documents are too generic" and try to delete them. The real cause is geometric — those documents sit closest to the mean of the space, so they are near-ish to everything. Centring fixes a good deal of it; a stronger objective fixes more.

Intrinsic dimension. Your vectors have 768 coordinates, but how many directions do they actually use? Run a PCA over a corpus sample and ask how many components explain 95% of the variance.

The middle row is why 384-dimensional models work nearly as well as 768-dimensional ones, and why Matryoshka truncation is possible at all: the extra coordinates were mostly carrying variance nobody was using. It is also a fast sanity check after any fine-tune — if the intrinsic dimension collapsed, so did your model, whatever the loss curve said.

Reading the columns, not just the average

Averages hide things. Two columns in Chapter 1's table deserve individual attention, because each teaches something the average erases.

SICK-R. Mean-pooled BERT scores 58.40 here — and averaged GloVe scores 53.76. This is the one dataset where naive BERT beats the GloVe baseline. Why? SICK-R was built by taking image-caption sentences and applying controlled linguistic transformations: passivisation, negation, quantifier substitution, word-order changes. Pairs differ by syntax at fixed vocabulary. A bag of word vectors is blind to that by construction — "a man is not playing guitar" and "a man is playing guitar" produce nearly the same GloVe average — while BERT's contextual vectors carry at least some of it. So on the one benchmark where structure matters more than vocabulary, the contextual encoder's advantage shows through even without pair supervision.

STS12. The hardest column for every model: GloVe 55.14, mean-BERT 38.78, SBERT 70.97. STS12 is the oldest and most heterogeneous of the sets — it mixes MSRpar news paraphrases with SMTeuroparl machine-translation outputs and WordNet gloss pairs, so a model must handle several genres with one metric. The 32-point spread between mean-BERT and SBERT on this column alone is the single strongest evidence for the paper's thesis, because heterogeneity is precisely what an unsupervised geometry handles worst.

That last row is the one to remember. SBERT loses to Universal Sentence Encoder on SICK-R by 3.78 points, in the paper's own headline table. The dataset that tests negation and syntactic transformation is the dataset where NLI-trained SBERT is weakest — and negation blindness is still, six years later, the most reliable complaint about sentence embeddings. The failure was visible on the day of publication, in a column most readers skip.

A note on what "worse than GloVe" does and does not mean

The headline comparison is rhetorically powerful and it can be over-read, so bound it carefully.

The distinction between rows 3 and 4 is the whole diagnosis. A representation can contain a fact and still not expose it to a particular readout. Cosine similarity is an extremely restricted readout — no parameters, no training, every coordinate weighted as given — and pretraining had no reason to accommodate it.

Which reframes what the fine-tune is doing. It is not adding knowledge; twenty minutes on a million pairs cannot teach a model what "guitar" means. It is reorganising knowledge the encoder already had into a form a parameter-free comparison can read. That is why it is so cheap, and why it works from a checkpoint rather than from scratch.

Two predecessors worth naming

InferSent (Conneau et al., 2017) is the direct ancestor. A BiLSTM with max-pooling, trained on the Stanford Natural Language Inference corpus, producing 4096-dimensional sentence vectors. It established the finding SBERT relies on: NLI is unreasonably good supervision for general-purpose sentence embeddings. SBERT is, in one sentence, "InferSent with BERT instead of the BiLSTM, plus a careful ablation of everything else."

Universal Sentence Encoder (Cer et al., 2018) is the strong contemporary baseline, and the one that makes SBERT's win non-trivial. USE trains a transformer (or a deep averaging network, for the fast variant) with multi-task supervision: SNLI plus a conversational input-response prediction task mined from web forums plus supervised classification data. It reaches 71.22 average STS — genuinely good, and it shipped as an easy TensorFlow Hub module, so it was the default choice at the time.

SBERT beats it by 3.67 points on average, and by 2.11 points on STS-B, using less supervision (NLI only) and about twenty minutes of fine-tuning on a pretrained encoder. The lesson generalises past this paper: when a strong pretrained encoder exists, the winning move is usually a small, well-shaped fine-tune, not a bigger multi-task training run from scratch.

The 2×2 that explains the whole table

There are exactly two factors at work in Chapter 1's results: how good the encoder is, and whether a pair-level loss was ever applied. Cross them:

Read it as an experiment with an interaction term. Along the top row, adding pair supervision to a weak encoder buys +3.69. Along the bottom row, it buys +20.08. Down the left column, upgrading the encoder costs you 6.51 points. Down the right column, it gains 9.88.

So the two factors are not additive at all — they multiply. A strong encoder without pair supervision is worse than useless: it has more capacity to arrange its space in a way nothing constrains, so it arranges it worse. Add the constraint and that same capacity becomes the largest gain in the table.

The five-minute experiment that makes all of this yours

Everything in this chapter is checkable on your own machine, and running it once is worth more than reading it twice. The measurement is: what does my model think two unrelated sentences score?

python — measure your own model's noise floor and coneimport numpy as np
from sentence_transformers import SentenceTransformer

sents = [# 200+ sentences sampled from YOUR corpus, not from a benchmark]
E = model.encode(sents, normalize_embeddings=True)        # (N, d)

S = E @ E.T                                                # (N, N) all-pairs cosine
off = S[~np.eye(len(E), dtype=bool)]                      # drop the diagonal of 1.0s
print("mean cosine of random pairs :", off.mean())        # the CONE. 0.0 would be ideal
print("sd                          :", off.std())         # the usable dynamic range
print("95th percentile             :", np.percentile(off, 95))  # your noise FLOOR

Ec = E - E.mean(axis=0, keepdims=True)                    # centre: subtract the corpus mean
Ec = Ec / np.linalg.norm(Ec, axis=1, keepdims=True)
Sc = Ec @ Ec.T
print("after centring              :", Sc[~np.eye(len(E), dtype=bool)].mean())

Three numbers come out, and each one answers a question you would otherwise guess at.

That last row deserves its own arithmetic, because it is where a mining job goes wrong. With 10,000 sentences there are 49,995,000 pairs. If 5% of unrelated pairs clear your threshold, that is roughly 2.5 million false positives — against perhaps a few thousand genuine duplicates. Precision would be well under one percent. On quadratic problems, a noise floor you would call "quite good" on a per-pair basis is a catastrophe in aggregate, and the only defence is to know exactly where the floor sits.

Where this leaves us

We now have a precise statement of the problem, in two halves. The speed half from Chapter 0: comparisons must happen after the network, not inside it. The quality half from this chapter: vectors compared after the network must have been trained under a loss that saw pairs. Chapter 2 puts the two halves into one design space and shows that the choice between them is not a ranking — it is a decision that depends on how many comparisons you need and how much latency you have.

Chapter 2: Cross-Encoder vs Bi-Encoder

It is tempting to read Chapter 0 as "cross-encoders are slow, use bi-encoders." That reading will cost you accuracy in production, because the cross-encoder is not merely faster-at-being-worse. On the hardest comparison tasks in this very paper, the cross-encoder wins, sometimes by a lot. This chapter maps the design space properly so that Chapter 8 can put both to work in the same system.

The cross-encoder, in tensor shapes

A cross-encoder takes both sentences as a single input sequence and outputs a score. The data flow, with real shapes for BERT-base scoring one pair:

The row that matters is the fourth. In layer 1 already, the token "crashes" in sentence A can attend to the token "force-closes" in sentence B, compare them, and write the result of that comparison into its own representation. By layer 12 the model has had twelve rounds of arbitrarily fine-grained cross-sentence reasoning: alignment of phrases, detection of negation on either side, resolution of "it" in B to "the app" in A.

The bi-encoder, in tensor shapes

A bi-encoder — also called a dual encoder, or two-tower model — runs each sentence through the encoder independently and compares the outputs.

The last row is the entire economic argument. A bi-encoder produces a per-sentence artefact. Artefacts can be stored, sorted, indexed by approximate-nearest-neighbour structures, shipped between services, and reused across every query for the rest of the model's life. That is what an index is.

Precomputability: the property that actually matters

Let us define the distinction cleanly, because "slow vs fast" is the wrong axis. Write the score function of each architecture:

The bi-encoder's score factorises: it is an inner product of two things that each depend on only one input. Every good scaling story in machine-learning systems is a factorisation story, and every factorisation buys speed by giving up expressiveness. The cross-encoder can represent any function of the pair. The bi-encoder can only represent functions expressible as an inner product of two independently-computed vectors — a strictly smaller family.

The expressiveness gap, proved

That last point can be made exactly, not just intuitively, and the proof is two lines of linear algebra worth knowing.

Consider the full score matrix S over a corpus: S_ij = s(A_i, B_j), an (n × m) table of every score the model would give. For a bi-encoder, stack the embeddings into E_A ∈ R^n×d and E_B ∈ R^m×d. Then:

The rank of a product is at most the smaller inner dimension. So no matter how good your encoder is, a bi-encoder with d = 768 can only ever produce score matrices of rank 768. A cross-encoder's score matrix has no such constraint — it can be full rank, up to min(n, m).

Make that concrete with the smallest possible counterexample. Suppose you want a scorer where A₁ matches B₁ only, A₂ matches B₂ only, and so on for n sentences — the identity matrix, rank n. With d = 2 and n = 3 you cannot do it: three points on a 2-D plane cannot each be closest to their own partner and far from the other two while satisfying the geometry, because the required score matrix has rank 3 and the achievable one has rank 2. The constraint is not about training difficulty. It is a dimension count.

The cost model, term by term

Let n be the corpus size, q the number of queries, d the embedding dimension, and F the cost of one encoder forward pass. Then:

Put numbers on the second row, because that is the row a product manager cares about. Interactive search over n = 1,000,000 documents, F ≈ 5 ms per pair on a V100:

Note the honest detail in the bi-encoder line: the brute-force scan is memory-bandwidth bound, not compute bound. One million fp32 vectors of 768 dimensions is 1,000,000 × 768 × 4 = 3.07 GB, and you have to stream all of it through the ALUs. At 50 GB/s of usable bandwidth that is 61 ms, which matches the estimate above. Store in fp16 and you halve it; store in int8 and you quarter it. This is the kind of realisation detail that decides whether the feature ships.

Batching, and why the reranker's cost is not linear in k

One more piece of realism, because "5 ms per pair" is not how a GPU behaves. A GPU is a throughput machine: it processes a batch of 32 pairs in barely more time than a batch of 1, because the fixed costs — kernel launches, weight loads from HBM — dominate until the arithmetic units are saturated.

Three consequences follow. First, reranking 32 candidates costs almost the same as reranking 1, so if you are going to pay the overhead at all, take k = 32 rather than k = 5 — the extra 27 candidates are nearly free. Second, going from k = 32 to k = 128 is a genuine 3× in latency, so that is where the real tradeoff sits. Third, the same effect makes index building vastly cheaper than a per-sentence estimate suggests: encode with batch 128 and sorted lengths (smart batching, Chapter 7) and you approach the model's peak throughput rather than its per-call latency.

Eight real products, and which design each needs

The abstract rule is "how many comparisons, and how much latency." Applied:

Two patterns fall out. When the comparison set is small and fixed — a handful of labels, one reference answer, two documents — the cross-encoder is not merely allowed, it is the correct default, because the quadratic never engages and you are leaving accuracy on the table by not using it. When the comparison set is the corpus, the bi-encoder is not one option among several; it is the only thing that runs.

So when does each win?

Worked latency budget. Suppose your p95 budget is 120 ms and the cross-encoder costs 5 ms per pair with a batch of 32 running in 12 ms total (batching wins hard here). Then reranking k candidates costs about ⌈k/32⌉ × 12 ms. With k = 100 that is 4 batches × 12 = 48 ms. Add 10 ms for retrieval and 5 ms for the query encode: 63 ms. You have 57 ms of headroom, so you could push k to 200 (7 batches, 84 ms → 99 ms total). Beyond that you are out of budget, and each doubling of k buys less because the retriever's recall curve flattens.

Where the cross-encoder's cost actually goes

One more detail decides real budgets. A transformer layer has two cost terms: the position-wise linear projections, which are linear in sequence length L, and the attention matrix, which is quadratic in L.

Now compare two designs on the same two 64-token sentences. The bi-encoder runs two passes at L = 64. The cross-encoder runs one pass at L = 128. The linear terms come out equal — 2 × 64 = 128 tokens either way. The attention terms do not:

So even for a single pair the cross-encoder costs more, by the factor that the quadratic term contributes. That is a second-order effect — the first-order disaster is still that it runs n² times instead of n — but it is the reason long-document cross-encoding is punishing, and the reason rerankers truncate aggressively.

What a query costs in money. Take a rented A10G at roughly $1/hour, and a reranker throughput of about 800 pairs per second at 128 tokens:

Four cents per thousand queries for near-cross-encoder quality, against a number with six digits in it. The architecture decision in this chapter is usually the largest single lever on the cost of a search product.

The cross-encoder you will actually build

Since the recommended architecture is "both," it is worth knowing what the second stage looks like in practice. A reranker is a smaller model than you expect and its training data is a different shape from the retriever's.

python — training and using a rerankerfrom sentence_transformers import CrossEncoder, InputExample

# Training data is (query, passage) -> relevance. The NEGATIVES must come from
# your retriever's own top-k, not from random sampling: the reranker only ever
# sees what stage 1 sends it, so that is the distribution it must be good on.
train = [InputExample(texts=[q, pos], label=1.0) for q, pos in positives] + \
        [InputExample(texts=[q, neg], label=0.0) for q, neg in retriever_top_k_wrong]

model = CrossEncoder('microsoft/MiniLM-L6-H384-uncased', num_labels=1, max_length=256)
model.fit(train_dataloader=loader, epochs=1, warmup_steps=1000)

# Inference: the scores are NOT probabilities and NOT comparable across queries.
scores = model.predict([(query, p) for p in candidates])   # use for ORDERING only
order  = np.argsort(-scores)

The middle ground, for completeness

The design space is not actually binary. Two families sit between the extremes, and knowing they exist keeps you from treating SBERT's tradeoff as a law of nature.

ColBERT keeps a vector per token and defines the score as the sum over query tokens of the maximum similarity to any document token. That recovers much of the fine-grained alignment a cross-encoder gets, while still factorising — the document's token vectors can be precomputed. The price is storage: a 100-token document costs 100 vectors instead of 1. The design space is a smooth trade between how much interaction you keep and how much you must store.

A vocabulary note, because the names are a mess

Four names for one thing, and the distinctions that do carry information are: tied versus untied weights, and whether the comparison happens before or after the encoder finishes. Everything else is which conference the author attends.

They also differ in what they need from you

One last axis, which decides more projects than accuracy does: the two architectures have different data requirements, and different failure modes when that data is thin.

Row 2 is the one that surprises people, and it is worth stating plainly because it inverts a normal intuition. For most models, batch size is a systems parameter: bigger is faster, and you adjust the learning rate. For a bi-encoder trained with in-batch negatives, doubling the batch doubles how many wrong answers each example is contrasted against, which makes the task strictly harder and the resulting space strictly better-separated. It is the one place where "we ran out of GPU memory" is a modelling problem, not an infrastructure one. Chapter 9 returns to it.

A worked recall/latency tradeoff

Chapter 2's central decision is "how many candidates does stage 1 hand to stage 2." Here is that decision as numbers, using a retriever whose recall curve has the shape every retriever's does — steeply rising, then flat.

Two shapes to read off. Going from 10 to 32 is free, because of the batching effect above — if you are reranking at all, never rerank fewer than one full batch. And beyond about 128 you are buying hundredths of recall with tens of milliseconds, which is where the curve says stop.

The subtler point is in the fourth column's heading: ceiling. Reranking cannot exceed the retriever's recall@k, so this table bounds what the whole pipeline can do no matter how good stage 2 becomes. If your product needs 0.95 end-to-end and your retriever tops out at 0.90 for any affordable k, the fix is a better retriever — or a hybrid that adds BM25's recall to the dense recall, which is usually the cheapest way to raise a ceiling.

The stage nobody puts in the diagram: caching

Retrieve-then-rerank is usually drawn as two stages. Production systems have three, and the missing one is free.

An exact-string cache catches "how do i reset my password" twice. A semantic cache also catches "password reset" and "cant log in need new password" — and query distributions are heavily skewed, so hit rates of 30–60% are ordinary. Every hit skips both expensive stages entirely.

The threshold is the whole design, and it is asymmetric in consequence: too low and users get someone else's answer, which is a correctness bug; too high and you merely lose hits, which is a cost. So set it high (0.97–0.99 measured against your own noise floor from Chapter 1's experiment), and note that this is one of the few places where an absolute cosine threshold is genuinely the right instrument — because both sides are queries, drawn from the same distribution, encoded by the same function, so the comparison is as apples-to-apples as this geometry ever gets.

The cousin: when you should untie the towers

Chapter 3 will argue hard for weight tying. It is worth knowing now when the opposite is right, because the same two-tower diagram appears in recommender systems with the weights deliberately separate.

The fourth row hides a real engineering advantage. Document encoding happens offline, so it can use a large model; query encoding happens in the request path, so it wants a small one. Untying lets you spend asymmetrically, exactly matching where the latency is.

Modern text embedders take a third option that is neither: tied weights with an asymmetric prefix. One encoder, but queries are prepended with "query: " and documents with "passage: ", so the same parameters compute two different functions selected by a token. You get the asymmetry without doubling the model or risking divergent towers — and, as Chapter 9 notes, it is worth several points of recall on retrieval tasks that SBERT-NLI handles poorly.

Chapter 3: Twins and the Pooling Layer

We know what we want: an encoder f whose outputs can be compared by cosine. We know why plain BERT does not give it: no pair-level pressure was ever applied. So apply some. The structure that applies it is sixty lines of code and one idea from 1993.

What a siamese network is

A siamese network (Bromley, Guyon, LeCun, Säckinger, Shah, 1993 — built to verify handwritten signatures) is a network applied twice, to two inputs, with the same weights both times, followed by a loss computed on the two outputs. The word "siamese" refers to twins, and the essential property is that the twins are not merely identical — they are the same object. There is one set of parameters, one gradient buffer, one saved checkpoint.

That last line is the whole mechanism, so read it as an instruction rather than an equation. Sentence A's pass says "adjust θ so my vector moves toward B's." Sentence B's pass says "adjust θ so my vector moves toward A's." Because θ is shared, the two demands are resolved inside a single parameter update. The encoder cannot satisfy them by specialising — it has to find a representation function under which semantically related inputs land near each other in general.

Where the pooling layer goes, with shapes

SBERT is BERT plus one parameter-free layer. Here is the full forward pass for a batch of b sentences padded to length L:

The pooling layer has zero parameters. That is a deliberate and slightly surprising choice: you might expect a learned attention-pooling head. The paper's ablation (below) says the simplest option wins, and there is a good reason — a parameterised pooler is one more thing that has to be learned from a small amount of pair data, and one more thing that can overfit the training genre.

The three pooling strategies, written out

Let H ∈ R^L×768 be the token vectors and m ∈ {0,1}^L the attention mask.

Read the MAX definition carefully, because the subscripts hide something important. The maximum is taken per dimension, independently. Dimension 0 of the output might come from token 3, dimension 1 from token 7, dimension 2 from token 3 again. The output vector is a Frankenstein assembled from different tokens' coordinates and corresponds to no token at all. That is fine — InferSent used exactly this and it worked well for a BiLSTM — but it is a stranger operation than it looks.

The ablation that chose MEAN

The paper tests all three, under both of its training regimes, and reports Spearman correlation on the STS benchmark dev set:

Two very different stories in these two columns, and the difference is more instructive than the winner.

Under NLI training, the three are close — 80.78, 79.80, 79.07, a spread of 1.7 points. The classification objective (Chapter 4) puts a trainable matrix on top of the pooled vectors, and that matrix can compensate for a mediocre pooling choice. It is the "forgiving consumer" from Chapter 1 all over again.

Under STS-B regression training, MAX collapses — 69.92 against MEAN's 87.44, a 17.5-point hole. The regression objective optimises cosine similarity directly, with no trainable head to absorb anything. So this column is the one that measures pooling quality honestly, and it says MAX produces a geometry that cosine cannot read.

Make that concrete with arithmetic. Suppose each coordinate of each token vector is roughly standard normal. The expected maximum of L standard normal draws grows like √(2 ln L):

Read down the second column: the max-pooled vector of a 128-token sentence has coordinates twice the size of a 6-token sentence's, purely from length, in every dimension at once — which is exactly a shared component. Read down the third: the mean's expectation does not move at all. Only its variance shrinks with length, which is a benign effect (longer sentences get more stable estimates, not systematically bigger ones).

Verify the direction of the bias with a two-token toy. Tokens (2, 0) and (0, 2). MEAN gives (1, 1), norm √2 = 1.41. MAX gives (2, 2), norm 2.83. Add a third token (1, 1): MEAN becomes (1, 1) still — unchanged — while MAX stays (2, 2). Add a fourth token (2.5, 0.5): MEAN goes to (1.375, 0.875), MAX jumps to (2.5, 2). The max only ever moves up, and it moves up every time you add a token. That is a ratchet, and it is pointed at the wrong quantity.

And why does CLS trail MEAN even under regression (86.62 vs 87.44)? Because [CLS] is a single position that has to have learned to aggregate. Fine-tuning does teach it to, which is why it is only 0.8 points behind rather than 30 points behind as it was without fine-tuning (Chapter 1's 29.19). But MEAN gets aggregation for free, from arithmetic rather than from parameters, and free things do not need to be learned from your small pair dataset.

Three experiments worth running before moving on. (1) Switch to MAX and add padding with the mask off — the cosine jumps, because both sentences inherit the same padding-driven outliers. (2) Switch to CLS and add padding: nothing changes, because [CLS] is position 0 and padding is at the end. CLS is the one strategy that is immune to the masking bug, which is a genuine argument in its favour that the accuracy table does not show. (3) Compare MEAN with and without the mask on the 2-pad setting; the shift is small here with 8 dimensions and 2 pads, and in a real batch with 40 pads out of 64 positions it is not small at all.

Masked mean, computed by hand

The formula is short enough that it hides its own subtlety, so do one all the way through. Three real tokens and two padding positions, in three dimensions:

Compare them with the tool we care about:

The same sentence, encoded two ways, at cosine 0.677 — which is roughly where an unrelated sentence would sit in a well-trained space. And there were only two padding positions. In a real batch padded from 6 tokens to 128, the buggy vector is 95% padding and the cosine to the correct one approaches whatever the [PAD] vector's own direction happens to be.

The complete SBERT module, in code

Nothing in this chapter needs a framework to understand, so here it is as it actually exists. The pooling layer is nine lines.

python — the entire SBERT moduleclass Pooling(nn.Module):
    def forward(self, token_embeddings, attention_mask):
        # token_embeddings: (b, L, 768)   attention_mask: (b, L)
        mask = attention_mask.unsqueeze(-1).float()      # (b, L, 1) - broadcasts over 768
        if self.mode == 'mean':
            summed = (token_embeddings * mask).sum(dim=1)    # (b, 768)
            counts = mask.sum(dim=1).clamp(min=1e-9)          # (b, 1) - never divide by zero
            return summed / counts
        if self.mode == 'cls':
            return token_embeddings[:, 0]                   # (b, 768)
        # max: kill the padded positions before the max, do not just ignore them
        masked = token_embeddings.masked_fill(mask == 0, -1e9)
        return masked.max(dim=1).values                    # (b, 768)

# The whole model:
model = nn.Sequential(Transformer('bert-base-uncased'), Pooling(mode='mean'))

And because the pooling mode is a contract rather than a weight, it is serialised alongside the model. A published sentence-transformer is a directory whose modules.json lists the pipeline in order:

modules.json — the model IS this list[
  {"idx": 0, "name": "0", "path": "",          "type": "models.Transformer"},
  {"idx": 1, "name": "1", "path": "1_Pooling", "type": "models.Pooling"},
  {"idx": 2, "name": "2", "path": "2_Normalize", "type": "models.Normalize"}
]
# 1_Pooling/config.json
{"pooling_mode_mean_tokens": true, "pooling_mode_cls_token": false,
 "pooling_mode_max_tokens": false, "word_embedding_dimension": 768}

Module 2 is worth noticing: a Normalize layer that divides by the L2 norm. Whether a checkpoint includes it decides whether model.encode() hands you unit vectors, and therefore whether a raw dot product equals cosine downstream. Two checkpoints that differ only in that third entry will behave identically on ranking within one query and differently the moment you compare scores across queries or apply a threshold. It is the highest ratio of consequence to visibility anywhere in this stack.

Count the new parameters: zero. SBERT-base has exactly the parameter count of BERT-base (110M), plus, during training only, the small classification matrix of Chapter 4 which is thrown away afterwards. The published model is a BERT with different weights and a documented pooling convention. That is why it drops into any BERT-shaped inference stack without changes — and why forgetting to apply the same pooling at query time silently ruins retrieval, since nothing errors.

What happens to the norms

Pooling determines not just direction but length, and length has consequences even though cosine is supposed to ignore it. Take a sentence of L tokens whose token vectors have typical norm r and are only partially correlated with each other. Two extremes bracket the answer:

Real sentences sit between: tokens share a large common component (the cone) plus individual content. So the pooled norm shrinks with length, but slowly — and it shrinks more for sentences whose tokens are semantically diverse than for repetitive ones.

Read the third row against the first: the paragraph that actually covers your query's topic is penalised for also covering four others, while a repetitive fragment is rewarded for saying one thing four times. That is a ranking driven by variance, not relevance, and it disappears entirely the moment you L2-normalise. It is also the deep reason chunking helps — a chunk has low internal diversity, so its pooled vector is both longer and more sharply directed.

The masking bug, in numbers

Chapter 3's callout claims that an unmasked mean makes an embedding depend on its batch-mates. Price it exactly. Take a 6-token sentence whose real tokens have mean vector r, batched with a 64-token sentence so it is padded to 64. Let p be the (nonzero, learned) [PAD] vector. The masked and unmasked means are:

Ninety-one percent of the "sentence embedding" is the padding vector. Every short sentence in that batch converges toward the same point, which is p — so all of them become mutually similar and nearly identical, and their cosine to each other approaches 1 regardless of content. If the same sentence is later encoded alone (L = 6, no padding) it gets u = r instead. Two encodings of one sentence, cosine between them possibly below 0.3.

What the pooling layer does to the gradient

A parameter-free layer still shapes learning, because it decides how the loss's gradient is distributed back over the tokens. Differentiate each strategy with respect to the token vectors:

Three genuinely different learning dynamics fall out of three one-line derivatives.

MEAN spreads the signal evenly, so every token in the sentence is nudged a little on every step. Dense gradients, stable, no token is privileged — and no token can be ignored either, which is precisely why negation is hard: "not" gets 1/L of the correction, exactly like "the".

CLS routes everything through position 0. Note the subtlety though: the gradient reaching other tokens is not zero, it just arrives indirectly, through the self-attention that built H₀ from them. So the model must learn to aggregate as well as to represent, which is more to learn from the same pair data. That extra burden is the 0.8-point gap in the ablation table.

MAX gives each dimension's entire gradient to a single token, and the winner takes everything. Early in training the argmax flips constantly as the encoder shifts, so the gradient path is discontinuous — a token that received a large update at step t may receive nothing at step t+1. Sparse, high-variance updates, which combine badly with the length bias already derived.

Two pooling variants you will be offered, and when to take them

The second row is worth internalising because it explains a real difference between model families. In a causal LLM used as an embedder, token i has only seen tokens 1…i, so the mean over positions averages a lot of half-formed representations — which is why last-token pooling is standard there. BERT is bidirectional, so position 3 has attended to position 40 since layer 1, and every position is a legitimate view of the whole. That is why MEAN is a good idea here and a poor one there. The pooling choice is downstream of the attention mask used in pretraining.

The pooling layer's real job, restated

Step back from the ablation for a moment. What is pooling actually being asked to do?

BERT gives you L vectors of 768 numbers — for a 64-token sentence, 49,152 numbers describing every token in its context. You must produce 768. That is a 64:1 compression, performed with no parameters and no knowledge of what will be asked.

Stated that way, the surprise is not that pooling loses word order and negation. The surprise is that it preserves anything useful at all. And the reason it does is that the token vectors are contextual: by layer 12, the vector above "bank" has already absorbed "river" or "money" from elsewhere in the sentence. The averaging is not over words, it is over 64 partially-redundant views of the whole sentence, each written from a different position's vantage. Averaging redundant estimates is exactly what averaging is good at.

Triplet structure: the same idea with three towers

For the triplet objective (Chapter 4) the network runs three times instead of two — anchor, positive, negative — still with one shared θ. Nothing else changes. It is the same siamese principle; "siamese" and "triplet" describe how many passes share the weights, not different architectures.

Where the 768 comes from, and whether it matters

One number in this architecture is inherited rather than chosen, and it is worth saying so. SBERT's embedding dimension is 768 because BERT-base's hidden size is 768; the pooling layer cannot change it, having no parameters. BERT-large gives 1024 for the same reason.

So the dimension was picked in 2018 for reasons about transformer capacity that have nothing to do with similarity search. It is not tuned, not optimal, and not sacred:

Chapter 2's rank bound says dimension is a hard ceiling on expressiveness, and Chapter 1's intrinsic-dimension measurement says real spaces use far less than they have. Both are true, which is why the practical answer is "as small as your recall target tolerates" — and why Matryoshka training, which lets one model serve several dimensions, was such a natural idea once someone thought of it.

Five assertions for a pooling implementation

Pooling is nine lines and every one of its failure modes is silent, which is an unusually bad combination. These five checks take ten minutes to write and cover all of them.

The second row is worth dwelling on. model.eval() disables dropout; forget it and every encode of the same sentence differs, typically at cosine 0.97–0.99 — close enough that nothing looks broken and far enough that your near-duplicate threshold becomes meaningless. It is the same mechanism SimCSE exploits deliberately to manufacture positive pairs, which is a pleasing symmetry: one paper's bug is another's training signal, and the only difference is whether you meant it.

Why no learned projection?

A design contrast worth drawing, because the neighbouring literature made the opposite choice. CLIP and CLAP — the vision-language and audio-language contrastive models — put a learned linear projection after their encoders, mapping each modality into a shared space of a chosen dimension. SBERT does not. Why?

The projection in CLIP exists to solve a problem SBERT does not have. Two encoders that have never met need a learned map into a common frame; two applications of one encoder are already in a common frame. Adding a projection anyway would only give the model an easy place to undo its own geometry — and would introduce a layer that has to be learned from the pair data, with everything that implies about overfitting.

Chapter 4: Three Objective Functions

The siamese structure delivers two vectors, u and v, and holds one set of weights responsible for both. Now we need a loss — a number to minimise that encodes what "these two sentences are related" should mean geometrically. The paper defines three, and it matters that they are three, because they attach to three different shapes of supervision you might have lying around.

Objective 1 — classification, and the feature that carries it

Given u and v (each 768-dimensional after pooling), form a single feature vector by concatenating three things: u, v, and the element-wise absolute difference |u − v|. Multiply by a trainable matrix and softmax:

Then cross-entropy against the gold label. Six thousand nine hundred and twelve parameters, on top of a 110-million-parameter encoder, and they are discarded after training — W_t exists only to create gradient pressure on the encoder. The shipped model is the encoder and the pooling layer, nothing else.

First, where δ = p − y comes from. This identity is used everywhere and derived almost nowhere, so derive it once. With logits z, probabilities p_i = e^z_i/∑_je^z_j, and loss L = −log p_c for the true class c:

Two terms: the first pulls the true class's logit up, the second pushes every logit down in proportion to how much probability it currently holds. That is the entire behaviour of softmax cross-entropy, and it is why a confidently-wrong class absorbs most of the correction while a class at p = 0.001 absorbs almost none. Chapter 9's InfoNCE inherits exactly this property, which is where its automatic hard-negative weighting comes from.

Now, why |u − v| is the load-bearing feature. Write the concatenated feature as x = [u; v; d] with d = |u − v|, split W_t into three column blocks W_u, W_v, W_d (each 768×3), and use δ = (p − y). We need ∂d/∂u: since d_i = |u_i − v_i|, and |x| has derivative sign(x), the Jacobian is diagonal with entries sign(u_i − v_i). So the gradient arriving at u is:

Two terms with completely different characters. The first, W_uδ, is the same vector whatever v is — it pushes u in a direction that depends only on the label. It says "sentences with an entailment partner should live over there," which is a weak, absolute instruction.

The second term is the interesting one. The factor sign(u − v) is +1 in every coordinate where u exceeds v and −1 where it does not. So the update is relative: in each coordinate it pushes u toward or away from v depending on which side of v it currently sits. That is a distance-shaping instruction — the first thing in this entire lesson that actually moves two sentences relative to each other.

The paper's ablation settles it. With MEAN pooling, trained on NLI, evaluated on STS-B dev:

Read rows one and two together, twice if necessary. Dropping u and v entirely and keeping only their absolute difference — halving the feature width, throwing away every absolute fact about either sentence — improves the result by 3.74 points. The information the classifier needed was never in the individual vectors; it was in their relationship.

And the last row is a lovely negative result: the model has all the information from row 6 plus the element-wise product, and it does no better at all. The reason is an identity. Coordinate-wise:

Given u, v, and |u − v|, the product u ∗ v is recoverable by squaring and rearranging — not by a linear map, which is why the ablation is not exactly zero-information, but by a computation the network above easily has. Adding a feature the model can already derive buys nothing, and the table confirms it to two decimal places (80.78 either way).

Objective 2 — regression on cosine, and its gradient

When your labels are graded — STS pairs come with human scores from 0 (unrelated) to 5 (equivalent) — you can dispense with the classifier entirely:

Zero new parameters. Train-time and test-time are now identical, which is the cleanest possible situation.

Derive the gradient, because its shape is genuinely surprising and it explains what cosine training does to a space. Let a = ‖u‖, b = ‖v‖, and c = cos(u,v) = (u·v)/(ab). Differentiate the quotient with respect to u, using ∂‖u‖/∂u = u/a:

where û and v̂ are the unit vectors. Now check something: what is the component of this gradient along u itself? Dot it with û:

Exactly zero, always. The cosine gradient is orthogonal to u. It can only rotate u; it can never lengthen or shorten it. That is a deep and useful fact: the regression objective sculpts directions and is completely blind to magnitudes. Vector norms drift under it as an unconstrained side effect, which is one reason practitioners L2-normalise at index time regardless of what the training did.

Note also the 1/a factor. A long vector receives a proportionally smaller angular update from the same error — longer vectors are harder to steer. If a subset of your sentences (say, long ones) systematically has larger norms, they will move more slowly during training. Normalising before the loss removes this asymmetry entirely, and is a small, real improvement over the paper's default.

Objective 3 — triplet, and the geometry of a margin

Sometimes you have neither labels nor grades, only structure: three sentences from the same Wikipedia section are related, one from another section is not. The triplet objective takes an anchor a, a positive p, and a negative n, and demands that the anchor be closer to the positive than to the negative by a margin:

Unpack the hinge. If the positive is already closer than the negative by more than ε, the bracket is negative, the max returns 0, and the gradient is exactly zero — that triplet is solved and contributes nothing. If it is not, the loss is the shortfall, and the gradient pulls s_p toward s_a while pushing s_n away.

Why a margin at all? Without ε, the loss ‖s_a−s_p‖ − ‖s_a−s_n‖ is minimised by making the second term enormous, and there is a degenerate solution where the encoder simply blows up the scale of everything. The margin makes the objective satisfiable: once the ordering holds with a gap of ε, stop. It converts an unbounded optimisation into a constraint, and the hinge is what turns "solved" into "silent."

The relation to cosine, which ties this objective back to inference. If all vectors are L2-normalised, then:

So on the unit sphere, squared Euclidean distance is a strictly decreasing function of cosine similarity: ranking by one is identical to ranking by the other. Training with Euclidean triplets and serving with cosine is therefore consistent — provided you normalise. Without normalisation the two rankings genuinely differ, and this mismatch is a classic silent bug in retrieval systems.

Worked numbers on a margin. Take normalised vectors with cos(a, p) = 0.80 and cos(a, n) = 0.50. Then:

Still a positive loss, even though the ordering is already correct by 0.368. With ε = 1 on unit vectors the largest achievable separation is ‖a−n‖ ≤ 2, so the constraint "positive distance + 1 ≤ negative distance" demands a very wide gap indeed. This is a real subtlety: a margin of 1 is aggressive for normalised embeddings and mild for unnormalised ones. The margin's meaning depends entirely on the scale of your space, which is why triplet training is famously fiddly, and why in-batch softmax contrastive losses (which are scale-free after normalisation, up to a temperature) largely replaced it after 2020.

The cosine gradient, on numbers. Take 2-dimensional unit vectors u at 0° and v at 60°, with gold y = 1. Then cos = 0.5, and:

Descent moves u by −η∂L/∂u, i.e. in the direction (0, +0.866) — straight up, perpendicular to u, which rotates it toward v at 60°. Exactly as the derivation promised: no component along u, so the length is untouched and only the angle changes. Take η = 0.1 and u becomes (1, 0.0866), whose angle is 4.95° — it has rotated about 5° of the 60° gap in one step, and its norm has grown to 1.004 only because a finite step leaves the tangent line, not because the gradient had a radial part.

That last detail is a real effect at scale: repeated finite steps along tangents slowly inflate norms even though every gradient is orthogonal. It is harmless if you normalise before comparing, and confusing if you do not.

Set the positive to 35° and the negative to 95° and step through the three objectives. Under regression with a gold score of 1.0 the loss is (cos 35° − 1)² = (0.819 − 1)² = 0.0327 and the gradient rotates v toward u. Under triplet with ε = 1 the loss is positive even though the ordering is correct — drag the negative out to 150° and watch it hit exactly zero, at which point that triplet stops teaching the model anything at all. That dead zone is why triplet training needs hard negative mining: as the model improves, a randomly chosen negative satisfies the margin almost always, gradients go to zero, and learning stalls. You must actively search for negatives that are still confusable.

Why the product feature alone reaches 70.54

One ablation row deserves an explanation rather than a shrug: (u ∗ v) alone scores 70.54, beating |u − v| alone at 69.78 and both raw vectors together at 66.04. What does an element-wise product know?

Sum its coordinates and you get the dot product: ∑_i u_iv_i = u · v. So the product feature is an un-summed dot product — it hands the classifier every term of the similarity separately, letting it learn which dimensions should count and how much. It is cosine similarity with per-dimension learned weights, which is strictly more expressive than cosine.

The difference feature carries related but not identical information. Compare what each says about a single coordinate:

Rows 1 and 2 are the crux: the difference cannot distinguish agreement-at-high-magnitude from agreement-at-low-magnitude, and the product can. Conversely, the product cannot distinguish (3, 1) from (1, 3) — it is symmetric — while the difference at least records the gap. They are complementary views, which is why the two together reach 78.37, well above either alone.

And yet the paper's choice, (u, v, |u − v|), beats that pair at 80.78. Why? Because with u and v both present, the classifier can recover magnitude information itself — the thing the difference was missing. The winning combination is not "the two best features"; it is a set that is jointly sufficient. Feature selection is about coverage, not about individual scores, and this ablation is an unusually clean demonstration.

The objective that replaced all three

Everything above is 2019. It is worth seeing the successor now, because it makes the three objectives' tradeoffs legible and because it is what you would actually write today. The multiple-negatives ranking loss (also called InfoNCE, or in-batch softmax) needs only positive pairs: (a₁, b₁), …, (a_N, b_N) in a batch. For row i, b_i is the positive and every other b_j is a negative.

Work a batch of three. Suppose the cosines of a₁ against the three candidates are (0.62, 0.55, 0.10) — the true partner first, a confusable one second, an easy one third. Multiply by 1/τ = 20:

Now read the gradient allocation from δ = p − y = (−0.1978, 0.1978, 0.0000244). The confusable negative absorbs 0.1978 of repulsive force; the easy negative absorbs 0.0000244 — eight thousand times less. Nobody mined those negatives, nobody set a margin, and nobody sorted them by difficulty. The softmax did it, because its denominator weights each competitor by the probability it currently holds. That is the property triplet loss lacks, and the reason triplet was abandoned.

Notice that temperature does the job the margin was doing, but on a quantity with a fixed scale. Cosine lives in [−1, 1] whatever your embeddings do, so τ = 0.05 means the same thing in every model — unlike ε = 1, which as the arithmetic above showed is aggressive on unit vectors and mild otherwise. Choosing a hyperparameter on a normalised quantity is a small design decision with an outsized effect on how transferable a recipe is.

The three objectives, compared as gradient machines

Set the losses side by side by the only property that matters during training: which examples still produce a gradient.

Row 3 is the operational one. A loss that goes exactly to zero is a loss that stops teaching, and as the encoder improves, an ever-larger fraction of randomly-sampled triplets fall into that dead zone. Late in training you can be computing thousands of forward passes per step that contribute literally nothing to the gradient — burning GPU on satisfied constraints. That is the specific pathology in-batch softmax losses were invented to avoid: their denominator always has something in it, so every example keeps contributing, weighted by how threatening it is.

What all three objectives have in common

Strip away the details and the same skeleton is underneath each one, which is the thing to remember when you meet a fourth.

Step 2 is the whole content of Chapter 1's diagnosis, appearing here as a design rule. Masked language modelling has steps 1, 3 and 4 and no step 2 — and that missing step is worth twenty Spearman points. Any new objective you encounter can be read against this skeleton: find its step 2, and you know what geometry it is building.

Training configuration, from the paper

Twenty minutes. That is the entire training cost of the fix, on top of a pretrained encoder. Chapter 0's 46,700× speed-up and Chapter 1's 20-point quality jump are both bought with a single-GPU job shorter than a lunch break, which is the strongest possible evidence that the bottleneck was never compute — it was that nobody had applied pair-level pressure to a good encoder.

Where the gradient goes after u and v

Everything derived so far stops at the pooled vectors. One paragraph on what happens below them, because it explains the learning-rate choice in the next section.

Pooling divides ∂L/∂u evenly among the L token positions (Chapter 3's derivative), so each token vector receives 1/L of it. That signal then flows back through twelve transformer layers to every weight in the model — and it arrives twice, once from each siamese pass, into one shared θ.

A uniform learning rate of 2e-5 across all of them is a compromise: small enough that the lower layers are not damaged, large enough that the upper ones can move. Layer-wise decaying rates (larger at the top, smaller at the bottom) are a standard refinement worth a fraction of a point — and a good illustration that "one learning rate" is a simplification everyone accepts rather than a principled choice.

Choosing the one hyperparameter each objective has

Each objective exposes exactly one scale knob, and the procedure for setting it is the same in all three cases: relate it to a quantity you can measure in your own data, rather than copying a number from a paper trained on different vectors.

Row two is the one people get wrong most often, and the second sentence explains why a naive regression fine-tune can make a model worse. Gold labels rescaled to [0, 1] tell the model that unrelated pairs must reach cosine 0.0. In an anisotropic space where the floor is 0.3, that instruction is unsatisfiable for most pairs, so the optimiser spends its capacity flattening the space in a way that damages the pairs it was getting right. Mapping the gold range onto the model's achievable range instead — or simply centring first — removes an impossible demand.

What the paper did not try

Reading a paper for what is absent is often more instructive than reading it for what is present, and in this case each gap became somebody's follow-up.

Row 4 is the subtlest and worth a moment. Chapter 7 evaluates with Spearman — a rank correlation — while objective 2 minimises squared error on cosine values. A model could rank every pair perfectly and still carry loss, because its values are systematically compressed; and a model could match the values well while inverting a few near-ties. The mismatch is small, which is why it took three years for anyone to attack it, but it is the same species of train-test gap the paper explicitly calls out for the classification objective. Every objective in this chapter has one; the question is only how big.

Chapter 5: Why Entailment Teaches Meaning

Here is a thing that should strike you as odd. To build a model that answers "do these two sentences mean the same?", the paper trains on a dataset that answers a different question entirely: "does the first sentence imply the second?" Those are not the same relation. Entailment is directional and asymmetric; similarity is neither. And yet NLI supervision produces the best general-purpose sentence embeddings of its era, and continues to do so — it is still in the training mix of modern models six years later. This chapter is about why that works.

What the data is

A concrete triple, of the kind that fills SNLI. Premise: "A man is playing a guitar on stage."

Compare with the alternative supervision available in 2019. Paraphrase corpora (positives only, and often near-duplicate positives that teach nothing). Question-answer pairs (positives only). Random negatives sampled from the corpus (trivially easy — a random other sentence is about a different topic, so the model learns topic matching and stops). NLI hands you a million examples with hard negatives already written by humans who were trying to be adversarial. That is an unusual gift.

What the model is forced to learn

Put the classification objective from Chapter 4 next to this data and follow the pressure. For the contradiction pair, the classifier sees [u; v; |u − v|] and must output "contradiction." For the entailment pair it must output "entailment." The two hypotheses have nearly identical surface forms. So the difference vector |u − v| must come out substantially different in the two cases, and the only way to achieve that is for the encoder to place "playing an instrument" near "playing a guitar" and "sleeping" far from it.

Ask what would happen if the encoder took the lazy route and encoded surface form. Then |u − v| would be small for all three pairs, since all three hypotheses look alike, and the classifier would face three different labels on nearly identical inputs. Its loss cannot go down. Backpropagation's only route to a lower loss runs through the encoder learning to separate the sentences by what they assert.

Real examples, across MultiNLI's genres

Abstract descriptions of a dataset are much less useful than a page of it. These are representative of what the encoder is being asked to separate:

Read the telephone row carefully. The premise is disfluent spoken transcript with no punctuation and a stammer; the hypothesis is clean written English. They mean the same thing and share almost no surface form. That is a hard positive, and it is the mirror image of the hard negatives — supervision that forces the encoder past style and register into content. SNLI alone, being all image captions, contains nothing like it. It is a concrete instance of why the genre mixture matters.

And the travel row shows the reasoning depth that is occasionally required: to see the contradiction you must know that "daily from nine" includes mornings. No amount of lexical overlap detects that, and honestly, no bi-encoder reliably will either — but the pressure to try is what pushes the representation toward content.

What happens if you train longer

One epoch looks like an under-training decision until you see what more does. The qualitative pattern, consistent across the literature on fine-tuning pretrained encoders on a narrow objective:

The shape of that table — target-task metric rising monotonically while transfer peaks early and then falls — is the signature of overfitting to a proxy. NLI accuracy was never the goal; it is scaffolding, exactly like the classifier matrix W_t. Optimising a proxy past the point where it correlates with the real objective is one of the most common ways to make a model worse while every number on your dashboard improves.

The artefact problem, and why the siamese structure survives it

Now the complication. NLI datasets are famous for annotation artefacts: statistical regularities in the hypotheses that a model can exploit without reading the premise at all. Because crowdworkers producing contradictions reach for negation, and workers producing entailments reach for generic superordinates, the hypothesis alone leaks the label.

The effect is large: a model given only the hypothesis, never seeing the premise, reaches roughly 67% on SNLI where chance is 33%. Two-thirds of the task is solvable without doing the task.

The ablation from Chapter 4 is the receipt: (u, v) alone scores 66.04, which is the shortcut regime — and the model is still reading a hypothesis-only signal for a large part of that. Adding |u − v| takes it to 80.78. The 14.74-point gain is the shortcut being closed off.

Why not paraphrase corpora, or web pairs?

An obvious alternative in 2019 was to train on data that matches the target task directly: paraphrase pairs. Here is the accounting of why NLI wins anyway.

Every row but the last is missing the same thing: supplied hard negatives. Random negatives from a corpus are almost always about a different topic, so the model can satisfy the objective by learning coarse topic separation, at which point the gradient dies and the fine structure never gets built. NLI's contradiction column is a million hand-written, same-topic, opposite-meaning negatives. That is the asset, and it is the reason a dataset built for a completely different research question turned out to be the best sentence-embedding corpus of its decade.

The transfer that shouldn't work, and roughly why it does

Be honest about the mismatch, because it is real. Entailment is asymmetric: "a man is playing a guitar" entails "a man is playing an instrument", but not conversely. Cosine similarity is symmetric by construction — cos(u, v) = cos(v, u), always. So the training relation has a property the inference metric structurally cannot express.

What survives the collapse is the part of entailment that is symmetric: topical and propositional compatibility. If A entails B then A and B are about the same situation and do not conflict. If A contradicts B they are about the same situation and do conflict. The direction is lost; the compatibility is kept. And compatibility, averaged over a million examples spanning ten genres, is a very good proxy for what humans mean by "these sentences are similar."

Why the classification objective, and not regression, on this data

A fair question, given Chapter 4 argued that regression has no train-test gap: NLI labels are ordered — entailment, neutral, contradiction runs from most to least compatible. You could map them to 1.0, 0.5, 0.0 and regress on cosine, closing the gap. Why did the authors not?

Rows 1 and 2 are the substantive ones. Contradiction does not mean "similarity zero" — a contradiction pair is about the same situation, so a sensible space places it moderately close, and demanding 0.0 fights the very structure you want. The classification objective sidesteps the question by never committing to a number, which is exactly the forgivingness Chapter 3's flat pooling ablation revealed.

The modern resolution is neither: use entailment pairs as positives and contradiction pairs as hard negatives in an InfoNCE loss, which asks only that the positive outscore the negatives — a ranking constraint, not a value one. Supervised SimCSE does precisely this on the same data and gains about 6.7 points. Same labels, third framing, best result.

What NLI does not teach

Symmetric across the earlier list of what NLI supplies, here is what it leaves out — and each gap predicts a specific production complaint.

Every row is fixable by the same move — include pairs of that kind in training — which is why modern embedders train on a deliberately heterogeneous mixture: web pairs, question-answer pairs, code, titles-and-bodies, multilingual, and NLI. The mixture is the model. And it is why "this embedder is bad at X" almost always means "X was not in the mixture" rather than anything about capacity.

The labels are not clean, and that is fine

SNLI's gold labels come from a vote. Each pair was shown to five annotators; the gold label is the majority, and pairs without a majority were discarded. Even among the kept pairs, agreement is imperfect — individual annotators match the gold label roughly 88% of the time, so something like one label in eight is contested by a competent human.

That should worry you more than it does, and the reason it does not is worth understanding.

So SBERT is comparatively lucky: its target geometry depends mostly on the well-agreed boundary, and the noisy boundary separates two labels it does not much care to distinguish. A model trained to do NLI is limited by the 88% ceiling. A model using NLI to shape a space is not, because it only needs the coarse structure the labels reliably encode.

Directionality, made concrete

The asymmetry is easy to wave at and worth pinning down with an example you can check. Take:

So one bit of the relation — which sentence is more specific — cannot survive into the embedding space at all. What does survive is the shared situation, and that is enough for similarity but not for inference. If your application needs the direction (does this document support this claim? does this log line satisfy this alert rule?) then a symmetric metric is the wrong instrument regardless of how good the encoder is.

There is a research line that fixes it — order embeddings and hyperbolic embeddings represent entailment as containment or as depth in a tree rather than as proximity, so asymmetry is native. They are a genuinely different geometry and outside this paper's scope, but knowing they exist prevents the mistake of expecting cosine to do a job it is mathematically incapable of.

Counting the hard negatives you get for free

Chapter 9 will claim that hard negatives are the highest-leverage variable in embedding training. Quantify what NLI hands you. Of roughly one million pairs, the label distribution is close to uniform across the three classes:

Now price the alternative. To mine negatives of that quality yourself you would retrieve top-k candidates for each positive with a bootstrap model, filter out the true positives (which requires labelling, or you poison your training set with false negatives), and repeat as the model improves so the negatives stay hard. That is a multi-week engineering effort producing lower-quality negatives, per domain.

The three labels, and what each one contributes

It is tempting to think the three-way label is incidental — that you could binarise to "related / not related" and lose nothing. Each class is doing a distinct job.

The neutral class is the underrated one, and it is what makes the resulting space usable for ranking rather than just matching. STS asks "which of these two pairs is more similar," a question that only has an answer if the space has intermediate distances. A model trained on positives and hard negatives alone tends toward a bimodal similarity distribution — a spike near 1 and a spike near the floor — which scores fine on binary duplicate detection and poorly on Spearman.

Modern supervised contrastive setups reproduce this by mixing in-batch negatives (easy, the "unrelated" end) with explicit hard negatives (the contradiction end) while positives supply the top. The three-way label is one way of specifying that mixture; it is not the only way, but the mixture itself is not optional.

Why MultiNLI matters as much as SNLI

SNLI's premises are all image captions. That is a peculiar and narrow genre: present tense, concrete, visually grounded, short, third person. A model trained on SNLI alone learns a space beautifully organised for describing photographs and less so for anything else.

MultiNLI was built to fix this — the same annotation protocol over ten genres including telephone transcripts, government reports, fiction and letters. Adding it to the mix is what makes SBERT a general-purpose encoder rather than a caption encoder. Chapter 9's failure mode — domain shift — is exactly what you get when the training genres do not cover your deployment genre, and the SNLI/MultiNLI mix is a partial, not complete, defence.

What one epoch actually changes

It is worth forming a picture of what the space looks like before and after, because "the geometry improves" is vague and the change is quite specific.

Row two is the one that reframes everything. The fine-tune does not make paraphrases score dramatically higher — they were already near the top of a compressed range. It makes non-paraphrases score lower. Chapter 1's five-minute experiment measures exactly this, and it is why Spearman (a rank metric) captures the improvement while a naive look at "similar pairs score 0.85 either way" would miss it entirely.

The arithmetic of one epoch

Something worth checking, because it is easy to assume the fine-tune must be doing something enormous:

Sixty-two thousand steps at learning rate 2e-5. For comparison, BERT's own pretraining was about a million steps at a much larger batch. This is a light touch — a nudge that reorganises the geometry without destroying the linguistic knowledge underneath. Push it harder (more epochs, larger learning rate) and you get catastrophic forgetting: STS scores keep improving on the NLI-flavoured evaluation while transfer to anything else collapses. The one-epoch choice is not laziness; it is regularisation by early stopping.

Worth pausing on the learning rate, because 2e-5 looks arbitrary and is not. Fine-tuning a pretrained encoder is a search for a nearby minimum, not a fresh optimisation. At 1e-3 — a normal rate for training from scratch — the first few hundred steps move the weights far enough that the pretrained features are destroyed, and you spend the rest of the run relearning English from a million sentence pairs, which is not enough data to do it. At 2e-5 the parameters travel a short distance and the linguistic knowledge survives. The warm-up is the same argument applied to the first few steps specifically: Adam's variance estimate is built from almost no samples at t = 1, so its step size is unreliable exactly when the model is most fragile.

The two-stage recipe, and why the order matters

Chapter 7's best SBERT numbers come from a two-stage sequence: fine-tune on NLI, then fine-tune on the STS benchmark's graded pairs with the regression objective. Both stages are cheap; the order is not arbitrary.

Reverse the order and stage 1's million coarse examples would overwrite stage 2's few thousand fine ones. This general shape — lots of weak pairs first, few strong pairs last — became the standard multi-stage recipe for every modern embedding model, where stage 1 is now hundreds of millions of mined web pairs and stage 2 is a curated supervised mixture. The pattern in this paper is the two-stage version of a pipeline that now has four.

Note how small that last gain is on this benchmark — +0.68 — and do not conclude that NLI was unnecessary. STS-B's test set resembles its training set; the NLI stage buys generality, which this table cannot see. It shows up in the 77.03 row (usable with no in-domain data at all) and in every downstream task the model was never tuned for. Benchmarks that share a distribution with their training data systematically undervalue pretraining, which is worth remembering whenever an ablation says a stage "does not help".

If you have no NLI data

Which is the normal case: a specialist domain, a language other than English, or a company corpus. The substitution to make is not "find NLI data" but "find the two properties NLI supplied" — non-trivial positives and same-topic negatives. Both can usually be manufactured from structure you already have.

Row three is the richest source most companies already own and almost none use. A search log's non-clicked impressions are same-query, same-topic, human-judged-irrelevant — the definition of a hard negative, generated for free by traffic. The caveats are real (position bias, clicks are noisy relevance) and manageable with standard debiasing, and it is still far better data than anything you can buy.

Chapter 6: The Whole Pipeline, By Hand

Everything so far has been described. Now we compute it. Three toy sentences, four tokens each, four dimensions instead of 768, and every single number visible. If you can follow this chapter with a pen, you understand Sentence-BERT completely — the real model differs only in that 4 becomes 768 and the token vectors come from twelve transformer layers instead of from this page.

Step 1 — the token vectors coming out of BERT

Pretend BERT has emitted these. Each sentence is four tokens: [CLS], two content words, [SEP]. Each token vector is 4-dimensional.

Sentence A — "a dog barks":

Sentence B — "a puppy is barking":

Sentence C — "the server crashed":

Step 2 — MEAN pooling

All four positions are real tokens, so the mask is all ones and the denominator is 4. Divide each column sum by 4:

Already you can see the structure in the numbers: A and B both put their mass in dimensions 0 and 1; C puts its mass in dimensions 2 and 3. If dimensions 0–1 encoded "animal / sound" and 2–3 encoded "software / failure", this is exactly what a working sentence encoder would produce.

Step 3 — cosine similarity, computed twice

The paraphrase pair first. Dot product:

Norms:

Now the unrelated pair:

0.9780 against 0.2223. The pooled vectors do the job: the paraphrase scores four times higher than the unrelated pair. This is what "the embeddings work" means, made of nothing but sums and square roots.

Step 3b — the same sentences under CLS and MAX pooling

Chapter 3 argued from statistics that MEAN should win. Here is the argument as arithmetic on these exact numbers.

CLS pooling takes row 0 of each table and nothing else:

The unrelated pair scores higher than the paraphrase — 0.913 against 0.897. Ranking inverted. Nothing is wrong with the arithmetic; the [CLS] rows simply do not carry the content, because in an untuned model they were never asked to. This is Chapter 1's 29.19 reproduced in four dimensions.

MAX pooling takes the largest value in each column:

The ranking is preserved here, but look at the norms: 1.800 and 1.536, against MEAN's 0.831 and 0.812. Max-pooled vectors are roughly twice as long, and — crucially — the ratio between the two sentences' norms has grown from 1.02 to 1.17 for no semantic reason. Add two more tokens to sentence A and the gap widens further, because the max can only ratchet up. Under a cosine objective that drift is uncorrectable, which is the 69.92 in Chapter 3's table.

Step 4 — the classification features

The training objective does not use cosine. It builds the concatenation [u; v; |u − v|]. First the difference:

And for the unrelated pair, so we can compare:

0.30 versus 1.90 — a factor of 6.3. That is the signal the classifier gets to work with, and it is far starker than the cosine gap. This is the practical answer to "why does a difference feature help": the difference amplifies the distinction the raw vectors only imply.

The full 12-dimensional feature vector for the paraphrase pair is:

In the real model this is 2304-dimensional (768 × 3). Here it is 12, and W_t is 3×12 instead of 3×2304.

Step 5 — the classifier and its logits

Take a toy W_t that has already learned something sensible, written as three blocks of four plus a bias. (The paper's equation omits a bias; every real implementation uses nn.Linear, which carries one, and we need it here to keep the numbers readable.)

Read the W_d column as a sentence: "the bigger the difference, the less entailment and the more contradiction." That is the geometry the classifier is enforcing on the encoder. Now compute the three logits for the paraphrase pair, where the difference coordinates sum to 0.30:

Step 6 — softmax and cross-entropy

Gold label is entailment, so the loss is the negative log of the probability assigned to entailment:

For calibration, a model guessing uniformly over three classes would score −ln(1/3) = 1.0986. Our toy classifier is doing better than chance but is far from confident — 43% on the right answer. Good: that means there is a gradient to follow, which is what we want for the next step.

Run the same machinery on the unrelated pair (u, w), gold label contradiction, with difference sum 1.90:

0.0631 against 0.8373. The model is already confident and correct on the easy pair and unsure on the hard one, which is exactly how a healthy mid-training loss distribution looks. Almost all of the learning signal is coming from the pair that is nearly right.

Step 7 — one gradient step, all the way through

The error signal for softmax with cross-entropy is beautifully simple: δ = p − y. For the entailment pair with y = (1, 0, 0):

Now push it back into u using the gradient we derived in Chapter 4:

The first term, coordinate by coordinate. Only the entailment row has a nonzero W_u entry in dimension 0 (0.1), and only the contradiction row in dimension 1 (0.1):

The second term. Every column of W_d is identical — (−2.0, −0.5, +1.5) down the three rows — so W_d^Tδ is the same scalar in all four coordinates:

And the sign vector: u − v = (0.10, −0.10, −0.10, 0.00), so sign(u − v) = (+1, −1, −1, 0). Multiply coordinate-wise and add:

Read that gradient against u − v = (0.10, −0.10, −0.10, 0). In every coordinate, the gradient points in the same direction as the difference — and gradient descent moves against the gradient, so u will move to reduce each coordinate's gap with v. The difference feature is doing precisely what Chapter 4 promised.

By symmetry (swap the roles, the sign vector flips):

Take a step with learning rate η = 0.05 — u ← u − η(∂L/∂u):

(In the real model this step lands on θ and propagates back through twelve transformer layers to every weight; here we take the shortcut of updating the pooled vectors directly, which is what the encoder would be nudged to produce.)

Step 8 — did the geometry improve?

The objective never mentioned cosine. Let us check what it did to cosine anyway.

And the loss, recomputed on the updated vectors. New difference: |u' − v'| = (0.0331, 0.0330, 0.0330, 0), summing to 0.0991 instead of 0.30. The logits become 1.1324, 0.5505, −0.1616, the softmax gives p_ent = 0.5456, and:

Step 9 — the other two objectives on the same numbers

Same three vectors, evaluated under the alternatives, so you can feel the difference between them concretely.

Cosine regression. With gold scores y = 1.0 for the paraphrase pair and y = 0.0 for the unrelated one:

Under this objective, almost all the gradient comes from the unrelated pair — 0.0494 against 0.0005, a hundredfold difference. Compare with the classification objective, where the paraphrase pair carried almost all the loss (0.8373 against 0.0631). Same vectors, same data, opposite allocation of learning effort. Which objective you pick decides which examples your model spends its capacity on, and that is a much more consequential choice than it appears.

Triplet. Anchor u, positive v, negative w, margin ε = 1, Euclidean:

The ordering is already correct by 0.85 — the positive is five times closer than the negative — and the loss is still positive, because ε = 1 demands a gap of a full unit. Drop ε to 0.8 and the loss becomes max(−0.0515, 0) = 0 and this triplet goes silent. Chapter 4's warning about margin scale, in two lines of arithmetic.

Step 9b — one step of the regression objective, for contrast

We took a gradient step under the classification objective and watched cosine rise as a side effect. Do the same under objective 2, where cosine is the target directly, and compare the character of the two updates.

Use the unrelated pair (u, w) with gold y = 0, since that is where this objective puts its gradient. Recall cos(u, w) = 0.2223, ‖u‖ = 0.8307, ‖w‖ = 0.8124. The chain rule gives:

Unit vectors: û = (0.8427, 0.4815, 0.0000, 0.2408) and ŵ = (0.1231, 0.0000, 0.8617, 0.4924). So:

Sanity-check the orthogonality claim by dotting with û:

Zero to four decimal places. The update cannot change ‖u‖ at all; it only rotates u away from w. Step with η = 0.05:

Down by 0.031, moving toward the gold value of 0. Note also ‖u″‖ = 0.8322 against the original 0.8307 — a 0.2% growth caused entirely by taking a finite step along a tangent, not by any radial component in the gradient. That is the norm-drift effect Chapter 4 mentioned, visible in the fourth decimal place.

Step 10 — check the normalised-distance identity

Chapter 4 claimed that on unit vectors, ‖a − b‖² = 2(1 − cos). Verify it here rather than trusting it. Normalise u and v by their norms 0.8307 and 0.8124:

Difference: (0.1041, −0.1340, −0.1231, −0.0054). Squared length:

They agree to four decimal places, the residual being our rounding. So on the unit sphere the two metrics are the same metric wearing different clothes, and a k-means that minimises squared distance is maximising cosine, exactly as Chapter 8 will assume.

Confirm the converse too — that the identity fails without normalisation. Unnormalised, ‖u − v‖² = 0.03 while 2(1 − cos) = 0.0440. Different numbers, and for pairs with very different norms they can rank differently. That is the whole content of "always normalise before you compare," made checkable.

The whole pipeline on one line

Row 8 is the one to memorise. Everything from step 3 onward exists only during training. At inference SBERT is a BERT forward pass and a mean.

Step 9c — and one triplet step

For completeness, the third objective's update on the same vectors: anchor u, positive v, negative w, ε = 1. The loss was 0.1485, so it is active and there is a gradient.

Differentiate ‖u − v‖ with respect to v. Writing d = u − v:

Descent moves v by −η∂L/∂v, i.e. along +(0.5774, −0.5774, −0.5774, 0) — straight toward u along the connecting line. With η = 0.05:

Distance fell by exactly η = 0.05, which is not a coincidence: the gradient of a Euclidean norm is a unit vector, so a step of size η moves the distance by exactly η regardless of how far apart the points are. Contrast that with the cosine gradient, whose magnitude scaled as 2(c − y)/‖u‖ and therefore shrank as the model got closer to correct.

Check yourself on a second pair

Do this one with a pen before reading the answers. Sentence D — "the dog is silent" — pools to:

Using u = (0.70, 0.40, 0.00, 0.20) from sentence A ("a dog barks"), compute: (1) cos(u, x); (2) |u − x| and its coordinate sum; (3) the three logits under the toy W_t; (4) the softmax and the loss if the gold label is contradiction.

Answers. (1) u · x = 0.455 − 0.040 + 0.000 + 0.030 = 0.445; ‖x‖ = √(0.4225 + 0.01 + 0.0025 + 0.0225) = √0.4575 = 0.6764; cos = 0.445 / (0.8307 × 0.6764) = 0.445 / 0.5619 = 0.792.

(2) u − x = (0.05, 0.50, −0.05, 0.05), so |u − x| = (0.05, 0.50, 0.05, 0.05), sum = 0.65.

(3) z_ent = 0.1(0.70) + 0.1(0.65) − 2.0(0.65) + 1.2 = 0.07 + 0.065 − 1.30 + 1.2 = 0.035; z_neu = −0.5(0.65) + 0.6 = 0.275; z_con = 0.1(0.40) + 0.1(−0.10) + 1.5(0.65) − 0.4 = 0.04 − 0.01 + 0.975 − 0.4 = 0.605.

(4) e^0.035 = 1.0356, e^0.275 = 1.3166, e^0.605 = 1.8313; sum = 4.1835. p = (0.2476, 0.3147, 0.4377). Loss = −ln(0.4377) = 0.8262.

Now interpret it, which is the actual exercise. Cosine says 0.792 — high, because "dog" is shared and the vector still lives in the animal region. The difference sum says 0.65, more than double the paraphrase pair's 0.30, so the classifier leans contradiction at 43.8%. The two views disagree in exactly the way you would hope: raw proximity is fooled by the shared subject, and the per-coordinate difference is not, because dimension 1 — where "barks" put its mass — differs by 0.50 all by itself. That single coordinate carries most of the signal, and it is precisely what a coordinate-wise feature can see and a single cosine cannot.

The same computation at full scale

Everything above used d = 4 so the numbers would fit on a page. Nothing changes at d = 768 except the size of the loops — which is worth stating precisely, because it is where the intuition "the real model must be doing something more complicated" gets corrected.

Only the last row is qualitatively different, and it is the part every autograd framework writes for you. Everything you computed by hand — the pooling, the difference, the logits, the softmax, the error signal, and its route back into u and v — is exactly what runs in production, at a different loop bound.

Worth noting what the toy scale hides, though, in fairness. At d = 4 there is no room for anisotropy to be a subtle effect — we saw it as an obvious 0.2223 floor. At d = 768 the cone is a statistical property of thousands of near-orthogonal directions, and your intuition from four dimensions will mislead you about how much room there is up there. The rule 1/√d from Chapter 1 is the bridge: in four dimensions random pairs scatter with sd 0.5, in 768 with sd 0.036. High dimensions are mostly empty, and everything is nearly orthogonal to everything, unless a training objective has arranged otherwise.

Chapter 7: The Numbers, Including the Bad Ones

A paper that only reports the results where it wins is an advertisement. This one reports a table in which the architecture it is arguing against beats it by three points, and that table is the most useful one in the paper. Let us go through the evaluation properly — what is being measured, why that measure, and what each row means for a decision you might make.

What Spearman correlation is, and why not Pearson

STS datasets give each sentence pair a human score, typically 0 to 5. A model gives each pair a cosine. You need a single number for "do these agree." Spearman rank correlation is Pearson correlation computed on the ranks rather than the raw values, which makes it sensitive only to ordering.

Work it on five pairs. Human scores and model cosines:

One adjacent swap out of five pairs costs ten points. Now you can read the tables with a feel for the scale: SBERT's 74.89 versus mean-BERT's 54.81 is not a marginal improvement, it is a different quality of ordering.

Result 1 — unsupervised STS (no STS training at all)

This is Chapter 1's table, and it measures the thing that matters most in practice: how good are the embeddings if you just download them and use them?

Three separate lessons live in this table, and it is worth pulling them apart.

The supervision is worth about 20 points; the encoder upgrade is worth about 2. Mean-BERT to SBERT-base is +20.08. SBERT-base to SBERT-large — three times the parameters — is +1.66. If you have a fixed budget, the order of operations is unambiguous: fix the objective before you enlarge the model.

RoBERTa does not help. SRoBERTa-large (76.68) versus SBERT-large (76.55) is a 0.13-point difference on a benchmark whose run-to-run standard deviation is larger than that. The paper says as much. RoBERTa is a clearly better model than BERT on GLUE-style fine-tuning tasks, and none of that advantage survives into sentence-embedding quality — another sign that the bottleneck is the training signal, not the encoder.

Beating USE is the real result. USE was trained on far more and far more varied data. SBERT beats it by 3.67 average points with NLI alone and twenty minutes of fine-tuning, which is the clearest statement of the paper's thesis: the right small intervention on a strong pretrained encoder outperforms a large bespoke training programme.

Result 2 — supervised STS, where the cross-encoder wins

Now train on the STS benchmark itself and evaluate on its test set. Reported as Spearman × 100 with standard deviations over ten random seeds.

The other detail worth extracting: look at the standard deviations. SBERT's are 0.17–0.35; the cross-encoder's run as high as 0.81. Bi-encoders trained this way are noticeably more stable across seeds, which is a real operational virtue when you are going to re-train quarterly and need results you can compare.

Result 3 — SentEval, and why the paper is sceptical of it

SentEval evaluates sentence embeddings by freezing them and training a logistic regression classifier on top for seven transfer tasks — sentiment (MR, CR, SST), subjectivity (SUBJ), opinion polarity (MPQA), question type (TREC), and paraphrase detection (MRPC). Accuracy, averaged:

SBERT wins here too, but the paper immediately qualifies the result, and the qualification is the interesting part. SentEval trains a classifier on top of the frozen embedding. That means it measures how much information is linearly extractable from the vector — not whether cosine distances in that space mean anything.

Where the STS datasets came from

Seven datasets get averaged into one number, and knowing what they are stops you over-reading it.

Two implications. First, "average STS" is an average over genres, so a model that is superb on headlines and poor on forum posts can post the same number as one that is even. Second, none of the seven contains a query, a document longer than a sentence, or an asymmetric pair — which is precisely the gap BEIR was built to fill, and precisely why an excellent STS score does not guarantee a usable retriever.

What a three-point Spearman gap is actually worth

Correlation numbers are hard to feel. Convert one. Suppose you use the model for top-1 retrieval over 1,000 candidates. A Spearman of 0.85 versus 0.88 does not translate to accuracy by any exact formula — the relationship depends on the score distribution — but the direction is clear from the rank mechanics we derived: a lower ρ means more rank inversions, and inversions near the top of the list are the ones that change your answer.

The practical consequence is that you should measure what you will ship. Spearman weights every pair equally, including the ones both models get obviously right at the bottom of the ranking. Retrieval cares only about the top few. A model that is 3 points worse on Spearman can be 10 points worse on precision@1 if its errors are concentrated at the top, or indistinguishable if they are not.

Result 4 — the Wikipedia triplet evaluation

The paper's third evaluation exercises the triplet objective directly, on a dataset from Dor et al. (2018): triples of sentences where the anchor and positive come from the same Wikipedia section and the negative comes from a different section of the same article. The task is to say which is which by distance alone.

Same story as STS-B: the cross-encoder is ahead by a few points, and the bi-encoder is the only one of the two that produces vectors you can put in an index. The value of the experiment is that it validates the third objective — triplet training on structural supervision, with no labels and no graded scores, produces a usable space. That is the recipe you reach for when all you have is "these things go together."

Result 5 — argument similarity, and the domain-shift warning

The paper also evaluates on the Argument Facet Similarity corpus: pairs of arguments about gun control, gay marriage, and the death penalty, scored 0–5 for whether they make the same point. Two evaluation splits, and the difference between them is the point:

Why does the gap appear only when the topic is unseen? Because of Chapter 2's structural argument. The cross-encoder gets to look at both arguments together and can reason about their relationship even for a topic it has never encountered — it does not need a good global map, only a good local comparison. SBERT must place an unseen-topic argument somewhere in a fixed space that was organised around other topics, with no chance to consult the other argument while doing so.

Reproduce Table 1 yourself

Every number in the unsupervised table is reproducible in about twenty lines and a few minutes, and doing it once converts the whole chapter from claims into measurements.

python — reproduce the paper's headline comparisonfrom sentence_transformers import SentenceTransformer, util
from scipy.stats import spearmanr
from datasets import load_dataset

sts = load_dataset('mteb/stsbenchmark-sts', split='test')
s1, s2 = sts['sentence1'], sts['sentence2']
gold   = sts['score']                       # human ratings, 0-5

def evaluate(model):
    a = model.encode(s1, normalize_embeddings=True)
    b = model.encode(s2, normalize_embeddings=True)
    pred = (a * b).sum(axis=1)                 # row-wise cosine, since rows are unit
    return spearmanr(pred, gold).correlation * 100

# The comparison that IS the paper:
print('mean-pooled BERT :', evaluate(SentenceTransformer('bert-base-uncased')))   # ~46
print('SBERT            :', evaluate(SentenceTransformer('all-MiniLM-L6-v2')))   # ~82

Note the first line's subtlety: passing a raw bert-base-uncased to SentenceTransformer constructs a Transformer plus a default MEAN Pooling module — which is exactly the "Avg. BERT embeddings" row, 46.35 on STS-B. Two model names, one function, and the paper's central claim on your own screen.

Two things to try immediately afterwards. Swap normalize_embeddings=True for False and use a raw dot product — Spearman falls, because you have reintroduced the norm confound. And centre the embeddings (subtract the column means over the test set) before scoring the mean-pooled BERT model — it should gain several points for free, which is Chapter 1's anisotropy fix, measured.

The ten-seed discipline

Every supervised number in this paper is reported as a mean over ten random seeds, with a standard deviation. That is not a formality, and the values themselves make the argument: BERT-STSb-base is 84.30 ± 0.76. Two standard deviations is 1.5 points, so a single run of that configuration can plausibly land anywhere from 82.8 to 85.8.

Now consider a hypothetical paper reporting a new method at 85.2, single run, "beating BERT's 84.30." The claim is inside the noise of the baseline. It would be indistinguishable from having run the baseline twice and reported the better number — which, done unintentionally, is how a great many small improvements enter the literature.

Read the second column vertically. Cross-encoder variances are two to four times the bi-encoder's. There is a plausible mechanism: a cross-encoder's score depends on a specific, delicate attention pattern learned over the pair, and which pattern the optimiser finds is seed-dependent; a bi-encoder's job is to arrange a global space, which is a more averaged, more constrained objective and therefore a more reproducible one.

Results in this area you should distrust

Having read a careful results section, it is worth calibrating on the careless ones you will meet elsewhere. Four patterns, each of which produces a real-looking improvement out of nothing.

Notice this paper's defences against all four: ten seeds with standard deviations, evaluation on STS sets that are explicitly excluded from training, a fixed protocol taken from prior work, and baselines quoted from their own papers. That is why its 2.98-point self-criticism is believable in a way most three-point claims are not — the same rigour that makes the wins credible is what makes the loss reportable.

How evaluation changed because of this paper

STS is the right benchmark for the claim the paper makes, and it is the wrong benchmark for what people went on to use SBERT for. That mismatch produced two successor benchmarks worth knowing, because if you evaluate an embedding model today you will be reading their leaderboards.

BEIR's founding result is the one to internalise: models that topped STS did not top zero-shot retrieval, and in several domains BM25 beat every dense model tested. That is Chapter 7's cross-topic result generalised to a whole benchmark — bi-encoders degrade under domain shift, and a benchmark whose test data resembles its training data cannot see it.

An evaluation checklist for your own model

Putting all four results in one sentence each

Notice that three of the five rows are about the method of evaluation rather than about SBERT. That is characteristic of a paper making a structural argument: most of the work is establishing that the comparison is fair, and the number that follows is almost incidental.

Result 6 — the speed numbers that started it

Sentences encoded per second, from the paper's efficiency table:

Smart batching is a simple and generalisable trick: sort the input by length before batching, so that each batch contains sentences of similar length and the padding overhead collapses. Nearly a 2× speed-up on CPU (44 → 83) and about 1.5× on GPU (1,378 → 2,042) for zero modelling change — because attention is quadratic in sequence length and you were computing it over [PAD] tokens.

Do the arithmetic once so the trick sticks. A batch of 16 with lengths {8, 9, 10, …, 120} pads everything to 120, so you compute 16 × 120 = 1,920 token-slots. Sorted into length-homogeneous batches, the same 16 sentences might need 16 × 12 = 192 slots in one batch and 16 × 120 in another — but averaged over the corpus the total slots computed drop toward the total real tokens. The saving is exactly the padding fraction, and on a corpus with a long tail of lengths that fraction is often 50–70%.

What the numbers do not tell you

Four things the results section cannot report, each of which turned out to matter more than a Spearman point.

Inference cost per unit of quality. The tables compare accuracy; they do not compare accuracy-per-millisecond. It emerged later that a 6-layer, 384-dimensional distilled SBERT retains nearly all of the quality at a fifth of the cost, which changed the deployment calculus far more than the base-to-large gain of +1.66 ever did.

How the model fails. Spearman is an average over pairs. It cannot tell you that your model's errors cluster on negation, or on numbers, or on named entities — and those are the errors users notice, because they are the ones a human would never make. Two models with identical Spearman can be very differently annoying.

Sensitivity to input formatting. Nothing in the evaluation says what happens if your text has markdown, HTML entities, inconsistent casing, or a 200-token boilerplate footer on every document. In production, that footer is a shared component added to every vector — Chapter 1's cone, self-inflicted. Strip boilerplate before embedding; it is one of the highest-return preprocessing steps there is.

Behaviour on inputs longer than the model. BERT truncates at 512 tokens, and SBERT's checkpoints often default to 128 or 256. Text beyond the limit is silently discarded — no warning, no error, just a vector that represents the first paragraph of a ten-page document. If your average document is longer than model.max_seq_length, your retrieval quality is being set by a configuration value most people never read.

Chapter 8: SBERT in Production

Sentence-BERT's real legacy is not its STS score — it is that a whole class of product features became buildable in an afternoon. Every one of them follows from the same fact: a sentence is now a point, and points can be indexed. This chapter walks the four canonical patterns with their real costs, then lists the four ways they fail silently.

Pattern 1 — semantic search

The whole system, in shapes:

Note what happened to the similarity computation: because both sides are unit-length, cosine similarity is the dot product, and all N of them together are one matrix-vector product. Your semantic search engine is a single BLAS call.

Memory, worked. For N = 1,000,000 documents:

Latency, worked. A brute-force scan reads the whole matrix, so it is bound by memory bandwidth rather than arithmetic:

That is why fp16 storage roughly halves your search latency even though the arithmetic is unchanged: you are moving half as many bytes. Below about a million vectors, brute force is genuinely fine and far simpler than an ANN index. Above it, an approximate nearest neighbour structure (HNSW, IVF-PQ) takes the scan from O(N) to something closer to O(log N) at the price of a small recall loss and a build step.

Pattern 2 — clustering

Clustering needs a vector per item. A cross-encoder cannot participate at all — there is no object to hand to k-means. With SBERT it is three lines.

One subtlety worth deriving. k-means minimises squared Euclidean distance, but you care about cosine. On L2-normalised vectors those are the same objective, by the identity from Chapter 4:

Minimising squared distance is therefore exactly maximising cosine. Running k-means on normalised embeddings is spherical k-means and it is the right algorithm — but only if you renormalise the centroids after each update, since the mean of unit vectors is not itself a unit vector (its length shrinks in proportion to how spread the cluster is). Skip that renormalisation and tight clusters get systematically longer centroids than loose ones, so they win assignments they should not.

Cost. k-means on N = 100,000 vectors of d = 768 with k = 50 clusters and 20 iterations:

Pattern 3 — paraphrase mining and deduplication

This is Chapter 0's original task, and the quadratic has not gone away — it has become cheap. All-pairs similarity over N sentences is E E^T, an (N, N) matrix. For N = 100,000 that matrix is 10¹⁰ entries = 40 GB in fp32, which does not fit anywhere, so you never materialise it:

python — chunked all-pairs, never materialising the N×N matriximport numpy as np

E = E / np.linalg.norm(E, axis=1, keepdims=True)     # (N, 768), unit rows
pairs, CH = [], 1024
for i in range(0, len(E), CH):
    block = E[i:i+CH] @ E.T                            # (CH, N) - 3 MB per row-block, fine
    r, c = np.where(block > 0.85)                       # threshold once, keep survivors only
    for a, b in zip(r + i, c):
        if a < b: pairs.append((int(a), int(b), float(block[a - i, b])))

The peak memory is one block: 1024 × 100,000 × 4 bytes = 410 MB, and only the survivors above the threshold are ever kept. This is the standard paraphrase_mining routine, and it turns Chapter 0's 65-hour job into a couple of minutes.

Pattern 3b — the blocking trick, revisited

Chapter 0 listed blocking as the database world's escape from the all-pairs problem. It has not been superseded by embeddings — it composes with them, and at large scale you need both.

Deduplicating 50 million product listings gives 1.25 × 10¹⁵ pairs. Even at 10¹⁰ dot products per second that is 34 hours of pure arithmetic, and the storage for the results does not exist. So block first:

The last row is the elegant one: an ANN index is a learned, soft version of a blocking key. Instead of a human deciding that "same brand" is the right partition, the embedding decides who is worth comparing to whom. Retrieve k neighbours for every item, score only those pairs, and the quadratic becomes O(N · k).

But note what you inherit along with the trick: blocking's recall failure. A duplicate whose two listings never appear in each other's top-50 will never be found, no matter how good the scorer is. Raising k trades compute for recall, and there is no setting of k that guarantees completeness. Exhaustive comparison was the only thing that did, and you gave it up in Chapter 0 for four orders of magnitude. It was the right trade; it is still a trade.

Pattern 4 — zero-shot classification, and RAG

Embed your label names ("billing issue", "shipping delay", "product defect"), embed the incoming text, take the arg-max cosine. That is a classifier with no training data and no gradient steps, whose class set can change at runtime — the same structural move CLIP and CLAP made in vision and audio, arriving here from a different direction.

And the pattern that came to dominate everything: retrieval-augmented generation. Chunk your documents, embed the chunks, store them; at query time embed the question, retrieve the top-k chunks, paste them into an LLM prompt. Every RAG system on earth has a Sentence-BERT-shaped component at its heart, and its quality ceiling is the retriever's recall — the point Chapter 2 made about rerankers applies at full force to generation, since a fact absent from the retrieved chunks cannot be generated except by luck.

When brute force stops being enough: ANN indexes

The 61 ms scan above is per query on one core. At 100 queries per second you need six cores doing nothing else, and at 100M vectors the matrix does not fit in RAM at all. That is where an approximate nearest neighbour index earns its complexity.

The decision rule is simpler than the table suggests. Under about a million vectors, use flat — it is exact, needs no tuning, and 30–60 ms is usually inside budget. Between one and fifty million, HNSW. Above that, or when memory is the binding constraint, quantise. Reaching for an ANN index at 50,000 documents is a very common way to spend a week acquiring a recall bug you did not need.

The four silent failures

None of these throw an exception. All of them return plausible numbers. Each has cost somebody a quarter.

The third row deserves a derivation because it is so common. Ranking by an unnormalised dot product means ranking by ‖u‖ ‖v‖ cosθ. For a fixed query, ‖u‖ is a constant, so you are ranking by ‖v‖ cosθ — the cosine weighted by each document's norm. Since mean-pooled norms grow modestly with content density and length, you have built a length-biased ranker and it will look almost right, which is worse than looking wrong.

Updating the index without downtime

Two of Chapter 8's silent failures are really one operational question: how does an index change while it is being served? There are three patterns and they are not interchangeable.

The rule is one line: a model change invalidates every vector, not the changed ones. Vectors from two models are coordinates in two different spaces, and a dot product between them is arithmetic without meaning. It will not error. It will return a plausible number, and old documents will quietly stop being competitive.

The cheapest enforcement is to make the model revision part of the index's identity — put it in the index name (docs_v3_bge-small_768), not in a metadata field somebody has to remember to check. Then a partial re-encode with a new model is impossible by construction, because the writer would have to target an index that does not exist yet.

Capacity planning, worked end to end

Take a concrete brief and size it, because the arithmetic is short and almost nobody does it before choosing an architecture.

"Semantic search over 5 million support articles. 50 queries per second at peak. p95 under 150 ms. Documents change: about 20,000 edits a day."

Two decisions fall straight out of this table and both are about dimension. Moving from 768 to 384 dimensions and from fp32 to fp16 takes storage from 61 GB to 15 GB — a 4× reduction that decides whether this is one machine or a cluster, for a cost of a point or two of retrieval quality. And the last row is why you version indexes: a full rebuild is hours, so it happens as a background job writing to a new index that you swap in atomically, never as an in-place mutation.

Chunking, and the arithmetic that makes it obvious

Chapter 9's "detail dilution" has a quantitative form worth having in advance. Mean pooling makes the sentence vector an average over tokens, so a single relevant sentence inside a long document contributes in proportion to its share of the tokens.

Signal share and index size move in opposite directions, and the sweet spot for most corpora is the paragraph — large enough to be self-contained when handed to a reader or a language model, small enough that the answer is a fifth of the vector rather than a hundredth. Two refinements matter in practice: overlap the chunks by a sentence or two, so a fact spanning a boundary is not split in half; and prepend the document title to each chunk, so a paragraph that says "it also supports SAML" still carries the name of the product it is about. Both are cheap, and between them they typically move recall more than swapping the embedding model does.

The reference implementation

python — the entire production patternfrom sentence_transformers import SentenceTransformer, util

model = SentenceTransformer('all-MiniLM-L6-v2')   # 384-dim, 6 layers, ~14k sent/s on GPU

# OFFLINE - once per corpus version
corpus_emb = model.encode(corpus, batch_size=128,
                          convert_to_tensor=True,
                          normalize_embeddings=True)   # (N, 384), unit rows

# ONLINE - per query
q = model.encode(query, convert_to_tensor=True, normalize_embeddings=True)
hits = util.semantic_search(q, corpus_emb, top_k=100)[0]   # chunked matmul + top-k

# OPTIONAL STAGE 2 - the cross-encoder from Chapter 2
from sentence_transformers import CrossEncoder
ce = CrossEncoder('cross-encoder/ms-marco-MiniLM-L-6-v2')
scores = ce.predict([(query, corpus[h['corpus_id']]) for h in hits])   # 100 joint passes

One detail in that snippet is a decision, not a default: all-MiniLM-L6-v2 is not SBERT-base. It is a 6-layer, 384-dimensional distilled descendant that is roughly 5× faster and half the storage, and on most retrieval tasks it is within a point or two of the 12-layer model. The lineage this paper started very quickly discovered that sentence-embedding quality is remarkably robust to shrinking the encoder — which is Chapter 7's finding (supervision matters far more than model size) cashed out as an engineering win.

Three incidents, and what each one actually was

The silent-failure table lists causes. Here is what they look like from the outside, because that is how you will first meet them.

"Search got worse after the deploy, but only for old articles." The model was upgraded and the reindex job was written to process documents modified since the last run — sensible for a content sync, catastrophic for embeddings. Old articles kept vectors from the previous model, and vectors from two models share no coordinate system, so old and new documents were being compared in different spaces. Nothing errored; the two populations simply stopped competing on equal terms. Fix: version the index; a model change forces a full rebuild, always.

"Duplicate detection flagged 400,000 pairs overnight." The threshold was tuned on a sample of English tickets at 0.85. A new locale launched, the corpus gained a large body of shorter, template-heavy text, and the mean similarity of that subpopulation was far above the old floor. The threshold had not moved; the distribution under it had. Fix: thresholds must be recomputed per population, from the measured noise floor — and the mining job should alert when its output volume moves by more than a factor of two.

"Latency is fine but the results feel random for long questions." max_seq_length was 128. Long questions were being truncated mid-sentence, so the embedded query was the first two clauses of a four-clause question — frequently the setup rather than the ask. Short questions worked perfectly, which made it look like a quality problem rather than a configuration one. Fix: log the fraction of inputs that hit the truncation limit; if it is above a percent, raise the limit or chunk the query.

The RAG failure taxonomy

Because retrieval-augmented generation is where most people meet SBERT, it is worth naming its failure modes precisely — users report all of them as "the AI got it wrong", and four of the five are retrieval bugs with distinct fixes.

Four of five rows are upstream of the language model entirely, which is why "our RAG is bad, let us try a bigger LLM" so rarely works. The instrumentation that separates them is one number: for a sample of real questions with known answers, did the correct chunk appear in the retrieved set at all? If it did not, nothing downstream can help; if it did, the problem is downstream and you have halved your search space.

Monitoring an embedding system

An embedding index has no natural error signal. It does not 500, it does not time out, it does not log a stack trace — it returns ten documents, always, ranked by a number. So the monitoring has to be built deliberately, and it is four cheap jobs.

Embedding drift deserves its own paragraph because it is subtle. Your model is frozen, so the function does not drift — but the distribution of inputs does. A support corpus in January is about last year's product; by July it is about the new release, with new terminology the encoder has never seen used this way. The vectors are still computed correctly and the neighbourhoods are still geometrically valid. What has changed is that the region of the space your queries land in is no longer the region your evaluation set measured.

Four questions before you add embeddings to a product

Not every search problem needs a vector index, and the failure mode of adding one unnecessarily is a permanent operational burden for no user-visible gain. Ask these first.

When the answers are good, the build order that minimises wasted work is: BM25 baseline → off-the-shelf bi-encoder, brute-force scan, measured against that baseline → cross-encoder reranker if the budget allows → ANN index only when the scan is too slow → fine-tuning only when the off-the-shelf model is demonstrably the bottleneck.

That order is deliberately the reverse of how these projects usually start. The common opening move — fine-tune a model and stand up a vector database — front-loads the two most expensive, most operationally sticky steps before anyone has established that the cheap ones were insufficient. Each stage above is a day or two and produces a number; skipping to the end produces a system nobody can evaluate.

Chapter 9: Where It Breaks, and What Came Next

Sentence-BERT is six years old, is still downloaded millions of times a month, and is comprehensively superseded by its own descendants. Understanding exactly which of its limits each descendant attacked is the fastest way to understand the modern embedding landscape — so this chapter is four limitations, each with its consequence and its successor.

Limit 1 — domain shift

The Argument Facet Similarity result from Chapter 7 was the early warning: in-domain, SBERT ties the cross-encoder; cross-topic, it loses by about seven points. The mechanism is structural and we derived it in Chapter 2 — a bi-encoder must commit to a representation of sentence A before it has seen B, so it needs a good global map, while a cross-encoder only needs a good local comparison.

The practical consequence is that "similarity" is not a domain-free notion. Two ICD-10 codes that differ by one digit are near-identical to a general encoder and clinically opposite. Two code snippets differing in a comparison operator are 0.99 apart in a text embedder and are a bug and its fix. Your domain's notion of "same" is a fact about your domain, and a model trained on image captions and telephone transcripts does not know it.

Limit 2 — anisotropy survives the fine-tune

Chapter 1 diagnosed the cone; the NLI fine-tune stretches it but does not remove it. Measure your own model: sample a thousand random pairs from your corpus and take the mean cosine. For SBERT-NLI it typically lands around 0.25–0.40 rather than 0. Everything sits in a wedge.

Three consequences follow directly. Thresholds are not portable across models or corpora (Chapter 8's day of work). Absolute similarity values are uninterpretable — 0.72 means nothing until you know the noise floor. And the usable dynamic range is compressed, so quantisation to int8 or binary loses more than the bit-count suggests, because the interesting variation occupies a small slice of the representable interval.

Two families of fix appeared almost immediately. Post-hoc: BERT-flow (Li et al., 2020) learns an invertible flow to a Gaussian; BERT-whitening (Su et al., 2021) does the same job with a mean subtraction and a linear whitening transform computed in closed form — roughly ten lines of NumPy, recovering several STS points on most models. In-training: make the objective itself punish anisotropy. Which brings us to the successor.

Limit 3 — the supervision bottleneck, and SimCSE

SBERT needs labelled pairs. NLI happens to exist for English; for most languages and nearly all specialist domains it does not, and annotating a million pairs is a serious programme. SimCSE (Gao, Yao, Chen, 2021) removes the requirement with a trick that is almost insultingly simple.

The loss is InfoNCE — softmax cross-entropy over in-batch similarities with a temperature:

Compare this to Chapter 4's triplet loss and the improvement is visible in the shape. Triplet: one negative, a hand-tuned margin, zero gradient once satisfied. InfoNCE: N−1 negatives at once, no margin, and the softmax automatically weights each negative by how threatening it currently is — a hard negative absorbs most of the repulsive gradient, an easy one almost none. Hard-negative mining stops being a separate engineering project and becomes an emergent property of the loss.

And it directly attacks the anisotropy. Wang and Isola (2020) decomposed contrastive objectives into two forces: alignment (positive pairs should be close) and uniformity (embeddings should spread over the sphere). The denominator of InfoNCE is the uniformity term — it is a sum over all other sentences that is minimised by pushing everything apart. So the cone gets flattened by the objective itself rather than by a post-hoc transform. SimCSE's paper makes exactly this argument, and shows the alignment-uniformity plot to prove it.

Read row 4 twice. Unsupervised SimCSE beats SBERT, which used a million human-labelled pairs. The labels were never the essential ingredient — what mattered was applying a pair-level force with enough negatives, and dropout noise is enough to define a positive. Then row 5: feed the same NLI data into the better objective, using contradictions as explicit hard negatives rather than as a third softmax class, and you gain another five points over that. Same data as SBERT, +6.7 points, purely from the loss.

Limit 3b — "similar" is not one relation

There is a limitation more fundamental than any of these, and it is not an engineering defect — it is a category error built into the premise. Take two sentences:

A single vector per sentence and a single cosine cannot answer four different questions with four different answers. SBERT-NLI is trained toward the first — semantic equivalence — and every deployment that actually wanted the second (question–answer relevance) is quietly using the wrong relation and blaming the model.

Limit 3c — English, and how the fix works

SBERT-NLI is English, because SNLI and MultiNLI are English. The obvious approach — find NLI data in fifty languages — does not scale, so Reimers and Gurevych solved it a different way in a 2020 follow-up, and the trick is elegant enough to state here.

The move is worth generalising: when supervision exists in one setting and not another, look for a cheap correspondence between the two (here, translation pairs) and distil across it rather than annotating again. The same trick multilingualises almost any encoder.

Limit 3d — you cannot ask the vector why

One more limitation, less discussed than the others and increasingly the binding one in products. A cosine of 0.83 is a number with no account of itself. It cannot tell you which part of the document matched, nor why one result outranked another.

Row four is the one that surfaces in regulated settings and in trust-and-safety work: there is no way to express a hard constraint over a learned metric. Everything you can do is a filter bolted on afterwards, which means the model can propose anything and your safety property lives entirely in the wrapper.

Partial mitigations exist. ColBERT's late interaction identifies which query token matched which document token, giving something like a highlight. Hybrid search lets the lexical component carry the explanation. And a cross-encoder reranker's attention can be inspected, with the usual caveats about attention as explanation. But none of them recovers the term-level auditability that a sparse index gives for free, and it is a real reason serious systems keep a lexical component rather than going purely dense.

Alignment and uniformity, computed

Wang and Isola's decomposition is worth making numerical, because it turns "the space is better" into two things you can measure on your own model in ten lines. For positive pairs (x, x⁺) drawn from your data and arbitrary points x, y drawn from the corpus:

The second one is a Gaussian-kernel energy, and the reason it measures spread is worth seeing. If every point collapses to the same place, all distances are 0, every exponential term is e⁰ = 1, the mean is 1, and the log is 0 — the worst possible value. If points are spread, distances are large, the exponentials are near 0, and the log is very negative. It is a repulsion energy read as a score.

Three toy configurations on the unit circle, each with four points, to see the two numbers move in opposition:

Now the whole story of this lesson in two coordinates. Mean-pooled BERT has decent alignment (related things are somewhat close) and terrible uniformity (everything is close, so "close" carries no information). SBERT improves both, mostly uniformity, because contradiction pairs get pushed apart. SimCSE improves uniformity a lot more, because its InfoNCE denominator is a uniformity term applied at every step to every pair in the batch. And a hypothetical model that maximised uniformity alone would spread everything evenly and destroy alignment — which is why the objective must contain both forces.

Limit 4 — one vector is a very small pipe

A 768-dimensional fp32 vector is 3 KB. A 512-token document is perhaps 2 KB of text carrying dozens of independent facts. Compressing it into 3 KB of floats with no knowledge of what will be asked is a hard information-theoretic bargain, and it produces three characteristic failures:

The third one has a purely operational fix that is the single most valuable piece of RAG advice: chunk your documents. Embedding paragraphs instead of documents raises the signal fraction from 1/500 to 1/50, and most retrieval quality complaints are chunking complaints wearing a costume.

The architectural fix is late interaction — ColBERT (Khattab & Zaharia, 2020) keeps one vector per token and scores with MaxSim, recovering fine-grained matching while staying precomputable, at 30–100× the storage. Chapter 2's design space, revisited with the benefit of knowing where the bi-encoder hurts.

The line of descent

Every arrow in that diagram preserves the structure Sentence-BERT established: encode independently, pool to one vector, compare with cosine, index the result. Six years of progress has been about the objective, the negatives, and the data — not the shape. That shape is the paper's real contribution.

What you would change if you wrote this paper today

A useful exercise for any paper, and here the answers are all things the following six years established.

Notice what is not in that table: the siamese structure, mean pooling, cosine similarity, storing one vector per sentence, and the retrieve-then-rerank pattern. Seven revisions to the training procedure and not one to the architecture. When a paper's method section ages badly and its structure section does not, the structure was the contribution — and that is the shape of a result worth studying six years later.

Which descendant should you actually use?

Reading this lesson should not end with you downloading bert-base-nli-mean-tokens. It is a 2019 checkpoint, and its own authors deprecated it. A short decision guide, expressed in the vocabulary this lesson has built:

The Matryoshka row is worth a sentence of its own, because it is a neat idea. Ordinary embeddings distribute information across all 768 dimensions with no ordering, so truncating to 128 destroys them. Matryoshka representation learning applies the training loss at several prefix lengths simultaneously — 64, 128, 256, 768 — so the model is forced to put the most important information first. One model then serves every storage budget, and you can retrieve cheaply at 64 dimensions and rescore the survivors at 768. It is the storage-side analogue of retrieve-then-rerank.

Cheat sheet

The four equations.

The numbers worth remembering

Where to go from here

Build it yourself — the afternoon recipe

References

1. Reimers, N. & Gurevych, I. "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks," EMNLP-IJCNLP 2019 — arXiv:1908.10084. The paper this lesson is built on.
2. Devlin, J., Chang, M.-W., Lee, K., Toutanova, K. "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding," NAACL 2019 — arXiv:1810.04805. The encoder, and the [CLS]/NSP design Chapter 1 dissects.
3. Cer, D. et al. "Universal Sentence Encoder," 2018 — arXiv:1803.11175. The strong contemporary baseline SBERT beats by 3.67 points.
4. Conneau, A. et al. "Supervised Learning of Universal Sentence Representations from Natural Language Inference Data" (InferSent), EMNLP 2017 — arXiv:1705.02364. Where "train on NLI" comes from.
5. Bowman, S. et al. "A large annotated corpus for learning natural language inference" (SNLI), EMNLP 2015 — arXiv:1508.05326; Williams, A., Nangia, N., Bowman, S. "A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference" (MultiNLI), NAACL 2018 — arXiv:1704.05426. The million pairs.
6. Gao, T., Yao, X., Chen, D. "SimCSE: Simple Contrastive Learning of Sentence Embeddings," EMNLP 2021 — arXiv:2104.08821. Dropout as augmentation; the successor from Chapter 9.
7. Ethayarajh, K. "How Contextual are Contextualized Word Representations?" EMNLP 2019 — arXiv:1909.00512. The anisotropy measurement behind Chapter 1's cone.
8. Li, B. et al. "On the Sentence Embeddings from Pre-trained Language Models" (BERT-flow), EMNLP 2020 — arXiv:2011.05864. The post-hoc geometric fix.
9. Wang, T. & Isola, P. "Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere," ICML 2020 — arXiv:2005.10242. The lens Chapter 9 uses on InfoNCE.
10. Khattab, O. & Zaharia, M. "ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT," SIGIR 2020 — arXiv:2004.12832. The middle ground of Chapter 2's design space.
11. Karpukhin, V. et al. "Dense Passage Retrieval for Open-Domain Question Answering," EMNLP 2020 — arXiv:2004.04906. In-batch negatives for retrieval, the bridge to modern embedders.
12. Muennighoff, N. et al. "MTEB: Massive Text Embedding Benchmark," EACL 2023 — arXiv:2210.07316. Where the descendants are measured today.