E5 and the Modern Text-Embedder Recipe

Chapter 1: CCPairs and Consistency

Contrastive learning has one non-negotiable input: pairs. Two texts that belong together, so the model has something to pull close, with everything else in the batch acting as the thing to push away. The method is simple. The bottleneck is arithmetic.

Count the world's supply of human-judged query–passage pairs as of 2022. MS MARCO passage ranking, built from real Bing queries with human relevance labels, contributes about 500,000 training pairs. Natural Questions contributes roughly 58,000 after filtering. Add NLI corpora, add TriviaQA, add HotpotQA, add every retrieval dataset anyone has released. You reach a few million pairs, and you have exhausted the entire annotated corpus of humanity.

A few million is not enough. CLIP trained on 400 million image–text pairs. The scaling behaviour of contrastive learning is unforgiving: the model learns the geometry of the pair distribution, and a small pair distribution yields a small geometry, correct in the neighbourhoods it saw and arbitrary everywhere else. That is exactly the BM25-loses-to-nobody failure from Chapter 0, restated as a data problem.

Pairs that already exist

Look at how text appears on the web and the pairs are obvious once you see them. A scientific paper has a title and an abstract: the title is a compressed query, the abstract is its answer. A Stack Exchange post has a question and an accepted answer. A Reddit post has a title and its top comment. A Wikipedia article has section headings and the paragraphs beneath them. A news article has a headline and a body.

None of these were written as retrieval training data. All of them encode the same underlying relation: a short piece of text that names a topic, and a longer piece of text that elaborates on it. That is exactly the relation a search engine needs to learn.

E5 harvests six such sources and calls the result CCPairs — Colossal Clean text Pairs, named in deliberate echo of the C4 web corpus.

After simple heuristic cleaning — length bounds, language identification, dropping pairs with too few alphanumeric characters, deduplication — this yields roughly 1.3 billion text pairs. Three orders of magnitude more than the world's labelled supply, obtained by scraping rather than annotating.

What "weakly supervised" actually means, precisely

The phrase gets used loosely, so pin it down. A weak label is a signal correlated with the thing you want but not equal to it. Here the thing we want is relevance — would a person say this passage answers this query? — and what we observe is co-occurrence: these two spans of text appeared adjacent on a web page.

The assumption is a probabilistic inequality, and stating it as one makes the risk visible:

The right-hand side is essentially zero: two random web spans are almost never relevant to each other. So the inequality is easy to satisfy, and it is all the inequality guarantees. It says nothing about how large the left side is. If 40% of Reddit title–comment pairs are jokes, the inequality still holds comfortably while 40% of your training positives are wrong.

Deriving the filter from zero

So: 1.3 billion pairs, an unknown but large fraction of them wrong, and no budget to look at them. What can you actually do?

Option 1: hand-label a sample and train a classifier. Reasonable, and this is what most industrial pipelines do. It requires an annotation guideline, annotators, agreement measurement, and a classifier that generalises across six wildly different sources. It is a project, not a step.

Option 2: heuristics. Drop comments under five words, drop pairs with high lexical overlap, drop Common Crawl pages with navigation-heavy text. Cheap, and it catches the dumbest failures — which is exactly why it was already applied to get from raw scrape to 1.3B. Heuristics cannot see semantics: "same" is five characters, and so is "Yes, use pandas.read_parquet".

Option 3 — the one E5 takes: ask a model that has read all of it.

Here is the move, and it is worth building up slowly because it looks circular at first and is not.

Train an embedder on the full noisy 1.3B. It will be mediocre — 40% of its positives were lies. But it will not be random, and the reason is statistical: a pattern has to recur to shape a model's weights. One post where "same" answers a pizza question contributes one gradient among a billion. The pattern "a title about a topic goes with a paragraph about that topic" appears in hundreds of millions of pairs. Gradient descent over a billion examples is an averaging machine, and the average of mostly-signal plus scattered-noise is signal.

Now make "agrees with the consensus" testable. Given a pair (q, p), hide p in a pool of about one million randomly sampled passages, embed everything with the noisy model, and rank the pool by cosine similarity to q. If p comes out near the top, the model — which learned from everyone else's pairs — independently agrees that p is the right answer to q. If p is buried at rank 40,000, the model does not see the connection, and the model is speaking for a billion other examples.

Keep the pair only if p lands in the top-k. E5 sets k very small; the paper's setting is k = 2. This is consistency-based filtering, and it takes 1.3 billion pairs down to roughly 270 million — about one pair in five survives.

Worked example: "top-2 out of a million" is secretly a score threshold

The rank criterion sounds sophisticated. Let us do the arithmetic and discover it is not, which is a much more useful thing to know than the criterion itself.

Fix a query q. Score it against the one-million-passage pool. Those scores are cosines between a query embedding and a million essentially unrelated passage embeddings. Each cosine is a sum of 768 coordinate products; by the central limit theorem the sum is approximately Gaussian. Empirically, mean-pooled encoders are anisotropic — their embeddings occupy a narrow cone rather than the whole sphere — so the background cosine sits well above zero. Take a concrete, realistic model:

The pair survives at rank ≤ 2 when fewer than two of the million pool passages beat p's score. The expected number that beat a score s is the pool size times the tail probability:

Set that expectation to 2 and solve for the score. We need P(Z > z) = 2 × 10⁻⁶. Reading the normal tail: P(Z > 4.61) ≈ 1.99 × 10⁻⁶. So z* = 4.61, and

Now run four candidate pairs through it, by hand.

Check the third row's tail by hand so you trust the table. The standard normal density at z is φ(z) = (1/√(2π)) e^−z²/2. At z = 4.167: z²/2 = 8.682, e^−8.682 = 1.70 × 10⁻⁴, so φ = 0.3989 × 1.70 × 10⁻⁴ = 6.78 × 10⁻⁵. The tail is well approximated by φ(z)/z for large z: 6.78 × 10⁻⁵ / 4.167 = 1.63 × 10⁻⁵, and the next correction factor (1 − 1/z²) = 0.942 brings it to 1.53 × 10⁻⁵. Times a million: 15.3 passages beat it. Rank 16. Dropped.

A ten-thousand-fold change in k moves the threshold by 0.14 cosine. The reason is that the Gaussian tail decays like e^−z²/2, so inverting it gives z ≈ √(2 ln(1/P)) — the threshold grows with the square root of the logarithm of the cutoff. Square root of a logarithm is the most inert function in applied mathematics. This is why the paper can pick k = 2 without an elaborate sweep: almost any small k lands in the same place.

What the filter costs: purity bought with recall

Twenty percent survival sounds like a lot of thrown-away data. It is. Let us quantify what was bought, using a model of pair quality chosen so that it reproduces the one number the paper publishes — 1.3B in, 270M out.

Model the 1.3 billion pairs as a mixture: 45% are genuinely aligned, with model scores distributed Normal(0.415, 0.10), and 55% are weak or wrong, with scores Normal(0.18, 0.09). (This is a modelling choice, not a paper result — E5 does not publish the score distribution. It is here so the trade-off becomes computable.) Apply the threshold s* = 0.4266:

Total survival: 0.2042 + 0.0017 = 0.2059, which is 20.6% of 1.3 billion ≈ 268 million pairs. The model is calibrated. Now read the two numbers that matter:

Read that last row again. The filter is not a scalpel. It discards the majority of the correct pairs in order to reach near-perfect purity. Is that the right trade?

Yes, and the argument is the asymmetry from earlier in this chapter. A discarded true positive costs one gradient you had 268 million replacements for. A retained false positive costs a gradient pointing the wrong way, plus the geometric distortion of resolving the conflict. When the two error types have wildly different costs, the optimal operating point is nowhere near the balanced one — you push hard toward the cheap error. Filtering aggressively is the same instinct as a spam filter with a low false-positive tolerance, and for the same reason.

Realization: could you actually run this?

A filter you cannot afford is a thought experiment. Walk the compute.

Step 1 — embed the pool once. One million passages through BERT-base at 128 tokens each. Store as fp16: 10⁶ × 768 × 2 bytes = 1.54 GB. That fits in the high-bandwidth memory of a single accelerator, which is the whole reason this design is affordable. The pool never changes, so this cost is paid once.

Step 2 — embed every pair. 1.3 billion queries and 1.3 billion passages, 2.6 billion forward passes. This, not the ranking, is the dominant cost, and it is unavoidable — you had to embed them to train the noisy model anyway, and you can cache.

Step 3 — rank. For each query embedding q of shape (768,), one matrix–vector product against the (1e6, 768) pool gives all million scores: 2 × 10⁶ × 768 ≈ 1.5 GFLOP. Batch 4,096 queries at a time and it becomes a (4096, 768) @ (768, 1e6) matmul — the shape GPUs are fastest at. Across 1.3 billion queries that is about 2 × 10¹⁸ FLOPs, which on a cluster achieving 10¹⁷ FLOP/s is a matter of hours, not weeks.

Step 4 — the early exit. You do not need the rank. You need the boolean "were fewer than 2 scores greater than s(q,p)?". That is a comparison and a count, fused into the same kernel that produced the scores, with no sort anywhere. Sorting a million floats 1.3 billion times would have been the actual bottleneck; noticing you never needed the sort is the difference between a feasible pipeline and an infeasible one.

What everyone else did with the same problem

Consistency filtering is E5's answer. It is not the only one, and the alternatives are informative about the design space.

Read the last two rows next to the first. E5 spends enormous effort finding and cleaning naturally occurring pairs. E5-Mistral, from the same lab fourteen months later, skips the entire problem by having a language model write the pairs. LLM2Vec skips it in the opposite direction by manufacturing positives out of dropout noise. Chapter 6 is about why both of those became possible, and what each one gives up.

Chapter 2: Batch Size Is the Miner

We have 270 million filtered pairs. Now: what is the loss?

Start from the question you actually want the model to answer, not from a formula. Hand the model a query and a shortlist of passages, exactly one of which goes with it. Ask: which one? That is an N-way classification problem — and it is a strange one, because the classes are not fixed. They change every step. There is no weight matrix with one row per class, because there is no fixed set of classes. The "class vectors" are just the passage embeddings, recomputed from scratch on every batch.

Once you see it that way, the loss writes itself. You want the softmax over the shortlist to put its mass on the correct passage, so you use cross-entropy, and the logits are similarities.

Symbol by symbol. q_i is the L2-normalised embedding of the i-th query in the batch. p_i is the embedding of its true partner passage. p_j ranges over every passage in the batch, including the correct one. τ (tau) is the temperature, a positive scalar that divides every similarity before the exponential — we will spend half this chapter on it. N is the batch size.

This loss has a name from the representation-learning literature: InfoNCE, for Noise-Contrastive Estimation of mutual information, introduced by van den Oord and colleagues in 2018. The name describes its origin — it lower-bounds the mutual information between q and p — rather than what it does day to day, which is: pick the right one out of N.

The economics of in-batch negatives

Put numbers on "free." A batch of N pairs requires 2N forward passes — N queries and N passages. From those 2N forward passes you obtain N × (N − 1) query–negative comparisons, because every query is compared against every non-matching passage. The comparison itself is a dot product, essentially free next to a transformer forward pass.

Check the last row: 32,768 × 32,767 = 1,073,709,056, just over a billion query–negative comparisons, from 65,536 transformer forward passes. Divide: 16,383.5 comparisons purchased per forward pass. The ratio grows linearly in N, which is why every lab that could afford a bigger batch took one.

But "more comparisons" is not automatically "more learning", and that is where the interesting part starts.

Why the number of negatives is the wrong thing to count

Here is the claim this chapter is built to earn: at the temperatures these models actually use, the loss is dominated by a single negative — the hardest one in the batch. The other 32,766 contribute almost nothing. So batch size does not matter because it gives you more negatives. It matters because it gives you a bigger pool to draw the hardest negative from.

Prove it with arithmetic. Take a partially trained model and a realistic batch. The correct passage scores cos = 0.75. There is one genuinely confusable passage at 0.65, one mildly confusable at 0.55, and 32,765 unrelated ones sitting near the background level of 0.15. Compute the softmax denominator, term by term, at three temperatures. Every term is measured relative to the positive, that is, exp((s − 0.75)/τ).

Now read the columns as three different worlds.

At τ = 0.01 — E5's setting — the entire negative mass is 4.54 × 10⁻⁵, and 99.995% of it belongs to the single hardest negative. The 32,765 background passages contribute 2.9 × 10⁻²², which is not a small number, it is an absent number: it vanishes below the resolution of the accumulator and does not exist in the computation at all. The loss behaves exactly like a triplet loss against one carefully chosen adversary.

At τ = 0.05 the picture flips in a genuinely surprising way. The hardest negative contributes 0.1353. The crowd of 32,765 easy negatives contributes 0.2013 — more. Each one is negligible; there are enough of them that negligible times thirty-two thousand is the dominant term. At this temperature, batch size matters for the reason people usually assume: volume.

At τ = 0.2 the crowd contributes 1,631 out of a denominator of 1,633. The hard negative is 0.037% of the signal. The model is being told, with overwhelming emphasis, "you are not any of these thousands of random things", which it already knew. The loss is 7.4 and mostly measures batch size, not model quality.

How hard is the hardest negative in a batch of N?

If the loss cares about the maximum over the batch, then the question "should I use a bigger batch?" becomes the question "how does the maximum of N − 1 similarity scores grow with N?" That is a question extreme-value statistics answers in closed form, and the answer is discouraging in an instructive way.

Take a fixed query embedding u on the unit sphere in d dimensions, and a passage embedding v drawn uniformly at random from that sphere. Their cosine is u · v, a sum of d coordinate products. It has mean 0 by symmetry, and its variance is exactly 1/d. For d in the hundreds, the central limit theorem makes it approximately Gaussian:

The maximum of M independent draws from Normal(0, σ²) concentrates near σ√(2 ln M). Substituting σ = 1/√d and M = N − 1:

Evaluate it for d = 768, doing one by hand so the rest are believable. At N = 32,768: ln(32,767) = 10.397, so 2 × 10.397 = 20.794, divided by 768 gives 0.027076, and the square root is 0.1646.

Read the last column with care. Multiplying the batch by 4,096 makes the hardest negative only 2.31 times harder. The scaling is √(ln N) — the square root of a logarithm again, the same inert function that made k irrelevant in Chapter 1. To keep making progress you must keep multiplying the batch, and the returns keep shrinking. Every lab discovered this the expensive way.

Realization: how do you put 32,768 sequences in a batch?

You do not. Not on one device, and not in the way "batch size" usually means. Three things have to be true at once, and each requires a specific engineering trick.

Problem 1: the similarity matrix. The loss needs all N² pairwise cosines. At N = 32,768 in fp32 that is 32,768² × 4 bytes = 4.295 GB for the forward matrix, and the backward pass wants its gradient, another 4.295 GB. Nearly 9 GB before a single transformer activation has been stored.

Problem 2: the activations. 65,536 sequences of a few hundred tokens through a 12- or 24-layer transformer, with all intermediate activations retained for backpropagation, is off by orders of magnitude from any accelerator's memory. The standard fix is gradient checkpointing: store only the layer boundaries and recompute the interiors during the backward pass, trading roughly 30% extra compute for a large memory reduction.

Problem 3: the negatives live on other machines. Split the batch across G devices and each device holds N/G pairs. Its local queries can only see N/G − 1 local negatives — you have not built a batch of 32,768, you have built G independent batches of N/G, which by the table above is dramatically weaker.

The fix for problem 3 is the one worth internalising, because it is what "large-batch contrastive training" actually means in code. After each device computes its local embeddings, an all-gather collects every device's embeddings onto every device. Now each device holds the full (N, d) matrix of passage embeddings and can score its local queries against all of them.

The communication cost is startlingly small. At N = 32,768 and d = 1024 in fp16:

Sixty-seven megabytes per step, over an interconnect built for hundreds of gigabytes per second. This is the asymmetry that makes the whole design work: embeddings are tiny compared with activations. You cannot ship activations between devices, but you can ship the 1024 numbers each sequence collapses to, and the 1024 numbers are all the loss ever needed.

The problem large batches create

Every in-batch negative is assumed to be a negative. It is an assumption, and at a batch of 32,768 drawn from 270 million pairs, it will sometimes be false. A false negative is a passage in the batch that would actually be a correct answer to the query but is being pushed away anyway.

Estimate the rate. Suppose a query has r genuinely valid passages somewhere in the 270M-pair corpus. The chance that a random batch of size N contains at least one of them is

Work two cases.

A specific technical query with r = 3 valid passages: 1 − (1 − 1.214×10⁻⁴)³ = 1 − 0.999636 = 0.00036. Roughly one batch in 2,700. Ignorable.

A common query like "what is machine learning", where r = 1,000 passages in a web-scale corpus would serve: 1 − (1 − 1.214×10⁻⁴)¹⁰⁰⁰. Since (1 − x)ⁿ ≈ e^−nx for small x, this is 1 − e^−0.1214 = 1 − 0.8857 = 0.114. Eleven percent of the time, this query is trained to push away a passage that answers it.

E5's stage one lives with this. It uses in-batch negatives only — no mined hard negatives at all — precisely because mined negatives make the false-negative problem dramatically worse. Chapter 3 is about the second stage, where hard negatives are introduced deliberately, and about the machinery required to survive them.

Push d to 4096 — the hidden size of a 7B decoder LLM — and watch the hardest random negative collapse. At d = 4096 and N = 32,768 it is √(20.794/4096) = 0.0713, the same hardness a 768-dimensional model gets from a batch of eight. High-dimensional spaces are emptier, random vectors are more nearly orthogonal, and luck stops producing hard negatives at all. That is a preview of why the 7B-parameter embedders in Chapter 6 lean on curated and synthesised hard negatives rather than on batch size.

Chapter 3: Hard Negatives and Distillation

Stage one produced a model that is very good at a task nobody needs: telling a passage about cooking apart from a passage about Kubernetes. Random in-batch negatives are, overwhelmingly, from a different topic than the query. Train on a billion of those comparisons and you get an excellent topic detector.

Real retrieval failures do not look like that. They look like this:

Both passages are about boiling water. Both share vocabulary with the query. A topic detector rates them nearly equally. Only B answers the question, and the difference lives in one clause. No batch of 32,768 random web passages will ever put A and B side by side and force the model to choose — the odds of drawing a passage that specific are essentially zero.

So you have to go and find them.

Hard negative mining, derived

The definition writes itself once you know what you are missing. A hard negative is a passage that is not the correct answer to a query but that the current model scores highly anyway. It is, by construction, a mistake the model is currently making.

And there is an obvious place to find them: run the model. For each training query, retrieve the top few hundred passages from the corpus with the stage-one model. Remove the known positive. What remains is a ranked list of the model's most confident errors, sorted by how confident it is.

Step 3 hides the entire difficulty. Which rank window? Rank 1 is the hardest negative available, so surely take rank 1?

No — and the reason is the most important practical fact in dense retrieval training.

The rank-1 "negative" is usually a positive

Retrieval datasets are sparsely labelled. MS MARCO, the workhorse of the field, was built by showing annotators the top Bing results for a real query and asking them to mark passages that contained the answer. On average roughly one passage per query ends up labelled relevant. But the corpus contains 8.8 million passages, and for a query like "how long does it take to boil an egg" there are certainly dozens of passages that answer it correctly. They are unlabelled, not irrelevant.

So when you retrieve the top 200 and call everything except the single labelled passage a negative, you are lying, and you are lying most at the top of the list — because a good retriever puts the genuinely relevant unlabelled passages there.

Quantify it. Suppose a typical MS MARCO query has about 7 truly relevant passages in the corpus, of which 1 is labelled. A decent stage-one retriever will rank most of the other 6 highly — say 4 of them land in the top 30 and 2 more in ranks 31–200.

Now recall Chapter 2's temperature arithmetic and see how bad 13.8% really is. Take a positive at cos 0.75, a false negative (secretly relevant) at 0.72, and the hardest genuine negative at 0.65, all at τ = 0.01. Relative softmax weights against the positive:

Ratio: 0.04979 / 0.0000454 = 1,097. The mislabelled passage receives about a thousand times more gradient than the hardest correct negative in the batch. You did not add a little noise to your training signal. You handed the loudest voice in the room to the one example that is wrong.

Cross-encoders: accurate and unusable

To denoise negatives you need a judge better than the model being trained. The field has one, and it has been sitting there since 2019.

Everything so far has been a bi-encoder: q goes through the network alone, p goes through the network alone, and the only interaction between them is one dot product at the very end. Call that the late-interaction constraint. It exists so that you can embed the corpus once, offline, and answer queries with a matrix multiply.

A cross-encoder throws that constraint away. It concatenates the two texts into one sequence — [CLS] query [SEP] passage [SEP] — runs the whole transformer over the pair, and reads a relevance score off the [CLS] position through a small linear head.

Do the last-but-one row's arithmetic, because the gap is the entire reason both architectures exist. A BERT-base forward pass over a 256-token pair is roughly 2 × 110M parameters × 256 tokens ≈ 5.6 × 10¹⁰ FLOPs. Over a million passages that is 5.6 × 10¹⁶ FLOPs, which at an achieved 10¹⁴ FLOP/s takes about 560 seconds per query. The bi-encoder's million dot products take about 15 microseconds. The ratio is roughly 3 × 10⁷.

Distillation: buy the accuracy, skip the cost

So we have a slow, accurate teacher and a fast, less accurate student. Knowledge distillation is the standard move: instead of training the student against the ground-truth label, train it to reproduce the teacher's distribution.

Concretely, for one query with a candidate list of M passages: run the cross-encoder on all M pairs to get teacher scores t_1..M; softmax them into a target distribution P; softmax the student's cosines into Q; minimise the Kullback–Leibler divergence

and blend it with the ordinary contrastive term:

E5's stage two is exactly this: fine-tune on a mixture of labelled datasets (natural language inference pairs, MS MARCO passage ranking, and Natural Questions) using mined hard negatives, with a linear interpolation of the cross-entropy contrastive loss and a KL term against a cross-encoder teacher.

Worked example: one query, four candidates, all the arithmetic

Take a single training instance. The candidate list is the gold passage plus three mined negatives, one of which is secretly relevant.

Teacher distribution. Exponentiate: e^8.2 = 3,640.95, e^5.1 = 163.99, e^4.6 = 99.48, e^−1.3 = 0.2725. Sum = 3,904.69. Divide:

Read that before continuing. The teacher is confident the gold wins, and it assigns the secretly-relevant passage 4.2% — a hundred times more than it gives the random one. The one-hot label would have assigned it 0. That 4.2% is the entire denoising signal, and it exists because the teacher is a relevance model rather than a lookup of what an annotator happened to click.

Student distribution at τ = 0.01. Logits are cosines divided by 0.01: (71, 68, 55, 12). Take everything relative to the largest: e⁰ = 1, e⁻³ = 0.049787, e⁻¹⁶ = 1.125×10⁻⁷, e⁻⁵⁹ ≈ 2.2×10⁻²⁶. Sum = 1.049787.

The contrastive loss with a one-hot target on the gold is just −ln Q₁ = −ln(0.95257) = 0.0486. By that measure the student is nearly perfect and there is almost nothing to learn.

The KL term, computed term by term:

Two things just happened that are worth more than the numbers.

First, the two losses disagree about whether the model is fine. Cross-entropy says 0.049 — nothing to see. KL says 0.294, six times larger, and 93% of it comes from row 3: the student has crushed a same-topic passage to a probability of 10⁻⁷ when the teacher thinks it deserves 2.6%. The student is not underconfident; it is catastrophically overconfident, and only the soft target can see it.

Second, that overconfidence is a direct consequence of τ = 0.01. Redo the student distribution at τ = 0.05: logits (14.2, 13.6, 11.0, 2.4), relative exponentials 1, e^−0.6 = 0.54881, e^−3.2 = 0.040762, e^−11.8 = 7.49×10⁻⁶, sum 1.58958:

Recompute the KL: 0.9325 × ln(0.9325/0.6291) = 0.9325 × 0.3935 = +0.3670; 0.0420 × ln(0.0420/0.3453) = 0.0420 × (−2.1066) = −0.0885; 0.0255 × ln(0.0255/0.0256) = −0.0002; 0.0000698 × ln(14.8) = +0.0002. Total 0.2785.

Almost the same magnitude — and pointing in the opposite direction. At τ = 0.05 the dominant term is row 1: the student now thinks gold and the secretly-relevant passage are nearly tied (0.629 versus 0.345) while the teacher is sure the gold wins, so KL pushes the student to separate them more. At τ = 0.01 the dominant term was row 3 and KL pushed the student to separate less.

Choosing α: what each term is actually for

With L = α L_InfoNCE + (1 − α) L_KL, the two extremes fail in different, predictable ways.

There is a subtler reason not to set α = 0. The teacher only ever saw the candidate list you gave it. It has no opinion about the other 32,767 in-batch passages, so a pure-KL objective supplies no signal at all against them — you would silently lose the entire in-batch negative mechanism from Chapter 2. The contrastive term is what keeps the global geometry honest; the KL term refines the local ordering inside each candidate list.

Realization: the shape of a stage-two training batch

Stage one's batch was flat: N queries, N passages, one similarity matrix. Stage two's is grouped, and the shapes change accordingly.

That last row is the quiet efficiency of the design. With G = 64 groups and H = 7, each query sees 64 × 8 = 512 passages: its own 8, plus 504 others that were mined as hard negatives for different queries. Those borrowed negatives are topically diverse but individually difficult, which is a better negative distribution than either pure random or pure own-query mining would give.

And note where the cross-encoder is: entirely offline. You score the mined candidates once, write the numbers to disk beside the training data, and never load the teacher into GPU memory during training at all. A distillation setup that runs the teacher online costs several times the training throughput for no benefit when the candidate lists are fixed in advance.

Chapter 4: query: and passage:

Here is E5's most-copied idea, in its entirety. Before encoding, prepend a literal string:

Two words and two colons. No new parameters, no architecture change, four tokens of extra compute. It looks like a formatting convention someone added for tidiness. It is load-bearing, and removing it breaks the model.

This chapter is about why — and the argument starts somewhere that seems unrelated: with the fact that cosine similarity is symmetric and relevance is not.

The impedance mismatch

A function is symmetric when swapping its arguments does not change the answer. Cosine similarity is symmetric by construction:

Now consider the relation retrieval actually needs: p answers q. Ask whether it is symmetric by trying it in both directions.

The relation is asymmetric, and it is asymmetric in a structural way, not a superficial one. A query is short, underspecified, ungrammatical, and often a fragment. A passage is long, self-contained, well-formed, and contains ninety percent material the query never mentioned. They are different kinds of object, and the relation between them runs one way.

So: we need to model an asymmetric relation with a symmetric kernel. That sounds impossible, and the resolution is worth stating carefully because it is the whole trick.

Three ways to make the two encoders different

Once you know you need f_Q ≠ f_P, there are exactly three implementations, and the field tried them in this order.

The third column explains why nobody uses two towers any more: it doubles the model for one capability. The fourth column explains why the prefix beat two towers, and this is the argument most summaries of E5 miss.

A two-tower model has permanently committed to asymmetry. Ask it to score sentence-similarity — "the cat sat on the mat" versus "a cat is sitting on a mat", where the correct relation is symmetric — and you must arbitrarily assign one sentence to the query tower and one to the passage tower. The score you get depends on a coin flip. A prefixed model simply uses "query: " on both sides, recovering a genuinely symmetric function from the same weights. This is not a hack; it is what the E5 model cards instruct you to do for symmetric tasks.

Worked example: watching a symmetric encoder fail

Abstract arguments about symmetry are easy to nod along to. Do the numbers.

Three texts, and a symmetric encoder trained on paraphrase and inference data (which is what Sentence-BERT-style models are):

A paraphrase-trained symmetric encoder produces, roughly:

Retrieval works by ranking, so what matters is the last two rows: the margin between the correct passage and its closest distractor is 0.41 − 0.39 = 0.02.

Is 0.02 enough? That depends on how noisy the scores are. Real embedding scores wobble by a few hundredths depending on phrasing, chunk boundaries, and truncation — call the per-score standard deviation 0.06. The difference of two such scores has standard deviation 0.06√2 = 0.0849. The probability the correct passage ranks first is

Fifty-nine percent. On a two-way choice. The model is barely better than a coin flip on a question a child could answer, and it is not because it fails to understand the text — look at row one, it understands the text beautifully. It has learned paraphrase geometry, and paraphrase geometry says two questions about Shakespeare plays are nearly the same thing while a question and its answer are quite different things. Which, as a statement about English, is true.

Now the prefixed model. Because it saw query:-marked short fragments paired with passage:-marked long documents through 270 million examples, it has learned a map that takes question-shaped text and points it at answer-shaped text:

Margin: 0.78 − 0.52 = 0.26, thirteen times larger. Rerun the probability:

From 59.3% to 99.89%. The error rate went from 40.7% to 0.11%, a factor of 370. Nothing about the model's language understanding changed. What changed is which relation the geometry encodes, and that was selected by four tokens of input.

Mechanism: how can four tokens do that?

Time to be suspicious. A mean-pooled embedding is the average of L token vectors. Adding two prefix tokens changes the average by a small amount. How does a small change to an average produce a 0.37 shift in cosine?

There are exactly two channels, and separating them is instructive.

Channel A — the direct contribution to the mean. WordPiece splits "query: " into two tokens, query and :. If the query body is 32 tokens, the pooled vector is now an average over 34 instead of 32, and the two prefix tokens own 2/34 = 5.9% of it. For a 200-token passage, "passage: " owns 2/202 = 1.0%.

Channel B — attention-mediated modulation. Inside every one of the twelve transformer layers, every content token attends to the prefix tokens. The value vectors flowing into "cancel" at layer 5 include a contribution routed from the query token. So the prefix does not merely get averaged in at the end; it changes what all the other token vectors are.

Which channel does the work? Rule one out by arithmetic. Suppose Channel A were the whole story: the prefix just adds a fixed vector c, scaled by about 0.06, to the query embedding and a fixed c′ scaled by 0.01 to the passage embedding. Then for unit vectors, a perturbation of relative size ε can move a cosine by at most about ε — roughly 0.06. We observed 0.37. Channel A is off by a factor of six and cannot be the explanation.

One more thing follows from Channel B being the dominant one: the prefix must be in the model's vocabulary and seen during training. It is not a magic token; it is a word whose embedding was shaped by 270 million gradient steps in which it reliably preceded question-shaped text. Invent a new prefix at inference — "search_query: " — and you get an untrained conditioning signal that the model has no reason to interpret correctly.

The production gotcha nobody warns you about

The prefix is not optional and it is not symmetric across your corpus. Both of those facts have caused real outages.

Failure 1: dropping the prefix. The E5 model cards state plainly that omitting the prefixes degrades quality substantially. It is easy to omit: your embedding library takes a list of strings, your prefix lives in a config file, and someone writes a batch backfill script that skips it. Nothing errors. Cosines are still in a plausible range. Recall quietly drops.

Failure 2 — the worse one: mixing prefixes within one index. Suppose half your corpus was ingested with "passage: " and, after a refactor, the other half with "query: ". The prefix is a strong conditioning signal, so it shifts the entire embedding cluster. Cross-prefix cosines are systematically lower than within-prefix cosines — the two halves of your index sit on nearly disjoint sub-manifolds.

Put numbers on it. If within-prefix neighbours score around 0.78 and cross-prefix neighbours around 0.55, then for any query:-embedded search, every document in the passage: half is penalised by roughly 0.23 relative to the other half. With a background score spread of 0.06, that is nearly four standard deviations. The affected half of your corpus is not merely down-ranked. It is unreachable, and there is no error, no exception, and no log line. The only symptom is that certain documents stopped being found.

Run the symmetric task with the prefix at full strength and the accuracy readout falls. That is not a bug in the simulation; it is the honest consequence of the design. A rotation that helps questions find answers actively hurts when the correct relation is "these two sentences mean the same thing." Which is precisely why E5 tells you to use query: on both sides for those tasks — applying the same rotation to both arguments leaves their angle unchanged.

That last sentence is worth making formal, because it is the cleanest statement of why one encoder plus a prefix beats two towers. If the prefix acts approximately as a rotation R, then in symmetric mode

because a rotation matrix satisfies R^TR = I. The prefix cancels out exactly when both sides carry it. A two-tower model cannot do this: its two encoders are different functions with no such algebraic relationship, so there is no way to make it symmetric on demand.

Chapter 5: Instructions as Geometry

Chapter 4 established something small and strange: four tokens of input select between two different similarity geometries computed by identical weights. If that is true, an obvious question follows, and two teams asked it independently in late 2022.

Why only two?

If "query: " means "encode this so it points at its answer", then nothing stops you writing "Represent this scientific paper title for finding papers that cite it" and expecting a third geometry. The prefix stops being a type tag and becomes an instruction: a sentence describing the relation you want the embedding to be good at.

INSTRUCTOR: one embedder, any task

Su and colleagues (arXiv:2212.09741) built INSTRUCTOR around a fixed instruction template:

Filled in, that becomes strings like "Represent the Wikipedia question for retrieving supporting documents:" or "Represent the Amazon review for classifying sentiment:" or "Represent the news article for clustering by topic:". The template makes the instruction space structured rather than free-form, which matters for a reason we will get to.

Training data is the interesting part. INSTRUCTOR is fine-tuned on MEDI — a collection assembled from roughly 330 datasets, each paired with a hand-written instruction. The datasets span retrieval, classification, clustering, and similarity, and the point of the mixture is that no single task dominates, so the model must actually read the instruction to know which behaviour to produce. The base model is GTR, a T5-encoder retriever. Evaluated on 70 tasks, 66 of which were unseen during training, it improved over the prior best by about 3.4% on average.

The pooling detail that makes it work

INSTRUCTOR mean-pools over the input text tokens only. The instruction tokens participate in self-attention — they condition every layer, exactly the Channel B mechanism from Chapter 4 — but they are excluded from the average that produces the final vector.

That sounds fussy. It is not; it is arithmetic, and the arithmetic is severe enough to be worth doing.

Suppose you pooled over everything. Write the instruction's mean token vector as c and the text's mean token vector as t, both unit length. With m instruction tokens and L text tokens, the pooled vector is a weighted sum with weights w = m/(m+L) and 1 − w:

Now take two texts a and b that carry the same instruction, so they share the same c. Assume for clarity that c is orthogonal to both text vectors. Then:

Work a short instruction: m = 12 tokens, L = 32 tokens, so w = 12/44 = 0.2727 and 1 − w = 0.7273. Then w² = 0.07437, (1−w)² = 0.52897, and ‖e‖² = 0.60334.

A 12% loss of dynamic range from a short instruction. Now a long one: m = 30, L = 32, so w = 0.4839, w² = 0.23416, (1−w)² = 0.26636, ‖e‖² = 0.50052.

Forty-seven percent of the usable range destroyed, and every score compressed into the top of the scale — the classic symptom of a badly-pooled embedder, where everything looks 0.8-similar to everything.

BGE: one instruction, and only on the query

Xiao and colleagues' C-Pack release (arXiv:2309.07597) — models, benchmark, and data together — took a deliberately narrower position. BGE uses a single fixed instruction, and applies it only to the query side:

Documents get nothing. No instruction, no prefix, raw text.

At first glance this looks like a weaker version of INSTRUCTOR. It is a different optimisation target, and the reason is a production constraint that a benchmark cannot see.

Read row two. A corpus of 400 million documents at 1024 dimensions in fp16 is 400M × 1024 × 2 = 819 GB. Multiply by T tasks and you are shipping a rack per task. Query-side-only instruction keeps the index task-agnostic, which is the difference between "conditioning is a feature" and "conditioning is a capacity plan."

Worked example: what an instruction actually costs

"Index size multiplies by the number of tasks" is the headline. Do the arithmetic on both sides, because the surprising part is which resource actually binds.

Take a 15-token instruction, an 8-token query, and a 200-token passage, on a BERT-base-class encoder. A transformer forward pass costs roughly 2 × (parameters) × (tokens) FLOPs, so at 110M parameters that is 2.2 × 10⁸ FLOPs per token.

The query side nearly triples, which sounds alarming and is not: 5.1 GFLOP is well under a millisecond on any accelerator, and query volume is small compared with corpus volume. The passage side rises by only 7.5% per document, which sounds negligible. Both intuitions are wrong about what matters.

Compute the full corpus pass for 400 million documents:

At an achieved 10¹⁵ FLOP/s that is about 4.9 GPU-hours — hours on a small fleet. Re-embedding an entire 400-million-document corpus is affordable. So compute is not the constraint, and every summary that says "instructions on documents are too expensive to compute" has the argument backwards.

The constraint is the other two numbers. Storage: 400M × 1024 dims × 2 bytes = 819 GB per index, times T tasks. And operations: each re-index is a migration — build, validate, dual-write, cut over, keep the old one until you are sure — and you now run T of those pipelines, each of which can drift out of sync with the others in exactly the silent way Chapter 4 describes.

Where BGE's [CLS] vector comes from: RetroMAE

One more BGE decision is worth unpacking, because it is a different answer to a question E5 answered with prefixes: how do you make the pooling slot carry the sentence?

E5 mean-pools, which is a safe default: no single position has to be special, because every position contributes. BGE pools the [CLS] position — a single token that was never, in ordinary BERT pretraining, trained to summarise anything (its original job was next-sentence prediction, which BERT variants later dropped entirely). Pooling an untrained slot is a bad idea, so BGE trains it first, with RetroMAE.

The design is an information bottleneck argued into existence by handicapping the decoder. Make the decoder strong and it will cheat by using surrounding tokens; make it one layer and mask most of the input, and the bottleneck becomes the only path. The result is a [CLS] position that already behaves like a sentence summary before contrastive training begins, which is why BGE can afford to pool it.

Worked example: five sentences, three geometries

The claim "one model, many geometries" deserves to be made concrete. Take five short texts from a restaurant-review corpus:

Three instructions, three cosine tables. These are representative magnitudes, not measurements from a specific checkpoint — the point is the pattern of which pairs win.

Instruction A: "Represent the review for clustering by cuisine type:"

Instruction B: "Represent the review for classifying sentiment:"

Look at the pair (S₁, S₂): 0.81 under instruction A, 0.22 under instruction B. Same two sentences, same weights, same forward pass architecture — a similarity that moves by 0.59 depending on a sentence of English prepended to the input. And (S₁, S₃) moves the opposite way, 0.28 to 0.79. The nearest-neighbour graph is completely rewired.

This is what "many geometries" means operationally. If you run k-means on the instruction-A vectors you get cuisine clusters. Run it on the instruction-B vectors and you get sentiment clusters. You did not train two models, and you did not train a classifier — you asked the same model a different question.

The honest question: does it read the instruction, or memorise it?

Time to be sceptical about the thing we just spent a chapter admiring.

There are two stories consistent with the evidence above. In the compositional story, the model genuinely parses the instruction and configures itself accordingly, so a novel instruction it has never seen produces the appropriate novel geometry. In the lookup story, the model has learned a few dozen instruction templates as effectively opaque task identifiers, and a novel instruction is snapped to the nearest memorised one — or produces garbage.

The evidence is mixed and leans uncomfortably toward the second. Instruction-tuned embedders are measurably sensitive to paraphrases of their instructions: rewording "Represent the news article for clustering by topic:" as "Group these news stories by subject:" can move scores materially, which is not what a model that understood the instruction would do. The structured template INSTRUCTOR uses — domain, text type, objective — is partly a response to this: it keeps novel instructions inside the grammar the model was trained on, so the lookup has something close to snap to.

None of this makes instruction conditioning a bad idea. It makes it a partially delivered idea, and knowing which part is delivered is the difference between deploying it well and being surprised in production. Chapter 6 shows what happens when the substrate is a model that genuinely does read instructions for a living.

Chapter 6: Decoders Become Embedders

By 2024 an obvious embarrassment had appeared. The best language understanding on the planet lived in decoder-only LLMs — seven billion parameters, trained on trillions of tokens, capable of following instructions in ninety languages. And the best text embedders were still 110-million-parameter BERTs trained on three billion words in 2018.

Everyone could see the arbitrage. Three papers in five months took three different routes to it, and comparing them is the single most efficient way to understand what an embedder actually requires.

Because you cannot simply use an LLM as an embedder. Three things are in the way.

Deriving the causal-mask problem

This obstacle is the deep one, so build it from zero.

In a bidirectional encoder like BERT, self-attention at every layer lets every position attend to every other. After twelve layers, the final state at position 3 has integrated information from all L tokens. Every position is a complete-information summary written from a different vantage point. Pool them however you like — mean, max, first — and you are averaging complete summaries.

In a causal decoder, position i can attend only to positions 1 through i. That constraint is not a bug; it is what makes next-token prediction a well-posed task. But it means the final states are not comparable objects.

Define the visibility of position i as the fraction of the sequence its final state has read: v_i = i / L. Now compute what mean pooling actually averages, for L = 8:

For a general L the sum is L(L+1)/2, so the average visibility is (L+1)/(2L), which tends to exactly one half. At L = 512: 131,328 / 262,144 = 0.5010.

So mean pooling is out. What about the last position, the only one with full visibility?

It works, and it has its own defect. In a causal LM the state at position L is read by the language-model head to produce a distribution over token L+1. Gradient descent has spent trillions of tokens shaping that state to answer "what word comes next?", which is a local, syntactic, high-frequency question. If a passage ends with "…and the", the final state is overwhelmingly about grammar. It is not a summary; it is a prediction.

Path 1 — E5-Mistral: change nothing, generate everything

Wang and colleagues (arXiv:2401.00368) took the most audacious position: leave the architecture almost alone, skip the contrastive pretraining stage entirely, and fix the data problem by having a language model write the data.

The pooling fix is minimal. Append an explicit [EOS] token to every input and pool that position. The [EOS] has full visibility, and because it is the terminal token, its "next token" is nothing — so contrastive fine-tuning is free to repurpose that state as a summary without fighting a strong pretrained prior about what should follow.

The data fix is the paper's real contribution, and it is a two-step prompting recipe:

Compare that with Chapters 1 and 3. E5 scraped 1.3 billion pairs and filtered them to 270 million to get clean positives; then it mined hard negatives from a corpus and distilled a cross-encoder to denoise them. E5-Mistral asks for a clean positive and a hard negative in the same API call. The two hardest problems in the first half of this lesson become prompt engineering.

The training fix is LoRA. Mistral-7B is fine-tuned with Low-Rank Adaptation at rank 16 for fewer than a thousand steps. LoRA replaces a full weight update with a low-rank one: instead of learning a whole new W of shape (4096, 4096), learn B of shape (4096, 16) and A of shape (16, 4096) and use W + BA. Count the parameters:

Why does this matter here specifically? Because the whole premise is that the LLM already knows things. A full fine-tune on 500k synthetic examples would happily overwrite trillions of tokens of pretraining with the idiosyncrasies of one generated dataset. LoRA's low-rank constraint makes catastrophic overwriting geometrically difficult — the update simply does not have the capacity to rewrite the model.

Path 2 — LLM2Vec: perform surgery on the mask

BehnamGhader and colleagues (arXiv:2404.05961) attacked the obstacle directly. If the causal mask is the problem, remove the causal mask.

Step 2's off-by-one is the kind of detail that separates a paper that works from one that does not. The pretrained language-model head was trained on exactly one mapping: hidden state at position i − 1 → token i. Ask it instead to map position i to token i and you are using a calibrated head for an uncalibrated purpose, and it has to relearn its entire input convention. By keeping the shift, MNTP reuses the pretrained head as-is and spends its gradient budget on the only thing that actually changed — how the attention layers behave without a causal mask.

Step 3 is worth dwelling on because it eliminates the entire first half of this lesson. SimCSE constructs a positive pair out of nothing: run the same sentence through the network twice, with dropout active, and the two resulting vectors differ slightly. Train them to be close, and everything else in the batch to be far. There is no CCPairs, no consistency filter, no annotation. The only requirement is a corpus of raw text — LLM2Vec uses Wikipedia.

Because mask flipping restored full visibility to every position, LLM2Vec can go back to mean pooling, which is now a legitimate operation: it averages complete summaries again, exactly as in BERT.

Path 3 — NV-Embed: build a pooler worth having

Lee and colleagues at NVIDIA (arXiv:2405.17428) accepted both prior moves — remove the causal mask, use a strong decoder — and asked the remaining question: if neither mean nor last-token pooling is right, what is?

Their answer is a latent attention layer, and it is the cleanest idea in this chapter. Introduce a trainable array of latent vectors — think of it as a learned dictionary of concepts, of shape (512, 4096). Then run one cross-attention step in which the sequence's hidden states are the queries and the latent dictionary supplies the keys and values:

Why is this better than plain mean pooling? Because plain mean pooling weights every token equally — the subject noun, the definite article, the trailing punctuation, the boilerplate footer. Latent attention lets each token first project itself onto a learned basis of 512 concept directions, so what gets averaged is not raw token states but dictionary coefficients. Uninformative tokens have nothing to retrieve and contribute near-uniform, low-magnitude mixtures; informative tokens light up specific latents. The averaging happens in a space where "unimportant" has a representation.

And why is it better than last-token pooling? Because last-token pooling routes an entire document through the state of one position — a bottleneck of 4096 numbers that also has to satisfy whatever the language model wanted that position to be. Latent attention reads all L positions.

Note the cost is linear in L: one cross-attention against a fixed 512-entry dictionary, not a quadratic self-attention. Pooling stays cheap.

NV-Embed's second finding: when to switch off in-batch negatives

NV-Embed trains in two instruction-tuned stages. Stage one is retrieval, with in-batch negatives plus mined hard negatives — the Chapter 2 and 3 recipe. Stage two blends in non-retrieval tasks (classification, clustering, semantic similarity) and disables in-batch negatives.

That sounds like an arbitrary knob. Derive it and it becomes forced.

Return to Chapter 2's false-negative calculation, but for a classification dataset with C classes. Two examples of the same class are, for the purposes of a classification embedding, positives — you want them close. In-batch negatives will treat them as negatives. The chance that a batch of N contains at least one same-class item as the anchor is

Check the binary row: 1 − 0.5³¹ = 1 − 4.66 × 10⁻¹⁰. Every single batch, half the "negatives" are same-class items you are actively teaching the model to separate. This is not a small amount of noise. It is a second, opposing objective running concurrently with the one you wanted.

The three paths side by side

Read down the rows and the point of this lesson appears. Three teams, one substrate family, and every one of them made a different choice in every slot. There is no single recipe. There is a design space, and each of these papers is one point in it with an ablation table attached.

What the seven billion parameters cost

Before you reach for a 7B embedder, price it against the 335M one honestly.

Twenty times the encoding compute and four times the storage for four to seven points of benchmark average. Whether that trade is correct is a product question, not a research question, and it depends entirely on whether your bottleneck is retrieval quality (Chapter 0's leverage argument says yes, often) or serving cost.

The dimension half of the problem has a standard mitigation worth knowing: Matryoshka Representation Learning (arXiv:2205.13147) trains a model so that the first 256 or 512 coordinates of its embedding are themselves a usable embedding, by adding loss terms on truncated prefixes. You store the short vector for the first-pass search and the full vector for reranking. It converts an all-or-nothing dimension choice into a runtime knob, and most 2024-onward embedders ship with it.

Chapter 7: InfoNCE, By Hand

Six chapters of machinery. This chapter runs one complete training step through all of it — normalisation, prefix conditioning, the similarity matrix, the temperature, the loss, and the gradient — with every number computed by hand. Nothing is skipped and nothing is approximated except where the arithmetic says so.

We work in two dimensions. Real E5 uses 768 or 1024; two is what fits on paper and on a circle you can look at. One consequence to hold on to: two dimensions is a crowded place. Six unit vectors in 2-D cannot get far from each other, so converged cosines will look implausibly high, around 0.99. In 768 dimensions the same relative structure appears at absolute cosines around 0.7–0.8. What transfers is the margin — the gap between the right answer and the best wrong one — not the absolute value.

The batch

Three query–passage pairs, which is our entire batch. N = 3.

The encoder produces these raw, unnormalised 2-D outputs. Read the pattern before the arithmetic: the three queries all point mostly along dimension 1, and the three passages all point mostly along dimension 2. That is not about topic. It is register — short interrogative fragments versus long declarative sentences — and it is exactly the surface-form geometry Chapter 4 warned about.

Check one normalisation by hand so the rest are trustworthy. For q₁ = [9, 3]: 9² + 3² = 81 + 9 = 90, and √90 = 9.48683. Then 9 / 9.48683 = 0.94868 and 3 / 9.48683 = 0.31623. Confirm it landed on the unit circle: 0.94868² + 0.31623² = 0.90000 + 0.10000 = 1.00000. Good.

Stage A: the similarity matrix with no prefix

Nine dot products. Two worked in full, the rest by the same recipe.

The full matrix. Rows are queries, columns are passages, and the diagonal is what we want to win:

Accuracy: 1 out of 3, and it is worse than that number sounds. Every query picks p₁. Not because p₁ is about eggs, but because p₁ is the passage whose direction sits closest to the whole query cluster. It is a hub.

Stage B: the prefix map

Chapter 4 argued that "query: " works by reconfiguring the encoder's processing, and that its effect is approximately a linear map applied to the query representation. In this toy we write that map down explicitly. Suppose training has produced

applied to the raw query output, before normalisation. Read what it does: it shrinks dimension 1 (the register axis), amplifies dimension 2, and mixes them. Apply it:

Normalise each:

Notice what the map achieved geometrically. The unprefixed queries were bunched between 18° and 36° — a spread of 17°. The prefixed ones sit between 50° and 82° — a spread of 31°, in the same neighbourhood as the passages. The prefix did two things: it moved the queries into passage territory, and it spread them out. The second is as important as the first, because a spread of 17° cannot possibly distinguish three passages spread over 30°.

New similarity matrix. One worked in full:

Three out of three. The encoder weights did not change — the passage embeddings are byte-for-byte the same numbers as in Stage A. All that changed is a linear map applied to the query side, which is what conditioning on a prefix buys you.

Stage C: the loss, at three temperatures

Recall the per-row loss from Chapter 2, written in the form that is easiest to compute: take every similarity relative to the positive, divide by τ, exponentiate, sum, and take the log.

That form is worth a sentence of its own: the loss depends only on differences from the positive. Absolute similarity is irrelevant. This is why comparing cosine values across models is meaningless and comparing margins is not.

Stage B (prefixed), τ = 0.01, row 1. Differences: 0.93119 − 0.99866 = −0.06747, and 0.83414 − 0.99866 = −0.16452. Divide by 0.01: −6.747 and −16.452. Exponentiate: e^−6.747 = 0.001177, e^−16.452 = 7.2×10⁻⁸. Sum with the 1:

Row 2. Differences −0.03690 and −0.03176; divided by τ, −3.690 and −3.176; exponentials 0.024960 and 0.041782. L₂ = ln(1.066742) = 0.06460.

Row 3. Differences −0.12052 and −0.01469; divided, −12.052 and −1.469; exponentials 5.8×10⁻⁶ and 0.230234. L₃ = ln(1.230240) = 0.20720.

Now the same computation on Stage A, the unprefixed matrix, and this is where the temperature earns its chapter.

Row 1's positive is already the row maximum, so every difference is negative and the loss is essentially zero: e^−22.22 + e^−40.32 = 2.2×10⁻¹⁰, giving L₁ = 2.2×10⁻¹⁰.

Row 2's positive is p₂ at 0.70711, but p₁ scores 0.89443 — higher. The difference is positive: 0.89443 − 0.70711 = +0.18732, divided by τ gives +18.732, and e^18.732 = 1.365×10⁸. The denominator is dominated by that single term, so

Look at that number and then look back at the difference: 18.732 is exactly 0.18732 / 0.01. Row 3 does the same thing: its deficit is 0.95323 − 0.66750 = 0.28573, and L₃ = 28.573.

Stage A: 15.768. Stage B: 0.09099. A factor of 173, from a linear map on one side of the batch. And now sweep the temperature to see how much of that gap is real and how much is the ruler:

At τ = 1 both models sit at roughly ln 3 and the loss cannot tell them apart — a model that gets 1 of 3 right and a model that gets 3 of 3 right differ by four percent. At τ = 0.01 they differ by a factor of 173. The temperature is not a detail of the loss; it decides whether the loss can see the thing you care about.

Trace one row to feel the mechanism. Stage A row 2 at τ = 1: the exponentials are e^0.18732 = 1.2060 for the wrong winner, 1 for the positive, and e^−0.1639 = 0.8488 for the third, summing to 3.0548 — the wrong answer holds 39.5% of the mass and the right one 32.7%. The model is losing, and the loss reports 1.117, barely distinguishable from the 1.099 of a model that has learned nothing. Divide the same three differences by 0.01 instead and the wrong answer holds 99.9999993% of the mass. Same embeddings. Same ordering. Completely different gradient.

Stage D: the gradient, and where it points

Differentiate. Writing s_ij = q_i · p_j and P_ij for the row softmax of s_i·/τ:

In words: the gradient is the softmax-weighted average of all passages, minus the correct passage, scaled by 1/τ. Gradient descent moves q in the opposite direction — toward the positive, away from whatever the model is currently betting on.

Compute it for Stage A, row 2, the badly wrong one. The row softmax at τ = 0.01 is dominated by p₁: relative weights are 1 for p₁, e^−18.732 = 7.3×10⁻⁹ for p₂, and e^−35.12 for p₃. So P_2· ≈ (1, 0, 0), and the weighted average is just p₁:

Magnitude: √(28.377² + 14.868²) = √(805.25 + 221.06) = √1026.31 = 32.04. The descent direction is (−28.377, +14.868): shrink dimension 1, grow dimension 2. That is precisely "become less query-shaped and more passage-shaped", and it is what accumulates, over 270 million examples, into the prefix map of Stage B.

Now Stage B, row 3, the least confident of the three correct rows. Its softmax is P_3· = (4.7×10⁻⁶, 0.18715, 0.81285). The weighted average of the passages is

Subtract p₃ = (0.11043, 0.99381) and scale by 100: (3.852, −0.845), magnitude 3.94. Eight times smaller than Stage A's, and pointing almost tangentially — the model is nearly right and is being nudged, not shoved.

Set the prefix to 0% and press Train. The queries crawl out of their register cluster and rotate toward their own passages — you are watching, in miniature, the thing that 270 million pairs and a 32,768-wide batch do over two weeks on a GPU fleet. Then set the temperature to 1.0 and press Train again: the same gradients, a hundred times weaker, and the diagonal never quite wins.

Chapter 8: Reading MTEB Honestly

Every model in this lesson is compared to every other one by a single number. That number governs which embedder ends up in your stack, which papers get written, and which research directions get funded. It deserves an hour of scepticism.

MTEB — the Massive Text Embedding Benchmark (Muennighoff et al., arXiv:2210.07316) — was a genuine service to the field. Before it, every embedding paper evaluated on a different handful of datasets, chosen after the fact. MTEB fixed a suite, published a leaderboard, and made results comparable. That is a real contribution and this chapter is not an attack on it.

It is an attack on reading only the first column.

What the average actually averages

The English MTEB suite is 56 datasets across seven task types. The headline number is the unweighted mean over the 56 datasets — not over the seven task types. That distinction is the whole chapter.

Read the third column as an editorial statement, because that is how the field uses it: retrieval matters fifteen times as much as summarization. Now ask where that ratio came from. Not from a survey of what embeddings are used for. Not from a weighting decision anyone defended in the paper. It came from how many suitable public datasets happened to exist for each task type in 2022.

Worked example: what does a one-point average gap mean?

Model B beats Model A by 1.00 on the MTEB average. What has actually been observed?

The average is a sum over 56 datasets divided by 56, so a 1.00 average gap is exactly 56 points of total dataset-level movement. That total can be distributed in wildly different ways, and the distributions are not equivalent for your purposes.

Check the adversarial row: 15 × 5.00 = 75.0 points gained; 41 × 0.463 = 18.98 points lost; net 56.02 points, divided by 56 = +1.0004 average. Model B climbs the leaderboard by a full point while being worse than A on forty-one of the fifty-six datasets. Both statements are true simultaneously, and one column can only show you one of them.

A real pair that makes the point

This is not hypothetical. Two models from the same lab, built on the same T5 backbone, differing only in training data:

Half a point apart on the headline. Six and a third points apart on retrieval. If you are building search or RAG, GTR-XXL is not marginally better, it is in a different class — and the leaderboard reports them as tied. The reason is visible in their training: Sentence-T5 was trained largely on symmetric community-QA and inference pairs, GTR on retrieval data. They learned two different relations, exactly as Chapter 4 predicted, and the average dissolves that distinction into a rounding error.

The contamination problem

MTEB's retrieval section is essentially BEIR, and BEIR was designed as a zero-shot benchmark: score well without having trained on these domains. That framing is now largely false, for a mundane reason. Several BEIR datasets have public training splits, and nearly everyone trains on them.

So a "zero-shot retrieval average" over 15 datasets is, for a typical modern model, an average over roughly 5 in-domain datasets and 10 genuinely held-out ones. Those two groups measure different things: the first measures fit, the second measures transfer. Averaging them produces a number that measures neither.

Is the average at least statistically stable?

A common complaint is that leaderboard gaps are "just noise." Let us check, because the honest answer is more interesting than the complaint.

Different task types have different amounts of run-to-run variance. Classification uses a logistic probe with a fixed split — nearly deterministic. Retrieval and STS are deterministic given the embeddings. Clustering is not: it runs k-means, whose initialisation is random, and V-measure genuinely moves between seeds. Assign plausible per-dataset standard deviations and propagate them:

So the average is reproducible to about ±0.13 at two standard deviations. A 0.5-point gap is real. A 0.05-point gap — and leaderboards do report two decimal places — is not.

What MTEB cannot see at all

Distribution across task types is the failure people can at least detect from published numbers. These are the ones no published number contains.

Look at row one and row four together. The two properties most likely to decide whether a model works for you — how it handles your document length, and what it costs to store — are both entirely absent. That is not a flaw anyone can fix by adding datasets; a benchmark measures what it measures.

Worked example: how big does your own evaluation need to be?

Everyone agrees you should evaluate on your own data. Almost nobody computes whether their evaluation can detect the difference they are looking for. Do it once, properly, and the answer is memorable.

Say recall@10 is around 0.75 and you want to detect a 5-point difference between two models.

Unpaired — different query samples for each model, or the same sample treated as independent. The standard error of one proportion at n = 200 is

The difference of two independent estimates has standard error 0.0306√2 = 0.0433, so the 95% confidence interval on the gap is ±0.085. You can only resolve differences larger than 8.5 points. Your 5-point effect is invisible.

Paired — both models on the same 200 queries. Now the query's intrinsic difficulty cancels, and only the queries where the two models disagree carry information. This is McNemar's test. Suppose B succeeds where A fails on b = 18 queries, and A succeeds where B fails on c = 8. The observed gap is (18 − 8) / 200 = +5.0 points, exactly the effect we wanted to detect.

Under the null hypothesis that the models are equally good, b is Binomial(26, 0.5) — 26 discordant queries, each a coin flip. Compute the exact tail by summing binomial coefficients:

Suggestive, not conclusive. So the honest verdict on 200 paired queries is: it will show you a 5-point effect and it will not let you swear to it.

What would settle it? Double to 400 queries with the same discordance rate: b = 36, c = 16, 52 discordant. The null mean is 26 with standard deviation √(52 × 0.25) = 3.606, so the continuity-corrected z is (36 − 0.5 − 26) / 3.606 = 2.634, giving a two-sided p of 0.008. Four hundred paired queries is conclusive.

The fourth problem: the leaderboard is a training signal

MTEB is public, fixed, and famous. Nobody needs to train on its test sets for it to be overfit. Selection is enough: you try eight data mixtures, four pooling strategies, and three instruction templates, and you keep the one with the best MTEB score. No test label ever touched a gradient, and you have still fitted the benchmark — through your own decisions.

Quantify the pressure. If you evaluate 100 candidate configurations and pick the best, you are taking a maximum over 100 draws. With a per-configuration standard deviation of 0.064, the expected inflation of the winner is roughly 0.064 × √(2 ln 100) = 0.064 × 3.03 = 0.19 points of pure selection luck — small on its own, but exactly the size of the gaps that reorder the top of the leaderboard. And that is the optimistic estimate: real configuration choices differ by far more than noise, so the selection is fitting genuine benchmark-specific structure, not just noise.

Push both sliders to the right. The headline stays at +1.00, the retrieval sub-average climbs past +5, and the number of datasets B wins falls to 15 of 56. Two models, one number, four completely different products.

The protocol: how to actually choose an embedder

None of the above means "ignore the benchmark." It means use it for what it is good at — a broad first-pass filter — and then do the ten minutes of work that determines your outcome.

Step 3 is where the worked example above pays off. Paired evaluation is not a refinement of unpaired evaluation; it is the difference between an experiment that can answer your question and one that cannot, at the same labelling cost. And the sample size is not a matter of taste — it is four lines of binomial arithmetic that tell you, before you label anything, whether the study you are about to run is capable of detecting the effect you are looking for.

Chapter 9: The Design Space

Nine chapters, six papers, one recurring structure. This chapter lays the structure out flat, because the point of reading six papers instead of one is to stop seeing recipes and start seeing a space with coordinates.

Every text embedder in this lesson is a choice in five slots plus three cross-cutting axes. Here is the whole grid.

The five slots, filled in

Slot 1 — Substrate. What network are you starting from?

Slot 2 — Data. Where do positive pairs come from?

Slot 3 — Objective. What does the loss say?

Slot 4 — Pooling. How do L token vectors become one?

Slot 5 — Conditioning. What tells the model which relation to encode?

Three cross-cutting axes

These are orthogonal to the five slots and are usually decided late, which is why they are usually decided badly.

Reading the grid as an invention procedure

Look back at how the papers in this lesson were actually produced. Each is a transplant.

The empty cells

Now use the grid the way it is meant to be used. Here are six combinations that the six papers between them never tried, with the reason each is plausible and the reason nobody has done it.

Cheat sheet: what to actually use

The one paragraph to keep

A text embedder is a function from a string to a direction. Everything in this lesson is an answer to two questions about that function: which pairs of strings should point the same way, and how do you get enough examples of that to matter. E5 answered the second with a scraper and a filter; E5-Mistral answered it with a prompt. E5 answered the first with two prefixes; INSTRUCTOR answered it with a sentence. The loss barely changed across six papers and four years. The data and the conditioning changed completely. That is the shape of progress in this field, and it is a useful thing to know before you go looking for a better loss.

References

1. Wang, L., Yang, N., Huang, X., Jiao, B., Yang, L., Jiang, D., Majumder, R., Wei, F. "Text Embeddings by Weakly-Supervised Contrastive Pre-training," 2022 — arXiv:2212.03533. The anchor paper: CCPairs, consistency filtering, the two-stage recipe, and the prefixes.
2. Xiao, S., Liu, Z., Zhang, P., Muennighoff, N. et al. "C-Pack: Packed Resources For General Chinese Embeddings," 2023 — arXiv:2309.07597. BGE, RetroMAE-first three-stage training, and query-side-only instruction.
3. Su, H., Shi, W., Kasai, J. et al. "One Embedder, Any Task: Instruction-Finetuned Text Embeddings," 2022 — arXiv:2212.09741. INSTRUCTOR, MEDI, and instruction-excluded pooling.
4. Wang, L., Yang, N., Huang, X., Yang, L., Majumder, R., Wei, F. "Improving Text Embeddings with Large Language Models," 2024 — arXiv:2401.00368. E5-Mistral: synthetic data, LoRA, EOS pooling, fewer than 1k steps.
5. BehnamGhader, P., Adlakha, V., Mosbach, M., Bahdanau, D., Chapados, N., Reddy, S. "LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders," 2024 — arXiv:2404.05961. Bidirectional-attention surgery, MNTP, unsupervised SimCSE.
6. Lee, C., Roy, R., Xu, M. et al. "NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models," 2024 — arXiv:2405.17428. Latent-attention pooling and two-stage instruction tuning.
7. Muennighoff, N., Tazi, N., Magne, L., Reimers, N. "MTEB: Massive Text Embedding Benchmark," 2022 — arXiv:2210.07316. The benchmark Chapter 8 takes apart.
8. Thakur, N., Reimers, N., Rücklé, A., Srivastava, A., Gurevych, I. "BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models," 2021 — arXiv:2104.08663. Where the BM25 baseline that E5 had to beat lives.
9. van den Oord, A., Li, Y., Vinyals, O. "Representation Learning with Contrastive Predictive Coding," 2018 — arXiv:1807.03748. The origin of the InfoNCE loss.
10. Chen, T., Kornblith, S., Norouzi, M., Hinton, G. "A Simple Framework for Contrastive Learning of Visual Representations" (SimCLR), 2020 — arXiv:2002.05709. Where large batches and low temperature became standard.
11. Radford, A. et al. "Learning Transferable Visual Models From Natural Language Supervision" (CLIP), 2021 — arXiv:2103.00020. The two-tower contrastive template everything here inherits.
12. Karpukhin, V. et al. "Dense Passage Retrieval for Open-Domain Question Answering" (DPR), 2020 — arXiv:2004.04906. The two-tower design the prefix replaced.
13. Gao, T., Yao, X., Chen, D. "SimCSE: Simple Contrastive Learning of Sentence Embeddings," 2021 — arXiv:2104.08821. Dropout as a positive-pair generator; LLM2Vec's step 3.
14. Xiao, S., Liu, Z., Shao, Y., Cao, Z. "RetroMAE: Pre-Training Retrieval-oriented Language Models Via Masked Auto-Encoder," 2022 — arXiv:2205.12035. How BGE's [CLS] slot gets trained.
15. Qu, Y. et al. "RocketQA: An Optimized Training Approach to Dense Passage Retrieval," 2020 — arXiv:2010.08191. The false-negative diagnosis behind Chapter 3.
16. Xiong, L. et al. "Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval" (ANCE), 2020 — arXiv:2007.00808. Hard-negative mining with periodic index refresh.
17. Hofstätter, S. et al. "Improving Efficient Neural Ranking Models with Cross-Architecture Knowledge Distillation," 2020 — arXiv:2010.02666. Cross-encoder to bi-encoder distillation.
18. Hofstätter, S. et al. "Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Sampling" (TAS-B), 2021 — arXiv:2104.06967. Batch composition as a design variable.
19. Reimers, N., Gurevych, I. "Sentence-BERT," 2019 — arXiv:1908.10084. The symmetric baseline Chapter 4 breaks.
20. Ni, J. et al. "Sentence-T5," 2021 — arXiv:2108.08877; and Ni, J. et al. "Large Dual Encoders Are Generalizable Retrievers" (GTR), 2021 — arXiv:2112.07899. The pair Chapter 8 uses to show what an average hides.
21. Kusupati, A. et al. "Matryoshka Representation Learning," 2022 — arXiv:2205.13147. Truncatable embeddings.
22. Hu, E. et al. "LoRA: Low-Rank Adaptation of Large Language Models," 2021 — arXiv:2106.09685. How E5-Mistral fine-tunes 7B without erasing it.
23. Li, Z. et al. "Towards General Text Embeddings with Multi-stage Contrastive Learning" (GTE), 2023 — arXiv:2308.03281; Nussbaum, Z. et al. "Nomic Embed," 2024 — arXiv:2402.01613. Two more points in the same design space, with different data-mixture choices.

Connections

This lesson sits in the middle of a chain. Below it are the primitives; above it are the systems that consume the vectors.