Dense Passage Retrieval and the Craft of Negatives

Chapter 1: Sparse vs Dense, Honestly

To understand precisely what dense retrieval buys, you have to first understand precisely what BM25 is — not as a black box labelled "the baseline", but as a specific vector space with specific properties. Let us derive it, because the derivation is where its ceiling becomes visible.

Building BM25 from three ideas

Start with the crudest possible scorer: count how many query terms appear in the passage. Query "irish sea between england ireland", passage about the Irish Sea — four matches. Passage about Irish literature — one match. Useful, and immediately broken in three ways.

Problem 1: not all terms are equally informative. The term the appears in essentially every passage, so matching on it tells you nothing. The term Thoros appears in perhaps a dozen passages out of 21 million, so matching on it tells you almost everything. We need a weight that rises as a term gets rarer. That weight is inverse document frequency:

where N is the number of passages in the corpus and n(t) is how many of them contain term t. Let us feel it with real magnitudes. With N = 21,015,324:

Check the first row by hand so the formula stops being decoration. n = 2.0×10⁷, N = 2.1015×10⁷. The numerator is 2.1015×10⁷ − 2.0×10⁷ + 0.5 = 1,015,324.5. The denominator is 2.0×10⁷ + 0.5. The ratio is 0.050766. Add 1 and take the log: ln(1.050766) = 0.04952. And the last row: (21,015,324 − 12 + 0.5)/12.5 = 1,681,225, plus 1, log = 14.3352. The point is the spread: matching "thoros" is worth roughly 290 times as much as matching "the".

Problem 2: repetition saturates. A passage that says "sea" ten times is more about the sea than one that says it once — but not ten times more. The tenth occurrence adds almost nothing. So we pass the raw term frequency f(t,D) through a saturating function:

With the standard k₁ = 1.2, walk the curve: f = 1 gives 2.2/2.2 = 1.000; f = 2 gives 4.4/3.2 = 1.375; f = 5 gives 11/6.2 = 1.774; f = 20 gives 44/21.2 = 2.075; f → ∞ approaches k₁+1 = 2.2. The first occurrence buys you 1.000. The next nineteen occurrences together buy you 1.075. That is the saturation, and it is the entire reason BM25 beats naive term counting.

Problem 3: long passages match everything. A 2,000-word passage contains more of every term by accident. So we normalise by length relative to the corpus average, with a knob b controlling how aggressively:

with the near-universal defaults k₁ = 1.2, b = 0.75. DPR's corpus is chunked into fixed 100-word passages, so |D| = avgdl = 100 for every passage and the length term collapses to exactly 1 — a small, quietly important detail: fixed-length chunking removes one of BM25's three ideas, and equally removes one of its failure modes.

A BM25 score, computed by hand

Question: "who is the bad guy in lord of the rings". After stopword removal the scoring terms are bad, guy, lord, rings. Two candidate passages, with |D| = avgdl so the denominator is f + 1.2:

Passage A, the one that answers the question:

Passage B, which is about ticket sales and answers nothing:

The wrong passage wins, by 14%, and it wins for the right reason under BM25's own logic: it says "Lord of the Rings" one more time. The two terms that would have decided the question — bad guy against villain — contributed a total of zero to both scores. There is no setting of k₁, b, or the IDF formula that fixes this, because the fix would require the term villain in the passage to contribute to the query term bad, and BM25's sum runs over query terms matching themselves.

The dense alternative, in one sentence

Learn a function that maps text into R⁷⁶⁸, where the axes are not terms but whatever directions turn out to be useful, and where the map is trained so that a question lands near the passage that answers it.

That is the whole idea. Note carefully what changed and what did not:

The last row is where a great many production systems have quietly stayed sparse, and Chapter 7 gives that its due weight.

Where BM25 still wins — four cases, with the mechanism

A lesson that only shows you the winning cases has taught you a marketing slide. Here are the four regimes where the sparse index is genuinely the better tool, each with the reason it happens.

1. Rare literal strings. The DPR paper's own example: "Who plays Thoros of Myr in Game of Thrones?" BM25 ranks the correct passage first, because Thoros has IDF ≈ 14.3 and occurs in about a dozen passages — the term itself is nearly a unique identifier. DPR ranks a passage about a different Game of Thrones character above it. The mechanism: a 768-dimensional vector is an average of a passage's content, and averaging is lossy in exactly the way that hurts rare tokens. A single unusual proper noun in a 100-word passage contributes roughly 1% of the pooled representation, whereas in the sparse space it contributes an entire high-IDF coordinate.

2. Exact-match constraints. Numbers, dates, model codes, error strings, chemical formulae. "ORA-01555" and "ORA-01552" are one character apart, which is one coordinate apart in the sparse space and, for a subword-tokenised transformer trained on natural language, frequently indistinguishable in the dense one.

3. Out-of-domain corpora. A dense retriever is a learned function, so it inherits the distribution it was trained on. Move it to legal contracts, clinical notes, or a private codebase and its geometry is a guess. BM25's "geometry" is recomputed from the new corpus's own term statistics, for free, on ingest. This is not a small effect — it is the reason the BEIR benchmark exists, and the reason Contriever (Chapter 5) is an important paper.

4. Questions written while looking at the passage. This one is subtle and it produces the single most interesting number in DPR's results table. On SQuAD, BM25 beats DPR: 68.8% versus 63.2% at top-20. SQuAD's annotators wrote each question with the paragraph in front of them, so they naturally reused its vocabulary. That gives an artificially high lexical overlap between question and gold passage — precisely the regime BM25 was built for. Chapter 6 shows this is not the only thing wrong with SQuAD as a retrieval benchmark, but it is the main one.

Hybrid: adding the two scores

DPR itself reports a hybrid. Retrieve a large candidate pool with each system, then rerank the union with a linear combination of the two scores:

The effect is exactly what the four cases above predict — it helps most where the two systems fail on different questions, and it helps least where one of them is simply better:

Look at the NQ row for a moment. Naive intuition says combining a 78.4 system with a 59.1 system should land between them or above the better one. It lands below the better one. Fusing scores is not free: you are adding a noisy opinion to a good one, and with a fixed λ you cannot tell, per query, which opinion to trust. That observation is the seed of the entire learned-reranking literature.

The sparse vector, written out

Abstract claims about orthogonality become obvious once you write the vectors down. Take a toy vocabulary of eight terms and index them as coordinates 1 through 8:

The question "who is the bad guy in lord of the rings", after stopword removal, is the vector (with IDF weights in place of raw counts):

Passage A, the one that answers it — "portraying the villain Sauron in the Lord of the Rings trilogy":

Passage B, about box office receipts, saying "Lord of the Rings" three times:

Now take the inner products, coordinate by coordinate. For p_A: coordinates 1 and 2 contribute 3.9×0 + 5.1×0 = 0; coordinates 3 and 4 contribute 4.6×1.375 + 5.4×1.375 = 13.75; coordinates 5 and 6 contribute 0×1.0 + 0×1.0 = 0. Total 13.75. For p_B: 4.6×1.571 + 5.4×1.571 = 15.71.

Look at coordinates 5 and 6. The passage has mass there — villain and sauron, the two words that make it the right answer — and the query has zero. Their product is zero. It is not "small because the words are only loosely related". It is zero because multiplication by zero is zero, and the query has zero in that coordinate because the user did not type that word.

Why earlier dense attempts failed

People had been embedding documents into low-dimensional spaces since the 1990s. Knowing why those attempts did not displace BM25 tells you what the missing ingredient actually was — and it was not "deep learning".

Read the "why" column downwards and one word repeats: unsupervised. Every pre-2019 method built its geometry from co-occurrence statistics, which give you topical similarity for free and answer-hood never. The two ingredients DPR combines are contextual pretraining (so "villain" and "bad guy" are already near each other before you start) and contrastive fine-tuning against confusable negatives (so the space learns the difference between "same topic" and "answers this"). Neither alone is sufficient, which is exactly what Chapter 4's ablation table shows from the inside.

How sensitive is BM25 to its two knobs?

Before declaring the baseline beaten, it is fair to ask whether it was tuned. BM25 has exactly two hyperparameters, and it is worth knowing how much they can move.

k₁ controls term-frequency saturation. Take a passage where a query term appears 3 times, and vary k₁ in the tf component f(k₁+1)/(f + k₁):

Check the k₁ = 1.2, f = 3 cell: 3 × 2.2 / (3 + 1.2) = 6.6/4.2 = 1.571. And k₁ = 3.0, f = 10: 10 × 4 / 13 = 3.077.

b controls length normalisation, and DPR's fixed 100-word chunks make it inert: |D|/avgdl = 1 for every passage, so 1 − b + b·1 = 1 regardless of b. The entire second hyperparameter has no effect on this corpus.

The other sparse family: query likelihood

BM25 is not the only classical scorer, and it is worth knowing its main rival exists so "sparse retrieval" does not collapse into one formula in your head. Query-likelihood language models score a passage by the probability it would generate the query:

The second term is smoothing: it stops a single missing query term from sending the whole product to zero, and it plays the same role IDF plays in BM25 — terms that are common in the corpus contribute little because the background term already explains them.

Different derivation, different intuition, near-identical behaviour in practice, and identical failure on the lexical gap: P("bad" | passage about Sauron) is whatever the background says, because the word is not there. The limitation is not BM25's; it belongs to the term basis, and every member of the family inherits it.

Measure your own lexical gap in an afternoon

Whether dense retrieval will help you is an empirical question about your query distribution, and it is cheap to answer before building anything.

# For each (query, gold passage) pair in your eval set:
def idf_coverage(query, gold, idf):
    terms = [t for t in tokenize(query) if t not in STOPWORDS]
    if not terms: return 1.0
    hit = sum(idf[t] for t in terms if t in tokenize(gold))
    return hit / sum(idf[t] for t in terms)   # IDF-weighted, not raw overlap

Plot the histogram. The mass near 1.0 is the population BM25 will serve well. The mass below about 0.4 is your lexical gap — queries whose most informative terms simply do not occur in the passage that answers them.

Fusing two rankings, worked

Chapter 1 asserted that hybrid fusion is not free. Here is why, with numbers, on five candidate passages.

Linear fusion, score = BM25 + 1.1 × DPR. For P1: 15.7 + 1.1×42.1 = 15.7 + 46.31 = 62.01. For P2: 13.8 + 1.1×71.4 = 13.8 + 78.54 = 92.34. P2 wins comfortably — here.

Now change one thing that has nothing to do with relevance: suppose this corpus's BM25 scores run ten times larger, as they do on a corpus with different length statistics, so P1 scores 157 and P2 scores 138. Recompute: P1 = 157 + 46.31 = 203.31, P2 = 138 + 78.54 = 216.54. Still P2, but the margin has collapsed from 49% of the smaller score to 6.5%. One more factor of two on the BM25 side and the ranking flips, with no change in either system's judgement.

Reciprocal rank fusion avoids the problem by discarding the scores entirely and using only positions:

For P1: 1/61 + 1/64 = 0.016393 + 0.015625 = 0.032018. For P2: 1/62 + 1/61 = 0.016129 + 0.016393 = 0.032522. P2 wins — and it wins by the same amount no matter what scale either system's scores are on, today or after any retraining. The constant 60 damps the influence of the very top ranks so that one system cannot dominate on a single confident hit.

The cost of that robustness is real: RRF cannot express "BM25 was extremely confident about this one", because it has thrown that information away. In practice the calibration-freedom wins, which is why RRF is the more common production choice despite being the cruder idea.

BM25 is not language-neutral either

One more limitation of the sparse basis, easy to miss because English hides it. BM25's coordinates are tokens, and tokens come from a tokeniser, and tokenisers are language-specific.

The last row is the lexical gap taken to its limit, and it is where dense retrieval's advantage is least arguable: a multilingual encoder places "villain" and "Bösewicht" near each other for the same reason it places "villain" and "bad guy" near each other, and no amount of term engineering can produce a non-zero score between two disjoint vocabularies.

The cleverest classical patch, and exactly where it breaks

There is one more sparse technique worth knowing, because it is the closest classical IR ever came to bridging the lexical gap, and its failure mode is instructive.

Pseudo-relevance feedback — RM3 is the best-known variant — runs in two passes. First retrieve the top ten passages with plain BM25. Then assume they are relevant, mine their highest-IDF terms, add those terms to the query with learned weights, and retrieve again.

If villain makes it into the expansion set, the second pass finds the Sala Baker passage and the lexical gap is bridged — with no learning and no neural network. This genuinely works, and it is why BM25+RM3 rather than plain BM25 is the honest sparse baseline in modern papers.

Now the failure, which is structural. The expansion terms come from the top ten of the first pass, so RM3 can only bridge a gap when a document already found by term matching happens to contain the bridging word. Chapter 0's failure was that the first pass returns ten box-office and plot-summary passages. If none of them says "villain", the term never enters the expansion set and the second pass repeats the first.

Chapter 2: The Dual Encoder

We want a function that scores how well a passage answers a question. The most accurate such function is obvious and unusable, so let us start there — understanding why it is unusable is what forces the architecture.

The model you would build if compute were free

Concatenate the question and the passage, feed the pair into a single BERT, and let every question token attend to every passage token through twelve layers of self-attention. Read a relevance score off the [CLS] position. This is a cross-encoder, and it is meaningfully better than anything we are about to build, because it can perform reasoning that requires seeing both texts at once: "the question asks when, and this passage's only date is in a sentence about a different person."

Now price it. To retrieve for one question you must score every passage in the corpus:

One BERT-base forward pass on a 128-token input costs roughly 2 × 110×10⁶ params × 128 tokens ≈ 2.8×10¹⁰ FLOPs — call it 28 GFLOPs. Multiply:

On an accelerator sustaining 150 TFLOP/s that is 3,930 seconds — 65 minutes per question, on a full GPU. And it cannot be precomputed, because the passage representation depends on the question. Cross-encoders are not slow; they are categorically the wrong shape for first-stage retrieval. They reappear in Chapter 5 as rerankers over a shortlist of 100, where 100 × 28 GFLOPs = 2.8 TFLOPs = about 19 ms is perfectly reasonable.

The architecture, with every tensor shape

DPR is two BERT-base-uncased models, trained jointly but with entirely separate parameters. Call them E_Q for questions and E_P for passages. Here is the exact data flow — and this is the part you should be able to reproduce from memory, because the shapes are the model.

The passage tower, run offline over the whole corpus:

The question tower, run online, once per question:

The interaction, and this is all of it:

A dot product. No projection head, no MLP, no normalisation, no learned temperature. The paper tried alternatives and reports that L2 distance performs comparably while cosine similarity performs worse; a plain inner product is the choice, and Chapter 3 explains why the missing normalisation is doing real work.

Feel the cost inversion

Here is the number that makes bi-encoders viable, and it is worth computing rather than reading. Scoring one question against the entire corpus, once the passage vectors exist:

That is 32 GFLOPs to brute-force score the whole of Wikipedia — almost exactly the cost of a single BERT forward pass on one passage. Put differently: the compute you would have spent looking at passage number one under a cross-encoder is the compute the bi-encoder spends looking at all twenty-one million.

Explore the two shapes

Why two towers and not one?

Nothing forces E_Q and E_P to be different networks. Tying them halves the parameter count and is what several later models do. DPR unties them. Three reasons, in ascending order of importance.

The distributions genuinely differ. A question is a short interrogative fragment: "who is the bad guy in lord of the rings", ten tokens, no verb agreement with the answer, often ungrammatical, frequently containing a question word that appears nowhere in the passage. A passage is 100 words of declarative encyclopaedic prose with a title. Asking one set of weights to be excellent at both is a real constraint, and untied towers simply remove it.

The two sides are used asymmetrically at inference. The passage tower runs 21 million times, offline, once. The question tower runs once per user request, on the latency path. Untied towers let you make different engineering choices per side — distil the question tower for speed, keep the passage tower large, quantise only the passage outputs.

Untying changes what the loss can express. With shared weights, sim(q,p) is symmetric-ish: the model is pressured toward "these two texts are about the same thing." With separate towers the model can learn an asymmetric relation — "this passage answers that question" — which is not the same relation and is not symmetric. A passage about Sauron answers "who is the bad guy in LOTR"; the string "who is the bad guy in LOTR" does not answer anything about Sauron.

And the counter-evidence, because it exists: Contriever ties the towers and works extremely well, and several strong modern embedders (E5, BGE, GTE) use a single encoder with different instruction prefixes — "query: " and "passage: " — to recover the asymmetry inside one set of weights. That is a strictly better trade when you want one model that also serves symmetric tasks like clustering and similarity. DPR's untied choice was right for DPR's single task; it is not a law.

Why [CLS], and what pooling throws away

Stage 6 in both tables is the quiet violence of the architecture: a passage of ~140 token vectors, each 768-dimensional, becomes a single 768-dimensional vector. The tensor goes from (140, 768) to (768,) — a 140× compression, performed by selecting one position.

Why is that not insane? Because BERT's [CLS] token is not a token of the text at all: it is a slot that attends to everything and is attended by everything, and during pretraining it is the position from which next-sentence-prediction is read. It is, by design, the position whose job is to summarise. Fine-tuning then reshapes what "summarise" means for this task.

Alternatives exist and matter. Mean pooling — averaging all token vectors — is what Contriever, E5, and most modern embedders use, and it is generally a little better out of the box, particularly without fine-tuning, because an untuned [CLS] from vanilla BERT is a famously poor sentence representation. Max pooling preserves peak activations. And no pooling at all is ColBERT: keep every token vector, and define similarity as a sum of per-token maxima. ColBERT recovers a great deal of the lexical precision we lost in Chapter 1, at roughly 100× the index size. Chapter 9 returns to it.

What pooling costs you, concretely: after stage 6 there is no way for the model to say "this passage contains the exact string Thoros of Myr." That fact has been averaged into a direction. Chapter 1's failure case is a direct consequence of stage 6, not of the training data or the loss.

Reading the model as one matrix multiply

At serving time, once every passage is encoded, the entire retriever is:

# offline, once — the index
P = encode_passages(corpus)          # (21_015_324, 768) float32  →  64.5 GB

# online, per question
q = E_Q(question)[0]                 # (768,)   the [CLS] row
scores = P @ q                       # (21_015_324,)
top_k = np.argpartition(-scores, k)[:k]  # (k,)  →  20 passage ids

Four lines. The research is entirely in how E_Q and encode_passages got their weights, and Chapters 3 and 4 are about nothing else.

Follow one passage all the way through

Abstractions hide arithmetic errors, so take a real chunk and walk it. Passage: "Sala Baker is a New Zealand actor and stuntman, best known for portraying the villain Sauron in the Lord of the Rings trilogy directed by Peter Jackson." Its article title is "Sala Baker".

Notice the WordPiece step, because it is where Chapter 1's rare-string problem is born. "Thoros" is not in BERT's 30,522-token vocabulary, so it becomes something like th ##oro ##s — three fragments, each of which appears in thousands of unrelated words. The sparse index kept "thoros" as a single coordinate with IDF 14.3. The dense encoder has already dissolved it into common subwords before layer 1, and then averages the whole passage into one vector. Two lossy steps, both applied to exactly the signal that mattered most.

The memory arithmetic of one training step

People are surprised that DPR needed eight 32 GB V100s to run batch size 128, given that the model is only 220M parameters. The model is not the problem.

Two readings. First, activations dominate by a factor of ten, and the passage side dominates the activations because there are twice as many passages as questions once you add a hard negative each. Second — look at the last row. The object this entire paper is about, the similarity matrix holding 32,768 comparisons, is 131 kilobytes. Negatives are free in memory as well as in compute; what is expensive is the encoder passes that produced the vectors.

Which immediately explains why gradient checkpointing and mixed precision are standard in every serious implementation: they attack the 36 GB of activations, and every gigabyte recovered goes straight into a larger batch, which goes straight into more negatives.

Truncation, and the passages you silently lose

Sequences are capped at 256 tokens. A 100-word English passage tokenises to roughly 130–160 WordPiece tokens, so the cap almost never bites — which is precisely why fixed-length chunking was worth doing. Had DPR indexed whole sections, a meaningful fraction would have been truncated, and the truncated tail is invisible to the model but still present in the passage a user is shown. Silent truncation is one of the most common bugs in home-built retrieval systems: the index and the display disagree about what the document says.

Where the 28 GFLOPs actually go

The per-passage cost has been used repeatedly, so it is worth opening. One BERT-base layer, at sequence length L = 128 and hidden size d = 768, does two things:

Two observations that pay off later. The feed-forward network is 65% of the cost, not attention — at L = 128 the quadratic term is only 2.7% of a layer, which is why "attention is quadratic" is the wrong worry at passage length and the right worry at document length. And the whole thing is linear in L, so the choice of 100-word chunks is directly a choice about your encoding budget: doubling chunk length roughly doubles the 8.8 hours in Chapter 7.

Why 768 dimensions?

Because that is BERT-base's hidden size and DPR added no projection. But it is worth asking what the number costs, because it is the one architectural knob that changes your index by an order of magnitude.

Memory and scan cost are both exactly linear in d, while retrieval quality is emphatically not — it saturates. That asymmetry is the entire motivation for Matryoshka representation learning, which trains one model whose first 64, 128, 256… dimensions are each independently usable, so you can pick your point on this table after training rather than before.

The interaction spectrum

DPR sits at one end of a well-defined spectrum, and seeing all four positions at once makes the trade legible.

Read down the "what is stored" column: it is a ladder of how much you refuse to throw away before the question arrives. DPR throws away everything except one direction. That is why it is fast and why Chapter 1's rare-string failure exists, and the two facts are not separable — they are the same decision.

What pooling choice actually does, on a tiny example

Take a four-token passage whose contextual vectors are 2-dimensional, so you can hold the whole thing in your head:

[CLS] pooling gives (0.8, 0.1) — whatever that position learned to summarise. Mean pooling over the three content tokens gives ((0.2+0.3+0.7)/3, (0.9+0.8+0.2)/3) = (0.4, 0.633).

Now score against a question vector q = (0.1, 0.9), which points in the "antagonist" direction:

Mean pooling wins here, and the reason generalises: the signal lives in two of the tokens, and averaging keeps it while an untuned [CLS] — which vanilla BERT trained for next-sentence prediction, not for summarising content — may be pointing anywhere. Fine-tuning teaches [CLS] to point where it should, which is why DPR can use it and why an off-the-shelf BERT [CLS] embedding is famously terrible.

And the reverse case, which is Chapter 1's rare-string failure in miniature: suppose "sauron" is the only informative token and the other three are boilerplate near (0.7, 0.2). The mean is dominated three-to-one by boilerplate and the one informative direction is diluted to a quarter of its magnitude. Averaging is a low-pass filter, and rare, decisive tokens are exactly the high-frequency content it removes.

The title, and what it is worth

Prepending the article title is a one-line change worth about a point, which is a large return for a line. The mechanism is easy to see once you look at a real chunk.

Chunk 4 of the Sala Baker article might read: "He also performed stunts in Xena: Warrior Princess and appeared in several other productions filmed in New Zealand during the same period." Nowhere does it say who "he" is. As a standalone passage this is nearly unretrievable — there is no question it can answer, because its subject is missing.

Two general lessons. First, chunking destroys coreference, and every chunk after the first in an article is full of pronouns whose antecedents were left behind. Any cheap way to reinject context — the title, the section heading, a one-line summary — buys accuracy for almost nothing. Second, this is the same problem BM25 has, and prepending the title helps it too; it is a corpus-preparation improvement rather than a dense-retrieval one, and honest comparisons give it to both systems.

Padding waste, and the batching detail that doubles your throughput

One implementation fact that is invisible in the equations and very visible in a profiler. Transformers process a batch as a rectangle: every sequence is padded to the longest one in the batch, and the padded positions still cost full attention and feed-forward compute.

Questions are short and highly variable — 6 tokens for "who is sauron", 30 for a verbose one. Batch them naively and:

Sorting by length before batching — bucketing — costs one sort and recovers most of the difference. On the 21M-passage encoding job in Chapter 7 this is worth hours, because passage lengths vary too once you account for short final chunks of articles.

The same effect at serving time is subtler and worth knowing: if your inference server batches concurrent requests, one unusually long query inflates the compute for every request batched with it, so your p99 latency is partly a function of other people's queries. Length-bucketed batching fixes it there too.

Making the online tower cheaper

The two towers are used with wildly different frequencies — the passage tower runs 21 million times offline and the question tower runs once per user request, on the latency path. Untied weights let you optimise them separately, and the standard moves are asymmetric:

Note the constraint that survives all of this: whatever you do to the question tower, its output must still live in the same 768-dimensional space as the index. Distillation here is not "make a smaller model"; it is "make a cheaper map into an existing, frozen geometry", which is a much easier target and is why it works so well.

Chapter 3: The Objective

We have two towers and a dot product. Now: what should the number they produce be trained to do?

The naive answer is "be large for matching pairs." Follow it honestly and it collapses in one line. If the only term in the loss is −sim(q, p⁺), then gradient descent will make both vectors longer without bound — scale every embedding by 10 and every similarity multiplies by 100 — and the resulting model, whose vectors all point the same way with enormous norm, scores every passage identically high. The loss goes to −∞ and the retriever is useless. This failure has a name, representational collapse, and the cure is structural, not a regularisation trick.

Reframe the task as a multiple-choice question

Here is the move. Instead of asking the model to produce a calibrated relevance score, ask it to win a competition. Present it with one question and n+1 passages, exactly one of which is the gold. Ask: which one?

Now the training instance is a classification problem with n+1 classes, and there is a completely standard tool for that: treat the similarities as logits and apply softmax cross-entropy. Written out, with the gold passage first:

That is DPR's entire objective. It is the negative log-likelihood of the positive passage under a softmax over the candidate set, and if you have met InfoNCE or the CLIP loss, this is the same functional form with two differences we will come back to: DPR uses no temperature, and DPR's loss is computed in one direction only.

Notice immediately what the denominator did. Because the score is now normalised across candidates, growing all your norms multiplies every term in the numerator and denominator by the same factor — the loss is invariant to a shared rescaling. Collapse is no longer an optimum. The only way to lower this loss is to make the gold's similarity large relative to the others.

Derive the gradient, and read what it says

Write s₀ = sim(q, p⁺) and s_j = sim(q, p⁻_j). Let

Differentiate. For the gold's own score, using ∂log p₀/∂s₀ = 1 − p₀:

And for any negative j ≥ 1, using ∂log p₀/∂s_j = −p_j:

Compact form: ∂L/∂s_k = p_k − y_k, where y is the one-hot label. That is the familiar softmax-cross-entropy gradient, and here it says something specific and important:

Worked example: one positive, two negatives

Take a question and three candidate passages, with these raw dot products (real DPR similarities land in this range):

Exponentiate, subtracting the max for numerical stability (which changes nothing, since softmax is shift-invariant):

So the probabilities are:

And the loss is −log(0.81211) = log(1.231360) = 0.20812.

Now read the gradient. ∂L/∂s = p − y = [−0.18789, +0.18121, +0.00668]. The hard negative receives 0.18121 of push-away and the random negative receives 0.00668. Their ratio:

One negative is doing 96% of the work of teaching this model what "wrong" looks like. Hold that number; Chapter 4 scales it to 255 negatives and it becomes the argument for everything that follows.

Why not a triplet loss?

The obvious competitor, and the standard in metric learning before this, is the triplet or margin loss:

which says "the positive must beat this negative by at least a margin m, and once it does, stop caring." DPR compares the two and reports NLL is better. Three reasons, and they are all visible in the formulas.

One negative at a time. Triplet loss compares the positive to a single negative per term. To use 255 negatives you sum 255 independent hinge terms, each with equal weight regardless of difficulty. NLL normalises over all of them jointly, so the weighting is automatic — that p_j factor we just derived.

The margin is a hyperparameter you must guess. m is in the units of your unnormalised dot products, whose scale the model is free to change during training. A margin that is meaningful at initialisation is meaningless after ten epochs of norm growth.

Hinge losses go flat. Once a triplet satisfies the margin, its gradient is exactly zero — the example is discarded. In NLL a satisfied negative contributes a small but nonzero p_j, so the ranking keeps being refined. Practically, triplet training needs its own hard-negative mining pipeline just to keep the loss from becoming all zeros; the softmax gets that for free.

Two things DPR deliberately does not do

No temperature. CLIP, CLAP, SimCLR and most contrastive models divide the logits by a learned temperature τ before the softmax, because their similarities are cosines and therefore trapped in [−1, 1] — a range far too narrow to produce a peaked softmax. DPR does not normalise, so its dot products are free to occupy any range, and the model can sharpen its own distribution by growing the norms of its embeddings. The norm growth is the temperature, learned implicitly and per-example rather than as one global scalar. This is why the paper found cosine similarity worse: cosine throws away the norm, which throws away the model's only means of expressing confidence.

Make it concrete. Scale both towers' outputs by α. Then every similarity scales by α², and the softmax over [6.00, 4.50, 1.20] with α² = 2 becomes a softmax over [12.0, 9.0, 2.4]:

The ranking is unchanged — scaling all logits by a positive constant cannot reorder them — but the loss fell by 77% with no improvement in retrieval whatsoever. Gradient descent will happily take that. It is a real and slightly uncomfortable property of the unnormalised objective: some of your training loss reduction is confidence inflation rather than learning. It is also, in a system whose only job is ranking, harmless.

No symmetric term. CLIP averages a text→image loss and an image→text loss because both retrieval directions are products. DPR computes only question→passage. There is no passage→question term, for a simple reason: the columns of the batch have no well-defined gold. Passage j's "correct question" is question j, yes — but in a batch of 128, several other questions may be equally well answered by passage j, and more importantly, "given this passage, which of these 128 questions did a user ask?" is not a task anyone deploys. DPR's loss matches DPR's product. Chapter 9 walks the matrix and you will see exactly which entries are used and which are ignored.

The loss floor you should monitor

A model that cannot tell the gold from the negatives assigns them all equal probability 1/(n+1), giving loss log(n+1). This is your sanity baseline, and it is the single most useful number to print during the first epoch:

If your loss sits pinned at 5.545 and never moves, the model is guessing and something is structurally wrong — a detached gradient, a shuffled label, a similarity matrix transposed. If it drops to 0.3 in fifty steps, your negatives are trivial and you are learning nothing useful. A healthy DPR run starts near ln(256) and settles somewhere in the 0.3–1.5 range, and the interesting question is always which negatives are keeping it up there.

The gradient, derived rather than asserted

The result ∂L/∂s_k = p_k − y_k is quoted everywhere and derived rarely. It takes four lines and it is worth having.

Write Z = ∑_m e^s_m, so p_k = e^s_k/Z and L = −s₀ + log Z.

Differentiate the second term first. Since ∂Z/∂s_k = e^s_k:

Now the first term. −s₀ depends on s_k only when k = 0, contributing −1 there and 0 elsewhere — which is exactly −y_k for the one-hot label y. Adding them:

Two facts fall out immediately. The gradients over a row sum to zero (∑p_k = 1 = ∑y_k), so the loss can only ever redistribute score between candidates — it is a purely relative objective, which is why the shared-rescaling invariance from earlier exists. And a candidate's influence is its assigned probability, which is the sentence Chapter 4 is built on.

Push one step further, to the embeddings themselves. Since s_k = q · p_k:

Read it as a physical statement: the question vector is pulled toward the gold passage with force 1, and pushed away from a probability-weighted average of all candidates including the gold. In the limit where the model is already perfect (p₀ = 1) the two terms cancel exactly and nothing moves. In the limit where one negative dominates, the update is almost exactly "move away from that one passage."

Numerical stability, and the bug that eats an afternoon

DPR's similarities are unnormalised dot products of 768-dimensional vectors whose norms grow during training. Scores of 20, 40, 60 are entirely ordinary late in a run. Now:

The fix is the log-sum-exp trick, which is what every worked example in this lesson has quietly used: subtract the row maximum before exponentiating. Since

choosing c = max_m s_m makes the largest exponent exactly 0, so nothing can overflow, and the most negative exponents underflow harmlessly to 0. F.cross_entropy does this internally; a hand-rolled -log(exp(pos)/exp(all).sum()) does not, and it will produce nan after a few thousand steps in a way that looks like a data bug and is not.

InfoNCE, and the ceiling on what the loss can tell you

This objective has a second life in representation learning under the name InfoNCE, where it is derived as a lower bound on the mutual information between the two views:

The practical content of that inequality is a warning about measurement. The bound saturates at log(n+1): with 256 candidates you can never certify more than log 256 = 5.545 nats of mutual information, no matter how good your model is. So a loss of 0.05 with 8 candidates and a loss of 0.05 with 256 candidates are not comparable quantities, and neither are two training runs with different batch sizes. If you compare loss curves across batch sizes without dividing through by the chance-level floor, you will reach the wrong conclusion — a bigger batch has a higher floor and will look worse for a long time while being better.

When one positive is not enough

The loss as written assumes exactly one correct answer among the candidates. Real questions frequently have several. The standard extension keeps the same shape and masks the known extras out of the denominator:

scores = q @ p.T                       # (B, M)
scores = scores.masked_fill(other_positives, -1e4)
loss = F.cross_entropy(scores, labels)   # the designated gold only

Where other_positives is a boolean (B, M) tensor marking every cell you know to be relevant except the designated one. Setting those cells to a large negative number removes them from the softmax entirely, so they neither receive gradient nor inflate the denominator. Chapter 6 explains why you will want this more often than you expect: the moment two questions in your batch are about the same entity, you have manufactured this situation yourself.

The road not taken: normalise, then add a temperature

DPR uses raw dot products. Almost every embedding model trained since 2022 does the opposite: L2-normalise both sides and divide by a small temperature. Since you will meet both, it is worth seeing exactly what changes.

With normalisation, sim(q,p) = cos(q,p) ∈ [−1, 1]. Now compute a softmax over three candidates at cosines 0.75, 0.65, 0.15 — a realistic spread for a trained model — with no temperature:

That is nearly uniform. The correct answer holds 40.8% and the loss sits close to the chance floor of log 3 = 1.0986 even though the model has ranked everything correctly. There is almost no gradient pressure and nothing distinguishes a good model from a mediocre one. Cosine similarities are too narrow to produce a usable softmax on their own — and this, not any subtlety about angles, is the concrete reason temperature exists.

Now divide by τ = 0.05, equivalently multiply by 20:

The same model, the same ranking, and a loss seven times smaller with a sharply concentrated gradient on the one confusable negative. Nothing about the representation changed; you changed how sharply the loss reads it.

Why a classification loss trains a ranking system

A fair objection: you deploy a ranker, but you train a classifier. Why should minimising cross-entropy improve top-20 accuracy?

Because minimising −log p₀ means maximising p₀, and p₀ exceeds one half exactly when s₀ exceeds the log-sum-exp of every other candidate — a smooth surrogate for the maximum. So the loss is a differentiable relaxation of "the gold's score is the largest", which is precisely top-1 accuracy within the candidate set. Cross-entropy is a soft-max in the literal sense: a smooth stand-in for the hard max that ranking metrics ask about.

What it does not optimise is position beyond the top of the list. Once the gold wins its row, additional loss reduction goes into widening a margin nobody measures. This is why the correlation between training loss and retrieval accuracy is strong early and weak late, and why the honest stopping criterion is a held-out retrieval metric rather than the loss curve.

One gradient step, computed all the way

The formulas are done. Take one actual step so the geometry stops being metaphorical. Work in two dimensions with one positive and one negative:

Similarities: s⁺ = 2.0×2.5 + 1.0×1.0 = 6.0; s⁻ = 2.0×1.0 + 1.0×2.0 = 4.0.

Softmax over (6.0, 4.0): subtract 6, giving e⁰ = 1 and e⁻² = 0.135335, sum 1.135335. So p = (0.880797, 0.119203) and L = log(1.135335) = 0.126928.

Now the gradient with respect to q, using the formula derived above:

Look at what that simplifies to. Factoring out the shared 0.119203:

The update direction is exactly the vector from the negative to the positive, scaled by the negative's own probability. With one positive and one negative, gradient descent moves the question along the line connecting them, and the step size is the negative's softmax mass. If the negative were hopeless (probability 0.001), the step would be a thousandth as long. That is Chapter 4's entire thesis, visible in two dimensions.

Take the step with learning rate 1:

Recompute. s⁺ = 2.178805×2.5 + 0.880797×1.0 = 5.447013 + 0.880797 = 6.327810. s⁻ = 2.178805×1.0 + 0.880797×2.0 = 2.178805 + 1.761594 = 3.940399. The gap widened from 2.000 to 2.387.

New loss: e^−2.387411 = 0.091868, sum 1.091868, L = log(1.091868) = 0.087890. Down from 0.126928, a 31% reduction, from one step on one training pair. Do this 58,880 times per epoch for 40 epochs and you have DPR.

What to stop on

Three chapters have now given reasons why the training loss is a poor progress signal: it is invariant to shared rescaling, it is exponentially sensitive to a margin whose units the model controls, and its floor depends on batch size. Collect the consequence.

The catch is that the honest signal requires re-encoding your evaluation corpus at every checkpoint, which is expensive. The standard compromise: evaluate against a fixed 100k-passage subsample of the index containing every gold passage plus a random remainder. It is not the real number, it moves with the real number, and it costs seconds instead of hours.

Should the loss run in both directions?

CLIP averages an image→text loss and a text→image loss. DPR uses question→passage only. The difference is not an oversight, and thinking it through sharpens what the objective is for.

The symmetric version would add a term over the columns of the score matrix: for each passage, a softmax over all questions in the batch, with the label being its own question.

The column term is right for CLIP because both directions are products — people search images with text and text with images — and because an image–caption pair is genuinely symmetric. A question and its answer passage are not: the passage answers the question, and the question does not answer the passage.

What a healthy loss curve looks like

The shape that matters is the long, slow second half. A run that reaches 0.3 in one epoch has not trained faster; it has been given an easier exam, and the ablation table's 47.0 is what that looks like when measured properly.

Chapter 4: The Craft of Negatives

This is the chapter. Everything before it was scaffolding; everything after it is consequence.

Recall the loss from Chapter 3. The numerator is fixed by the data — the gold passage is whatever the annotation says it is. The denominator is chosen by you. Every question in your training set comes with a pile of passages you have declared to be wrong, and the model's entire concept of "wrong" is the empirical distribution of that pile. Change the pile and you change what the model learns, more than you change it by swapping BERT for RoBERTa or 768 dimensions for 1024.

Type 1: random negatives — and why they stop teaching

The simplest choice: sample n passages uniformly from the 21 million. Cheap, unbiased, trivially parallel. Let us compute exactly how informative they are.

Suppose a given question has roughly 200 passages in Wikipedia that are meaningfully on-topic — other paragraphs about Lord of the Rings characters, say. The probability that one uniform draw lands on-topic is:

Draw seven negatives, as one of DPR's ablation settings does. The expected number of on-topic negatives per question is 7 × 9.52×10⁻⁶ = 6.7×10⁻⁵. Over an entire epoch of 58,880 NQ questions, the expected count of on-topic random negatives across the whole dataset is 58,880 × 6.7×10⁻⁵ ≈ 3.9. Four. In an entire epoch.

So a model trained on random negatives has, over its whole training life, essentially never been asked to distinguish two passages about the same subject. It learns the coarse topical geometry of Wikipedia extremely well, and stops. Now recall Chapter 3's gradient result: a negative's influence is its softmax probability p_j. Once the model has learned topics, every random negative sits far below the positive, its p_j is near zero, and the training signal vanishes — not because the model is good, but because the exam got too easy.

DPR's number for this configuration: top-20 accuracy of 47.0 on NQ. BM25 gets 59.1. Random negatives produce a dense retriever meaningfully worse than the twenty-year-old baseline.

Type 2: BM25 hard negatives

The fix is almost embarrassingly direct. Run BM25 on the question. Take the top-ranked passages. Throw away any that contain the answer string. What is left is a set of passages that are lexically excellent matches and factually wrong — exactly the confusions a deployed retriever will face.

For "who is the bad guy in lord of the rings", BM25's top passages are all about the franchise: the film's box office, the Two Towers plot summary, Tolkien's biography, a list of Middle-earth locations. Every one shares heavy vocabulary with the question. None contains "Sauron" in an answering role. These are the passages the model must learn to rank below the Sala Baker passage, and no amount of topical geometry will do it.

DPR's number: adding BM25 negatives moves top-20 from 47.0 to 50.0 in the seven-negatives-no-in-batch setting. A real gain, and a modest one — because seven negatives is seven negatives. The explosion comes when you combine them with the next idea.

Type 3: in-batch negatives — the free lunch, priced exactly

Here is the observation, and it is one of those ideas that is obvious the instant you see the matrix.

You are training on a batch of B questions. To compute the loss you must encode B questions and their B gold passages. Now look at what you are holding: a (B, 768) matrix of question vectors and a (B, 768) matrix of passage vectors. One matrix multiply gives you every pairwise similarity:

The diagonal S_ii holds the B positives. The off-diagonal holds B² − B entries, every one of which is a question paired with a passage that is not its gold. Those are negatives. You did not fetch them, encode them, or store them. They were already in your GPU.

Price the lunch. With B = 128 and one gold each:

Check the middle row by hand. 128 × 128 = 16,384 passage encodings; at 2.816×10¹⁰ FLOPs each that is 16,384 × 2.816×10¹⁰ = 4.613×10¹⁴. On an accelerator sustaining 150 TFLOP/s that is 3.08 seconds of forward compute per training step, before the backward pass. The in-batch version needs 128 × 2.816×10¹⁰ = 3.60×10¹² FLOPs, which is 24 milliseconds. Same 16,384 similarity scores either way.

And the similarity matrix itself is nothing: 128 × 128 × 1,536 FLOPs = 2.5×10⁷, seven orders of magnitude below the encoding cost. Put another way, the marginal cost of one additional in-batch negative is 1,536 FLOPs versus 2.8×10¹⁰ for a freshly encoded one — a factor of 18.3 million.

The combination that produced the headline

Now put the two together, and read the tensor shapes carefully because the arithmetic surprises people.

Take a batch of B = 128 questions. For each, include its gold passage and one BM25 hard negative. The passage side of the batch is now 256 rows, not 128:

Each row of S has 256 entries: one positive and 255 negatives. Break down where those 255 come from:

Read the last column and something uncomfortable appears: you added 128 hard negatives to the batch, but each question only receives one of them as a hard negative. The other 127 hard negatives are hard for someone else and easy for you. The hardness does not pool. And yet:

(These are the paper's ablation runs, which use a shorter training schedule than the headline configuration — that is why the absolute numbers sit below 78.4. The relative ordering is the point.)

Now do the accounting that turns this table into a design principle. Going from 7 in-batch negatives to 127 — adding 120 negatives — buys 55.8 − 51.1 = 4.7 points. Adding one BM25 hard negative to the 31-negative setting buys 65.0 − 52.1 = 12.9 points. Per negative:

Explore the matrix

The dose-response curve, and where it turns

Look again at two adjacent rows of the ablation table:

Doubling the hard negatives makes it worse. If hard negatives are worth 330× a random one, why does the second one hurt?

Because "the top BM25 passage that does not contain the answer string" and "the second-ranked BM25 passage that does not contain the answer string" are not drawn from the same distribution as far as correctness goes. As you walk down the BM25 list, you are walking into passages that are increasingly likely to be relevant but unlabelled. Which brings us to the failure mode that has cost the field more accuracy than any other.

Type 4: false negatives — the poison in the well

A false negative is a passage you placed in the denominator that actually answers the question. The label says "wrong". The world says "right". And the loss believes the label.

Watch what it does to the gradient. Return to the worked example from Chapter 3: scores [gold 6.00, hard negative 4.50, random 1.20], loss 0.20812. Now add a fourth candidate — a passage that also correctly answers the question, which the model (correctly!) scores high at 5.80, but which we have labelled a negative:

Three separate harms, all visible in those numbers.

The loss inflated 3.4× for a model that did nothing wrong. Your training curve now reports failure where there was success, and if you are early-stopping or tuning on it, you are tuning on noise.

39.9% of the gradient now pushes a correct passage away. That is not a missing update; it is an update pointing in the wrong direction, with more force than the genuine hard negative (10.9%) receives. The model is being actively taught that a correct answer is wrong.

The damage generalises. Embeddings are shared across examples. Pushing this question away from this passage moves a whole region of the space, degrading every other question that lives nearby.

Now the crucial asymmetry, and the reason this chapter is called a craft rather than a procedure. How likely is a false negative? It depends entirely on where you drew from.

Check the first row's arithmetic: 50 relevant passages out of 21,015,324 is a rate of 2.38×10⁻⁶; across 255 in-batch negatives the expected count is 255 × 2.38×10⁻⁶ = 6.1×10⁻⁴. Six in ten thousand. Utterly safe.

Why the answer-string filter is not enough

DPR's filter for BM25 negatives is: exclude any top passage that contains the answer string. This is a real defence and it is leaky in both directions.

It misses paraphrases. Question: "when did the United States enter World War II?" Answer string: "1941". A passage saying "the attack on Pearl Harbor brought America into the war the following December" answers the question, contains no "1941", survives the filter, and becomes a hard negative. The model is taught that the single best paraphrastic answer is wrong — which is exactly the capability you were building a dense retriever to obtain.

It over-rejects on common strings. Answer "France" excludes every passage mentioning France for any reason, including genuinely good hard negatives. Answers that are years, small integers, or common nouns are the worst offenders. Question "how many players are on a basketball team?" with answer "5" excludes every passage containing the character 5.

Both leaks are direct consequences of using string matching to approximate semantic relevance, which is the exact thing Chapter 0 told you does not work. The pipeline that trains a model to escape lexical matching is itself built on lexical matching. Chapter 5's RocketQA is the paper that noticed and paid to fix it.

A practitioner's decision table

Why the denominator prevents collapse — with numbers

Chapter 3 asserted that a positives-only objective collapses. Watch it happen.

Suppose every question and every passage maps to the same unit direction u scaled by α. Then sim(q,p) = α² for every pair. With a positives-only objective L = −sim(q, p⁺) = −α², and gradient descent drives α → ∞. Loss → −∞; retrieval accuracy is exactly chance, because every passage ties.

Now put the denominator back. Every candidate scores α², so:

The α has cancelled completely. Collapse now sits at the chance-level loss — the worst value the model can achieve rather than the best — and every gradient step moves away from it. That cancellation is the entire structural reason contrastive objectives work, and it is why the negatives are load-bearing rather than decorative.

Counting the negatives that actually count

"How many negatives?" is the wrong question, and there is a better one you can compute. Given the softmax probabilities of the negatives, define the participation ratio:

which equals n when all negatives are equally weighted and approaches 1 when a single one dominates. It answers: how many negatives is this loss actually distinguishing between?

Case A — two negatives, one hard (the Chapter 3 example, p = 0.18121 and 0.00668):

Case B — 255 easy negatives at 0.00152 each:

Read that pair carefully, because it is more interesting than it first looks. The easy batch has 255 effective negatives and the hard batch has 1.07 — and the hard batch trains a better model. The participation ratio measures diversity, not usefulness. What 255 easy negatives deliver is a diffuse push away from all of Wikipedia at once, which after a few thousand steps is information the model already has. What one hard negative delivers is a specific, sharp instruction about a distinction the model is currently getting wrong.

The negative pipeline, in code

Everything in this chapter is about twenty lines of data preparation. Here it is, with the decisions marked.

def build_batch(questions, bm25_index, k_hard=1):
    qs, ps = [], []
    seen = set()                                # DECISION 3: dedup by passage id
    for q in questions:
        qs.append(q.text)
        if q.gold_id in seen: continue          # same passage twice = an
        seen.add(q.gold_id)                       # unsatisfiable constraint
        ps.append(q.gold_passage)                   # the positive

        cands = bm25_index.search(q.text, top_k=100)
        hard  = [c for c in cands
                 if not contains_answer(c, q.answers)  # DECISION 1: leaky filter
                 and c.id != q.gold_id]
        ps.extend(hard[:k_hard])                    # DECISION 2: k_hard = 1, not 4
    return qs, ps                                 # (B,) and (<= B*(1+k_hard),)

Decision 1 is Chapter 4's leaky answer-string filter — it misses paraphrases and over-rejects common strings. Decision 2 is the dose-response result: one, not four. Decision 3 is the quiet one that bites people. If two questions in a batch share a gold passage, that passage appears in the batch once as a positive and once as a negative, and the loss is being asked to make one cell simultaneously the largest and the smallest in its column. It is an unsatisfiable constraint, it produces a persistent noise floor in the loss, and on datasets like Natural Questions — where popular entities generate many questions — it happens often enough to matter.

What each negative type actually teaches

Read the middle column top to bottom: each row is a strictly harder question, and each row is closer to the question the model will actually be asked at serving time. That is the whole progression, and the rest of this lesson is the engineering required to climb it without poisoning yourself on the way up.

A negative curriculum

One question the ablation table cannot answer: should the negatives be the same throughout training? Modern pipelines say no, and the reasoning follows directly from the gradient identity.

At initialisation the model is random. Every negative is equally confusing, so hard negatives are wasted — the model cannot yet tell a hard one from a random one, and the loss is near log M regardless. By mid-training the topical structure is learned, and now random negatives contribute nothing while hard ones carry everything. Staging follows:

The transition signal in each arrow is a measurement, not a step count. That matters: the right moment to escalate is when the current negatives have stopped producing gradient, which the max-off-diagonal-probability diagnostic tells you directly.

Does batch size have a scaling law?

Fit the in-batch rows of the ablation table and see. Negatives 7, 31, 127 give 51.1, 52.1, 55.8:

The per-doubling gain is small and, over this range, not even monotone — three points is far too few to claim a law. What the numbers do support is a bound: extrapolating the best-case rate of ~1.8 points per doubling, you would need to go from 127 to about 4,000 negatives to buy the ten points that one BM25 hard negative delivered. That is five doublings of batch size, which is a cluster, against one BM25 lookup per question, which is an afternoon.

Instrumenting the negatives

Four things you can log from the score matrix, at no cost, that tell you what your negatives are doing:

The strangest row in the table

Go back to the ablation and look at a comparison that should stop you:

Other questions' gold passages, used as explicit negatives, are worse than random Wikipedia passages — and the very same passages, used as in-batch negatives, are better than both. Same texts. Same count. Opposite verdicts.

The most consistent reading has two parts, and both are worth internalising because they generalise well beyond this paper.

Gold passages are a biased sample of the corpus. The ~59,000 designated positives in Natural Questions are not a uniform draw from 21 million passages. They are the passages that happen to answer the kind of question people type into Google: densely factual, entity-heavy, well-formed, drawn from popular articles. Train the model to push away only those, and it learns to discriminate against a narrow distribution it will never face at inference. Random negatives, whatever their weakness, at least sample the distribution the model must actually rank against. Your negatives are a claim about what the model will see at serving time, and a biased claim is worse than a weak one.

In-batch negatives are not just cheaper — they are structurally different. When passage j is question j's positive and question i's negative in the same step, one embedding receives both forces at once. It must be close to one question and far from another, simultaneously, from a single vector. That is a joint constraint on the geometry, and it is impossible to express when the negatives come from a separate pool that is never anyone's positive. The reuse is not an optimisation of the same objective; it changes what the objective can say.

Every failure mode on one page

The gradient-accumulation trap

One gotcha that catches nearly everyone the first time, and it follows directly from what in-batch negatives are.

You want batch 128 and your GPU fits 32. The standard trick is gradient accumulation: run four micro-batches of 32, accumulate the gradients, then step. For ordinary supervised training this is mathematically identical to a batch of 128.

For contrastive training it is not. The loss is computed within each micro-batch, so each question sees only the 31 other passages in its own micro-batch. Accumulating gradients across four such losses does not create a 128-way softmax; it creates four 32-way softmaxes.

Nearly four points of accuracy, silently lost, with a loss curve that looks better because its floor is lower. The tell is exactly that: if accumulation "improved" your loss, you shrank your negative pool.

The real fixes are the ones from Chapter 5. Cross-GPU all-gather increases the pool without increasing per-device memory. A MoCo-style queue holds old embeddings at 3 KB each instead of full activations. Or gradient checkpointing, which trades compute for the activation memory that was the binding constraint in the first place. Accumulation solves a different problem, and reaching for it here is a category error.

Chapter 5: ANCE, RocketQA, Contriever

DPR left the field with a clear, uncomfortable insight and an obvious open problem. The insight: negatives are the lever. The problem: DPR's best negatives came from BM25, a model with no idea what the dense retriever currently finds confusing, computed once before training started and never updated.

Three papers over the following eighteen months attacked that from three different angles. Together they are the reason modern retrievers exist, and each one is a different answer to Chapter 4's tension.

ANCE: mine negatives with the model you are training

ANCE (Xiong et al., 2020 — Approximate nearest neighbour Negative Contrastive Estimation) begins with a theoretical argument that sharpens Chapter 3's gradient identity into a statement about variance.

Here is the argument in the form you can verify. Consider a training step where the positive scores 6.0 and you have 255 negatives, all easy, all scoring around 0.0. The denominator:

Total negative gradient mass is 1 − 0.6127 = 0.3873, spread across 255 negatives, so each carries an average of 0.00152. Compare the hard-negative case from Chapter 3, where a single negative carried 0.18121. One hard negative delivers 119× the gradient of one easy negative, and the entire easy batch of 255 delivers about twice what that single hard negative does — while costing 255 slots.

ANCE's framing: the loss is an expectation over a negative distribution, and sampling from a distribution that concentrates on uninformative points gives an estimator with terrible variance relative to its cost. The fix is to sample negatives from the distribution that actually matters at inference — the top of the model's own ranking over the whole corpus.

Which sounds impossible. To retrieve the model's current top-k for a question you must have every passage encoded by the current model, and the model changes every step. Encoding 21 million passages takes hours. You cannot do it per step, or per hundred steps.

Price the refresh. Re-encoding an 8.8M-passage corpus at 28 GFLOPs per passage:

On four accelerators sustaining 150 TFLOP/s each — 600 TFLOP/s aggregate — that is 2.48×10¹⁷ / 6.0×10¹⁴ = 413 seconds of pure compute, call it 10–15 minutes with data loading and the ANN build. Meanwhile the Trainer is doing steps. If a step takes 0.4 s, the index is roughly 2,000 steps stale by the time it is replaced.

That staleness has a real cost, and it is worth naming precisely: a passage that was the model's top-ranked confusion 2,000 steps ago may now be correctly ranked low. The negative that was hard is now easy, and its gradient contribution has decayed back toward the useless regime. The refresh interval is therefore a genuine tradeoff — refresh too often and the Inferencer becomes your bottleneck and your dominant compute cost; refresh too rarely and you are back to static negatives with extra machinery.

ANCE's reported result on MS MARCO passage ranking is MRR@10 = 0.330 on the dev set, against BM25's 0.187 and a DPR-style statically-mined baseline in the low 0.30s. The paper also shows the shape that matters more than the number: models trained on static negatives plateau, while ANN-mined negatives keep improving, because the negative distribution keeps tracking the model.

RocketQA: more negatives, and cleaner ones

RocketQA (Qu et al., 2020) attacks the same problem from the data-quality side, with three changes that each address a specific defect in Chapter 4.

1. Cross-batch negatives. In-batch negatives are limited by what fits in one GPU's memory. But data-parallel training already has A GPUs each holding B passages. Add an all-gather: every GPU broadcasts its passage embeddings to every other. Now each question sees A × B − 1 negatives instead of B − 1.

The cost is one all-gather of (B, 768) float tensors per step — at B = 512 that is 512 × 768 × 4 bytes = 1.57 MB per GPU, trivial on NVLink or InfiniBand. The embeddings are already computed; you are only moving them. This is the in-batch free lunch extended across the whole cluster, and it is now standard in every large embedding training run.

2. Denoised hard negatives. This is the one that resolves Chapter 4's tension, and it does so by spending money. Train a cross-encoder — the expensive question-and-passage-together model from Chapter 2 that you can never deploy for retrieval. Use it to score the top-ranked candidates mined by the bi-encoder. Keep as hard negatives only those the cross-encoder confidently rates as not relevant; discard the ambiguous ones entirely.

The audit that motivates this is the striking part. RocketQA's authors manually examined top-retrieved passages that were not labelled positive, and report that roughly 70% of them actually contained the answer. Seventy percent of your hardest negatives were positives. Chapter 4 predicted the direction of that number; the size of it is what makes denoising mandatory rather than optional.

3. Data augmentation. Run the cross-encoder over unlabelled questions and high-confidence retrieved passages, and promote the confident ones to training positives. This is knowledge distillation wearing a data-engineering hat: an accurate, slow model manufactures supervision for a fast, deployable one.

Results: MS MARCO dev MRR@10 0.370, and NQ top-20 of 82.7 against DPR's 78.4.

Contriever: invent the positives too

DPR, ANCE, and RocketQA all assume labelled question–passage pairs exist. Contriever (Izacard et al., 2021) asks the harder question: what if you have none at all?

Its answer is independent cropping. Take a single document. Sample two random contiguous spans from it, independently. Declare them a positive pair.

The assumption is that two spans from the same document are about the same thing, which is weak, frequently wrong, and — averaged over hundreds of millions of documents — sufficient. The spans are sampled independently so they may overlap, may not, and the model cannot solve the task by matching a shared boundary.

Note what has happened to Chapter 4's tension. There are no labels, therefore no false negatives in the DPR sense — only pairs that are noisy in both directions. The craft has moved from "which negatives?" to "what counts as a positive?", and the answer comes from document structure rather than from annotators.

Contriever's other choices are all in service of scale:

The MoCo queue deserves a sentence, because it is the third distinct answer to "where do negatives come from". Keep a FIFO buffer of the last 131,072 passage embeddings produced by a slowly-updated momentum copy of the encoder. Every batch enqueues its passages and dequeues the oldest. Negatives are drawn from the queue. The cost is 131,072 × 768 × 4 bytes = 402 MB of memory and no extra forward passes at all. The staleness problem returns — queued embeddings are old — and the momentum encoder exists precisely to keep them from drifting too fast to be usable.

The headline result is a zero-shot one: on BEIR, a benchmark of fifteen retrieval datasets from domains nobody trained on, Contriever beats BM25 on recall@100 for the majority of them — the first time an unsupervised dense retriever had done so. That directly attacks Chapter 1's third BM25 advantage, out-of-domain robustness.

The three answers, side by side

Read the last row down the columns. Nobody removed the tension; each paper moved it somewhere it could be afforded. That is what "craft" means here — a set of trades whose right resolution depends on the compute, the labels, and the corpus you actually have.

The all-gather, priced

RocketQA's cross-batch negatives are the cheapest of its three ideas and the one you should copy first, so it is worth seeing the communication cost written down.

Data-parallel training already replicates the model across A GPUs, each processing its own B passages. After the passage tower runs, GPU a holds a (B, 768) float tensor. An all-gather makes every GPU hold the full (A·B, 768):

A tenth of a millisecond, against a training step that takes hundreds. You have octupled the negative pool for free, because — exactly as in the in-batch case — the embeddings already existed and you are only moving them.

The one real subtlety: the gathered embeddings must stay in the autograd graph, or the gradient flows only through your local shard and you have silently turned off seven eighths of the signal. Frameworks provide a differentiable all-gather for precisely this reason, and forgetting it is one of the classic silent bugs in distributed contrastive training — the loss looks fine, the model is simply worse than it should be.

ANCE's argument, stated as variance

It is worth having the theoretical claim in a form you can check, because "hard negatives are better" is folklore and ANCE turned it into an estimator argument.

The exact objective sums over the whole corpus. You cannot afford that, so you replace it with a sample — which makes the training gradient a random variable whose mean should be the true gradient and whose variance determines how many steps you need. Now recall that a negative's contribution is weighted by p_j, which is near zero for almost every passage in the corpus. So the true gradient is dominated by a tiny, high-probability subset, and uniform sampling almost never draws from it.

Concretely: if 100 passages out of 21 million carry 90% of the negative gradient mass, then a uniform sample of 255 hits one of them with probability

Roughly one batch in 800 contains any of the passages that matter. The estimator is unbiased and its variance is enormous, which shows up in practice as a loss that stops decreasing while retrieval accuracy stops improving. Sampling from the model's own top-k inverts that probability to nearly 1, at the price of the entire Inferencer apparatus.

Contriever's cropping, and the momentum encoder

Two details of Contriever repay a closer look, because both reappear in every modern unsupervised embedder.

Independent cropping. Sample the two span lengths independently, and their start positions independently, from the same document. They may overlap, may be adjacent, may be far apart. Independence matters: if you always took adjacent halves, the model could exploit sentence-boundary continuity and learn a much easier task than "same topic".

The MoCo momentum encoder. A queue of 131,072 old embeddings is only useful if those embeddings are still comparable to what the current model produces. If the encoder changes quickly, a queue entry from 5,000 steps ago lives in a different geometry and is noise. The fix is to generate queue entries with a slowly-moving copy of the encoder:

With m = 0.999 the momentum weights are an exponential moving average with a time constant of about 1,000 steps, so the geometry drifts slowly enough that a queue spanning thousands of steps remains coherent. It is the same staleness problem ANCE solved with a refresh schedule, solved instead with a low-pass filter — and the queue costs 131,072 × 768 × 4 = 402 MB and zero extra forward passes.

Choosing among the four

The ordering matters: denoising before ANN mining, not after. ANCE mines from exactly the region where false negatives live, so adding it to an undenoised pipeline amplifies the poison faster than it adds signal. Chapter 4's tension is not a metaphor — it is an ordering constraint on your roadmap.

What the recipe became

By 2023 the three ideas in this chapter had fused into a single standard pipeline, and every strong open-weight embedder is a variation on it. Seeing the whole shape at once tells you where DPR's parts ended up.

Stage 3 deserves a note, because it is the cleanest resolution of Chapter 4's tension anyone has found. Rather than asking the cross-encoder a binary question ("is this negative honest?") and discarding the ambiguous cases, you ask it a graded one and fit the whole distribution — typically with a margin-MSE loss that matches the student's score difference to the teacher's:

A false negative is no longer a catastrophe. If the teacher also thinks the "negative" is relevant, the target margin is near zero and the student is simply told these two passages are comparable — which is true. The labelling error has been converted from a wrong instruction into a soft, approximately correct one.

The two things that were genuinely new after 2021

Generated queries. If labelled pairs are the bottleneck, generate them: hand a passage to a language model and ask for a question it answers. This inverts the mining direction — instead of searching for the passage that matches a query, you manufacture the query that matches a passage, which is trivially correct by construction and free of the BM25 circularity from Chapter 6. The failure mode moves too: generated queries are fluent, well-formed and lexically close to their passage, which is exactly SQuAD's problem in a new costume, so the generator has to be prompted toward realistic, terse, underspecified user language.

Instructions. Prefixing "query: " and "passage: " lets one shared encoder serve the asymmetric relation DPR needed two towers for, and richer instructions ("Retrieve a scientific paper that supports this claim") let one model serve many retrieval relations at once. The asymmetry moved out of the parameters and into the input.

What each successor cost to run

Papers report accuracy and rarely report price. The price is what decides which of these you can actually adopt, so here it is side by side, in units of "how much extra compute per unit of training compute".

Read the "engineering surface" column against the accuracy gains. Cross-batch negatives are nearly free in every column and were worth real points. ANCE and denoising each roughly double the complexity of your training system. That ordering — free things first, expensive things only when the free ones are exhausted — is the same ordering the decision flow gave, arrived at from cost rather than from accuracy, which is a good sign it is right.

The staleness question, answered properly

ANCE's refresh interval is the one hyperparameter here that people get badly wrong, so it is worth stating the principle rather than a number.

Refresh when the negatives have stopped being hard, and not before. Concretely: track the mean softmax probability assigned to your mined negatives. Immediately after a refresh it will be high — the negatives were selected precisely because the model ranks them near the top. As training proceeds the model learns to reject them and the probability decays. When it approaches the level of an ordinary in-batch negative, the mining has expired and a refresh will pay for itself. When it has not, a refresh buys almost nothing and costs a full corpus encode.

This turns a guessed schedule into a measured trigger, and it automatically adapts: early in training the model changes fast and refreshes should be frequent; late in training it barely moves and refreshes can be rare. A fixed interval gets both ends wrong.

The other half of the system: reranking

RocketQA trains a cross-encoder to clean its training data, and the same model has a second life at serving time. It is worth pricing, because "add a reranker" is usually a better first move than "improve the retriever" once your retriever is decent.

The division of labour is clean and follows directly from Chapter 2. The bi-encoder's weakness is that it compressed each passage before seeing the question; the cross-encoder's weakness is that it cannot look at 21 million passages. Put the cheap one first to get recall, and the expensive one second to get precision, and each covers the other's failure exactly.

And note where the ceiling sits: reranking can only reorder what retrieval returned. If R@100 is 85.4%, no reranker on earth pushes end-to-end recall past 85.4%. This is Chapter 0's multiplicative bound again, one level down — which is why the two-stage split does not eliminate the need for a good retriever, it just changes what "good" means from "ranks perfectly" to "recalls at 100".

Measuring your own false-negative rate

Chapter 4's drill showed that the optimal number of hard negatives depends entirely on f(r), your false-negative curve. Here is how to measure it rather than guess.

Fifty questions gives a noisy curve and it is enough to distinguish "f(1) is 10%" from "f(1) is 60%", which is the decision you are actually making. RocketQA's roughly-70% figure is what this procedure returns on Natural Questions; there is no reason to assume your corpus behaves the same way, in either direction.

Reading these four papers, in order

If you go to the sources, this order minimises confusion, and each one has a single thing worth taking away.

Read in that order they tell one continuous story: negatives are the lever (DPR) → so are positives, and both can be synthesised (Contriever) → the right negatives are the model's own top-ranked errors (ANCE) → and those are exactly the ones most likely to be mislabelled, so pay a second model to check (RocketQA). Each paper is an answer to the problem the previous one created.

Chapter 6: Positives Are Subtle Too

Four chapters on negatives can leave the impression that the numerator of the loss is settled. It is not. Where the positive comes from determines what the model believes "relevant" means, and DPR's five datasets get their positives in three quite different ways — which turns out to explain the single anomalous row in the results table.

Three sources of positives

Gold, from an annotator. In Natural Questions, a human was shown a real Google query and the Wikipedia page that Google returned, and asked to highlight a long answer — the paragraph containing the answer — and a short answer span within it. That paragraph, chunked, is a genuine positive: a person judged that this text answers this question.

Distant supervision. TriviaQA, WebQuestions and CuratedTREC ship question–answer pairs with no passage attached. DPR manufactures positives: run BM25 for the question, take the top 100 passages, and select the highest-ranked one that contains the answer string. If no passage in the top 100 contains the answer, the question is discarded.

Question written from the passage. SQuAD inverts the whole process: the annotator was given a paragraph and asked to write a question it answers. The passage is a positive by construction — and the question is a function of the passage, which is a very different statistical object.

Read that list again with Chapter 4's eyes. The second one — distant supervision — selects positives using BM25, and the third one produces questions that share vocabulary with the passage by construction. Both of them push lexical overlap into the training signal of a model whose entire purpose is to stop depending on lexical overlap.

Failure mode 1: the spurious positive

Answer-string matching says a passage is relevant if it contains the answer text. That is a necessary condition, not a sufficient one, and the gap is large.

Take: "When did the Beatles release Abbey Road?" Answer: "1969". Now run BM25 over Wikipedia and scan the top 100 for a passage containing "1969". Candidates that qualify:

If the second row ranks higher under BM25 than the first — entirely possible, since it contains both "Abbey Road" and "1969" and mentions the studio repeatedly — it becomes the designated positive. The model is now trained to place this question next to a passage about recording equipment.

Short, common answers make this far worse. Answers that are years, single digits, country names, or common nouns collide constantly. "How many players are on a basketball team?" with answer "5" will string-match essentially any passage containing the digit.

The harm is the mirror image of a false negative, and it is arguably worse. A false negative pushes the model away from something correct. A spurious positive pulls the model toward something incorrect — it corrupts the numerator, which receives the full ∂L/∂s₀ = p₀ − 1 gradient, the single largest term in the update.

Failure mode 2: filtering removes the hard questions

DPR discards any question whose answer does not appear in BM25's top 100. On Natural Questions the training set goes from 79,168 questions to 58,880:

Which 25.6%? Precisely the questions where BM25 failed. Which is precisely the population dense retrieval was invented to serve. The training set has been filtered by the baseline you are trying to beat, so the hardest quarter of the distribution — the "bad guy"/"villain" cases — is gone before the first gradient step.

Failure mode 3: the question is a function of the passage

SQuAD deserves its own section because it produces the only row in DPR's table where the paper loses, and the explanation is entirely about positives.

A SQuAD annotator reads: "The Amazon rainforest, also known in English as Amazonia, covers most of the Amazon basin of South America. This basin encompasses 7,000,000 square kilometres, of which 5,500,000 square kilometres are covered by the rainforest." They write: "How many square kilometres of the Amazon basin are covered by rainforest?"

Count the overlap. Square kilometres, Amazon, basin, covered, rainforest — five content terms, all present, several of them rare. A question written this way is close to a copy of the passage with one span replaced by a wh-word. It is the ideal input for a term-matching retriever.

Compare a Natural Questions query, which is a real thing a real person typed into Google before knowing any answer existed: "who is the bad guy in lord of the rings". No passage was in front of them. The vocabulary is theirs, not the encyclopaedia's.

The last row compounds the first. SQuAD's paragraphs come from a few hundred articles, so training questions cluster in a tiny slice of the corpus while evaluation retrieves over all 21 million. The dense model learns a geometry fitted to that slice, and the paper says so explicitly: the distribution is biased toward a very small set of Wikipedia articles, which is out of distribution for the full-corpus training setup.

Which positive, when there are many?

One more subtlety, and it links straight back to Chapter 4. Most questions have several correct passages. Wikipedia states the Abbey Road release date in the album article, the discography article, and the 1969-in-music article. DPR designates one of them the positive.

The other correct passages are now, from the loss's point of view, unlabelled. If one of them lands in the batch — as another question's gold, or as a mined hard negative — it enters the denominator as a negative. Multi-positive questions manufacture false negatives automatically. The two problems are one problem viewed from opposite ends, and the modern fix is the same in both directions: a multi-positive loss that masks out all known-relevant passages from the denominator rather than treating exactly one as correct.

Concretely, in the (B, M) score matrix from Chapter 4, instead of F.cross_entropy(scores, labels) you build a boolean mask of every (question, passage) pair known to be relevant, set those entries to −∞ except the designated positive, and only then take the softmax. Three lines, and it removes an entire class of self-inflicted gradient error.

A checklist for building positives

The positive-mining pipeline, in code

Chapter 4 showed the negative side. Here is the positive side, with its two leaks marked, because most teams write this function without noticing either.

def mine_positive(question, answers, bm25_index):
    cands = bm25_index.search(question, top_k=100)   # LEAK 1: BM25 chooses
    for c in cands:                                    # the candidate pool
        if any(a in c.text for a in answers):        # LEAK 2: string match
            return c                                  # stands in for relevance
    return None                                      # question is DISCARDED

Leak 1 means the positive can only ever be a passage BM25 already ranked in its top 100. Any question whose answer is phrased differently from the question is invisible to this function, so it is discarded (the 25.6%) rather than learned from. Leak 2 means "contains the answer text" stands in for "answers the question", which is the spurious-positive problem above.

The single highest-value upgrade, and the one RocketQA formalises: replace the loop body with a cross-encoder score, keep the highest-scoring candidate above a threshold, and widen the candidate pool by unioning BM25's top 100 with a dense retriever's top 100. Both leaks shrink at once, and the discard rate falls.

How often is a spurious positive actually chosen?

Worth a rough estimate, because it calibrates how much to care. Consider a question whose answer is a four-digit year. Of the 100 BM25 candidates, suppose 30 contain that year somewhere — unsurprising for a topically-related set of passages about the same subject — and only 3 of those 30 state it as the answer to this question. The pipeline picks the highest BM25-ranked of the 30. The probability that this is one of the 3 correct ones, if BM25's ranking were uninformative about correctness, is 3/30 = 10%.

BM25's ranking is not uninformative, so the real rate is much better than 10%. But the estimate makes the shape clear: short, common answers have a high spurious-positive rate and long, distinctive answers have a low one. If your domain's answers are entity names or long phrases you can mostly ignore this. If they are numbers, dates, yes/no, or single common words, string matching is close to useless as a relevance criterion and you need a model in that loop.

Masking the extras, concretely

The multi-positive fix from Chapter 3 needs a relevance map. Build it once, offline, from whatever you know:

# relevant[qid] = set of passage ids known to answer this question
mask = torch.zeros(B, M, dtype=torch.bool)
for i, qid in enumerate(batch_qids):
    for j, pid in enumerate(batch_pids):
        if pid in relevant[qid] and j != gold_col[i]:
            mask[i, j] = True                     # a known positive in a negative slot
scores = (q @ p.T).masked_fill(mask, -1e4)
loss   = F.cross_entropy(scores, gold_col)

Where does relevant come from? Three sources, in increasing order of quality: answer-string matches within your training corpus; the union of every gold passage across questions that share an answer and a topic; and a cross-encoder's high-confidence judgements. Even the cheapest version removes real gradient noise, because the most common false negative in Natural Questions is another paragraph of the same article.

The asymmetry between the two kinds of label error

The inverse move: generate the question

Every problem in this chapter comes from starting with a question and hunting for its passage. Turn it around and most of them evaporate.

Take a passage. Ask a generative model: what question does this answer? The resulting pair is a positive by construction. There is no BM25 in the loop, so Leak 1 is gone. There is no answer-string matching, so Leak 2 is gone. There is no discard rate, so the 25.6% survivorship bias is gone. And you can generate a pair for every passage in your corpus, including the ones no annotator would ever have looked at.

Look at the last row against Chapter 6's whole argument. A model asked to write a question about a passage does what a SQuAD annotator did: it reuses the vocabulary in front of it. Unprompted, you regenerate the exact bias you were trying to escape — a training set of high-overlap pairs that teaches lexical matching. The mitigations are the obvious ones and they are not optional: prompt for short, underspecified, colloquial queries; sample with high temperature for diversity; and filter with round-trip consistency, keeping the generated query only if a retriever actually finds the source passage for it.

What good looks like: a positives audit

A concrete, repeatable procedure. It takes an hour and it is the highest-value hour in the project.

Steps 4 and 5 are the ones nobody does, and they are the ones that most often change the plan. A 70% label ceiling means no architectural change can take you past roughly 70%; a 60% false-negative rate at rank 5 means your hard-negative pipeline is subtracting value and the fix is a cross-encoder, not a bigger batch.

How to build an evaluation set that is not lying to you

Chapter 6's whole argument reduces to one operational instruction, so make it explicit and checkable.

A cheap sanity check if you suspect leakage: compute the IDF-coverage histogram from Chapter 1 on your eval set and on a sample of real query logs. If the eval histogram is shifted right, you have built a lexical-matching benchmark and any dense retriever will look worse on it than it will in production.

The whole chapter as one decision table

Note that every row has a "watch out for" entry, and every one of those entries is a way that the proxy differs from relevance. There is no clean source. The discipline is knowing which distortion you accepted and measuring whether it matters, which is exactly the discipline the SQuAD row in Chapter 8 demonstrates.

Chunking is a positives decision too

Chapter 0 presented 100-word chunks as an indexing choice. It is equally a labelling choice, and the interaction is worth naming.

Suppose a question's evidence spans a chunk boundary: the entity is named at the end of chunk 3 and the fact is stated at the start of chunk 4. Now every option is bad. Chunk 3 contains the subject but not the answer — and it will be a high-scoring, answer-free hard negative, forever. Chunk 4 contains the answer string, so distant supervision designates it the positive, but read alone it says "he was born there in 1889" and cannot actually be used. You have simultaneously manufactured a spurious positive and a poisoned hard negative from one boundary.

The middle row's warning is the one people hit in practice. Overlap by 50% and every chunk has two near-identical neighbours that the model is being asked to rank apart on evidence that is genuinely the same. Deduplicate by content hash, or add near-duplicates to the multi-positive mask.

Click logs, and the bias that trains your own ranker

The most tempting positive source in any product with traffic is clicks: the user searched, saw ten results, clicked one. Surely that is a relevance judgement.

It is a relevance judgement conditioned on having been shown, and the conditioning is where the damage lives. Users examine high positions far more often than low ones, so click rate confounds relevance with position.

Work the consequence. Two passages are equally relevant; your current ranker happens to put one at position 1 and the other at position 5. Over 1,000 impressions each, the first collects 300 clicks and the second collects 60. Train on clicks as positives and the first appears in your training set five times as often — not because it is better, but because your current ranker preferred it.

The standard correction is inverse propensity weighting: weight each click by the reciprocal of the probability that its position was examined.

A click at position 5 now counts 1/0.20 = 5.0 while a click at position 1 counts 1/1.00 = 1.0, which exactly cancels the ratio computed above. The remaining problem is estimating P(examined | k), which needs either result randomisation on a slice of traffic or an intervention like swapping adjacent positions — both of which cost a little user experience to buy an unbiased training signal.

Chapter 7: Index and Serve

A trained dual encoder is not a retrieval system. It is half of one. The other half is 64.5 gigabytes of float vectors and a search structure over them, and the engineering there determines whether your beautiful +19 points ever reaches a user.

Step 1: encode the corpus

Every one of the 21,015,324 passages goes through E_P exactly once. Cost:

The paper reports this took 8.8 hours on 8 GPUs. Sanity-check that against the theoretical floor: eight V100s at a realistic 30 TFLOP/s each is 240 TFLOP/s aggregate, so 5.92×10¹⁷ / 2.4×10¹⁴ = 2,467 seconds = 41 minutes. The measured time is 12.8× the floor, implying roughly 8% utilisation.

That gap is not incompetence, and understanding it is genuinely useful. Passages are short (~140 tokens), so kernels are small and launch overhead is significant; tokenisation of 21M strings on CPU competes with the GPU for the data pipeline; the encoding runs in fp32 in the original implementation; and every output vector must be moved to host memory and written to disk, which at 3,072 bytes × 21M is 64.5 GB of I/O. Encoding jobs at this scale are almost always data-pipeline bound, not FLOP bound, and the fix is batching, fp16, and more CPU workers, not more GPUs.

The output:

Step 2: choose an index

Retrieval is now maximum inner-product search (MIPS): given q, find the k rows of P with the largest q · p. Three families, and the choice is a three-way trade between memory, latency, and recall.

Flat / brute force (faiss.IndexFlatIP). Compute all 21M dot products. Exact, no build time, no tuning. Cost per query in FLOPs is 32 GFLOPs, as computed in Chapter 2 — but that is not the binding constraint. The binding constraint is memory bandwidth: you must stream all 64.56 GB through the arithmetic units, and at 1 TB/s of HBM that alone is 64.6 ms per query, before any other work. That is fine for offline evaluation and far too slow for a product at any real QPS.

Graph-based (IndexHNSWFlat). HNSW builds a navigable small-world graph over the vectors: each node links to a few dozen neighbours, arranged in layers so that search descends from coarse to fine. A query walks the graph greedily, touching a few thousand vectors instead of 21 million. The paper reports the resulting system serving 995 questions per second, against 23.7 q/s for the Lucene BM25 baseline — a 42× throughput advantage that surprises people who assume dense retrieval is the slow option.

The price is memory and build time. The graph edges must be stored alongside the vectors; the paper reports an index of roughly 151 GB and about 8.5 hours to build on a single high-memory server. Compare the BM25 index: about 0.5 hours to build and a small fraction of the memory. The index is where dense retrieval is expensive, not the query.

Quantised (IndexIVFPQ). If 151 GB is not available, compress. Product quantisation splits each 768-dimensional vector into m subvectors and replaces each subvector with the id of its nearest centroid from a learned codebook of 256:

Scoring works directly on the codes: precompute, per query, a 96 × 256 table of partial inner products, then a passage's score is 96 table lookups and adds — no floating-point multiplies at all. You lose exactness, typically a few points of recall at these compression rates, and you gain the ability to hold the whole index in one machine's RAM.

Step 3: the full serving path, with a latency budget

Here is the request, end to end, with realistic per-stage costs. This is the diagram to have in your head during a design review.

Read the proportions. The retriever — the component this entire paper is about — is 15% of the latency. Reading twenty passages costs five times as much as finding them. This is why every production system fights over k: dropping from k = 100 to k = 20 cuts the dominant cost by 80%, and the whole value of a better retriever is that you can afford a smaller k for the same recall.

The operational cost nobody warns you about

You retrain the encoder. Maybe you added RocketQA-style denoised negatives and gained three points. Now what?

Every one of the 21,015,324 stored vectors was produced by the old E_P. They are now meaningless: your new E_Q lives in a different space, and dot products across the two are noise. You must re-encode the entire corpus (8.8 GPU-hours), rebuild the index (8.5 hours), and swap it — ideally without downtime, which means holding two indexes, which means 302 GB of RAM.

Compare BM25: a model update is not a concept, because there is no model. A new document is appended to a handful of posting lists in microseconds and is searchable immediately.

None of this argues against dense retrieval. It argues that the +19 points has an operational price tag, and that the price is paid in index lifecycle rather than query latency. Teams that budget only for the second one get an unpleasant surprise on their first model update.

Sharding, and why it is easy here

64.56 GB of vectors does not fit in one GPU, and 151 GB of HNSW does not fit comfortably in many single hosts. Split the corpus across N shards; each host holds 21M/N vectors and its own index. A query is broadcast to all shards; each returns its local top-k; a coordinator merges the N×k results and takes the global top-k.

This is exactly correct — not an approximation — for one specific reason: the score is a plain dot product, computed independently per passage. There is no cross-passage normalisation anywhere in the scoring function, so a passage's score does not depend on which shard it landed in, and merging local top-k lists yields the true global top-k. Had DPR used a softmax over the corpus, or any score that normalises across candidates, sharding would have required a second communication round. The architectural simplicity of Chapter 2 pays off again here.

Product quantisation, derived

Compressing 64.56 GB to 2 GB sounds like magic. It is a very simple idea applied 96 times.

Step 1 — split. Cut each 768-dimensional vector into m = 96 contiguous subvectors of 8 dimensions each. A passage is now 96 little vectors instead of one big one.

Step 2 — learn a codebook per slice. For slice s, run k-means with k = 256 over the 21 million 8-dimensional subvectors that occupy that slice. You now have 256 representative 8-dimensional centroids for slice s. Storage for all codebooks: 96 slices × 256 centroids × 8 dims × 4 bytes = 786 KB. Negligible, and shared across the whole corpus.

Step 3 — encode. Replace each subvector by the index of its nearest centroid: one byte, since 256 = 2⁸. A passage is now 96 bytes.

Step 4 — score without decompressing. This is the elegant part. The inner product decomposes over slices:

Given a query, precompute the table T[s][c] = q^(s) · c_s,c for all 96 slices and all 256 centroids — that is 96 × 256 = 24,576 eight-dimensional dot products, about 393,000 FLOPs, done once per query. Then scoring any passage is 96 table lookups and 95 additions. No multiplications at all. This is called asymmetric distance computation: the query stays exact, only the database is quantised, which is why the accuracy loss is far smaller than compressing both sides.

Cost per passage, compared honestly: 1,536 FLOPs of exact float arithmetic against 96 byte-indexed lookups. Both are cheap; the difference that matters is that 2 GB fits in cache-friendly memory on one machine and 64.56 GB does not.

Sharding arithmetic

Suppose you cannot find a 151 GB machine, or you want redundancy. Split into N shards:

Latency is the maximum over shards, not the sum, so fan-out costs you tail latency rather than mean latency — and with 16 shards you are exposed to the slowest of 16 machines on every request, which is the classic tail-amplification problem. The merge itself is trivial: 320 scores, one partial sort.

The correctness argument bears repeating because it is not automatic. Merging local top-k gives the true global top-k only because the score is computed per passage with no cross-passage normalisation. Had DPR normalised scores within a candidate set — a softmax over the corpus, say — a passage's score would depend on which shard it landed in, and local top-k lists would not compose. Simple scoring functions shard trivially; clever ones do not.

The cost of a model update, in hours and dollars

The compute is not the painful part — $560 is nothing. The painful part is the 1.7-day cycle time, which sets how fast your team can iterate, and the operational rule it implies: you cannot A/B test two retrievers cheaply, because each arm needs its own full index. Teams that want fast retrieval iteration either shrink the corpus for development or keep a small representative shard permanently indexed under both models.

A rollout runbook

Latency is a distribution, not a number

The 26 ms budget in the flow above is a mean, and means are how retrieval systems get shipped and then get paged about. Every stage has a tail, and the tails compose badly.

The hydration row is the instructive one and it catches people every time. Twenty independent lookups means your p50 is one lookup's p50 but your latency is the maximum of twenty draws. If a single lookup has a 1% chance of taking 25 ms, the probability that none of twenty does is 0.99²⁰ = 0.818 — so 18% of requests hit at least one slow read. A per-lookup p99 becomes a per-request p82. The fix is a batched multi-get, or storing the passage text alongside the vector so the two travel together.

Caching, and why it works better here than you expect

Query distributions are heavily skewed — a small set of questions accounts for a large share of traffic. Three cache layers, in increasing order of hit rate and decreasing order of usefulness:

Note the invalidation column: it is the index-lifecycle problem from earlier, propagating outward. A model update does not just invalidate 64.56 GB of vectors; it invalidates every cache derived from them, so your first minutes after a swap run at 0% hit rate and full cost. Capacity-plan for the cold cache, not the warm one.

A hybrid serving architecture

Chapter 1 concluded that the right production answer is usually both retrievers. Here is what that costs to run.

Two practical notes. DPR's λ = 1.1 score combination requires both scores to live on comparable scales, and they do not — BM25 is unbounded and corpus-dependent, dot products are unbounded and model-dependent — so any fixed λ is a per-corpus constant you must re-tune. Reciprocal rank fusion sidesteps this entirely by using only positions, which is why it is the more common production choice despite being cruder.

Failure modes to alarm on

Cheaper storage before you reach for quantisation

Product quantisation is powerful and it is not the first thing to try. Two simpler moves come first, and both are nearly free in quality.

fp16. Halve the bytes per dimension. The index goes from 64.56 GB to 32.28 GB, and the accuracy cost is essentially nil — fp16 has about 3 decimal digits of precision, and your dot products are sums of 768 terms whose ranking gaps are far larger than that. This is the free 2×, and many systems never bother.

int8 with a per-vector scale. Store each dimension as one byte plus a float scale per vector: 21,015,324 × (768 + 4) bytes = 16.2 GB, a 4× reduction. Reconstruct as scale × int8. Typical recall loss is under a point, and integer dot products are fast on modern hardware.

The ordering to work down: fp16, then int8, then a smaller d if your model supports it, then PQ. Reaching for PQ while still storing fp32 elsewhere in the stack is a common and slightly embarrassing sequencing error.

Two-stage retrieval within the index itself

One more idea that costs nothing and is widely used: search a compressed index for a generous candidate set, then rescore that small set with the exact vectors.

The quantisation error only has to preserve membership in the top 1,000, not the ordering within it, and membership is a far weaker requirement. Most of PQ's accuracy cost disappears, for 3 MB of reads and about a microsecond of arithmetic. If you take one implementation trick from this chapter, take this one.

What to alarm on, and at what threshold

The first row is worth the whole table. A version mismatch between the query encoder and the passage index produces results that are syntactically valid, latency-normal, and semantically random. There is no exception, no partial degradation, and no metric except relevance itself will notice. Pin the version, assert it at startup, and refuse to serve on mismatch.

"Why not just use a vector database?"

You will be asked this, so it is worth being precise about what a managed vector store does and does not remove from the list above.

Read the bottom three rows. A vector database removes the distributed-systems work, which is real and worth paying for. It removes none of the work this lesson has been about: the chunking, the encoder, the index lifecycle, and the version discipline. "We use a vector database" answers a storage question and leaves every retrieval-quality question exactly where it was.

Chapter 8: The Numbers, Including the Bad One

Five datasets, two metrics, and one row that the paper could have hidden and did not. Let us read all of it.

The training setup, so the numbers mean something

The training-set sizes matter for reading the results, because they vary by a factor of fifty:

Retrieval accuracy — the table

Four wins, one loss, and the loss is the informative one. Chapter 6 explained it fully: SQuAD's questions were written by people reading the answer paragraph, which manufactures the lexical overlap BM25 thrives on, and its passages come from only a few hundred articles, which puts the training distribution badly out of line with a 21-million-passage index. The paper states both reasons plainly rather than dropping the row.

Read the curve, not the point

Two readings worth making explicitly. First, the curves converge as k grows: at k = 100 the gap on NQ is 11.7 points, at k = 20 it is 19.3. A better retriever helps most exactly where you need it most, at small k, which is where reader cost lives. Second, on SQuAD the DPR curve sits below BM25 across the whole range — this is not a crossover artifact of one k, it is a genuinely worse retriever on that distribution.

How little data DPR needs

The paper's most under-quoted result: train on 1,000 NQ examples and DPR already beats BM25 on top-20. One thousand.

Why is that possible, when the model has 220M parameters? Because almost none of them are being learned from those thousand examples. BERT arrived already knowing English — that "villain" and "bad guy" occupy nearby regions is a fact about the pretraining corpus, not about your labels. What the thousand examples teach is a much smaller thing: which of the many similarity relations BERT encodes corresponds to "answers", and how to read it out of the [CLS] position. That is closer to fitting a projection than to learning a representation.

End-to-end question answering

Retrieval accuracy is instrumental. The product metric is exact-match on the final answer string:

DPR beats REALM on NQ without REALM's retrieval-oriented pretraining, which is the claim from Chapter 0 finally cashed. On the two smallest datasets, WebQuestions and CuratedTREC, DPR does not lead — REALM comes out ahead, and that is exactly the pattern you would predict: expensive pretraining buys the most where task supervision is scarcest, and 1,125 training questions is scarce.

Also note the sign flip in the last row: hybrid helps TriviaQA (+1.1) and hurts NQ (−2.5), matching the retrieval-level result from Chapter 1. Fusion is not a free improvement; it is a bet that the two systems fail differently.

Cross-dataset generalisation

A dense retriever trained on Natural Questions and evaluated on WebQuestions and CuratedTREC — without seeing a single training example from either — loses only a few points against models trained directly on those datasets, and still comfortably beats BM25 on them. Two implications.

The optimistic one: the model is learning a general "this passage answers this question" relation, not memorising NQ's topics. The cautionary one, which the BEIR benchmark would make painful two years later: all three of those datasets are Wikipedia-based factoid QA. Generalising across three samples of the same distribution is a much weaker claim than generalising to legal contracts or biomedical abstracts. Contriever exists because the second claim turned out to be false for early dense retrievers.

What the ablations settled

Reading the errors, not just the averages

The paper includes a qualitative analysis, and it is the part that generalises furthest, because it names which questions each system gets right. Two representative cases, both from the paper:

These two rows are the whole argument of Chapter 1 in concrete form, and they are also a design instruction: the errors are complementary, which is why hybrid retrieval helps at all, and why it helps most exactly where each system is weak.

What top-k accuracy does not tell you

Every number in this chapter counts a retrieval as successful if some returned passage contains the answer string. Three ways that overstates things, all of which you will meet in your own evaluation:

Presence is not evidence. A passage containing "1969" for an unrelated reason counts. The reader then has to distinguish real evidence from coincidental string presence, and some of the gap between retrieval accuracy (78.4%) and end-to-end EM (41.5%) is exactly that.

One-passage answers are assumed. Questions needing two passages combined — multi-hop questions — are scored as if a single passage sufficed. HotpotQA exists because this class of question defeats every single-vector retriever, and DPR's benchmarks mostly avoid it.

Unanswerable questions are absent. The datasets are filtered so that every question has an answer in Wikipedia. A production system receives questions with no answer anywhere, and neither the metric nor the training objective has anything to say about them — the softmax will confidently rank something first, exactly as the fixed-head classifier from Chapter 0 always names a class.

The ablations you should run on your own data

DPR's ablations are a template. Six experiments, in the order of return per unit of effort, each answering a question you cannot answer by reasoning:

Note that experiments 1, 2 and 5 are diagnostics rather than improvements. They are also the three most often skipped, and each of them can invalidate months of modelling work in under a day.

Where these numbers sit in 2026

DPR's 78.4 top-20 was a landmark in 2020. Putting it beside what came after is not a way of diminishing it — it is the clearest picture of what the negative-mining lineage was actually worth.

Read the gaps. BM25 to DPR is +19.3 on NQ — the paradigm shift. DPR to RocketQA is +4.3 — three more years of negative engineering. The first jump came from having hard negatives at all; everything after came from having better ones, and the returns were exactly the diminishing kind you would predict from Chapter 4's dose-response curve.

What generalisation actually looked like, later

Chapter 8's cross-dataset result — NQ-trained DPR transferring to WebQuestions and CuratedTREC — reads as encouraging until you notice all three are Wikipedia factoid QA. BEIR, published a year later, evaluated dense retrievers on eighteen datasets spanning scientific claim verification, biomedical search, financial question answering, argument retrieval and duplicate-question detection.

The finding that mattered: a dense retriever trained on MS MARCO frequently lost to BM25 on datasets far from its training distribution, sometimes badly. Chapter 1's third BM25 advantage was not a footnote; at the time it was the single largest practical objection to dense retrieval, and it is what Contriever, and then the instruction-tuned multi-domain embedders, were built to answer.

The lesson for your own work is the same one Chapter 6 gives from the label side: a dense retriever is a fitted function, and its quality is a statement about a distribution, not about text in general. Measure on your own corpus, with your own queries, before believing any number in this chapter applies to you.

The reader, briefly, because the end-to-end numbers depend on it

Every EM number in this chapter came through a reader, so a paragraph on what it is prevents mis-attributing credit.

DPR's reader takes the k retrieved passages and does two things at once. A passage selection head scores each passage's [CLS] to pick which one holds the answer, and a span extraction head predicts start and end token positions within each passage. Both are linear layers on top of a BERT encoder that reads each (question, passage) pair. The final answer is the best span from the best-scoring passage.

Note the reader is itself a cross-encoder — the architecture Chapter 2 ruled out for retrieval. That is the whole point of the two-stage design: the expensive joint model becomes affordable once something cheap has narrowed 21 million candidates down to 20. Improving the retriever lets you shrink k, which shrinks the only cost that scales.

Reproducing this, if you want to

The last row is the one worth doing. Half a million passages fits comfortably in one GPU's memory as an exact index (500,000 × 768 × 4 = 1.5 GB), so you can skip FAISS tuning entirely and still watch the in-batch-only model land near BM25 and the plus-one-hard-negative model pull clearly ahead. That single before-and-after is the whole paper, at a scale you can run tonight.

What the paper did not test

Reading a results section well includes noticing the experiments that are absent. Five, each of which became somebody's next paper.

None of these is a criticism — a paper that tested all five would have been five papers. The point is a reading habit: the scope of the experiments is the scope of the claim, and the most common way to be burned by a benchmark result is to apply it outside the distribution it was measured on.

How many evaluation questions do you need?

Every improvement in this chapter is a difference between two percentages measured on a finite sample. It is worth knowing when such a difference means anything.

Top-k accuracy is a proportion, so on n independent questions its standard error is

At p = 0.78 and n = 1,000: p(1−p) = 0.1716, divided by 1,000 is 1.716×10⁻⁴, square root 0.0131. So a single measured number of 78.0% carries a 95% interval of roughly ±2.6 points. And comparing two systems on different question samples inflates it by √2:

Which means a 2-point improvement measured this way is indistinguishable from noise. That number should be sobering: a great deal of published and internal retrieval progress lives inside that band.

The fix is not a bigger eval set — reaching ±1 point at 95% would need n = 1.96² × 0.1716 / 0.01² ≈ 6,600 questions. The fix is to evaluate both systems on the same questions and analyse only the disagreements, which is McNemar's test. If the two systems agree on 880 of 1,000 questions and disagree on 120, the standard error of the difference is

Better, from the same data, purely by not throwing away the pairing. And the practical instruction that follows is the useful part: keep your eval questions fixed across experiments, and report the count of questions won and lost, not just the delta.

Reporting retrieval results honestly

A short checklist, assembled from every trap this lesson has walked through. If a retrieval result — yours or someone else's — does not answer these, you cannot tell what it means.

Chapter 9: By Hand, End to End

One training step, computed entirely by hand, to a final decimal. If you can reproduce this on paper you understand the objective, and if you cannot, some part of Chapters 3 and 4 has not landed yet.

The setup

A batch of three questions with their three gold passages. No BM25 hard negatives yet — pure in-batch. The towers have produced six 768-dimensional vectors; we only need the nine dot products they induce.

The similarity matrix S = Q P^T, shape (3, 3). Rows are questions, columns are passages. Raw dot products — unnormalised, no temperature, exactly as DPR computes them:

Before computing anything, read it. The diagonal (6.0, 5.4, 4.8) holds the positives and is the largest entry in each row — the model is already roughly right. The entry S₁₂ = 4.5 is the interesting one: question 1 is about Lord of the Rings, and so is passage 2, so the model finds it plausible. That is a genuine hard negative, arriving free from the batch. S₁₃ = 1.2 is the Irish Sea passage, which shares nothing with question 1 — an easy negative.

Row 1, in full

The label for row 1 is column 1. Subtract the row maximum (6.0) before exponentiating; softmax is shift-invariant, so this changes nothing except the numerical stability:

Verify the log by hand if you like: log(1.2) = 0.182322, and 1.231360/1.2 = 1.026133, whose log is 0.025796. Sum: 0.208118. Good.

Row 2

Label is column 2, the maximum is 5.4:

Row 3

Label is column 3, the maximum is 4.8:

The batch loss

Compare it to the chance-level floor for three candidates, log 3 = 1.098612. The model is at 13% of the guessing loss, so it is clearly learning. Compare it also to zero: the remaining 0.144 is almost entirely row 1's 0.208, which is almost entirely the single cell S₁₂ = 4.5.

Where the gradient goes

Using ∂L/∂S_ij = (p_ij − y_ij)/3 — the division by 3 because we averaged over the batch:

Check row 1's split: 0.181209 / (0.181209 + 0.006683) = 0.181209/0.187892 = 0.96444. Ninety-six percent of everything this question teaches the model about "wrong" comes from one cell. That single number is Chapter 4 in miniature: the batch contributed two negatives and delivered the gradient of approximately one.

Now plant a false negative and watch it break

Extend the batch to four passages by adding one that also correctly answers question 1 — a different Wikipedia paragraph naming Sauron as the antagonist — but which is labelled as question 4's gold, not question 1's. The model scores it 5.8 for question 1, correctly recognising it as relevant. Row 1 becomes [6.0, 4.5, 1.2, 5.8]:

Nothing about the model got worse. Its similarity for a genuinely correct passage was high, which is the behaviour you want. The label disagreed, and the loss punished it for being right. Nearly 40% of the update now pushes question 1 away from a passage that answers it — more force than the honest hard negative receives (10.9%).

The same computation, in code

import torch, torch.nn.functional as F

S = torch.tensor([[6.0, 4.5, 1.2],
                  [2.1, 5.4, 3.0],
                  [0.9, 2.4, 4.8]])       # (B, M) = Q @ P.T
labels = torch.arange(3)                        # gold of question i is column i
loss = F.cross_entropy(S, labels)              # tensor(0.1445)

# the gradient split, per row
P = S.softmax(dim=1)
G = P - F.one_hot(labels, 3).float()      # row 0: [-0.1879, 0.1812, 0.0067]

Five lines, and every number in this chapter falls out of them. When you build this yourself, print G for one batch and look at where the mass sits. If it is spread evenly across all columns, your negatives are too easy. If one non-diagonal column dominates every row, go and read that passage — it is either a superb hard negative or a false one, and only your eyes can tell.

What came after: the two directions the field took

DPR compressed each passage into one vector. Two families of successors relaxed exactly one of its constraints each.

ColBERT — relax the pooling. Keep every token vector instead of pooling to one, and define similarity as a sum over query tokens of the maximum similarity against any passage token (the MaxSim late-interaction operator). This restores the lexical precision that Chapter 2's stage 6 destroyed — the Thoros-of-Myr failure largely goes away, because a rare token can now match a rare token directly. The price is the index: instead of one 768-dim vector per passage you store one per token, roughly a hundred times more, which ColBERTv2 then attacks with residual compression and the PLAID engine. It is the same trade as always: more of the interaction retained, more of the index paid for.

SPLADE — relax the density. Keep a sparse, vocabulary-sized output, but learn which terms to activate, so a passage about villains can light up the "bad guy" coordinate. This gives you a learned representation that still runs on an inverted index, inheriting decades of sparse-retrieval infrastructure. The lexical basis stops being a prison once the model gets to decide what goes in it.

And downstream, RAG: DPR's retriever wired to a generator, with the retrieved passages placed in the generator's context. That paper shares authors with this one and uses this index directly. Every retrieval-augmented system you have used inherits the loss you just computed by hand — which means it also inherits the negative-sampling decisions of whoever trained its embedder.

The cheat sheet

Where to go from here

Build it yourself — the weekend recipe

References

1. Karpukhin, V., Oǧuz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., Yih, W. "Dense Passage Retrieval for Open-Domain Question Answering," EMNLP 2020 — arXiv:2004.04906. The paper this lesson is built on.
2. Xiong, L. et al. "Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval" (ANCE), ICLR 2021 — arXiv:2007.00808. Asynchronous index refresh and the variance argument for global negatives.
3. Qu, Y. et al. "RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering," NAACL 2021 — arXiv:2010.08191. Cross-batch negatives, cross-encoder denoising, data augmentation.
4. Izacard, G. et al. "Unsupervised Dense Information Retrieval with Contrastive Learning" (Contriever), 2021 — arXiv:2112.09118. Independent cropping, MoCo queue, zero-shot BEIR.
5. Khattab, O. & Zaharia, M. "ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT," SIGIR 2020 — arXiv:2004.12832; and Santhanam, K. et al. "ColBERTv2," 2021 — arXiv:2112.01488.
6. Lewis, P. et al. "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks" (RAG), NeurIPS 2020 — arXiv:2005.11401. DPR's index, wired to a generator.
7. Lee, K., Chang, M.-W., Toutanova, K. "Latent Retrieval for Weakly Supervised Open Domain Question Answering" (ORQA / Inverse Cloze Task), ACL 2019 — arXiv:1906.00300. The pretraining approach DPR showed was unnecessary.
8. Guu, K. et al. "REALM: Retrieval-Augmented Language Model Pre-Training," ICML 2020 — arXiv:2002.08909. The strongest pretraining-heavy baseline DPR beats on NQ.
9. Johnson, J., Douze, M., Jégou, H. "Billion-scale similarity search with GPUs" (FAISS), 2017 — arXiv:1702.08734. The index behind every number in Chapter 7.
10. Malkov, Y. & Yashunin, D. "Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs," 2016 — arXiv:1603.09320. HNSW.
11. Thakur, N. et al. "BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models," 2021 — arXiv:2104.08663. Where the out-of-domain claim from Chapter 1 gets measured properly.
12. Kwiatkowski, T. et al. "Natural Questions: A Benchmark for Question Answering Research," TACL 2019. The dataset whose gold long answers make DPR's positives honest.

A second worked example: the 2×2 batch

Smaller, so you can do it entirely in your head after a little practice. Two questions, two golds, diagonal similarity 5.0 and off-diagonal 3.0:

Row 1: subtract the max, 5.0. e⁰ = 1, e⁻² = 0.135335. Sum = 1.135335.

Row 2 is the mirror image, so L₂ = 0.126928 and the batch loss is 0.126928. The chance-level floor is log 2 = 0.693147, so the model is at 18.3% of guessing.

Now change one thing: raise the gap from 2.0 to 4.0 by pushing the diagonal to 7.0. Then e⁻⁴ = 0.018316, sum = 1.018316, L = 0.018150 — a 7× reduction in loss from a 2-point change in a single similarity. This is the exponential sensitivity of the objective to the margin, and it is why loss values are hard to compare across runs while retrieval accuracy is not: loss is an exponential function of a gap whose units the model itself controls.

What to print during a training run

The last two rows deserve emphasis. In-batch accuracy is cheap and misleading: distinguishing a gold from 255 batch-mates is a far easier task than distinguishing it from 21 million passages, so it saturates near 1.0 long before real retrieval is good. Track it for debugging, never for decisions.

Five things to remember when everything else fades

A design exercise, before you close the tab

You have been asked to build search over 4 million internal engineering documents. Users are engineers typing things like "why does the ingest job OOM on large parquet files" and "ORA-01555 during nightly rollup". You have no labelled pairs, two GPUs, and six weeks. Work through it against the chapters.

Notice that four of the six answers are decisions about data and evaluation, and none is a decision about architecture. That distribution is not an accident of this exercise; it is the distribution the DPR paper itself found, and it is the reason its ablation section is more valuable than its method section.

The three sentences to keep

One complete training step, in code

Everything in ten chapters, assembled. This is the loop; every earlier chapter is a decision inside it.

for batch in loader:                                  # CH 4: batch = the negative pool
    q_ids, p_ids, gold_col = batch                      # (B,L) (M,L) (B,)

    q = E_Q(**q_ids).last_hidden_state[:, 0]           # CH 2: [CLS] → (B, 768)
    p = E_P(**p_ids).last_hidden_state[:, 0]           # (M, 768), M = 2B

    if world_size > 1:                                 # CH 5: cross-batch negatives
        p = all_gather_with_grad(p)                     # (A*M, 768) — grad MUST flow
        gold_col = gold_col + rank * M                  # reindex into the gathered block

    S = q @ p.T                                       # CH 2: raw dot product, no temperature
    S = S.masked_fill(known_positives, -1e4)           # CH 6: multi-positive mask
    loss = F.cross_entropy(S, gold_col)                 # CH 3: NLL, logsumexp inside

    loss.backward(); opt.step(); opt.zero_grad()

    if step % 100 == 0:                                # CH 4: the diagnostics that matter
        P = S.softmax(1)
        log(loss=loss, floor=math.log(S.shape[1]),
            inbatch_acc=(S.argmax(1) == gold_col).float().mean(),
            max_offdiag=P.masked_fill(is_gold, 0).max(),
            qnorm=q.norm(dim=1).mean())

Fourteen lines of substance. The research of the last five years is almost entirely in two places: what loader put into p_ids, and what known_positives knows. Neither is in the model.

If you remember one table