Retrieval-Augmented Generation

You run an office hours queue every week, and a student asks a chat assistant built on a language model: “Which office hour this week is least crowded?” The model has never seen this week's queue. It cannot have. So it guesses, and it guesses in a very specific, confident voice: “Try the Wednesday afternoon session, midweek hours are usually quietest.” That sentence is fluent, reasonable-sounding, and might be wrong tonight.

Now hand the same model one more thing before it answers: the actual queue log from the last three weeks. “Tuesday 4pm: 14 waiting on average. Thursday 9am: 3.” With that text sitting in its context, the model writes a different answer: “Thursday 9am, about 3 students ahead of you.” Same model, same question, one added ingredient: the right few lines of real data, placed in front of it before it writes a word.

That one move, fetch the right text first, then let the model write with it in view, is retrieval-augmented generation, usually shortened to RAG. The lecture's own name for the failure the first answer shows is hallucination: a fluent, confident sentence with no real evidence behind it. RAG does not make a model smarter. It changes what the model is allowed to look at while it writes.

Why not just make the model remember everything?

One answer to “the model does not know this” is to train it on more data, so the fact becomes one of its memorized weights. That approach has two costs a builder feels immediately. First, it is static: the moment the queue log updates next Tuesday, the trained-in fact is stale, and retraining a model to fix one fact is absurd. A retrieval system, by contrast, is dynamic: add or update one document and the very next query can use it. Second, a memorized answer is unverifiable: the student has no way to check where “Thursday is quietest” came from. A retrieved answer is interpretable: the model can point at the exact line it used, a citation, and the student, or you, can check it.

The shape every RAG system shares: retriever, then reader

This idea predates today's language models. A 2017 system called DrQA (Chen et al., “Reading Wikipedia to Answer Open-Domain Questions”) answers “How many of Warsaw's inhabitants spoke Polish in 1933?” by splitting the job into two pieces in a straight line. A document retriever searches all of Wikipedia and returns a short list of pages likely to contain the answer. A document reader then reads only those few pages closely and pulls out the answer, 833,500. Neither piece does the other's job: the retriever never reads carefully, and the reader never searches the whole encyclopedia.

The lecture writes this shape as two functions, and every RAG system in this lesson, however elaborate, is still built from exactly these two pieces.

f(𝒟, Q) → P₁…P_K

The retriever f takes the whole document collection 𝒟 = D₁, D₂, …, D_N and a query Q, and returns the K passages it judges most relevant. K is a number you pick ahead of time, say 100.

g(Q, {P₁…P_K}) → A

The reader (or generator, once it is a language model instead of a classifier) takes the query and those K passages and produces the answer A. This is a reading-comprehension problem: the answer is already sitting in the text, somewhere.

DrQA's own retriever was a fixed, non-learned module (word counting, Chapter 6 names it TF-IDF) paired with a trained neural reader. Nothing here requires either half to be a neural network at all. That separation, something cheap and fast narrows millions of documents down to a few dozen, something careful and expensive reads only those few, is what makes retrieval affordable at any scale. Try it below: narrowing is the whole job of the retriever, and it happens before the reader ever sees a word.

Picture the collection as Wikipedia's entire text, and the retriever's job as finding the handful of sentences that matter among billions of words. However the retriever decides what counts as relevant, and the rest of this lesson covers several different answers, the reader downstream never knows the difference. It only ever sees K passages and a question.

The road

The lecture covers RAG in six parts, and this lesson follows them in order: why retrieval helps at all, the original RAG pipeline trained end to end, how to index a document collection, how to search that index with several different retrievers, how an agent decides when and what to retrieve, and where the whole pipeline breaks.

1. The RAG pipeline: the original paper, trained retriever and generator together.
2. Chunking strategies: fixed-size, semantic, and context-enriched splits.
3. Contextual retrieval and late chunking: giving a chunk back its lost context.
4. RAPTOR and GraphRAG: trees and graphs for questions one chunk cannot answer.
5. Vector search and HNSW: finding the nearest chunk among millions.
6. BM25: scoring by word overlap, still winning on exact terms.
7. DPR and scoring a retriever: dense vectors, and Hit@k, MRR, Recall@k, nDCG.
8. Hybrid search, reranking, ColBERT: combining retrievers, then late interaction.
9. Agentic RAG: Self-RAG, CRAG, and Search-R1 deciding when to search.
10. Where RAG breaks: nine named failure modes, sorted into three places.
11. The whole pipeline on one page: numbers, code, and what comes next.

Chapter 0's retriever and reader were two separate, independently built pieces: a fixed word-counter and a reading-comprehension network trained on its own. The 2020 paper that gave this lesson its name, Lewis et al., “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks,” asks a sharper question: what if the retriever and the generator are trained together, as one system, so each gets better at the job the other needs from it?

Two probabilities, written precisely

The paper names its retriever p(z|x): given a query x, it returns a probability over passages z, and in practice the system keeps only the top-K, the K passages p(z|x) rates highest. It names its generator p(y|x, z): given the query and one retrieved passage, it writes the answer y one token at a time, exactly the chain-rule machinery from Lecture 2. Chapter 7 derives p(z|x) in full; for now, treat it as a black box that ranks passages.

The question the paper has to answer is how these two probabilities combine into one answer, when there is no single “correct” passage, only K candidates of varying relevance.

p(y|x)

The final probability of answer y given only the query x: what the whole RAG system outputs, once the retrieved passage is marginalized away.

∑_z∈top-K

Sum over the K retrieved passages only, not the whole collection; everything outside the top-K is treated as having zero chance of being the right evidence.

p(z|x)

How much the retriever trusts passage z for this query: its weight in the sum.

p(y|x,z)

How likely the generator is to write answer y, given the query and that one specific passage.

Read in words: weight each passage's candidate answer by how much the retriever trusts that passage, then add the weighted answers together. A passage the retriever is unsure about contributes a quieter vote; a passage it is confident about dominates. The paper trains both p(z|x) and p(y|x,z) jointly, end to end, by minimizing the negative marginal log-likelihood summed over training examples, −log p(y_j|x_j) for each example j: exactly Chapter 2's cross-entropy idea, just with a retrieval step folded inside it.

One design choice remains: does the generator commit to one passage for the whole answer, or is it allowed to change its mind mid-sentence? The paper offers both. RAG-Sequence picks one passage and writes the entire answer conditioned on it alone, then compares whole answers across the K passages. RAG-Token re-picks the passage, effectively re-weighting all K of them, at every single token of the answer, so different parts of one answer can draw on different evidence. Try both below.

The pieces that made this specific paper run

The paper's retriever is DPR (dense passage retrieval, Chapter 7 in full): p(z|x) ∝ exp(d(z)^Tq(x)), where d(z) = BERT_d(z) encodes the passage and q(x) = BERT_q(x) encodes the query, both with BERT, and top-k(p(·|x)) is solved with Maximum Inner Product Search (MIPS, Chapter 5 shows how this scales to millions of vectors). Its generator is BART, an encoder-decoder model; the query x and the passage z are simply concatenated and handed to BART together, so from the generator's point of view, retrieval has already happened by the time it starts writing.

Did training the two together actually help?

On four open-domain question-answering benchmarks (Natural Questions, TriviaQA, WebQuestions, CuratedTrec), measured by exact match, RAG-Sequence scores 44.5 on Natural Questions and 68.0 on TriviaQA, well above a closed-book T5-11B model with no retrieval at all (34.5 on Natural Questions). RAG also beats DPR plus a separate, non-joint reader on most of these sets, which is the paper's real claim: training the retriever and generator together, so the generator's needs can shape what counts as a relevant passage, beats bolting two independently trained pieces together.

The paper's most telling result is not a benchmark number but a human judgment. Asked to generate original Jeopardy-style questions, human raters found RAG's output more factual than BART's alone 42.7% of the time, versus BART only 7.1%, and more specific 37.4% of the time versus BART's 16.8%. Both were rated equally good or equally poor the rest of the time, but when they disagreed, retrieval almost always won, and almost always on the two properties grounding is supposed to buy you: getting facts right, and being concrete rather than vague.

Before any retriever can find anything, the document collection 𝒟 has to become a list of fixed-size pieces, each with one vector. That piece is a chunk: the unit retrieval actually operates on. Get chunking wrong and nothing downstream, no matter how clever, can recover the information you threw away.

What makes a good retriever: an embedding per chunk

An embedding is a fixed-length list of numbers, a vector, standing in for a piece of text so that similar meanings land near each other in that number space; our vector databases lesson builds one from zero. Different kinds of embedding cover different units of text, and RAG needs the passage kind specifically.

Retrieval needs two things from this table's passage row: a fixed-dimension vector per chunk, so every chunk can be compared on equal footing no matter its original length, and a similarity metric, almost always cosine similarity or the inner product, to say how close two vectors are. Chapter 5 shows how that comparison scales to millions of chunks.

Fixed-size chunking, and its two failure directions

Fixed-size chunking is the simplest approach: cut the document every 100 to 500 tokens, regardless of where a sentence or idea ends. It fails in two opposite directions. Too small, and a chunk's vector has almost no context to describe. a lone sentence like “The company's revenue grew by 3% over the previous quarter” does not even say which company. Too large, and one vector has to average many different topics into a single point, blurring all of them.

The standard mitigation is overlap: let consecutive chunks share 10 to 20% of their tokens, so a sentence that would otherwise be cut exactly at a chunk boundary still appears whole inside at least one chunk. Overlap does not fix the averaging problem, only the sentence-splitting one.

The lecture's own example makes the trade-off concrete. Fixed-size chunking at 50 tokens slices mid-sentence, “due Week 6, Thursday” ends one chunk and “11:59pm” begins the next, so a query about the deadline's exact time might retrieve the chunk that is missing the number. Adding overlap keeps that one sentence whole in at least one chunk, at the cost of storing (and later reading) some tokens twice. Structure-based chunking, splitting on headings instead of a token count, keeps “Grading — Milestone: Week 6, Thu 11:59pm” and “Final report: Week 10” as two clean, complete facts, because it respects the document's own boundaries instead of a fixed count.

Three more strategies, each solving a different piece of this

Semantic chunking segments on logical boundaries the text already has, sentences, paragraphs, sections, rather than a token count, so a chunk never ends mid-idea by accident. Context-enriched chunking carries extra metadata or a short summary alongside each chunk (which document it came from, which section), so a reader downstream knows more about a chunk than the chunk's own words say. AI-driven dynamic chunking goes one step further: it asks a language model to find the document's natural breakpoints itself, rather than relying on a fixed rule. Chapter 3 takes the context-enriched idea and pushes it further still, by writing a whole sentence of context, generated by a model, into every single chunk before it is ever embedded.

Chapter 2 named the problem a small chunk has: its vector carries almost no context. Here is exactly how that bites. A query asks, “What was the revenue growth for ACME Corp in Q2 2023?” and the retriever's candidate chunk reads: “The company's revenue grew by 3% over the previous quarter.” Nothing in that sentence says the company is ACME, or that the quarter is Q2 2023. A retriever comparing embeddings has no way to know this chunk answers the query; the words that would prove it were left behind in some earlier chunk.

Contextual retrieval: write the context back in, with a model

Anthropic's contextual retrieval fixes this with one extra step at indexing time, before any query ever arrives. For every chunk, it sends the whole document plus that one chunk to a language model, with a prompt asking for a short, succinct sentence of context situating the chunk within the document, for the sole purpose of improving search. That generated sentence is prepended to the chunk before it gets embedded. The same sentence, “The company's revenue grew by 3%…,” becomes “From ACME Corp's Q2 2023 SEC filing; Q1 2023 revenue was $314M. The company's revenue grew by 3% over the previous quarter.” Now ACME and Q2 2023 are in the chunk's own words, and its embedding can finally match the query.

Anthropic pairs this with hybrid retrieval at the same step: the contextualized chunk is embedded for meaning, and also indexed by TF-IDF or BM25 (Chapter 6) for exact terms, and the two ranked lists are merged (Chapter 8 covers merging in full). Embeddings catch paraphrase and meaning; TF-IDF catches an exact product ID or an error code an embedding might blur past.

The result, measured as the average percentage of relevant chunks a top-20 retrieval fails to find, across the method's test suite: plain embeddings fail 5.7% of the time. Adding BM25 to plain embeddings brings that to 5.0%. Switching to contextual embeddings alone brings it to 3.7%. Contextual embeddings plus BM25 together bring it to 2.9%, roughly half the plain-embedding failure rate. Each ingredient, context and a second retriever, removes a different kind of miss.

Late chunking: let one long encoder carry the context instead

Contextual retrieval pays for its context with an extra language-model call per chunk, at indexing time, and a few dozen extra tokens stored per chunk forever. Late chunking (Günther et al., 2024) gets a similar benefit for a different price. Instead of embedding each chunk on its own, it feeds the whole document into a long-context encoder first, producing one contextual token vector for every token in the document, the way Lecture 2's attention lets every token see every earlier token. Only afterward does it decide where the chunk boundaries fall, and for each chunk, it mean-pools (averages) just the token vectors that fall inside that chunk's span.

The naming matters: ordinary chunking embeds each chunk first and chunks the document second (chunk, then encode); late chunking encodes the whole document first and chunks it second (encode, then chunk, hence “late”). Because every token's vector was computed while attending to the entire document, a chunk's pooled vector already reflects the rest of the document, with no extra text and no extra model call added to the chunk itself. Against contextual retrieval, the trade is exact: no LLM call and no added tokens per chunk, versus needing a long-context encoder that can see the whole document at once.

Measured by nDCG@10 (Chapter 7 defines this score) across several retrieval benchmarks (SciFact, NFCorpus, FiQA, TRECCOVID), late chunking's average score beats naive chunking's average at the same chunk boundaries: 54.0 versus 52.2 with fixed-size, 256-token boundaries, and 54.3 versus 52.4 with sentence boundaries. The gain is consistent but modest, because late chunking fixes exactly one failure, lost context, while leaving every other chunking decision (where the boundaries fall) unchanged.

Every fix so far still indexes one flat list of chunks. That is enough when the answer lives inside one chunk. It is not enough for a question like “What is the central theme of this story?”, because no single paragraph states a story's theme; the theme is a property of the whole document, built from many chunks at once. A single chunk's vector cannot represent a fact that only exists at the level of the whole collection.

RAPTOR: build a tree, bottom up

RAPTOR (Sarthi et al., 2024) answers this by indexing the document at several levels of summary at once, not just its raw chunks. The construction runs bottom up, and repeats: embed the leaf chunks, cluster chunks whose embeddings sit near each other (so clustering here means grouping similar vectors, not supervising anything), summarize each cluster with a language model into one short passage, then embed those summaries as a new, smaller layer, and repeat the whole process on that layer. The result is a tree: raw chunks as leaves, summaries of groups of chunks one level up, summaries of summaries above that, up to one or a few root nodes.

Retrieval then searches across every level at once, not just the leaves. A narrow, detail question (“how did she find a happy ending?”) tends to match a leaf or a low-level node, where the specific sentence lives. A thematic, whole-document question (“what is the central theme of the story?”) tends to match a high-level summary node, where many chunks have already been condensed into one idea. On the paper's own Cinderella example, RAPTOR's retrieved context for both questions usually already contains everything a flat retriever like DPR would have found at its leaves, either directly, as one of the same chunks, or indirectly, folded inside one of RAPTOR's summaries.

GraphRAG: build a graph instead of a tree

GraphRAG (Edge et al., 2024) takes a different structure to the same problem: query-focused summarization over a graph an LLM itself builds from the source text. The pipeline runs in stages: source documents are split into text chunks; a language model extracts entities and relationships from those chunks (people, places, organizations, and how they relate); those become a knowledge graph; a community-detection algorithm groups the graph into communities, tightly connected clusters of entities; and a language model writes a community summary for each one. A question is then answered by gathering the relevant community summaries and having a model write one global answer from them, rather than from any single chunk.

Where RAPTOR's layers come from clustering raw text embeddings, GraphRAG's communities come from the structure of named entities and their relationships, so GraphRAG tends to suit questions that are fundamentally about how things connect (“who worked with whom, and on what”), while RAPTOR suits questions that are fundamentally about a document's overall gist or structure. Later variants (Han et al., 2025) extend the same idea across text and image data together, and swap in different graph-building and retrieval organizers, but the shape, extract structure, summarize it, answer from the summaries, stays the same.

Every chapter so far asked what to index and how to shape it. This chapter asks the question underneath all of them: once you have N chunk vectors sitting in a database, and a new query vector arrives, how do you actually find the top-k closest ones, fast enough to answer in real time?

The honest answer, and why it does not scale

Exact search is a brute-force scan: compare the query vector against every one of the N stored vectors, each comparison costing d operations for a d-dimensional vector, for a total of O(N · d) work per query. This is exactly correct, every time, and for a few thousand chunks it is fast enough that nothing more is needed. It stops being fast enough once N reaches the millions a production corpus actually holds; scanning every vector for every query becomes the single slowest part of the whole pipeline.

The fix is approximate nearest neighbor search (ANN): trade a little recall, how often the true nearest neighbors actually appear in the returned top-k, for a lot of speed. Three structures do this in different ways. An inverted file index (IVF) first clusters the whole index into regions, then at query time probes only the handful of regions nearest the query, skipping everything else entirely. Product quantization (PQ) compresses each vector into a short code, so comparisons run on tiny compressed numbers instead of the full vector, at some cost in precision. HNSW builds a graph structure that lets a search hop its way to the answer instead of scanning anything.

HNSW: a multi-layer map, searched from the top down

HNSW, hierarchical navigable small world, organizes every stored vector into several stacked layers. Layer 0, the bottom, holds every single vector. Each layer above it holds only a thin, random sample of the vectors in the layer below, so the very top layer might hold only a handful of the whole collection. Think of it as a road map: the bottom layer is every street in a city, and each layer above is a coarser map, interstates only, then a single cross-country highway.

A search starts at the top, sparse layer. From wherever it currently sits, it takes a greedy hop to whichever neighbor is closer to the query than its current position, repeating until no neighbor at that layer is any closer, at which point it has gotten “stuck,” and it drops down one layer and resumes the greedy hop there, where the map is denser and the steps get finer. By the time it reaches layer 0, it is already close, and a short, precise search there finds the true nearest neighbors. This takes roughly log N hops in total rather than N comparisons, which is the entire source of HNSW's speed, and in practice it keeps high recall despite never looking at most of the collection.

How the graph gets built in the first place

HNSW's graph is built by inserting one vector at a time, never all at once. Each new vector is assigned a random top layer: the chance of reaching a high layer is deliberately small, so most vectors live only in layer 0, and only a thin, exponentially shrinking sample climbs higher, which is exactly what keeps the top layers sparse and the search fast. At every layer the new vector appears in, it is linked to its M nearest already-inserted neighbors (M is a fixed, chosen number of connections), plus one extra heuristic link deliberately placed across to a different, distant cluster. That cross-cluster link exists for a specific reason: without it, a greedy walk that starts inside one tight cluster of similar vectors can get trapped circling that cluster forever, and the heuristic link gives it an escape route to the rest of the graph.

Chapter 5 asked how to search an index fast. The next three chapters ask a different question: what should actually count as “relevant” in the first place? The lecture sorts every retriever into three families: word-overlap methods like BM25, which score a passage by which query words it literally contains; vector retrieval methods like DPR (Chapter 7), which score by meaning in embedding space; and neural, token-level methods like ColBERT (Chapter 8), which sit between the two. This chapter is the oldest and, by a wide margin, still the most used: BM25.

BM25: count the words that matter, and only up to a point

BM25 scores a document D against a query Q by summing a contribution from each query term q_i, built from two ideas: how rare that term is across the whole collection, and how often it actually shows up in this one document.

score(D,Q)

The final relevance score for document D against query Q: higher means more relevant, and the number itself only matters for ranking, not as a probability.

∑_i=1ⁿ

Add up one contribution per query term, from the first of n query terms to the last.

IDF(q_i)

Inverse document frequency: how rare term q_i is across the whole collection. A term almost every document contains (“the”) gets a low, even negative, weight; a term only a few documents contain gets a high weight.

f(q_i,D)·(k₁+1)

How often q_i occurs in D (its raw term frequency f), scaled by a constant k₁+1 that sets how steeply frequency can boost the score before the saturation term below caps it.

[… + k₁·(1−b+b·|D|/avgdl)]

The denominator: it divides down the frequency term, with the ratio |D|/avgdl (this document's length over the collection's average length) scaled by b. A long document that merely repeats a word many times because it is long, not because it is relevant, gets discounted.

The IDF term itself has its own formula: IDF(q_i) = ln[(N − n(q_i) + 0.5) / (n(q_i) + 0.5)], where N is the total number of documents in the collection and n(q_i) is how many of them contain q_i at all. A term that appears in almost every document (n close to N) pushes the fraction inside the logarithm toward a small number, and for an extremely common term it can even go negative, actively penalizing a document for containing a word so common it carries no information.

The frequency term is where BM25 earns its name: term frequency saturation. A document mentioning the query term once is clearly more relevant than one mentioning it zero times, but a document mentioning it 50 times is not 50 times more relevant than one mentioning it once; after a point, more repetitions tell you almost nothing new. The k₁·(…) term in the denominator enforces exactly this curve: the score rises quickly at first and then flattens, saturating, as frequency keeps climbing.

A second graph from the lecture compares BM25's own IDF curve against a plain, unsmoothed IDF: BM25's version stays better-behaved across the full range of document frequency, including a defined (if negative) value for terms that appear in almost every document, where a naive formula can blow up toward infinity. The same figure then overlays saturation curves at several k₁ values (0.5, 1.2, 2.0, 5.0): a low k₁ saturates almost immediately, so repetitions beyond the first one or two add nearly nothing, while a high k₁ keeps rewarding extra repetitions for much longer before flattening.

Why does a formula this old still matter, when dense retrieval exists? Chapter 7's own numbers answer this directly: BM25 is a fixed, non-learned module, needs no training data at all, runs in tens of milliseconds, and it is unbeatable at the one thing embeddings are weakest at, matching an exact literal string, a product ID, an error code, a name, that happens not to resemble anything in the training data an embedding model saw.

BM25 scores by matching literal words. Dense passage retrieval (DPR, Karpukhin et al., 2020), the retriever Chapter 1's RAG paper actually used, scores by meaning instead: it trains two separate BERT encoders so that a query and the passage that answers it land near each other in the same vector space, even when they share almost no words in common.

Two encoders, trained to agree

DPR keeps a passage encoder E_P and a query encoder E_Q, both BERT models, but kept separate so their jobs can specialize. Every passage in the collection is run through E_P exactly once, offline, before any query arrives, producing one fixed vector per passage, stored in a vector index. At query time, only the query is run through E_Q, online, and that one new vector is compared, by a simple dot product, against every stored passage vector, returning the top-k closest.

p(z|x) ∝ exp(d(z)^Tq(x)), d(z) = BERT_d(z), q(x) = BERT_q(x): this is the exact formula Chapter 1 already showed, and now you can see why splitting the two encoders apart matters for engineering, not just math. Because passage vectors never depend on the query, they are computed once and reused for every future query forever; only the cheap query encoding has to run live.

Training pushes a query's vector toward its one true passage and away from everything else, using a contrastive loss: for query q_i and its true passage p_i⁺, minimize −log of exp(sim(q_i,p_i⁺)) divided by the sum of exp(sim(q_i,p)) over p⁺ and every negative passage in the batch. The negatives come from two sources: in-batch negatives, simply every other training example's true passage, which are free to collect, and BM25 hard negatives, passages BM25 itself rated highly for this query but which are not actually the right answer, forcing the model to learn distinctions a weaker, word-overlap retriever cannot make.

Measuring whether a retriever actually works

Before looking at DPR's results, the metrics that produced them need defining, because every number in this chapter and the next is one of these five.

Hit@k

Averaged over every query q in the set Q: 1 if any relevant passage lands in the top k, 0 otherwise. Position inside the top k does not matter at all; Failed@k is just 1 − Hit@k, the fraction of queries with zero relevant passages anywhere in the top k.

nDCG@k

Normalized discounted cumulative gain: it rewards relevant passages that rank near the top of the k results more than ones buried near the bottom, then divides by the best possible ordering's score (IDCG) so the final number sits between 0 and 1.

MRR@k

Mean reciprocal rank: for each query, take 1 divided by the position of the first relevant passage (so rank 1 scores 1.0, rank 4 scores 0.25), counted as 0 if nothing relevant appears in the top k, then average.

Recall@k

What fraction of all the relevant passages for a query (not just whether at least one appeared) were actually found in the top k.

Did it actually beat BM25?

Trained with just 1,000 question-answer pairs and no extra pretraining beyond ordinary BERT, DPR already beats BM25's top-20 retrieval accuracy, and matches or beats two heavier pretrained systems, ORQA and REALM, trained end to end. Measured on top-20 passage retrieval accuracy across five datasets, DPR wins on four of five: 78.4 versus BM25's 59.1 on Natural Questions, 79.4 versus 66.9 on TriviaQA, 73.2 versus 55.0 on WebQuestions, 63.2 versus 68.8 on SQuAD (DPR's one loss, where BM25's exact word matching suits SQuAD's artificially worded questions). Combining both, BM25 plus DPR, beats either alone on TREC (85.2) and SQuAD (71.5), previewing Chapter 8's hybrid search: the two retrievers' strengths do not fully overlap, so merging them recovers ground either one gives up alone.

Chapter 7 closed on a fact worth sitting with: BM25 and DPR each win roughly where the other loses. BM25 misses paraphrase (different words, same meaning) and dense retrieval misses rare literals (an error code, a product ID, a name an embedding model never learned to treat as special). Neither failure is a bug to fix; it is what each method is built to do. The practical answer is hybrid search: run both retrievers on every query, and fuse their two ranked lists into one.

Fusing ranks, not scores

BM25's scores and DPR's cosine similarities live on entirely different, incomparable scales, so averaging them directly is meaningless. Reciprocal rank fusion (RRF) sidesteps the problem by throwing away the raw scores and fusing positions instead: RRF(d) = ∑_r∈R 1/(k+rank_r(d)), summing, over every retriever r in the set R, the reciprocal of that document's rank in retriever r's list, with a constant k = 60 that softens how much rank 1 dominates rank 2. A document that both retrievers rank highly accumulates a large score from both sums; a document only one retriever found at all still gets some credit. This needs no per-corpus tuning, and the same formula works whether you are fusing two retrievers or five.

Reranking: pay more, but only for a short list

A bi-encoder, DPR's shape, encodes the query and each passage completely separately, which is exactly what makes precomputing every passage vector offline possible. A cross-encoder encodes the query and a candidate passage together, in one pass, letting every query word attend to every passage word directly (Humeau et al., 2020). This full interaction makes a cross-encoder far more accurate at judging one specific pair, but it means nothing can be precomputed; every query-passage pair needs a fresh forward pass. The standard pattern is therefore a pipeline: a cheap bi-encoder or BM25 narrows millions of passages down to a short list, and a cross-encoder reranks only that short list, paying its higher cost on a few dozen candidates instead of the whole collection.

Late interaction: ColBERT's middle path

ColBERT (Khattab & Zaharia, 2020) asks whether there is a third option between a bi-encoder's one-vector-per-passage (fast but coarse) and a cross-encoder's full joint attention (precise but slow). Its answer is late interaction: keep one vector per token, not one per passage, encoding the query and the document completely independently (so document token vectors can still be precomputed offline), and let them interact only at scoring time, through a specific operation called MaxSim.

S_q,d

The final relevance score between query q and document d, the number ColBERT ranks documents by.

∑_i∈q

Add up one contribution for every token in the query, so a 6-token query makes 6 separate contributions.

max_j q_i·d_j

For this one query token q_i, find whichever single document token d_j it matches best (the highest dot product over all j), and keep only that best match, discarding every other document token's score for this query token.

In words: every query token gets to pick its own single best-matching word anywhere in the document, and the final score adds up each query token's best match. A query token that matches nothing well drags the sum down; a query token with a strong match anywhere in a long document, even buried far from the other matches, still contributes fully. This is what “token-level evidence at scale” means: ColBERT can see that a document contains the right words in the right senses, spread across the document, without ever needing one single vector to summarize the whole passage at once.

The cost of all this token-level detail is index size: ColBERT stores one vector per token, not per passage, so its index is far larger than DPR's. The lecture's own latency comparison, measured on MS MARCO, lines the four methods up on exactly this trade-off. BM25 (sparse, an inverted index, fully precomputed offline): 62ms, wins on exact terms like IDs and error codes, breaks on paraphrase and synonyms. DPR (bi-encoder, one vector per passage): tens of milliseconds, wins on semantic match at corpus scale, breaks on rare literals and needs labeled training pairs. ColBERT (late interaction, one vector per token): 458ms end-to-end, in the middle on speed, wins on token-level evidence at scale, breaks on index size since it stores tokens, not documents. A cross-encoder (BERT-base, nothing precomputed): 10,700ms per 1,000 passages, by far the slowest, wins on accuracy once you already have a short shortlist, cannot scan a whole corpus at all. The right system is rarely one of these alone; it is BM25 or DPR (or both, fused) narrowing the field, with ColBERT or a cross-encoder reranking what survives.

Every retriever so far answers one question well, once, with however many passages you asked for. None of them decides whether retrieval was worth doing at all, what exact words to search for, or whether the passages that came back are actually good enough to answer from. Agentic RAG puts retrieval inside an agent's loop (Lecture 2's agent patterns) as one action among several, governed by three decisions the lecture names directly: retrieve, or answer directly from what the model already knows; what query to issue, since the user's literal words are not always the best search query; and whether the retrieved result is good enough to stop on, or whether to try again.

Self-RAG: retrieve, generate in parallel, then critique

Self-RAG (Asai et al., 2024) trains a single model to make all three decisions itself, using special tokens it learns to emit. Facing a question, it first decides whether to retrieve at all. If it retrieves, it generates several candidate continuations in parallel, one per retrieved passage, then critiques its own outputs, scoring each for relevance and factual support, and selects the best-supported segment to keep. The paper's own example, “How did US states get their names?”, shows the model retrieving, drafting several candidate explanations grounded in different passages, and keeping the one its own critique rates highest for overall generation quality, factuality and verifiability, the three properties the paper explicitly trains it to self-assess.

CRAG: let a critic grade the retrieval itself

CRAG, corrective RAG (Yan et al., 2024), separates the retrieval decision into its own explicit step: a trained critic evaluates the retrieved documents against the query and estimates a confidence, sorting the result into one of three buckets, correct, ambiguous, or incorrect, each triggering a different action. A correct result is used as retrieved. An incorrect result triggers a fresh search (the paper uses a web search) instead of trusting the bad passage. An ambiguous result triggers knowledge refinement: decomposing the passage into smaller pieces, re-filtering them, and recombining only the parts the critic still trusts. Where Self-RAG bakes the decision into one model's own tokens, CRAG makes it an explicit, separately gradeable step that can be inspected and tuned on its own.

Search-R1: learn the whole policy with reinforcement learning

Self-RAG and CRAG are each trained with their own supervised signal (critique tokens, a graded critic). Search-R1 (Jin et al., 2025) instead trains the entire multi-turn search policy with reinforcement learning, the same family of method Lecture 2's GRPO uses for verifiable answers. The model reasons inside explicit <think> and <search> tags, issuing as many search calls as it decides it needs, reads each result back inside <information> tags, and only commits to a final <answer> once it is satisfied, with no limit fixed in advance on how many searches it may issue.

Trained this way on Qwen2.5-7B, Search-R1 clearly beats fixed, non-learned alternatives across six QA benchmarks (Natural Questions, TriviaQA, PopQA, HotpotQA, 2WikiMultiHopQA, Musique): Search-R1-base scores 0.480 on Natural Questions and 0.638 on TriviaQA, well above plain retrieval-augmented generation with no learned search policy (0.349 and 0.585 on the same two) and far above direct inference with no retrieval at all (0.134 and 0.408). The gain comes specifically from letting the model learn when and how often to search, rather than searching a fixed number of times regardless of the question's difficulty, which is exactly the third decision this chapter opened with: whether the evidence gathered so far is good enough to stop on.

Every chapter so far built one piece of a RAG pipeline. This closing chapter does the opposite: it takes the lecture's own list of nine named failures and sorts each one into the one place it actually comes from, retrieval, the reader, or the loop around both, because the fix for each failure lives exactly where its cause does.

Retrieval-side: the index itself is the problem

A recall ceiling is the simplest failure: the relevant passage was never in the collection, or never made it into the top-k, so no amount of clever reading can recover it. Symptom: the model answers confidently wrong, or declines to answer, on a question whose evidence exists somewhere in the corpus. Measurement: Chapter 7's Recall@k, computed on a held-out set of queries with known relevant passages, directly reports this ceiling. Fix: a better retriever, hybrid search (Chapter 8), or simply a larger k, whichever the measurement shows is actually the bottleneck.

A stale index happens when the underlying documents change but the embeddings were never refreshed, so the retriever confidently returns last month's answer to a question about today. Symptom: an answer that was correct once and is now wrong, with no error anywhere in the model. Measurement: compare retrieved passage timestamps against the query's intent; a gap signals staleness directly. Fix: a re-indexing schedule tied to how often the source documents actually change, not a fixed calendar interval chosen without checking.

A permission leak is a retriever returning content the asking user was never allowed to see, because the index itself does not encode access control. Symptom: a correct, well-grounded answer that should never have been given to this particular user. Measurement: an access audit on retrieved passages against the requester's permissions, not a quality metric at all. Fix: filter candidates by permission before ranking, inside the retriever, never as an afterthought on the generated answer.

Reader-side: the passages are fine, the reading of them is not

Lost in the middle, also called position bias, is a reader paying less attention to passages placed in the middle of a long context than to ones at the very start or end, regardless of which passage is actually most relevant. Symptom: the right passage was retrieved, sits in the context, and the model still misses it. Measurement: hold the retrieved set fixed and vary only where the correct passage sits in the ordering; accuracy that swings with position, not content, confirms it. Fix: put the most relevant passages at the edges of the context, or rerank (Chapter 8) so the best evidence is never buried in the middle.

Distraction is similar but opposite in cause: passages that are superficially similar to the query, enough to rank well, but actually irrelevant, pull the reader's attention away from the passage that truly answers it. Symptom: an answer that blends in details from a near-miss passage. Measurement: swap in a deliberately similar-but-wrong distractor passage and watch accuracy drop. Fix: a reranker (Chapter 8) or a tighter relevance threshold that drops near-misses instead of passing every retrieved passage straight to the reader.

Evidence conflict arises when retrieved sources disagree, or disagree with the model's own training-time knowledge, and the reader has no principled way to pick a side. Symptom: an answer that quietly picks one source's number over another's with no acknowledgment that a conflict existed at all. Measurement: deliberately retrieve two passages with contradictory facts and check whether the answer flags the conflict or silently resolves it. Fix: surface the conflict explicitly in the answer, with both sources cited, rather than silently resolving it.

Loop-side: the problem is in the agent's control flow, not any single step

Compounding error is a mistake made early in a multi-step agentic process, Chapter 9's territory, shaping every decision that follows it, so a small early slip becomes a large final one. Symptom: a final answer far wrong in a way no single step looks obviously broken. Measurement: trace the step where the trajectory first diverges from a correct one; errors compound forward from there, never backward. Fix: checkpoints that re-verify intermediate conclusions against retrieved evidence before building further steps on top of them.

No stopping rule leaves an agent without a clear condition for when it has gathered enough evidence to answer, so it either stops too early, on thin evidence, or searches indefinitely. Symptom: wildly inconsistent numbers of search calls for equally hard questions. Measurement: track search-call counts against question difficulty; Search-R1's learned policy (Chapter 9) exists specifically to fix this, by learning the stopping decision instead of fixing it by hand. Fix: an explicit, trained or hand-specified confidence check before the agent is allowed to stop.

Prompt injection is retrieved text that contains instructions of its own, written by whoever controls that document, and the model follows them as if they came from the user. Symptom: the agent's behavior changes in a way no part of the actual conversation asked for. Measurement: plant an instruction inside a test document and check whether the agent obeys it. Fix: treat every retrieved passage as data to read, never as instructions to execute, the same boundary Chapter 9's CRAG critic exists to help enforce before a passage ever reaches the generator.

Every RAG system, however it is built, is still Chapter 0's two functions: a retriever that narrows a whole collection down to a handful of passages, and a reader that writes an answer from them. Everything in between is choices: how you chunk the collection (Chapters 2-4), how you search the index (Chapters 5-8), and how much control you hand to an agent deciding when to retrieve at all (Chapter 9). Chapter 10's nine failures are the reminder that each of those choices can fail independently, and the fix for each lives exactly where its cause does.

Here is Chapter 0's retriever-reader pair, written as runnable code, with a BM25-style score standing in for the retriever (Chapter 6) and a dense top-k search (Chapter 5) commented alongside it, since a real system typically fuses both (Chapter 8):

python
def retrieve(query, chunks, k=5):
    # chunks: list of (text, bm25_score_fn, embedding) built at indexing time
    bm25_ranked = sorted(chunks, key=lambda c: c.bm25_score(query), reverse=True)
    dense_ranked = sorted(chunks, key=lambda c: c.embedding.dot(embed(query)), reverse=True)
    # hybrid: reciprocal rank fusion, k=60 (Chapter 8)
    scores = {}
    for rank, c in enumerate(bm25_ranked):  scores[c] = scores.get(c, 0) + 1 / (60 + rank + 1)
    for rank, c in enumerate(dense_ranked): scores[c] = scores.get(c, 0) + 1 / (60 + rank + 1)
    return sorted(scores, key=scores.get, reverse=True)[:k]

def answer(query, chunks):
    passages = retrieve(query, chunks, k=5)
    context = "\n\n".join(p.text for p in passages)
    return generate(prompt=f"Context:\n{context}\n\nQuestion: {query}\nAnswer:")

Where this sits in the field: RAG was Chapter 1's answer to one problem, a model that cannot know what it was never trained on; everything after that chapter is the engineering of making retrieval actually good, fast, and robust at real scale. The rest of CS 329Z builds on exactly this foundation: an agent with memory, tool use, and multi-agent coordination is, underneath, a system that decides what to retrieve, when, and from where, at every single step it takes.

Keep going: our RAG paper Veanor walks Lewis et al.'s full paper chapter by chapter; our ColBERT Veanor does the same for late interaction; our RAG Gleam builds a minimal pipeline from zero in more depth than this lecture had time for; our vector databases lesson goes deeper into HNSW, IVF and PQ; and multimodal RAG extends every idea here to retrieval over images.

Go back to the top and try every retriever on the same question again, now that you have watched the pipeline, the index, and nine ways it can still fail. Then try Present or Teach on this lesson: explain to someone else why a RAG system with a perfect retriever can still answer wrong, and why a RAG system with a mediocre retriever but a good reranker often beats it.