Retrieval-Augmented
Generation

Chapter 1: The Pipeline

RAG is not one algorithm — it's a pipeline of six steps. Each step has its own literature, trade-offs, and failure modes. Understanding the pipeline as a whole is the prerequisite for fixing any single part of it.

Chapter 2: Indexing

Before you can retrieve anything, you need to turn raw documents into searchable chunks with semantic vector representations. This is the indexing pipeline: load → chunk → embed → store.

1. Document Loading

Documents come in many formats: PDF, DOCX, HTML, Markdown, CSV, database rows. Each needs a parser that extracts clean text while preserving structure cues (headings, table relationships). Libraries like LangChain's DocumentLoader and LlamaIndex abstract this, but the output quality depends entirely on parser quality. A PDF with scanned images needs OCR; a DOCX with embedded tables needs special handling.

2. Chunking

You can't embed a 100-page manual as one unit — the vector would average over everything and represent nothing specific. Chunking splits documents into semantically coherent passages, each independently embeddable. The two main strategies are:

3. Embedding

An embedding model maps text to a dense vector where semantically similar texts are close in cosine distance. Common choices: OpenAI text-embedding-3-small (1536-dim, API), all-MiniLM-L6-v2 (384-dim, local, fast), bge-large-en (BAAI, strong retrieval-focused model).

python
# Offline indexing pipeline
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_openai import OpenAIEmbeddings
from langchain_community.vectorstores import Chroma

# 1. Chunk — 512 tokens, 50-token overlap
splitter = RecursiveCharacterTextSplitter(
    chunk_size=512, chunk_overlap=50, separators=["\n\n", "\n", " "]
)
chunks = splitter.split_documents(raw_docs)  # list[Document]

# 2. Embed + 3. Store — all in one call with Chroma
vectorstore = Chroma.from_documents(
    documents=chunks,
    embedding=OpenAIEmbeddings(model="text-embedding-3-small"),
    persist_directory="./chroma_db"
)
# Each chunk is now: { text, metadata, embedding_vector }

Chapter 3: Retrieval

At query time, you have a user question and a vector store with millions of embedded chunks. Retrieval means: find the K chunks most semantically relevant to this query in milliseconds. This is harder than it sounds.

Query Embedding

The user's query is embedded with the same model used to embed the documents. This is critical — if you indexed with text-embedding-3-small, you must retrieve with text-embedding-3-small. Different models produce incompatible vector spaces.

Approximate Nearest Neighbor (ANN)

Brute-force cosine similarity over 10 million vectors takes seconds. ANN algorithms (HNSW, IVF, ScaNN) trade a tiny accuracy loss for 100-1000x speedup by partitioning the vector space and only searching nearby regions. HNSW (Hierarchical Navigable Small World) is the default in most vector stores — it builds a navigable graph where each node links to its nearest neighbors at multiple granularities.

Dense vs. Sparse vs. Hybrid

python
# Hybrid retrieval: dense + sparse, fused with RRF
from langchain_community.retrievers import BM25Retriever
from langchain.retrievers import EnsembleRetriever

bm25 = BM25Retriever.from_documents(chunks, k=10)
dense = vectorstore.as_retriever(search_kwargs={"k": 10})

# EnsembleRetriever applies RRF internally
hybrid = EnsembleRetriever(
    retrievers=[bm25, dense],
    weights=[0.4, 0.6]  # tune based on your domain
)
results = hybrid.invoke("What is the refund policy?")
# returns: list[Document] sorted by fused relevance

Chapter 4: Reranking

You retrieved the top-20 chunks with fast ANN search. Fast is good, but ANN uses a bi-encoder: the query and each document are encoded independently. That means the model never sees them side-by-side. It's like scoring job candidates by their LinkedIn headline alone.

A cross-encoder reranker does something slower but smarter: it takes the query and a single document concatenated together — "[CLS] query [SEP] document [SEP]" — and outputs a single relevance score. The model can attend across both texts simultaneously. It's like reading the candidate's full application together with the job description.

python
# Cross-encoder reranking with sentence-transformers
from sentence_transformers import CrossEncoder

reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")

# retrieved: list[Document] from the bi-encoder step (top-20)
pairs = [(query, doc.page_content) for doc in retrieved]
scores = reranker.predict(pairs)  # shape: (20,) — relevance score each

# Sort by score descending, keep top-5 for context
ranked = sorted(zip(scores, retrieved), reverse=True)
top_docs = [doc for _, doc in ranked[:5]]

Chapter 5: Generation

You have the top-K retrieved and reranked chunks. Now you need to turn them into a grounded answer. This is where the LLM does what it's actually good at: reading, synthesizing, and writing.

Prompt Construction

The prompt has three parts: a system instruction (your persona, tone, and citation rules), the retrieved context (the chunks, each labeled with a source), and the user query. The context goes between the system and the query so the model reads it before the question.

python
def build_prompt(query: str, docs: list[str], sources: list[str]) -> str:
    context_block = ""
    for i, (doc, src) in enumerate(zip(docs, sources)):
        context_block += f"[{i+1}] ({src})\n{doc}\n\n"

    return f"""You are a helpful assistant. Answer ONLY using the provided context.
If the context does not contain the answer, say "I don't have enough information to answer that."
Cite sources as [1], [2], etc.

CONTEXT:
{context_block}
QUESTION: {query}

ANSWER:"""

Context Window Budget

Every token in the context costs: latency, money, and attention. The LLM has a fixed context window (4K to 128K tokens). You must fit: the system prompt (~200 tokens), retrieved chunks (~300 tokens each × K), the query (~50 tokens), and leave room for the answer (~500 tokens). With a 4K window, you can fit roughly 10 chunks of 300 tokens each.

"I Don't Know" is an Answer

A RAG system that makes up an answer when retrieval fails is worse than no system at all. Explicitly instruct the model: if the retrieved context does not contain sufficient information, say so. You can reinforce this with a threshold: if no chunk scores above 0.5 relevance, decline to answer and return a "not found" response rather than hallucinating.

Chapter 6: Evaluation

How do you know if your RAG system is good? "It seems to work" is not an answer. You need metrics that measure the three things that can go wrong independently: retrieval might fetch the wrong chunks, the model might ignore the right chunks, and the answer might not address the question.

The Three RAGAS Metrics

RAGAS (Retrieval Augmented Generation Assessment) is the standard evaluation framework. It decomposes quality into three orthogonal scores, each measured by an LLM judge:

python
from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_recall
from datasets import Dataset

# Build evaluation dataset
eval_data = {
    "question": ["What is the refund window?"],
    "answer": ["Enterprise licenses have a 30-day refund window."],
    "contexts": [["Refunds: Enterprise: 30 days. Consumer: 14 days."]],
    "ground_truth": ["30 days for enterprise licenses"]
}
ds = Dataset.from_dict(eval_data)

result = evaluate(ds, metrics=[faithfulness, answer_relevancy, context_recall])
# result: { faithfulness: 1.0, answer_relevancy: 0.97, context_recall: 1.0 }

Chapter 7: Failure Modes

RAG fails in ways that are specific, diagnosable, and fixable — but only if you know what to look for. Here are the five most common failure modes, each with a distinct signature in your RAGAS scores.

Chapter 8: Advanced RAG

Once you've got vanilla RAG working, there are four research-backed extensions that each solve a specific limitation. None of them are mandatory — add them when the vanilla system fails in the specific way they fix.

HyDE — Hypothetical Document Embedding

Problem: query embeddings and document embeddings live in slightly different parts of vector space (queries are short, documents are long). Solution: use the LLM to generate a hypothetical answer to the query, then embed that hypothetical answer for retrieval. Hypothetical answers are document-shaped — so they match real documents better.

python
# HyDE: embed a hallucinated answer, not the query
hypo_answer = llm.invoke(f"Write a short passage that answers: {query}")
query_vector = embed(hypo_answer)  # embed the hypothesis, not the query
results = vectorstore.similarity_search_by_vector(query_vector, k=10)

RAPTOR — Recursive Abstractive Processing for Tree-Organized Retrieval

Problem: individual chunks lack global context. A chunk about a product feature doesn't know it's part of a larger enterprise offering. Solution: cluster similar chunks, generate LLM summaries of each cluster, embed the summaries, and add them to the index. Build this tree recursively. Now you can retrieve at multiple granularities — fine-grained chunk OR high-level summary depending on query type.

Agentic RAG

Problem: one retrieval step is often insufficient. A question may require looking up three separate facts, combining them, then verifying. Solution: give an LLM agent the vector store as a tool. The agent decides when to retrieve, what query to use, and can iterate (retrieve → read → decide to retrieve again with a refined query). This is RAG inside a ReAct loop.

Self-RAG

Self-RAG trains the LLM itself to decide when retrieval is necessary, evaluate the retrieved chunks' relevance, and critique its own generated answer for faithfulness — all using special [Retrieve], [Relevant], [Supported] tokens inserted during fine-tuning. The model becomes its own RAG controller.

Chapter 9: Interactive RAG Pipeline

This is the full system in action. Type a query, watch it flow through retrieval, reranking, and generation. Adjust top-K and toggle the reranker to feel the difference each component makes.

Chapter 10: Connections

RAG is not a terminal architecture — it's the retrieval layer in a larger system. Here's how it connects to adjacent ideas you'll encounter as you go deeper.

RAG → Agents

In a ReAct agent, the vector store is just another tool: search_knowledge_base(query). The agent decides when to call it, can call it multiple times with refined queries, and can combine retrieved facts with other tool outputs (web search, code execution). This is agentic RAG — the retrieval is no longer a fixed pipeline step but a dynamic capability.

See: Agents & Tool Use

RAG → Multimodal

Text is not the only modality you might want to retrieve. Multimodal RAG indexes images, PDFs with figures, audio transcripts, and video frames. Embedding models like CLIP embed images and text into the same space — enabling "retrieve images relevant to this text query." The pipeline is identical; only the embedding model and the chunk definition change.

RAG → Enterprise Production

Deploying RAG at enterprise scale introduces new concerns: access control (who can see which documents?), incremental indexing (re-embed changed docs without re-indexing everything), multi-tenant isolation (company A cannot retrieve company B's data), and observability (log every retrieval + generation for debugging and compliance). These are engineering problems, not ML problems — but they're what makes or breaks a production system.