Train a tiny KV cache offline that replaces 100K tokens of in-context learning. 38.6x less memory, 26.4x higher throughput — no quality loss.
You are building a coding assistant. A developer pastes their entire repository — 80,000 tokens of source code — into the prompt, then asks: "Where is the authentication bug?" The model reads everything, reasons over it, and gives a perfect answer.
Now a hundred developers do the same thing, each with their own codebase. Every single one needs a full KV cache — the key-value pairs that attention computes over the context. For LLaMA 8B at 128K tokens, that cache alone is ~10 GB per user. On a single H100 with 80 GB of VRAM, you can serve maybe 4–6 users simultaneously.
And it gets worse. As the context grows, throughput collapses. LLaMA 8B's peak throughput drops by 77x when increasing context from 1K to 120K tokens. Long context is not just expensive — it is the primary bottleneck of modern LLM serving.
What if you could pre-compute a much smaller KV cache that captures everything the model needs to know about a corpus? Train it once offline, store it, and load it for every future query — no matter what the user asks. That is exactly what a Cartridge is.
Drag the slider to increase context length. Watch how KV cache memory grows linearly — and how a Cartridge stays flat.
Here is the observation that makes Cartridges possible: in most real deployments, many different queries reference the same corpus. A legal team sends dozens of questions about the same filing. Developers ask different questions about the same codebase. A doctor queries the same patient record repeatedly.
If the corpus is shared, the cost of preparing its representation can be amortized across all future queries. Standard ICL computes the KV cache from scratch on every request (or caches it, but still stores the full thing). Cartridges take a different path: spend more compute once, offline, to train a much smaller KV cache that captures the corpus's information.
But there's a catch. You can't just train the small KV cache by having the model predict the next token of the raw corpus. That approach memorizes the text but fails to answer diverse questions. The solution — Self-Study — is what makes Cartridges actually work. We'll get there in Chapter 4.
A Cartridge is a trained set of key-value vectors that you prepend to the model's KV cache at inference time. Think of it as a virtual prefix — the model "sees" it as if those tokens were at the start of the context, even though they were never real tokens. The model's own weights are completely frozen; only the Cartridge's key-value vectors are trained.
For a Transformer with L layers, a Cartridge Z has shape:
Where L is the number of layers, 2 is for keys and values, p is the number of virtual tokens (the Cartridge size — typically 128 to 8192), and d is the head dimension times the number of heads. The memory footprint is exactly that of a KV cache for a prompt with p tokens.
In standard ICL, the KV cache looks like this:
The first nC pairs are the corpus. The rest are the query. With a Cartridge, you replace those nC pairs with just p trained pairs:
Random initialization of the Cartridge's key-value vectors leads to unstable training and poor results (29.9% accuracy on LongHealth vs. 55.3% with proper init). The authors initialize Z with the KV cache of the first p tokens of the corpus. This gives the optimizer a sensible starting point — real key-value vectors from actual tokens — and converges much faster.
Drag the slider to set corpus length. The visualization shows the KV cache layout for ICL (top) vs. Cartridge (bottom). Notice how the Cartridge stays small.
The most obvious way to train a Cartridge is next-token prediction (NTP) on the raw corpus. You freeze the model, prepend the trainable KV cache Z, feed in the corpus tokens, and backpropagate the language modeling loss into Z's key-value vectors. This is essentially prefix-tuning with the corpus text as the training objective.
The result? The Cartridge memorizes perfectly. Given the trained Cartridge, the model can regurgitate the corpus with near-zero perplexity. It uses 107x less memory than ICL while doing so. Sounds great, right?
Wrong. Try asking a question the Cartridge wasn't trained on — "Summarize this filing" or "Compute the D&A margin for FY15" — and performance collapses. The NTP-trained Cartridge is a one-trick pony: it can complete the corpus text, but it cannot reason about the corpus.
On the GenConvo benchmark (diverse questions about AMD's 10-K filing), the NTP-trained Cartridge matches ICL only on "memorization" queries (complete-the-passage tasks). On factual questions, synthesis, creative tasks, and mathematical reasoning, it falls far behind.
| Query Type | ICL | NTP Cartridge | Self-Study Cartridge |
|---|---|---|---|
| Memorization | Good | Good | Good |
| Factual Q&A | Good | Poor | Good |
| Synthesis | Good | Poor | Good |
| Creative | Good | Poor | Good |
| Math reasoning | Good | Poor | Good |
| Data structuring | Good | Poor | Good |
Self-Study is the training recipe that makes Cartridges actually work. The idea is beautifully simple: instead of training on the raw corpus, make the model quiz itself about the corpus and train the Cartridge on those synthetic conversations.
The corpus C may be hundreds of thousands of tokens — far longer than the context window. So we first split it into chunks of 512–4096 tokens. Each chunk becomes a subcorpus that fits in context.
We randomly sample one of five seed prompt types. These are generic — the same five prompts are used for every corpus, whether it's medical records, legal filings, or code:
Two participants A and B (both are the same model) have a conversation. A gets the subcorpus plus the seed prompt in context. A generates a message (e.g., a question). B gets the subcorpus plus A's message and generates a response. This produces a synthetic query-response pair grounded in the corpus.
algorithm # Self-Study Data Generation Input: C (corpus), F (model) Output: {a1, b1, ..., ak, bk} (conversation) c_tilde = chunk(C) # get subcorpus that fits in context s = get_seed_prompt() # random from {structuring, summary, ...} for i = 1 to k: ai ~ F(. | c_tilde + s + a1 + ... + bi-1) # A asks with seed bi ~ F(. | c_tilde + a1 + ... + ai) # B answers with corpus return {a1, b1, ..., ak, bk}
We don't just do next-token prediction on the synthetic conversations. Instead, we use context distillation (covered in the next chapter). The teacher is the model with the full subcorpus in context. The student is the model with only the Cartridge. We minimize the KL divergence between their output distributions.
Click "Generate" to simulate the Self-Study pipeline. Watch how the corpus is chunked, a seed prompt is selected, and a synthetic conversation is generated. Each "Step" button advances one phase.
Self-Study generates synthetic conversations. But how do we train the Cartridge on them? The answer is context distillation — a technique from the model distillation literature adapted to this setting.
The teacher is the model F with the full subcorpus in context. Given a synthetic conversation x and the subcorpus that generated it, the teacher computes:
The student is the same model F augmented with the Cartridge Z (no corpus in context). It computes:
The teacher "knows" the answer because it has the actual text. The student must learn to produce the same answers using only the Cartridge.
We minimize the KL divergence between the teacher's and student's next-token distributions, summed over all positions in the synthetic conversation:
Only Z (the Cartridge's key-value vectors) is updated. The model's weights are completely frozen.
This is worth emphasizing: the only trainable parameters are the key-value vectors in Z. That is p × d values per layer, per head, for both keys and values. For a Cartridge with p=2048 on LLaMA 8B (32 layers, GQA with 8 KV heads, dh=128):
That is 134 million parameters — about 1.7% of LLaMA 8B's 8 billion parameters. And these parameters are stored as a KV cache, so existing inference servers already know how to manage them.
The teacher (with corpus) and student (with Cartridge) produce distributions over the vocabulary. Drag the slider to adjust how well-trained the Cartridge is. Watch the KL divergence shrink as the student's distribution approaches the teacher's.
The authors evaluate Cartridges on three challenging long-context benchmarks, each pairing a large corpus (100K–484K tokens) with diverse queries:
On LongHealth and QASPER with LLaMA 3B, Cartridges trained with Self-Study match or exceed ICL quality while using dramatically less KV cache memory. Compression baselines like DuoAttention degrade rapidly beyond 2x compression. Cartridges achieve 10–100x compression without quality loss.
On MTOB, the full Kalamang textbook is 484K tokens — far beyond LLaMA 8B's 128K window. ICL simply cannot use the full book. But Self-Study's chunking strategy lets a Cartridge train on the entire 484K text. The resulting Cartridge outperforms ICL on the first 130K tokens by 11.0 chrF points and matches ICL on the hand-curated 60K subset.
Two independently trained Cartridges (e.g., one for AMD's 10-K, one for Pepsi's 10-K) can be concatenated at inference time. The model can then answer questions requiring information from both documents — without any joint training. This composition outperforms both single-Cartridge and truncated-ICL baselines on multi-document questions.
Compare methods across different KV cache sizes. Cartridges maintain quality at much smaller cache sizes, while compression baselines degrade rapidly.
Cartridges enable a new class of context-aware applications where the corpus is large but relatively static, and many queries reference it over time.
An entire repository (50–200K tokens) is trained into a Cartridge. Every developer query — "find the auth bug," "explain the payment flow," "write tests for the API" — runs against the same pre-trained Cartridge. No need to stuff the entire repo into every prompt.
SEC filings, contracts, and regulatory texts are perfect Cartridge candidates: they are long, queried many times by different analysts, and rarely change. A 10-K filing (~100K tokens) becomes a tiny Cartridge that any analyst can query instantly.
A patient's full medical history (labs, imaging reports, clinical notes) can be compressed into a Cartridge. Doctors can query it from different angles — drug interactions, lab trends, surgical history — without re-processing the entire record each time.
Long conversational histories (months of messages) can be distilled into a Cartridge, giving the model persistent memory without an ever-growing context window.
For corpora where the full text is available and relatively static, a Cartridge can replace Retrieval-Augmented Generation entirely. Instead of retrieving chunks and hoping you got the right ones, the Cartridge captures all the information holistically.
How does Cartridge quality change as we adjust its size and training compute? The ablations reveal clear scaling patterns.
The authors test p ∈ {128, 512, 2048, 8192}. Across all benchmarks, larger Cartridges perform better — but with diminishing returns. Going from p=128 to p=512 gives a large jump. Going from p=2048 to p=8192 gives a smaller improvement. The sweet spot for most tasks is p=2048, which achieves ICL-matching quality with ~12.5x compression.
Performance scales steadily with the number of training steps. The authors train with batch size 64 and max sequence length 1024. No synthetic data is reused (one epoch). Across all Cartridge sizes, more steps = better quality, with no sign of overfitting in the regime tested.
The authors compare Cartridges parameterized as a KV cache (prefix-tuning) vs. as LoRA adapters. Key finding: prefix-tuning wins on both in-domain and out-of-domain tasks. Critically, LoRA causes catastrophic forgetting — MMLU accuracy drops from 54.7 to 45.3 as LoRA rank increases. Prefix-tuning barely touches MMLU (54.7 to 54.3), because it only adds virtual tokens without modifying the model's weights.
Using 5 diverse seed prompts vs. a single generic prompt: +4.8 accuracy on LongHealth, +7.9 chrF on MTOB. The diverse prompts force the synthetic data to cover different query types, which translates directly to better generalization.
Holding the synthetic data constant, context distillation outperforms NTP by +3.7 accuracy on LongHealth and +8.6 chrF on MTOB. The soft targets from distillation carry more information than hard next-token labels.
Drag the slider to set the Cartridge size (p). The bar chart shows relative quality on LongHealth and MTOB, plus memory usage. Watch the diminishing returns as p grows.
Cartridges sit at the intersection of several active research threads. Here is how the ideas connect.
Modern inference servers already cache KV states for shared prefixes (e.g., system prompts). Cartridges take this further: instead of caching the exact KV cache from prefill, they train a compressed one. Cartridges are drop-in compatible with prefix caching infrastructure.
Retrieval-Augmented Generation retrieves relevant chunks and puts them in context. Cartridges compress the entire corpus into the KV cache, avoiding the retrieval step entirely. RAG risks missing relevant information; Cartridges represent everything (at the cost of offline training).
Methods like DuoAttention, H2O, and SnapKV drop or merge KV pairs at inference time with minimal compute. They achieve 2–4x compression before quality degrades. Cartridges use much more compute (offline training) to achieve 10–100x compression. These approaches are complementary — you could even apply KV compression to a Cartridge.
LoRA-based knowledge injection (e.g., Plug-n-Play Knowledge Modules) also stores corpus information in parameters. Cartridges use prefix-tuning instead, which avoids catastrophic forgetting and integrates with standard serving infrastructure.
Architectures like Titans and TTT use gradient descent at test time to compress context into a fixed-size memory. Cartridges apply the same principle (gradient-based compression) but to standard Transformers without architectural changes — any pre-trained model works.
| Method | Compression | Compute Cost | Quality at High Compression | Infrastructure |
|---|---|---|---|---|
| Full ICL | 1x | Prefill only | Baseline | Standard |
| KV Compression | 2–4x | Minimal | Degrades | Custom kernels |
| RAG | Varies | Retrieval | Retrieval quality | Vector DB |
| LoRA Injection | High | Training | Good, but forgetting | Custom serving |
| Cartridges | 10–100x | ~30 min train | Matches ICL | Standard KV cache |
"What I cannot create, I do not understand." — Richard Feynman. Cartridges embody this: instead of passively reading the corpus at inference time, they actively learn it through self-quizzing and distillation.