Pre-compute KV caches for your document corpus. At query time, blend them with selective re-computation. 2-3x faster TTFT, less than 1% quality loss.
You are running a RAG system. A user asks: "Who scored more goals at FIFA World Cups, Messi or Ronaldo?" Your retriever finds two documents — one about Messi's career stats, one about Ronaldo's — and concatenates them with the query into a single prompt. The LLM reads the whole thing and answers correctly.
But here is the cost. Before the model generates a single token, it runs prefill: a forward pass through the entire input to compute the KV cache — the key and value tensors at every layer for every input token. Prefill cost grows super-linearly with input length. For a 4,000-token RAG context on Llama-70B, prefill takes about 6 seconds on a single A40 GPU. That is 6 seconds of silence before the user sees the first word.
Now consider: your corpus has 10,000 documents. Many queries retrieve the same documents. Every time a document appears in a new query, you re-compute its KV cache from scratch. If a document about Messi's stats appeared in 500 queries today, you computed the same KV tensors 500 times.
The idea of caching KV states is not new. Prefix caching (used by vLLM, SGLang, RAGCache) stores the KV cache of the first text chunk in the input — the prefix. Since the prefix's KV cache does not depend on what follows it, you can reuse it exactly. But in RAG, only the first retrieved document is the prefix. All subsequent documents cannot reuse their cached KV states because they were computed in isolation, without attending to the documents that came before them.
The alternative, full KV reuse (PromptCache), concatenates independently pre-computed KV caches and adjusts positional embeddings. This is fast — but it ignores cross-attention between chunks entirely. The resulting answers can be wrong.
Drag the slider to set the number of retrieved chunks (512 tokens each). Watch how prefill time grows. The orange bar shows full prefill; the teal bar shows CacheBlend (15% selective re-compute). The gap widens with more chunks.
CacheBlend's insight is simple but powerful: you do not need to re-compute the entire KV cache to recover cross-attention. You only need to re-compute a small fraction of tokens — typically 10-15% — on each layer. The rest can be reused as-is.
Why does this work? Because of attention sparsity. In a typical attention matrix, most tokens have very low attention to tokens in other chunks. Only a small subset of tokens have high cross-attention with preceding chunks. These are the tokens whose KV states differ significantly between the independently computed cache and the full-prefill cache. CacheBlend calls them High KV Deviation (HKVD) tokens.
The system is built on top of vLLM and adds about 3,000 lines of Python. The key components are a KV cache store (hashes text chunks to their cached KV states), a loading controller (picks the re-compute ratio and storage device), and a fusor (performs selective re-computation layer by layer, pipelined with KV loading).
Let us make the failure concrete. You have two text chunks:
The query is: "Who scored more goals at FIFA World Cups, Messi or Ronaldo?"
With full prefill, the model processes [Chunk 1 | Chunk 2 | Query] as a single sequence. Every token in Chunk 2 can attend to every token in Chunk 1. The model sees "13 goals" and "8 goals" in the same context and answers "Messi scored more."
With full KV reuse, each chunk's KV cache was computed independently. Chunk 2's keys and values were computed as if Chunk 1 did not exist. The cross-attention — the attention between tokens in Chunk 2 and tokens in Chunk 1 — is zero. The model cannot compare the two numbers. It rambles: "The question is asking about FIFA World Cups. The names of Messi and Ronaldo are well-known..."
To quantify this, CacheBlend defines two metrics:
This is the KV deviation: for layer i and token j, how different is the pre-computed KV from the full-prefill KV? Tokens with high cross-attention to other chunks will have high KV deviation.
This is the attention deviation: on layer i, how different is the forward attention matrix (between context tokens and the final tokens that determine generation) from the full-prefill version? High attention deviation means the model will generate different — usually worse — tokens.
This shows a simplified attention matrix for [Chunk 1 | Chunk 2 | Query]. The left panel shows full prefill (cross-attention present). The right panel shows full KV reuse (cross-attention missing). Toggle the slider to see a CacheBlend blend at different re-compute ratios.
As the paper shows empirically (Figure 2 in the paper), the quality gap between full prefill and full KV reuse grows as you add more chunks. With 5 chunks, the F1 gap is modest. With 25+ chunks, full KV reuse produces significantly worse answers because the missing cross-attention accumulates.
This is the core mechanism. On each transformer layer, CacheBlend does not run the full attention computation over all tokens. Instead, it applies a mask to select only the HKVD tokens, re-computes their Q, K, V projections, merges the new K, V values back into the cached KV, and runs attention. The un-selected tokens keep their pre-computed KV values unchanged.
For each layer i in the model:
The selection of HKVD tokens uses a gradual filtering scheme. On the first layer, CacheBlend performs full prefill (all tokens) and identifies which tokens have the highest KV deviation. It selects slightly more than r% of tokens as candidates. On subsequent layers, it re-computes only those candidates and further filters: tokens whose KV deviation decreases are dropped, and the selection narrows toward the true HKVD set.
Watch the layer-by-layer blending process. Each column represents a token. Orange tokens are HKVD (selected for re-computation). Gray tokens reuse their cached KV. Adjust the re-compute ratio to see how the selection narrows across layers. Click "Step" to advance layer by layer, or "Run" to animate.
Let us trace exactly what CacheBlend does from query arrival to first generated token.
The LLM input is split into text chunks (e.g., 512 tokens each, following RAG convention). Each chunk is hashed. The KV cache store looks up whether a pre-computed KV cache exists for that hash.
For chunks with cached KVs: load them from storage (CPU RAM, SSD, or slower disk) into a GPU-side queue. For new chunks: run standard prefill and add the resulting KVs to the store.
Each chunk was pre-computed with positional embeddings starting at position 0. When chunks are concatenated, their positions change. CacheBlend corrects this by rotating the K vectors with the appropriate RoPE offset. For RoPE, this is a simple multiplication by a rotation matrix — negligible cost.
Layer by layer, the fusor does the following:
This pipelining is critical. The re-compute time per layer (e.g., 3 ms for Llama-7B at 15% ratio on 4K tokens) is often shorter than the KV loading time per layer (e.g., 16 ms from SSD). So the re-compute is completely hidden behind the I/O. You pay zero extra latency.
The fused KV cache is passed to the standard vLLM decoder. Autoregressive generation proceeds normally.
| Component | What it does | Where it runs |
|---|---|---|
| KV Cache Store | Maps chunk hashes to stored KV tensors. LRU eviction when full. | CPU RAM / SSD / Disk |
| Loading Controller | Picks re-compute ratio r% and storage device to balance cost and quality. | CPU |
| Fusor | Layer-by-layer selective re-computation, pipelined with KV loading. | GPU |
The question at the heart of CacheBlend is: which tokens should be re-computed? You need to find the HKVD tokens without knowing the full-prefill KV values in advance (computing those would defeat the purpose).
If you could peek at the full-prefill KV cache (which you cannot in practice), you would find that re-computing the tokens with the highest KV deviation gives the biggest reduction in attention deviation. Figure 6 in the paper shows this clearly: as you re-compute more tokens sorted by KV deviation, the attention deviation drops rapidly at first (the top few % have outsized impact) and then flattens. This is the signature of sparsity.
Tokens with high KV deviation on layer 5 tend to also have high KV deviation on layer 6. The Spearman rank correlation between consecutive layers is typically 0.6-0.8 (Figure 8 in the paper). This makes sense: the input embedding of each token changes slowly between layers (a well-known property of deep transformers), so the KV cache, which is a linear projection of the embedding, also changes slowly.
CacheBlend exploits both insights:
This shows the KV deviation per token on a single layer (simulated). Most tokens cluster near zero — they have negligible cross-attention. A small tail has high deviation — these are the HKVD tokens. Drag the threshold slider to select which tokens would be re-computed.
The paper evaluates CacheBlend on three models (Mistral-7B, Yi-34B, Llama-70B), four datasets (2WikiMQA, Musique, MultiNews, SAMSum), and multiple storage configurations. Here are the headline numbers.
| Metric | vs. Full Prefill | vs. Prefix Caching |
|---|---|---|
| TTFT reduction | 2.2-3.3x faster | 2.2-3.3x faster (prefix only caches 1st chunk) |
| Throughput gain | 2.8-5x higher | 2.8-5x higher |
| Quality loss (F1) | ≤0.02 absolute | ≤0.02 absolute |
| Quality loss (Rouge-L) | ≤0.02 absolute | ≤0.02 absolute |
Full KV reuse is faster than CacheBlend (it does zero re-computation). But its quality is substantially worse: 0.15-0.35 lower in F1/Rouge-L compared to full prefill. CacheBlend closes this gap to within 0.01-0.02 of full prefill, at the cost of 15% more compute than full reuse.
Figure 16 in the paper sweeps the re-compute ratio on Yi-34B. The quality drop from full prefill is at most 0.002 in F1/Rouge-L with 5-18% re-computation. At 15%, the TTFT reduction is 4.1-6.6x compared to full prefill. Going beyond 18% gives diminishing quality returns but costs more compute.
Drag the model-size slider. The chart shows TTFT for four approaches: Full Prefill, Prefix Caching, CacheBlend (15%), and Full KV Reuse. Quality scores are shown below each bar.
Is CacheBlend worth deploying? The answer depends on how often your documents get reused. Let us do the math.
For each document chunk (512 tokens), you run a single full prefill to generate its KV cache. This is a one-time cost. For Llama-7B on an A40, prefilling 512 tokens takes about 0.5 seconds. For a corpus of 10,000 chunks, that is ~83 minutes of one-GPU time.
Storage: each chunk's KV cache occupies (number of layers) × (2 × dmodel × ntokens × 2 bytes for fp16). For Llama-7B (32 layers, d=4096), a 512-token chunk takes ~256 MB. For 10,000 chunks, you need ~2.5 TB. An NVMe SSD handles this easily; CPU RAM requires more planning.
Every time you reuse a chunk's KV cache instead of re-computing it, you save (1 - 0.15) = 85% of the prefill compute for that chunk. If a chunk appears in N queries, the total compute saved is N × 0.85 × (prefill cost). The pre-computation cost is 1 × (prefill cost). So the break-even point is:
After just 2 reuses, you are already saving compute. In practice, popular RAG documents appear in hundreds or thousands of queries, so the amortized cost is negligible.
The loading controller chooses the cheapest storage device where Tload ≥ Trecompute. If loading from SSD takes 16 ms per layer and re-compute takes 3 ms, the SSD is fast enough and cheaper than CPU RAM. If you use a very fast storage (CPU RAM, ~1 ms load), the re-compute cannot be fully hidden, but you still save compute overall.
CacheBlend is not a silver bullet. Here are the real constraints.
If documents in your corpus change (edits, updates), their cached KV states become stale. You must re-compute the KV cache for any modified document. For rapidly changing corpora, this erodes the savings. CacheBlend works best when documents are relatively stable — knowledge bases, legal docs, scientific papers.
The KV cache for each chunk is large. For Llama-70B, a 512-token chunk's KV cache is roughly 2.5 GB (80 layers, much larger dmodel). For 10,000 chunks, that is 25 TB. You need fast, large-capacity storage. The loading controller helps by picking cheaper devices when possible, but the total footprint is non-trivial for very large corpora.
The quality of full KV reuse (and therefore the baseline CacheBlend improves on) depends on chunk ordering. Different orderings produce different cross-attention patterns. CacheBlend does not optimize the ordering — it takes whatever order the retriever provides and blends accordingly. Combining CacheBlend with ordering optimizations is an open direction.
CacheBlend relies on attention sparsity, RoPE positional embeddings, and the transformer's layer-by-layer KV computation. It does not apply to non-transformer architectures like Mamba or Griffin, which have fundamentally different state representations. The authors acknowledge this as a limitation.
The current system stores KV caches on a single storage tier (CPU RAM or SSD, not a hierarchy). A multi-level cache (hot chunks in GPU HBM, warm in CPU RAM, cold on SSD) could further improve performance but is left for future work.
CacheBlend sits at the intersection of several active research areas in LLM systems.
CacheBlend extends prefix caching to non-prefix chunks. Prefix caching is a special case of CacheBlend where only the first chunk is cached and r = 0% for that chunk. CacheBlend's contribution is enabling reuse of all chunks, not just the first one.
Techniques like GQA, MQA, MLA (used in DeepSeek-V3), quantized KV caches, and token pruning reduce the size of each KV cache entry. CacheBlend is complementary: it reduces the number of times you compute KV caches. Combining both gives compounding savings — smaller caches that are computed fewer times.
Systems like Sarathi and DistServe split prefill into smaller chunks to interleave with decoding. CacheBlend is complementary: it reduces the total prefill work, and the remaining selective re-computation can still be chunked.
CacheBlend assumes the "stuff" RAG mode (all chunks concatenated into one prompt). Other modes like MapReduce or MapRerank process chunks separately. The paper shows CacheBlend outperforms both: MapReduce has higher TTFT (extra LLM calls for summarization), and MapRerank has lower quality (chunks processed independently).
CacheBlend focuses on prefill acceleration, while speculative decoding focuses on decoding acceleration. They attack different phases and can be combined. Anthropic's and OpenAI's prompt caching features are conceptually related but operate at the API level, caching exact prefix matches rather than doing selective re-computation.
| Approach | What it caches | Where it helps | Limitation |
|---|---|---|---|
| Prefix caching | First chunk's KV | Prefill | Only 1st chunk reused |
| Full KV reuse | All chunks' KV | Prefill | No cross-attention → quality loss |
| CacheBlend | All chunks' KV + selective re-compute | Prefill | Storage cost, transformer-only |
| KV compression | Smaller KV per token | Memory | Doesn't reduce prefill compute |
| Speculative decoding | Draft tokens | Decoding | Doesn't help prefill |