TransMLA & Sparse Latent Attention

Chapter 1: GQA, Precisely

Before you can prove GQA is a special case of anything, you need GQA written down precisely enough that “special case” is a claim you could check by hand, not a vibe. This chapter does that, using exactly the notation TransMLA's own paper uses, so nothing shifts under you in Chapter 2.

One knob: how many groups

Let D be the model's hidden dimension, h the number of query heads, and d = D/h the dimension of each head — the same per-head width Session 03 used. Standard multi-head attention gives every one of the h query heads its own key head and value head: h query heads, h key heads, h value heads, one-to-one. Group-Query Attention (GQA) changes exactly one thing: it divides the h query heads into g groups, and every query head inside a group shares one key head and one value head with the rest of its group.

Notice the query projection still produces h full-width heads — nothing about how many things ask questions changes. Only the key and value projections shrink, from h heads' worth of output down to g. Attention for query head i, at position t, looks up the key and value belonging to its group:

The only unfamiliar piece is the ceiling expression, ⌈i/(h/g)⌉, and it says something simple: it maps query head i to whichever of the g groups it belongs to. If h = 32 and g = 8, then h/g = 4 query heads share every group, and head 5 (the first head of the second group) maps to group ⌈5/4⌉ = 2. Nothing exotic — it is integer division dressed up in ceiling-bracket notation.

A toy attention pass, computed by hand

Formulas convince the eye; arithmetic convinces the hand. Shrink to h = 4 query heads, g = 2 groups, d = 1 (a single scalar per head, so nothing hides inside a vector), and a two-token history. Group 1's key/value pair at position 1 is (k_1,1 = 2, v_1,1 = 10); group 2's is (k_1,2 = 1, v_1,2 = 20). Query head 3 belongs to group ⌈3/(4/2)⌉ = ⌈3/2⌉ = 2, so it looks up group 2's key and value, not group 1's, regardless of what its own query value happens to be. Say query head 3's query at the current step is q_t,3 = 3. Skipping the softmax normalization for a single-token history (softmax of one score is trivially 1), the output is just:

Query head 1, in group 1, would use k_1,1 = 2 and v_1,1 = 10 instead — a completely different key and value, for the identical position in the sequence, purely because it belongs to a different group. That's the entire mechanism: which key and value a query head sees is decided once, by which group it was assigned to at initialization, and never by anything about the query's own content. Real models have d = 128, not d = 1, and softmax genuinely mixes multiple positions once the history has more than one token — but the group-lookup step this toy example isolates works identically at any scale.

Where GQA came from, briefly

GQA (Ainslie et al., 2023) was introduced as a middle ground, not an endpoint. MQA (g = 1) gives the smallest possible cache but forces every query head in the model to share literally one key and one value — a severe constraint that measurably costs some model quality. MHA (g = h) gives every query head its own key and value, no compromise, but the cache scales with the full head count. GQA's contribution was showing that intermediate values of g trace out a smooth trade-off between those two extremes, letting a model designer choose a cache budget without going all the way to MQA's most severe compression. That framing — a knob you can dial, not a binary choice — is exactly what makes GQA the right family to formalize before Chapter 2 asks whether MLA can absorb the entire knob's range at once.

Why does forcing query heads to share keys and values cost quality at all? Each query head, under full MHA, learns to specialize — one head might track syntactic agreement, another long-range coreference, another local phrase structure, each with its own key and value space tuned to what that head looks for. Force four such heads to share one key/value projection and the shared projection has to be a compromise across everything all four heads need simultaneously, rather than a specialist for any one of them. The fewer groups you have, the more heads are compressed into that same compromise, which is exactly why MQA (all heads sharing a single group) trades away the most quality for the smallest cache, and why the paper's own related-work section states plainly that both GQA and MQA “sacrifice model performance” relative to MHA, in exchange for a smaller cache. Chapter 2's whole point is that MLA breaks this specific trade-off: it reaches MQA's cache size without forcing that same compromise, because the up-projection doesn't have to be a fixed, shared copy — it can be a learned matrix that reconstructs something different for every head from the same compressed latent.

Why the sum only runs from 1 to t, not to h

One more piece of the attention formula worth making explicit, since Chapter 6 leans on it directly: the sum in o_t,i = ∑_j=1^t runs over token positions j from 1 to t, the current position — not over the full sequence length L. This is causal masking: token t is only allowed to attend to itself and everything that came before it, never anything that hasn't been generated yet. It means the number of candidates a query actually compares against grows as generation proceeds — the very first token has exactly one candidate (itself), the last token in a 131,072-token (128K) sequence has up to 131,072. Chapter 6's arithmetic, and every O(L²) claim in this session, comes directly from summing this growing candidate count across every position in the sequence, not from any single step being expensive on its own.

GQA is a slider, not a fork

Here is the detail that makes the rest of this session possible: g is not a binary choice between “grouped” and “not grouped.” It is a single integer that can take any value from 1 to h, and the two extremes are architectures you already have names for.

That last row is the one worth sitting with. MHA is not a different mechanism from GQA — it is GQA with the group count turned all the way up to g = h, so that every “group” happens to contain exactly one query head. This is not a cute reframing for its own sake: it means Llama-2-7B, which uses ordinary multi-head attention with no grouping at all, is already an instance of the same family this chapter is formalizing. Everything TransMLA does to a “true” GQA model with g < h applies just as well to an MHA model with g = h — which is exactly why Chapter 4 can convert Llama-2-7B, an MHA model, using the identical machinery.

Two real models, two ends of the same slider

This session leans on two concrete models, and it matters that they sit at different points on that slider. Llama-3-8B (hidden size 4,096, 32 layers) uses genuine grouping: 32 query heads split across just 8 key/value groups, head width 128 — the paper uses this exact model for its RoPE-concentration experiments in Chapter 3. Llama-2-7B (also hidden size 4,096, also 32 layers) uses g = 32 — ordinary MHA, no grouping at all — and it is the model behind every headline number in Chapters 4 and 5: the 93% compression, the 6-billion-token fine-tune, the 10.6× speedup. Keep both in mind: Chapter 2's proof needs a model with real grouping to be interesting (Llama-3-8B), while Chapter 4's numbers all come from the g = h = 32 case (Llama-2-7B), because that is what the paper actually measured.

Cache savings you already get, for free, before MLA ever enters the picture

Worth pricing out before moving on: how much does grouping alone save, with zero compression, just from sharing key/value heads across a group? Using Session 08's byte arithmetic (2 bytes per fp16 number), compare Llama-2-7B (g = 32, no sharing) against what Llama-3-8B's grouping structure would cost at the same hidden size and layer count:

A 4× reduction — exactly the ratio of the group counts, 32/8 = 4 — and this costs nothing: no compression, no low-rank anything, just fewer key/value heads to begin with. That 4× is the ceiling of what grouping alone can buy you. Everything TransMLA adds on top, starting in Chapter 4, is additional compression within whatever cache GQA already produced.

The same saving, seen as fewer projections to compute

The 4× reduction in Chapter 1's byte arithmetic has a mirror-image saving on the compute side, worth naming so it doesn't get confused with the cache-byte number later: GQA doesn't just cache fewer key/value vectors, it computes fewer of them in the first place. Under MHA (g = 32), producing keys for one token means evaluating the projection W^K ∈ ℝ^{32·128×4096} — 32 separate 128-wide outputs. Under GQA with g = 8, the same step needs only W^K ∈ ℝ^8·128×4096 — a matrix a quarter the size, producing a quarter as many numbers, at prefill time as well as at every future decoding step. This is a real, additional saving on top of the cache-memory one, and it is why GQA earns its place in a production model even before anyone talks about MLA: it isn't just a smaller thing to store, it's less work to produce in the first place.

In code, so the parameter is undeniable

python
def gqa_cache_bytes_per_token(g, d=128, n_layers=32, dtype_bytes=2):
    # g = 1  -> MQA   (every query head shares one KV head)
    # g = h  -> MHA   (no sharing at all)
    return 2 * g * d * dtype_bytes * n_layers

print(gqa_cache_bytes_per_token(1)  / 1024)   # 16.0 KiB  -- MQA
print(gqa_cache_bytes_per_token(8)  / 1024)   # 128.0 KiB -- Llama-3-8B's real g
print(gqa_cache_bytes_per_token(32) / 1024)   # 512.0 KiB -- MHA, g=h, Llama-2-7B

What the slider costs you as g shrinks

It's worth being explicit about the price side of this slider, not just the savings side, before Chapter 2 shows how MLA sidesteps it. Every step down in g merges more query heads onto a shared key and value, and the model has no way to opt out selectively — the sharing is uniform across every token, every layer, decided once at architecture-design time. A model trained with g = 8 groups from the start can, in principle, learn to distribute its 32 query heads' specializations so that heads sharing a group aren't relying on wildly different information — but it cannot make head-by-head exceptions token by token. That rigidity, baked in before a single gradient step of training happens, is the quality cost this chapter's table trades against the memory savings, and it's a cost paid by design, not a cost that shows up only under certain workloads.

The rest of the toy example, filled in

Finish what the toy pass above only showed for query head 3. Query head 1 belongs to group ⌈1/2⌉ = 1, the same group as head 2; query head 4 belongs to group ⌈4/2⌉ = 2, the same group as head 3. Say query head 1's query is q_t,1 = 7 and query head 2's is q_t,2 = −3 — deliberately different query values, to make the next point sharp.

Query heads 1 and 2 produce the identical output, 10, despite starting from completely different query values (7 versus −3). That isn't a coincidence of this particular toy example; it's the mechanism working exactly as designed, just exposed in its most extreme form. With only one cached position, softmax over a single score is trivially 1 no matter what that score is — so the query's own content has zero influence on the output at this history length; only which group's value gets selected matters at all. Real multi-token histories don't collapse this way: once there are two or more candidate positions, softmax genuinely redistributes weight across them based on each query's actual content, and heads sharing a group start producing genuinely different outputs from one another, weighted differently over the same shared pool of keys and values. What never changes, at any history length, is that pool itself — heads 1 and 2 always draw from group 1's key and value, heads 3 and 4 always draw from group 2's, decided once, never by content.

The other bookend: MQA's maximum saving

The byte arithmetic above compared GQA (g = 8) against MHA (g = 32) and found a 4× reduction. Run the same formula at the slider's other extreme, g = 1 (MQA — all 32 query heads sharing a single key/value head), to see the ceiling of what this slider can buy on its own, with zero compression stacked on top:

That 32× is a genuine ceiling, not a floor TransMLA later improves on — it's the most grouping, by itself, can ever save, at any g. Notice it sits well above the 14.2× TransMLA's own most aggressive rung reaches in Chapter 4. That isn't a contradiction: TransMLA's compression happens on top of whatever g a real model already shipped with (g = 32 for Llama-2-7B, the MHA case), and it's solving a different problem than “pick the smallest possible g.” A model already trained at g = 1 would already sit at this 32× ceiling — at the fixed quality cost this chapter's table already named as MQA's price for getting there.

Why nobody just ships every model at g = 1

If MQA gets you the largest cache saving this slider can produce, and Session 03's MLA proves you don't have to accept MQA's quality cost to get MQA's cache size (Chapter 2 is about to prove exactly that), a fair question is why a real production model — Llama-3-8B included — chooses an intermediate g instead of going straight to g = 1. The honest answer, stated plainly by GQA's own paper: g = 1 forces every one of a model's query heads through one shared key/value projection, the most severe version of the compromise this chapter's table warned about — every head's specialization gets averaged into the same bottleneck, with no way for the model to recover any of the lost differentiation. GQA's g = 8 was a deliberate hedge against that risk, chosen before anyone training Llama-3-8B had MLA on the table as an alternative. Chapter 2 makes MQA's cache size safe to want again, by showing you can reach it without paying MQA's quality cost — the reason this session exists.

Chapter 2: GQA Is MLA in Disguise

Chapter 1 gave you GQA as a one-parameter family. This chapter proves the claim the whole conversion depends on: any GQA model, at any group count g, can be exactly rewritten in MLA's mathematical form — same outputs, same cache size, not one bit of information gained or lost — just by choosing a very particular, very unglamorous up-projection matrix. Once that's true, converting the weights stops being a leap of faith and becomes a concrete linear-algebra exercise.

Recall MLA's shape, from Session 03

MLA caches a compressed latent c_t^KV, then reconstructs full-width keys and values from it on demand via learned up-projections:

Session 03 treated W^UK and W^UV as learned matrices, trained by gradient descent from scratch. This chapter asks a different question: forget learning them — can you write down, by hand, a specific choice of W^DKV, W^UK, and W^UV that makes this whole pipeline compute exactly what GQA already computes? If the answer is yes, then GQA was secretly doing MLA's kind of computation the entire time, just with those matrices frozen at one particular, structured setting instead of learned freely.

The latent: just concatenate what GQA already has

Define the latent to be nothing more exotic than every one of GQA's g key heads and g value heads, stacked into a single vector:

This is exactly GQA's existing W^K and W^V stacked on top of each other — nothing new computed, nothing new stored. The latent's width is 2gd, the same as what GQA was already caching per token. There is no compression at this step; that is deliberate, and Chapter 4 is where real compression finally enters.

The up-projection: a selector, not a learned matrix

Now the key move. For query head i, belonging to group j = ⌈i/(h/g)⌉, define W_i^UK ∈ ℝ^d×2gd to be almost entirely zero — a block of zeros for every group except group j, where it is the d×d identity matrix:

Multiply this against the latent and every block except block j gets annihilated by the zeros; block j gets passed through unchanged by the identity. The result:

Do the identical construction for W_i^UV, and every query head reconstructs, via this “up-projection,” precisely the value it was already using. Nothing about the attention scores, the softmax, or the output changes. The forward pass is bit-for-bit the same computation GQA was already doing — it has just been re-described as: down-project to a latent, then up-project per head. That re-description is MLA's computational template.

A toy example, by hand

Formulas convince the eye; arithmetic convinces the hand, exactly as Session 03 insisted. Shrink to h = 4 query heads, g = 2 groups, d = 2 (real scale: h = 32, g = 8, d = 128 for Llama-3-8B). Heads 1–2 share group 1; heads 3–4 share group 2. Say group 1's key is k_t,1 = [3, 1] and group 2's key is k_t,2 = [0, 5]. The latent is their concatenation:

Query head 1 belongs to group 1, so its selector matrix is I_2×2 in the first block, zeros in the second:

Query head 3 belongs to group 2, so its selector is zeros then identity, and it recovers [0, 5] = k_t,2 just as exactly. No approximation anywhere in this step — the identity blocks make the reconstruction lossless by construction.

Finish the same toy example on the value side, so the construction is complete, not half-shown. Say group 1's value is v_t,1 = [4, −2] and group 2's is v_t,2 = [1, 1]. The full latent, now carrying both K and V, is their concatenation:

Query head 1's value-selector W₁^UV is built the same way as its key-selector, just aimed at the V half of the latent: zeros everywhere except an identity block over positions 5–6:

Query head 1's full reconstructed attention input is now (k_t,1, v_t,1) = ([3,1], [4,−2]) — bit-for-bit what plain GQA would have handed that query head directly, arrived at through two extra matrix multiplications that, at this stage, exist purely to make the computation look like MLA's template. Chapter 4 is what makes those two extra multiplications start paying rent.

Why the cache size doesn't shrink yet

Check the cache size honestly before celebrating. The latent is 2gd wide, and that is exactly what plain GQA was already caching — nothing has been compressed. The paper is explicit about this: reforming GQA into MLA's shape, on its own, changes nothing about memory. What it changes is that each query head's effective dimensionality has grown — W_i^UK is d×2gd instead of the original d×D per-group projection, and running these extra selector matmuls at inference time actually adds a little compute, for zero cache benefit, if you stop here.

Why this is worth doing: a strictly bigger hypothesis space

TransMLA's Appendix A proves something sharper than “GQA fits inside MLA's template.” It proves an ordering of expressive power, holding the KV cache size and query head count fixed:

The GQA ≤ MLA_Factorized half is exactly the identity-selector construction this chapter just walked through, formalized: GQA's key projection has rank at most gd (Chapter 1's cache-size ceiling), and setting r_kv = 2gd in MLA's factorized form with the identity-selector W^UK reproduces it exactly — so anything GQA can compute at that cache budget, factorized MLA can compute too, by construction. The strict inequality comes from the other direction: a learned, dense W^UK at that same rank budget can represent combinations across groups that a block-identity selector structurally cannot — the selector can only ever copy one group's key verbatim, never blend information across groups the way a free matrix can. The MLA_Factorized ≤ MQA half follows a similar logic one level up: MQA's single shared head, at the same total width 2gd, uses one unconstrained dense projection with no low-rank bottleneck at all, which can represent anything MLA_Factorized's two-matrix product can and more. Practically, this ordering is the entire justification for bothering with reformation in the first place: it moves you into a strictly larger space of functions at the exact same cache cost, before a single gradient step of fine-tuning happens.

Making "blend across groups" concrete

Reuse the toy latent from above, c_t^KV = [3, 1, 0, 5, …] (group 1's key [3,1], group 2's key [0,5]). The identity selector for query head 1 can only ever output exactly [3,1] or exactly [0,5] — whichever block its identity sits over — because every entry of the selector matrix is fixed at 0 or 1 by construction, never anything in between. Now imagine a learned, dense 2×4 up-projection instead, with no zero/identity constraint at all — say the model discovers, through training, that this particular query head does best attending to the average of both groups' keys, [1.5, 3]. A dense matrix can represent exactly that: a row like [0.5, 0, 0.5, 0] would produce [0.5×3+0.5×0, 0.5×1+0.5×5] = [1.5, 3] directly. No identity-selector matrix can ever produce that output, for any choice of which block holds the identity — blending is structurally outside what a selector can express, no matter how it's arranged. That is the concrete content behind “the inequality is strict”: it isn't that learned matrices are strict supersets in some abstract sense, it's that there are specific, nameable outputs — like this average — a selector can never produce and a dense matrix can.

In code: the reformation, done to a real GQA layer

python
import torch

def build_selector(query_head_i, h, g, d):
    """W_i^UK: d x (g*d), identity in block j, zero elsewhere."""
    group_j = query_head_i // (h // g)          # which group this query head belongs to
    W = torch.zeros(d, g * d)
    W[:, group_j*d:(group_j+1)*d] = torch.eye(d)   # identity block, everything else stays 0
    return W

# sanity check: reconstructing group j's key should equal the original, exactly
h, g, d = 32, 8, 128
c_KV = torch.randn(g * d)          # stand-in for the concatenated group keys
W5 = build_selector(4, h, g, d)     # query head index 4 (0-indexed) -> group 4//(32//8) = group 1
k_reconstructed = W5 @ c_KV
assert torch.allclose(k_reconstructed, c_KV[d:2*d])  # exactly group 1's slice, bit-for-bit

Why one merged latent, and not g separate small ones?

A natural question: why concatenate every group's key and value into a single wide latent, rather than just keeping g separate, smaller latents — one per group — and skipping the selector-matrix machinery entirely? The honest answer is that a per-group latent scheme would look almost identical to what GQA already does, which is exactly the point Chapter 2 needed to avoid: the goal isn't to describe GQA in new words, it's to land in MLA's specific computational template, one shared latent per token with per-head up-projections reading out of it, because that's the template Chapter 3's Absorb-compatibility argument and Chapter 4's joint compression both depend on operating over. Merging into one latent isn't a cosmetic choice; it's what makes “treat this as an MLA layer” a literally true statement rather than an analogy, and every mechanism downstream in this session builds on that being literally true, not just similar in spirit.

What the reformation actually costs, in extra arithmetic

Quantify the “a little compute” the previous section flagged, rather than leaving it vague. Plain GQA, at inference, needs zero extra arithmetic to fetch query head i's key: it's an array lookup, group j's key sitting exactly where the model already stored it, no matrix multiply involved at all. The reformed version, by contrast, computes k_t,j = W_i^UKc_t^KV, an honest d×2gd matrix-vector product, even though every entry of that matrix is a fixed 0 or 1:

Put that number in context rather than leaving it to sound alarming on its own: 16.8 million multiply-adds is tiny next to the roughly 202 million parameters (Chapter 0's per-layer breakdown) a single Llama-3-8B-shaped layer already runs through on every token, and it's a cost paid specifically so that a dense, learned matrix can be substituted in later without changing anything else about the computational shape. This chapter's proof was never claiming reformation is free in absolute terms — only that it's exact, and that whatever small overhead it adds buys access to a strictly larger function space in return.

Reformation doesn't undo Chapter 1's quality cost either

One more honest bookkeeping point, easy to miss if this chapter's proof reads as unconditionally good news: rewriting a g = 8 GQA layer into MLA's identity-selector shape produces a model that computes bit-for-bit the same outputs as before, which means it inherits exactly the same quality cost Chapter 1 attributed to g = 8's sharing — no more, no less. The reformed model isn't a better model than the GQA model it came from; it's the identical model, in a different mathematical costume. The identity selector cannot blend across groups (the “making blend across groups concrete” section above proved that explicitly), so it cannot recover whatever specialization g = 8's grouping already discarded during pretraining. What the reformation buys is not quality recovered; it's a shape that makes recovering some of that quality possible later, once the identity selector gets replaced by a learned, dense one that can blend — and it's that replacement, with its own separate fine-tuning, that actually pays down some of Chapter 1's quality debt. The reformation step itself pays down none of it.

Chapter 3: RoPE Breaks the Trick

Chapter 2 built a reformation that is exact and lossless. Chapter 4 is where real compression happens. Between them sits a genuine obstacle, and skipping past it would make Chapter 4's numbers look like they came from nowhere: Session 03's Absorb operation, the trick that makes MLA fast at inference time, quietly assumes something that is false for almost every GQA model you'll actually find in the wild. This chapter is about what breaks, and TransMLA's fix for it.

Recall Absorb, precisely

Session 03 showed that a score q_t^Tk_t^C, once k_t^C is written as W^UKc_t^KV, regroups by associativity:

The critical fact this relies on: W^Q and W^UK are plain, static, learned matrices, with no dependence on which token t or j you're scoring. That means W^QTW^UK is a single fixed matrix, computable once, offline, before any request arrives — and reused, unchanged, against every cached latent the model will ever see, regardless of that latent's position in the sequence. That is exactly what lets MLA skip reconstructing full-width keys at inference: fuse the weights once, score directly against the small cached latent forever after.

What RoPE actually does to that argument

Rotary Position Embedding does not touch the hidden state before projection — it rotates the key after the up-projection, and the rotation angle depends on the token's position:

where R(t) is the block-diagonal rotation matrix Session 08's RoPE formula builds, one 2×2 rotation per frequency pair, all driven by the token's absolute position t. Redo the associativity argument with R(t) sitting in the middle:

Look at the middle term now: W^QTR(t)W^UK is not a fixed matrix anymore — it depends on t, the position of the token being scored, which is different for every cached latent in the sequence. You cannot precompute one fused matrix offline and reuse it against every position, because a different R(t) sits inside the product for every single cached token. The entire premise Absorb needs — one fusion, computed once, valid forever — is gone the instant RoPE sits between the up-projection and the score. Concretely: reconstructing keys explicitly, at full width, on every single decoding step is exactly the computational overhead Session 03's Absorb trick exists to avoid, and RoPE forces you right back into it.

Two positions, two different rotations, made concrete

Verify “R(t) is different for every position” on numbers small enough to see, rather than take it on faith. Session 08's RoPE formula rotates each dimension pair by an angle t·θ. Take one such pair, θ = 1 radian for simplicity, and compare position t = 1 against position t = 5:

Two entirely different 2×2 matrices, from the same formula, purely because t changed from 1 to 5. Now ask what “fuse W^QTR(t)W^UK once, offline” would even mean here: there is no single matrix that equals W^QTR(1)W^UK and W^QTR(5)W^UK at the same time, because those are two different products with two different middle terms. A cached sequence has as many distinct R(t) values as it has positions — fusing “once” would require picking one t and being wrong for every other position in the cache.

DeepSeek's own way out, and why it doesn't transfer directly

DeepSeek-V2 sidestepped this problem entirely by never letting it arise: they designed decoupled RoPE from the start of pretraining — a small, separate part of each key (dimension d^R) carries positional rotation and is never compressed, computed fresh every step at full width; the much larger remaining part carries no rotation at all (“NoPE”) and gets compressed through the latent, fully Absorb-compatible. That works beautifully — if you get to choose it before a single token of pretraining happens. A GQA model converted after the fact was never trained with that split. Every dimension of every key already has RoPE baked into it, indiscriminately, by the pretraining that already happened. TransMLA's job is to manufacture that same clean split, after training, from a model that was never built with one.

RoRoPE: rotate first, then split

The fix starts from an observation: not every dimension of a key carries equal positional signal. TransMLA runs principal component analysis (PCA) on the key activations and finds a rotation matrix U that concentrates most of the positional information into the first attention head's dimensions, leaving the rest nearly position-free. The rotation is not arbitrary — the paper proves the query-key inner product is provably invariant under any U satisfying two conditions: it rotates only within the same dimension index across every head (not mixing dimension 3 of head 1 with dimension 7 of head 2), and it rotates the real and imaginary halves of each RoPE pair identically. Any rotation obeying those two constraints can be applied for free — it changes nothing about what the attention scores compute, only how the positional signal is distributed across dimensions. FreqFold pushes the same idea further, exploiting the fact that RoPE dimensions with similar rotation frequencies carry redundant, foldable information, concentrating things even more tightly into that first head.

Why those two specific conditions on U

Neither condition is arbitrary bookkeeping; each one protects a specific piece of machinery from breaking. The first — rotate only within the same dimension index, never mixing dimension 3 of head 1 with dimension 7 of head 2 — matters because RoPE's rotation angle t·θ_i is different for every dimension pair i. Mixing two dimensions that rotate at different frequencies would smear together information that's supposed to stay separable by position, corrupting the very signal RoRoPE is trying to concentrate. The second — rotate the real and imaginary halves identically — matters because RoPE pairs dimensions into (real, imaginary) components (recall Section 3.1's complex-number framing: odd-indexed dimensions are the real part, even-indexed the imaginary part). Rotating one half of a pair differently from the other would break the 2D rotation structure RoPE relies on to encode relative position at all — it would no longer be a rotation in the plane, just an arbitrary linear map that happens to touch the same coordinates. Both constraints exist to guarantee the thing being rotated is still, afterward, a valid RoPE-compatible representation — concentrated differently, but not broken.

FreqFold's contribution sits one level below that: even after RoRoPE's rotation, some residual positional signal still leaks into dimensions outside the designated first head, concentrated in dimensions whose rotation frequency θ_i happens to be numerically close to a frequency already inside that first head. Because nearby frequencies rotate at nearly the same rate over any short span of positions, their contributions are close to redundant — two dimensions telling you almost the same thing about relative position, just phrased slightly differently. FreqFold identifies those near-duplicate frequency pairs and folds them together, squeezing a little more positional signal into the same fixed-size RoPE head without adding a single new dimension to it.

Once positional signal is concentrated into one head's worth of dimensions, the split becomes possible: that one head keeps its RoPE, uncompressed, computed fresh every step exactly like DeepSeek's decoupled positional key. Every other head's now-nearly-position-free dimensions get merged with the values and compressed through the latent, fully eligible for Absorb. The RoPE-carrying head Chapter 4 leaves alone; the rest is where the real compression numbers come from.

Does the concentration actually work? A real number

Measured on Llama-3-8B against WikiText-2, removing RoPE from an increasing share of dimensions and tracking log-perplexity: at a 90% removal ratio — keeping positional rotation in only the last 10% of dimensions — RoRoPE combined with FreqFold holds log-perplexity around 2. A competing method that instead selects which dimensions to drop purely by the raw magnitude (norm) of each dimension, evaluated at the same 90% removal ratio, reaches log-perplexity nearly 6 — a model the paper describes as no longer generating meaningful text at all. Recall from Session 08: perplexity is e^loss, so this gap is exponential, not additive — e² ≈ 7.4 versus e⁶ ≈ 403, roughly a 54× difference in “how surprised the model is,” from the same removal ratio, purely because of which dimensions got chosen to carry the remaining positional signal.

How small is the RoPE-carrying slice, on the real model?

Put the split in concrete proportions for Llama-2-7B, the model Chapter 4's headline numbers come from: 32 heads of 128 dimensions each, 4,096 dimensions total. RoRoPE concentrates positional signal into just the first head:

Almost the entire key vector becomes fair game for Chapter 4's low-rank compression; only a sliver — about one thirty-second of it — has to stay uncompressed, full-width, and recomputed fresh every step to preserve positional information. That lopsided split is exactly why the compression ratios in Chapter 4's table (−68.75% and beyond) are possible at all: compressing 96.9% of the key hard, while leaving 3.1% alone entirely, easily nets out to a large overall reduction.

Sanity-check that arithmetic against Chapter 4's headline −92.97% figure directly, so the two chapters' numbers visibly agree rather than sitting side by side unconnected. If the untouched RoPE slice (3.1% of the key) survives every compression pass completely, while the remaining 96.9% (merged with the full value vector) gets compressed down aggressively by Chapter 4's PCA, the overall reduction has to land somewhere below a full 96.9% — it can never exceed the fraction that was ever eligible for compression in the first place. Chapter 4's actual measured −92.97% sits comfortably under that 96.9% ceiling, consistent with a real, lossy compression applied to the eligible share rather than some impossible reduction squeezed out of the untouched RoPE head too.

python
# conceptual shape, not the real PCA machinery -- the point is what gets kept vs merged
def split_after_rope_concentration(key_full, d_h=128, n_heads=32):
    # after RoRoPE + FreqFold: dims 0..127 (head 0) carry ~all positional signal
    k_rope    = key_full[:d_h]           # kept full-width, uncompressed, RoPE applied every step
    k_nope    = key_full[d_h:]           # (n_heads-1)*d_h dims, ~position-free -> merge with V, compress
    return k_rope, k_nope   # k_rope: Absorb-incompatible but tiny; k_nope: Absorb-compatible and large

What specifically goes wrong with the naive baseline

The comparison table above named a naive alternative — select dimensions to drop by raw magnitude — and showed it collapsing toward incoherence at a 90% removal ratio. Worth being precise about which naive alternative that is, because it's a real, published method, not a strawman: MHA2MLA, a concurrent GQA-to-MLA conversion approach, determines which RoPE dimensions to remove solely by the norm of the query and key vectors at each dimension. TransMLA's own comparison identifies two specific problems with that choice. First, norm alone doesn't track how foldable a dimension's positional signal is — two dimensions can carry similar norms while holding very different amounts of genuinely non-redundant positional information, so a norm-only cutoff throws away real signal, the same failure mode Chapter 4's PCA-bias discussion warns against for keys and values, just applied here to RoPE dimensions instead. Second, and more of an engineering problem than a statistical one: the dimensions norm-based selection ends up keeping aren't contiguous — they're scattered unevenly across the key's width, which means serving them efficiently requires sparse indexing (gathering non-adjacent dimensions on every decoding step) rather than a clean contiguous slice a GPU kernel can read in one pass. RoRoPE and FreqFold's rotation solves this second problem too, as a side effect of solving the first: by construction, the dimensions that end up carrying positional signal after the rotation are one head's worth, a single contiguous block, not a scattered handful chosen after the fact by inspecting norms.

Watching the fusion actually fail, on numbers

Push the R(1)-versus-R(5) comparison one step further than “two different matrices” and watch what happens to an actual attention score. Take a toy fused query direction, after W^QTW^UK has already been applied — call it q′ = [1, 0] for simplicity — and score it against the same key direction k = [1, 0], cached at two different positions:

Same query direction, same key direction — and two different scores, 0.540 versus 0.284, purely because the cached position changed from t = 1 to t = 5. A single fixed “fused” matrix multiplied against the same key vector can only ever produce one score, by definition of what a fixed linear map does — it cannot output 0.540 and 0.284 from the identical input k. That's the concrete failure this chapter has been describing abstractly: Absorb's whole premise is one fused matrix, valid everywhere it's applied; RoPE hands you a family of matrices, one per position, and no single member of that family is correct for every cached token at once.

Chapter 4: The Compression Ladder

Chapter 3 built the split: a small RoPE-carrying head left alone, and a much larger NoPE remainder free to compress. This chapter does the actual compressing, fixes one more real obstacle along the way, and then shows the payoff in the same currency Session 08 used: kibibytes per token, on the exact same Llama-2-7B this whole course keeps coming back to.

One more obstacle: keys and values don't weigh the same

The NoPE keys and the values are meant to be compressed jointly, into one shared low-rank latent, the same way Session 03's MLA compresses K and V together. Joint compression via PCA works by finding the directions of greatest shared variance — but TransMLA's authors measured something inconvenient on Llama-3-8B's first layer: even after removing the RoPE-carrying head, the remaining keys' activation norms are substantially larger than the values' norms. PCA, run naively on the two stacked together, ends up finding directions that mostly track the keys, because the keys simply dominate the shared variance by magnitude — and whatever distinguishes one value from another gets discarded as numerically insignificant, even when it isn't semantically insignificant at all.

See the bias on numbers small enough to trust by eye. Suppose (toy scale) a key dimension varies across tokens as [10, −10, 8, −8] and a value dimension, carrying genuinely useful distinctions between tokens, varies as [1, −1, 0.9, −0.9]. Stack them and ask PCA to find the single direction of greatest variance across both. The key dimension's spread (variance around 88) swamps the value dimension's spread (variance around 0.9) by roughly a factor of 98× — so the top principal component will point almost entirely along the key axis, [1, 0, …], contributing almost nothing to reconstructing the value axis at all. Compress down to one dimension using that component and the value information — genuinely present, genuinely distinguishing one token from another — is thrown away almost entirely, not because it doesn't matter, but because it was numerically quieter than the key sitting right next to it in the same PCA call. Balanced KV rescales exactly this — bringing both streams' variance to a comparable magnitude before PCA ever runs — so the top components reflect actual shared structure, not just whichever input happened to be louder.

The fix, Balanced Key-Value (BKV), is exactly what the name says: rescale the norms of keys and values to be comparable before running the joint PCA, so the compression algorithm sees two inputs on equal footing instead of one drowning out the other. The paper confirms this matters empirically: joint low-rank compression with balancing consistently loses less than the same compression without it, whether the PCA is run on attention weights directly or on the actual activation outputs — and activation-based PCA with balancing is the combination TransMLA ships with.

The compression ladder, with real numbers

Put RoRoPE, FreqFold, and Balanced KV together and TransMLA can target any compression ratio on the ladder. The paper's headline experiment runs Llama-2-7B (pretrained on 2 trillion tokens, average score 59.85 across six benchmarks: MMLU, ARC, PIQA, HellaSwag, OpenBookQA, WinoGrande) through three compression levels:

Read the last row against the first non-zero row carefully: compressing the cache to less than a fourteenth of its original size, then fine-tuning on 6 billion tokens — not 2 trillion, 6 billion — lands within 1.17 points of the fully-trained original. Divide the two token counts to see exactly how little this costs relative to the sunk cost Chapter 0 opened with:

For comparison, a competing conversion method (MHA2MLA) at the same −68.75% compression, with zero fine-tuning, scores 37.90 — a drop of nearly 22 points, more than thirteen times worse than TransMLA's 1.65 at the identical cache budget. That gap is the empirical proof that RoRoPE, FreqFold, and Balanced KV are not decorative refinements; they are the difference between a compressed model that is still usable out of the box and one that mostly isn't.

The fine-tune budget is itself a ladder

The paper's full table has one more layer of structure worth surfacing: TransMLA doesn't spend a flat 6 billion tokens at every compression level — the fine-tune budget grows with how aggressive the compression is, exactly the way you'd expect if more compression destroys more information that needs re-teaching.

Compare TransMLA's 500M-token recovery at −68.75% against MHA2MLA's own recovery at the identical compression level: MHA2MLA needs a full 6 billion tokens — twelve times TransMLA's budget — to reach 59.51, a score TransMLA already exceeds (59.82) using a twelfth of the fine-tuning data. The compression ratio being equal makes this a clean, apples-to-apples comparison of just the conversion technique: RoRoPE, FreqFold, and Balanced KV aren't only about surviving zero-shot with less damage — they leave the model close enough to its original behavior that whatever fine-tuning does happen goes much further, much faster, per token spent.

The same ladder, in Session 08's units

Convert every rung of that ladder into the same KiB-per-token currency Session 08 used for this exact model's dense cache. Recall the baseline: 512 KiB/token for Llama-2-7B under plain MHA, derived from 2×4,096×2 bytes×32 layers.

Run Chapter 0's day-long-chat numbers back through the last rung: 48,000 tokens at 36.0 KiB each is 1,728,000 KiB ≈ 1.65 GiB for a full 8-hour working day of cache — against the 23.4 GiB Session 08 computed for the same day under dense attention. The wall Session 08 spent an entire chapter deriving is now comfortably below a single gigabyte of headroom for a routine conversation.

python
def compressed_kib_per_token(reduction_pct, baseline_kib=512.0):
    return baseline_kib * (1 - reduction_pct)

for pct in [0.6875, 0.8750, 0.9297]:
    print(f"-{pct*100:.2f}% -> {compressed_kib_per_token(pct):.1f} KiB/token")
# -68.75% -> 160.0 KiB/token
# -87.50% -> 64.0  KiB/token
# -92.97% -> 36.0  KiB/token

A parameter saving too, not just a cache saving

Everything so far has been about the cache produced at inference time. Following the same low-rank logic Session 03 applied to MLA's weight matrices, TransMLA's compression also shrinks the up-projection weights themselves — a storage and loading saving, separate from and in addition to the per-token cache saving. Before compression, reconstructing the full-width NoPE keys and values from the merged latent (Chapter 2's 2gd-wide representation, g = 32 for Llama-2-7B's MHA case) needs up-projection matrices of shape hd×2gd = 4,096×8,192 each for K and V:

Compress the shared latent down to the −92.97% rung — roughly 576 dimensions once the RoPE head is set aside, matching the scale of DeepSeek's own d_c + d_h^R from Chapter 6 — and the same up-projections shrink to 4,096×576 each:

This is a smaller checkpoint to load and less activation memory during any future fine-tuning — a real, additional benefit, on top of and separate from the KiB-per-token number the rest of this chapter has been building toward.

Which rung would you actually pick?

Put yourself back in Chapter 0's infrastructure-lead seat and make the call. If your team has no budget or schedule for a fine-tuning run at all — a hard constraint, not a preference — the −68.75% rung is the only one on this ladder that's usable immediately, at a 1.65-point cost, with 4× the free cache budget Chapter 1 already showed grouping alone buys, stacked on top. If a modest fine-tuning run is available — and 500M tokens, per Table 1, is a run measured in hours on a modern cluster, not weeks — the same rung recovers to within 0.03 points of the original, essentially free performance-wise. Only reach for the aggressive −92.97% rung when the deployment target genuinely needs that much headroom back: Chapter 5's 16K-and-beyond context lengths on memory-constrained hardware are exactly the case where the extra squeeze is worth the larger (still comparatively tiny) 6-billion-token fine-tuning investment. The ladder isn't a recommendation to always go as far as possible — it's a menu, and the right rung depends on what your actual memory ceiling and fine-tuning budget look like.

What actually moves during that fine-tuning, closing the loop from Chapter 0

Chapter 0 computed that roughly 15.9% of Llama-2-7B's parameters — the key and value projections — are the slice that needs new structure after conversion. This chapter's fine-tuning numbers are about that same slice, not the whole model: the embeddings, the feed-forward blocks, W^Q, and W^O all started this process already correct, already trained on 2 trillion tokens, and the recovery fine-tuning run is overwhelmingly spent nudging the newly-constructed W^DKV, W^UK, and W^UV (and, indirectly, letting the rest of the model adapt slightly to whatever small numerical differences the compressed reconstruction introduces) — not re-teaching grammar, facts, or reasoning from nothing. That is the mechanical reason 6 billion tokens is enough: the fine-tuning run's job was always narrower than the number “2 trillion vs. 6 billion” makes it sound, because most of what determines a language model's competence was never touched by the conversion in the first place.

A cross-check: does -92.97% actually fit inside Chapter 3's ceiling?

Chapter 3 established that only about 3.1% of a key's width has to stay uncompressed (the RoPE-carrying head), meaning at most 96.9% of the key, plus the entire value vector, is ever eligible for compression. Verify Chapter 4's headline −92.97% cache reduction is internally consistent with that ceiling rather than exceeding a budget Chapter 3 said wasn't available. A 92.97% overall reduction, applied to a representation where roughly half is key (partly protected) and half is value (fully eligible), works out to compressing the eligible portion down by a comparable double-digit-times factor — well inside “compress 96.9% of the key hard, don't touch the rest,” not a contradiction of it. Two chapters' worth of independently-derived numbers landing in a mutually consistent range is exactly the kind of agreement worth checking for, rather than assuming, whenever a session builds one chapter's arithmetic on top of another's.

Generalizes beyond Llama-2-7B: same ladder, a different model

Every number in this chapter's headline table so far comes from one model. Worth checking the paper's second worked example, because a technique that only reproduces on the exact model it was tuned against is a much weaker claim than one that generalizes. The paper runs the identical compression ladder on SmolLM-1.7B (pretrained on 1 trillion tokens, baseline average score 55.90 across the same six benchmarks), a model roughly a quarter Llama-2-7B's size:

Two things to read off this second table. First, the shape repeats: zero-shot conversion already lands close, and a comparatively small fine-tuning run closes almost the entire remaining gap — the same story as Llama-2-7B's table, on an unrelated model. Second, the fine-tune budget itself scales down with the model: SmolLM-1.7B needs only 300 million tokens at this compression level, against Llama-2-7B's 500 million at the identical −68.75% rung — both a small fraction of their respective pretraining budgets (300M ÷ 1T = 0.03% for SmolLM, close to the 0.025% that Llama-2-7B's 500M ÷ 2T works out to), consistent with the same underlying claim: fine-tuning re-adapts, it doesn't re-teach.

The paper states the comparison against MHA2MLA even more sharply on this exact model: compressing SmolLM-1.7B's cache to 31.25% of its original size, TransMLA needs only 4.9% of the training data MHA2MLA needs to reach the same quality — and completes that fine-tuning run in roughly 2 hours. That is not a small efficiency edge; it's close to two orders of magnitude less data for the same outcome, on a second model, independent of every one of Llama-2-7B's specific numbers.

Which PCA, exactly? An ablation the paper ran directly

Balanced KV fixes which directions PCA finds, by rescaling the inputs first. A separate question is what PCA runs on in the first place, and the paper checks this directly rather than assuming one choice: PCA applied to the attention weight matrices themselves (“W-based”) versus PCA applied to the actual activation outputs those weights produce on real data (“WX-based”). The result is unambiguous in the paper's own comparison — activation-based PCA yields meaningfully lower loss than weight-based PCA, with or without balancing stacked on top, and Balanced KV improves both variants consistently. TransMLA ships the combination that wins on both axes: PCA on activations, with balancing applied first. The intuition behind why activations beat raw weights is the same idea this chapter has leaned on throughout: weights describe what a matrix is capable of producing across every possible input; activations describe what it actually produces on the data the model will really see. A compression method that only has to preserve the directions that matter for real inputs, rather than every direction the weight matrix could theoretically express, has an easier and more targeted job.

Chapter 5: The vLLM Payoff (showcase)

Every number so far has been about bytes and benchmark scores. Neither one is what an infrastructure team actually gets paged about. This chapter asks the question that matters at 3am: does a smaller cache turn into more tokens per second, on real GPUs, running real inference software — or is this all a paper exercise that evaporates the moment it meets vLLM?

The experiment

TransMLA's authors benchmarked the −92.97%-compressed Llama-2-7B against the original, using the vLLM serving framework, on three real consumer/prosumer-tier accelerators of increasing capability: 165.2 TFLOPS with 24 GB of memory, 312 TFLOPS with 40 GB, and 320 TFLOPS with 64 GB. Both models generated output at matched input/output lengths (a context length of 8K means roughly 4K tokens in, 4K tokens out), and throughput — output tokens per second — was measured directly, not estimated.

Two things to read off this table, and they are different kinds of result. First, the ratio: at 8K context on the smallest GPU, dividing TransMLA's throughput by the original's gives 832.69 ÷ 85.80 ≈ 9.7× — computed directly from the paper's own reported numbers. The paper's abstract quotes a rounded headline of “10.6×” for this same 8K/24GB configuration, likely reflecting a slightly different measurement pass (the paper separately reports results for a “low-rank Q” variant that also compresses the query path); both numbers describe the same order of magnitude, and either way the honest, directly-computable figure from the table above is already close to an order of magnitude.

Second, and more strikingly: at 16K context on the 24 GB card, the original model does not run at all — it exhausts memory before it can even produce output. The compressed model produces 414 tokens per second on the exact same hardware. No ratio can be computed for that row, because the denominator is zero capability, not a slower number. This is the day-long-chat wall from Session 08, made concrete on real inference hardware: it isn't that TransMLA makes long-context serving somewhat cheaper — on this GPU, it is the difference between serving 16K context at all and not.

Why the speedup grows with context, instead of staying flat

Notice the ratio itself is not constant across the table — it grows as context grows: roughly 4.7× at 1K, 7.6× at 4K, 9.7× at 8K, and then undefined (infinitely better, in the only sense that matters) at 16K. This is exactly the behavior you should expect once you know why MLA helps: the KV cache is what grows with context length, and the smaller that cache is, the more of the GPU's fixed memory stays free for larger batches and longer sequences before anything hits a ceiling. At short context, the cache was never the bottleneck to begin with, so compressing it buys comparatively little. At long context, the cache becomes the dominant cost, and shrinking it dominates the speedup. The gain compounds exactly where it is needed most, which is the opposite of a coincidence — it's the direct, mechanical consequence of what got compressed.

The same story, confirmed from the other direction: more headroom, smaller advantage

The three-hardware-tier table lets you check this mechanism from the opposite angle too. At 32K context, compute the ratio on each of the two GPUs that can actually complete the run:

The ratio shrinks as the GPU gets more memory headroom — 6.4× on the 40 GB card, 5.0× on the 64 GB card, both measured at the identical 32K context length. That is the same mechanism as the growing-with-context finding above, seen from the memory-headroom axis instead of the context-length axis: on a card with more spare memory, the original dense model was never as close to its ceiling in the first place, so compressing the cache has less slack to reclaim. Both directions point at the same underlying cause — the speedup tracks how close the original model was to running out of memory, not some fixed multiplier TransMLA applies unconditionally.

What the compressed model actually says

Numbers on a throughput table are one kind of evidence. The paper also published raw generation samples, which are worth reading precisely because they are the least abstract evidence available: does the compressed model still sound like a language model, or like noise? Prompted with “Hello, my name is” on the −92.97%-compressed Llama-2-7B:

Even at zero fine-tuning — the row that scored 43.26 on the benchmark table, sixteen-and-a-half points below the original — the output is recognizably English, grammatically mostly intact, just wandering and inconsistent (a dog that is also a “3600 year-old tribe,” reptiles introduced from nowhere). That matches the benchmark number's story without needing the number at all: a model that is damaged, not destroyed. By the fine-tuned row, the output reads like an ordinary, coherent chatbot response. Numbers and qualitative samples are telling the same story from two different angles, which is exactly the kind of cross-check worth doing before trusting either one alone.

One more detail worth naming honestly: this section's table numbers come from one specific configuration. The paper separately reports a “Low-rank Q” variant — additionally compressing the query path the way Session 03 flagged as a training-time optimization — alongside a “Full-rank Q” variant that leaves queries uncompressed; the two aren't identical in throughput, and the abstract's rounded 10.6× headline likely reflects whichever variant that specific measurement used. The table above is internally consistent (every number in it comes from Table 4 of the paper, the same measurement pass), which is what makes the 6.4× and 5.0× comparison above trustworthy even without knowing exactly which Q-variant produced the abstract's separate headline number.

Why there's already a real speedup at 1K, long before anything OOMs

Table 4's 1K row shows TransMLA at 4.7× the original's throughput, on hardware nowhere near its memory ceiling. That number needs its own explanation, since “more free memory” can't be the reason yet — there's plenty of memory to spare on all three tiers at 1K context. The real reason is memory bandwidth, not memory capacity: autoregressive decoding generates one token at a time, and at every single step the GPU has to read the entire KV cache back off memory to compute that step's attention, even when there's room to spare for storing it. For small batch sizes — the common case in interactive serving — this repeated cache read, not raw compute, is usually what limits how fast tokens come out at all, a well-known property of autoregressive decoding independent of anything this session has covered. Shrinking the cache by 92.97% doesn't just delay an eventual OOM; it shrinks how many bytes have to move off memory on every single decoding step, from the very first token onward — which is exactly why the speedup shows up even where capacity was never the constraint, and why it then compounds further once capacity does become binding at longer context.

Put a number on it using figures already derived in this session. At 8K context, a single decoding step under dense Llama-2-7B has to read all 8,000 cached tokens' worth of keys and values back off memory, at 512 KiB per token from Chapter 1's baseline:

The −92.97%-compressed model, at 36.0 KiB per token from Chapter 4's ladder, moves:

Nearly 4 gibibytes shuttled off memory to produce a single output token is not a hypothetical cost — it happens again for the next token, and the one after that, for as long as generation continues. That per-step data-movement gap, not the eventual OOM, is what a 4.7×–9.7× throughput advantage at context lengths well short of any memory ceiling is actually measuring.

What this table doesn't show: batching

Table 4's numbers describe one request's throughput on otherwise-idle hardware — the simplest possible serving scenario, and worth naming as a simplification. Session 08's Chapter 1 introduced the batch-size multiplier in the general KV-cache formula: total cache bytes scale with batch_size directly, on top of whatever seq_len is doing. A production server rarely runs one request at a time; it batches many concurrent conversations onto the same GPU to keep the hardware busy. Every gigabyte this chapter's compression frees up at a given context length is a gigabyte a busy server can instead spend on more concurrent requests at that same context length, not just a faster single request. The throughput ratios in Table 4 are therefore a conservative lower bound on TransMLA's real production value — the advantage compounds again, in a dimension this specific experiment didn't even measure, the moment a service has more than one user.

How much more context does the same 24 GB card actually buy?

Read the OOM boundaries themselves as a measurement, not just a warning sign. On the 165.2 TFLOPS / 24 GB tier, the original model runs at 8K but OOMs by 16K — its ceiling sits somewhere in that range. TransMLA runs at 16K but OOMs by 32K — its ceiling sits one whole doubling further out, on the identical card. That's at least a 2× increase in maximum servable context length, even though the cache itself shrank by roughly 14× (Chapter 4's 512-to-36-KiB-per-token ratio). The gap between “14× smaller cache” and “2× more context” is explained by everything else sharing that same 24 GB budget: Session 08 priced Llama-2-7B's weights alone at roughly 13 GiB in fp16, a fixed cost paid before a single token of cache is ever allocated, plus activation memory on top. Shrinking the cache doesn't shrink that fixed overhead — it just means what memory remains after the fixed costs stretches much further than it used to, which is exactly what “doubling the usable context ceiling, on a cache 14× smaller” looks like once weights are accounted for honestly.

What this means for the next context-length doubling

Extrapolate one step further, cautiously. If a 14× cache reduction bought roughly a 2× extension in usable context length on the smallest tier here, going after an even larger compression ratio wouldn't buy a proportionally larger context extension — the fixed weight and activation overhead this section just priced out doesn't shrink alongside the cache, so it increasingly dominates the remaining budget as the cache's own share of memory keeps falling. This is the same diminishing-returns shape Chapter 0 used to rule out “just buy a bigger GPU” as a permanent fix, seen from the opposite direction: past a certain point, squeezing the cache further runs into a floor set by everything else that has to fit on the card, not by the cache itself.

Completing the ratio table, cell by cell

Table 4 has fifteen cells; the prose above worked through four of them by hand. Finish the rest, so every number the paper reports gets checked, not just a convenient sample:

Read down each column rather than across, and a second pattern shows up alongside the growing-with-context one already named above: on the smallest, most memory-constrained card (165.2 TF | 24 GB), the ratio climbs fastest and furthest — 4.7× up to an undefined, infinitely-better 16K — while the two larger cards top out lower, at 6.4× and 5.0× respectively, even at the longest context both can still complete. Less headroom to begin with means the compressed model's advantage has more room to grow before any card runs out of memory to reclaim.

Putting a number on the batching advantage

Turn “every gigabyte freed is a gigabyte for more concurrent requests” into an actual concurrent-request count, using figures already established elsewhere in this session. Session 08 priced Llama-2-7B's weights at roughly 13 GiB in fp16; on the 24 GB (165.2 TFLOPS) card from Table 4, call the remaining roughly 11 GiB (ignoring activation memory, for a clean order-of-magnitude estimate) the budget available for concurrent request caches at 8K context:

That's the number Table 4's single-request throughput figures were always a conservative floor for. A card that could barely hold three simultaneous 8K conversations under the original model can hold roughly forty under the compressed one — not because any single conversation got faster in this framing, but because the same fixed memory budget now stretches across far more of them at once, which is exactly what a production server, busy with many users rather than one, actually needs.

Why the paper opens by calling this a communication problem

TransMLA's own abstract opens with a framing worth sitting with, because it's the thesis this entire chapter has been demonstrating with real numbers: “modern large-language models often face communication bottlenecks on current hardware rather than computational limitations.” Table 4's numbers are exactly what that sentence predicts — the FLOPs a GPU can execute per second barely factor into where these throughput gaps come from; moving cached bytes on and off memory does. And because the conversion targets DeepSeek's own codebase directly, a converted model doesn't just get the vLLM speedup measured here — the paper notes it becomes eligible for DeepSeek's further optimizations too, including FP8 quantization and Multi-Token Prediction, and serving through SGLang as well as vLLM. Table 4 measures the floor of what conversion buys on stock vLLM alone, not the ceiling of what the wider DeepSeek ecosystem eventually makes possible once a model is speaking its format.

Chapter 6: DeepSeek's Second Wall

Picture an agentic coding assistant working through a large repository: it has read 100,000 tokens of source files, commit history, and its own prior tool calls before it writes a single new line. Every next token it generates has to weigh that entire history — not summarize it, not sample from it, actually compare against every cached key in it, at every single decoding step, for as long as the session runs. This is not a contrived edge case; it is what a long-context agentic workload looks like in production, and it's exactly the regime DeepSeek-V3.2 targets with a 128K (131,072-token) context window.

Everything through Chapter 5 was about the cache: how many bytes get stored per token, and how much that costs in memory. Chapter 0 promised a second wall, orthogonal to the first, and this is where it becomes concrete. Even a model with an already-tiny, already-compressed cache still does one thing at every single decoding step that a small cache does nothing to fix: it compares the current query against every cached key, one by one, to decide what to attend to. At 131,072 tokens of context, that is 131,072 comparisons, for every new token generated, no matter how few bytes each cached entry occupies.

Even MLA's own cache, priced out honestly

Ground this in DeepSeek's own real architecture, the one both this session's papers build on. DeepSeek-V3's published configuration: 61 transformer layers, hidden dimension 7,168, n_h = 128 query heads, d_h = 128 per head, KV compression dimension d_c = 512, and decoupled RoPE dimension d_h^R = 64. What gets cached per token, per layer, is the latent plus the small RoPE key:

Compare that against Session 08's dense Llama-2-7B baseline, 512 KiB/token:

MLA is already doing real work here, before DeepSeek Sparse Attention (DSA) enters the picture at all. And it is still not enough at 128K context, because none of these 68.6 KiB/token numbers say anything about how many of those cached entries get compared against on every step. That number is still, for plain dense attention, the full length of the sequence so far — L.

Scale matters here too, and it's worth naming: DeepSeek-V3's underlying architecture is a 671-billion-parameter Mixture-of-Experts model, of which 37 billion are activated for any single token, spread across 256 routed experts plus one shared expert (8 routed experts active per token). At this scale, every inefficiency is multiplied by an enormous serving footprint — a compute wall that costs a few extra milliseconds per token on a small model becomes a genuinely large recurring bill across a production deployment of this size, run across however many concurrent 128K-context sessions a service like this actually handles.

The lightning indexer

DeepSeek-V3.2 (arXiv:2512.02556) introduces exactly one architectural change over its predecessor, DeepSeek-V3.1-Terminus: DeepSeek Sparse Attention (DSA). Its first component is a lightning indexer — a small, cheap side-computation that scores every candidate key against the current query, before the expensive main attention ever runs:

H^I is the number of indexer heads — a small number, 64 in DeepSeek-V3.2's actual configuration, tiny next to the main model's 128 attention heads. Each indexer query q_t,j^I and key k_s^I live in a modest d^I = 128-dimensional space. The paper explicitly chooses ReLU instead of the usual softmax for throughput — ReLU is one comparison and a clamp, no exponentials, no normalization pass over the whole row before you can even start comparing candidates. The indexer is deliberately built to be run in FP8, the cheapest numeric format the hardware supports, precisely because it is going to be run a lot.

Notice what ReLU buys, specifically, over the softmax the main attention mechanism uses. Softmax needs every candidate score computed first, then a max subtracted from all of them for numerical stability, then an exponential applied to each, then a sum across the whole row, then a division — a full second pass over every candidate before you have a usable score. ReLU needs none of that: max(0, x) is computed the instant a single score arrives, independently of every other candidate, with no row-wide reduction required at all. For a computation that has to touch L candidates at every one of L query steps, skipping a row-wide normalization pass is not a minor implementation detail — it's the difference between an algorithm whose per-candidate cost depends only on that candidate, and one that can't finish scoring any candidate until it's seen all the others.

python
import torch

def indexer_score(q_I, k_I, w_I):
    # q_I: (H_I, d_I) indexer queries for this token, one per indexer head
    # k_I: (d_I,)     indexer key for one candidate position s
    # w_I: (H_I,)     per-head weights for this token
    dots = (q_I * k_I).sum(dim=-1)          # (H_I,) -- one dot product per indexer head
    return (w_I * torch.relu(dots)).sum()  # scalar I_{t,s} -- no softmax, no row-wide pass

# H_I=64, d_I=128 in DeepSeek-V3.2's real configuration -- tiny next to n_h=128 main heads

Fine-grained token selection

Given the index scores for query token h_t against every earlier position s, keep only the k highest-scoring entries and run real attention over just those:

DeepSeek-V3.2's configuration sets k = 2,048. Read that against the full 131,072-token (128K) context: at the far end of a full context window, a query attends to roughly 1.56% of its available history (2,048 ÷ 131,072) — chosen freshly, differently, for every single query, by the indexer's scores, not by any fixed positional rule.

Worked arithmetic: how many fewer comparisons, at 128K, precisely

Two honest ways to measure the saving, both from real, stated numbers. First, a single decoding step at the very end of a full 128K context: dense attention compares the new query against every one of the 131,072 cached positions; DSA compares it against only the top 2,048.

Second, sum the comparisons across an entire 131,072-token generation, causally — query t can only ever see positions 1 through t, so early queries see fewer candidates than late ones. Dense attention's total, summed over every position:

DSA's total: every query still sees min(t, k) candidates — the full history until the cache fills past k = 2,048 positions, capped at k after that:

Both numbers describe the same phenomenon from different angles: a single late-context step sees the full 64× (L/k, exactly — a clean power of two, since both 131,072 and 2,048 are themselves powers of two), while the whole-sequence average is smaller (32.3×) because early positions, where t < k, aren't sparse yet at all — there simply aren't more than k tokens to select from until the context grows past k.

A sanity check on the formula itself, at a scale you can verify by hand

Before trusting a formula at billions of pairs, run it at a scale small enough to count by hand. Take L = 10, k = 3 — a toy 10-token sequence with a top-3 selection budget. Dense: ∑_t=1¹⁰ t = 10×11/2 = 55 pairs. DSA: the first 3 queries (t = 1, 2, 3) each see fewer than k candidates anyway, contributing 1+2+3 = 6; the remaining 7 queries (t = 4 through 10) are each capped at exactly k = 3, contributing 7×3 = 21. Total: 6 + 21 = 27 pairs, and 55 ÷ 27 ≈ 2.04×. Small L, small savings — exactly consistent with Chapter 6's earlier point that sparsity barely matters until L grows well past k. Plug L = 131,072, k = 2,048 into that same two-term structure and the identical formula produces the 266,339,328 above; the arithmetic didn't change between the toy and the real scale, only the numbers feeding it.

That last point is worth being honest about as a limitation, not just a footnote: for any conversation shorter than k = 2,048 tokens, DSA has nothing to filter — every cached token is already within the selection budget, so the top-k step selects everything, identically to dense attention, while still paying the indexer's own (small, but nonzero) O(t²) scoring cost to reach that conclusion. Sparse attention's entire benefit is a long-context story; on short contexts it adds a bit of pure overhead for zero selection benefit, which is exactly why the paper frames this as a fix for “long-context scenarios” specifically, not a universal free upgrade to every request regardless of length.

Prefill and decode aren't the same computation

Everything in this chapter's worked arithmetic describes the causal, position-by-position accounting of a full generation. It's worth separating two phases that get lumped together casually but behave differently in practice. Prefill processes an entire prompt at once, before generation starts — all L prompt positions' worth of attention computed together, highly parallelizable, because every position's key and value are already known simultaneously. Decode generates one token at a time afterward, each step sequential, each step paying for exactly the query-key comparisons this chapter counted position by position. The paper is explicit that it treats these differently: for short-sequence prefilling specifically, it runs a specially implemented masked dense-attention mode simulating DSA's selection pattern, rather than paying the indexer's overhead on a phase that's already efficient at short lengths. The 62.5× and 31.5× savings this chapter derived are decode-side numbers — the phase where sequential, one-token- at-a-time generation makes every avoided comparison a real, unhideable latency cost, unlike prefill's parallel-friendly structure.

A question worth sitting with before Chapter 7 answers it

There's a much simpler alternative sitting right there, and it's worth asking why DeepSeek didn't take it: why not just keep the most recent 2,048 tokens, the way Session 08's StreamingLLM keeps a fixed window, instead of building an entire trained indexer to dynamically pick which 2,048 tokens to keep? A fixed window is dramatically simpler to implement, requires no training at all, and produces the identical O(Lk) compute saving this chapter just derived — k tokens attended per step, capped, flat cost. If the only goal were the compute number, the window would already get you there for free.

The gap between those two designs is entirely about which k tokens survive, not how many. A fixed window's k tokens are chosen by one rule — recency — applied identically at every step, regardless of content. The indexer's k tokens are chosen freshly by content, different for every query, and nothing is ever permanently evicted the way a window discards its oldest entries. Whether that difference is worth an entire training pipeline — the lightning indexer, the two-stage distillation, all of Chapter 7 — depends on whether long-range, content-driven recall actually matters for the workloads a 128K-context model is built to serve. Chapter 8 makes that comparison explicit; for now, hold onto the fact that DeepSeek chose the harder-to-build option on purpose, not by default.

What this costs in real dollars, on DeepSeek's own service

Everything in this chapter's arithmetic so far has been comparisons of counts — keys examined, query-key pairs, ratios. DeepSeek-V3.2's own paper puts an actual price on the difference, not just a count: the inference costs it reports for both DeepSeek-V3.1-Terminus (dense) and DeepSeek-V3.2 (with DSA) are measured directly off a real deployed service, running on H800 GPU clusters rented at 2 USD per GPU-hour, tracked separately for prefill and decode — because, as the next section makes explicit, those two phases behave differently under sparsity. This is worth pausing on precisely because it's unusual: most of what this session has covered is a research paper's own reported benchmark. This one number — a real-world hourly rental rate applied to a real production deployment — is where the arithmetic stops being an academic exercise about complexity classes and becomes a literal, auditable operating cost a customer of that service is paying, or not paying, per token.

One detail the paper is careful to flag alongside those cost curves, because it would otherwise look like an inconsistency: for short-sequence prefilling specifically, DeepSeek doesn't run the trained lightning indexer at all. It runs a specially implemented masked dense-attention mode that simulates DSA's selection pattern, because at short prompt lengths that's more efficient than paying the indexer's own overhead on a phase that's already cheap. Sparsity, in other words, isn't applied uniformly by rule — it's applied where the paper's own measurements say it actually pays for itself, and skipped where it wouldn't.

A third point on the curve, between the two extremes

The worked arithmetic above jumped straight from L = 1,000 to L = 131,072 — a useful bookend comparison, but worth filling in a middle value to see that the growth isn't a step function, it's continuous. Take L = 16,384 (16K), a context length real conversations and moderate agentic sessions actually reach on any given day, well short of the full 128K ceiling:

Three points now sit on the same curve: roughly 2× savings at a small toy scale, 4.3× at L = 16,384, and 32.3× at the full L = 131,072. The savings don't arrive all at once past some threshold; they compound smoothly as context grows, because the gap between L and k widens continuously, not in jumps. A service that mostly handles 16K-scale sessions is already capturing a meaningful fraction of DSA's benefit, well before any session reaches the 128K ceiling this chapter has used as its headline number.

Chapter 7: Teaching an Indexer to Watch

A 64-head, 128-dimensional indexer deciding which 2,048 of 131,072 tokens matter, for a query it has never seen before, sounds like it should require its own labeled dataset of “correct” attention patterns. It doesn't. The model already produces exactly that signal, every time it runs dense attention — the indexer just has to learn to imitate it.

The teacher is the model's own dense attention

For a given query token, sum the real attention weights across every one of the main model's heads, then normalize that sum so it forms a proper probability distribution over positions (an L1 normalization — divide by the total so everything sums to 1). Call this target distribution p_t,:. The indexer is trained to match it, via KL-divergence, treating its own scores as a softmax distribution to compare against:

This is distillation: a small, cheap network learning to reproduce the output of a large, expensive one, using the large one's own behavior as ground truth — no external labels required, because the “correct answer” is just whatever the model was already going to pay attention to.

KL-divergence, on numbers small enough to compute by hand

See what this loss actually penalizes before trusting the formula. Suppose, for a toy 3-position history, the model's real (target) attention distribution is p = [0.7, 0.2, 0.1] — it mostly attends to position 1. Say the indexer, before training, produces roughly uniform scores that softmax into q = [0.34, 0.33, 0.33]. KL divergence sums p_i·log(p_i/q_i) across positions:

Now suppose training has nudged the indexer toward q′ = [0.65, 0.22, 0.13], closer to the true distribution:

The loss dropped by roughly 43× from a small shift toward matching the target — because KL divergence penalizes disagreement most heavily exactly where the target distribution places the most weight (the 0.7 term dominates both sums). That's precisely the behavior you want from this loss: get the indexer to agree with the model most where the model cares most, position 1 in this toy example, and don't spend gradient budget perfecting agreement on positions the model was never going to attend to strongly anyway.

Two stages, so the target never lies to the indexer

There's a subtlety worth surfacing: once sparse attention is switched on, the main model's own attention pattern changes too, since it's now computed only over the selected tokens. Training the indexer against a moving target from the very first step would be unstable. DeepSeek-V3.2's continued pretraining, starting from a DeepSeek-V3.1-Terminus checkpoint already extended to 128K context, splits into two stages to avoid exactly that.

Stage 1 keeps dense attention running and freezes every parameter except the indexer, so the target distribution p_t,: is stable and honest — a real, unmodified dense-attention pattern — while the indexer learns to predict it. Only after the indexer is already a reasonable predictor does Stage 2 turn sparsity on and unfreeze the rest of the model, so the main model can adapt to actually running on a sparse subset of tokens, while the indexer keeps refining against the (now sparsity-restricted) target. One more detail the paper is explicit about: the indexer's input is detached from the main computational graph — the indexer trains only on its own KL loss, the main model trains only on the ordinary language-modeling loss, and neither gradient contaminates the other.

Stage 2's loss isn't quite the same formula as Stage 1's — it has to change once sparsity is active, because there is no longer a full dense distribution to compare against for every position. Instead, it restricts the comparison to exactly the selected set S_t = {s | I_t,s ∈ Top-k(I_t,:)}, the same set that Stage 2's real attention is now actually computing over:

Read the subscripts carefully: both p and the indexer's softmax are now evaluated only over the k selected positions, not the full sequence. This keeps the training signal honest in a second way, beyond Stage 1's frozen-model trick — it asks the indexer to correctly rank which of its own top-k picks matter most relative to each other, rather than continuing to grade it against a full L-wide distribution that the model, running sparse now, isn't actually using at inference time anyway.

What "not detached" would actually look like

Make the failure mode concrete rather than abstract. If the KL loss were allowed to flow gradients back into the main model's attention weights, the main model would have two competing incentives at once: produce the attention pattern that best predicts the next token, and produce an attention pattern the small 64-head indexer can easily approximate. Those two goals are not the same thing — a model under pressure to satisfy both might learn to flatten out genuinely useful but hard-to-approximate attention patterns (say, a sharp, unusual spike of attention onto one distant token that a cheap indexer would need many heads to capture accurately) simply because smoothing it out makes the indexer's job easier and the joint loss lower. That is a real quality cost, paid by the main model, to make a side-computation's approximation task more convenient — exactly backwards from what the whole system is for. Detaching removes the incentive entirely: the main model never sees the indexer's loss, so it has no reason to distort its own attention for the indexer's benefit.

Why FP8 specifically, and not just "a smaller format"

It's worth being precise about why cutting the indexer down to FP8 is a reasonable choice rather than a reckless one. The indexer's job is selection — ranking candidates well enough to identify the top 2,048, not computing an exact numeric attention output that downstream layers depend on bit-for-bit. Selection tasks tolerate more numeric noise than value-computation tasks: if FP8's reduced precision nudges a borderline candidate's rank by a small amount, the practical consequence is, at worst, a slightly different set of tokens entering the top-k cut — not a corrupted output the way rounding error in the main attention computation itself would be. That asymmetry — cheap precision where you're choosing among options, full precision where you're computing the answer — is the same principle behind quantizing other selection or routing mechanisms in modern MoE systems, and it's why the indexer, specifically, was a good candidate for the most aggressive numeric format available.

The indexer keeps its own small cache, implicitly

One structural detail the scoring formula implies, even though it's not the paper's own explicit focus: to score I_t,s for a past position s, the indexer needs that position's own key k_s^I — which means it has to have been computed and kept somewhere the moment position s was first processed, just like the main model's cached latent c_s^KV. In other words, DSA introduces a second, much smaller side-cache riding alongside the main one: d^I = 128 dimensions per token, against the main cache's 576 (d_c + d_h^R). At roughly 128⁄576 ≈ 22% of the main cache's per-token width, this second cache is a real but comparatively minor addition to the memory budget — the price of keeping the indexer able to re-score the entire cached history at every future step, exactly the non-destructive property Chapter 8 leans on.

The two ways this can go wrong, and why they aren't equally bad

Any trained selector makes mistakes sometimes, so it's worth reasoning about what an indexer mistake actually costs, in each direction. A false positive — the indexer ranks an irrelevant token into the top-k, bumping out something that mattered less — wastes a small slice of the attention budget on a token that didn't need it, a minor efficiency loss, recoverable the very next step once the indexer re-scores everything fresh. A false negative — the indexer fails to rank a genuinely relevant token into the top-k for this step — means the model's output at that step is computed without information it needed, a correctness cost, not just an efficiency one. That asymmetry is exactly what the KL-divergence training objective is aimed at minimizing: matching the real attention distribution isn't about getting every score exactly right, it's about not systematically under-ranking the tokens dense attention would have leaned on most heavily — the high-probability mass in p_t,: that a good indexer has to reproduce faithfully, even if it's a little noisy everywhere else.

Bridging back to Session 03: DSA runs in MLA's MQA mode

One more connection worth making explicit. Recall from Session 03 that MLA can operate in two computational paradigms: an MHA-like mode (used during training and prefill, where the full-width reconstruction is cheap relative to the parallel compute available) and an MQA-like mode via the Absorb operation (used during decoding, where one cached latent is shared across all query heads at once). DeepSeek-V3.2's paper states plainly that DSA is instantiated specifically under MLA's MQA mode, because at the kernel level, each selected key-value entry needs to be efficiently shared across every query head that selected it — exactly the sharing pattern the MQA mode was already built for. DSA is not a separate attention mechanism bolted onto MLA; it is a selection rule layered on top of the exact decoding path Session 03 already derived.

Why the indexer, despite O(L²), stays cheap

The paper is upfront that the indexer itself still costs O(L²) — it has to score every candidate pair to know which ones to keep, so it can't avoid looking at the whole quadratic space. What it avoids is doing expensive work over that whole space. Compare the raw arithmetic width of one score computation in each path. The indexer sums across H^I = 64 heads of width d^I = 128:

The main model, in its MQA/Absorb decode mode, must dot each of its n_h = 128 query heads against the shared cached representation of width d_c + d_h^R = 512 + 64 = 576:

Scoring a candidate with the indexer costs roughly a ninth of what actually attending to it costs — on top of running in a cheaper numeric format. That gap is why it is affordable to run the indexer over the full 131,072² candidate space every step, and spend real attention's much larger per-pair cost only on the 2,048 candidates that survive.

Does the imitation actually hold up? DeepSeek's own parity check

Everything in this chapter so far has been about how the indexer is trained — the loss it optimizes, the two stages, the detached gradient. None of that answers a more basic question: once training finishes, does the resulting sparse model actually perform as well as the dense one it's imitating, on real evaluations, not just on a KL-divergence number going down in training? DeepSeek-V3.2's paper runs exactly this check, calling it a parity evaluation, comparing the newly-sparse DeepSeek-V3.2-Exp directly against its dense predecessor, DeepSeek-V3.1-Terminus. Three separate pieces of evidence, three different methodologies:

Read that middle row carefully, because it's the strongest kind of evidence available for a claim like this: ChatbotArena scores aren't produced by the paper's own authors grading their own model against a benchmark they chose; they come from real users, blind-comparing model outputs, aggregated into an Elo rating the same way chess rankings work. A sparse model whose Elo score is indistinguishable from its dense predecessor's is a sparse model that real people, with no idea which version they're talking to, can't tell apart from the original in ordinary use. The bottom row goes one step further — on an independent long-context reasoning benchmark run by a third party, after the fact, the sparse model doesn't just match the dense one, it edges ahead. A separate long-context evaluation (Fiction.liveBench) reports the same story: the sparse model consistently outperforms the dense one across multiple metrics there too.

Connect this back to the false-negative risk named earlier in this chapter: if the indexer were systematically dropping genuinely relevant tokens from its top-k picks, that failure would show up first and most visibly on exactly the kind of benchmark AA-LCR measures — long-context reasoning, where getting the right distant token wrong actually breaks the answer. That the sparse model instead scores higher than dense attention on this specific benchmark is the empirical version of this chapter's shared claim with Chapter 6: the indexer's false negatives, in aggregate, are not derailing the tasks that would be most sensitive to them.

Put this next to the KL-divergence toy example earlier in this chapter. A shrinking training loss is evidence the indexer is learning to imitate the target distribution; it is not, by itself, proof the resulting system performs as well downstream. Parity evaluation is that separate, harder-to-fake check — the same discipline Chapter 5 applied to TransMLA's throughput claims (cross-checking the paper's own numbers before trusting them), now applied to DSA's quality claims instead of its speed claims.

The indexer's training is distillation, in the strict textbook sense

Worth naming this precisely, since the vocabulary matters and this session used it once already, in a different context. Chapter 0 dismissed full model distillation — training a brand-new, smaller MLA model from scratch to imitate a GQA model's outputs — as too expensive an alternative to conversion, because a student learning an entire language model's behavior from nothing is a genuinely large undertaking. The lightning indexer's own training is also distillation, in the exact same technical sense: a small network learning to reproduce a large network's output distribution, using the large network's own behavior as the only supervision signal, no external labels anywhere. What makes it cheap where full-model distillation was expensive isn't a different technique; it's a dramatically narrower target. The indexer isn't learning to write coherent text, reason, or represent a task; it's learning a single, comparatively low-dimensional scoring function — which of up to 131,072 candidates a given query should attend to. A 64-head, 128-dimensional scoring function is a vastly smaller object to distill than an entire language model, which is exactly why this particular distillation costs a warm-up stage measured in billions of tokens, not the multi-trillion-token budget Chapter 0 ruled out for the harder version of the same idea.

Chapter 8: Latent, Sparse, or Both

Eight chapters ago, this session opened with an infrastructure lead deciding whether to retrain a GQA model from scratch. It closes with a sharper, more useful question: given a real deployment, which of these two techniques — TransMLA's latent compression, or DeepSeek's sparse selection — do you actually reach for? The honest answer is that they solve different problems, and the clean way to see that is to be precise about what each one shrinks.

Two different nouns being shrunk

Look at DeepSeek-V3.2's §2 again with this framing in hand, and a fact from Chapter 6 becomes obvious rather than surprising: nowhere in that section does the paper claim DSA reduces memory. It can't — the indexer needs every cached latent c_s to remain available as a candidate, every single step, in case this step's query happens to select it. Sparsity here lives entirely in which computations run, never in what stays resident. That is precisely why DSA is instantiated under MLA rather than beside it or instead of it: MLA already shrank what each candidate costs to store; DSA shrinks how many of those already-cheap candidates get the expensive full attention treatment.

Stack the two multipliers and see what DeepSeek's own production model gets

Chapter 6 computed two separate numbers for DeepSeek-V3's real, published configuration: MLA's cache runs 7.5× smaller than a dense-Llama-2-7B-style baseline (512 KiB/token down to roughly 68.6 KiB/token), and DSA's top-k selection runs decode-side attention over roughly 32.3× fewer query-key pairs across a full 131,072-token generation than dense attention over that same cache would need. Those are two independent multipliers, on two independent axes — storage and compute — and nothing about deriving either one assumed the other. Neither one combines into a single clean product the way Chapter 4's compression ladder multiplied cleanly against Session 08's baseline, because they answer different questions for different consumers of the number: a systems engineer sizing a GPU fleet's memory footprint cares about the 7.5× storage figure; the same engineer estimating latency and GPU-hour cost per generated token cares about the 32.3× compute figure.

The honest way to state DeepSeek's achievement isn't one multiplier at all — it's that the model pays roughly an eighth of the storage cost and roughly a thirty-second of the comparison cost that a dense, uncompressed 128K-context model would have paid, and it pays both of those reduced prices independently, because the two techniques attack genuinely different resources. A team that only converted to MLA and stopped there would still be paying the full quadratic comparison cost on every long session; a team that somehow had sparse selection without any cache compression would still be paying the full per-token storage cost on every cached entry, sparse or not. Each technique closes exactly the gap the other one leaves open.

A sharper contrast: DSA vs. Session 08's window attention

Both DSA and Session 08's StreamingLLM cap how many keys get attended to per step at some constant, and both therefore turn a growing-with-context cost into a flat one. It would be easy to mistake them for the same idea wearing different notation. They are not, and the difference matters for what each one can and can't do.

Concretely: if a conversation returns to a topic from 90,000 tokens ago, StreamingLLM has no way to recover that content — it was evicted from the cache long ago, and Session 08's Chapter 9 was explicit that this is a real, permanent limitation, not a bug to be patched. DSA's indexer, by contrast, re-examines that 90,000-token-old latent on every new query, right alongside everything else, and will select it again the moment it scores highly — because it was never thrown away, only skipped on steps where something else scored higher. Windowing trades away long-range recall for its flatness; sparse selection keeps long-range recall available, and pays for its flatness with the cost of scoring the whole history every step (Chapter 6's affordable O(L²) indexer, not a free O(L) truncation).

Choosing, in practice

Two deployment scenarios, worked through with this session's own numbers

Abstract comparison tables are useful for orientation; a concrete scenario, run through the arithmetic this session has actually built, is what turns “it depends” into an answerable question.

Scenario A: a customer-support fleet, 500 concurrent 4,000-token conversations, on 165.2 TFLOPS | 24 GB cards. At 4K context, one dense conversation's cache costs 4,000 × 512 KiB = 2,048,000 KiB (≈ 1.95 GiB); the TransMLA-compressed version costs 4,000 × 36.0 KiB = 144,000 KiB (≈ 137 MiB). Scale both to the full 500-conversation fleet, against the roughly 11 GiB of per-card headroom Chapter 5 estimated after weights:

88.8 ÷ 6.24 ≈ 14.2× — the identical ratio Chapter 4's compression rung produced, now carried all the way through to an actual GPU count: roughly 89 cards' worth of cache headroom under the original model versus roughly 7 under the converted one, for the exact same 500-conversation workload. That's not a minor efficiency tweak at this scale; it's the difference between a rack-scale deployment and a handful of cards, purely from what each conversation's cache costs to hold resident, before compute is even part of the conversation.

Scenario B: one 128,000-token overnight code-review agent session, on whatever hardware is already provisioned. Here the calculus is different, because there's one session, not five hundred. A single dense-MHA cache at 131,072 tokens costs 512 × 131,072 KiB ≈ 64.0 GiB — large, but not impossible on a multi-GPU node. DeepSeek's own native MLA cache at 68.6 KiB/token shrinks that same session to roughly 8.6 GiB, comfortably fitting almost anywhere. Cache capacity, in other words, was never really the bottleneck for this single-session scenario the way it was for Scenario A's five hundred concurrent conversations. What actually dominates this scenario's cost is Chapter 6's 32.3× reduction in query-key comparisons across the full generation — because a single long agentic run's cost isn't set by whether its cache fits in memory, it's set by how many decoding steps that one session has to pay for, sequentially, one at a time, for as long as it keeps running. A fleet optimized purely for cache compression, with no sparse selection layered on top, would still pay the full quadratic comparison cost this single long session accumulates; DSA is the piece built specifically for this scenario, not the cache-compression half of this session.

The two scenarios don't just illustrate the table above — they explain, with real numbers from this session's own arithmetic, exactly why a company running mostly Scenario-A-shaped traffic reaches for latent compression first, why a company running mostly Scenario-B-shaped traffic reaches for sparse selection first, and why DeepSeek, whose production traffic almost certainly includes both shapes at once, ships both stacked together rather than picking one.

Turning the decision table into code

This session's own concept-plus-realization discipline means writing that decision down as code, not leaving it as a qualitative table entry. Here's the same “choosing in practice” table above, as an actual function an infrastructure team could run against their own traffic numbers — using nothing but the coefficients this session already derived:

python
def recommend_technique(concurrent_requests, avg_context_tokens, gpu_mem_gib=24, weights_gib=13):
    """Rough sizing heuristic built from this session's own derived ratios --
    not a substitute for real profiling, but the right FIRST question to ask."""
    headroom_gib = gpu_mem_gib - weights_gib
    dense_kib_per_tok, compressed_kib_per_tok = 512.0, 36.0   # Chapter 1 / Chapter 4
    dense_cache_gib = (concurrent_requests * avg_context_tokens * dense_kib_per_tok) / 2**20
    fits_dense = dense_cache_gib <= headroom_gib

    long_context = avg_context_tokens > 16384          # Chapter 6: DSA's benefit compounds past here

    if not fits_dense and not long_context:
        return "latent compression (TransMLA) -- memory-bound fleet, moderate context"
    elif fits_dense and long_context:
        return "sparse selection (DSA) -- compute-bound single sessions, memory has headroom"
    elif not fits_dense and long_context:
        return "both, layered -- DeepSeek-V3.2's own situation"
    else:
        return "neither is urgent yet -- dense fits, context is short"

print(recommend_technique(500, 4000))    # Scenario A above -> latent compression
print(recommend_technique(1, 131072))    # Scenario B above -> sparse selection

Run Scenario A's numbers through it: 976.6 GiB of dense cache against 11 GiB of headroom doesn't fit, and 4,000 tokens of context isn't long by this session's own definition — the function returns latent compression, matching the worked arithmetic above exactly. Run Scenario B's: a single 64 GiB dense cache fits inside even a modest multi-GPU node's headroom, but 131,072 tokens is well past the long-context threshold — the function returns sparse selection, again matching. The two scenarios weren't chosen to make the heuristic look good after the fact; the heuristic is just this session's own reasoning, written down as a function instead of left as prose.

Closing the loop: Chapter 0's day-long chat, one more time

Chapter 0 never used a number this large; Chapter 4 revisited it once, briefly. Worth returning to it a final time, now that both techniques are on the table together, because it's the same scenario this entire session opened with. An 8-hour working day of steady conversation, Session 08's original estimate, runs to roughly 48,000 tokens — short of even this chapter's 16,384-token “long context” threshold, let alone DSA's 131,072-token design point. Under TransMLA's most aggressive rung alone, that day's cache costs roughly 1.65 GiB (Chapter 4's own number) — comfortably resident, no sparse selection required, no compute wall in sight. This is worth being honest about directly: most of what this session built is not, in fact, needed for most conversations most people have with a model. It becomes necessary exactly at the scale this session kept pointing toward — concurrent fleets in the hundreds, single sessions in the hundreds of thousands of tokens — and comfortably unnecessary below it. Knowing where that line sits, concretely, in gigabytes and token counts rather than vaguely, is what turns “use MLA and sparse attention” from a cargo-culted default into an actual engineering decision.

What this session did not cover, honestly

TransMLA's own conclusion names its remaining limitation directly: the Balanced Key-Value technique from Chapter 4, while effective, is described by the authors themselves as “relatively trivial,” and they pose an open question — whether more powerful mathematical tools could close the gap further and make training-free conversion viable even at the most aggressive compression ratios. DeepSeek-V3.2's paper, for its part, spends most of its length on reinforcement learning and agentic tool-use training that this session deliberately left out — this session covered exactly §2, the architecture, and nothing about how the model was subsequently trained to reason or use tools. Both omissions are intentional scoping, not gaps in the source material.

TransMLA's own paper points at exactly this stacking

It's worth closing the loop with what TransMLA's authors themselves name as left to do, because it points directly at this session's second half. The paper's own conclusion states that TransMLA “should be integrated with pruning, quantization, token selection, and other optimization techniques to fully explore the upper bounds of inference acceleration” — naming token selection specifically, by name, as a complementary direction alongside latent compression. That sentence was written about the idea in general, not about DeepSeek Sparse Attention by name — TransMLA and DeepSeek-V3.2 are separate papers, on different timelines, neither citing the other as a dependency. But the shape of the claim is exactly what Chapters 6 through 8 just demonstrated concretely: latent compression and token selection are not competing answers to the same question, they're compatible answers to two different questions, and DeepSeek's own production stack is a real, shipped instance of stacking them, not a hypothetical the authors were merely gesturing toward.

For completeness, DeepSeek-V3.2's own conclusion names three further limitations, and it's worth being explicit that none of them are about DSA or the architecture this session covered: narrower breadth of world knowledge than frontier closed-source models (attributed to fewer total training FLOPs, not to sparse attention), lower token efficiency in reasoning chains (needing more generated tokens to reach comparable answer quality), and weaker performance than frontier models on the hardest tasks. All three are properties of the post-training, reasoning, and pretraining-scale choices this session deliberately scoped out from the start — not consequences of DSA or MLA. Keeping that distinction sharp matters: a reader who conflated “DeepSeek-V3.2 has a smaller knowledge base than a frontier closed model” with “sparse attention hurts quality” would be drawing a causal link the paper itself never makes, and Chapter 7's parity evaluation directly contradicts.

Continue with Session 03: MLA & Architectural Choices for where the down/up-projection and the Absorb trick were first derived, or Session 08: Attention Sinks & Streaming for the destructive alternative to sparse selection this chapter contrasted against.

“Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away.”

TransMLA takes that literally at the level of a matrix: the same information, represented in fewer stored numbers. DSA takes it literally at the level of a step: the same quality of answer, computed by looking at fewer of the numbers you already have. Neither one adds capability the model didn't already have. Both simply refuse to pay for what the model was never going to use.