Architecture teardown · 2019 → 2026

From GPT-2 to Kimi K3: one memory, seven eviction policies.

The headline number says 22,580 GPT-2s fit inside Kimi K3. The honest numbers are more interesting — and the real story isn't scale at all. It's that a language model is an associative memory, a fixed-size memory must eventually evict, and seven years of architecture research is the search for a learned eviction policy. Every architecture in this lesson is one edit to a single update equation, and you will watch the same four-dimensional memory succeed, blur, overwrite, fade, and survive under each one.

SPAN 6 architectures DEPTH hand-arithmetic to code WIDGETS 10 interactive CHAPTERS 9

00Concept constellation

Every idea in this lesson, drawn as a map. Amber is the memory frame, terracotta is update rules, blueprint is hardware & parallelism, moss is the full system. Hover to preview; click to jump to its chapter. Nodes with a blue ring have their own dedicated lesson on this site — shift-click (or ⌘-click) to open it.

The timeline reads like a parade of paper titles: linear attention, fast weight programmers, DeltaNet, Gated DeltaNet, Kimi Delta Attention, Kimi K3. A parade tells you what happened. It never tells you why each thing had to happen — what specific failure of the previous idea forced the next one. That is what this map is for. Every arrow means "this problem created that solution."

the memory frame update rules hardware & parallelism the full system has its own lesson
The single sentence that organizes everything on this map: a fixed-capacity associative memory needs an eviction policy, and the last seven years of sequence-model research is the search for a learned one. GPT-2 evicts nothing and pays rent on every token forever. Linear attention evicts by accident. The delta rule evicts one fact at a time. Gates evict everything a little. Per-channel gates evict with resolution. And Kimi K3 runs four different policies at once, each where it is cheapest.

00bThe four questions

Every architecture in this lesson will be forced to answer the same four questions. The answers are the whole story.

question 1

What does it store?

A growing list of every key and value it has ever seen? Or a fixed-size matrix that must summarize?

question 2

How does it write?

Append a new slot? Add an outer product on top of everything? Erase first, then write?

question 3

How does it read?

Softmax over every stored key — sharp, expensive? Or one matrix multiply against the summary — cheap, blurry?

question 4

What gets evicted?

Nothing? Everything, slowly? One fact, precisely? Each channel at its own speed? This question is the plot.

As each chapter answers them, a running scorecard accumulates one row per architecture. By the coda you will read the completed table and see the entire seven-year arc in eight rows. Here is the row every other row is measured against:

The scorecard · row 1 of 8
ArchitectureStoresWriteReadEviction
GPT-2 (softmax + KV cache)every K,V ever seen — grows O(N)append a slotsoftmax over all N keysnone. Perfect memory, unbounded rent.

00c22,580 — held honestly

The number is real. It is also three different comparisons wearing one trenchcoat. Let's take the coat off before we start.

Kimi K3 reports 2.8 trillion total parameters. The GPT-2 that ships in nanoGPT — the one whose code opens Chapter 1 — is 124 million. Divide and you get 22,580. True, and worth holding. But hold the asterisks with it:

  • 124M is GPT-2 small. The model the 2019 paper actually headlines is the 1.5B-parameter XL. Against that, the ratio is about 1,867×.
  • 2.8T is a sparse total. K3 is a mixture-of-experts that activates roughly 16 of 896 routed experts per token (plus shared ones). Per-token active parameters are on the order of tens of billions — so the compute ratio against GPT-2 small is on the order of hundreds, not tens of thousands. Chapter 7 has a live calculator where you can do this arithmetic yourself.
  • Neither number is the interesting one. The interesting number is how much of the gap is new ideas versus more of the same. By raw parameter count, K3 is ~95%+ expert FFNs — an idea whose structure dates to 2017. The sub-1% — the eviction policy, the hybrid read path, the depth-wise attention — is what decides whether the other 95% is even reachable at million-token contexts. Scale bought the library. The new ideas bought the librarian.
This lesson is a rebuild of a genuinely good worklog by ali (@waterloo_intern) that traced this same arc. We keep its spine and its best moments, fix its bugs (every code listing here actually runs — several of the original's did not), spell out every skipped step, and replace every static figure with something you can drag. Where a Kimi K3 detail comes only from the worklog or Kimi's release notes and we could not independently verify it, it is marked reported.

00dHow to read this lesson

Nine chapters, ten widgets, one running example that never changes.

Chapters 1–3 are the memory story: what attention actually stores (Ch 1), what happens when you compress the store into a fixed square (Ch 2), and how to erase from that square with linear algebra (Ch 3). Chapter 4 is the hardware interlude — how the eraser was made trainable at scale; if you only care about the memory story you can skim it and keep its one-line conclusion. Chapters 5–6 give the memory knobs for forgetting — one knob, then one per channel. Chapters 7–8 assemble the full system: Kimi K3's stack, and the beautiful final twist — the residual stream itself has the same disease, and gets the same cure.

The running example: Ada and Mochi

One tiny story is re-processed by every architecture in this lesson:

"Ada adopted a cat named Mochi. [… 10,000 tokens of other things …] Ada's cat is named ____."

A fact is written early and queried late. Whether the answer survives — and what it costs to keep it alive — is exactly what distinguishes these architectures. We run it in a toy dimension of $d=4$ so every matrix fits on your screen and every multiplication can be checked by hand. Real models use $d_{\text{head}} = 64$–$128$; nothing about the story changes, only the size of the square.

Prerequisites, honestly stated. You should know what a transformer is at the level of "attention mixes information across positions, MLPs transform it, gradient descent trains it." If that sentence is shaky, read the attention & transformers lesson first. Everything else — linear attention, delta rules, gating, MoE, MLA — is built from zero right here.

Notation, fixed once. Batch and head dimensions are dropped everywhere (they just come along for the ride). A key $k$, query $q$, value $v$ are row vectors of size $1 \times d$. The state $S$ is a $d \times d$ matrix. Writing is $S \mathrel{+}= k^{\top} v$ (an outer product), reading is $o = qS$. Sequence length is $T$, chunk size is $C$. Some papers write everything transposed ($S$ as $d_v \times d_k$, read as $Sq^{\top}$) — same math, flipped convention; we flag it once in Chapter 3 and then never mix them.

00eConcept index

The eighteen ideas on the map, one line each.

CH 1
memory frame

Attention is a memory

K rows are labels, V rows are contents, softmax is the read. The reframe everything else stands on.

CH 1
memory frame

The KV cache

Past keys and values never change, so store them. Decode goes quadratic→linear per step.

CH 1
memory frame

The memory wall

37 MB at 1K tokens is cute. At 1M tokens the cache outweighs the model — and you re-read it every step.

CH 2
update rules

Re-association

Remove the softmax and the parentheses move: $(qK^{\top})V$ becomes $q(K^{\top}V)$. History folds into a $d{\times}d$ square.

CH 2
update rules

Interference

A $d{\times}d$ matrix holds at most $d$ clean facts. Fact number $d{+}1$ lands on top of the others.

CH 2
hardware

FlashAttention, correctly

An IO-aware exact kernel — it changes memory traffic, not asymptotics. The KV cache is what kills per-step quadratic cost.

CH 3
update rules

The delta rule

Read what the key currently retrieves, subtract it, write the difference. An eraser made of algebra.

CH 3
update rules

Fast weight programmers

Schlag's 2021 name for the same idea: the state is a weight matrix that inference itself trains.

CH 4
hardware

Chunked training

Exact attention inside C-token tiles, one folded state across tiles. C is a dial between recurrence and attention.

CH 4
hardware

The T-matrix trick

The delta rule's within-chunk dependency chain is a small triangular solve. That one observation buys parallel training.

CH 5
update rules

Decay gating

One scalar $\alpha_t$ multiplies the whole state each step. Forget-everything-a-little, on demand.

CH 5
update rules

Gated delta rule

Gate for bulk forgetting, delta for precise replacement. Each fixes exactly what the other cannot.

CH 6
update rules

KDA: a knob per channel

$\alpha_t$ becomes a $d$-vector. Fast facts and slow facts share one matrix at different half-lives.

CH 6
full system

MLA

Full attention with a compressed cache: store one small latent per token, expand K and V on demand.

CH 6
full system

The 3:1 hybrid

Three KDA layers per MLA layer. Constant-state memory plus periodic photographic recall.

CH 7
full system

Latent MoE

898 experts, ~18 awake per token, running in a compressed space. Capacity where compute isn't.

CH 7
full system

Kimi K3

23 macrocycles of KDA·KDA·KDA·MLA with latent MoE everywhere and AttnRes taps up the spine.

CH 8
full system

AttnRes

The residual stream is an additive memory over depth — same disease as Chapter 2, same cure: attention.

01The memory that keeps everything

Before any architecture, decide what a language model must do at token 10,000: answer a question about token 3. Where does token 3 live?

Every debate in this lesson — softmax versus linear, cache versus state, gate versus delta — is secretly a debate about that one question. So we will not start with GPT-2's block diagram. We will start with the thing the block diagram is for: remembering.

Here is the claim this chapter earns: the attention layer is an associative memory. The keys are labels. The values are contents. The softmax is a read operation. And GPT-2's answer to "where does token 3 live?" is the most honest answer possible: token 3 lives in a list, in its own slot, untouched, forever. Perfect recall — with a rent bill that grows every single step.

01bAttention, reframed as a memory

Same equation you already know. Different noun. The noun is the whole trick.

At step $t$, a decoder-only transformer computes three row vectors from the current token's hidden state $x_t \in \mathbb{R}^{1 \times d_{\text{model}}}$:

the three projections — shapes on everything $$ q_t = x_t W_Q, \qquad k_t = x_t W_K, \qquad v_t = x_t W_V \qquad \in \mathbb{R}^{1 \times d} $$
  • $k_t$ — the label this token files itself under. "I am a pet-name fact."
  • $v_t$ — the content it files. "The name is Mochi."
  • $q_t$ — the lookup request this token makes against everything filed so far. "Any pet-name facts back there?"
  • $d$ — the head dimension ($d_{\text{head}}$, typically 64–128; our toy uses 4). Multi-head just runs $h$ of these memories side by side; we track one.

Why can $q$ find $k$? Because $W_Q$ and $W_K$ read the same residual stream. Training shapes them jointly, so the query direction a later token emits for "pet-name?" lines up with the key direction the earlier token filed under. Nobody hand-assigns addresses; gradient descent invents the addressing scheme.

The read at step $t$ is a softmax-weighted average over every slot ever written:

the read — softmax attention $$ o_t = \mathrm{softmax}\!\left(\frac{q_t K_{1:t}^{\top}}{\sqrt{d}}\right) V_{1:t}, \qquad K_{1:t}, V_{1:t} \in \mathbb{R}^{t \times d} $$
  • $q_t K_{1:t}^{\top} \in \mathbb{R}^{1 \times t}$ — one similarity score per stored slot. Cost: $t \cdot d$ multiplies.
  • $\mathrm{softmax}$ — exponentiate, normalize. The exponential is why the read can be nearly one-hot: a score gap of 4 becomes a weight ratio of $e^4 \approx 55$. Softmax attention can commit hard to a single slot. Remember this — it is exactly what Chapter 2 gives up.
  • $V_{1:t}$ — the contents shelf. The weighted average pulls out (mostly) the value whose key matched.

Here is nanoGPT's attention, annotated in the memory vocabulary — every line, every shape:

python · nanogpt attention, memory glosses
B, T, C = x.size()  # batch, sequence length, d_model

# every token simultaneously prepares: a lookup (q), a label (k), a content (v)
q, k, v = self.c_attn(x).split(self.n_embd, dim=2)
k = k.view(B, T, nh, C // nh).transpose(1, 2)  # (B, nh, T, d)  labels shelf
q = q.view(B, T, nh, C // nh).transpose(1, 2)  # (B, nh, T, d)  lookups
v = v.view(B, T, nh, C // nh).transpose(1, 2)  # (B, nh, T, d)  contents shelf

# score every lookup against every label: (B,nh,T,d) @ (B,nh,d,T) -> (B,nh,T,T)
att = (q @ k.transpose(-2, -1)) * (1.0 / math.sqrt(k.size(-1)))
att = att.masked_fill(self.bias[:,:,:T,:T] == 0, float('-inf'))  # no reading the future
att = F.softmax(att, dim=-1)              # the read weights — can be nearly one-hot
y = att @ v                              # (B,nh,T,T) @ (B,nh,T,d) -> (B,nh,T,d)
y = y.transpose(1, 2).contiguous().view(B, T, C)  # heads side by side again

The $T \times T$ score matrix is the price tag: during training, every token reads against every earlier token. GPT-2's config — and where its 124M parameters actually sit:

python · gpt-2 small config
vocab_size: int = 50304  # 50257 padded up to a multiple of 64 for tensor cores
n_layer: int = 12
n_head: int = 12
n_embd: int = 768       # d_head = 768/12 = 64
Where do 124M parameters live? Do the arithmetic once. Embeddings: $50304 \times 768 \approx 38.6\text{M}$ (weight-tied with the output head). Per block: attention $4 \times 768^2 \approx 2.36\text{M}$ (the $W_Q, W_K, W_V$, and output projections), MLP $2 \times 768 \times 3072 \approx 4.72\text{M}$, so $\approx 7.1\text{M}$ per block, $\times 12 \approx 85\text{M}$. Total $\approx 124\text{M}$. Two-thirds of the model is MLPs. Keep that ratio in mind — it explodes when we reach the mixture-of-experts in Chapter 7.

01cDeriving the KV cache in four steps

Not "an optimization." A consequence, forced by four observations in a row.

  1. Decode step $t$ needs $q_t$ against ALL past keys and values.

    Generating token $t{+}1$ requires $o_t$, and $o_t$ reads over $k_1 \ldots k_t$, $v_1 \ldots v_t$. The past is load-bearing; there is no shortcut around touching it.

    why it matters: the read set is the whole history

  2. Past keys and values never change.

    $k_3 = x_3 W_K$ depends only on token 3's hidden state and frozen weights. Nothing that happens at step 10,000 alters it. Recomputing it every step would produce bit-identical output.

    why it matters: identical recomputation is pure waste

  3. So: store them. That storage is the KV cache.

    Two tensors per layer, each $t \times d$ per head, appended once per step, never edited. Per decode step the model computes one new $(q_t, k_t, v_t)$, appends $(k_t, v_t)$, and reads.

    why it matters: per-step compute drops from O(t²·d) to O(t·d)

  4. The cost didn't vanish. It moved.

    Per-step compute is now linear in $t$ — but so is per-step memory traffic: the whole cache must be streamed from GPU HBM through the compute units every single step, and the cache itself grows $O(N)$. Long-context decode is bandwidth-bound: the GPU spends its time carting the shelf around, not multiplying.

    why it matters: this bottleneck is the villain of the next five chapters
A correction worth internalizing (the original worklog stumbled here, and it's a common stumble): with a KV cache, per-step decode cost is $O(N)$ — and this was true from GPT-2's own 2019 release code, which already shipped `past` caching. FlashAttention did not fix this and does not claim to. FlashAttention (2022) is an IO-aware, mathematically exact attention kernel: it tiles the $N \times N$ score matrix through fast on-chip SRAM so it never materializes in slow HBM. It changes the memory-traffic constant — dramatically — for training and prefill. It changes no asymptotics, and it is not what makes decode linear. The cache is. Full FlashAttention lesson here.

01dThe memory wall, in your own units

"The cache can become a bottleneck" is a sentence. 2.9 terabytes is an argument. Drag the context slider.

The formula is four multiplications long. Per token of context, the cache holds K and V (that's the 2), for every layer, for every KV head, $d$ numbers each, at some precision:

kv-cache bytes per token $$ \text{bytes/token} = 2 \times n_{\text{layers}} \times n_{\text{kv-heads}} \times d_{\text{head}} \times \text{bytes}_{\text{dtype}} $$
GPT-2 small, by hand: $2 \times 12 \times 12 \times 64 \times 2 = 36{,}864$ bytes $\approx$ 36 KB per token (fp16). At its full 1,024-token context: $36{,}864 \times 1024 \approx 37.7$ MB. Cute. Now hold the formula fixed and grow only the context: at 128K tokens it is 4.7 GB; at 1M tokens, 37 GB — for one sequence, before you store the model itself. And a hypothetical 2.8T-scale model with vanilla multi-head attention would need terabytes of cache at 1M tokens. That is not an engineering annoyance. That is a wall.
gpt-2 (mha) llama-70b (gqa) dense 2.8t-scale (mha, hypothetical) k3-style: 23 mla layers only fixed-state (kda layers) gpu hbm ceiling

Read the green line. The fixed-state architectures of Chapters 2–6 are horizontal: their memory does not grow with context at all. The entire remainder of this lesson is the story of what that flatness costs, and how to buy the cost down.

01eMochi replay #1: the growing list

Our running example meets its first architecture. Result: flawless. Rent: due monthly, forever.

"Ada adopted a cat named Mochi" writes (among others) one slot we care about. In our $d=4$ toy, say the fact tokens produce:

the fact, as a key-value pair (toy d = 4) $$ k_{\text{cat}} = [\,1, 0, 0, 0\,], \qquad v_{\text{Mochi}} = [\,2, 1, 0, 1\,] $$

Ten thousand tokens later, "Ada's cat is named ____" emits the lookup $q = [\,1, 0, 0, 0\,]$. The softmax memory scores $q$ against all 10,000 stored keys. Suppose the best distractor scores $q \cdot k_j = 0.5$ while our fact scores $q \cdot k_{\text{cat}} = 1.0$ (after the $\sqrt{d}$ scaling, a gap of 0.25). One score gap, exponentiated across thousands of slots, still concentrates the read; with a trained model's much sharper gaps the read weight on the Mochi slot is effectively 1, and

$$ o \approx 1.0 \cdot v_{\text{Mochi}} = [\,2, 1, 0, 1\,] $$

Perfect retrieval at distance 10,000 — because the slot was never touched. That is the gold standard every later architecture will be judged against. The bill: 10,000 slots $\times$ 36 KB = 368 MB of cache (GPT-2-sized), all of it re-read on every one of the next steps.

The scorecard · row 1 of 8
ArchitectureStoresWriteReadEviction
GPT-2 (softmax + KV cache)every K,V — grows O(N)append a slotsoftmax over all N keys — can be one-hot sharpnone. Perfect memory, unbounded rent.
Payoff: GPT-2's memory is perfect because it cheats — it never forgets because it never compresses. Handoff: can we get retrieval from a memory whose size does not grow? What is the smallest change to the attention formula that lets the past be summarized instead of stored? Chapter 2 finds it hiding in a pair of parentheses.

01fConcept check

Chapter 1 · one question
With a KV cache, what is the per-step cost profile of decoding at context length N, and what actually bottlenecks long-context decode?

02The memory that blurs

Take softmax(qKᵀ)V and ask: which single operation forces us to keep every k and v around? Then remove it and watch what the algebra does.

Write one output of attention as a sum over slots, ignoring the $\sqrt{d}$ scale for a moment:

$$ o_t = \sum_{i=1}^{t} \frac{\exp(q_t \cdot k_i)}{\sum_{j=1}^{t} \exp(q_t \cdot k_j)} \; v_i $$

Look at where $q_t$ sits: inside the exponential, welded to every $k_i$ individually. You cannot precompute anything about the keys before the query arrives, because $\exp(q \cdot k)$ is a function of the pair. The exponential is a rigid rod between the query and each key. $t$ keys, $t$ rods, no factoring. That is the operation that forces the shelf.

Katharopoulos et al. (2020) asked the deceptively simple question: what if the nonlinearity were applied to $q$ and $k$ separately, before they meet? Pick any feature map $\phi$ that keeps scores non-negative — their choice was $\phi(x) = \mathrm{elu}(x) + 1$, which is smooth, positive, and near $x{+}1$ for positive inputs — and replace $\exp(q \cdot k)$ with $\phi(q) \cdot \phi(k)$.

Honesty about what $\phi$ is: $\mathrm{elu}{+}1$ is not an approximation of the exponential, and the paper never claims it is (that's the Performer/FAVOR+ lineage, a different research branch). It is simply a different, weaker similarity function that happens to be non-negative — chosen because non-negativity is all the "weighted average" contract needs, and separability is what the next equation needs. The score $\phi(q) \cdot \phi(k)$ is non-negative because the features are positive, not because a score got exponentiated. This distinction is exactly why linear attention is less expressive: $\exp(q \cdot k)$ is not an inner product of any finite-dimensional features of $q$ and $k$ alone.

02bThe parenthesis moves — the whole pivot, one line per step

This is the load-bearing derivation of the entire seven-year arc. It gets every line spelled out.

step 1 — swap the similarity $$ o_t = \frac{\sum_{i \le t} \big(\phi(q_t) \cdot \phi(k_i)\big)\, v_i}{\sum_{j \le t} \phi(q_t) \cdot \phi(k_j)} $$
step 2 — a scalar times a row is a matrix product: $(\phi(q_t)\phi(k_i)^{\top}) v_i = \phi(q_t)\,(\phi(k_i)^{\top} v_i)$ — shapes: $(1{\times}d)(d{\times}1)(1{\times}d)$ regrouped as $(1{\times}d)\,\big[(d{\times}1)(1{\times}d)\big]$ $$ o_t = \frac{\phi(q_t) \sum_{i \le t} \phi(k_i)^{\top} v_i}{\phi(q_t) \cdot \sum_{j \le t} \phi(k_j)} $$
step 3 — name the two sums. they are the memory. $$ S_t = \sum_{i \le t} \phi(k_i)^{\top} v_i \;\in \mathbb{R}^{d \times d}, \qquad z_t = \sum_{j \le t} \phi(k_j) \;\in \mathbb{R}^{1 \times d}, \qquad o_t = \frac{\phi(q_t)\, S_t}{\phi(q_t) \cdot z_t} $$

Stop and stare at what just happened. In step 2 the associative law of matrix multiplication — $(AB)C = A(BC)$, the most boring fact in linear algebra — moved $q$ outside the sum. Because $q$ no longer touches each $k_i$ through a nonlinearity, the sum over history can be computed before the query exists. And a sum can be maintained incrementally:

the update rule — version 1 of the equation this whole lesson edits $$ S_t = S_{t-1} + \phi(k_t)^{\top} v_t, \qquad z_t = z_{t-1} + \phi(k_t) $$
  • $\phi(k_t)^{\top} v_t$ — an outer product: a $d \times 1$ column times a $1 \times d$ row gives a full $d \times d$ matrix. One rank-1 stamp per token.
  • $S_t$ — the entire history, compressed into $d^2$ numbers. It never grows. Token 10,000 costs exactly what token 10 cost.
  • $z_t$ — the running normalizer: the sum of all feature-mapped keys. Note it sums over tokens — a $1 \times d$ row, not a matrix. (The original worklog's code accumulated the un-summed keys here, which crashes for any sequence longer than one token. Small bug, load-bearing line.)

Per decode step: read $S$ (that's $d^2$ numbers), do two small matrix products, write $S$ back. Compare Chapter 1's freight train: read $2 N d$ numbers, growing forever. The memory-bandwidth curve went from a rising line to a flat one. The transformer became an RNN — that is literally the Katharopoulos paper title.

02cWatch history fold

Left: the shelf grows and every read touches all of it. Right: the same tokens stamp into one 8×8 square. Same story, two memories.

keys k values v state cell magnitude bytes read this step

02dWhat the square actually holds — and why it must blur

"A fixed D×D state" is a phrase. Here is the object, cell by cell, and its capacity limit derived in two lines.

$S = \sum_i k_i^{\top} v_i$ (dropping $\phi$-marks from here on; read $k$ as "feature-mapped key"). What is row $r$ of this matrix? Row $r$ of $k_i^{\top} v_i$ is $k_i[r] \cdot v_i$ — the whole value, scaled by how much key $i$ points along channel $r$. So:

Row $r$ of $S$ = the sum of every value ever written, each weighted by its key's component along channel $r$. $S$ is a filing cabinet with $d$ drawers. A key that is exactly channel $r$'s unit vector files its value cleanly into drawer $r$. A key spread across channels smears its value across drawers, proportionally. Reading with $q$ means: take $q$'s mix of drawers, sum their contents.

Read-back, derived. Store one association $S = k^{\top} v$, then read with the same key:

exact recovery needs unit keys $$ kS = k(k^{\top} v) = (k k^{\top})\, v = \lVert k \rVert^2 \, v $$

$k k^{\top}$ is a $1{\times}1$ matrix — a scalar, the squared norm. If $\lVert k \rVert = 1$, read-back is exactly $v$. This is why every delta-rule architecture from Chapter 3 onward L2-normalizes its keys: it makes the memory's read operation an identity for stored facts.

Now store two associations and read with the first key:

the interference law $$ k_1 S = k_1(k_1^{\top} v_1 + k_2^{\top} v_2) = \underbrace{v_1}_{\text{what you wanted}} + \underbrace{(k_1 \cdot k_2)\, v_2}_{\text{crosstalk}} $$

The crosstalk coefficient is the key overlap. Orthogonal keys ($k_1 \cdot k_2 = 0$): zero interference. And here is the capacity theorem you can prove to yourself in one sentence: a $d$-dimensional space holds at most $d$ mutually orthogonal directions, so $S$ holds at most $d$ facts with zero crosstalk. Fact number $d{+}1$ must overlap something. In our toy, 4 clean facts. In a real head with $d = 128$: 128 clean facts — against contexts of a million tokens. The regime that makes linear attention attractive, $N \gg d$, is exactly the regime that breaks it.

Schlag, Irie & Schmidhuber saw this in 2021 and said it plainly: when sequence length exceeds capacity, "the model should learn to dynamically interact with the memory contents and selectively decide which key-value associations to keep and which ones to delete. The purely additive [update] may be inappropriate for this purpose." Hold that quote. It is the hinge of the entire lesson: addition is not a policy. Chapters 3, 5, and 6 are three successively better answers to it.

02eThe 4-slot whiteboard — interference you can compute

Write facts into a real 4×4 S. Read them back. Watch clean retrieval degrade into blur — with the actual numbers, checkable by hand.

The board is blank. Write a fact.
positive cell read result true value crosstalk error
Check the widget's arithmetic by hand (this is the whole point of $d=4$). Write cat→Mochi: $S = k_{\text{cat}}^{\top} v_{\text{Mochi}}$ puts $[2,1,0,1]$ in row 0. Write dog→Rex with $k_{\text{dog}} = [0.6, 0.8, 0, 0]$, $v_{\text{Rex}} = [0,3,1,2]$: row 0 gains $0.6 \times [0,3,1,2] = [0, 1.8, 0.6, 1.2]$, row 1 gains $0.8 \times [0,3,1,2] = [0, 2.4, 0.8, 1.6]$. Now read "cat": $qS$ with $q = [1,0,0,0]$ returns row 0 $= [2, 2.8, 0.6, 2.2]$. The true answer was $[2, 1, 0, 1]$. The error is exactly $(k_{\text{cat}} \cdot k_{\text{dog}})\, v_{\text{Rex}} = 0.6 \, v_{\text{Rex}}$ — the interference law, verified to the second decimal. Rex bled into Mochi because their keys overlap by 0.6, and nothing in the additive rule can ever get him back out.

02fWhat we gave up: the sharpness budget

Attention's contract survives. Its precision doesn't. Know exactly which part broke.

Both mechanisms honor the same three-step contract: (1) make query-key scores non-negative, (2) normalize them to sum to one, (3) take the weighted average of values. What differs is step 1's dynamic range:

PropertySoftmax attentionLinear attention (elu+1)
score function$\exp(q \cdot k)$ — on the pair$\phi(q) \cdot \phi(k)$ — features meet later
can the read be one-hot?yes — exp turns score gaps into weight ratios exponentiallyno — polynomial-ish scores give heavy-tailed, blended reads
state per stepgrows: $2Nd$ numbersfixed: $d^2 + d$ numbers
per-step decode reads$O(N)$ from HBM — bandwidth-bound$O(d^2)$ — constant
exact retrieval at distancealways (slot untouched)only until keys collide ($N \le d$, orthogonal)
capacityunbounded$\approx d$ clean facts per head

This table is the honest version of "less expressive." Linear attention cannot commit: where softmax snaps to the best-matching slot, $\phi$-scores blend neighbors. Retrieval-heavy tasks (recall a name, match a bracket, copy a passkey) feel this first — which is exactly why the hybrid architectures of Chapter 6 keep a few softmax layers around like reading glasses.

02gCode that actually runs

The reference worklog's snippet had four bugs, including one that silently breaks causality. Here is the corrected minimal implementation, with the traps labeled.

python · linear attention, decode step (the RNN view)
def decode_step(x_t, S, z):
    # x_t: (1, d_model) — ONE new token. S: (d, d). z: (d,)
    q, k, v = proj_q(x_t), proj_k(x_t), proj_v(x_t)   # each (1, d)
    q, k = F.elu(q) + 1, F.elu(k) + 1                  # feature map BEFORE they meet

    S = S + k.T @ v       # outer product (d,1)@(1,d) -> (d,d): the stamp
    z = z + k.squeeze(0)  # TRAP 1: sum over tokens -> (d,). Accumulating
                          #         un-summed keys here crashes for t>1.
    o = (q @ S) / (q @ z[:, None] + eps)  # read, then normalize (scalar denom)
    return o, S, z
python · linear attention, causal prefill (the parallel view)
def prefill(Q, K, V):
    # Q,K,V: (T, d). TRAP 2: you may NOT do S = K.T @ V once and read q@S —
    # that folds FUTURE keys into the state before early queries read it.
    # Causality needs per-position prefix sums (cumsum over the T axis):
    Q, K = F.elu(Q) + 1, F.elu(K) + 1
    S_prefix = torch.cumsum(K[:, :, None] * V[:, None, :], dim=0)  # (T,d,d)
    z_prefix = torch.cumsum(K, dim=0)                             # (T,d)
    O = torch.einsum('td,tde->te', Q, S_prefix)
    denom = (Q * z_prefix).sum(-1, keepdim=True) + eps
    return O / denom
    # (materializing (T,d,d) is memory-hungry — Chapter 4 fixes exactly this)
Trap 2 deserves one more sentence, because the original article shipped it: if you compute the full-sequence state first and then read every query against it, token 5 reads a memory that already contains tokens 6…10,000. During training that model can "predict" by reading the future through the state; at deployment the future doesn't exist, and quality collapses. Causality bugs in recurrent memories are silent — the shapes all check out. The fix is always the same: the state a query reads must be the prefix state at its own position.

02hMochi replay #2: the blur

Same story, new memory. The fact survives — smeared.

"Ada adopted a cat named Mochi" stamps $k_{\text{cat}}^{\top} v_{\text{Mochi}}$ into $S$. Ten thousand tokens follow, each stamping its own outer product onto the same 16 cells. By the query, row 0 holds Mochi plus the channel-0 component of ten thousand other stamps. The read returns Mochi's silhouette under wet tissue paper: $[2, 2.8, 0.6, 2.2]$-style blends instead of $[2,1,0,1]$. And the memory cost was flat the whole time — 16 cells at token 10 and at token 10,000.

The scorecard · rows 1–2 of 8
ArchitectureStoresWriteReadEviction
GPT-2 (softmax + KV cache)every K,V — O(N)append a slotsoftmax, one-hot sharpnone — perfect memory, unbounded rent
Linear attentionfixed $d{\times}d$ + normalizeradd outer product $k^{\top}v$$qS$ — blended, can't commitaccidental — interference buries old facts
Payoff: we made the memory free by converting eviction from a policy into an accident. Nothing is ever deleted; everything is slowly buried. Handoff: if addition is what buries old facts, can a write ever subtract? What would "overwrite" even mean for a matrix memory? Chapter 3 answers with three lines of algebra and an eraser.

02iConcept check

Chapter 2 · one question
Why exactly does removing the softmax allow the O(N) key-value history to be replaced by a fixed d×d state?

03The memory that overwrites

You're at a whiteboard with four regions. A new fact arrives for a region that's already written on. What does a human do?

Not scribble on top — that's Chapter 2's additive update, and you watched what it does. A human reads what's there, erases it, and writes the new thing. The delta rule is exactly that gesture, translated into linear algebra. It needs three lines, and each line is one of the whiteboard verbs.

One prerequisite we've already earned: from Chapter 2's read-back derivation, if keys are L2-normalized then reading a stored fact with its own key returns it exactly ($kS = \lVert k \rVert^2 v = v$). That's why DeltaNet-family models normalize their keys — the eraser is only accurate if the read is.

03bRead, subtract, write — then collapse to one line

Three named steps, then the algebra folds them into the form every later paper quotes.

step 1 — READ: what does this key currently retrieve? $$ v_{\text{old}} = k_t \, S_{t-1} \qquad \in \mathbb{R}^{1 \times d} $$
step 2 — SUBTRACT: keep only what's actually new, scaled by write strength $$ u_t = \beta_t \, (v_t - v_{\text{old}}) \qquad \beta_t = \sigma(x_t w_{\beta}) \in (0, 1) $$
step 3 — WRITE: the same outer-product stamp as before, but stamping the difference $$ S_t = S_{t-1} + k_t^{\top} u_t $$
  • $\beta_t$ — the write strength, a per-token scalar the model computes from the current token. $\beta = 0$: ignore this token entirely. $\beta = 1$: fully replace whatever this key retrieved. In between: blend. It is a learned "how sure am I" dial on every write.
  • Note what step 2 subtracts: not "the old fact" in the abstract, but whatever the key currently retrieves — including any crosstalk that has accumulated there. The eraser erases what a reader would actually see. (This subtlety matters in the worked example below.)

Now substitute step 1 and 2 into step 3 and watch it collapse:

the collapse, line by line $$ S_t = S_{t-1} + k_t^{\top} \beta_t (v_t - k_t S_{t-1}) $$ $$ = S_{t-1} - \beta_t \, k_t^{\top} k_t \, S_{t-1} + \beta_t \, k_t^{\top} v_t $$ $$ = \big(I - \beta_t \, k_t^{\top} k_t\big)\, S_{t-1} + \beta_t \, k_t^{\top} v_t $$

Version 2 of the equation this lesson keeps editing. Compare version 1 (Chapter 2): the identity matrix was silently there all along — additive linear attention is just $S_t = I \cdot S_{t-1} + k_t^{\top} v_t$. The delta rule replaces that $I$ with a transition matrix $(I - \beta k^{\top} k)$, and that one factor is the eraser:

The geometry of $(I - \beta\, k^{\top} k)$, one sentence each. With unit $k$ and $\beta = 1$ it is the projection that deletes the $k$-direction from every column of $S$ — total erasure along that key, everything orthogonal untouched. With $\beta = 2$ it is a Householder reflection (flips the $k$-direction). With $\beta \in (0,1)$: partial erasure. This is why the literature calls DeltaNet's transitions "generalized Householder" matrices — a family of controlled edits along one direction at a time, where linear attention's transition was the do-nothing identity.
Convention flag, once and never again. We write row-vector convention: $S$ is $d_k \times d_v$, update $S_t = (I - \beta k^{\top} k) S_{t-1} + \beta k^{\top} v$, read $o = qS$. The DeltaNet papers write the transpose: $\mathbf{S}$ as $d_v \times d_k$, update $\mathbf{S}_t = \mathbf{S}_{t-1}(I - \beta k k^{\top}) + \beta v k^{\top}$, read $o = \mathbf{S} q^{\top}$. Same mathematics, mirror image. Pick one and never mix — mixing them mid-derivation is precisely how the source worklog's hardest section became confusing.

03cThe delta stamp, one step at a time

The board holds two facts. A correction arrives: Ada's cat is actually named Biscuit. Step through read → subtract → write and watch the eraser work — with the real numbers on every cell.

state cell being erased being written

03dBy hand, every digit — including the honest caveat

The clean case, then the case nobody shows you: erasing on a contaminated board.

Clean overwrite (the widget's numbers). Board: row 0 holds $v_{\text{Mochi}} = [2,1,0,1]$, key $k = [1,0,0,0]$, new value $v_{\text{Biscuit}} = [1,0,2,2]$, $\beta = 1$.
READ: $v_{\text{old}} = kS = [2,1,0,1]$.
SUBTRACT: $u = 1 \cdot ([1,0,2,2] - [2,1,0,1]) = [-1,-1,2,1]$.
WRITE: row 0 becomes $[2,1,0,1] + [-1,-1,2,1] = [1,0,2,2] = v_{\text{Biscuit}}$, exactly. Read-back: perfect. Row 2's Silver: untouched, because $k$ has no component along channel 2.
Overwrite on a contaminated board. Suppose Rex (Chapter 2) already bled into row 0, so $kS = [2, 2.8, 0.6, 2.2]$ instead of clean Mochi. The delta rule subtracts that — the contaminated read — and writes Biscuit in its place. Result: reading "cat" now returns clean Biscuit (good!), but the crosstalk that Rex deposited in row 0 has been consumed in the process, which slightly perturbs what reading "dog" returns (its row-0 component changed). The eraser guarantees the fact you wrote reads back true; it does not guarantee zero side effects on overlapping facts. Only orthogonal keys give you both. Capacity is still capacity — the delta rule manages it; it cannot repeal it.

And the limitation that motivates Chapter 5, visible right in the formula: the transition $(I - \beta k^{\top} k)$ only edits along $k$. The delta rule can replace the fact it is currently touching. It has no way to act on the 40 facts of a finished subplot, or to fade everything at a topic switch — there is no key pointing at "everything."

03eWhy Schlag called them "fast weight programmers"

The name sounds exotic. The observation is one line of calculus, and it earns the section its title.

Look at the delta update as an optimization step. Define a tiny loss measuring how badly the memory stores the current pair: $\mathcal{L}(S) = \tfrac{1}{2} \lVert k_t S - v_t \rVert^2$. Its gradient with respect to $S$ is $k_t^{\top}(k_t S - v_t)$. One step of gradient descent with learning rate $\beta_t$:

$$ S_t = S_{t-1} - \beta_t \, k_t^{\top} (k_t S_{t-1} - v_t) = S_{t-1} + k_t^{\top} \beta_t (v_t - v_{\text{old}}) $$

— which is exactly the delta rule (this is Widrow & Hoff's 1960 least-mean-squares update, rediscovered as a memory). So the picture is: the state $S$ is a weight matrix, and inference itself trains it, one SGD step per token, with a learned per-token learning rate $\beta_t$. The "slow" weights ($W_Q, W_K, W_V, w_\beta$) are trained over weeks by backprop; they program a "fast" weight matrix that is rewritten within every single sequence. That's Schlag, Irie & Schmidhuber's 2021 framing, resurrecting an idea Schmidhuber first published in 1991 — and it is why $\beta$ has the semantics of a learning rate: how hard should this one example update the memory?

The scorecard · rows 1–3 of 8
ArchitectureStoresWriteReadEviction
GPT-2 (softmax + KV cache)every K,V — O(N)append a slotsoftmax, one-hot sharpnone
Linear attentionfixed $d{\times}d$add $k^{\top}v$$qS$, blendedaccidental (interference)
DeltaNetfixed $d{\times}d$erase-then-write: $k^{\top}\beta(v - kS)$$qS$, blendedtargeted replace — needs a replacement in hand

03fMochi replay #3: the correction sticks

The story gains a twist, and the memory finally handles it.

"Ada adopted a cat named Mochi. … Actually, she renamed him Biscuit. … [10,000 tokens] … Ada's cat is named ____."

Chapter 2's additive memory answers this query with a Mochi-Biscuit smoothie: both stamps sit in row 0, and the read sums them. The delta memory read row 0 before writing, subtracted Mochi, and deposited Biscuit — so the query returns Biscuit, clean. The correction didn't pile on top of the error; it replaced it. That is the difference between a memory that accumulates and a memory that maintains.

Payoff: the delta rule gives the memory an eraser — but the eraser only works where you're about to write. Handoff, with an honest gear change: this chapter fixed what the memory does. It broke how we train it: step 1's read needs $S_{t-1}$, which needs $S_{t-2}$, all the way down — a strictly sequential chain, and GPUs despise chains. Chapter 4 is about buying the parallelism back without changing a single answer the memory gives. If you only care about the memory story, read its payoff line and skip ahead; the scorecard gains no row there, because nothing about the memory changes.

03gConcept check

Chapter 3 · one question
With unit-normalized keys and β=1, what does the transition matrix (I − βkᵀk) do to the state S?

04Making the eraser fast

The author of the worklog this lesson rebuilds spent seven hours on this material and said so — the single most-quoted line in his replies. Our goal: the same understanding in fifteen minutes, because we arrive with Chapter 3's algebra already in hand.

State the problem as a compute-shape problem, because that's all it is. The delta rule is a chain: $u_t$ needs $S_{t-1}$, which needs $u_{t-1}$, which needs $S_{t-2}$… Running it as written means, at training time, a for-loop of tiny $d \times d$ operations — one per token, a million deep. A GPU running a million sequential small matmuls achieves a few percent of its throughput; the same total arithmetic delivered as a few hundred large matmuls runs near peak. Same FLOPs, up to ~100× different wall-clock. Nothing in this chapter changes what the memory computes — we are only repackaging the arithmetic into GPU-shaped boxes.

04bPass 1: chunking, without the eraser

First solve the easy version — plain additive linear attention — because its trick carries over.

Split the $T$ tokens into chunks of $C$ (say 64). For a query $q_t$ inside chunk $c$, split its history at the chunk boundary:

every output = one state read + one small attention $$ o_t = \underbrace{q_t \, S_{[c-1]}}_{\text{everything before my chunk, folded}} \; + \underbrace{\sum_{\substack{i \in \text{chunk } c \\ i \le t}} (q_t \cdot k_i)\, v_i}_{\text{my chunk: real masked attention}} $$
  • $S_{[c-1]}$ — the state at the last chunk boundary. Updated once per chunk: $S_{[c]} = S_{[c-1]} + K_c^{\top} V_c$, where $K_c, V_c \in \mathbb{R}^{C \times d}$ are the chunk's stacked keys and values. One big matmul.
  • The second term is a genuine $C \times C$ masked attention — $\mathrm{tril}(Q_c K_c^{\top}) V_c$ — computed within the chunk. Another big matmul. (Within a chunk it's score-first, $(q k^{\top}) v$; across chunks it's state-first, $q (k^{\top} v)$ — both orders are legal by Chapter 2's associativity, and chunking simply uses each where it's cheapest.)
A correction to the source worklog: it claimed "setting C = N recovers standard O(N²) attention." Not quite — there is no softmax anywhere in this equation. C = N recovers the quadratic parallel form of linear attention: $\mathrm{tril}(QK^{\top})V$, same answers as the recurrence, different packaging. Softmax attention is not on this dial at all — the exponential is exactly the thing that can't be folded into $S$, which was the whole lesson of Chapter 2. What IS true: at $C = N$ the cost shape matches full attention ($O(N^2 d)$), and at $C = 1$ you're back to the pure recurrence. The dial interpolates cost shapes, not semantics.

04cThe cost splits in two — derive it, then feel it

One fixed piece, one piece that grows with C. Every symbol defined; check it against the widget.

Per chunk, with $L$ total tokens, chunk size $C$, head dimension $d$ (and $L/C$ chunks):

  • State work (fold + read): $K_c^{\top} V_c$ is $(d \times C)(C \times d)$ ≈ $C d^2$ multiply-adds, and $Q_c S$ is $(C \times d)(d \times d)$ ≈ $C d^2$. Per token: $2d^2$. Total: $\mathbf{2 L d^2}$ — independent of C.
  • Intra-chunk attention: $Q_c K_c^{\top}$ is $(C \times d)(d \times C)$ ≈ $C^2 d$, and the masked product with $V_c$ is another $C^2 d$. Per chunk $2C^2 d$, times $L/C$ chunks: $\mathbf{2 L C d}$ — linear in C.
the cost law of chunked linear attention $$ \mathrm{FLOPs}(C) \approx 2Ld^2 + 2LCd \qquad \Rightarrow \qquad C{=}1: 2Ld^2 \quad \big| \quad C{=}L: \approx 2L^2 d $$

So smaller C is always fewer FLOPs — and yet nobody trains at $C = 1$. Because wall-clock is FLOPs divided by achieved throughput, and achieved throughput is a cliff function of matmul size: tensor cores eat $64{\times}64$ tiles; feed them $1 \times d$ crumbs and they idle. $C = 64$ or $128$ sits at the bottom of the U: big enough that the hardware runs hot, small enough that the quadratic term stays trivial. Drag the dial:

04dThe chunk dial

Left: the T×T work matrix — amber tiles are real attention, blueprint is history folded into S. Right: FLOPs falls as C shrinks; wall-clock doesn't. The gap between the two curves is the whole chapter.

intra-chunk attention tiles (2LCd) folded into state (2Ld²) wall-clock (relative) flops (relative)

04ePass 2: why the eraser resists — and the T-matrix that tames it

The delta rule's subtraction couples every token in a chunk to its predecessors. The escape: that coupling is triangular, and triangular systems solve fast.

Chunking plain linear attention worked because the chunk's contribution to the state, $K_c^{\top} V_c$, is a sum — order inside the chunk doesn't matter. The delta rule breaks this: each token's correction depends on the state including all earlier writes in the same chunk. Write it out for tokens $1, 2, 3$ of a chunk entering with boundary state $S_0$ (all keys unit, all inside one chunk):

the within-chunk dependency, made explicit $$ u_1 = \beta_1 (v_1 - k_1 S_0) $$ $$ u_2 = \beta_2 (v_2 - k_2 S_0) - \beta_2 (k_2 \cdot k_1)\, u_1 $$ $$ u_3 = \beta_3 (v_3 - k_3 S_0) - \beta_3 (k_3 \cdot k_1)\, u_1 - \beta_3 (k_3 \cdot k_2)\, u_2 $$

Read the pattern: each $u_i$ equals its "naive" correction against the boundary state, minus overlap-weighted copies of every earlier correction in the chunk. (Where does that come from? Token 2's read passes through $S_1 = S_0 + k_1^{\top} u_1$, so $k_2 S_1 = k_2 S_0 + (k_2 \cdot k_1) u_1$ — the crosstalk term from Chapter 2, reappearing as bookkeeping.) The coupling coefficients $(k_i \cdot k_j)$ for $j < i$ form a strictly lower-triangular $C \times C$ matrix. So the whole system is

a unit lower-triangular solve — the entire trick $$ \big(I + \mathrm{tril}(\,\mathrm{diag}(\beta)\, K_c K_c^{\top}, -1)\big)\; U = \mathrm{diag}(\beta)\,\big(V_c - K_c S_0\big) $$

— solvable by forward substitution: $u_1$ directly, then $u_2$ using $u_1$, then $u_3$ using both — $C$ small steps, all of them dense row operations that batch beautifully, or equivalently one $C \times C$ triangular inverse. $C = 64$: a $64 \times 64$ solve. Trivial. Once $U$ is known, everything else is Pass-1 machinery: outputs read $Q_c S_0$ plus intra-chunk attention against the corrected pseudo-values $U$, and the boundary state advances by $K_c^{\top} U$ — big matmuls all the way down. (In the literature this triangular-solve packaging is the WY representation of products of Householder-like factors — the same trick QR decompositions have used for decades. That's your search term.)

Tiny numeric check ($C = 2$). Chunk keys $k_1 = [1,0,0,0]$, $k_2 = [0.6, 0.8, 0, 0]$, overlap $k_2 \cdot k_1 = 0.6$, both $\beta = 1$, $S_0 = 0$, values $v_1 = [2,1,0,1]$ (Mochi), $v_2 = [0,3,1,2]$ (Rex). Then $u_1 = v_1$, and $u_2 = v_2 - 0.6\, u_1 = [0,3,1,2] - [1.2, 0.6, 0, 0.6] = [-1.2, 2.4, 1, 1.4]$. Sequential check: after writing Mochi, token 2 reads $v_{\text{old}} = k_2 S_1 = 0.6 \cdot [2,1,0,1] = [1.2, 0.6, 0, 0.6]$, so $u_2 = v_2 - v_{\text{old}} = [-1.2, 2.4, 1, 1.4]$. Identical. The triangular solve is the sequential rule, re-ordered — not an approximation of it.

04fThe corrected listing

The source worklog's version crashes on line 8 (a 3-D tensor has no .t()) and its substitution loop indexes chunks as matrix rows. Here is the per-chunk version that runs — and the assertion that keeps it honest.

python · chunked delta rule, single head, runnable
def chunked_delta(Q, K, V, beta, C):
    # Q,K,V: (L, d) with K rows unit-norm; beta: (L,). Returns O: (L, d)
    L, d = Q.shape
    S = torch.zeros(d, d)
    O = torch.empty_like(V)
    for s in range(0, L, C):
        q, k, v, b = Q[s:s+C], K[s:s+C], V[s:s+C], beta[s:s+C, None]  # (C,d),(C,1)
        A = torch.tril(b * (k @ k.T), -1)          # (C,C) strictly-lower couplings
        rhs = b * (v - k @ S)                      # naive corrections vs boundary S
        U = torch.linalg.solve_triangular(
            torch.eye(len(k)) + A, rhs, upper=False)  # forward substitution
        O[s:s+C] = q @ S + torch.tril(q @ k.T) @ U # state read + intra-chunk attn
        S = S + k.T @ U                            # advance the boundary state
    return O

# the sentence that makes this listing trustworthy — run it:
assert torch.allclose(chunked_delta(Q, K, V, beta, C=4),
                      sequential_delta(Q, K, V, beta), atol=1e-5)
Payoff: chunking is a dial — $C{=}1$ is a recurrence, $C{=}L$ is the quadratic form, 64 is where the silicon smiles — and the delta rule joins the party because its within-chunk tangle is a triangular solve. Nothing about the memory changed in this chapter; the scorecard gains no row. Handoff, back in memory-land: the eraser still needs a target. When Ada's story ends and a new document begins, what erases the forty facts of a finished subplot? Nothing points at "everything." Chapter 5 installs the missing knob.

04gConcept check

Chapter 4 · one question
Why is C=1 the FLOP-optimal chunk size, yet C=64–128 the wall-clock-optimal one?

05The memory that fades

Ada's story ends at token 11,999. Token 12,000 opens a tax form. Forty facts about Ada, Mochi, and Biscuit are now dead weight in a 16-cell memory. Who takes out the trash?

Not the delta rule. Watch it try: to erase "cat→Biscuit" it must write something at the cat key — the eraser only works where you're about to write, and the tax form will never mention the cat. The delta rule can replace; it cannot release. Forty stale facts keep squatting in the state, radiating crosstalk into every future read, because there is no key that points at "everything about the previous document."

So ask the cheapest-possible-fix question: what is the smallest edit to the update equation that lets the model free capacity on demand?

05bThe cheapest possible knob: multiply by a scalar

One number per step, between 0 and 1. That's the entire mechanism. The interesting part is who invented it and what it can't do.

gated additive update — the knob in isolation $$ S_t = \alpha_t \, S_{t-1} + k_t^{\top} v_t, \qquad \alpha_t = \sigma(x_t w_{\alpha}) \in (0, 1) $$
  • $\alpha_t = 1$: yesterday's memory survives intact — back to plain linear attention.
  • $\alpha_t = 0$: amnesia. The board is wiped and the new token writes onto a blank state.
  • $\alpha_t = 0.9$ held for $n$ steps: everything old shrinks by $0.9^n$ — a half-life. The state can no longer grow without bound, because old content is taxed every step it stays.
  • Crucially $\alpha_t$ is data-dependent — computed from the current token. The model can learn that "\n\n## New Document" means slam the gate.
Lineage, stated carefully (the source worklog flattened this into "this is the Mamba-2 contribution / it's what Mamba does," which is off twice). Fixed scalar decay: RetNet (2023), $\alpha$ a constant per head. Data-dependent per-channel decay: GLA (Yang et al., 2023) — note: per-channel, before Kimi. Mamba-1's decay is likewise per-channel (via its discretized state matrix), not uniform. What Mamba-2 (SSD) did was collapse decay to one data-dependent scalar per head and show that this simplification unlocks matmul-friendly training at no quality loss — and that scalar-gate form is what the Gated DeltaNet paper explicitly imports. So: the knob has many parents; Mamba-2 is the one Gated DeltaNet cites for the scalar version. Mamba's own lesson is here.

05cGate meets eraser: the gated delta rule

Each tool fixes exactly the failure the other cannot touch. The combination is one line.

Put the two capabilities side by side and the complementarity is exact:

Delta rule aloneScalar gate alone
replace one fact preciselyyes — erase-then-write at its keyno — can only shrink everything
bulk-forget at a context switchno — needs a key to write againstyes — one small α wipes the board
failure modestale facts squat foreverthe fact you needed fades with the junk

The Gated DeltaNet update — version 3 of the one equation this lesson keeps editing — simply installs both:

gated delta rule — decay first, then erase-and-write $$ S_t = \alpha_t \big(I - \beta_t k_t^{\top} k_t\big)\, S_{t-1} + \beta_t \, k_t^{\top} v_t $$

Read it as a story, right to left: yesterday's memory is first faded by $\alpha_t$, then the eraser clears the $k_t$-direction, then the new fact is stamped at full strength. Set $\alpha_t = 1$: pure DeltaNet. Set $\beta_t = 0$: pure gated linear attention. Set both to their idle values ($\alpha = 1, \beta = 0$): the memory coasts untouched through filler tokens. Every token, the model chooses a point in this 2-D policy square — and the code change from Chapter 3 is literally one multiplication:

python · the entire diff from deltanet
alpha = torch.sigmoid(self.w_alpha(x_t))   # NEW: one scalar per token, per head
v_old = k @ S
u = beta * (v - v_old)
S = alpha * S + k.T @ u                     # CHANGED: alpha * S   (was: S + ...)

(One nuance for the sharp-eyed: with the decay applied before the read as the displayed equation has it, $v_{\text{old}}$ should read the already-faded state $\alpha_t S_{t-1}$; implementations fold this in. The chunked version composes with Chapter 4's T-matrix by carrying a running product of $\alpha$'s across the chunk — details live in the flash-linear-attention repo; an honest pointer beats a hand-waved figure.)

05dSurvival arithmetic — what a fact is worth after n steps

Decay compounds multiplicatively. Three numbers make it visceral.

A fact written at step $s$ and read at step $t$ has been multiplied by every gate in between:

cumulative survival $$ \text{strength at read} = \prod_{i = s+1}^{t} \alpha_i $$
Worked: Mochi is written at step 3. The gates at steps 4, 5, 6 are $\alpha = 0.9$, then $0.5$ (a paragraph break made the model cautious), then $0.9$. Read at step 6: strength $= 0.9 \times 0.5 \times 0.9 = \mathbf{0.405}$. The read returns $0.405 \cdot v_{\text{Mochi}}$ plus whatever fresher, less-faded content shares its channels — a fact at 40% volume in a room of louder voices. This product structure is also why real kernels track $\log \alpha$ prefix sums instead of raw products: $10{,}000$ multiplications of numbers like 0.98 underflow float16; sums of their logs don't.

And the cost of the whole mechanism? One extra projection ($w_\alpha$), one multiply per step. The scorecard-relevant consequence: the state finally has a rent system. Old content pays $\alpha$ every step it stays; content the model marks important (gates near 1) lives long; junk after a slammed gate is gone in a step or two.

05eThe fade curve — one gate, everything on it

Mochi is written at token 2. A context switch hits at token 12. Drag how hard the gate slams and watch who survives to the query at token 24 — the fact and the junk share one fate.

mochi (written t=2, still needed) stale subplot junk (written t=6) tax-form fact (written t=14) context switch

05fMochi replay #4: collateral damage

The gate solves the tax form and shoots the cat.

Run the story: Mochi written at token 2; subplot junk accumulates; the document switches at token 12 and the model, quite reasonably, slams $\alpha = 0.2$ to clear the board for tax vocabulary. It works — the junk drops to 20% instantly, the tax facts land on a nearly-clean state, crosstalk plummets. And Mochi — still needed at token 24 — is at 20% × baseline decay too, because one scalar cannot tell a load-bearing fact from a dead subplot. Everything that crossed the switch crossed it together.

The scorecard · rows 1–4 of 8
ArchitectureStoresWriteReadEviction
GPT-2 (softmax + KV cache)every K,V — O(N)appendsoftmax, sharpnone
Linear attentionfixed $d{\times}d$add $k^{\top}v$$qS$, blendedaccidental (interference)
DeltaNetfixed $d{\times}d$erase-then-write$qS$, blendedtargeted replace
Gated DeltaNetfixed $d{\times}d$fade · erase · write$qS$, blendedtargeted replace + global decay (all-or-nothing)
Payoff: the memory now has an eraser and a fade knob — but the knob is wired to the whole board at once. Handoff: one $\alpha$ for a 128-channel state means choosing a single decay rate for names, syntax, style, and stale junk simultaneously. What if each channel got its own knob? That question, plus a very fast kernel, is a 2025 paper by Moonshot AI — and it's in the title of this lesson.

05gConcept check

Chapter 5 · one question
Why does Gated DeltaNet need BOTH the gate α and the delta rule's β-write, rather than either alone?

06KDA: a knob per channel

A single α is a thermostat for the whole house. Kimi Delta Attention installs one per room.

Chapter 5 ended on an indistinguishability: one scalar gate cannot fade the junk without fading Mochi. But look at where facts live — Chapter 2 told us: different facts occupy different channels (rows) of $S$, to the extent their keys differ. Junk written under tax-form keys and Mochi written under pet-name keys occupy different rows of the board. A per-row gate could tax them differently. That is the entire idea of Kimi Delta Attention (KDA), the linear-attention half of Moonshot's Kimi Linear architecture: replace the scalar $\alpha_t$ with a vector $\boldsymbol{\alpha}_t \in (0,1)^{d}$, one learned decay per channel per token.

06bThe one-line upgrade — and honest credit

Version 4 of the equation. Diff highlighted. Lineage stated without marketing.

kimi delta attention — per-channel decay inside the delta rule $$ S_t = \mathrm{Diag}(\boldsymbol{\alpha}_t)\, \big(I - \beta_t k_t^{\top} k_t\big)\, S_{t-1} + \beta_t\, k_t^{\top} v_t, \qquad \boldsymbol{\alpha}_t \in (0,1)^{d} $$
  • $\mathrm{Diag}(\boldsymbol{\alpha}_t)$ — a diagonal matrix: row $r$ of the (post-erase) state is multiplied by $\alpha_t[r]$. Channel 0 can keep 99% of its content while channel 3 keeps 10% — in the same step.
  • $\boldsymbol{\alpha}_t$ comes from its own small projection of the current token (in Kimi Linear's parameterization, a low-rank projection squashed through a sigmoid-like map) — a few extra parameters per layer, reported at roughly a 1% parameter overhead versus the plain delta layer.
python · the entire diff from gated deltanet
alpha = torch.sigmoid(self.w_alpha(x_t))   # CHANGED: now shape (d,), was a scalar
v_old = k @ (alpha[:, None] * S)            # decay applied per ROW of S
u = beta * (v - v_old)
S = alpha[:, None] * S + k.T @ u            # same rule, d knobs instead of 1
Credit where it's due: per-channel, data-dependent decay is not new with KDA — GLA (2023) had exactly that, and Mamba-1's discretized state matrix is per-channel too. What KDA contributes is the combination: fine-grained diagonal decay inside the delta rule (GLA has no eraser), plus a chunked kernel (a specialization of the DPLR family) that makes the combination train fast — reported at roughly twice the throughput of a general DPLR implementation, because the decay and erase directions share the same $k_t$. The math of that kernel is Chapter 4's T-matrix trick with a running per-channel decay product folded through it; we stop at the update rule, and the FLA kernels are where the folding lives.

06cTwin boards, one context switch

Same story on both boards. At the switch, the left board gets one knob; the right board gets four. Read "cat" at the end and compare what survives.

Press Run the story: Mochi is written on channel 0, junk on channels 2–3, then the context switches and both boards try to clear the junk.
state cell gate slams (low α) gate holds (α≈1)

06dMLA in one box — because "hybrid" is meaningless without it

The other half of Kimi Linear is Multi-head Latent Attention. The source worklog promised to explain it and never did. One box, all the shapes.

Multi-head Latent Attention (DeepSeek, 2024) is ordinary softmax attention with a compressed cache. Instead of storing full $k_t, v_t$ for every head ($2 \times n_{\text{heads}} \times d_{\text{head}}$ numbers per token per layer), the layer stores one small shared latent per token: $c_t = x_t W_{\downarrow} \in \mathbb{R}^{1 \times d_c}$, with $d_c \approx 512$, plus a small positional key ($\approx 64$ dims for RoPE). At read time, every head reconstructs its keys and values from the latent on the fly: $k_t^{(h)} = c_t W_{\uparrow K}^{(h)}$, $v_t^{(h)} = c_t W_{\uparrow V}^{(h)}$ — and the up-projections can be algebraically absorbed into the query/output weights, so the reconstruction is (mostly) free.

The arithmetic that matters: a 64-head, $d_{\text{head}}{=}128$ layer stores $2 \times 64 \times 128 = 16{,}384$ numbers per token under vanilla MHA, versus $\approx 576$ under MLA — a ~28× cache compression — while remaining full softmax attention: every token still individually addressable, reads still one-hot sharp. The shelf got smaller; it did not become a summary. Chapter 1's widget has an MLA curve — go look at how much lower it sits.

06eThe 3:1 weave — Kimi Linear

Three KDA layers, then one MLA layer, repeat. Not hedging — a division of labor with a dial on it.

Kimi Linear (Moonshot, 2025) interleaves the two memories: 3 KDA layers for every 1 MLA layer, with the MLA layers reportedly run without positional encoding (NoPE) — position is KDA's job, since a recurrence is ordered by construction. The division of labor is exactly our scorecard's two columns come to life:

  • KDA layers carry the running, constant-cost, per-channel-managed memory — the workhorse for flow, syntax, and recent context. 75% of layers pay $O(d^2)$ per step, forever flat.
  • MLA layers keep a real (compressed) shelf — the reading glasses. When the model must find a passkey verbatim from 800K tokens back, some layer needs slot-exact softmax recall; one in four turns out to be enough.

The reported headline results (from the Kimi Linear tech report, under matched training recipes at 1.4T tokens): the hybrid matches or beats the all-MLA baseline on standard evals — and at 1M-token context, decode throughput is up to 6× the full-attention baseline, with the KV cache cut by ~75% (only every 4th layer keeps one, and it's MLA-compressed on top). The 6× is a long-context number — at short context the cache never dominates and the gap shrinks; always attach the regime to the claim.

The scorecard · rows 1–7 of 8
ArchitectureStoresWriteReadEviction
GPT-2 (softmax + KV)every K,V — O(N)appendsoftmax, sharpnone
Linear attentionfixed $d{\times}d$add $k^{\top}v$$qS$, blendedaccidental
DeltaNetfixed $d{\times}d$erase-then-write$qS$targeted replace
Gated DeltaNetfixed $d{\times}d$fade·erase·write$qS$replace + global decay
KDAfixed $d{\times}d$per-channel fade·erase·write$qS$replace + per-channel decay
MLAcompressed latents — O(N), ~28× smallerappend latentsoftmax, sharpnone (smaller rent)
Kimi Linear (3 KDA : 1 MLA)bothbothbothdivision of labor: policy where cheap, shelf where sharp

06fMochi replay #5: survival with resolution

The switch hits. The junk dies. The cat lives. And one layer in four never forgot anything anyway.

Run the story through Kimi Linear: in the KDA layers, the context switch drops $\alpha$ hard on the channels carrying subplot junk while the channels carrying named-entity facts hold near 1 — Mochi rides through the switch at ~95% strength while the junk dies at 10%. And if the pet-name channels had been sacrificed — capacity is still capacity — the query "Ada's cat is named ____" can still be rescued by the next MLA layer, whose compressed shelf kept token 2's actual key and value, individually addressable, all along. Two memories, two failure modes, interleaved so each covers the other.

Payoff: fine-grained forgetting isn't a luxury — it's what lets a fixed-size memory hold fast facts and slow facts at once; and pairing it with a periodic exact shelf covers the residue. Handoff: Kimi Linear is the memory system. Kimi K3 is what happens when you build a 2.8-trillion-parameter product around it. Time to audit the whole machine — and the 22,580 number with it.

06gConcept check

Chapter 6 · one question
In the 3:1 KDA:MLA hybrid, why keep ANY full-attention layers, given that KDA is strictly cheaper?

07Kimi K3: the full system — and the 22,580 payoff

The hook from page one comes due. Where do 2.8 trillion parameters actually live, and how many of them are new ideas?

Kimi K3's language backbone reported: 23 macrocycles of four layers each — KDA, KDA, KDA, MLA — for 92 layers total. The first layer's FFN is dense; every other layer's FFN is a latent mixture-of-experts with 898 experts (2 shared + 16 of 896 routed per token). Blockwise AttnRes taps sit every 12 layers. Gated MLA, MLA query LoRA, and a new activation (SiTU) round out the change list. Several of these specifics come only from the source worklog and Kimi's release notes; where we could not independently verify a number, it wears the reported tag rather than false confidence.

Now the audit the hook demanded. If you decompose 2.8T by where parameters sit, the split is brutally lopsided: at these expert counts, an estimated ~98% of parameters are MoE expert FFNs (19 in 20 is the conservative floor) — a structure whose core idea (sparsely-activated expert mixtures) dates to Shazeer et al., 2017. Attention — all of it, KDA's projections and decay knobs, MLA's latents, AttnRes's queries — is a rounding error next to the experts. By raw parameter count, K3 is overwhelmingly "more of the same."

And that is exactly why the parameter count is the wrong lens. The sub-1% is the part that decides whether a million-token context is reachable at all: the eviction policy (KDA), the compressed shelf (MLA), the depth-wise read (AttnRes). Scale bought the library; the new ideas bought the librarian. The accountant below lets you flip between the three honest views of the 22,580 number.

07bThe parameter accountant

Three views of one model: what it stores, what it runs per token, and how each compares to GPT-2. All arithmetic shown in the readout — check it.

MoE expert FFNs attention (KDA + MLA + AttnRes) embeddings & other active slice

07c92 layers, drawn to scale

The macrocycle isn't a metaphor — here is the actual layer map, with the AttnRes taps where they land.

KDA layer (69 of 92) MLA layer (23 of 92) AttnRes tap (every 12th boundary) dense FFN (layer 1 only; all others latent MoE)

One arithmetic note the source worklog left dangling: 92 layers with a boundary every 12 gives seven interior boundaries (12, 24, …, 84) — the eighth snapshot is the top of the stack. That's how "every 12 layers" and "eight AttnRes blocks" reconcile reported.

07dSix changes, triaged by what they're for

A release-notes list gives every change equal billing. A teardown shouldn't. Three buckets, called out loud.

Bucket A — the memory story (full treatment)

  • The 3:1 macrocycle at scale — Chapter 6's hybrid, now the spine of a 2.8T model. KDA supplies constant-state recurrent memory; every 4th layer retains full softmax retrieval over the (compressed) context.
  • Blockwise AttnRes — the genuinely new memory idea in K3, big enough to be Chapter 8. Teaser: it applies this lesson's central insight to a dimension we haven't touched yet.

Bucket B — capacity (the honest middle)

  • Latent-space MoE reported — 898 experts: 2 shared (process every token) + 16 of 896 chosen per token by a dot-product router. The K3 twist: experts operate in a compressed latent space — inputs are down-projected before the shared experts and the sum is up-projected after — reportedly nearly halving expert FLOPs. Same librarian's trick as MLA, applied to the FFN: do the heavy lifting in a smaller room. This bucket is where ~95% of the parameters live, and its core idea is the oldest thing in the model.

Bucket C — engineering (one honest paragraph each, not the plot)

  • Gated MLA — element-wise output gate on the attention result, projected from the input: how much of each retrieved feature actually enters the residual stream. A volume knob on the reading glasses (same spirit as $\beta$ on writes: learned trust, everywhere).
  • MLA query LoRA reported — low-rank query projections in the MLA layers; a parameter-efficiency tweak, not a mechanism change.
  • SiTU activations — next section, because it carries a lesson about kernels that generalizes.

07eSiTU and the kernel tax

A three-line activation change, and the systems lesson hiding inside it.

Standard MoE experts use SwiGLU: split the up-projection into a gate half and a value half, $\mathrm{SiLU}(\text{gate}) \odot \text{up}$, then down-project. K3 swaps in SiTU reported, which wraps both halves in scaled $\tanh$ soft-clamps:

python · situ (as reported)
d = x.shape[-1] // 2
gate = x[..., :d].to(torch.float32)
up   = x[..., d:].to(torch.float32)
a = beta * torch.tanh(gate / beta) * torch.sigmoid(gate)   # SiLU with a tanh cap
if linear_beta is not None:
    up = linear_beta * torch.tanh(up / linear_beta)          # cap the value path too
return (a * up).to(x.dtype)

Why cap? $\beta \tanh(x / \beta) \approx x$ for small $x$ but saturates at $\pm\beta$ — a soft ceiling on activation magnitudes, which is exactly what you want when training and serving in very low precision (K3 reportedly ships quantization-aware in MXFP4): outliers are what break low-bit formats. The float32 casts in the middle of a bf16 model are part of the same discipline.

The kernel tax, a recurring systems lesson: as reported, without a fused kernel this activation runs almost 3× slower than the SwiGLU path it replaces — not because tanh is expensive, but because a naive implementation makes multiple round-trips to HBM for tensors that should never leave registers. The offset: latent-space experts nearly halve the FLOPs the activation sits inside. Every architecture change in this lesson eventually meets this same accountant — Chapter 4 was nothing but this meeting, scheduled early.

07fScorecard row 8

The system, in one row — and the thesis, now earned.

The scorecard · row 8 of 8
ArchitectureStoresWriteReadEviction
Kimi K369 fixed $d{\times}d$ states + 23 compressed shelves + 898-expert library + 8 depth snapshotsper-channel fade·erase·write; append latents; route to 18 experts$qS$ cheap · softmax sharp · router's pick · depth-attentionfour policies, each where it's cheapest

Look back at the accountant. "Is it just scale?" now has a three-part answer: by storage, 22,580×, and ~95% of that is the oldest idea in the building. By per-token compute, hundreds. And by mechanism — the thing this lesson actually traced — K3 is six edits to one update equation plus a division of labor, all of it living in the smallest slice of the parameter budget. Scale is real; it is also the least informative axis along which K3 differs from GPT-2.

Handoff: one change is left, and it closes the loop. Look one more time at the equation of a residual network — not the attention, the skip connections — and you will find Chapter 2's additive memory hiding in plain sight, running vertically. Same disease. Same cure.

07gConcept check

Chapter 7 · one question
K3 reports 2.8T total parameters with 2 shared + 16-of-896 routed experts per token. What's the right way to compare it to a 124M dense GPT-2?

08The same disease in a new organ

Forget attention for a minute. Look at the skip connections.

In every pre-norm transformer — GPT-2 included, since Chapter 1 — each block adds its output to a running stream: $h_{l+1} = h_l + f_l(h_l)$. Unroll the recursion from the embedding up:

the residual stream, unrolled — look familiar? $$ h_l = h_1 + \sum_{i=1}^{l-1} f_i(h_i) $$
  • $h_1$ — the current token's embedding, at the bottom of the stack.
  • $f_i(h_i)$ — what block $i$ (attention or MLP/MoE) computed and deposited.
  • The sum — everything every block ever wrote, weighted equally, nothing ever removed.

You have seen this equation before. It is Chapter 2's additive memory — $S_t = \sum_i k_i^{\top} v_i$ — running vertically over depth instead of horizontally over time. The residual stream is a fixed-size vector ($d_{\text{model}}$ wide) into which 92 layers pour their outputs by pure addition. And it inherits the additive memory's exact pathologies, translated one-for-one:

Additive memory over time (Ch 2)Residual stream over depth
facts interfere as N outgrows dearly-layer features drown as depth outgrows $d_{\text{model}}$ — "residual dilution"
nothing can be evicted, only buriedno block can remove a predecessor's contribution — only shout over it
late writes must be large to register over accumulated contentlate layers learn ever-larger outputs to influence the pile — hidden-state norms grow, destabilizing training
the fix: learned, selective access (delta, gates, KDA)the fix: …the same one. Selective, learned access. Attention.

08bAttnRes: attention over depth

One learned query per boundary, keys made from earlier snapshots, softmax over the depth axis. The trick you already know, rotated 90 degrees.

AttnRes replaces the equal-weight sum with a learned, per-token weighting of earlier states:

attnres — the residual sum, with learned weights (using w to avoid clashing with the decay α of Ch 5–6) $$ h_l = w_0 \cdot h_1 + \sum_{i=1}^{l-1} w_i \cdot f_i(h_i), \qquad \mathbf{w} = \mathrm{softmax}\big(\text{depth-scores}\big) $$

Where do the scores come from? A dot product, of course: each boundary owns a small learned query vector; the keys are normalized versions of the earlier snapshots; the softmax runs over the snapshot axis, per token. A layer that needs the raw embedding can reach down and pull it up at full strength, 80 layers later, without it having to survive 80 additions. Selective retrieval from earlier depth, exactly as attention is selective retrieval from earlier time.

Doing this at every one of 92 layers would be expensive and mostly redundant, so K3 does it blockwise reported: a snapshot is banked every 12 layers (plus the top — 8 total, from the Chapter 7 strip), and the mixing happens over that small library. As reported, the machinery adds under 2% compute overhead while improving training efficiency by roughly 25% — i.e., reaching equal loss with about $1/1.25$ of the compute. (The source worklog rendered this as "2% inference latency" against "a 1.25× compute advantage," which garbles what's being bought with what; the clean statement is: a small constant tax on every forward pass, purchasing a large discount on the training bill reported.)

08cThe depth lens

Eight snapshots feed the top of the stack. Plain residuals weight them equally; AttnRes lets different tokens pull different depths. Toggle and compare.

snapshot weight hidden-state norm growth

08dThe einsum, annotated to death

Four lines carry the whole mechanism. Every tensor named, every axis explained.

python · blockwise attnres core (as reported), with shapes
V = torch.stack(blocks + [partial_block])   # (N+1, B, T, D): the library —
                                             # N banked snapshots + the in-progress one
K = norm(V)                                  # normalized keys: direction, not magnitude,
                                             # decides relevance (so loud layers can't cheat)
logits = torch.einsum('d, nbtd -> nbt', q, K)
                                             # q: (D,) — ONE learned query for this boundary.
                                             # one score per snapshot, per token position
h = torch.einsum('nbt, nbtd -> btd', logits.softmax(0), V)
                                             # softmax over axis 0 = the SNAPSHOT axis —
                                             # NOT over tokens. each position mixes depths;
                                             # positions never mix with each other.

Note what's deliberately small: one query vector per boundary (a few thousand parameters — the whole mechanism is measured in kilobytes), softmax over at most 9 snapshots. This is the cheapest attention in the entire model, and per reported numbers among the highest-leverage. The pattern to take away: wherever an unweighted sum is silently acting as a memory, a dot-product softmax read is waiting to replace it.

08eCoda: the completed table

Eight rows, seven years. Read it top to bottom and the arc tells itself.

The scorecard · complete
ArchitectureStoresWriteReadEviction policy
GPT-2 (2019)every K,V — O(N)appendsoftmax, sharpnone — perfect memory, unbounded rent
Linear attention (2020)fixed $d{\times}d$add $k^{\top}v$$qS$, blendedaccidental — interference buries
DeltaNet (2021)fixed $d{\times}d$erase-then-write$qS$targeted replace
+ chunked training (2024)no change to the memory — the eraser became trainable at scale (T-matrix)
Gated DeltaNet (2024)fixed $d{\times}d$fade·erase·write$qS$replace + global decay
KDA / Kimi Linear (2025)fixed $d{\times}d$ (+1 MLA shelf per 4 layers)per-channel fade·erase·write$qS$ + periodic softmaxreplace + per-channel decay + division of labor
Kimi K3 (2026)69 states + 23 shelves + expert library + 8 snapshotsall of the above + routingall of the above + depth-attentionfour learned policies, composed
the residual stream itselfone $d_{\text{model}}$ vector over depthevery block addswas: equal-weight sum → now: AttnResthe ninth row the table didn't see coming

So: was it just scale? 22,580× the stored parameters, yes — and roughly 98% of them are 2017's idea, sparsely woken. The arc that actually distinguishes 2026 from 2019 fits in one sentence, and by now you can verify every clause of it by hand: a fixed-capacity associative memory needs an eviction policy; purely additive updates make eviction an accident; so the field learned to erase (delta), to fade (gates), to fade with resolution (KDA), to keep a compressed exact shelf for what must never blur (MLA), and finally to notice that the residual stream was one more additive memory in need of the same medicine (AttnRes). Learned selection — gating, routing, decay — is the recurring answer, and a dot-product softmax read is its sharpest instrument.

The number to leave with isn't 22,580. It's four — the four eviction policies running concurrently inside one model, each spending its capacity exactly where its failure mode is cheapest. Twenty-two thousand GPT-2s was never the story. One GPT-2 that finally learned to forget — that's the story.

Where to next on this site

08fFinal check

Chapter 8 · the lesson in one question
What is the structural parallel between AttnRes and the linear-attention story of Chapters 2–6?