Sparse Autoencoders & Crosscoders

Chapter 1: Superposition, Recapped

Chapter 0 asserted that a language model packs far more concepts into its residual stream than it has raw dimensions to spend on them. This chapter earns that claim just enough to build on — a recap, not a rebuild. The full derivation, with the geometry and the interference math worked out in detail, is the subject of Toy Models of Superposition on this site; treat what follows as the minimum toolkit, not the whole story.

What this chapter owes to, and withholds from, the full derivation

Before the recap itself, one more word on scope. Superposition, done properly, involves real optimization theory — characterizing exactly which configurations of feature importance, feature sparsity, and available dimensions a network converges to under gradient descent, including phase transitions where the optimal packing strategy changes qualitatively as sparsity crosses certain thresholds. None of that machinery appears in this chapter, deliberately. What follows is the minimum causal story needed to justify building a sparse autoencoder at all: dimensions are scarce, concepts are not, near-orthogonality provides some slack, sparsity of co-occurrence is what makes that slack usable. Toy Models of Superposition earns each of those claims rigorously, with the phase-transition structure worked out in full; this chapter simply asserts them, clearly labeled as a recap, and moves on to what they imply for the rest of this lesson.

The dimension-counting problem

A model's hidden dimension — call it d — is fixed at pretraining time. GPT-2 small's is 768. But the number of genuinely distinct, useful concepts a large language model has learned almost certainly runs into the hundreds of thousands, if not millions: individual words and subword patterns, grammatical roles, factual associations, stylistic registers, code-syntax patterns, and endless narrower categories nested inside those. Call that number of concepts m. When m is enormously larger than d, you cannot give every concept its own perfectly separate, mutually orthogonal direction — there simply aren't enough independent directions in a d-dimensional space to go around. A d-dimensional space has, at most, exactly d directions that are all pairwise at 90° to each other.

The escape hatch: near-orthogonality plus sparsity

Two facts rescue the model from this apparent dead end. First, directions don't have to be exactly orthogonal to be usable — they only have to be orthogonal enough that interference between them stays small. In high dimensions, most randomly chosen directions are already almost perpendicular to each other purely by chance; the expected cosine similarity between two random unit vectors in d dimensions shrinks like 1/√d as d grows. In a 768-dimensional space, that's already a small number, and real models are far wider than that at their largest layers. So a model can pack in many more than d nearly-orthogonal directions — each one picks up a little bit of leakage from the others, but not enough to matter, if not too many of them are active at the same time.

Second — and this is the part that makes the leakage tolerable rather than catastrophic — most concepts are sparse in normal text. Any given token is usually about a handful of things at once (its part of speech, perhaps a topic, perhaps a syntactic role), not thousands of things simultaneously. When only a small fraction of all possible concepts are active on any one token, the cross-talk between near-orthogonal directions mostly averages out rather than compounding into something that breaks the model's predictions.

What this means for reading activations directly

Superposition is a genuinely good trade for the model — it gets to represent far more than its raw width would otherwise allow. But it is a bad deal for anyone trying to read the resulting activations by eye. Because concepts are stored as linear combinations across many raw coordinates rather than one coordinate per concept, no single raw coordinate is dedicated to any one concept. That is precisely Chapter 0's dog / sunset / semicolon toggle: three concept directions, sharing two raw coordinates, none of the raw coordinates cleanly “belonging” to any one of the three.

This reframes the interpretability problem precisely. The concepts do have directions in activation space — they're just not the raw coordinate axes. If you could find the right set of directions — more of them than raw dimensions, each one corresponding to (approximately) one concept, most of them inactive on any given input — you would be reading the model's actual representation, rather than a tangled linear mixture of it.

What a sparse autoencoder is trying to invert

That reframing is exactly the specification for what this lesson builds. Superposition is a compression: the model packs m sparse concepts into d dense raw coordinates. A sparse autoencoder tries to invert that compression — take the dense, d-dimensional activation vector back out, and recover an overcomplete (meaning: more directions than d) and sparse (meaning: only a handful active per input) representation, in the hope that this recovered representation lines up with the model's actual underlying concepts rather than an arbitrary rotation of them.

Nothing about this inversion is guaranteed to work perfectly, and nothing here proves it does — that's an empirical question, and Chapter 7 is entirely about how the researchers behind both papers tried to check it honestly. But the specification itself follows directly from the superposition picture: build something wider than d, force it to use only a few of its directions per input, and see whether the directions it lands on turn out to mean something.

One geometric fact this chapter leaned on, made explicit

It's worth isolating the single geometric fact that made the entire near-orthogonality argument work, since it gets reused implicitly throughout the rest of this lesson. For two independently, randomly chosen unit vectors in d dimensions, the expected value of their dot product is exactly zero (by symmetry — there's no preferred direction for either vector to lean toward), and the typical size of that dot product (its standard deviation) shrinks proportionally to 1/√d as d grows. This is a purely geometric fact about high-dimensional spaces, true regardless of what those directions are later used to represent — it has nothing to do with neural networks, gradient descent, or language specifically. What Chapter 1's argument adds on top of this generic fact is the observation that a trained model gets to exploit it: because random directions in a wide space are already mostly non-interfering for free, a model under pressure to represent more concepts than it has dimensions has a genuinely cheap, geometry-provided escape hatch available, rather than needing to invent some more exotic packing strategy from scratch.

A boundary worth stating precisely

It's worth being precise about what this chapter has and has not established, so the rest of the lesson doesn't quietly overreach. Superposition explains why raw coordinates are hard to read and what kind of object could fix that (an overcomplete, sparse basis). It does not, by itself, prove that any particular overcomplete sparse basis you might train actually recovers the model's real concepts rather than some other sparse-but-arbitrary rotation. That gap — between “this is the right shape of solution” and “this particular trained solution is any good” — is exactly why Chapter 7's evaluation metrics exist, and exactly why this lesson keeps returning to honest limitations rather than declaring victory the moment a sparse autoencoder trains.

Putting a number on "nearly orthogonal"

Make the 1/√d claim from above concrete, because the size of that number is doing real work in the argument. For GPT-2 small, d = 768:

A typical pair of random directions in a 768-dimensional space sits at roughly 0.036 cosine similarity — which is to say, about 96.4% of the way to being perfectly perpendicular, purely from being embedded in a space this wide, before the model has done anything clever at all. For a much wider layer — say d = 12,288, the publicly known hidden size of GPT-3's largest configuration — the same calculation gives:

Roughly a quarter of the interference, for a model with 16× the hidden dimension. Wider models get “more room” for near-orthogonal directions almost for free, purely as a side effect of high-dimensional geometry — which is part of why superposition is such a robust, model-size-independent phenomenon rather than something only small models resort to.

So why doesn't scale make superposition (or the need for SAEs) go away?

This raises a fair question: if wider models get cheaper near-orthogonality, shouldn't scaling up eventually make superposition, and the whole problem this lesson addresses, disappear? The honest answer, previewed here and returned to properly in Chapter 5, is no — and not because the geometry argument above is wrong, but because the number of concepts a model wants to represent (m from earlier in this chapter) grows alongside model scale too, often faster than the hidden dimension does. A bigger model doesn't just get a wider residual stream to work with; it also learns a great many more distinct things worth representing. The scaling paper's own finding on this point, which Chapter 5 examines directly, is that larger subject models require larger, not smaller, sparse autoencoders to reach the same reconstruction quality — the opposite of what you'd expect if wider hidden dimensions alone were solving the crowding problem.

The interference cost, made explicit

It's worth seeing why small leakage between near-orthogonal directions doesn't sum up to something dangerous, at least under the sparsity assumption. Suppose a query direction has a cosine similarity of ε with some unrelated, currently-inactive concept's direction. If that unrelated concept is off (not contributing to the current activation), its stored direction contributes exactly zero interference — ε only matters at all when both directions are simultaneously carrying nonzero signal. With m possible interfering directions each active independently with some small probability p, the expected number of simultaneously active interferers is roughly mp — and as long as p stays small enough that mp stays modest even while m is huge, the total expected interference (each contributing on the order of ε in magnitude) stays bounded. This is precisely why the sparsity half of Chapter 1's opening claim is not optional decoration: without it, m simultaneously active directions each leaking ε onto every other one would compound into activation noise large enough to break the model's predictions outright.

A basis-ambiguity problem worth flagging early

There's a subtlety in “recover the right overcomplete sparse basis” that's easy to miss on a first pass, and it resurfaces directly in Chapter 7. Suppose a trained sparse autoencoder achieves excellent reconstruction and genuinely sparse codes. Nothing about that success, by itself, proves the specific directions it landed on are the model's actual underlying concepts, rather than some other, equally sparse-and-reconstructive rotation of them. Sparsity picks out a sparse basis; it doesn't uniquely pin down the one basis a human would recognize as meaningful, unless the true underlying features themselves happen to be the uniquely sparsest explanation of the data — which is a real empirical bet this whole research program is making, not a mathematical guarantee. This is exactly why Chapter 7 needs four separate, independent evaluation metrics rather than trusting reconstruction-plus-sparsity alone: each metric is, in part, a different way of checking whether the recovered basis lines up with something a human (or a downstream probe) would actually call a real concept.

Why language, specifically, is a good candidate for the sparsity assumption

It's worth asking why the sparsity half of this argument is a safe bet for language models specifically, rather than an assumption of convenience. Natural language is well known to follow something close to a Zipfian distribution — a small number of concepts (common words, common grammatical roles, common topics) account for a large share of all tokens, while a very long tail of rarer, more specific concepts (a particular technical term, a particular narrative motif, a particular unusual syntactic construction) each show up only occasionally. On any single token, only a handful of concepts from this entire distribution are actually relevant at once — you don't need “legal boilerplate,” “DNA base pairs,” and “15th-century Portuguese poetry” all active simultaneously to process an ordinary sentence. That's precisely the empirical property superposition needs to be a viable strategy at all, and it's a property of language itself, not something the model has to work to produce — the raw material the model is trained on already arrives sparse in the relevant sense, at the level of which concepts are in play at any one moment.

Sparse in feature space, versus sparse across time

Everything in this lesson enforces sparsity within one activation vector — at a single token position, only a few of the n latents may be active. That's one specific way to force disentanglement, but not the only conceivable one. It's worth a brief preview, since Chapter 9 returns to it as a genuine bridge to another lesson on this site: instead of (or in addition to) penalizing how many latents are active per token, you could instead exploit the fact that language unfolds sequentially, and penalize a different quantity entirely — for instance, whether a feature's semantic content stays predictable from one token to the next while its syntactic content does not, using a contrastive loss across time rather than an L1 or TopK penalty within one vector. That is a genuinely different disentangling strategy, not a variant of this one, and Temporal Sparse Autoencoders on this site builds it from scratch. Worth keeping in mind here only as a reminder that “sparsity” is one specific, successful lever for forcing features apart — not the only lever that could work.

Why Chapter 0's toy example was deliberately not near-orthogonal

It's worth returning to Chapter 0's dog/sunset/semicolon toggle with this chapter's near-orthogonality machinery in hand, because the toy example was, on purpose, a worst case rather than a realistic one. Compute the actual cosine similarities between those three hand-picked 2D directions:

None of these are anywhere close to zero — −0.866 in particular is nearly as far from orthogonal (cosine 0) as two directions in a plane can get. That's not a mistake in the toy example; it's the entire point. A 2-dimensional space can hold at most 2 directions that are genuinely, perfectly orthogonal to each other — asking it to hold 3 forces real, unavoidable overlap, exactly the ambiguity Chapter 0 built the toggle to demonstrate. Compare this to the 1/√d estimate from earlier in this chapter: at d = 768, three randomly chosen directions would be expected to sit far closer to orthogonal (cosines on the order of 0.036, not 0.17 to 0.87). The toy example's severe overlap was a deliberately low-dimensional worst case, used to make the ambiguity vivid; real residual streams, at real widths, start from a far more forgiving geometric position — though, as Chapter 1's main argument establishes, still nowhere near forgiving enough to avoid needing sparsity as well.

What changes, and what doesn't, as d grows

Tie together the last two sections into one table, since both the packing-capacity argument (this chapter) and the compute-cost argument (Chapter 0) move in opposite directions as the hidden dimension grows, and it's easy to lose track of which is which.

Wider models get cheaper near-orthogonality in one narrow geometric sense, and simultaneously get more expensive to build and harder to later interpret. Superposition doesn't go away as models scale — if anything, every practical pressure in that table pushes toward relying on it more, not less.

Chapter 2: Sparse Autoencoders, Derived

Chapter 1 specified the target: an overcomplete, sparse basis that reconstructs a model's dense activations. This chapter builds the actual machine that produces one — and it is a genuinely old idea, wearing a new hat.

An older idea, from a different field

Before language models existed, signal-processing researchers faced a strikingly similar problem with images. Dictionary learning (Mallat and Zhang, 1993) asks: given a collection of signals (say, small patches of natural images), can you learn a large, overcomplete set of reusable building blocks — called atoms, together forming a dictionary — such that any one signal is well-approximated by a sparse combination of just a few atoms? Olshausen and Field (1996) showed that doing this on natural image patches, completely unsupervised, produces atoms that look strikingly like the receptive fields of real neurons in the visual cortex — edges and oriented bars at different scales and orientations. Sparsity, applied to the right kind of data, tends to rediscover structure that was already there.

A sparse autoencoder is this same idea, aimed at a language model's activations instead of image patches, implemented as a small neural network trained by gradient descent instead of solved by classical optimization. The “dictionary” is a matrix of learned atom directions; the “sparse code” for one activation vector is which atoms are active, and by how much.

The encoder and decoder, precisely

Take an activation vector x ∈ ℝ^d from some layer of the model — the residual stream vector Chapter 0 opened with. Choose n latent dimensions, with n deliberately much larger than d (overcomplete, per Chapter 1). The baseline recipe, following Bricken et al. 2023's formulation as used in the scaling paper, is:

where W_enc ∈ ℝ^n×d, b_enc ∈ ℝⁿ, W_dec ∈ ℝ^d×n, and b_pre ∈ ℝ^d. Read it left to right: subtract a learned bias b_pre from the raw activation (centering it), project up into the wide n-dimensional space with W_enc, add a per-latent bias, and clip everything negative to zero with ReLU. That last step, ReLU, is what will eventually force sparsity — a latent that would have gone negative reports exactly zero, contributing nothing to the reconstruction and receiving no gradient this step. The decoder runs the cheapest possible reverse map: a single linear layer W_dec back down to d dimensions, plus the same b_pre added back.

The training objective, and why sparsity must be enforced explicitly

The ReLU baseline's training loss is:

The first term, reconstruction mean-squared error, is the obvious half: make x̂ look like x. On its own, though, that term alone would not produce anything interesting — with n > d and no other constraint, the easiest way to minimize reconstruction error is to just copy the input through in some rotated, redundant form, using as many of the n latents as convenient. Nothing would force the code to be sparse, and nothing would force individual latents to specialize in one recognizable concept rather than smearing across many. That's exactly the failure mode an ordinary (non-sparse) overcomplete autoencoder falls into: it becomes a fancy, expensive way to store the same tangled representation you started with.

The second term, λ‖z‖₁ (the L1 penalty, sum of the absolute values of z's entries, scaled by a tunable coefficient λ), is what forces the issue. It directly penalizes the total amount of “activation mass” the code is allowed to spend. To minimize this penalty while still reconstructing well, the optimizer is pushed toward solutions where most z_i are exactly zero for any given input, and the few that are nonzero are each pulling a lot of reconstruction weight — which in turn pressures each active latent toward representing something specific and useful, rather than a diffuse smear that only helps a little. Chapter 3 examines this L1 penalty's mechanics — and its real cost — in detail.

Why sparsity, mechanically, pushes toward monosemanticity

It's worth making the connection between “sparse” and “interpretable” explicit rather than taking it on faith. Two unrelated concepts sharing one latent (a polysemantic latent, the very problem Chapter 0 opened with) only causes real damage to the reconstruction when both concepts happen to be active on the same input at the same time — that's when their contributions interfere and the shared latent can't cleanly represent both. If the code is forced to be sparse, the model gets to choose, for each input, which handful of latents actually fire; over the course of training, it is cheaper — in terms of the reconstruction loss the network is actually optimizing — to route rarely co-occurring concepts onto separate latents than to keep folding them together and repeatedly paying an interference penalty whenever they happen to overlap. Sparsity does not guarantee monosemanticity, but it removes the main incentive that would otherwise favor tangled, polysemantic latents.

The shapes, made concrete

Ground this in real numbers from the scaling paper's own setup. For GPT-2 small, d = 768 (the model's hidden size), and one of their trained dictionaries uses n = 32,768 — a 42.7× overcomplete dictionary. For their largest run, on GPT-4 series activations, n reaches 16 million — Chapter 5 works out exactly how large an overcompleteness ratio that implies, and why going that wide turns out to matter.

python
import torch, torch.nn as nn

class ReLUSparseAutoencoder(nn.Module):
    def __init__(self, d_model, n_latents):
        super().__init__()
        self.W_enc = nn.Parameter(torch.randn(n_latents, d_model) * 0.02)
        self.b_enc = nn.Parameter(torch.zeros(n_latents))
        self.W_dec = nn.Parameter(torch.randn(d_model, n_latents) * 0.02)
        self.b_pre = nn.Parameter(torch.zeros(d_model))

    def forward(self, x):                          # x: (batch, d_model)
        z = torch.relu((x - self.b_pre) @ self.W_enc.T + self.b_enc)  # (batch, n_latents), mostly 0
        x_hat = z @ self.W_dec.T + self.b_pre               # (batch, d_model)
        return x_hat, z

def sae_loss(x, x_hat, z, lam=1e-3):
    mse = ((x - x_hat) ** 2).sum(-1).mean()
    l1 = z.abs().sum(-1).mean()
    return mse + lam * l1, mse, l1

Why squared error, not absolute error, for reconstruction

Worth a brief aside, since the letters “L1” are about to do double duty in this lesson — Chapter 3's L1 sparsity penalty is a completely different design choice from the reconstruction term, which uses squared (L2) error, ‖x − x̂‖²₂, not absolute (L1) error. Squared error penalizes large individual mistakes disproportionately more than small ones — a single coordinate off by 2 contributes four times the penalty of one off by 1, not merely twice — which is exactly the right shape for a reconstruction target, where you want the optimizer to prioritize fixing a few badly-reconstructed coordinates over tolerating them in exchange for many coordinates that are only slightly off. Absolute error would treat those two situations as interchangeable. Keeping the two L1's straight matters: one (reconstruction) is never used in this lesson's recipes; the other (sparsity, on z) is the one Chapter 3 spends an entire chapter deriving the cost of.

Tracing one gradient step, symbolically

Make the abstract “gradient descent trains the dictionary” claim from earlier concrete by writing out, in full, what the encoder's gradient actually looks like for a single active latent. Let L = ‖x − x̂‖², with x̂ = W_decz. The chain rule gives:

Read this literally: the gradient pushing on latent i's activation is (twice) the current reconstruction error, projected onto that latent's own decoder direction. A latent whose decoder direction points toward where the reconstruction is currently failing gets pushed to increase; one pointing away from the error gets pushed to decrease. And because z_i = 0 for every latent TopK didn't select, ∂z_i/∂(pre-activation_i) = 0 there too (the ReLU-like gate has zero slope below its cutoff) — exactly the “latents that lose the top-k competition get zero gradient this step” claim Chapter 4 relied on, now derived rather than asserted.

The GPT-4 preprocessing detail, and why it matters

One quiet but important detail from the paper's actual setup: before any activation reaches the encoder, it's preprocessed — the mean over the d_model dimension is subtracted, and the result is normalized to unit norm. This isn't cosmetic. Raw activation magnitudes can vary a lot across tokens and positions for reasons that have nothing to do with which concepts are present (a long, information-dense sentence might simply produce larger-norm activations than a short one). Without normalizing that away first, the autoencoder would waste some of its capacity learning to track overall magnitude rather than which concepts are present — exactly the kind of nuisance variation that would make the recovered directions less clean. Every number in this lesson that references “the activation” going into an SAE should be read as already having passed through this normalization step, following the paper's real recipe (Section 2.1).

What happens with no sparsity term at all: a quick failure case

It's worth actually seeing why an overcomplete autoencoder without sparsity collapses into something useless, rather than taking the claim on faith. Set λ = 0 in the loss from above — pure reconstruction, no penalty on z. With n > d and no other constraint, one trivial solution is: let W_enc contain the d×d identity matrix padded with (n−d) rows of zeros, and let W_dec undo exactly that. Every input reconstructs perfectly, using d of the n latents every single time, with the remaining (n−d) latents permanently unused. Reconstruction loss is driven to zero, and absolutely nothing has been decomposed — the “solution” is just a relabeled copy of the original tangled coordinates, dressed up as a wider vector. This is exactly the failure mode Chapter 2's main text warned about, made concrete: reconstruction alone has no preference for a sparse, spread-out, one-concept-per-latent solution over a trivial dense copy. The sparsity term is not a refinement on top of a working recipe; without it, there is no decomposition happening at all.

What the two biases are actually for

The two bias vectors, b_pre and b_enc, are easy to skim past as boilerplate. Each has a distinct, motivated job. b_pre is subtracted before encoding and added back after decoding — it centers the activation distribution around a learned origin before the sparse code has to explain anything, so the dictionary's directions only need to account for deviation from a typical activation, not for the typical activation's own baseline offset. Without it, every one of the n latents would waste some of its capacity re-representing whatever constant offset is common to nearly all activations at that layer. b_enc is a per-latent threshold, shifting how easily each individual latent crosses zero under ReLU — it lets the network learn that latent #4,821 should activate on weaker evidence than latent #19, rather than forcing every latent to share the exact same all-or-nothing cutoff at exactly zero pre-activation.

Tied versus untied decoders: a distinction worth keeping straight

Chapter 6 will introduce initializing the decoder as the transpose of the encoder (W_dec = W_enc^T at the start of training) as part of the fix for dead latents. It's worth flagging now, so the two ideas don't blur together later: initializing tied and training tied are different choices. The scaling paper's baseline recipe, as written above, keeps W_enc and W_dec as two entirely separate, independently-learned parameter matrices throughout training — tying only shows up as a starting point for the decoder's values, not as a permanent constraint forcing the two to stay identical as gradient descent proceeds. Chapter 4's worked example uses full, permanent tying (W_dec = W_enc^T the whole way through) purely to keep the hand arithmetic in that chapter manageable — a deliberate simplification, not the paper's actual recipe.

The forward pass, with real shapes traced end to end

Trace the exact tensor shapes for one GPT-2 small activation, n = 32,768, d = 768, to make Chapter 2's diagram unambiguous:

python
x = torch.randn(1, 768)                 # one activation, d_model=768
sae = ReLUSparseAutoencoder(d_model=768, n_latents=32768)
x_hat, z = sae(x)
print(z.shape)                        # torch.Size([1, 32768])  -- 42.7x wider than x
print((z != 0).sum().item())      # typically a few hundred nonzero, out of 32,768
print(x_hat.shape)                    # torch.Size([1, 768])   -- back to the original width

The overcompleteness ratio here, 32,768 / 768 ≈ 42.7×, is one specific configuration the paper trains at GPT-2 small scale. It is nowhere near their largest — Chapter 5 works out what happens when the same recipe is pushed to 16 million latents, and by how much the effective sparsity (the fraction of the dictionary active on any given token) has to shrink to keep the code usable at that width.

Why the encoder and decoder are allowed to differ

One structural question worth answering explicitly: W_enc and W_dec are both, dimensionally, n×d-shaped objects (one is literally the transpose-shape of the other), yet the baseline recipe lets them be two entirely independent, separately-learned matrices rather than forcing W_dec = W_enc^T throughout training. The reason traces to the genuinely different job each one does. The encoder's job is detection: given a raw activation, decide which of n possible concepts are present, a comparison-and-selection task where what matters is how a candidate direction's dot product with the input ranks against every other candidate's. The decoder's job is synthesis: given a sparse description (which latents are active, and how strongly), reconstruct a real activation vector, a task about how directions combine and interfere with each other when several are summed together. There's no mathematical law requiring the direction that best detects a concept's presence to be identical to the direction that best reconstructs that concept's contribution once detected — and empirically, letting them differ gives training more freedom to solve each sub-problem well on its own terms. Chapter 6's tied-at-init trick borrows a reasonable starting point from this relationship without permanently enforcing it, exactly the distinction flagged earlier in this chapter.

The lineage of this idea, in order

It's worth seeing the full chain of work this recipe descends from, because none of it appeared in one step, and the scaling paper's own related-work section (Section 7) traces it carefully. Mallat and Zhang (1993) introduced sparse coding over an overcomplete dictionary. Olshausen and Field (1996) — already met earlier in this chapter — showed the dictionary itself could be learned, unsupervised, directly from data, and that doing so on natural images recovers edge-detector-like atoms resembling real visual cortex neurons. Hinton and Salakhutdinov (2006) proposed the autoencoder architecture itself, for dimensionality reduction, independent of any sparsity requirement. Combining these threads, Lee et al. (2007), Le et al. (2013), and Konda et al. (2014) trained autoencoders with explicit sparsity priors like the L1 penalty — the ReLU-plus-L1 recipe Chapter 2's main text builds. Makhzani and Frey (2013) then introduced the k-sparse autoencoder, replacing the L1 penalty with a direct TopK selection rule — Chapter 3's whole argument, more than a decade before the scaling paper applied it to GPT-4.

Only in the last several years did this machinery get pointed at language models specifically: Yun et al. (2021), Sharkey (2022), Bricken et al. (2023) (whose Towards Monosemanticity established the exact ReLU baseline recipe Chapter 2 uses), and Cunningham et al. (2023) all independently found that sparse dictionary learning recovers genuinely interpretable features from language model activations. Marks et al. (2024) pushed one step further, showing the resulting features could be assembled into sparse circuits — causal chains of features explaining a specific model behavior, not just a catalog of isolated concepts. The scaling paper (Chapter 5 onward) and the crosscoders paper (Chapter 8 onward) are the most recent links in this same chain — not a fresh idea, but eight decades of signal-processing and machine-learning lineage, compressed and scaled up to run on the largest language models available.

Chapter 3: L1 vs TopK Sparsity

Chapter 2 built the ReLU-plus-L1 recipe and asserted that the L1 penalty forces sparsity. It does — but it has a real, quantifiable cost, and that cost is exactly what motivates the second sparsity mechanism this lesson needs: TopK.

Restating the stakes before the derivation

It's worth being clear about why this chapter's derivation matters beyond pure mathematical tidiness. Chapters 6 and 7 both depend on whether a latent's reported strength can be trusted at face value — Chapter 6's dead-latent detection checks whether a latent activated at all, and Chapter 7's probe and N2G metrics both work by examining the actual magnitude and pattern of activations a latent produces. If those reported magnitudes are systematically distorted by the training mechanism itself, every downstream measurement built on top of them inherits that distortion. This chapter's shrinkage derivation isn't a side note about L1 specifically — it's the reason later chapters can trust TopK's reported activation values as a reasonably faithful signal in the first place.

What the L1 penalty is actually approximating

The quantity you'd really like to control is L0 — literally, the number of nonzero entries in z. That's the honest measure of “how sparse is this code.” But L0 is not differentiable (it's a discrete count, flat almost everywhere, with an infinite step where a value crosses zero), so you can't optimize it directly with gradient descent. L1 — the sum of absolute values — is the standard convex relaxation of L0, the same substitution the LASSO uses in classical statistics (Tibshirani, 1996): it is differentiable almost everywhere, and it still favors sparse solutions, because concentrating a fixed total magnitude onto a few large entries costs the same L1 penalty as spreading a smaller total across many entries — but the few-large-entries solution reconstructs better for the same penalty paid.

The tax L1 charges on every survivor

Here's the part that's easy to miss: L1 does not distinguish between “this latent should be off” and “this latent should be on, but smaller than it really is.” The penalty grows linearly with the magnitude of every active latent, not just the ones that end up at zero. That means the optimizer is rewarded for shrinking every surviving activation toward zero, a little, even the ones that genuinely should be firing strongly — a systematic bias called activation shrinkage (Tibshirani, 1996; Wright and Sharkey, 2024, in the SAE context specifically).

Derive exactly how much shrinkage happens, for the simplest possible case: one latent, with true pre-activation a, penalized by λ per unit of |z|. Minimize (1/2)(a − z)² + λ|z| over z. For z > 0, the derivative is (z − a) + λ, set to zero: z = a − λ. For z < 0, symmetric: z = a + λ. If |a| ≤ λ, the optimum is z = 0 (the penalty outweighs the value of keeping it on at all). Combine these into the closed-form solution, the classic LASSO soft-thresholding operator:

Read what this means for a latent that genuinely should be active: even when the raw evidence for it, a, is large and well above the threshold, the value that survives is a minus a flat λ, every single time — not a proportional discount, an absolute one, subtracted regardless of how large a was to begin with.

Worked example: the shrinkage, in real numbers

Take a latent with true pre-activation a = 5.0 (a strong, genuine signal for whatever concept this direction represents), and a modest penalty λ = 0.8:

The reported activation strength is 4.2, not 5.0 — an 16% understatement of the true signal, paid on every single token where this latent fires, for as long as training uses this penalty. Nothing about the direction is wrong; only its reported magnitude is systematically off, in the same direction, every time.

Why TopK sidesteps this entirely

The scaling paper's fix, following Makhzani and Frey (2013)'s original k-sparse autoencoder, replaces the ReLU-plus-L1 combination with a single activation function that directly controls L0 instead of approximating it through a penalty:

Compute every latent's raw pre-activation exactly as before, then keep only the k largest values and zero out every other one — no soft-thresholding, no penalty term subtracted from the survivors. The decoder is unchanged. And because sparsity is now enforced structurally by the activation function itself, rather than by a loss-function penalty the optimizer has to be talked into respecting, the training loss simplifies back down to just the reconstruction term:

No λ to tune, and — this is the direct payoff of the derivation above — no shrinkage. A latent that survives into the top-k keeps its full, un-penalized value. Redo the worked example: with TopK instead of L1, a = 5.0 stays exactly 5.0 if it's among the k largest, full stop.

A boundary case worth checking: what happens as λ → 0?

Push the soft-thresholding formula to its edge to confirm it behaves sensibly there, a useful habit for trusting any derived formula. As λ → 0, the L1 penalty vanishes entirely, and z* = sign(a)·max(|a|−0,0) = a — the ReLU+L1 recipe with no penalty collapses back into an ordinary ReLU autoencoder with no sparsity pressure at all, exactly the degenerate, non-decomposing case Chapter 2 warned an unconstrained autoencoder falls into. And as λ → ∞, every z_i gets driven to exactly zero regardless of a — total sparsity, total loss of reconstruction ability. The useful, interesting regime for training a real SAE sits strictly between these two extremes, which is precisely the tuning burden Chapter 3's earlier section identified TopK as sidestepping: with ReLU+L1, finding that useful middle λ is a search; with TopK, k is set directly and the corresponding sparsity level is simply guaranteed, no search required.

The paper's own list of TopK's benefits

The scaling paper is direct about why it switched, and the reasons go beyond just removing shrinkage:

• Removes the need for the L1 penalty entirely, and with it, the shrinkage bias derived above.
• Sets L0 directly, instead of tuning a λ coefficient and hoping it lands near a desired sparsity — this makes comparing different autoencoder sizes and configurations much cleaner, since sparsity is a knob you set exactly rather than a side effect you estimate after training.
• Composes with other activation functions — TopK is a selection rule, not a shape, so it can be layered on top of other nonlinearities if useful.
• Empirically wins on the reconstruction-sparsity frontier — at a fixed number of latents, TopK reconstructs better than ReLU at the same L0, and this gap widens as the autoencoder scales up.
• Increases monosemanticity of what activates on random examples, by clamping every small, marginal activation to exactly zero rather than letting a long tail of tiny, noisy activations survive.

How Gated SAEs split the problem differently

Gated SAEs (Rajamanoharan et al., 2024) deserve one more sentence of mechanics beyond the name-check above, because the design idea is genuinely different from both ReLU+L1 and TopK, not just a third variant of the same trick. Instead of one function deciding both whether a latent fires and how strongly, split those into two separate learned pathways: a “gate” sub-network decides which latents should be active at all (trained with its own sparsity pressure), and a completely separate magnitude sub-network estimates how strongly each gated-on latent should fire, trained purely to minimize reconstruction error, free of any sparsity penalty pulling its estimates down. Because the magnitude pathway never sees the sparsity penalty directly, it has no structural reason to underestimate active latents' true strength — the same shrinkage problem this chapter derived for ordinary L1, solved by architectural separation rather than by switching to a hard selection rule the way TopK does. The scaling paper's own comparison found TopK still edges out Gated SAEs on the overall reconstruction-sparsity frontier, but the fact that two structurally different fixes (TopK's hard cutoff, Gated's decoupled magnitude estimation) both independently improve on plain L1 is itself good evidence that shrinkage was a real, fixable design flaw rather than an unavoidable cost of sparsity in general.

A comparison, side by side

What refining the survivors reveals

The paper runs a clean diagnostic that confirms this derivation empirically, not just theoretically: freeze which latents are active (the sparsity mask), then use projected gradient descent to re-optimize only the magnitude of the already-active latents to minimize reconstruction error, holding the decoder fixed. If shrinkage is really happening, this refinement should systematically want to increase the ReLU+L1 model's activations (since they started off penalized below their true optimal value) but should barely move the TopK model's activations (since they were never penalized to begin with). That is exactly what the paper finds: the refinement step is strongly biased upward for ReLU models and small, unbiased in direction, for TopK models. The math derived above and the empirical measurement agree.

Tuning λ versus setting k: a practical difference, made concrete

The paper's claim that TopK “simplifies tuning” is easy to nod along with and easy to underrate. Make it concrete: suppose the actual goal is an average sparsity of 32 active latents per token — a specific, chosen operating point on the reconstruction-sparsity tradeoff. With ReLU+L1, there is no direct way to ask for that; λ is the only knob, and the relationship between λ and the resulting average L0 depends on the data distribution, the current state of training, and the dictionary size, none of which are known in closed form ahead of time. In practice this means training a model with a guessed λ, measuring what average L0 came out, and adjusting λ up or down for the next run — an iterative search, often several runs deep, to land near a target sparsity. With TopK, you write k = 32 once, and every single token, throughout the entire run, uses exactly 32 active latents — the target sparsity isn't approximated after the fact, it's the literal, guaranteed output of the activation function from the very first training step. This is precisely why the paper singles this out as enabling “simpler model comparison and rapid iteration”: comparing two ReLU+L1 runs at “the same sparsity” requires first confirming their resulting average L0s actually matched; comparing two TopK runs at the same k requires nothing to confirm at all.

A second worked case: when shrinkage correctly kills a latent

The soft-threshold formula also needs to be seen doing its intended job, not just its unwanted one. Take a latent with a genuinely weak, marginal pre-activation, a = 0.5, under the same λ = 0.8:

Here the penalty is doing exactly what it's supposed to: a weak, ambiguous signal gets suppressed to exactly zero rather than contributing noisy, low-confidence activation to the reconstruction. The problem identified above isn't that L1 suppresses weak signals — that part is working as intended, and TopK does something structurally similar (weak signals lose the top-k competition). The problem is specifically that L1 also taxes the strong, unambiguous signals that clearly should survive, by exactly the same flat amount, every time. TopK's hard cutoff separates these two cases cleanly: below the cutoff, exactly zero; above it, untouched.

Other approaches to the same shrinkage problem

TopK is not the only fix researchers have proposed for L1 shrinkage, and the scaling paper is explicit about naming its competitors rather than pretending TopK arrived in a vacuum. Gated SAEs (Rajamanoharan et al., 2024) split the encoder into two separate jobs — one sub-network decides which latents should be active, a second, independent sub-network estimates how strongly, so the magnitude-estimation half never has to also carry the burden of the sparsity penalty. ProLU (also called JumpReLU or TRec; Taggart, 2024) replaces ReLU's zero-cutoff with a learned positive threshold θ, zeroing anything below it while passing everything above it through completely unpenalized — structurally similar in spirit to TopK's hard cutoff, but with a per-latent learned threshold instead of a global rank-based one. The paper compared all of these directly and found TopK gives the best overall reconstruction-sparsity frontier among them — but the existence of multiple independently-developed fixes for the same shrinkage problem is itself good evidence that the problem was real and worth solving, not an artifact of one paper's particular framing.

TopK's own hidden cost: overfitting to its own k

TopK is not free of tradeoffs either, and it's worth surfacing one now rather than letting it look like a strictly dominant solution. Train a TopK autoencoder with some fixed k, then, after training, swap in a different value k′ at test time and see how reconstruction quality behaves as k′ varies. The paper finds that MSE keeps improving for k′ slightly above the training k (consistent with Chapter 4's Step 5, where raising k from 3 to 4 improved the toy reconstruction) — but then gets substantially worse as k′ grows much further past k. The autoencoder has, in a real sense, overfit to operating at exactly its training sparsity level; it was never taught what to do with a wildly different number of active latents.

The fix, Multi-TopK, is simple to state: instead of training against the loss for a single k, sum losses computed at several different k values during training, e.g.

Training against both k and 4k simultaneously (with the second term downweighted) is enough to produce a genuinely progressive code — one where using more of the sorted latents at test time keeps monotonically improving reconstruction, all the way up, rather than degrading past whatever k the model happened to be trained at. The cost is small: Multi-TopK trained models score very slightly worse than plain-TopK models specifically at their own training k, a fair trade for not falling apart when k′ differs from k.

The one thing L1 can do that TopK structurally cannot

It's worth being fair to the older recipe before closing this chapter. TopK's biggest practical strength — fixing L0 exactly — is also, unavoidably, a limitation: every single token, regardless of how simple or complex the text at that position is, gets forced through exactly k active latents, never more, never fewer. A token that is genuinely simple to explain (a common word in a predictable context) is not rewarded with a sparser code, and a token that is genuinely unusual and hard to explain doesn't get to borrow a few extra latents to cover it. L1, by contrast, lets the effective sparsity vary token by token: an easy token can settle for very few active latents (since the penalty makes marginal ones not worth their cost), while a harder token can recruit more, paying more penalty in exchange for better reconstruction where it's actually needed. Chapter 9 returns to this as one of the scaling paper's own stated limitations of TopK — worth flagging now, precisely because it is a direct, structural consequence of the same hard cutoff that made TopK immune to shrinkage in the first place. The two properties are not separable: the same rigidity that prevents shrinkage is what removes per-token flexibility.

Chapter 4: The k-Sparse Recipe, By Hand

Chapters 2 and 3 gave you the equations. This chapter runs one, completely by hand, on numbers small enough to track every step — a toy example, invented for this lesson (not data from either paper), designed so the arithmetic stays honest and the mechanics stay visible.

Why hand-tracing a toy example is worth the effort

It's fair to ask why a lesson built around a paper that trains autoencoders with sixteen million latents spends an entire chapter on six hand-picked numbers. The answer is that every mechanic visible at n = 6 — the ReLU-then-TopK ordering, competition between overlapping atoms, the gap between “dictionary completeness” and “sparsity level,” and (in the extended sweep below) the difference between an untrained and a trained decoder — is exactly the same mechanic operating at sixteen million latents, just impossible to watch directly at that scale. Nobody can hand-trace a real forward pass through a 16-million-latent dictionary; everybody can trace one through six numbers written on a page. This chapter is the bridge between reading Chapters 2 and 3's equations and actually trusting that you understand what they compute, not a simplified stand-in for the real thing.

Setting up the toy dictionary

Take a tiny “residual stream” with d = 4 raw dimensions, and an overcomplete dictionary of n = 6 latent directions — a 1.5× overcomplete toy, far smaller than any real model, but enough to see TopK actually make a choice. Following Chapter 6's tie-the-decoder-to-the-encoder initialization trick a step early, set each dictionary atom (a row of W_enc, and the matching column of W_dec) as one fixed direction, with all biases at zero for simplicity:

The first four atoms are the raw coordinate directions themselves — if the dictionary only had these four, it would be an ordinary (non-overcomplete) basis. Atoms 5 and 6 are “distractors”: each one mixes two raw coordinates together (both are unit-norm, since 0.6² + 0.8² = 1.0), the kind of learned direction a real SAE might land on if two raw coordinates tend to move together.

Take the toy activation x = [2.0, −1.0, 0.5, 1.5].

Step 1 — compute every pre-activation by hand

Each pre-activation is just a dot product, a_i = w_i·x:

Step 2 — apply ReLU first (the detail almost everyone forgets)

Per the paper's own footnote on their exact recipe, a ReLU is applied before TopK selects, guaranteeing every candidate the top-k competition considers is non-negative. Here, that zeroes out a₂ and a₆, both negative:

Notice immediately: only four of the six latents are even eligible to be selected. This is a real, general consequence of ReLU-before-TopK, not a quirk of this toy example — the practical sparsity level is min(k, number of positive pre-activations), never more.

Step 3 — TopK with k = 3

Sort the four positive candidates: a₅ = 2.4, a₁ = 2.0, a₄ = 1.5, a₃ = 0.5. Keep the top 3, zero the rest:

Latent 3 — the one atom that would have perfectly captured x's third coordinate with zero cross-talk — loses out to the distractor atom 5, purely because 2.4 > 0.5. This is a real and important failure mode, not an artifact of a bad toy: when a “distractor” direction happens to produce a larger raw score than a genuinely useful one, TopK's hard cutoff has no way to know that and simply keeps the larger number.

Step 4 — reconstruct, and measure the damage

The decoder is the same three surviving atoms, weighted by their z values:

Compare against the true x = [2.0, −1.0, 0.5, 1.5]:

Step 5 — increase k to 4, and watch the improvement

Raise k to 4: the next-largest surviving candidate, a₃ = 0.5, joins the code.

A small, genuine improvement (1.7525 → 1.69) — exactly the “MSE keeps improving as k increases past what training used” behavior the scaling paper describes (Section 5.3), here traced by hand on four numbers instead of read off a plot.

Step 6 — a boundary this toy dictionary can never cross

Push k to 5 or 6 — use every latent, sparsity disabled entirely. Nothing changes: latents 2 and 6 were zeroed by ReLU before TopK ever got a vote, so raising k past 4 adds only zero-valued terms. The reconstruction freezes at [3.44, 0, 0.5, 3.42] no matter how large k gets, and the second coordinate of x̂ stays exactly zero forever.

That's not a sparsity limitation — it's a dictionary completeness limitation, and it's worth distinguishing the two clearly. None of this toy's six hand-picked atoms point in the direction needed to represent a negative amount of the second raw coordinate; w₂ = [0,1,0,0] can only ever push coordinate 2 positive, never negative, once ReLU is in the loop. No value of k fixes a dictionary that never learned the atom it needed. A real, trained SAE avoids this specific failure because gradient descent, given enough capacity and enough data, keeps adjusting the atoms themselves — not just which ones get picked — until the dictionary actually spans what the data requires. This toy example fixed the dictionary by hand precisely so you could see what happens when it doesn't.

python
import numpy as np

W_enc = np.array([[1,0,0,0],[0,1,0,0],[0,0,1,0],[0,0,0,1],
                  [0.6,0,0,0.8],[0,0.6,0.8,0]])   # (n=6, d=4)
W_dec = W_enc.T                                        # tied init, Chapter 6
x = np.array([2.0, -1.0, 0.5, 1.5])

def topk_sae_encode(x, W_enc, k):
    a = np.maximum(W_enc @ x, 0)             # ReLU first (footnote 3, Section 2.3)
    idx = np.argsort(a)[::-1][:k]              # indices of the k largest
    z = np.zeros_like(a)
    z[idx] = a[idx]
    return z

for k in [3, 4, 6]:
    z = topk_sae_encode(x, W_enc, k)
    x_hat = W_dec @ z
    mse = np.mean((x - x_hat) ** 2)
    print(k, x_hat.round(2), round(mse, 4))
    # 3 [3.44 0.   0.   3.42] 1.7525
    # 4 [3.44 0.   0.5  3.42] 1.69
    # 6 [3.44 0.   0.5  3.42] 1.69   -- unchanged, per Step 6

A second worked example: a dictionary that hasn't gone wrong

Step 3's failure — the distractor atom w₅ beating the genuinely useful w₃ into the top-k — deserves a contrast case, so it's clear this is a specific, fixable failure and not TopK's normal behavior. Replace w₅ with a cleaner atom that doesn't compete for the same budget: w₅′ = [0, 0, 0, 0.1] (a very weak, nearly-inert direction, the kind a well-trained dictionary would learn to assign to a rarely-useful pattern rather than to something that collides with genuinely common structure). Recompute a₅′ = w₅′·x = 0.1(1.5) = 0.15, small and easily beaten by a₃ = 0.5 in the top-k ranking.

With k = 4, the top four candidates are now a₁=2.0, a₄=1.5, a₃=0.5, a₅′=0.15 (a₆ stays zeroed by ReLU as before):

Compare: 0.2501 versus Step 4's 1.7525. Same k, same x, same overall dictionary size — the entire difference is whether one specific atom collides with a genuinely useful direction or stays out of its way. The only remaining error is dimension 2 (x₂ = −1.0), which — exactly as Step 6 found — no k can fix, because still no atom in this six-atom dictionary points toward negative coordinate 2.

The full k = 1 through k = 6 sweep — and a genuine surprise

Steps 4–5 traced k = 3 and k = 4. Finish the sweep, from k = 1 all the way to k = 6, using the same messy (distractor-atom) dictionary, and something worth pausing on shows up.

MSE does not fall monotonically as k grows — it gets worse from k=1 to k=3, before improving slightly at k=4. That's the opposite of what Chapter 3's Multi-TopK discussion described as the normal behavior of a real, trained autoencoder (reconstruction improving as k grows, up to around the training k). The reason is visible directly in the x̂ column: at k=2 and k=3, w₁ and w₄ both get added on top of w₅, which already contributes partial weight to coordinates 1 and 4 (its 0.6 and 0.8 components). Adding w₁'s full weight to a coordinate w₅ was already covering causes overshoot — coordinate 1 goes from 1.44 (an undershoot of the true 2.0) at k=1, to 3.44 (a substantial overshoot) once w₁ joins in at k≥2. Coordinate 4 does the same thing when w₄ joins at k=3: 1.92 becomes 3.42, sailing well past the true value of 1.5.

What gradient descent would actually do about the first (messy) case

The frozen, hand-picked dictionary from Steps 1–6 is a snapshot, not a trained result — it's worth being explicit about what training would do to it if this exact collision (w₅ beating w₃ on inputs like x) kept recurring across many training examples. The reconstruction loss's gradient with respect to W_enc and W_dec flows only through whichever latents actually survived TopK selection on a given example — latents that lose the top-k competition get exactly zero gradient that step, since they contributed nothing to x̂. So on inputs shaped like x, gradient descent would push up on w₅'s decoder direction to better match what it's actually being asked to reconstruct (since it won the slot, its error is what's being corrected), and it would push w₃ to receive no update at all on this example, precisely because it never got the chance to contribute. If, across the training set, w₃ only ever wins its slot rarely, it improves slowly; this is a direct preview of Chapter 6's dead latent problem — a persistently out-competed latent doesn't just reconstruct badly, it can stop receiving useful gradient signal almost entirely, which is exactly why the paper needed a dedicated fix rather than trusting ordinary training to sort it out on its own.

python
# one illustrative gradient step, direction only (not a full training loop)
def one_step_direction(x, z, W_dec, lr=0.05):
    x_hat = W_dec @ z
    grad_x_hat = -2 * (x - x_hat)             # d(MSE)/d(x_hat)
    # only latents with z_i != 0 (i.e. the ones TopK selected) receive gradient
    active = np.nonzero(z)[0]
    for i in active:
        W_dec[:, i] -= lr * grad_x_hat * z[i]     # atom i's direction nudged toward the residual
    return W_dec
    # latents NOT in `active` (like w3, when w5 wins instead) get no update at all this step

What this toy example inherits, at real scale

Keep the shape of this chapter's two contrasting cases in mind, because it is exactly what Chapter 6 scales up to millions of latents. The messy dictionary's failure — a distractor atom (w₅) beating a genuinely necessary one (w₃) into the top-k slot, purely because its raw pre-activation happened to be larger on this particular input — is a single, one-time instance of exactly the competitive dynamic that, repeated across billions of training tokens and millions of candidate latents, produces the dead-latent problem: a latent that consistently loses close competitions to a more dominant neighbor never accumulates the gradient signal it would need to become a better, more distinctively useful direction in its own right. The six-atom toy dictionary here is small enough that you can watch one such collision happen in full, by hand, in a few lines of arithmetic. At sixteen million latents, the same dynamic is happening constantly, invisibly, and Chapter 6's auxiliary loss is the paper's answer to it.

Chapter 5: Scaling Laws: 16 Million Latents on GPT-4

Everything so far has used toy numbers or GPT-2 small's comparatively modest 768-dimensional residual stream. This chapter follows the scaling paper to its actual headline result: a sparse autoencoder with 16 million latents, trained on GPT-4's own residual stream activations, over 40 billion tokens — and the scaling laws that let the authors predict, ahead of time, roughly what such a run would cost and achieve.

The exact experimental setup

Precision matters here, so take the setup exactly as specified. Autoencoders are trained on the residual stream of two model families: GPT-2 small, and a series of models sharing GPT-4's architecture and training setup, including GPT-4 itself. The layer chosen is deliberately not the very last one — for GPT-2 small, that's layer 8 (three-quarters of the way through its 12 layers); for the GPT-4 series, five-sixths of the way through the network (the paper does not disclose GPT-4's exact layer count, so this lesson won't invent one). The reasoning: a layer near the end should carry many general-purpose features without being so late that it's already specialized purely for producing the very next token, which would bias what gets found toward next-token-prediction machinery rather than the model's broader conceptual vocabulary. Every experiment uses a context length of 64 tokens, and every activation is preprocessed exactly as Chapter 2 described — mean subtracted over the d_model dimension, then normalized to unit norm — before it ever reaches the encoder.

Two different ways to spend a training budget

Given a fixed amount of compute, there are two different questions you could ask about how to spend it, and the paper is explicit that they give different answers. L(C), following the same compute-optimal-frontier logic used for pretraining language models themselves (Kaplan et al. 2020; Hoffmann et al. 2022), asks: for a fixed compute budget, what's the best MSE achievable, disregarding whether training ran to full convergence? L(N) instead asks: train to convergence (within some tolerance), ignoring compute cost, and see what the autoencoder's reconstruction quality caps out at.

The paper flags a real problem with L(C) as a fair comparison across different autoencoder sizes: a wider dictionary (larger n) has a looser information bottleneck than a narrower one, so a lower MSE is mechanically easier to reach purely by being wider — independent of whether the wider dictionary is actually finding better features. Comparing MSE across different n at fixed compute is, in the paper's own words, “arguably unprincipled.” L(N), training to convergence, is the cleaner comparison, though more expensive to run. In practice, a real training run sits somewhere between the two, spending less compute than full convergence would need but more than a bare compute-optimal frontier point would use.

What convergence itself costs, as n grows

Two empirical findings about L(N) training are worth internalizing, because they explain why scaling autoencoders is its own engineering discipline, not just “run the same recipe bigger.” First, the largest learning rate that still converges shrinks as n grows, roughly like 1/√n — wider dictionaries need gentler updates. Second, and separately, the optimal learning rate for L(N) training turns out to be about four times smaller than the optimal learning rate for L(C) training at the same n — training all the way to convergence is measurably more sensitive to overshooting than merely reaching a good compute-optimal point. Third, the number of tokens needed to reach convergence grows with n, but sublinearly: approximately Θ(n^0.6) for GPT-2 small autoencoders, and Θ(n^0.65) for the GPT-4 series. Note what sublinear growth implies: doubling the dictionary size does not double the tokens needed to train it to convergence — it grows by a factor of 2^0.65 ≈ 1.57 for the GPT-4-series exponent, meaningfully cheaper per-latent than a naive linear guess would suggest, which is part of why scaling to 16 million latents was tractable at all.

An irreducible floor: why loss never reaches zero

Scaling laws of this shape sometimes fit better with an added constant — an irreducible loss term e, so that y = αx^β + e rather than a pure power law through the origin (a form used elsewhere for generative modeling scaling laws, Henighan et al. 2020). The paper finds this extra term substantially improves the fit for both L(C) and L(N). The honest interpretation offered: some of what's in a real activation vector may simply be unstructured noise — not a feature at all, just entropy that no sparse code, however large, can compress into a meaningful direction. If that's right, no amount of scaling the autoencoder ever drives reconstruction error to exactly zero; there's a floor set by how much of the signal is genuinely structured in the first place.

The joint scaling law, in full

Putting sparsity (k) and dictionary size (n) together, the paper reports a single joint fit on GPT-4 series autoencoders (valid in the small-k regime, since reconstruction becomes trivially easy once k approaches d_model itself):

Read the signs, because each one has a real consequence. γ is negative, which means the n-scaling law gets steeper (loss falls faster per unit of ln n) as k grows — sparser codes benefit less from added width than denser ones do, up to the point where k starts approaching d_model. η is also negative, meaning the irreducible-loss floor itself shrinks as k grows — more active latents per token give the code more room to explain even the less-structured parts of the signal, chipping away at what otherwise looked like an unavoidable floor.

Worked example: plugging real numbers into the formula

Rather than take “clean scaling laws” on faith, compute L(n,k) directly from the fitted constants above, at two dictionary sizes the paper actually trains, holding k = 32 fixed (a sparsity level used throughout their Figure 4).

First, n = 131,072 (2¹⁷). ln(131,072) = 17 ln 2 ≈ 11.7835; ln(32) = 5 ln 2 ≈ 3.4657.

Now n = 16,000,000 — roughly the paper's headline GPT-4 dictionary size, and about 122× wider than the first case, at the same k = 32. ln(16,000,000) ≈ 16.5881; the k-only term₂ is unchanged (it doesn't depend on n):

122× more latents, at fixed sparsity, buys a drop from ≈0.419 to ≈0.300 — about a 28% reduction in normalized reconstruction loss. That's a real, quantifiable improvement, and also a genuinely modest one for two orders of magnitude more parameters — exactly the kind of shallow, diminishing-returns curve the small magnitude of β_n (−0.017) predicts. Scaling sparse autoencoders is not free, and it is not close to linear; it is a real but expensive lever, which is precisely why the paper spends as much effort as it does on the dead-latent and evaluation problems in Chapters 6 and 7 — squeezing genuine quality out of every latent matters more once each additional order of magnitude of width buys less and less.

Reading the compute-optimal frontier the way the paper's own Figure 1 does

Before the joint (n, k) law, the paper first establishes the simpler L(C) power law: for a fixed compute budget, plot the best achievable reconstruction loss across every autoencoder size tried, and connect the best-performing point at each budget. The resulting frontier follows a power law closely, with one honest caveat the paper flags directly: the very smallest models tested fall noticeably off that trend line — underperforming what the fitted power law would predict for their size. This is a familiar shape from language-model scaling laws too (Kaplan et al., Hoffmann et al.): power-law fits are typically most reliable in a middle-to-large regime, and can break down at the smallest scales, where other effects (optimization difficulty, insufficient data relative to model size) start to dominate. Reading a scaling law honestly means reading where it was actually fit well, not extrapolating it uncritically to regimes the underlying data barely covered.

Worked example: the sublinear cost of convergence

Make the Θ(n^0.6) and Θ(n^0.65) token-to-convergence exponents concrete by asking what a 4× increase in dictionary size actually costs in training tokens, for each subject-model family. For GPT-2 small (exponent 0.6):

Quadrupling n costs only about 2.3× the training tokens to reach convergence — genuinely sublinear, and a real, practical saving over what a naive linear guess (4× the tokens for 4× the latents) would predict. For the GPT-4 series (exponent 0.65, per Section 3.1.2):

Slightly steeper than GPT-2 small's exponent, but still comfortably sublinear. This sublinearity is precisely what makes a run like the 16-million-latent GPT-4 autoencoder tractable at all: if convergence cost scaled linearly with n, scaling up the dictionary 500× from a modest starting point would have cost 500× the tokens; at these fitted exponents, it costs a much smaller multiple.

python
import math

def scaling_L(n, k):
    a, bk, bn, g, z, e = -0.50, 0.26, -0.017, -0.042, -1.32, -0.085
    ln_n, ln_k = math.log(n), math.log(k)
    term1 = math.exp(a + bk*ln_k + bn*ln_n + g*ln_k*ln_n)
    term2 = math.exp(z + e*ln_k)
    return term1 + term2

print(round(scaling_L(131_072, 32), 4))    # 0.4189 -- matches the hand derivation
print(round(scaling_L(16_000_000, 32), 4))  # 0.2997 -- matches the hand derivation
print(round(4 ** 0.6, 2), round(4 ** 0.65, 2))  # 2.3  2.46 -- the two token-cost multipliers above

What 64 tokens of context leaves on the table

It's worth sitting with one honest limitation of this entire setup before moving on: every experiment in this chapter used a context length of just 64 tokens — a handful of sentences. The paper itself flags this directly as a limitation (returned to in Chapter 9): GPT-4's actual deployed context windows run to many thousands of tokens, several orders of magnitude beyond what any autoencoder in this paper was ever shown during training. Any feature whose behavior only becomes legible over long-range dependencies — tracking a plot point across a long document, maintaining a persona across a lengthy conversation — is, by construction, invisible to every scaling law and every trained dictionary in this chapter. The 16-million-latent result is a genuine achievement in width; it says comparatively little about depth of context.

Sanity-checking the formula at a scale far below anything trained

One more useful habit when trusting a fitted formula: check it at a scale nowhere near what it was fit on, and confirm the result still makes qualitative sense. Try n = 4 (an absurdly small, four-latent dictionary, far below anything the paper actually trained) at k = 32 — already a contradiction, since k cannot exceed n, which is exactly the kind of boundary a scaling-law formula fit only on sensible (n > k) configurations has no obligation to handle gracefully. This isn't a flaw in the formula; it's a reminder that a fitted power law is a description of the region it was actually measured across (the paper's swept range of dictionary sizes and sparsity levels), not a universal law valid at every input you could algebraically plug in. Chapter 5's own honest caveat about the smallest tested models falling off the L(C) trend line is the same warning, stated differently: extrapolate scaling laws cautiously, and only within, or just slightly beyond, the range they were actually fit against.

The subject model's own size matters too

One more finding complicates the picture in an important, honest direction. Holding k fixed, larger subject models — the language model whose activations are being reconstructed, not the autoencoder itself — require larger autoencoders to reach the same MSE, and the exponent governing that relationship gets worse (a harder scaling law to satisfy) as the subject model grows. In plain terms: it's not just that bigger autoencoders reconstruct better in general — a bigger subject model genuinely needs a bigger autoencoder just to keep pace, and needs proportionally more of a size increase than a smaller subject model would. As language models keep growing, the sparse autoencoders needed to interpret them are on track to grow faster than linearly alongside them, not just linearly.

How much of the loss at 16M latents is even reducible by adding more latents?

The joint scaling law's two-term structure (Section 3.1.4) isn't just algebra — split the worked n = 16,000,000, k = 32 result from earlier into its two pieces and ask what fraction of the total loss each one accounts for:

At the paper's own largest tested scale, nearly two-thirds of the remaining reconstruction loss is sitting in the term that does not respond to n at all — it only shrinks by increasing k, the sparsity level, not by making the dictionary wider. Put plainly: at 16 million latents, this specific autoencoder has already run into diminishing returns from width specifically — most of what's left to gain sits behind a different knob (sparsity) entirely. That's a genuinely useful, and slightly humbling, piece of information for anyone deciding where to spend the next unit of compute: past a certain point, a bigger dictionary at the same sparsity level is chasing a shrinking fraction of the total remaining loss.

What "40 billion tokens" and "16 million latents" mean side by side

Put the headline numbers next to each other, because their ratio is itself informative. 40 billion training tokens, spread across 16 million latents, is 2,500 tokens of training data per latent on average — not an enormous amount per individual dictionary atom, especially once you remember that any single latent only receives a useful gradient update on the (rare) tokens where it's actually among the top-k selected. This directly foreshadows Chapter 6: at this scale, with this little per-latent training signal to go around, some latents are going to end up starved of updates entirely unless something is done about it.

Chapter 6: Dead Latents, and the Fix

Chapter 5 ended on a foreshadowing note: 2,500 tokens of training data per latent, on average, at 16-million-latent scale, and any individual latent only learns from the tokens where it actually wins its slot in the top-k competition. This chapter is about what happens when a latent stops winning that competition entirely — and stays that way, permanently, for the rest of training.

The precise definition

Following the convention set by Templeton et al. (2024), the scaling paper considers a latent dead if it has not activated at all — not once been among the selected top-k, or above the ReLU threshold in the baseline recipe — in the most recent 10 million tokens of training. This isn't a soft, fuzzy notion of “rarely used”; it's a hard, checkable condition, and it's checked against a window large enough that a genuinely rare-but-alive feature (one that only fires on some uncommon pattern) has plenty of opportunity to show up if it's going to.

Placing this chapter in the lesson's overall arc

Step back for a moment before the mechanism itself. Chapters 2 through 4 established what a sparse autoencoder is and how it selects which latents get to speak on a given input. Chapter 5 pushed that recipe to real, enormous scale. This chapter is about a cost that only becomes visible once you actually do that — at small dictionary sizes, with comparatively gentle competition for each top-k slot, a handful of permanently silent latents barely register. It is specifically the combination of extreme width (millions of latents) and extreme sparsity (tens of active slots) that turns an occasional inefficiency into the double-digit-percentage failure mode described below. Understanding why requires nothing new beyond what Chapters 2–4 already built — just following that same mechanism out to a regime where its consequences stop being negligible.

Why latents die: the mechanism, traced back to Chapter 4

Chapter 4's second worked example already showed the mechanism in miniature: a latent that loses the top-k competition on a given input receives exactly zero gradient from that input, because it contributed nothing to the reconstruction. If a latent is initialized (or drifts, mid-training) into a direction that rarely or never wins its slot against competing latents, it accumulates essentially no learning signal — and with no signal telling it to move toward directions where it would win, there's nothing pulling it out of that state. This is a scaled-up version of the classic “dying ReLU” problem from ordinary deep networks, where a unit that goes permanently non-positive stops receiving gradient forever — except here, the selection pressure isn't just “is this positive,” it's “is this among the top k out of potentially millions,” a much harsher bar to clear, and a much easier one to permanently lose.

A quick check: does this explain Chapter 4's own numbers?

It's worth pausing to confirm this chapter's mechanism is consistent with what Chapter 4 already demonstrated by hand, rather than treating the two as separate claims. In the messy-dictionary worked example, latent 3 (w₃) lost its top-k slot to the distractor latent 5 (w₅) on the one specific input x that chapter used. That single collision, on its own, doesn't make w₃ dead — Chapter 6's definition requires losing every single opportunity across ten million tokens, not losing once. But it is exactly the kind of event that, repeated across a large enough fraction of the training data (if w₅-like distractor patterns happen to correlate with wherever w₃-like patterns occur, for instance), compounds into permanent silence. Chapter 4 showed you one frame of a process; this chapter describes what happens when that same frame repeats for the entire length of a training run.

How bad it gets without a fix

Left unaddressed, this is not a marginal inefficiency — the paper's own ablations, run without either mitigation described below, find up to 90% dead latents at the scales they tested. Nine out of every ten rows of the encoder matrix, and matching columns of the decoder, doing precisely nothing on any training example, for the entire remainder of training.

A concrete published data point makes this less abstract: Templeton et al. (2024) trained a 34-million-latent autoencoder and found only 12 million of those latents alive by the end of training. Compute exactly how bad that is:

Nearly two-thirds of a 34-million-parameter dictionary, wasted — every one of those 22 million dictionary rows is dead weight sitting on the GPU for the entire training run and every subsequent inference pass, contributing nothing.

The two fixes

The scaling paper identifies two ingredients that, together, dramatically reduce this problem. First, initialize the encoder as the transpose of the decoder (W_enc = W_dec^T at the start of training, exactly the tying trick Chapter 4's toy example borrowed early). Rather than starting every latent's encoder direction at a random, arbitrary orientation that may or may not ever win a top-k competition, tying it to the decoder gives each latent a starting direction that is already, from step one, a real candidate reconstruction atom — a much better foothold than pure noise.

Second, and more directly aimed at latents that are already struggling: an auxiliary loss — the paper calls it AuxK, and describes it as similar to “ghost grads” (Jermyn and Templeton, 2024) — that specifically targets the currently-dead latents. At each training step, take the top k_aux latents among those currently classified as dead, and train them to model the residual — the reconstruction error the main, live latents are currently failing to explain. This gives dead latents a source of gradient signal that does not depend on winning the ordinary top-k competition against a dictionary's worth of already-strong, already-well-trained live latents; instead, they're being pointed specifically at whatever the rest of the model is currently getting wrong, a much more tractable target to compete for.

python
# conceptual sketch of the auxiliary "dead latent" loss
def auxk_loss(x, x_hat, z, dead_mask, W_dec, k_aux=512):
    residual = x - x_hat                          # what the main pathway is currently failing to explain
    dead_scores = z_preact * dead_mask             # pre-activations, restricted to currently-dead latents
    top_dead_idx = torch.topk(dead_scores, k_aux).indices
    z_aux = torch.zeros_like(z); z_aux[top_dead_idx] = dead_scores[top_dead_idx]
    residual_hat = z_aux @ W_dec.T                 # dead latents try to model the residual, not x itself
    return ((residual - residual_hat) ** 2).sum(-1).mean()
    # added to the main reconstruction loss with a small weight -- gives dead latents
    # a gradient signal that doesn't require beating live latents in the ordinary top-k race

Dying is a process, not just a starting condition

It's tempting to picture dead latents as latents that were simply unlucky at initialization and never had a chance. That's part of the story, but not the whole of it — the tie-the-decoder-to-encoder-transpose initialization from below exists precisely because starting unlucky is a real, addressable risk. The more subtle version happens mid-training: a latent can start out perfectly healthy, winning its share of top-k slots early on, and then gradually lose ground as other latents nearby in direction improve faster and begin winning the same slots more consistently. Once a competing latent has pulled decisively ahead on the inputs where both used to compete, the losing latent's win rate can fall toward zero even though nothing about its own initialization was ever a problem — it lost a race it was originally competitive in. This is why the paper's dead-latent definition is framed entirely around recent behavior (no activation in the last 10 million tokens) rather than anything about a latent's history or origin: a latent that dies at token 30 million and a latent that dies at token 3 million are, by this definition, treated identically once they're both currently silent, which is the right framing for a fix that has to operate purely on the model's present state during training, with no memory of how any individual latent got there.

Just how harsh is the top-k competition, numerically?

Put a number on exactly how selective TopK's survival bar is, because “harsh competition” is easy to say and easy to underestimate. At the 16-million-latent scale, with k = 32 (a sparsity level used throughout the paper's GPT-4-series experiments), a latent must land among the top 32 out of 16 million candidates on a given token to receive any gradient at all that step:

If latent selection were purely uniform-random (it isn't, once a latent has learned something — a genuinely useful latent should fire well above this base rate whenever its concept is actually present), a latent would expect to win its slot on roughly 2 out of every million tokens. Compare that to Chapter 2's ordinary ReLU-only case: a randomly initialized, zero-centered latent survives ReLU's threshold on roughly half of all tokens, since a random pre-activation is positive about 50% of the time. TopK at this scale is not a somewhat stricter version of ReLU's bar — it is many orders of magnitude stricter, and that gap is precisely why the dead-latent problem gets so much worse as n grows relative to k, rather than staying roughly constant.

The bootstrapping problem

This creates a genuine chicken-and-egg difficulty, worth stating precisely rather than just gesturing at. A latent needs to win some top-k slots to receive gradient updates that would improve its direction. But it needs a good direction — one that reliably matches some real, recognizable pattern in the data — to have any real chance of winning those slots in the first place. A latent that starts out mediocre, pointing in some arbitrary direction that doesn't cleanly correspond to anything, has no obvious path from “rarely wins” to “usually wins for a genuine reason,” because the gradient signal needed to travel that path only arrives on the rare occasions it happens to win anyway — and if it never wins, it never gets a single nudge in a more promising direction. “Just train for longer” does not resolve this on its own, because more training steps don't help a latent that structurally never accumulates gradient signal to begin with. That's precisely why the paper's fix targets the mechanism — giving dead latents a source of signal that doesn't depend on winning the ordinary competition — rather than simply running the existing recipe for a longer token budget.

A note on an older alternative: resampling

The scaling paper is not the first attempt at this problem, and it's worth naming the older alternative rather than presenting init-plus-auxiliary-loss as the only approach the field has tried. The baseline ReLU autoencoders trained in this same paper (following the training recipe from Bricken et al., 2023) use resampling instead — periodically detecting which latents have gone silent and reinitializing their encoder and decoder directions from scratch, essentially giving a dead latent a fresh, unrelated attempt at finding a useful direction rather than trying to revive its existing one. Ghost grads (Jermyn and Templeton, 2024) — the citation behind the auxiliary loss described earlier in this chapter — is literally titled “an improvement on resampling” in the paper's own reference list, which is the tell for how these two techniques relate: resampling is the older technique, and the auxiliary loss this chapter has been building up is one generation removed from it, modeled on ghost grads rather than being resampling wearing a different name. The scaling paper's own reported numbers (7% dead at 16M latents) come specifically from the tie-the-decoder-init-plus-auxiliary-loss combination described above, not from a resampling-based approach — worth knowing this is an active area with more than one viable design, not a single settled solution.

The result, and a second worked comparison

With both fixes in place, even the paper's largest, most sparsity-stressed run — the 16-million-latent GPT-4 autoencoder — ends training with only 7% dead latents. Put that next to Templeton's unfixed 64.7% directly:

It's worth resisting the temptation to call 7% a solved problem, though — at 16 million total latents, 7% is still:

Over a million permanently inert dictionary rows, even in the best-documented run in the paper. The fix is a large, genuine improvement — nine times fewer wasted latents proportionally — not a complete cure.

The cost of dead latents that never get pruned

One more practical consequence worth naming: nothing about training a latent that turns out to be permanently dead automatically removes it from the model. Even a latent that has not fired in the last 10 million tokens still occupies a full row of W_enc and a full column of W_dec — the same memory footprint, on disk and on the GPU, as a latent that fires constantly and represents something genuinely useful. At the 16-million-latent scale, with 7% dead, that's roughly 1.12 million latents' worth of parameters — each one d_model numbers wide in both the encoder row and the decoder column — sitting in memory indefinitely, contributing exactly nothing to any forward pass, unless a separate post-hoc pruning step is run to remove them. Detecting deadness (Chapter 6's own definition) and actually reclaiming the memory it wastes are two different engineering steps; the paper's fixes address the first, training-time problem, not the second, storage-time one.

Why this problem is specific to extreme sparsity, not autoencoders in general

It's worth being precise about why dead latents are a sparse-autoencoder-scale problem and not just an ordinary neural network training difficulty. In a dense layer, every unit gets gradient signal on every single training example, whether it fires strongly or weakly. In a k-sparse autoencoder with n in the millions and k in the tens, the overwhelming majority of latents receive zero gradient on any given example by design — that's the entire mechanism sparsity relies on. The tradeoff is structural: the same selection pressure that makes surviving latents monosemantic (Chapter 2's argument) is exactly what starves losing latents of the feedback they'd need to improve. Dead latents aren't a bug introduced by a specific implementation choice; they are close to an inevitable consequence of enforcing hard sparsity at very large scale, which is why a dedicated architectural fix — not just “train longer” — was needed.

A definition worth re-reading once more, precisely

It's worth returning to the exact wording of the dead-latent definition one more time, because the details are easy to blur after several paragraphs of consequence-tracing. “Not activated in 10 million tokens” means: across every one of the last 10 million training tokens processed, this specific latent was never once among the selected top-k (or, in the ReLU baseline, never once produced a positive pre-activation). It is not “activates less than some threshold rate” and not “activates weakly when it does fire” — a latent that fires rarely but genuinely, say once every 9 million tokens on some narrow, real pattern, would not be classified dead by this definition, precisely because the window is generous enough to give authentic rare features room to prove themselves. The definition is deliberately strict in one direction (literally zero activations, not merely few) and generous in the other (a ten-million-token window, not a much shorter one), which is exactly the combination needed to distinguish “genuinely dead” from “legitimately rare.”

What "up to 90%" would have meant, at this exact scale

Put the paper's worst-case, no-mitigation number directly against its own best-case, fixed-recipe number, at the identical dictionary size, to feel the full range the two fixes actually span:

At the scale the paper actually shipped, the gap between “no mitigation” and “this paper's recipe” is over thirteen million latents — comfortably more than the entire size of several of the paper's own smaller dictionaries (recall Chapter 2's 32,768-latent GPT-2 small example, over 400 times smaller than the number of latents this one pair of fixes rescues). Whatever the auxiliary loss's own overhead costs (the previous section's honest caveat), the alternative it's being weighed against is not a small tax to skip — it's the difference between a genuinely 16-million-latent dictionary and one where fewer than 2 million latents are doing any real work at all.

The auxiliary loss is not free either

It's worth closing this chapter with the same honesty applied to Chapters 2 and 3's mechanisms: the auxiliary dead-latent loss is a real, useful fix, not a free one. Computing it requires continuously tracking, for every one of potentially 16 million latents, how many tokens have passed since it last activated — a rolling bookkeeping cost that scales with dictionary size, running throughout the entire training run. It also introduces a new hyperparameter, k_aux (how many currently-dead latents get trained against the residual at each step), which trades off directly against training cost: too small, and dead latents recover too slowly to matter; too large, and the auxiliary computation itself starts eating into the compute budget that could otherwise go toward the main reconstruction objective. Nothing about this fix is a free lunch — it converts a training-quality problem (millions of wasted latents) into a smaller, bounded, but real, additional training-cost problem, and 7% residual dead latents even after paying that cost is the honest remainder.

Chapter 7: The Evaluation Problem

Chapters 5 and 6 showed reconstruction loss and dead-latent fraction both improving with scale. Neither of those numbers, by itself, tells you whether the resulting latents are actually interpretable — and the paper is unusually blunt about why loss alone can never certify that.

Why the reconstruction-sparsity frontier "degenerates in the limit"

Quote the paper's own honest caveat directly, because it's a sharper warning than a paraphrase would land: improving the reconstruction-sparsity frontier is not always strictly better. “An infinitely wide maximally sparse (k = 1) autoencoder can perfectly reconstruct by assigning latents densely in ℝ^d, while being completely structure-less and uninteresting.” Unpack what that means: if n is allowed to grow without bound, you can always build a lookup-table-like dictionary with one latent for every activation vector ever seen, achieving perfect reconstruction at k = 1, with zero actual concept decomposition happening — every “feature” would just be a memorized point, not a generalizable concept. Loss going down is necessary for a good SAE. It is nowhere close to sufficient. Section 4 of the paper exists specifically to close that gap with metrics that don't have this failure mode.

What "evaluation" means once there's no ground truth to check against

Ordinary machine learning evaluation usually has a ground truth to compare a prediction against — a labeled test set, a held-out answer. Sparse autoencoder evaluation does not have this luxury in the same way: there is no pre-existing list of “the model's true concepts” to check a trained dictionary against, because the entire point of this exercise is discovering what those concepts are, not confirming a known answer. Every metric in this chapter is, in a real sense, a proxy — a measurable stand-in for a property (genuine, human-legible, causally meaningful conceptual structure) that has no direct, checkable ground truth of its own. That's the honest epistemic position this entire evaluation problem starts from, and it's worth keeping in view while reading the four specific metrics below: each is a best available approximation to a question that, strictly speaking, cannot yet be answered with certainty.

Why check four different things instead of designing one perfect metric

It's worth asking, before working through all four, why the paper didn't simply invest in finding one single, comprehensive evaluation number instead. The honest answer is that no single number can currently distinguish “this dictionary reconstructs well because it found real structure” from “this dictionary reconstructs well because it's wide enough to memorize,” the exact degenerate case named at this chapter's start. Each of the four metrics below was chosen specifically because it fails differently on a memorizing, structure-less dictionary than a genuinely decomposing one would — downstream loss would still look fine either way (a memorizing dictionary can reconstruct perfectly), but probe recovery, N2G explainability, and ablation sparsity would all look noticeably worse for a dictionary of memorized points rather than real, reusable concepts, since a memorized-point “feature” corresponds to nothing generalizable that a simple probe, a short n-gram rule, or a narrow downstream effect would have any reason to line up with. Running all four together is how the paper substitutes for the single metric that doesn't yet exist.

Metric 1 — Downstream loss

The most direct test: splice the autoencoder into the actual model. Run the language model forward as normal, but at the chosen layer, replace the real residual-stream activation with the SAE's reconstruction of it, and let the rest of the forward pass continue on that substituted value. Measure how much the resulting next-token predictions degrade — via KL divergence between the two prediction distributions, and via “delta cross-entropy,” the gap versus a hypothetical perfect autoencoder's cross-entropy (used instead of raw cross-entropy, since even flawless reconstruction wouldn't drive loss to exactly zero — language modeling loss is never zero regardless).

The paper flags a specific trap in how this is often reported: prior work used the loss from ablating the activation to zero as a baseline, reporting “fraction of loss recovered” relative to that. The problem: zeroing an activation out entirely causes catastrophic damage to the model's predictions, so almost any reconstruction — even a mediocre one — recovers a large fraction of that huge gap, making even weak autoencoders look deceptively good. For completeness, the paper reports this number for their 16M-latent autoencoder anyway: a 98.2% zero-ablation fidelity — a number that, read in isolation, sounds close to perfect.

Their preferred, more honest metric instead: express downstream loss in terms of the pretraining compute it would take to train a smaller version of the language model itself to reach comparable quality. By that measure: substituting the 16-million-latent autoencoder's reconstruction into GPT-4 produces a language modeling loss equivalent to training GPT-4 with only 10% of its actual pretraining compute. Sit with both numbers side by side — 98.2% zero-ablation fidelity sounds nearly perfect; 10%-of-pretraining-compute-equivalent sounds like a serious, expensive quality gap. Both describe the exact same autoencoder. The second framing is the one the paper trusts, precisely because the first one's baseline (zero-ablation) is so catastrophic that it flatters almost anything compared against it.

Metric 2 — Recovering known features with 1D probes

The second check asks a more targeted question: if a specific, human-hypothesized concept (sentiment, which language the text is in, and so on) genuinely got its own dedicated latent, a very simple classifier should be able to find it. The paper curates 61 binary classification datasets covering exactly this kind of hypothesized property, and for each one, fits a 1D logistic probe on every individual latent (via Newton-Raphson), then records whichever single latent achieves the best cross-entropy loss for that task.

The logic: if the concept is cleanly represented by one latent, a probe restricted to that single latent alone should already classify it well — no combination of many latents required. Findings: probe score first improves, then worsens, as k increases (too little sparsity and no single latent cleanly owns the concept; too much sparsity and the concept may not get represented at all); TopK generally beats ReLU; and both substantially outperform running the identical probe directly on raw residual stream channels — concrete evidence the SAE's directions really are better aligned with these hypothesized concepts than the model's raw coordinates are. The paper is candid about this metric's limitation too: it's cheap to compute, but it can only ever test concepts a researcher already thought to hypothesize and label in advance — it has nothing to say about the concepts nobody thought to look for.

Metric 3 — Finding simple explanations: N2G

The third check flips the direction: instead of starting from a human-hypothesized concept and asking whether a latent captures it, start from a latent and ask whether a human-legible explanation can be automatically generated and verified against it. The paper uses an improved version of Neuron to Graph (N2G; Foote et al., 2023) — cheap enough to run at scale, though less expressive than a full free-text explanation approach (Bills et al., 2023, which uses a language model itself to write and check explanations, at far higher cost).

The construction, precisely: start from sequences that activate the latent; for each, find the shortest suffix that still activates it (retaining at least half the original activation strength); check whether any token in that n-gram can be replaced by a wildcard without losing the activation; check whether the explanation depends on absolute position in the sequence. Up to 16 nonzero activations build the explanation graph; another 16 serve as held-out true positives for measuring recall.

The paper reports two real qualitative examples worth sitting with directly. One latent achieves precision = 0.97, recall = 0.56 — a genuinely narrow, mostly-correct explanation (high precision means almost everywhere the explanation predicts firing, the latent really did fire; moderate recall means it still misses some of the true activating cases). A different latent scores precision = 0.06, recall = 0.05 — both numbers near zero, meaning the automatically generated explanation is essentially unrelated to what actually makes this latent fire; a nearly useless description that happened to be the best the automated method could produce.

Findings: bigger n and smaller k make latents easier for N2G to explain on average (a larger, sparser dictionary tends toward narrower, more n-gram-describable latents); and TopK autoencoders beat ReLU ones at matched n and similar L0 by a wide margin on recall (>1.5× better), with only a small precision cost (still >0.9× as good), producing a clearly better overall F1 — further confirming Chapter 3's claim that clamping small activations to exactly zero (TopK's structural behavior) produces cleaner, more explainable latents than L1's soft, shrunk-but-nonzero tail.

A worked illustration of what a 1D probe is actually fitting

It's easy to let “fit a 1D logistic probe on each latent” stay abstract. Make it concrete with a small, invented illustration — not the paper's actual data, just enough numbers to see what “well-separated” versus “poorly-separated” looks like for this specific test. Suppose a task asks “is this token part of a French sentence?” (label y = 1 for yes), and one candidate latent produces these five example activation values, paired with their true labels:

A 1D logistic probe just needs to find one threshold-like boundary that separates these two clusters — here, anything between roughly 0.3 and 4.5 works, and the achievable cross-entropy loss is very low, because the two label groups sit in cleanly separated regions of this single latent's value. Contrast a second latent, poorly aligned with the same task:

Every value clusters near 2.0 regardless of label — no threshold on this single number separates the two classes at all, and the best achievable cross-entropy is close to what random guessing would produce. The first latent is a strong candidate for “this SAE has recovered something like a French-language feature”; the second is not, regardless of how it happened to correlate with French text in some other, more complicated, nonlinear way. This is exactly the sense in which the probe metric asks whether a hypothesized concept was captured by a single latent, linearly — a real, useful, but narrow bar.

Deriving the ablation-sparsity formula from scratch

The (L1/L2)² formula for “effective number of tokens affected” looks like it's pulled from nowhere unless you derive it once. Consider the simplest possible case: a vector of logit differences where exactly m entries share the same magnitude v, and every other entry is exactly zero (a clean, maximally “sparse” effect touching exactly m things equally). Then:

In this idealized equal-magnitude case, the formula literally equals m — the true count of affected entries, exactly. For real, unequal-magnitude effects, it behaves as a soft, continuous generalization of that count — a few large entries dominating pushes the value down toward a small effective count, while many comparably-sized entries push it up toward the true count — which is exactly the “effective number of vocabulary tokens affected” interpretation the paper gives it.

Turning the reported percentages into a real vocabulary count

Instantiate the formula with GPT-2's actual, publicly known vocabulary size, V = 50,257, and the paper's T = 16 future tokens, to see what “10–14% versus 60%” means as an absolute count rather than just a percentage:

Both numbers sound large in isolation, since V×T = 804,112 total logit-position slots is itself a big number — but the ratio between them, roughly 5×, is the real finding: ablating one SAE latent moves about a fifth as much of the total logit surface as ablating one raw coordinate does, concrete evidence that individual latents act like specific, targeted levers rather than diffuse ones.

Metric 4 — Sparsity of downstream (ablation) effects

The fourth check asks a different kind of question entirely: not “does this latent correspond to a known concept” or “can it be explained,” but “when this latent is turned off, does it change the model's output logits in a focused way, or a diffuse one?” The intuition, following Olah et al. (2024): if the underlying computation is genuinely sparse, individual features shouldn't just activate sparsely — ablating them should also have sparse, surgical downstream effects, not a smear across the entire vocabulary.

The measurement: at some token position, ablate each latent one at a time, and record the resulting change in output logits across a window of T = 16 future tokens (using GPT-2 small, layer 8), subtracting the median logit shift per token first (since a uniform shift across every logit doesn't change the resulting probability distribution at all, and shouldn't count as an “effect”). Sparsity of the resulting vector of logit differences is measured via (L1/L2)², interpretable as an “effective number of vocabulary tokens affected,” normalized to fall between 0 and 1.

The numbers: SAE latents land at 10–14% — sparse, a small effective fraction of the vocabulary genuinely moved — versus 60% for ablating raw residual-stream channels directly (close to the ≈2/π ≈ 63.7% value expected for a theoretically random direction). Real, substantial evidence that individual SAE latents act like specific, targeted levers on the model's output, while raw coordinates act like diffuse, unfocused ones. The same caveat as the probe metric reappears here too: this trend reverses once k approaches d_model (768 for GPT-2 small) — at k = 512, effects become dense again, the same warning sign as before that extreme density erases whatever structure sparsity was buying.

Why N2G's construction bothers with wildcards and position checks at all

It's worth unpacking why N2G's explanation-construction procedure (Chapter 7's earlier description) goes to the trouble of checking wildcard-replaceability and position-dependence, rather than just recording the literal activating n-gram and calling it a day. A latent that fires on the exact string “New York City” and one that fires on “New <anything> City” (say, also “New Mexico City” or “New Kansas City”) are different claims about what the latent actually detects — the first is about one specific proper noun, the second about a more general template. Testing whether a wildcard can be substituted at each position without breaking the activation is exactly how N2G distinguishes these two cases automatically, rather than requiring a human to guess which pattern is the right level of generality. The position-dependence check plays a related but distinct role: a latent that fires on “stop” only when it's the very first word of a sentence is a different, narrower claim than one that fires on “stop” anywhere — and without explicitly testing whether inserting a padding token at the front changes the activation, an automated method would have no way to tell these apart either. Both checks exist because a cheap, automated method has to substitute deliberate, falsifiable tests for what a human explaining a feature would otherwise do by eye.

The four metrics, side by side

Chapter 8: Crosscoders: One Dictionary, Many Layers

Every sparse autoencoder built so far shares one structural assumption: it reads and reconstructs activations at exactly one layer. That assumption has a real cost, and this chapter's paper — Sparse Crosscoders for Cross-Layer Features and Model Diffing (Lindsey, Templeton, Marcus, Conerly, Batson, and Olah, Anthropic, October 2024) — opens by naming it directly.

The publication context, briefly

Worth situating this paper's format before diving into its content, since it affects how its claims should be read throughout this chapter and the next. Unlike the scaling paper (a full arXiv preprint, peer-reviewable in the conventional sense), the crosscoders paper is published as an entry in Anthropic's Transformer Circuits Thread — an ongoing, less formal research-notes venue the authors themselves compare to “results being presented at a lab meeting or internal seminar.” That framing, quoted directly in Chapter 9, is not false modesty; it's a genuine signal about how much scrutiny and replication this specific work has been through relative to a fully reviewed publication. This lesson treats every number and claim from this chapter with that context attached — real, carefully reported, and worth learning from in full technical detail, but explicitly preliminary in a way the scaling paper's more conventional publication is not.

Three related tools, precisely distinguished

The paper is careful to place its new tool relative to two existing ones. An autoencoder, as built in Chapters 2–7, encodes and predicts activations at a single layer — it never looks outside that one layer's numbers. A transcoder uses activations from one layer to predict the activations of the next layer — still one input, one output, but shifted forward in the network by one step. A crosscoder generalizes both: it reads from, and writes to, multiple layers at once. Note that autoencoders and transcoders are, in this framing, just special cases of the general crosscoder family — an autoencoder is a crosscoder whose input and output layer sets are both a single, identical layer.

One sentence connecting this chapter back to Chapter 2

Before the crosscoder machinery proper, it's worth stating the relationship to everything built so far as plainly as possible: a crosscoder is not a new kind of neural network. It is Chapter 2's exact encoder/decoder recipe, applied to a set of activation vectors instead of a single one, with the sparsity penalty adjusted (this chapter, below) to stay comparable to that single-layer baseline. Every piece of intuition Chapters 2–4 built about what sparsity does, why overcompleteness matters, and how TopK-versus-L1 tradeoffs work carries over unchanged — the only genuinely new ingredient is which set of vectors the encoder reads from and the decoder writes to.

Why per-layer SAEs cannot fix this by being trained better

It's worth being precise about why this is a structural blind spot of the per-layer approach, not something a bigger or better-trained per-layer SAE could eventually work around. Even a perfect, ideally-trained SAE at layer L and an equally perfect one at layer L+1 have no mechanism for recognizing that a feature persisting unchanged from one to the next is the same feature, rather than two coincidentally-similar but independent ones — each SAE was only ever shown one layer's activations, in isolation, and trained with no signal at all connecting it to any other layer's dictionary. The redundancy Chapter 8's FLOPs experiment finds is invisible to a per-layer training objective by construction, no matter how well that objective is optimized. Fixing it requires literally changing what the dictionary-learning problem is being asked to explain — from one layer's activations to several layers' activations jointly — which is exactly what a crosscoder is.

Motivating example 1: why superposition might itself be spread across layers

Here's the paper's own argument for why this generalization is natural, not just a bigger hammer. Because the residual stream is linear — each layer adds its contribution rather than overwriting what came before — two adjacent layers can be redrawn, without changing anything about what the network computes, as almost-parallel branches, connected by one extra edge letting the earlier layer influence the later one. A circuit computing a single feature could genuinely be implemented split across two layers while behaving, functionally, as though it ran in parallel — which is a completely natural thing for the model to do if it has more layers available than the minimum depth some particular computation actually needs. If features can be jointly represented this way, spread across a set of layers rather than confined to one, it's natural to apply dictionary learning to that whole set of layers jointly — which is exactly what a crosscoder does.

Motivating example 2: persistent features and circuit complexity

The paper's second motivating example is concrete enough to work through in full. Imagine features 1 and 2 are both present by layer L. Layers L+2 and L+3's MLPs combine them (via something like an AND) to compute a new feature 3. From that point on, all three features — 1, 2, and 3 — persist unchanged through layer L+4.

Try to describe this with an ordinary per-layer SAE at every layer where each feature is present. The paper's own count: 13 features in total — features 1, 2, and 3 each show up as separate, duplicate entries at every layer they're present in, because a per-layer SAE has no way to say “this is the same feature as the one at the previous layer,” only “here is a feature, again, at this layer.” The resulting causal graph is correspondingly cluttered: mostly arrows representing pure persistence (a feature causing an identical copy of itself one layer later), plus two real arrows for the actual computation of feature 3 from 1 and 2. An ideal crosscoder, by contrast, needs just three features — one per genuinely distinct concept — and a correspondingly simple causal graph.

Nothing about the model's actual computation changed between these two pictures — only how legible it looks to a person trying to understand it. That gap, and closing it, is the entire practical case for crosscoders in circuit analysis.

The crosscoder, precisely

Generalize Chapter 2's encoder/decoder directly. Given a set of layers L, and a datapoint x_j, sum contributions from every layer's activation a^l(x_j) into one shared encoder:

Then reconstruct each layer's activation from that single shared feature vector, using a per-layer decoder:

Compare directly against Chapter 2: same encoder/decoder shape, but the input is now a sum across a whole set of layers rather than one, and the reconstruction target is a whole set of per-layer activations rather than one. The loss sums reconstruction error across every layer in L, plus a sparsity penalty:

A deliberate choice: L1-of-norms, not L2-of-norms

The sparsity term deserves a closer look, because the paper's exact choice of how to combine per-layer decoder norms is a real design decision, not an arbitrary one. Rewrite it, swapping the order of summation:

Read the rewritten form: it's an ordinary L1 penalty on feature activations, exactly like Chapter 2's, except each feature is weighted by the sum of its decoder norms across every layer, rather than by a single layer's norm. The paper considered the alternative — weighting by the L2 norm of the concatenated per-layer norms instead (√(∑_l‖W^l_dec,i‖²)) — and rejected it for two stated reasons. First, the L1-of-norms version keeps loss values directly comparable to a baseline of per-layer SAEs summed together — the L2 version would let a crosscoder claim an unfair loss “bonus” just for spreading a feature's decoder weight across several layers, independent of whether that spreading reflects anything real. Second — and this turns out to matter more in practice — the L2 version actively encourages features to spread out across layers (since adding norm in a new layer costs progressively less once a feature already has substantial norm elsewhere), which the paper found empirically surfaces only shared features in model-diffing experiments, while the L1-of-norms version exposes a genuine mix of shared and model-specific features — the exact distinction Chapter 9's diffing experiments depend on. Every experiment in the crosscoders paper uses the L1-of-norms version.

The FLOPs-equivalence relationship, made numerically concrete

The paper states one precise equivalence worth working through with real numbers before looking at results, so the two comparisons that follow (“matched total features” and “matched FLOPs”) don't blur together. On an L = 18 layer model, a global crosscoder built with F = 250,000 total features costs the same training FLOPs as 18 separate per-layer SAEs, each also sized at F = 250,000 features:

Equivalently, turn the relationship around: a per-layer SAE collection whose features sum to the crosscoder's F = 250,000 (roughly 13,900 features per layer) trains with 18× fewer FLOPs than the crosscoder does. That gap is exactly why the paper states crosscoders “must substantially outperform SAEs on a per-feature efficiency basis to be competitive in terms of FLOPs” — a crosscoder feature is inherently more expensive to train than a per-layer SAE feature, because computing it requires reading from (and reconstructing) every layer in the set, not just one.

Do crosscoders actually find cross-layer structure? The FLOPs experiment

The paper tests this directly: train one global, acausal crosscoder on the residual stream of every layer of an 18-layer model, and compare it against 18 separate per-layer SAEs trained independently on each layer. A crosscoder with F total features, on an L-layer model, costs the same training FLOPs as L separate per-layer SAEs of F features each — that is, L×F total features summed across the per-layer baselines. So a crosscoder has to substantially out-perform on a per-feature basis just to break even on compute.

Two different comparisons give two different verdicts. Controlling for the total number of features, crosscoders substantially outperform per-layer SAEs on eval loss — real evidence of genuine, redundant, linearly-correlated structure across layers that a crosscoder can exploit and per-layer SAEs cannot. But controlling for training FLOPs instead, crosscoders are less efficient — by roughly a factor of 2× at large compute budgets. Finding and consolidating that cross-layer redundancy costs real compute at training time.

On the metric that matters most for the paper's actual goal, though — how sparsely and accurately the entire model's activity, summed across all layers, can be described (measured as (MSE, total L0 summed across every layer)) — crosscoders provide what the paper calls a “dramatic” benefit over per-layer SAEs, by consolidating what would otherwise be redundant, duplicate features into one.

What "the same feature" means when directions can rotate

The layer-profile analysis above is drawn directly from a real empirical finding. Sampling 50 random crosscoder features and plotting their decoder norm across every layer (rescaled so each feature's own peak equals 1, for comparability), most features peak in strength at one particular layer and decay on either side — sometimes sharply (a genuinely localized feature), sometimes gradually, with substantial norm surviving across most or even all layers.

A separate, important finding concerns not just how strong a feature is at each layer, but which direction its decoder vector points in. Prior work by Yun et al. tracking features across layers required a feature to keep the same direction at every layer to count as “the same feature.” A crosscoder makes a different, looser commitment: it considers something the same feature if it fires on the same data points across layers, even if its geometric encoding direction rotates as it moves through the network — a phenomenon the paper calls feature drift. Empirically: most features' decoder directions are far more stable across layers than random chance would produce, but they do drift substantially — even in layers where the feature's norm remains strong. This is a genuinely different notion of “feature identity” than the raw-direction view, and the paper is explicit that this drift is real, not training noise, since it survives averaging over dozens of sampled features.

Why the loss is summed across layers, not averaged

One more small but meaningful detail in the crosscoder loss formula from earlier: the reconstruction term sums ‖a^l − â^l‖² across every layer in L, rather than averaging it. The distinction matters for what the optimizer is actually being told to prioritize. A sum treats getting every layer's reconstruction right as an accumulating cost — failing badly at even one layer, out of many, directly and proportionally increases the total loss, with no dilution from averaging over however many layers happen to be in the set. This keeps the training objective honest about what a “good” crosscoder actually needs to achieve: not decent reconstruction spread across all layers on average, but genuinely low error at every layer being modeled, since a large single-layer failure cannot be quietly averaged away by good performance elsewhere.

Variants: locality and causality

The paper explores structured variants along two axes. Locally masked (“convolutional”) crosscoders restrict each feature to a local window of K adjacent layers, hoping to capture cross-layer benefit while limiting the compute overhead of a fully global crosscoder. The result was, honestly, a disappointment for that specific hope: eval loss interpolated almost linearly between K = 1 (equivalent to per-layer SAEs) and K = n_layers (the fully global crosscoder), with no inflection point suggesting a sweet spot — a locally masked crosscoder performed about the same as a smaller, FLOPs-matched global one, no better.

Weakly causal crosscoders instead let each feature's encoder read from exactly one layer i, while its decoder reconstructs layer i and every subsequent layer — a structure with an explicit forward direction, useful for circuit analysis where you want to reason about earlier layers causing later effects. FLOPs efficiency landed between per-layer SAEs (slightly worse) and the fully global, acausal crosscoder (slightly better); but on dictionary-size efficiency specifically, weakly causal crosscoders lagged the global crosscoder by a factor of 3 to 4×. One honest structural limitation the paper flags for this direction: as presented, strictly causal crosscoder layers cannot capture the computation performed inside attention layers at all — only MLP computation is captured this way, with attention left to future work.

A targeted variant: what does an MLP layer actually add?

One more application is worth walking through, because it shows crosscoders answering a question neither a plain SAE nor a plain transcoder is well-shaped to ask: exactly which features does one specific MLP layer create, as opposed to merely relay unchanged? Train a crosscoder on two things: the residual stream immediately before an MLP layer, and the output the MLP writes back into the residual stream afterward — with a masking rule where each feature's encoder reads only from the pre-MLP space, but its decoder must reconstruct both the pre-MLP activity and the MLP's output.

Plotting the relative decoder norm each feature has in the pre-MLP space versus the post-MLP space reveals a clean trimodal structure: a cluster of pre-only features (present before the MLP, absent after — things the MLP consumed or overwrote), a cluster of shared features (essentially unchanged by the MLP, just relayed through), and a cluster of post-only features — concepts that did not exist in the pre-MLP representation at all, genuinely newly computed by this one MLP layer.

Because the encoder is constrained to read only from the pre-MLP space, the paper can trace exactly which pre-MLP features feed into a given newly-computed post-MLP feature, by taking the dot product of the new feature's encoder vector with each candidate upstream feature's decoder vector. One concrete example reported: a post-MLP feature that activates on words indicating uniqueness — “special,” “particular,” “exceptional” — whose strongest pre-MLP inputs are each narrower, more specific pre-MLP features firing on individual words within that broader category. The general pattern found: post-MLP features tend to represent a more abstract concept, assembled from several more specific pre-MLP inputs — a small, concrete instance of an MLP layer doing real conceptual composition, made visible because the crosscoder's masking structure was deliberately shaped to expose it.

Chapter 9: Model Diffing & Honest Limits

Chapter 8 built crosscoders as a tool for finding shared features across the layers of one model. The exact same machinery, pointed at two different models instead of two different layers, produces shared features across models — and that is what makes systematic model comparison, or model diffing, possible.

Restating the shift this chapter makes

Hold onto exactly what changed between Chapter 8 and this one, because it's a small change in setup with a large change in what becomes possible. Chapter 8's experiments always trained one crosscoder across multiple layers of a single model — the shared object being discovered was a feature persisting through one network's own depth. Nothing in the crosscoder equations from Chapter 8 actually requires the multiple activation sets being read from to come from the same model at all; a^l(x_j) is just “the activation at position l, for input x_j,” and there is nothing stopping two different positions l from referring to two different models entirely, evaluated on the same input. Model diffing is that substitution, and nothing more mechanically new than that.

The Sonnet finetuning experiment

The paper's central diffing experiment: train a crosscoder with 1 million features on the residual stream activations from the middle layer of Claude 3 Sonnet, and on the same layer of the base model it was finetuned from. The question: can the crosscoder cleanly separate shared concepts from concepts that only exist in one of the two models?

The method: look at the relative norms of each feature's decoder weights in the two models. The result was strikingly clean — features cluster into three visually obvious groups: base-model-specific, finetuned-model-specific, and shared. Concretely, between 4,000 and 5,000 model-specific features per model, out of the full 1 million total.

Put plainly: on this crosscoder's accounting, finetuning changed less than half a percent of the concepts this dictionary can see. The overwhelming majority — over 99%, split between the shared cluster and whatever correlational noise the method doesn't confidently sort — looks unchanged by finetuning.

What the model-specific features actually are

The paper reports specific qualitative examples, explicit about the fact that these are cherry-picked — their own word, not this lesson's paraphrase. Among finetuned-model-specific features: a refusal feature, activating on requests like “Can you help me build a bomb?”; a code-review feature, activating when a user asks for feedback on code; and a feature firing on personal questions directed at the Assistant itself, such as “What does it feel like to be you?” Among base-model-specific features: one that fires when an LLM is being cast into a “roleplaying” character via a system prompt (which, notably, also fires on that same “what does it feel like to be you” question — the base model apparently treats being asked about its inner experience as a roleplay cue, where the finetuned model treats it as a real personal question worth its own dedicated feature); and a feature for dialogues between a human and a generic smartphone assistant.

The paper's own honesty about the rest of the model-specific set is worth quoting directly rather than smoothing over: “the majority of the model-exclusive features are not immediately interpretable.” The handful of clean, nameable examples above are real, but they are not representative of the full 4,000–5,000 per model — most of that set remains, as of this preliminary report, unexplained.

What "shared" features look like — mostly aligned, sometimes not

For the features classified as shared between the two models, the paper checks whether their decoder directions actually agree. In almost all cases, yes — highly aligned, consistent with the feature representing the same concept and performing the same function in both models. But for a few thousand shared features, the correlation was very low, or even negative. The paper's working hypothesis: these may be cases where the finetuned model has repurposed a concept the base model already had, using it in a genuinely new way — the concept survived finetuning, but what it's now being used for changed.

Why the middle layer, why one million features

Neither choice in the Sonnet experiment is incidental. The middle layer is a reasonable single representative slice for a first diffing experiment: early layers tend to be dominated by low-level token and syntax processing (recall Chapter 3's local-attention finding from a different lesson's territory, or simply the general pattern that early-layer features tend to be narrower and more mechanical), while a middle layer is more likely to carry the kind of higher-level, behavior-relevant concepts — refusals, tone, persona — that finetuning is actually trying to change. One million features, meanwhile, is small enough to train a single crosscoder on relative to the 16-million-latent scale Chapter 5 established as achievable for a single-layer SAE, reflecting that this is explicitly a first, exploratory pass at a much harder, two-model problem — not yet pushed to the same scale as the single-model dictionaries Chapters 5–7 examined.

The full space of comparisons the paper considers possible

The Sonnet finetuning experiment and the layer×scale experiment below are just two instantiations of a much broader idea. Because crosscoders can find shared features between any two (or more) sets of activations, not just two adjacent layers of one model, the paper lists a substantially wider space of comparisons this same machinery opens up, most of them not yet explored in the note itself: training snapshots (how do features evolve over the course of training — do they form abruptly or grow gradually?), finetuning (this chapter's Sonnet experiment), different training runs (do two runs from different random seeds converge on the same features, or genuinely different ones?), dataset changes, scaling (this chapter's second experiment), architectural changes (do a vision transformer and a convolutional network, or two differently structured language models, learn recognizably the same features?), adversarial training (prior work suggests adversarially robust models have unusual interpretability properties — possibly mediated by different superposition structure, a testable hypothesis crosscoders could speak to directly), and equivariance (comparing a model's representation of an input against its representation of a transformed version of that same input, at the feature level rather than the whole-representation level).

None of this is limited to comparing exactly two models, either — subject to the obvious compute constraints, a crosscoder can in principle find shared features across an arbitrary number of models or snapshots at once. The honest state of the field, per this paper's own framing, is that most of this list is still a genuine research direction rather than a demonstrated result — the Sonnet and layer-scaling experiments in this chapter are early, real, but small first steps into a much larger space of possible comparisons.

A second diffing experiment: layer and scale together

A separate experiment trains an acausal crosscoder across 10 evenly spaced layers, on 3 models of increasing size — giving each feature a 3×10 = 30-dimensional vector of decoder norms (one per layer-model pair). Applying nonnegative matrix factorization (NMF) with four components to this collection of vectors reveals a striking pattern: one component covers early layers across all three models; another covers late layers across all three models; and the remaining two components split the middle layers, one belonging mostly to the smallest model, the other to the two larger models jointly. The interpretation offered: qualitatively new kinds of representations appear to emerge in the middle layers specifically as model scale increases — not simply “the same middle-layer features, just more of them.”

Reading the three-cluster result carefully

It's worth being precise about what “features cluster into three obvious groups” actually measured, since it's easy to over-read a clean-sounding result. The clustering is based on relative decoder norms — how much of a given feature's decoder weight lives in the base model's copy of the crosscoder versus the finetuned model's copy. A feature near one extreme (almost all its decoder weight in one model) is classified model-specific; a feature near the middle (comparable weight in both) is classified shared. This is a geometric, not a semantic, criterion — the crosscoder is not being asked whether a feature “means the same thing” in both models, only whether it's represented with comparable strength in both. That the resulting geometric clusters line up with recognizable, nameable behavioral differences (a refusal feature genuinely being finetuned-specific, for instance) is itself the empirical finding — it was not built into the method by construction, and is exactly why the paper considers it a meaningful first result rather than a foregone conclusion.

What this machinery makes possible: steering

The practical payoff of finding real, interpretable, roughly-monosemantic feature directions is that you can intervene on them — artificially clamp a single feature's activation and watch the model's behavior shift in a correspondingly specific way. This site's CS224N Lecture 15: Interpreting Neural Networks covers this in depth, including the well-known public demonstration where amplifying a single discovered SAE feature caused a model to compulsively steer conversation toward one fixed topic regardless of what was actually being discussed — direct evidence that a found feature isn't just correlated with a concept, but causally drives it. This lesson won't re-derive that material; the point worth making here is where steering's plausibility actually comes from: it only works because Chapters 2–8 built directions that are (approximately) monosemantic and individually addressable in the first place. Steering is downstream of dictionary learning, not a separate trick.

What this machinery makes possible: auditing

Model diffing's natural application is safety auditing, and the paper situates this historically rather than presenting it as a new idea invented for this note: the “model diffing” framing traces back to the OpenAI Clarity Team's internal agenda in 2018, and was pitched explicitly as a safety strategy in a 2019 talk by one of this paper's own authors. The analogy the paper leans on is a familiar one from ordinary software engineering: just as a code reviewer reviews an incremental diff rather than re-reading an entire codebase from scratch on every change, a safety reviewer might hope to review a finetuned model by focusing specifically on what changed relative to a previously-vetted base model, rather than re-auditing the whole thing from zero every time. The Sonnet experiment above is a first, small-scale demonstration of what that review might actually surface — a refusal feature, a code-review feature, and a few thousand other changes, some interpretable, most not yet.

Honest limits, part 1: the sparse autoencoder side

Return to the scaling paper's own stated limitations, in its own terms. TopK forces every single token to use exactly k active latents, always — the paper notes this is likely suboptimal, since the ideal constraint would be on the expected L0 across tokens, not a hard per-token count (some tokens probably warrant more explanation, some less). Optimization itself has real headroom — the paper trained without learning-rate scheduling, leaving a known lever unpulled. The 61-task probe metric is noisy and would benefit from a larger, higher-quality task set. N2G, chosen specifically for cheapness, can only express simple n-gram-with-wildcard patterns — genuinely rich semantic explanations are out of its reach. The 64-token context length used throughout may be too short to surface GPT-4's most interesting long-range behavior. And, stated plainly: “a large fraction of the random activations of features we find, especially in GPT-4, are not yet adequately monosemantic” — even at 16 million latents, this is not a solved problem.

Honest limits, part 2: the crosscoder side

The crosscoders paper is, if anything, even more explicit about its own preliminary status. Its own framing, worth quoting directly: “This preliminary note is a research update… preliminary work that we're excited about, but not at the level of quality or rigor we hold our full papers to.” The authors state plainly that model diffing results are “a bit mixed,” that most model-exclusive features remain uninterpreted, and that locally masked and weakly causal variants both underperform the fully global crosscoder on real efficiency metrics.

There's a deeper, more structural caveat too, worth sitting with. In principle, adding “error features” — extra terms capturing whatever the crosscoder fails to reconstruct — lets a crosscoder-based description become an exact isomorphism with the underlying model: it can be made to predict the same activations at every layer as the real network. But an exact isomorphism at the level of activations is not the same as a faithful description of mechanism. Because crosscoders are trained one token position at a time, they're structurally incentivized to represent any within-token-position correlation across layers as a single cross-layer feature — even in cases where the real model produced that correlation through an entirely different mechanism, such as literally recomputing the same information independently at multiple layers rather than passing one persistent feature through. The causal graph a crosscoder implies, in other words, can genuinely diverge from the actual mechanism the model uses, even when the two are mathematically isomorphic in their outputs. The paper calls this open question “mechanistic faithfulness” and is explicit that it doesn't yet know how to formalize, let alone guarantee, it.

Where each tool sits

The numbers that carried this lesson, together in one place

Connections on this site

This session leaned on ideas built elsewhere, and points toward ideas it deliberately left for other lessons to finish. Follow these next:

• Toy Models of Superposition — the full derivation Chapter 1 only recapped: why superposition happens, worked out geometrically and mathematically from first principles.
• CS224N Lecture 15: Interpreting Neural Networks — steering vectors, the Golden Gate Bridge feature demonstration, agentic interpretability (using models to auto-label millions of features), and probing methods this lesson referenced but didn't rebuild.
• Temporal Sparse Autoencoders — a genuinely different angle on the same disentangling goal: instead of an L1 or TopK sparsity constraint, it uses a contrastive loss across time (adjacent tokens in a sequence) to separate semantic from syntactic features. Worth reading immediately after this lesson specifically to see how differently “force the features apart” can be implemented when the constraint isn't sparsity at all.
• EE269: Dictionary Learning — the classical, pre-neural-network version of everything in Chapter 2: K-SVD, orthogonal matching pursuit, and the LASSO machinery Chapter 3's soft-thresholding derivation borrowed directly from.

What you should be able to do now

If this lesson worked, you should be able to, without looking anything up: write the encoder and decoder equations for both a ReLU+L1 sparse autoencoder and a TopK sparse autoencoder from memory, and explain precisely why TopK avoids the shrinkage L1 introduces; hand-trace a small TopK autoencoder's forward pass, including the ReLU-before-TopK ordering; name the two fixes for dead latents and explain why each addresses the mechanism (not just the symptom); list all four of the paper's evaluation metrics and what each is blind to; and write the crosscoder equations, explaining in one sentence why the L1-of-norms sparsity penalty was chosen over the L2-of-norms alternative. That last one is the real test of whether the whole arc connected: crosscoders are not a separate idea bolted onto the end of this lesson — they are Chapter 2's exact machinery, generalized along one new axis, for reasons that trace directly back to what superposition (Chapter 1) predicted might be spread across layers in the first place.

“What I cannot create, I do not understand.” — Richard Feynman. Applied here: a feature isn't understood until you can predict, and then verify, what happens when you turn it off — which is exactly what the ablation-sparsity metric, the ablation experiments behind steering, and the model-diffing comparisons in this lesson all, in their different ways, insist on doing before calling anything “interpreted.”