Toy Models of Superposition

Chapter 1: The Toy Model, Symbol by Symbol

The model has three moving parts: a data generator you control completely, a bottleneck that is too small on purpose, and a loss that says which mistakes hurt. Every one of them is three lines of code. The interesting behaviour is entirely emergent.

Part one: the features are the data

In a real network, a "feature" is a hypothesis — some property you believe is represented, which you then have to go and find. In the toy model we invert this. The features are the input. We declare that the world contains exactly n independent properties, and we hand the model a vector listing their intensities:

Two properties of this vector are chosen by us, and they are the only two knobs that matter in the entire paper.

Sparsity S. Each coordinate is zero with probability S, and otherwise drawn uniformly from [0, 1], independently of the others:

We will write d = 1 − S for the density — the probability a given feature is on — because almost every formula in this lesson is cleaner in d. Sparsity S = 0.9 means density d = 0.1: one feature in ten is active in a given sample. S = 0 means d = 1: every feature always on.

Importance I_i. Not all features matter equally. In a language model, "is this English" is worth more than "does this mention the specific brand of stapler". We encode this as a per-feature weight in the loss. The paper's default is geometric decay:

With r = 0.7 and n = 5 the importances are 1, 0.7, 0.49, 0.343, 0.2401. With r = 1 they are all equal, which the paper calls the uniform case and which produces the cleanest geometry.

Part two: the bottleneck

The model has exactly one job: reproduce its input. That sounds pointless until you notice the shape of the hidden layer.

Read that in shapes. Input x has n entries. The hidden vector h has m entries with m < n, so information must be lost — this is the whole experiment. Then the same matrix, transposed, maps back up to n, a bias is added, and a ReLU clips negatives.

Every piece of that is a decision. Let us justify each one, because a toy model is only as good as the honesty of its simplifications.

The paper calls this the ReLU output model, and it also studies a linear variant with no output ReLU. The comparison is the cleanest experiment in the whole paper: the linear model never superposes, at any sparsity. Nonlinearity is not decoration here, it is the enabling condition.

Part three: the loss

An importance-weighted mean squared error. Nothing more. But look at what it implies: because I_i multiplies the squared error of feature i, and because a feature the model completely ignores contributes I_i E[x_i²], the loss has a natural unit — the cost of giving up.

We will need E[x_i²] constantly, so compute it now. The feature is 0 with probability S and uniform on [0,1] with probability d. For U ~ Uniform[0,1], E[U²] = ∫₀¹ u² du = 1/3. So

The forward pass, with real numbers

Abstractions are cheap. Let us push actual numbers through the smallest interesting instance: n = 3 features into m = 2 dimensions.

Suppose training has produced these three columns for W (each column is one feature's direction; each has unit length):

Features 1 and 2 got clean axes. Feature 3 was squeezed in between them, 53.13° from the first axis. Check its length: √(0.36 + 0.64) = √1 = 1. Good.

Case A: only feature 3 is active, at strength x₃ = 0.7, with b = 0 for now.

Now read all three features back out with W^Th, which is just the dot product of h with each column:

Feature 3 comes back perfectly. But features 1 and 2, which are not present at all, report 0.42 and 0.56. Those are hallucinations — pure interference — and with importance weights of 1 and 0.7 they cost

Now switch on the bias. Set b = (−0.45, −0.60, 0) and redo the last step:

Loss: zero. The interference did not go away — the geometry is unchanged — it was filtered. This is the central mechanism of the entire paper, and you have just computed it by hand: a negative bias plus a ReLU is a noise gate.

Case B: features 1 and 3 both active, x = (0.5, 0, 0.7), still with b = (−0.45, −0.60, 0).

Weighted loss: 1 × 0.03² + 0.49 × 0.30² = 0.0009 + 0.0441 = 0.045. Compare with the 0.000 we got when only one feature fired.

The whole model, in tensor shapes

If you were to implement this — and you should, it is about twenty lines — here is exactly what flows where for a batch of B samples.

pytorch# n features, m hidden dims, B batch. n > m is the entire point.
W = nn.Parameter(torch.randn(m, n) / math.sqrt(n))   # (m, n)
b = nn.Parameter(torch.zeros(n))                     # (n,)

# ---- data generator: this IS the ground truth ----
u    = torch.rand(B, n)                    # (B, n)  magnitudes in [0,1]
mask = (torch.rand(B, n) > S).float()    # (B, n)  1 with prob d = 1-S
x    = u * mask                            # (B, n)  sparse, non-negative

# ---- forward ----
h  = x @ W.T                               # (B, n) @ (n, m) -> (B, m)   THE BOTTLENECK
xp = torch.relu(h @ W + b)                # (B, m) @ (m, n) -> (B, n)

# ---- importance-weighted MSE ----
I    = r ** torch.arange(n)               # (n,)   e.g. 1, .7, .49, ...
loss = (I * (x - xp) ** 2).sum(-1).mean()  # scalar

Two lines of that code deserve a second look.

h = x @ W.T is where the information is destroyed, and it is a sum: h = ∑_j x_j W_j. Every active feature adds its own direction, scaled by its intensity. The hidden vector is a superposition in the literal, physics sense of the word — a linear combination of contributions that have been added into the same medium.

xp = relu(h @ W + b) is where the model tries to un-add them. Written per feature: x′_i = ReLU( W_i · h + b_i ) = ReLU( ∑_j (W_i · W_j) x_j + b_i ). The whole model, therefore, is governed by one n × n matrix:

This is the Gram matrix of the feature directions. Its diagonal G_ii = ‖W_i‖² is how strongly feature i is represented at all. Its off-diagonal G_ij is exactly the interference of j onto i. The ideal — and the impossible — is G = I_n, the identity. But G = W^TW with W of shape m × n has rank at most m, and the identity has rank n. That rank gap, in one line, is the pigeonhole argument restated in matrix form — and everything the model does is an approximation strategy for a matrix it cannot reach.

Training it: the gradient, derived

The simulations in this lesson run real gradient descent, so it is worth deriving the gradient once. It is short, and its structure explains how the geometry organises itself.

Write the forward pass per feature: p_i = W_i · h + b_i, then x′_i = ReLU(p_i). Define

The indicator is the ReLU's derivative: a clipped output receives no gradient at all. Then ∂L/∂b_i = g_i, which is the easy one. The interesting one is W_i, because it appears twice in the computation graph — once writing feature i into h, once reading it back out. Both paths contribute:

The read path says: if feature i was read out too high, move its column away from the current hidden vector. The write path says something much more interesting: if feature i was active, move its column against the error-weighted sum of every other feature's direction. Feature i is being pushed away from exactly those features it is currently damaging, in proportion to the damage.

One gradient step by hand. Take the chapter-1 setup, b = 0, and the sample x = (0, 0, 0.7) from Case A, with I = (1, 0.7, 0.49). We computed h = (0.42, 0.56) and the readouts (0.42, 0.56, 0.70). Errors: e = (0.42 − 0, 0.56 − 0, 0.70 − 0.70) = (0.42, 0.56, 0). All three pre-activations are positive, so no gradient is clipped, and

Now the write-path term for feature 3, which is the only active feature:

Gradient descent moves W₃ in the negative of that: from (0.6, 0.8) toward (0.6, 0.8) − η(0.588, 0.549). It is being pushed down and to the left — away from both axis features, into the third quadrant. Run that for a while and the three columns end up 120° apart. The triangle is not designed; it is the fixed point of this push.

And notice what the bias does to this story. Once b₁ and b₂ are negative enough that p₁ and p₂ go below zero on this sample, the indicators fire zero, g₁ = g₂ = 0, and the repulsion switches off entirely. The model stops pushing the features apart because it is no longer being hurt. That is the whole mechanism, visible in the gradient.

What the model can and cannot express

Since everything routes through G = W^TW, it is worth being precise about which matrices are reachable.

G is symmetric, positive semidefinite, and of rank at most m. Conversely, every symmetric PSD matrix of rank ≤ m is W^TW for some m × n matrix W (take the eigendecomposition and keep the non-zero part). So the model's expressible set is exactly:

The target — perfect reconstruction — is G = I_n, which has rank n. Since m < n, it is unreachable, and the model must pick the best available compromise. Two families of compromise exist, and chapters 2 and 3 are exactly the story of choosing between them:

Nothing in linear algebra picks between them. What picks between them is the distribution of the inputs, through the ReLU, and that is a fact about the data rather than about the matrices. Which is why the same architecture produces PCA in one regime and pentagon packing in another.

What "superposition" will mean, precisely

We can now define the word without hand-waving. Given a trained W:

• A feature is represented if ‖W_i‖ is meaningfully non-zero.
• Two represented features are in superposition if W_i · W_j ≠ 0 — they are not orthogonal, so they interfere.
• The model as a whole is in superposition if the number of represented features exceeds m. This is forced: more than m non-zero vectors in m dimensions cannot be mutually orthogonal.

Chapter 4 replaces this three-way qualitative split with a single continuous number per feature. But the qualitative version is enough to run the first experiment, which is the subject of chapter 2: what happens when nothing is sparse at all?

Five misreadings to head off now

This model is small enough that people form opinions about it quickly, and a few of those opinions are wrong in ways that matter later.

One more that is subtler and worth stating on its own. It is tempting to read the toy model as saying "the network is compressing its inputs". It is not: the inputs to a real layer are not the features. The features are properties the network has computed, elsewhere, and this model is about the storage medium they have to fit through — a residual stream, an MLP activation, an embedding vector. The right mental picture is a bus with limited width carrying signals produced upstream, not a compressor applied to raw data.

Chapter 2: The Dense Regime, or PCA in Disguise

Set S = 0. Every feature is on, every sample, with an intensity uniform on [0,1]. This is the world our intuitions were built in — the world where the pigeonhole argument is not just valid but binding — and it is the right place to start, because it establishes the baseline against which everything strange later will be measured.

The question: given n = 5 features, m = 2 dimensions, and importances 1, 0.7, 0.49, 0.343, 0.2401, what is the best the model can do?

The candidate that ought to win

Here is the obvious strategy. Pick a subset A of at most m features. Give each one its own orthogonal unit direction. Set the columns of every other feature to zero.

The Gram matrix G = W^TW is then diag(1, 1, 0, 0, 0). Features 1 and 2 are reconstructed exactly: x′₁ = ReLU(x₁) = x₁, and no other feature can touch them, because their columns are orthogonal to everything. Features 3, 4, 5 output whatever the bias says.

What should the bias say for a feature the model has given up on? Not zero. If W_i = 0, then x′_i = ReLU(b_i) is a constant, and the constant that minimises E[(x_i − c)²] is the mean:

So an abandoned feature costs its variance, not its second moment. For a dense feature uniform on [0,1]: Var = 1/3 − (1/2)² = 1/3 − 1/4 = 1/12.

That "independent, therefore diagonal" step is the one to remember. It is exactly what will fail to matter later: superposition is not driven by correlations between features. It is driven by their sparsity, which is a property of each feature's marginal distribution, not of any relationship between them.

Making the optimality argument properly

"Weighted PCA, so top-m" is a slogan. Here is the calculation behind it, because it takes ten lines and it makes the later comparisons trustworthy.

Drop the ReLU (in this regime it never activates on the solutions we care about) and let A = W^TW, a symmetric positive semidefinite matrix of rank at most m. Let μ be the vector of feature means and let the bias take its optimal value b = (I − A)μ, which centres the problem. Writing Λ = diag(I₁…I_n) and using that the centred covariance is σ²I_n (independent features, identical marginals):

Now restrict to the natural family: A = P_S, the orthogonal projection onto the coordinate subspace spanned by a set S of at most m features. Projections satisfy A² = A, so the last two terms collapse:

Which is minimised by taking S to be the m largest importances. Clean, and it reproduces the number we guessed. (That the coordinate projection also beats every non-projection rank-m matrix takes a little more work; the paper settles it empirically, and so does the simulation below.)

Tabulating for n = 5, r = 0.7, d = 1 so σ² = 1/12:

Each extra dimension buys exactly the importance of the next feature, divided by twelve. Predictable, monotone, boring — and worth writing down precisely so that chapter 3's departure from it is unmistakable.

The challenger, and why it loses

Now try the greedy alternative: cram three features into the plane. Put W₁, W₂, W₃ at 120° from each other, all unit length — an equilateral triangle. Their pairwise dot products are all cos 120° = −1/2, so the Gram matrix has −0.5 everywhere off the diagonal of its top-left 3×3 block.

Take b = 0 for this candidate and read out feature 1:

The error is x₁ − x′₁. If x₁ > z the ReLU is inactive and the error is exactly z. If x₁ ≤ z the output is clipped to zero and the error is x₁. Both cases at once:

That is a genuinely pretty identity, and it lets us compute the expected squared error exactly. First, for any fixed z ∈ [0,1] and x ~ Uniform[0,1]:

Now average over z = (x₂ + x₃)/2, the average of two independent uniforms. Its mean is 1/2 and its variance is (1/12 + 1/12)/4 = 1/24, so

And for the cube, expand E[(x₂+x₃)³] = E[x³] + 3E[x²]E[x] + 3E[x]E[x²] + E[x³] = 1/4 + 1/2 + 1/2 + 1/4 = 3/2, so E[z³] = (3/2)/8 = 3/16 = 0.1875. Therefore

Exactly one sixth. Each of the three superposed features pays 1/6, and features 4 and 5 are abandoned at 1/12 each (with the mean bias):

Why the ReLU cannot help when nothing is sparse

Look again at error = min(x₁, z). The ReLU helps precisely when x₁ ≤ z, because then it converts an error of z into an error of x₁. In chapter 1 we saw the dream case: x₁ = 0, so the error becomes min(0, z) = 0 and interference vanishes completely. In the dense regime x₁ = 0 never happens. The gate is jammed open.

This also explains why a negative bias does not rescue the triangle. A bias of −β changes the readout to ReLU(x₁ − z − β), which subtracts β from every true activation as well. In a world where the true activation is present every time, that is a pure loss.

Three experiments worth running before moving on.

One. At S = 0, slide m from 1 to 4 and watch the identity block grow. The model always keeps the most important features and always keeps them orthogonal. Count the survivors: it is exactly m, every time.

Two. At S = 0, push the importance decay r toward 1.0 so all five features matter equally. The model still keeps exactly two — but now which two is arbitrary, and if you nudge the slider you may see it switch. Ties are broken by initialisation noise, not by the objective.

Three. Now cheat and look ahead: push sparsity to 0.9 and watch the Gram matrix light up off the diagonal. That is chapter 3, and it should look, at first glance, like a bug.

The linear model never superposes — at any sparsity

The claim in the box above deserves its own argument, because it is the paper's cleanest structural result and it is easy to prove with what we already have.

Remove the output ReLU. The model is now x′ = Ax + b with A = W^TW, and the loss is a quadratic form in A that depends on the data only through its mean and covariance. Look at what sparsity does to those two objects:

The covariance is still a multiple of the identity. Sparsity changed the scalar σ² and nothing else. So the minimiser of the loss is the same matrix A at every sparsity level, and only the loss value shrinks. The linear model does weighted PCA at S = 0, at S = 0.9, and at S = 0.999, and it represents exactly m features every time.

A summary of the dense world, so we can watch it break

Note the second-to-last row, because it is the one this whole research programme is about. In the dense regime, interpretability is trivial: read off the hidden units and each one is a feature. If real networks lived here, mechanistic interpretability would be a solved problem and this paper would not exist.

Chapter 3: Sparsity Flips the Phase

Everything in chapter 2 was correct and everything in chapter 2 was about a world that does not exist. Real features are rare. "Mentions a legal obligation" is off in the overwhelming majority of sentences. So let us turn the one knob we have not touched and redo the arithmetic.

Nothing about the model changes. Same W, same tied decoder, same ReLU, same loss. The only edit is to the data generator: each feature is now zero with probability S. And the solution the model converges to changes completely.

Redo the triangle, this time with holes in the data

Recall the exact error identity from chapter 2, which did not assume anything about density:

Now walk the cases, and watch how many of them are free.

Case 1: feature 1 is off (probability S = 1 − d). Then x₁ = 0, so min(0, z) = 0 — the error is exactly zero no matter what features 2 and 3 are doing. The interference is entirely absorbed by the ReLU: it pushes the readout negative, and negative is clipped. This case costs nothing and it is the majority of samples.

Case 2: feature 1 is on, features 2 and 3 both off (probability d(1−d)²). Then z = 0 and the error is min(x₁, 0) = 0. Perfect reconstruction. Also free.

Case 3: feature 1 on, exactly one of 2 or 3 on (probability 2d²(1−d)). Now z = U/2 with U ~ Uniform[0,1], so z is uniform on [0, 0.5]. Using the same integral as before, ∫ min(x,z)² dx = z² − (2/3)z³, and averaging with density 2 on [0, 0.5]:

Case 4: all three on (probability d³). This is the chapter-2 calculation: expected squared error 1/6.

Assemble. Multiplying each case by its probability:

The crossover, solved exactly

Put the two candidate solutions side by side at general density d, with n = 5, m = 2, I = (1, 0.7, 0.49, 0.343, 0.2401).

Subtract. The features 4 and 5 term is identical in both and cancels, leaving a startlingly clean statement:

In words: what you gain is feature 3 no longer being thrown away. What you pay is interference, charged to all three features that now share the plane. Superposition wins when the green term exceeds the red one.

Substitute the numbers. Using (1−d)/8 + d/6 = 0.125 + 0.0416667d:

Divide both sides by d (the d = 0 root is trivial) and expand:

Quadratic formula. The discriminant is 0.39625² + 4(0.09125)(0.163333) = 0.157014 + 0.059617 = 0.216631, whose square root is 0.465436. So

Below density 0.379 — that is, above 62% sparsity — the triangle beats the orthogonal pair. Let us verify at two points.

The full ladder — configurations compete, not features

We compared two solutions and found a boundary. But two is not all of them, and the extra candidates hold a genuine surprise. In two dimensions with five features there is a handful of natural configurations, and it is worth pricing every one — including two that use a trick we have not yet met.

The trick both of the new candidates use is the antipodal pair: two features on the same axis pointing opposite ways, c_ij = −1. Chapter 5 derives its cost from scratch; the result is that such a pair errs only when both members fire, and then by

C3b, an axis plus a pair. Give the most important feature its own private axis and hang an antipodal pair off the other. W₁ = (1,0); W₂ = (0,1); W₃ = (0,−1). Feature 1 is orthogonal to everything and reconstructs exactly; features 2 and 3 pay the pair cost; features 4 and 5 are abandoned.

C4, the square. Two antipodal pairs on perpendicular axes: W₁ = (1,0), W₃ = (−1,0), W₂ = (0,1), W₄ = (0,−1). Four features represented; each pays the pair cost; only feature 5 is abandoned.

Two boundaries, each a one-line solve. C3b overtakes the orthogonal pair when

and the square overtakes C3b when

Now put every candidate on one table and read down the columns.

Why are the antipodal arrangements so strong? Because every off-diagonal Gram entry in them is 0 or −1 — there is no positive interference anywhere. A zero costs nothing because there is no crosstalk at all; a −1 costs nothing unless both partners fire, and then the ReLU still salvages the smaller of the two. Positive dot products, which the triangle and the pentagon are forced into, are the expensive kind, and chapter 6 puts a number on exactly how expensive.

Where the pentagon comes from

So when does a five-sided figure ever appear? When importance stops decaying. Set r = 1 so all five features matter equally, and redo the comparison at small d.

In a pentagon each feature has two neighbours at cos 72° = +0.309 and two at cos 144° = −0.809. The positive ones are the expensive ones: when a +0.309 neighbour fires alone at strength U, feature i's readout becomes 0.309U whether or not feature i is present, so the error is charged with probability d — linear, not quadratic:

Five features means a total positive-interference bill of 5 × 0.06366d = 0.3183d. The benefit of admitting the fifth feature rather than abandoning it is I₅d/3 = 0.3333d at uniform importance. Benefit beats cost, narrowly — and a negative bias (chapter 6) widens the margin. So the pentagon appears.

Now put the decaying importances back: the fifth feature is worth I₅ = 0.2401, so the benefit falls to 0.0800d while the bill stays near 0.176d (the sum of importances is smaller, but so is the payoff). The pentagon loses, and the square stands.

Why it is a phase change and not a gradual blend

You might expect feature 3 to "fade in" — a column that grows continuously from zero as sparsity rises. That is not what happens, and the reason is structural.

Consider a tiny column W₃ = δu for a small δ and unit direction u. The benefit of representing feature 3 with such a stub is proportional to δ² at best (the reconstruction is δ²x₃, so the residual barely moves), while the cost — the interference this stub sprays onto features 1 and 2 — is also order δ². The two are the same order, and near W₃ = 0 the loss surface is nearly flat. So the model sits at zero until the balance tips, and then falls into a qualitatively new configuration.

Empirically this shows up in three ways the paper documents carefully:

• Discrete jumps in the sparsity sweep. The number of represented features is a staircase in S, not a ramp.
• Sticky plateaus during a single training run. Loss falls, sits flat for thousands of steps, then falls again — each drop is a reorganisation of the geometry, not a refinement of it. The paper likens these to energy levels.
• Hysteresis and seed dependence near boundaries. Close to a phase boundary, two runs from different initialisations can land in different geometries with nearly identical loss.

What to look for, in order.

At S = 0, two arrows at right angles and three at the origin. Exactly chapter 2.

At S ≈ 0.60 (density 0.398, inside the middle band of the table) a third arrow appears — and look at the dimensionality panel rather than the picture. Feature 1 reads D = 1.00, features 2 and 3 read about 0.50 each. That is C3b: one private axis plus one antipodal pair, exactly as priced, and the trained loss should land near 0.086 against the predicted 0.0857.

At S = 0.90 a fourth arrow joins and the four settle into a cross — two antipodal pairs, D = 0.50 each, ∑D_i = 2.00, loss near the predicted 0.0116. C4, as advertised.

Push r to 1.0 at S = 0.99 and the fifth arrow joins, spreading the five into a pentagon with D = 0.40 each. Pull r down toward 0.4 and the tail features stop being worth their bill, so the model keeps fewer of them and gives the survivors longer columns. Importance buys privacy; sparsity buys seats.

Press "new random seed" near a boundary and watch the geometry sometimes change while the loss barely moves. Those are two nearly-degenerate minima, and they are the reason the paper says these solutions have the flavour of physical phases rather than of unique optima.

Reproduce it yourself in forty lines

Nothing in this chapter should be taken on trust, and the whole experiment fits on one screen. This is the complete program — data generator, model, training loop, and the two measurements that matter.

pythonimport torch, math

n, m, S, r = 5, 2, 0.9, 0.7          # features, dims, sparsity, importance decay
I = r ** torch.arange(n)                # (n,)  1, .7, .49, .343, .2401
W = torch.randn(m, n, requires_grad=True)
b = torch.zeros(n, requires_grad=True)
opt = torch.optim.Adam([W, b], lr=0.05)

def batch(B=1024):
    u = torch.rand(B, n)
    return u * (torch.rand(B, n) > S).float()   # sparse, non-negative

for step in range(4000):
    x  = batch()
    h  = x @ W.T                          # (B, m)   THE BOTTLENECK
    xp = torch.relu(h @ W + b)            # (B, n)
    loss = (I * (x - xp) ** 2).sum(-1).mean()
    opt.zero_grad(); loss.backward(); opt.step()

# ---- measurement 1: which features survived, and how long are they? ----
norms = W.norm(dim=0)                     # (n,)  one per feature
print("column norms:", norms.detach().round(decimals=2).tolist())

# ---- measurement 2: the Gram matrix — diagonal is signal, off is crosstalk ----
G = (W.T @ W).detach()
print("Gram:\n", G.round(decimals=2))

# ---- measurement 3: feature dimensionality D_i = |W_i|^2 / sum_j (What_i . W_j)^2 ----
Wn = W / norms.clamp(min=1e-9)               # unit columns
D  = norms**2 / ((Wn.T @ W) ** 2).sum(1)
print("dimensionality:", D.detach().round(decimals=2).tolist(), "total", float(D.sum()))

Run it at S = 0 and you should see column norms like [1.00, 1.00, 0.00, 0.00, 0.00], a Gram matrix that is a 2×2 identity block, and dimensionalities [1, 1, 0, 0, 0] summing to 2. Run it at S = 0.9 and the norms come back with four non-zero entries and the dimensionalities cluster near 0.5, still summing to about 2.

The staircase, predicted

Using the ladder above, we can write down in advance what the sweep should look like for n = 5, m = 2, r = 0.7 — a falsifiable prediction rather than a description.

Every row keeps ∑D_i = 2. Capacity is conserved; only its division changes. Now open the simulation and try to break the table — the four rows correspond to S = 0, S ≈ 0.60, S = 0.90, and S = 0.99 with r = 1.

Why negative dot products, every time?

Look at the arrows once superposition sets in: they always spread out to make the pairwise angles obtuse. The triangle sits at 120°, not at 60°. There is a specific reason, and it comes back to the ReLU.

If W_i · W_j < 0, then when feature j fires alone the readout for feature i is pushed below zero, and ReLU clips it to exactly the right answer. A negative dot product produces interference that is free to fix.

If W_i · W_j > 0, feature j firing alone pushes feature i's readout up, producing a false positive that the ReLU cannot remove — only a negative bias can, and the bias damages true activations as collateral. Positive interference is expensive; negative interference is nearly free.

This asymmetry is the single most consequential detail in the toy model, and it is a direct consequence of choosing non-negative features. It is what makes the shapes in the next chapter be polytopes with negative pairwise dot products rather than an arbitrary near-orthogonal spray.

Chapter 4: The Geometry Zoo

Once the model starts superposing, "how many features are represented" stops being a good question. Feature 1 might own a whole dimension, features 2 and 3 might split one between them, and features 4 through 8 might be crammed into the leftovers. We need a way to say how much of a dimension each feature got, as a number.

Feature dimensionality, defined and sanity-checked

The paper's definition. Write Ŵ_i = W_i/‖W_i‖ for the unit vector along feature i's direction. Then

Unpack it slowly, because the shape of the expression is the whole idea. The numerator is how much magnitude feature i has. The denominator asks: along feature i's own direction, how much total feature-mass is piled up? The sum includes j = i, which contributes ‖W_i‖², plus a term for every other feature that leans into i's direction. So D_i is the fraction of that direction that belongs to i.

Check the two extremes.

Orthonormal. W_i is a unit vector perpendicular to everything. Numerator 1. Denominator 1 + 0 + 0 + … = 1. So D_i = 1: a whole dimension, privately owned.

Not represented. W_i = 0. Numerator 0, so D_i = 0. (The unit vector is undefined; take the limit, or just declare it zero.)

Antipodal pair. One dimension, two features: W₁ = +1, W₂ = −1. Then Ŵ₁ = +1, numerator 1, denominator 1² + (−1)² = 2, so D₁ = 1/2. Half a dimension each. That is the answer we wanted and it fell out of the formula.

The general rule, derived

The arrangements the model converges to are almost always tight frames: sets of k equal-length vectors in m dimensions that are as evenly spread as vectors can be. Formally, a set of unit vectors {w₁…w_k} in R^m is a tight frame when

which says the vectors have no preferred direction: summed as outer products, they look isotropic. Now compute the dimensionality of any member. Take any unit ŵ_i:

Every named plateau in the paper is now a one-line calculation.

Verify the pentagon by hand, because it is the one case where the frame is not obviously "even". Five unit vectors at 72° apart. From feature 1, the others sit at 72°, 144°, 216° and 288°. Cosines: cos 72° = 0.309017 (twice, by symmetry) and cos 144° = −0.809017 (twice).

The two irrational cosines conspired to give exactly 5/2. That is not luck — it is the tight-frame identity, and the fact that it holds is a good check that the regular pentagon really is one.

Why these shapes and not others

Two separate forces pick the geometry, and separating them explains everything in the table.

Force one: minimise total squared interference. The loss is quadratic in the errors, and the errors are built from the crosstalk terms W_i · W_j. So to first order, the model wants to minimise

This quantity has a name in frame theory: the frame potential. A classical theorem (Benedetto and Fickus, 2003) says its minimum over unit vectors is k²/m, and the minimisers are precisely the tight frames. So the very structure the paper observes is the known solution to the "spread k vectors as evenly as possible" problem — the same family of problems as the Thomson problem for electrons on a sphere.

Force two: prefer negative interference. Chapter 3 showed why: negative crosstalk is clipped away for free by the ReLU, positive crosstalk becomes a false positive. So among the tight frames, the model prefers ones whose pairwise dot products are non-positive.

Now here is the fact that makes the table make sense:

That single fact explains the sticky plateaus at 1/2, 2/3 and 3/4: they are m/(m+1) for m = 1, 2, 3. And it explains the model's favourite trick when it has more features than a simplex can hold: split the space into orthogonal blocks and put a simplex in each one. Two orthogonal dimensions can hold two antipodal pairs (4 features, D = 1/2 each, no positive interference anywhere). A 3-dimensional space can hold a triangle in a plane plus a dedicated axis (4 features, D = 2/3, 2/3, 2/3, 1).

Only when sparsity is extreme — when even positive interference is cheap because co-activation is so rare — does the model start accepting shapes with positive dot products: pentagons, square antiprisms, and beyond. The zoo is ordered by how much positive interference the sparsity budget can absorb.

Flip between the two modes at k = 4. The even packing is a square: four vectors 90° apart, which is the same thing as two antipodal pairs on perpendicular axes. Every off-diagonal entry is either −1 or 0 — no positive interference at all, and D = 1/2. Now go to k = 5 and watch red appear for the first time: in the plane, five features cannot avoid leaning into each other.

Working the tetrahedron, and a table to check yourself against

The plane is easy to draw and therefore easy to over-trust. Do one three-dimensional case by hand so the general rule is not just a formula you accepted.

Four unit vectors in R³ pointing at the vertices of a regular tetrahedron from its centre have every pairwise cosine equal to −1/3. (Quick check that this is forced: if ∑w_j = 0 then 0 = ‖∑w_j‖² = 4 + 12c, giving c = −1/3, where 12 is the number of ordered distinct pairs.) So

That trick — expanding ‖∑w_j‖² = 0 — gives the simplex cosine in any dimension: m+1 unit vectors summing to zero force (m+1) + m(m+1)c = 0, hence c = −1/m. Antipodal pair: −1/1 = −1. Triangle: −1/2. Tetrahedron: −1/3. All three rows of the zoo, from one line of algebra.

Read the last column together with the fifth. Every all-negative configuration sits at D ≥ 1/2, and the moment a shape needs positive dot products its dimensionality drops below a half. That is not a coincidence: k > 2m features cannot be tiled into antipodal pairs, and antipodal pairs are the last all-negative option. D = 1/2 is the frontier of free superposition.

Sum of squares, not maximum — and why that matters

There is a competing notion of "spread out as evenly as possible", and the difference between the two is instructive.

The Welch bound states that for k unit vectors in R^m,

For the triangle (k=3, m=2) this gives √(1/4) = 0.5, which the triangle achieves exactly — it is optimal for both criteria at once. For the pentagon (k=5, m=2) the bound is √(3/8) = 0.6124, and the pentagon's worst cosine is 0.809. So the pentagon is not the arrangement that minimises the largest overlap.

It is nonetheless what the model finds, because the loss is a sum of squared errors, and squared error is driven by the total crosstalk energy ∑c_ij², not by the single worst pair. Minimising the sum is the frame-potential problem, whose answer is the tight frame. Minimising the max is the Grassmannian packing problem, whose answer is different. The geometry a model chooses is a fingerprint of its loss function.

What the learning dynamics look like

Because the geometries are discrete, training does not glide toward them. The paper reports loss curves with long flat stretches punctuated by sudden drops, and shows that each drop corresponds to a change of shape — a feature switching on, a pair collapsing into a digon, a square reorganising into a pentagon. The authors describe these as energy levels, and the analogy is apt: the model spends most of its time in one configuration and occasionally tunnels into a lower one.

Two practical consequences follow, and both should feel familiar if you have ever trained anything.

Plateaus are not always a bug. A flat loss curve can mean the optimiser is stuck at a saddle between two geometries, and pushing through it — more steps, a learning-rate bump, a different seed — produces a discrete improvement rather than a gradual one.

Seeds matter more near boundaries. When two configurations have nearly equal loss, which one you land in is decided by initialisation. This is a mild but real reproducibility hazard for interpretability work: the "features" you find may be a property of the run, not just of the task.

Correlation changes the shape, but is not the cause

The paper also studies features that are not independent, and the result is a nice consistency check on the whole mechanism.

Chapter 5: The Phase Diagram

Chapter 3 found one boundary by comparing two hand-built solutions. This chapter does the whole map. To make that possible we shrink the model until every candidate solution can be written down and every expected loss integrated in closed form.

Two features. One dimension. n = 2, m = 1. That is the smallest system in which the question "should these two share?" can even be asked.

Two parameters in W, two in b. And because a one-dimensional space has exactly two directions, there are only three qualitatively distinct things the model can do.

The three candidate solutions, priced

Solution A — keep feature 1, drop feature 2. Set w = (1, 0) and b = (0, d/2).

Then h = x₁, so x′₁ = ReLU(x₁) = x₁: exact, always. And x′₂ = ReLU(d/2), the constant mean of feature 2, which by chapter 2's argument costs its variance:

Solution B — keep feature 2, drop feature 1. By symmetry,

Solution C — superposition. Put them at opposite ends of the single dimension: w = (1, −1), b = (0, 0). Now h = x₁ − x₂, and

Check the easy cases first, because they are the whole reason this works. If only feature 1 fires: x′₁ = ReLU(x₁) = x₁ exactly, and x′₂ = ReLU(−x₁) = 0, also exactly. Symmetrically if only feature 2 fires. If neither fires, both outputs are 0. Three of the four cases are perfect.

Only when both fire does anything go wrong, and the error is our old friend:

For two independent uniforms on [0,1], the minimum has density 2(1−t), so

Both features fire with probability d², and both pay the same expected error, so

Solving for the boundaries

Superposition beats dropping feature 2 when L_C < L_A:

Divide through by d (positive) and gather the d terms on the left:

Multiply everything by 12 to clear denominators:

And by the mirror argument, superposition beats dropping feature 1 when d < 4I₁/(5I₁ + 2I₂). Superposition is optimal exactly when both hold.

Worked check at equal importance. Set I₁ = I₂ = 1. Both conditions become d < 4/7 = 0.5714. Verify by evaluating all three losses at exactly d = 4/7:

Identical to six decimal places, as an exact boundary should be. In sparsity terms: with two equally important features and one dimension to share, superposition is optimal above 42.9% sparsity and dropping one is optimal below it.

A second check, with lopsided importance. I₁ = 1, I₂ = 0.1. Then d < 4(0.1)/(2 + 0.5) = 0.4/2.5 = 0.16. A feature ten times less important must wait until 84% sparsity before it is worth letting into the dimension at all. Importance buys you a seat; sparsity decides how many seats there are.

Why there is no fourth region

In one dimension, if both features are represented, w₁ and w₂ are two non-zero scalars, so they either have opposite signs or the same sign. Opposite signs is solution C. Same sign is strictly worse: when feature 2 fires alone, feature 1's readout is ReLU(w₁w₂x₂) > 0 — a false positive on a sample where the antipodal arrangement was exactly right. So there is nothing else to consider, and the phase diagram genuinely has three regions and no more.

In higher dimensions this exhaustiveness disappears — that is why chapter 4 needed a zoo — but the qualitative picture survives: a dense region with orthogonal representation of the top features, a sparse region with superposition, and a boundary that moves with importance.

Click around the diagram and check three things.

Deep in the blue region (dense, lopsided importance), the trained weights come back as roughly (1, 0): one feature owns the dimension, the other's weight is pinned at zero. The trained loss should match L_A.

Deep in the warm region (sparse), the weights come back with opposite signs and similar magnitudes: an antipodal pair. Trained loss matches L_C.

On the boundary, the optimiser is genuinely uncertain, and the trained loss sits slightly below both predictions — because gradient descent is free to use intermediate solutions our three candidates did not include, such as a shortened second column with a small negative bias. That gap is a feature of the exercise, not a bug: closed-form candidates give bounds, and the optimiser finds the interior.

Three features, one dimension: a hard ceiling appears

Add a third feature and keep m = 1. Now something qualitatively new happens: the third feature can never be admitted, at any sparsity. Here is why, and it is a useful counterweight to the impression that sparsity buys unlimited capacity.

A one-dimensional space has exactly two directions. Features 1 and 2 take them, antipodally. Feature 3 must share a direction with one of them — say with feature 1, so c₁₃ = +1 (or, with a shorter column, some positive number). Positive interference at full strength. When feature 3 fires alone at strength u, feature 1's readout is ReLU(u − β), a false positive of essentially the whole magnitude.

Price it at leading order in d. With β = 0 the false-positive cost charged to feature 1 is

and the entire benefit of admitting feature 3 is at most I₃d/3 — the variance we stop discarding. Since importances are ordered, I₁ ≥ I₃, so the cost charged to feature 1 alone already equals or exceeds the whole benefit, before we count feature 3's own errors or feature 2's. The optimal threshold recovers at most 41% of it (chapter 6), which is not enough to flip an inequality this lopsided.

What happens on the boundary, and why runs disagree

Exactly at d* the three solutions cost the same, and interesting things happen in a neighbourhood around it.

The loss landscape flattens. Two distinct configurations have equal loss, and the path between them passes through intermediate weights — a shortened second column with a compensating bias — that are only slightly worse. Gradient descent has little pressure to prefer either.

Optimisers find the interior. Our three candidates are corners of a continuum. The trained loss at the boundary usually comes in a little below all three closed forms, because a partial solution — say w = (1, −0.55) with b₂ = −0.06 — can beat both corners. That is a general lesson about closed-form analysis: it gives you correct bounds and correct asymptotics, and it under-states what an optimiser will do in between.

Seeds diverge. Run the same configuration twice near the boundary and you can get an antipodal pair one time and a collapsed column the next. Not a bug; a genuine degeneracy in the objective.

Reading the diagram the way the paper draws it

The paper plots the horizontal axis as 1/(1−S) = 1/d on a log scale, which stretches the interesting sparse region out and compresses the dense corner. The consequence is worth stating plainly, because it is the practical takeaway for real systems:

Chapter 6: The Cost of Sharing

We have priced two specific arrangements exactly and watched a phase boundary come out of the algebra. Now we do the general accounting: for an arbitrary geometry, what exactly does interference cost, what does the bias buy, and when is the trade worth making?

The anatomy of a readout

Write c_ij = W_i · W_j for the entries of the Gram matrix, and let g_i = c_ii = ‖W_i‖² be the gain on feature i's own signal. Take the bias to be b_i = −β_i with β_i ≥ 0 — a threshold. Then

Y_i is the interference hitting feature i on this sample. Three separate things can now go wrong, and it is worth naming them because they trade against each other:

The statistics of the interference term

Y_i is a weighted sum of independent sparse features, so its first two moments are immediate. Using E[x] = d/2 and Var(x) = d/3 − d²/4:

Both are linear in d to leading order, which looks alarming until you remember that the loss only charges you when the interference actually causes an error. Still, these two numbers tell you the shape of the problem, and they connect straight back to chapter 4.

For a tight frame of k unit vectors in m dimensions arranged symmetrically about the origin, ∑_j W_j = 0, so

The decisive computation: positive versus negative sharing

Here is where the chapter earns its title. Take two unit-length features at angle θ, so c₁₂ = cos θ. Consider only the samples where exactly one of them fires — those have probability 2d(1−d), and they are the majority of interesting samples in a sparse world. Compare two angles.

θ = 180°, antipodal. c₁₂ = −1. Feature 2 fires alone at strength u. Feature 1's readout is ReLU(−u) = 0, which is the correct answer. Cost: exactly zero. Errors occur only when both fire, at probability d², and cost 1/6 each — the chapter 5 result.

θ = 60°. c₁₂ = +0.5. Feature 2 fires alone at strength u. Feature 1's readout is ReLU(0.5u − β). With no threshold (β = 0) that is 0.5u: a hallucination with expected square

and it is charged on every sample where the other feature fires alone — probability d(1−d), which is linear in d. Positive interference does not enjoy the d² discount at all. This is the quantitative version of chapter 3's claim, and the factor is enormous: at d = 0.01, linear-in-d is a hundred times worse than quadratic-in-d.

What the threshold buys, optimised exactly

The model is not helpless against positive interference — it has β. Let us find the best β for the 60° case and see how much it recovers. Two terms compete, and by symmetry they carry the same probability weight d(1−d):

False positives. Feature 2 alone at strength U. Cost E[ReLU(0.5U − β)²]. Substituting t = 0.5u − β:

Missed signal. Feature 1 alone at strength X. Readout ReLU(X − β), so the error is min(X, β), and from chapter 2's integral its expected square is β² − (2/3)β³.

Total per-feature cost factor:

Differentiate and set to zero:

Substitute back, carefully:

Against f(0) = 0.083333, the optimal threshold recovers 41% of the damage. Real, useful — and completely unable to change the scaling. The cost is still d(1−d) × 0.0488, still linear in d.

The general "does it pay" rule

Now we can state the decision the model faces for any new feature j it is considering admitting into an already-crowded space.

where κ⁺ collects the false-positive terms from positive dot products (paid whenever one feature fires alone) and κ⁻ collects the co-activation terms (paid only when two fire together). Everything the paper observes follows from which term dominates:

• If a geometry with all-negative dot products is available, κ⁺ = 0, cost is pure d², and admission is a formality at any real sparsity. Hence simplices.
• If the space is too crowded for that, κ⁺ > 0 and the benefit I_jd/3 must beat a cost that is also linear in d — a genuine contest, decided by importance and by how small the unavoidable positive cosines are.
• As d → 0 with κ⁺ > 0, the comparison becomes I_j/3 versus ∑_iI_iκ⁺_ij — a fixed threshold. This is why superposition does not increase without limit: past a point, adding features stops being free even in the sparse limit.

Capacity: the budget you cannot exceed

Chapter 4 showed that the feature dimensionalities of a tight frame sum to exactly m. The paper turns this into a general accounting notion it calls capacity: each feature consumes some fraction of a dimension, and the total consumption is bounded by the number of dimensions you have. Superposition is a reallocation, not a free lunch.

What sparsity changes is not the size of the budget but how finely it can be divided. In the dense regime the only permitted allocations are 0 and 1 — a feature either owns a dimension or gets nothing. As sparsity rises, halves become available, then thirds, then fifths. The paper's capacity plots show exactly this: sharp bands at 1, then 1/2, then lower, with the transitions at the phase boundaries we derived in chapter 5.

How many features fit? A usable scaling law

Chapter 0 gave a geometric answer — exponentially many directions fit. That answer is useless on its own, because it ignores whether the resulting code can be read. Here is the answer the loss actually cares about, and it is much smaller and much more useful.

Suppose k unit features sit in m dimensions in a roughly symmetric arrangement, so the typical squared cosine between two of them is about 1/m — the value you get from random directions and the value a tight frame with k ≫ m approaches. Then, from the variance formula above,

Now impose an engineering requirement: the standard deviation of the interference must stay below some tolerance τ, chosen relative to the size of a real activation (which averages 1/2 in our generator). Setting Var(Y_i) ≤ τ² and solving for k:

Worked example. Take m = 768 and τ = 0.2 (interference standard deviation of 0.2 against typical activations near 0.5 — noticeable but recoverable).

Sanity-check the middle number against intuition: 92,000 features in 768 dimensions is 120 features per dimension. That sounds absurd until you notice that at d = 0.001 only about 92 of them are on at once, so the hidden vector is a sum of 92 terms in a 768-dimensional space — comfortably underdetermined, but nowhere near saturated.

And note which bound is binding. The geometric capacity from chapter 0 at these dimensions runs into the thousands and beyond; the number above is what the reconstruction quality permits. Superposition is limited by interference budget, not by packing. That is why the paper is about a loss function rather than about sphere packing.

Chapter 7: Computing in Superposition

Everything so far has been about storage. The model's only job was to hold n features in m slots and hand them back. A real network does something harder: it computes. Attention heads compose, MLPs apply nonlinear functions, information gets transformed on its way through.

So the paper asks the sharper question. Not "can you store more features than you have dimensions" but "can you compute functions of more features than you have neurons?" The answer is yes, with the same mechanism and the same currency, and the consequences for interpretability are worse.

A function simple enough to fully understand

The paper picks absolute value. Features are now signed — x_i is zero with probability S and otherwise uniform on [−1, 1] — and the target is

with the model being a one-hidden-layer ReLU network:

Why absolute value? Because it is the simplest function that a linear model cannot do at all. A linear map cannot produce |x|; you need the nonlinearity. So any success here is unambiguously computation, not clever routing.

The monosemantic circuit, built by hand

ReLU keeps positives and zeroes negatives. So ReLU(x) is the positive part of x, and ReLU(−x) is the magnitude of its negative part. Add them:

Check both branches. If x = 0.6: ReLU(0.6) + ReLU(−0.6) = 0.6 + 0 = 0.6 ✓. If x = −0.6: 0 + 0.6 = 0.6 ✓. If x = 0: 0 + 0 = 0 ✓.

So for n features you can build an exact circuit with m = 2n neurons: rows of W equal to +e_i and −e_i, and each output summing its own pair. Every neuron responds to exactly one feature; every output is exact everywhere, not just on average. This is the monosemantic solution, and it is what an interpretability researcher dreams about.

Where it breaks: four rays, three neurons

Now cut the budget. Two features, three neurons. Can we still be exact?

Here is the tool that makes this tractable. At high sparsity the data does not fill the plane — it concentrates on the coordinate axes, because at most one feature is usually non-zero. There are exactly four rays to get right:

Every neuron is ReLU(w · x), which is positively homogeneous: along a ray, its output is just t times a constant. So the whole circuit is described by a 3 × 4 table — three neurons, four rays — and the required outputs are also fixed: y₁ must be t on both e₁ rays and 0 on both e₂ rays, and vice versa.

Try a circuit. Take n₁ = ReLU(x₁), n₂ = ReLU(−x₁ − x₂), n₃ = ReLU(x₂), and tabulate:

Look at rows two and four. The neuron activations are identical — (0, t, 0) in both cases — but the required outputs are different: (t, 0) versus (0, t). No readout, linear or otherwise, can produce two different answers from the same input. The circuit has collided two rays, and one of them must be sacrificed.

Try to fix it and you will find the fix moves the collision rather than removing it. Each neuron fires on at most one of the two e₁ rays (a ReLU cannot be positive at both +x₁ and −x₁) and at most one of the two e₂ rays, so covering all four rays with three neurons forces at least one neuron to be mixed. A mixed neuron contributes positively to two rays that need opposite answers, and the readout must then either cancel it — killing a ray that needed it — or accept it, producing a false positive.

What the paper actually observes

Given that, does anything interesting happen? Yes, and the paper documents it carefully.

Trained models do compute absolute value for more features than they have neuron-pairs, at high sparsity. The solutions have a characteristic form the paper calls asymmetric superposition: the positive and negative halves of a feature are handled differently. One half gets clean treatment from a nearly-dedicated neuron; the other half shares a neuron with a different feature and pays a leak. Which half is favoured, and which features share, is decided by importance and by initialisation.

The result is a layer of polysemantic neurons: units whose weight rows have several substantial entries, and which therefore light up for several unrelated features. Poke such a neuron with a dataset and you get a jumble — "fires for Python code, Korean text, and DNA sequences" — which is precisely the phenomenon that motivated this line of research.

Why leaky circuits are good enough: the data lives on a skeleton

Here is the reframing that makes superposed computation stop feeling like a hack.

A circuit's correctness is usually judged over its whole input domain. But the loss does not integrate over the domain — it integrates over the data distribution. And a sparse distribution puts almost all of its mass on a thin skeleton: with n features at density d, the probability that exactly one feature is non-zero is nd(1−d)ⁿ⁻¹, and the probability that two or more are non-zero shrinks like d².

Worked numbers. Take n = 100 features at density d = 0.01. The number of active features is Binomial(100, 0.01), so:

Now drop the density to 0.001: P(two or more) = 1 − 0.999¹⁰⁰ − 100(0.001)(0.999⁹⁹) = 1 − 0.90479 − 0.09057 = 0.00464. A hundred-fold reduction in the region where any interaction can go wrong, for a ten-fold reduction in density. The volume of input space where a superposed circuit is broken can be almost all of it, while the probability of ever visiting it is under half a percent.

Two observations to take away from that map.

The monosemantic circuit is black everywhere. Exactness on the whole square, at the cost of a fourth neuron. Notice that its loss curve is flat at zero regardless of density — sparsity buys it nothing, because it was never wrong.

The three-neuron circuits are black along most of the cross and bright off it. Slide the density down and watch the sample dots retreat onto the arms of the cross, abandoning the bright regions. The loss curves fall accordingly. That falling curve is superposition paying for itself — and it is the same shape of argument as chapters 3 and 5, applied to a circuit rather than a code.

Reading a polysemantic neuron

Take the mixed circuit above and interrogate its second unit the way an interpretability researcher would — by asking what makes it fire.

Feed it inputs and tabulate.

Every row is a true observation and the summary is a mess. The honest description — "this unit computes the positive part of −x₁ − x₂" — is not a feature, it is a coordinate of a rotated basis with a hinge on it. Asked "what does this neuron mean?", the correct answer is that the question has no answer, because meaning lives in the directions and this unit is not aligned with one.

Notice too that the last row is the killer for maximum-activating-example methods. A dataset search will surface inputs where x₁ and x₂ are both negative, because those maximise the response — and it will conclude the neuron detects the conjunction, which is exactly wrong. The unit is a linear projection, not an AND gate, and the top-k examples cannot tell the difference.

Why gradient descent settles for a leak

One more thing is worth saying, because it explains why the leaks land where they do rather than anywhere.

The gradient on the readout weights is proportional to the error times the neuron activation, averaged over the data distribution. On a broken ray, the error is large but the ray's probability weight is d(1−d)ⁿ⁻¹ — small. On a correct ray the error is zero and the gradient is zero. So the optimiser's incentive to fix the broken ray is proportional to how often that ray is visited, which means the leak migrates to the least-visited, least-important ray and stays there.

That produces a specific, checkable prediction about real models: the errors of a superposed circuit should concentrate on rare feature combinations and on unimportant features, not spread evenly. It is also a warning about evaluation. A benchmark built from typical inputs samples the same distribution the model optimised against, so it will not find these errors. You have to go looking off the skeleton.

What this does to interpretability

Chapter 2 ended by noting that in the dense regime, interpretability is trivial: read the neurons. Now we can say precisely what superposition costs us.

Chapter 8: What This Means for Embeddings

Everything so far has been about a five-parameter toy. This chapter cashes it out against something you actually ship: a dense embedding vector from a text encoder, sitting in a vector database, being compared by cosine similarity a million times a second.

The claim is simple and it reorganises how you should think about that vector:

Let us take the predictions one at a time, and be explicit about which are well-supported and which are just plausible.

Prediction 1: there are far more features than dimensions, and they are faint

Chapter 5 gave the admission rule: a feature with importance ratio r relative to the primary feature earns a place once density falls below roughly 2r. Real linguistic features are extremely sparse — "mentions a specific chemical compound" might have density 10⁻⁴ — so the rule admits features that are thousands of times less important than the dominant ones.

The practical consequence is a claim about norms: those admitted features have small ‖W_i‖, so they contribute little to the vector's length and are easily swamped. They are present, and they are quiet. If you probe for one and get 70% accuracy where you expected 95%, the toy model says: the feature may well be there, occupying a few percent of a dimension, exactly as its importance and sparsity purchased.

Prediction 2: truncation is not a haircut, it is a collision

This is the one with the sharpest practical teeth, so let us do it by hand.

Take the triangle code from chapter 3: three features in two dimensions, unit columns at 120°.

Fire feature 2 alone at strength 0.8. The code is

Read all three features out of the full vector:

Flawless. Now truncate to the first coordinate only — keep h₁ = −0.400, throw away h₂. The readout vectors truncate with it, to the scalars 1, −0.5, −0.5:

Read those last two lines again. After truncation, features 2 and 3 produce exactly the same number, for every input. They have not been degraded, they have been merged. Their truncated directions are identical, and no downstream consumer can ever separate them again.

Contrast the ordered code from chapter 2: features on their own axes, sorted by importance. Truncating to the first k dimensions removes features k+1 onward completely and leaves the rest bit-for-bit exact. No merges, no confusions, and you know precisely what you lost.

This is exactly what Matryoshka Representation Learning buys. MRL trains an embedding so that every prefix is itself a valid embedding, by adding a loss term at each truncation length. In the language of this lesson: it forces the leading dimensions into the dense-regime solution — important features on nearly private, ordered directions — while allowing superposition in the tail. Truncation then behaves like the second case rather than the first.

Prediction 3: anisotropy has a mechanism, and it is importance

Embedding spaces are famously anisotropic: vectors from a trained encoder do not spread evenly over the sphere but cluster in a narrow cone, so that two randomly chosen texts have a cosine similarity of 0.6 rather than 0. A handful of "rogue" dimensions dominate the variance, and post-processing tricks — centring, removing top principal components, whitening — measurably improve retrieval.

The toy model explains both halves of this, and it separates them.

The rank-one part. The hidden vector is h = ∑x_jW_j with all x_j ≥ 0. Take expectations: E[h] = (d/2) ∑_j W_j. Unless the feature directions happen to sum to zero, every embedding carries a common offset along that sum, and the second-moment matrix picks up a rank-one term (d/2)² (∑W_j)(∑W_j)^T. That common direction is the "top principal component" that centring and all-but-the-top removal delete. Non-negative features plus an uncentred code is all it takes.

The unequal-variance part. Beyond the mean, the covariance is ≈ Var(x) W W^T. For a perfect tight frame, W W^T = (k/m)I — isotropic. So a uniformly-superposed code is not anisotropic. Anisotropy comes from the columns having unequal lengths, which is precisely what unequal importance produces: chapter 5's admission rule gives important features long columns and marginal features short ones.

Prediction 4: there is a floor under cosine similarity, and it is interference

Two texts that share no features still have a non-zero dot product, because their feature directions are not orthogonal. Chapter 0 gave the scale: random directions in m dimensions have typical cosine 1/√m, so with m = 768 the floor from geometry alone is about 0.036 per feature pair, and with tens of active features on each side the accumulated similarity is not small.

It is worth computing that floor properly, because the answer is unexpectedly clean and it separates two effects people usually conflate.

Let text A activate a features and text B activate b features, with no overlap — genuinely unrelated documents. Their codes are h_A = ∑_i∈Ax_iW_i and h_B = ∑_j∈Bx_jW_j. With random unit directions, E[W_i · W_j] = 0 and E[(W_i · W_j)²] = 1/m. So the dot product has mean zero and variance

and the norms are ‖h_A‖ ≈ √(a/3), ‖h_B‖ ≈ √(b/3). Divide:

The a and b cancel completely. However many features are active on either side, the interference-driven cosine floor is just 1/√m — 0.036 at 768 dimensions, 0.051 at 384, 0.026 at 1536.

Two operational consequences follow.

Absolute cosine thresholds do not transfer. A cut-off of 0.8 that works for one encoder is meaningless for another with a different dimension, feature count or importance profile. Rank-based retrieval is robust to the floor; thresholding is not.

Bigger is quieter. Going from 384 to 1536 dimensions cuts the typical interference cosine by a factor of two, and does so without needing any more features. Much of the quality gain from larger embedding dimensions is not "more capacity for concepts" but "less crosstalk between the concepts already there".

Prediction 5: aggressive quantisation should work, and does

Binary and int8 embeddings routinely retain 95–99% of retrieval quality. Under the superposition picture this is unsurprising: the information lives in which directions are active, and direction identity survives coarse quantisation far better than magnitude does. Quantisation adds a noise term of roughly uniform size across dimensions — which is exactly the kind of isotropic perturbation that a near-orthogonal code is robust to, and exactly the kind that truncation is not, because truncation is anisotropic by construction.

The asymmetry is worth internalising: quantise before you truncate. Cutting 768 floats to 768 bytes damages a superposed code far less than cutting 768 floats to 192 floats, even though the second saves the same memory.

Prediction 6: dictionary learning should recover features from embeddings

If an embedding is a sparse linear mixture of an overcomplete set of directions, then the classical way to recover those directions is sparse dictionary learning — the same problem Olshausen and Field solved for natural images in 1996. Train an autoencoder with a wide hidden layer and an L1 penalty, and its dictionary elements should line up with the underlying features.

That is exactly what sparse autoencoders do, and they were applied first to language-model activations and then, successfully, to embedding vectors themselves. The toy model is what tells you that this should work at all, and — importantly — what tells you the dictionary must be overcomplete: if there are more features than dimensions, a dictionary of size m cannot possibly suffice.

What to do differently on Monday

A lesson that only reorganises your beliefs has done half a job. Here is the operational half.

Measure your feature density before choosing a dimension. The scaling law from chapter 6, k ≈ 3mτ²/d, says capacity is dimensions over density. If the properties your downstream task cares about are extremely rare, a smaller embedding will hold more of them than you expect; if your corpus is narrow and its features fire constantly, extra dimensions buy less than the price list suggests.

Prefer quantisation to truncation, and truncate only with a matryoshka model. Same memory saving, radically different failure mode: isotropic noise versus feature merges. If you must truncate a non-matryoshka embedding, measure which pairs of concepts collapse, not just average retrieval quality — the average will look fine while a handful of pairs become indistinguishable.

Centre before you compare, and know why. Subtracting the corpus mean removes the rank-one common direction that non-negative features inevitably create. That is a different intervention from whitening, which additionally flattens the norm imbalance that encodes importance. Centring is nearly free and nearly always right; whitening trades information for isotropy and should be measured, not assumed.

Treat a weak probe as ambiguous evidence. A feature can be present at a few percent of a dimension and still be genuinely there. Before concluding "the model does not represent X", check whether a probe trained on the residual after removing the top principal components does better — the faint features live under the loud ones.

Do not read neurons. If you find yourself inspecting individual embedding coordinates and looking for meaning, the whole of this lesson says you are looking in a basis the model had no reason to prefer. Learn a dictionary instead.

Chapter 9: Limits and What Came After

This paper became one of the most-cited artefacts in mechanistic interpretability, and it deserved to. It also has real limits, and the authors are unusually direct about them. Reading a paper properly means being able to say what it does not show, so let us do that first.

What the toy model does not establish

The immediate consequence: sparse autoencoders

Chapter 7 ended with the specification: to interpret a model in superposition you need to recover an overcomplete set of directions from their sparse mixtures. The tool for that is dictionary learning, and the neural version is the sparse autoencoder.

The construction is almost embarrassingly direct. Take activations h ∈ R^m. Train

Look at the shapes and you can see the toy model being run in reverse. The toy model took a sparse n-vector and squeezed it into m dimensions; the sparse autoencoder takes the m-dimensional result and asks for the sparse M-vector that generated it, with the L1 penalty supplying the sparsity assumption the toy model started from. The columns of W_dec are the recovered feature directions — the estimate of W.

The results, in order of appearance:

• Towards Monosemanticity (Bricken et al., 2023) applied this to a one-layer transformer's MLP and recovered thousands of features that were interpretable, causally relevant, and far more numerous than the neurons. The toy model's prediction, confirmed on a real network.
• Sparse Autoencoders Find Highly Interpretable Features (Cunningham et al., 2023) independently reached the same conclusion on larger models.
• Scaling Monosemanticity (Templeton et al., 2024) took it to Claude 3 Sonnet, extracting millions of features including abstract ones — code errors, deception, the Golden Gate Bridge — and demonstrated causal steering by clamping them.
• Architecture refinements: gated sparse autoencoders (Rajamanoharan et al., 2024) and TopK autoencoders (Gao et al., 2024) attack the shrinkage bias that the L1 penalty introduces — the same "threshold damages true signal" trade-off we optimised by hand in chapter 6, reappearing in the tool built to undo it.

Open problems the field is still chewing on

Feature absorption and splitting. Train a wider dictionary and features split into finer ones; some features get absorbed into others. Is there a "true" feature set, or is the decomposition scale-dependent all the way down? Nobody knows.

Are all features linear? Engels et al. (2024) found multi-dimensional, genuinely circular representations — days of the week, months — that no set of independent directions captures. The linear representation hypothesis is a good approximation, not a law.

Superposition of computation, at scale. Chapter 7's story is well established in toys and much less so in real models. How attention heads and MLP circuits superpose is largely open.

Does the phase-diagram picture transfer? The boundaries we derived assumed a reconstruction loss on uniform features. Whether a next-token-prediction objective produces the same phase structure is an assumption, not a result.

What to actually take away

Four things, in order of how confidently you should hold them.

One, nearly certain. Nearly-orthogonal directions are exponentially plentiful, and interference between them is only paid when features co-occur. That is mathematics, not modelling.

Two, very well supported. Given sparse features and a bottleneck, gradient descent will superpose — and it does so via a sharp phase change, not a gradual blend, with geometry drawn from a small zoo of polytopes.

Three, well supported empirically. Real language models are in superposition, and sparse dictionary learning recovers directions from them that are interpretable and causally effective.

Four, a working hypothesis. The specific quantitative structure — phase boundaries, feature dimensionalities, capacity conservation — carries over to real models in a form you can compute with. Treat this as a source of predictions, not conclusions.

Connections

References

1. Elhage, Hume, Olsson, Schiefer, Henighan, Kravec, Hatfield-Dodds et al. "Toy Models of Superposition." Transformer Circuits Thread, 2022. transformer-circuits.pub — the paper this lesson is about.
2. Olah, Cammarata, Schubert, Goh, Petrov, Carter. "Zoom In: An Introduction to Circuits." Distill, 2020. distill.pub
3. Elhage et al. "Softmax Linear Units." Transformer Circuits Thread, 2022. transformer-circuits.pub
4. Bricken, Templeton, Batson et al. "Towards Monosemanticity: Decomposing Language Models With Dictionary Learning." 2023. transformer-circuits.pub
5. Templeton, Conerly, Marcus et al. "Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet." 2024. transformer-circuits.pub
6. Cunningham, Ewart, Riggs, Huben, Sharkey. "Sparse Autoencoders Find Highly Interpretable Features in Language Models." 2023. arXiv:2309.08600
7. Gao, la Tour, Tillman et al. "Scaling and Evaluating Sparse Autoencoders." 2024. arXiv:2406.04093
8. Rajamanoharan, Conmy, Smith et al. "Improving Dictionary Learning with Gated Sparse Autoencoders." 2024. arXiv:2404.16014
9. Engels, Liao, Michaud, Gurnee, Tegmark. "Not All Language Model Features Are Linear." 2024. arXiv:2405.14860
10. Scherlis, Sachan, Jermyn, Benton, Shlegeris. "Polysemanticity and Capacity in Neural Networks." 2022. arXiv:2210.01892
11. Kusupati, Bhatt, Rege et al. "Matryoshka Representation Learning." 2022. arXiv:2205.13147
12. Ethayarajh. "How Contextual are Contextualized Word Representations?" EMNLP 2019. arXiv:1909.00512
13. Gao, He, Tan, Qin, Wang, Liu. "Representation Degeneration Problem in Training Natural Language Generation Models." ICLR 2019. arXiv:1907.12009
14. Timkey, van Schijndel. "All Bark and No Bite: Rogue Dimensions in Transformer Language Models." EMNLP 2021. arXiv:2109.04404
15. Mu, Viswanath. "All-but-the-Top: Simple and Effective Postprocessing for Word Representations." ICLR 2018. arXiv:1702.01417
16. Olshausen, Field. "Emergence of simple-cell receptive field properties by learning a sparse code for natural images." Nature 381, 1996 — the origin of the dictionary-learning idea.
17. Benedetto, Fickus. "Finite Normalized Tight Frames." Advances in Computational Mathematics 18, 2003 — the frame-potential theorem behind chapter 4.
18. Johnson, Lindenstrauss. "Extensions of Lipschitz mappings into a Hilbert space." Contemporary Mathematics 26, 1984 — the almost-orthogonality result behind chapter 0.
19. Park, Choe, Veitch. "The Linear Representation Hypothesis and the Geometry of Large Language Models." 2023. arXiv:2311.03658