Linear Algebraic Structure of Word Senses

Chapter 1: One Word, One Vector?

Before we can understand why superposition works, we need to understand what a word vector is and why it captures meaning at all.

What word vectors encode

Word2vec and GloVe learn vectors by looking at co-occurrence patterns. Words that appear in similar contexts get similar vectors. "King" and "queen" appear near words like "royal," "throne," "crown" — so their vectors end up close together. "King" and "bicycle" appear in very different contexts — so their vectors are far apart.

The remarkable thing is that these vectors capture relational structure. The famous example: vec("king") - vec("man") + vec("woman") ≈ vec("queen"). The difference between "king" and "man" encodes the concept of royalty, and adding it to "woman" lands you near "queen." This means directions in embedding space correspond to semantic relationships.

Other examples of linear structure: vec("Paris") - vec("France") + vec("Japan") ≈ vec("Tokyo"). vec("walking") - vec("walk") + vec("swim") ≈ vec("swimming"). These relationships hold because the co-occurrence statistics of "Paris" relative to "France" mirror those of "Tokyo" relative to "Japan." The vector arithmetic captures regularities in how words relate to each other.

But here's the tension: if word vectors are so good at capturing meaning through linear structure, how do they handle the fundamentally non-linear phenomenon of polysemy? A word with two unrelated meanings shouldn't satisfy any simple linear relationship — yet it does. Understanding why requires diving into the mathematics of high-dimensional geometry.

Why didn't people check this before?

The superposition property seems like something someone would have noticed earlier. But there were two obstacles:

1. No theoretical framework: Without the random walk discourse model (Arora et al. 2016), there was no reason to expect linear superposition. People assumed the combination would be nonlinear because word2vec's training objective involves nonlinear functions (sigmoid, softmax). The insight that these nonlinearities wash out in high dimensions was not obvious.
2. Hard to verify directly: To check superposition, you need sense-specific vectors — but those don't exist in standard word2vec outputs. You need a sense-tagged corpus (rare and expensive) or a clever indirect method (like the sparse coding approach). The paper provided both the theory to predict superposition and the tools to verify it.

The problem with polysemy

But consider "bank." It co-occurs with "money," "account," "deposit" (financial sense) AND with "river," "water," "fishing" (river sense). The word2vec training procedure sees ALL these contexts and produces a single vector that tries to satisfy both sets of relationships simultaneously.

A concrete experiment

To see why a single vector is problematic, try this thought experiment. Take the word "cell" (biology: a living cell; technology: a cell phone; prison: a jail cell). Compute its nearest neighbors in word2vec:

1. cells (biology)
2. cellular (technology)
3. tissue (biology)
4. phone (technology)
5. prison (prison)

The neighbors are a mixed bag from all three senses. This means any task that relies on nearest neighbors — information retrieval, word sense disambiguation, analogies — gets confused. You can't tell which meaning of "cell" is relevant without decomposing the vector.

python
import gensim.downloader as api

# Load pre-trained word2vec vectors
model = api.load('word2vec-google-news-300')

# Nearest neighbors for "bank" — a mix of senses
for word, sim in model.most_similar('bank', topn=10):
    print(f"  {word:15} {sim:.3f}")
# Output mixes financial ("banking", "lender") and river ("riverbank")

# The vector for "bank" is somewhere between both senses
# cos(bank, money) ≈ 0.48, cos(bank, river) ≈ 0.33
print(model.similarity('bank', 'money'))   # 0.48
print(model.similarity('bank', 'river'))   # 0.33

What "recovery" means

Imagine the word "bank" has two senses with vectors v₁ (financial) and v₂ (river). If the combined vector v_bank is a linear combination:

where α₁ and α₂ are weights proportional to how often each sense appears in the training corpus, then we might be able to decompose v_bank to recover v₁ and v₂ separately. But first, we need to prove that this linear relationship actually holds.

Chapter 2: The Superposition Claim

Here is the central claim of the paper, stated precisely:

Several things to unpack here:

1. Linear, not nonlinear

The combination is a plain weighted sum — no activation functions, no complicated transforms. This is surprising. You might expect the word2vec training objective (which involves sigmoid functions and dot products) to produce some nonlinear mixing of senses. But it doesn't.

2. Frequency-weighted

The weights are not arbitrary. They depend on how often each sense appears. If 90% of "bank" occurrences are financial, then α₁ is much larger than α₂, and v_bank is much closer to the financial sense vector.

3. The exponent ζ

The weights are not simply the frequencies — they are frequencies raised to a power ζ between 0 and 1. This is a sublinear weighting. Why? Because word2vec's implicit weighting of co-occurrence statistics isn't linear in frequency. The paper shows ζ depends on the specific embedding algorithm (word2vec vs GloVe) and its hyperparameters.

4. "Would have" vectors

The sense vector v_si is defined as the vector the sense would get if we could train word2vec on a corpus where every occurrence of "bank" was replaced by "bank_financial" or "bank_river" depending on the intended sense. These hypothetical vectors are well-defined even though we never explicitly train them.

This is a subtle but critical point. We're NOT saying you need to manually disambiguate the corpus. The claim is that v_bank can be decomposed into components that match what you'd get from a disambiguated corpus. The sense vectors exist as latent directions within the embedding space — we just need a way to find them.

Testing the claim empirically

How do you verify that superposition actually holds? The paper uses a clever trick. Take a polysemous word like "bank." Find all sentences in the corpus containing "bank" and manually label them as financial or river. Compute the average context vector for each sense (the average of neighboring word vectors). These context-based sense vectors approximate the "would have" vectors v_s1 and v_s2.

Then check: is v_bank ≈ α₁ v_s1 + α₂ v_s2? The paper finds this holds with cosine similarity > 0.95 for most polysemous words tested. The linear model is an excellent fit.

python
# Verify superposition empirically
import numpy as np

# v_bank: the actual word2vec vector for "bank"
# v_s1: average context vector for financial sense
# v_s2: average context vector for river sense
# f1, f2: frequency of each sense in corpus

def verify_superposition(v_word, sense_vecs, freqs, zeta=0.7):
    """Check if v_word ≈ sum(alpha_i * v_si)"""
    alphas = np.array([f**zeta for f in freqs])
    alphas /= alphas.sum()  # normalize

    # Weighted sum of sense vectors
    v_combined = sum(a * v for a, v in zip(alphas, sense_vecs))

    # Cosine similarity with actual word vector
    cos_sim = np.dot(v_word, v_combined) / (
        np.linalg.norm(v_word) * np.linalg.norm(v_combined))
    return cos_sim, alphas

# Typical result: cosine_sim > 0.95

Empirical evidence

The paper verifies this on real word vectors. For words with known senses (from WordNet), they compute the cosine similarity between v_w and the best linear combination of sense vectors estimated from sense-tagged corpora. The fit is strong — typically r > 0.9. The linear model explains over 80% of the variance in the word vector.

The verification procedure works as follows:

1. Select a set of polysemous words with well-separated senses from WordNet (e.g., "bank," "crane," "bat," "spring"). WordNet provides a standardized sense inventory for each word.
2. For each word, use a sense-tagged corpus (SemCor or OMSTI) to identify which sense is used in each occurrence. SemCor annotates approximately 234,000 instances across 352 texts.
3. Compute the sense vector as the centroid of context vectors for occurrences of that sense. The context vector for an occurrence is the average of word2vec vectors of the 5 words on each side.
4. Find the optimal weights α_i by least-squares regression: minimize ||v_w − ∑ α_i v_si|| subject to α_i ≥ 0 (nonnegative constraint ensures the combination is physically meaningful).
5. Report the R² (fraction of variance explained) and cosine similarity between the predicted combination and the actual word vector.

The results are consistently strong across different embedding algorithms (word2vec CBOW, word2vec skip-gram, GloVe), different vector dimensions (100, 200, 300), and different corpora (Wikipedia, Google News, Common Crawl). The superposition relationship is robust — it's not an artifact of a particular training setup. Even negative sampling rate (k = 5 vs k = 15) and window size (5 vs 10) don't meaningfully change the quality of the linear fit. This universality strongly supports the theoretical explanation based on high-dimensional geometry rather than algorithm-specific properties.

What ζ tells us

The exponent ζ in the formula α_i = f_i^ζ has a value around 0.6-0.8 for word2vec and 0.5-0.7 for GloVe. What does this mean intuitively?

If ζ = 1, the weights would be proportional to the raw frequencies — a 90% dominant sense would contribute 90% to the word vector. If ζ = 0, all senses would contribute equally regardless of frequency.

The fact that ζ ∈ (0.5, 0.8) means the combination is frequency-weighted but democratized. A rare sense (10% frequency) contributes more than 10% but less than an equal share. Specifically, with ζ = 0.7: a sense with frequency 0.1 gets weight 0.1^0.7 ≈ 0.20 (twice its frequency share), while a sense with frequency 0.9 gets weight 0.9^0.7 ≈ 0.93 (slightly less than its frequency share). This "compression" of the frequency distribution is what makes rare senses recoverable — they get more representation in the word vector than their raw frequency would suggest.

This is actually good news for sense recovery. If rare senses were completely dominated by frequent senses (ζ = 1), they'd be invisible in the word vector. The sublinear weighting (ζ < 1) ensures that even a 5% sense leaves a detectable trace.

Chapter 3: The Random Walk Model

The superposition result doesn't appear out of thin air. It follows from a specific theoretical model of how text is generated and how word vectors arise from that generative process. This model was introduced in an earlier paper by Arora et al. (2016) and forms the mathematical backbone of the polysemy result.

The discourse vector

Imagine you're writing a document. At each moment, there's a discourse vector c_t — a vector in the same space as word embeddings — that represents "what you're currently talking about." When you're writing about finance, c_t points in the direction of financial concepts. When you're describing nature, c_t shifts toward nature-related directions.

The discourse vector evolves over time as a random walk:

where η_t is a small random perturbation. This means the topic drifts slowly and smoothly — consecutive words tend to be about the same thing, but the topic gradually shifts over the course of a document. This matches how real text works.

Word generation

At each position t, the probability of emitting word w is:

Words whose vectors align with the current discourse vector are more likely to appear. If c_t points toward finance, words like "deposit," "account," "interest" have high dot product with c_t and are likely to appear.

Connection to PMI

The model implies that the pointwise mutual information (PMI) between two words relates to their vector dot product:

where d is the embedding dimension. This is exactly the relationship that GloVe and word2vec implicitly optimize. The random walk model thus unifies different embedding algorithms under one theoretical roof.

Why the random walk is realistic

The random walk model captures three key properties of real text:

1. Local coherence: Consecutive words tend to be about the same topic. The small step size of the random walk ensures c_t and c_t+1 are close.
2. Topic drift: Over longer stretches, the topic changes. A paragraph about cooking might transition to nutrition, then to health. The cumulative effect of many small random steps creates these slow topic shifts.
3. Context-dependent word choice: The probability of each word depends on the current "context" (discourse vector), which matches the fundamental assumption of word embedding algorithms — that context determines word meaning.

The key insight for polysemy: when the random walk visits the "financial district" of the embedding space, it generates financial-sense occurrences of "bank." When it visits the "nature" region, it generates river-sense occurrences. The word vector for "bank" is the average of all these positions, weighted by how often the walk visits each region. This is exactly the superposition formula.

How fast does the walk mix?

An important parameter is the step size of the random walk. If the walk takes tiny steps, consecutive words are about exactly the same topic — strong local coherence but slow topic changes. If the walk takes large steps, topics change rapidly — weak coherence but more diverse co-occurrence patterns.

The step size determines the context window of the embedding algorithm. Word2vec uses a context window of 5-10 words. Within this window, the discourse vector hasn't moved much, so all words share approximately the same context. Across windows, the discourse vector has drifted, creating the contrastive signal that makes different words distinguishable.

For polysemy, the mixing time determines how often the walk transitions between sense regions. If "bank" appears in a financial document, the walk stays in the financial region for many paragraphs before potentially wandering toward a river context. The relative time spent in each region determines the sense frequencies f₁, f₂.

Chapter 4: Why Superposition Holds

Now we can prove why the word vector for a polysemous word is a linear combination of its sense vectors. The proof follows directly from the random walk model, and understanding it reveals exactly when and why superposition works.

Setup

Suppose word w has k senses. In the corpus, fraction f_i of occurrences of w use sense i. Each sense i has its own "sense vector" v_si — the vector w would get if only sense-i occurrences existed.

Step 1: Co-occurrence decomposes by sense

The key observation is that when word w co-occurs with some other word w', it's because one particular sense of w is active in that context. If "bank" appears next to "deposit," the financial sense is active. If "bank" appears next to "river," the river sense is active. So the total co-occurrence count decomposes by sense:

In the random walk model, the probability of w and w' co-occurring when sense i is active depends on v_si · v_w'. So the total log co-occurrence rate is:

Step 2: The log-sum-exp approximation

This is where the key mathematical step happens. We need to show that the function log(∑ f_i exp(x_i)) is approximately linear in the x_i values. Why would a clearly nonlinear function (log of sum of exponentials) behave linearly?

The answer lies in the regime we're operating in. In high dimensions (d = 300), the dot product v_si · v_w' is typically small — on the order of 1/√d ≈ 0.06. When the arguments to exp() are this small, exp(x) ≈ 1 + x + x²/2. The quadratic and higher terms are negligible, making the whole expression approximately linear.

Formally, the paper proves a lemma: when the x_i values have variance σ² that is small (specifically, σ² · max|x_i| << 1), then:

where ζ = 1 − σ²/2 + O(σ⁴). In the high-dimensional regime, ζ ∈ (0.5, 1.0) typically. The exponent ζ corrects for the curvature of the exponential — it's a first-order adjustment that makes the linear approximation much more accurate.

Concentration of measure — the key ingredient

Why are dot products small in high dimensions? This is one of the most important facts in high-dimensional geometry, and it's counterintuitive. In 2D, two random unit vectors can easily have a dot product of 0.7 or 0.8. In 300D, the dot product between two random unit vectors is almost always between −0.1 and +0.1.

Formally, if v and w are independent random vectors on the unit sphere in R^d, then:

In d = 300, the standard deviation of the dot product is 1/√300 ≈ 0.058. So most dot products are within 2 standard deviations of zero: between −0.12 and +0.12. This means the arguments to exp() in the log-sum-exp are small, and the linear approximation is excellent.

This is why superposition ONLY works in high dimensions. In d = 2 or d = 10, the dot products would be large, the linear approximation would fail, and polysemous word vectors would not be recoverable superpositions. The high dimensionality of word embeddings (d = 300) is not just a hyperparameter choice — it's a mathematical prerequisite for the superposition structure.

Step 3: From PMI to vectors

Since PMI(w, w') ≈ v_w · v_w' / d for any w', and we showed the PMI decomposes as a weighted sum of sense-specific terms, we can conclude:

where α_i ∝ f_i^ζ. Since this holds for all w', we conclude:

The word vector is a weighted sum of sense vectors. QED.

Worked example

Let's trace through the proof with numbers. Suppose "bank" has two senses with frequencies f₁ = 0.7 (financial) and f₂ = 0.3 (river). In 300 dimensions, a typical dot product v_s1 · v_w' ≈ 0.02 (small, due to near-orthogonality). So:

Compare with the linear approximation:

The linear approximation is very close. The error is on the order of (dot product)³, which in 300 dimensions is around 10⁻⁶. This is why superposition works so well in practice — the approximation error is negligible in high dimensions.

Chapter 5: Recovering Senses

We've proven that the word vector is a linear combination of sense vectors. Now the practical question: given just the word vectors (no sense labels), can we actually recover the individual sense vectors?

The sparse coding approach

The paper proposes using sparse coding to decompose word vectors into atomic "sense atoms." The idea is elegant: most word vectors should be expressible as a sparse combination of a small set of basis vectors (the "atoms"). Each atom represents a basic semantic unit — what the paper calls a sememe.

The algorithm

The sparse coding objective finds a dictionary of atoms A = [a₁, a₂, ..., a_m] and sparse coefficient vectors x_w such that:

subject to each x_w being sparse (most entries near zero). The optimization problem is:

The first term is the reconstruction error — how well can we reconstruct each word vector from its atoms? The second term is the sparsity penalty — the L1 norm encourages most coefficients to be exactly zero. The parameter λ controls the tradeoff: larger λ means sparser (fewer active atoms per word), smaller λ means denser (more active atoms, better reconstruction).

This is solved by alternating minimization:

From atoms to senses

Once we have the sparse decomposition, how do we go from atoms to senses? Each word w has a sparse coefficient vector x_w with, say, 5 nonzero entries. For a polysemous word, these nonzero entries naturally cluster into groups — one group per sense. The atoms used by "bank" (financial) are different from the atoms used by "bank" (river).

The paper identifies senses by looking at which atoms have the largest coefficients and clustering them. For a word with two well-separated senses, the atoms cleanly partition into two groups, and each group's weighted sum gives the sense vector.

Concrete walkthrough

Suppose sparse coding gives us this decomposition for "bank":

The first three atoms (institution, money, storage) are semantically related — they form a cluster. The last three (water, edge, nature) form another cluster. We identify two senses:

The relative magnitudes tell us that the financial sense dominates (larger coefficients), consistent with "bank" appearing more often in financial contexts in the training corpus.

For a monosemous word like "guitar," all active atoms belong to one coherent cluster (music, instrument, strings, acoustic). No separation is needed — and the algorithm correctly doesn't produce multiple senses.

The sememe hypothesis

The paper's use of sparse coding connects to a long-standing hypothesis in linguistics: that meanings are built from sememes — atomic units of meaning that combine to form complex concepts.

In traditional linguistics, a sememe is the smallest unit of meaning that cannot be further decomposed. For example, the concept "bachelor" might be composed of the sememes {male, unmarried, adult}. "Spinster" would be {female, unmarried, adult}. These share two of three sememes, explaining why they're semantically similar despite referring to different genders.

The sparse coding atoms ARE computational sememes. Each atom is a direction in embedding space that represents one atomic semantic concept. Words are composed of small numbers of these atoms, and the composition is linear (just addition of weighted directions). This is a computational vindication of the sememe hypothesis: the atoms learned by sparse coding correspond to interpretable semantic units.

Connection to modern sparse autoencoders (SAEs)

In 2023, researchers at Anthropic and elsewhere began applying the same idea — sparse decomposition of learned representations — to the hidden states of large language models. They train sparse autoencoders (SAEs) that decompose each residual stream vector into a sparse combination of learned features:

where h is the hidden state, f_i are learned feature directions, and c_i are sparse coefficients. This is mathematically identical to Arora et al.'s sparse coding of word vectors. The features learned by SAEs are interpretable — one might fire for "mentions of the Golden Gate Bridge," another for "Python code syntax," another for "sarcasm." They are the sememes of the neural network's internal language.

python
# Sparse autoencoder: same idea as sparse coding for word vectors,
# but applied to transformer hidden states
import torch
import torch.nn as nn

class SparseAutoencoder(nn.Module):
    def __init__(self, d_model, n_features, sparsity_coeff=1e-3):
        super().__init__()
        self.encoder = nn.Linear(d_model, n_features)  # overcomplete
        self.decoder = nn.Linear(n_features, d_model)
        self.sparsity_coeff = sparsity_coeff

    def forward(self, h):
        # h: (batch, d_model) hidden states
        c = torch.relu(self.encoder(h))  # sparse coefficients
        h_hat = self.decoder(c)            # reconstruction
        # Loss = reconstruction + L1 sparsity
        loss = (h - h_hat).pow(2).mean() + \
               self.sparsity_coeff * c.abs().mean()
        return h_hat, c, loss

# Same math as Arora et al.'s sparse coding!
# encoder.weight rows = atoms (sememes)
# c = sparse coefficients per input

Why sparsity is essential

You might ask: why not just use PCA or standard matrix factorization? The answer is that PCA gives you dense coefficients — every atom contributes to every word. This makes it impossible to cluster atoms into sense groups, because every sense blends into every other sense. Sparsity is what creates the clean separation.

Think of it like mixing paint colors. If you mix every color a little bit to get a word's representation (dense coding), you end up with brown — indistinguishable. If each word uses only a handful of pure colors (sparse coding), you can identify the original pigments. The sememe hypothesis says that meanings ARE built from a sparse set of semantic "pigments."

Mathematically, the advantage of sparsity comes from the overcomplete dictionary. In PCA, you have exactly d basis vectors for d dimensions — a complete but minimal basis. Every vector must be expressed using ALL basis vectors (dense). In sparse coding, you have m >> d dictionary atoms — far more atoms than dimensions. This redundancy means each word can "choose" a small subset of atoms that best match its meaning, ignoring the rest. Different senses choose different subsets, creating the clean separation we need.

Convergence guarantees

The paper proves that the alternating minimization algorithm converges to a local minimum of the objective under mild conditions on the dictionary. In practice, convergence takes 50-200 iterations and produces dictionaries that are reproducible across random initializations — the solution is robust, not a fragile local minimum. Running the algorithm on standard word2vec embeddings (300 dimensions, 50K vocabulary) takes about 10 minutes on a laptop. No GPU needed.

Choosing the number of atoms

The dictionary size m (number of atoms) is a hyperparameter. The paper finds that m should be significantly larger than the vocabulary size — typically m ≈ 2000 for a 50K-word vocabulary. This overcomplete dictionary ensures there are enough atoms to capture fine-grained semantic distinctions. If m is too small, different senses are forced to share atoms. If m is too large, atoms become redundant.

Quantitative results

On standard word sense benchmarks (SCWS — Stanford Contextual Word Similarity), the recovered sense vectors outperform the original single vectors. For highly polysemous words (3+ senses), the improvement is 8-12% in Spearman correlation with human similarity judgments.

The sparse recovery approach outperforms both the original single vectors AND the multi-prototype approach of Huang et al., despite using the same pre-trained embeddings as input. No retraining needed.

The improvement is most dramatic for words with well-separated senses. For "bank" (financial vs. river), the similarity between "bank" and "money" jumps from 0.48 (using the single combined vector) to 0.82 (using the recovered financial sense vector). The similarity between "bank" and "river" simultaneously jumps from 0.33 to 0.79 (using the recovered river sense vector). By decomposing the combined vector, we get much more accurate similarity judgments for each individual sense.

For words with related senses (like "bright" meaning luminous vs. intelligent), the improvement is smaller — about 3-5% — because the senses share semantic features and are harder to separate. This matches the theory: related senses have correlated vectors, making decomposition harder.

python
import numpy as np
from sklearn.decomposition import DictionaryLearning

# Learn sparse dictionary from word vectors
# word_vectors: (n_words, dim) array of word embeddings
dl = DictionaryLearning(
    n_components=2000,     # number of atoms (sememes)
    alpha=1.0,              # sparsity penalty
    max_iter=100,
    fit_algorithm='cd',    # coordinate descent
    transform_algorithm='lasso_cd'
)
# Fit dictionary and get sparse codes
sparse_codes = dl.fit_transform(word_vectors)
dictionary = dl.components_  # (2000, dim) atom vectors

# For a polysemous word, find its nonzero atoms
word_idx = vocab["bank"]
coeffs = sparse_codes[word_idx]  # (2000,)
active = np.where(np.abs(coeffs) > 0.1)[0]
print(f"Active atoms: {len(active)}")  # typically 3-8

# Sense vectors = cluster atoms by semantic similarity
# then sum weighted atoms within each cluster

Chapter 6: Sense Explorer

Let's put it all together. This interactive simulation demonstrates the full pipeline: a polysemous word is embedded as a superposition of sense vectors, and sparse coding recovers the individual senses.

The full pipeline in action

Let's trace the complete sense recovery pipeline step by step for the word "crane" (which can mean a bird or a construction machine):

1. Input: The single word2vec vector for "crane," trained on 1B tokens of English text.
2. Dictionary learning: Run sparse coding on the entire vocabulary (50K vectors) to learn a dictionary of 2000 atoms.
3. Sparse decomposition: Express v_crane as a sparse combination of atoms. Result: 6 atoms with nonzero coefficients.
4. Sense clustering: Three atoms (bird, flying, wildlife) cluster together. Three atoms (construction, machine, lifting) cluster together. Two distinct sense groups.
5. Sense vectors: Sum the atoms within each cluster to get v_bird and v_machine.
6. Verification: cos(v_bird, "heron") = 0.78 (high — correct sense). cos(v_machine, "excavator") = 0.72 (high — correct sense). cos(v_{crane-original}, "heron") = 0.42 (lower — mixed signal).

The decomposition correctly separates the two unrelated senses, and the recovered sense vectors are much more useful for sense-specific tasks.

Reading the visualization

In the explorer above, each scenario shows:

• Open circles: The TRUE sense vectors — where each sense "lives" in embedding space. These are what we want to recover.
• White filled dot: The COMBINED word vector — the weighted superposition of sense vectors. This is what word2vec gives us.
• Colored filled dots (after recovery): The RECOVERED sense vectors from sparse coding. If recovery works, these should land on or near the open circles.
• Red dashed lines: The error between recovered and true senses. Shorter = better recovery.
• Gray dots (when atoms visible): The dictionary atoms used by sparse coding.

When does recovery fail?

The paper identifies three failure modes:

1. Correlated senses: If two senses are semantically similar (e.g., "run" meaning "jog" vs. "sprint"), their sense vectors point in nearly the same direction. The combined vector is barely different from either sense, and there's nothing to decompose.
2. Rare senses: If one sense appears in only 1% of occurrences, its contribution to the combined vector is tiny. It's drowned out by noise and nearly impossible to recover.
3. Many senses: As the number of senses grows, the superposition becomes crowded. With k senses in d dimensions, reliable recovery requires d >> k (which is usually satisfied — 300 >> 5).

Recovery quality by word type

The paper reports recovery quality across different categories of polysemous words:

The best recovery happens for homonyms — words whose senses are completely unrelated. This makes sense: unrelated senses have orthogonal vectors, making them easy to separate. The hardest cases are related senses (where the vectors point in similar directions) and rare senses (where the signal is weak).

python
# Evaluate recovery quality for a polysemous word
import numpy as np

def recovery_quality(v_word, true_senses, recovered_senses):
    """R² between true and recovered sense vectors."""
    # Match recovered to true senses (greedy by cosine sim)
    matched = []
    remaining = list(range(len(true_senses)))
    for r in recovered_senses:
        sims = [np.dot(r, true_senses[j]) / (
            np.linalg.norm(r) * np.linalg.norm(true_senses[j]))
            for j in remaining]
        best = remaining[np.argmax(sims)]
        matched.append((r, true_senses[best]))
        remaining.remove(best)

    # Compute average cosine similarity
    avg_cos = np.mean([np.dot(r, t) / (
        np.linalg.norm(r) * np.linalg.norm(t))
        for r, t in matched])
    return avg_cos  # Higher = better recovery

Chapter 7: Connections

What this paper built on

Word2vec (Mikolov et al., 2013): The embedding algorithm whose vectors this paper analyzes. Word2vec trains by predicting co-occurring words using shallow neural networks (skip-gram or CBOW).

GloVe (Pennington et al., 2014): An alternative embedding algorithm that explicitly factorizes the co-occurrence matrix. The superposition result applies to both word2vec and GloVe, because both algorithms implicitly perform the same matrix factorization — the only difference is in how they approximate the optimization. This universality is a strong point: the superposition property is a consequence of the co-occurrence structure of language, not the details of the training algorithm.

A latent variable model for NLP (Arora et al., 2016): The random walk discourse model from the same group. This earlier paper introduced the generative model that makes the superposition proof possible. The polysemy paper is essentially a corollary of the discourse model. Without the random walk framework, there would be no principled way to connect the generative process of text to the structure of word vectors — and no way to prove that superposition arises naturally from co-occurrence statistics.

The Johnson-Lindenstrauss lemma (1984): The classical result that random projections approximately preserve distances in high dimensions. This is the mathematical foundation for why random vectors are nearly orthogonal — and therefore why the linear approximation in the superposition proof works. The lemma states that n points in high dimensions can be projected to d = O(log n / ε²) dimensions with all pairwise distances preserved up to a (1 ± ε) factor. For word embeddings, this means d = 300 dimensions can preserve distances between up to n = exp(300) ≈ 10¹³⁰ points — far more than any vocabulary.

Sparse coding (Olshausen & Field, 1996): The computational tool for recovering sense vectors. Originally developed for understanding V1 neurons in visual cortex — neurons respond to oriented edges, which form a sparse, overcomplete representation of visual scenes. The analogy is fitting and deep: both brains and word embeddings seem to use sparse superposition to pack multiple meanings into limited capacity.

Sememes in computational linguistics (Dong & Dong, 2003): The HowNet project in Chinese NLP manually annotated sememes (atomic semantic units) for over 100K Chinese and English words. This paper provides a computational analog — the sparse coding atoms correspond to automatically discovered sememes, bypassing the need for manual annotation.

What this paper enabled

Superposition hypothesis in neural networks (Elhage et al., 2022): The idea that neural network activations represent many features simultaneously through superposition is a direct descendant of this work. The mechanisms paper at Anthropic explicitly cites the analogy: neurons represent features as sparse linear combinations, just as word vectors represent senses as sparse linear combinations.

Sparse autoencoders for interpretability (Cunningham et al., 2023; Bricken et al., 2023): The use of sparse coding to extract interpretable features from neural network activations is methodologically identical to extracting sense vectors from word embeddings. Same math, different scale.

Contextualized embeddings (ELMo, BERT): These models produce different vectors for each occurrence of a word, effectively solving the polysemy problem by construction. In BERT, the vector for "bank" in "river bank" is completely different from "bank" in "bank account." This is the architectural solution to polysemy — instead of storing a superposition and recovering senses, generate sense-specific vectors on the fly using context.

But understanding WHY static embeddings superpose senses illuminates what contextual models are learning to separate. BERT's self-attention mechanism can be viewed as a dynamic sense disambiguation system: it reads the context, determines which sense is active, and generates a vector that matches that specific sense. The "sense vectors" that Arora et al. recover from word2vec are approximations to what BERT computes dynamically for each occurrence.

Polysemous embeddings (Athiwaratkun & Wilson, 2017): Instead of point vectors, represent each word as a Gaussian mixture in embedding space — one Gaussian per sense. This is an explicit probabilistic version of the linear superposition. The connection is direct: the mixture component means are the sense vectors, and the mixture weights are the sense frequencies. Arora et al.'s result explains WHY this probabilistic model works — because the point-estimate word vector already encodes the mixture structure linearly.

Word sense induction (Reisinger & Mooney, 2010): Early work on automatically discovering word senses from corpus statistics. Unlike Arora et al., this approach requires retraining embeddings with a sense-clustering step integrated into the training loop. The sparse coding approach is post-hoc — it works on any pre-trained vectors.

Implications for modern models

The superposition result has profound implications beyond word vectors. If meanings superpose linearly in 300-dimensional word2vec, what about the 12,288-dimensional hidden states of GPT-4? The same mathematical structure — high dimensions creating near-orthogonality, enabling many features to coexist in one vector — likely operates at massive scale in modern transformers. The difference is that transformers learn to dynamically compose and decompose these superpositions through attention, while word2vec just stores the static average.

The numbers are striking. In 300 dimensions, you can pack roughly d²/2 ≈ 45,000 nearly-orthogonal directions (this is a consequence of the Johnson-Lindenstrauss lemma). That's more than enough for a vocabulary of 50K words. In 12,288 dimensions, you can fit over 75 million nearly-orthogonal features. This explains why modern transformers can represent such complex, multifaceted concepts — the representational capacity grows quadratically with dimension.

Implementation: sense-aware similarity

Here's a practical application. Given a polysemous word and a context sentence, you can compute which sense is active by projecting the context vector onto each recovered sense direction:

python
def disambiguate(word, context_words, sense_vecs, embeddings):
    """Which sense of `word` is active given context?"""
    # Average context word vectors
    ctx_vec = np.mean([embeddings[w] for w in context_words
                       if w in embeddings], axis=0)

    # Project onto each sense direction
    scores = []
    for sv in sense_vecs:
        cos = np.dot(ctx_vec, sv) / (
            np.linalg.norm(ctx_vec) * np.linalg.norm(sv))
        scores.append(cos)

    return np.argmax(scores)  # index of best-matching sense

# Example: "I deposited money at the bank"
# context = ["deposited", "money"]
# Returns sense 0 (financial) with high confidence

Timeline of related work

The intellectual line from "word senses superpose linearly" to "neural network features superpose linearly" is direct and unbroken. The tools changed — from sparse coding on 300-dimensional word vectors to sparse autoencoders on million-dimensional transformer activations — but the mathematical principle is identical.

Why this paper matters for AI safety

The connection to interpretability is not just intellectual. If features in neural networks are stored as superpositions (as the evidence strongly suggests), then understanding the math of superposition is critical for AI safety. Arora et al.'s work provides foundational theory for:

• When features are recoverable: The paper gives conditions (high dimensionality, sparsity, sense separation) under which individual components can be extracted from a superposition. These same conditions determine when mechanistic interpretability can succeed in identifying individual features in neural networks.
• When they're NOT recoverable: The failure modes (correlated senses, rare senses, too many senses) map directly to failure modes of sparse autoencoders. Features that are rare or correlated with other features are harder to isolate.
• The role of dimensionality: The paper proves that superposition only works in high dimensions. This has implications for model design — wider models can pack more features via superposition, which is both useful (more capacity) and concerning (harder to interpret).

Limitations

Several limitations of the paper deserve mention:

• Static embeddings only: The theory applies to word2vec/GloVe, not to contextualized models like BERT or GPT. Contextualized models handle polysemy through multiple layers of nonlinear processing, not static superposition.
• Small number of senses: The superposition formula works well for 2-5 senses, but accuracy degrades for words with 10+ senses (like "set" or "run"). In practice, most applications don't need to recover all 30 senses of "run" — the top 3-4 suffice.
• Assumes approximate isotropy: The proof relies on word vectors being roughly isotropic (uniformly distributed in direction). Some recent work shows word2vec vectors have a systematic bias toward certain directions, which could affect the approximation quality.
• Evaluation difficulty: Evaluating sense recovery is hard because there's no universal ground truth for what the "senses" of a word are. WordNet provides one decomposition, but it's often too fine-grained.
• No dynamic disambiguation: The sparse coding recovers the senses statically — it tells you what the senses ARE, but not which sense is intended in a specific sentence. For sentence-level disambiguation, you still need contextual features (which is what models like BERT provide).

Recommended reading order

To fully understand this paper and its context, read in this order:

1. Mikolov et al. (2013) — word2vec: Understand what word vectors are and why they encode meaning.
2. Levy & Goldberg (2014) — word2vec as matrix factorization: Understand why word2vec is implicitly doing linear algebra.
3. Arora et al. (2016) — random walk model: Understand the generative model that produces word vectors with their observed structure.
4. This paper (2018) — superposition and recovery: The polysemy result follows naturally from the previous three.
5. Elhage et al. (2022) — superposition in neural nets: See how the same principle applies to modern deep networks.

What would have happened differently?

If Arora et al.'s paper had been published in 2022 instead of 2018, it would have been framed very differently — as a foundational result for the superposition hypothesis in mechanistic interpretability. The mathematical tools (sparse coding, dictionary learning, overcomplete representations) are identical to what the interpretability community independently developed. The delay in making this connection cost the field several years of potential progress.

The lesson: theoretical work in one subfield (word embeddings) can have unexpected applications in another (neural network interpretability). The math of superposition is universal — it doesn't care whether you're decomposing a 300-dimensional word vector or a 12,288-dimensional transformer hidden state. The principles are the same. The tools are the same (sparse coding / sparse autoencoders). Even the failure modes are the same (correlated features, rare features, overcrowded representations).

Key numbers to remember

Cheat sheet