Inkling, Dissected — Engineermaxxing

Property	Value	Where we dissect it
Total parameters	975 billion — a Mixture-of-Experts transformer	Ch 1, 2
Active per token	41 billion — ~4.2% of the model wakes up per token	Ch 1, 2
Context window	up to 1 million tokens	Ch 3
Training data	45 trillion tokens of text, images, audio, and video	Ch 8
Optimizer	Muon for matrix weights, Adam for the rest	Ch 4
Post-training	small SFT bootstrap, then RL at 30M+ rollouts	Ch 5
Reasoning	controllable thinking effort, 0.2 to 0.99	Ch 6
Hardware	trained on NVIDIA GB300 NVL72 systems	Ch 4
Weights	Hugging Face (`thinkingmachines/inkling`), incl. an NVFP4 checkpoint	Ch 10

02Anatomy of one MoE layer

Replace the transformer's one big MLP with 258 small ones and a receptionist. That's the whole trick — everything else is keeping the receptionist honest.

In a dense transformer, every layer has one MLP (also called the feed-forward block): two matrix multiplies with a nonlinearity between them, applied identically to every token. It is where most parameters live and — the standard reading — where most of the model's stored knowledge lives.

Inkling's MoE layers replace that single MLP with:

256 routed experts — small independent MLPs. Any given token will use exactly 6 of them.
2 shared experts — MLPs that process every token, no routing involved.
A router — a tiny linear layer that looks at the token's hidden state and scores all 256 experts: "how useful would each of you be for this token?"

The layer's output is a weighted sum: the 2 shared experts' outputs plus the 6 chosen experts' outputs, each scaled by a gate weight from one jointly normalized score vector — all 8 contributing experts share a single normalization. Two design questions hide in that sentence, and they are the two ideas of this chapter: how do you score (sigmoid, not softmax), and how do you keep the choosing fair (a bias, not a loss).

Why shared experts exist at all

If routing is so great, why keep 2 experts that everyone visits? Because some computation is common: basic syntax, token-level bookkeeping, the boring glue of language. Without shared experts, every routed expert must relearn that glue — 256 redundant copies of "how commas work" — wasting exactly the capacity the MoE was built to win. Give the common work a guaranteed home and the routed experts are free to specialize. (This is the DeepSeek lineage of MoE design, which Inkling explicitly follows; the same paper introduced the balancing trick we meet below.)

Mental model: the layer is a hospital. The two shared experts are triage — everyone passes through. The 256 routed experts are specialists, and the router is the front desk deciding which 6 specialists each patient sees. The chapter's problem, in hospital terms: the front desk keeps sending everyone to the same famous surgeon, the surgeon has a fixed number of slots per day, and patients who overflow get no specialist at all.

02bSigmoid, not softmax

A one-word change in the router with real consequences: experts stop competing for probability mass.

The classic MoE router (Shazeer 2017, Switch, Mixtral) computes a softmax over expert logits: 256 numbers that must sum to 1, of which the top-k survive. Inkling — following DeepSeek-V3 — scores each expert with an independent sigmoid instead:

Inkling's router, one token $$ s_i = \sigma(\mathbf{x}^\top \mathbf{w}_i) \qquad \text{keep top-}6\text{ by } (s_i + b_i) \qquad g_i = \frac{s_i}{\sum_{j \in \text{kept} \,\cup\, \text{shared}} s_j} $$

$\mathbf{x}$ — the token's hidden state entering the layer.
$\mathbf{w}_i$ — expert $i$'s routing vector: a learned direction meaning "tokens like this are my specialty."
$s_i = \sigma(\cdot)$ — the affinity: an independent 0-to-1 score. One expert scoring high does not force another to score low — that is the break from softmax.
$b_i$ — the balancing bias, the star of this chapter. Note precisely where it appears: inside the top-6 selection, and nowhere else.
$g_i$ — the gate: the 6 chosen routed experts' and 2 shared experts' affinities, jointly renormalized — all 8 contributing experts share one normalization. The bias never touches this — the mixture is weighted by honest scores.

Why does independence matter? Two practical reasons. First, gradients stop fighting: under softmax, pushing one expert's score up mechanically pushes every other expert's score down, so 256 experts play tug-of-war through one normalizer. Sigmoid scores train independently. Second, the scores mean something absolute: $s_i = 0.9$ says "strong match" regardless of what the other 255 scores are, which makes them usable as gate weights directly — and makes the balancing story below much cleaner.

the whole router, numpy

def moe_layer(x, all_experts, W_r, b):             # x: [d] one token
    s   = sigmoid(x @ W_r.T)                      # [258] affinities: 256 routed + 2 shared
    top = argtop6(s[:256] + b)                    # bias steers WHO is picked (routed only)...
    idx = concat([top, [256, 257]])                # ...plus the 2 always-on shared experts
    g   = s[idx] / s[idx].sum()                   # ONE joint normalization over all 8
    out = 0
    for g_k, e in zip(g, all_experts[idx]):
        out += g_k * e(x)                         # 8 contributors, honest weights
    return out

02cA routing decision, by hand

Four experts, top-2, every number visible. Watch what the bias does — and what it refuses to do.

Shrink the layer to 4 experts choosing top-2 so we can track every digit. (A routed-only toy: we leave out the 2 shared experts' share of the joint normalization to keep the arithmetic on one hand — the bias mechanics are identical either way.) A token arrives whose hidden state gives router logits $z = [0.5,\; 0.45,\; 0.44,\; -0.5]$. Sigmoids:

Step 1 — affinities. $\sigma(0.5)=0.622$, $\sigma(0.45)=0.611$, $\sigma(0.44)=0.608$, $\sigma(-0.5)=0.378$. Experts 1, 2, 3 are nearly tied — this token would be happy with any pair of them; expert 4 is a poor match.

Step 2 — selection, no bias. Top-2 by affinity: experts 1 and 2. Gates: $g_1 = 0.622/(0.622+0.611) = 0.504$, $g_2 = 0.496$. Nearly a 50/50 blend.

Step 3 — now suppose expert 1 has been overloaded all morning, so the balancer has drifted its bias to $b_1 = -0.02$ (and, say, $b_3 = +0.001$ from mild underload). Selection scores: $[0.602,\; 0.611,\; 0.609,\; 0.379]$. Top-2 is now experts 2 and 3 — the tie broke the other way.

Step 4 — gates for the new pair, computed from the raw affinities, not the biased scores: $g_2 = 0.611/(0.611+0.608) = 0.501$, $g_3 = 0.499$. The output quality barely moved — 0.608 was almost as good a match as 0.622 — but one unit of load just migrated off the hot expert.

That is the entire mechanism, and its elegance is worth saying out loud: the bias only ever flips near-ties. A token that strongly needs expert 1 ($s_1 = 0.95$, everyone else 0.5) sails through a $-0.02$ bias untouched. Tokens that are roughly indifferent get gently redistributed. Load balancing happens exactly where it is cheapest, and the mixing weights — what actually multiplies the experts' outputs — never learn that any of this happened.

02dThe rich-get-richer problem

Why routers collapse without help, and what it costs when they do.

Real token streams are lumpy: code, English, numbers, and boilerplate do not arrive in equal measure, so some experts' specialties are simply in higher demand. Left alone, this compounds. A popular expert sees more tokens, so it trains more, so it gets better, so its affinities rise, so it is selected even more — the rich-get-richer loop. Meanwhile unpopular experts starve: barely trained, barely useful, dead weight in your 975B.

The acute symptom is worse than the chronic one. Every expert has a capacity — a maximum number of tokens per batch it can process, set by memory and the all-to-all communication schedule (Chapter 1's logistics tax). When a hot expert's queue exceeds capacity, the overflow tokens are dropped: they simply do not get that expert's contribution, and the layer output is computed from whatever remains. Dropped tokens are silent quality damage — nothing crashes, the model just gets a little dumber on exactly the popular topics.

The classic fix (Switch Transformer and its descendants) is an auxiliary load-balancing loss: add a term to the training objective that punishes uneven expert usage. It works — but look at what it touches. The aux loss's gradient flows into the router's weights, the same weights that produce the affinities that become gate values. You are deliberately corrupting the router's opinion of which expert is best, in order to fix a logistics problem. The two objectives — "route to the best expert" and "route evenly" — fight inside one set of parameters, and the tug-of-war costs model quality.

02eThe aux-loss-free bias

DeepSeek-V3's answer, adopted wholesale by Inkling: separate the two jobs. Selection gets a knob; the gates stay honest.

The balancing rule (after each batch) $$ b_i \leftarrow b_i + \gamma \cdot \mathrm{sign}\!\left(\bar{L} - L_i\right) $$

$L_i$ — expert $i$'s load this batch: how many tokens selected it.
$\bar{L}$ — the ideal load: total selections divided by 256.
$\mathrm{sign}(\cdot)$ — overloaded experts get their bias lowered by $\gamma$; underloaded experts get it raised. No gradients, no loss term — a plain feedback controller bolted onto the selection step.
$\gamma$ — a small step (the lab below uses 0.01 on affinities that live in 0–1). It accumulates: an expert overloaded for 30 straight batches carries $b_i = -0.3$.

Three properties make this better than it looks:

It cannot corrupt the mixture. The bias appears in $\text{top-}6(s_i + b_i)$ and nowhere else. Gate weights are computed from raw $s_i$. However hard the controller pushes, the quality of the blend for whichever experts are chosen is untouched — the failure mode is bounded by construction.
It acts on near-ties first. As the hand example showed, a bias of $\pm0.02$ only flips decisions where the affinities were within 0.02 anyway — the tokens that had the least to lose from being rerouted.
It fails gently. Crank $\gamma$ ten times too high and selection gets a bit sloppy — measurably, in the lab below, a few percent of drops return. Crank an aux-style penalty ten times too high and the whole routing system oscillates. When you are training one model for months on GB300 clusters, "how does this fail when mistuned" is not an academic question.

The design pattern generalizes far beyond MoE and is worth stealing: when one signal must serve two masters, split the signal. Selection needed fairness; mixing needed honesty. The bias gives each job its own channel instead of making one number lie to serve both.

02fInteractive: nudge the router

Eight experts, one token, and a bias slider on the hottest expert. Find the flip point — and watch what the gates do.

The bars are one token's affinities $s_i$ (amber). The outline shows the selection score $s_i + b_i$ as you drag expert 1's bias. Ticks mark the top-3 selection; the readout prints the gates — computed, as always, from the raw affinities of whoever is selected. Press new token to resample.

bias on expert 1 0.00

affinity s (sets the gates) selection score s + b selected (top-3)

Two things to try. First, drag the bias down slowly and notice selection flips happen one near-tie at a time — there is no cliff. Second, watch the printed gates at the moment of a flip: the newcomer enters with nearly the same gate weight the evicted expert had, because near-ties are precisely the tokens for which the substitution is cheap.

02g⚖ Run the experiment: the Router Balance Lab

We built the whole routing floor in 80 lines of numpy — lumpy tokens, capacities, drops — and raced three balancing strategies. Run it yourself, then break it.

⚖ The Router Balance Lab ✓ verified run

64 experts, top-6, Zipf-lumpy tokens, hard capacities — no balancing vs. score penalty vs. selection-only bias.

Hypothesis. Without balancing, hot experts overflow capacity and drop tokens. A score penalty and a selection-only bias can both restore balance — but the penalty bends the same scores that weight the output mixture, so it should degrade quality or destabilize when pushed, while the bias cannot touch the gates by construction.

Our verified run: no balancing → MaxVio ≈ 1.2 (hottest expert at 2.2× its fair share), 7.1% of expert slots dropped, layer quality 93.5%. Score penalty (α=0.6) → MaxVio 0.25, 0% dropped, 99.2%. Selection-only bias (γ=0.01) → MaxVio 0.26, 0% dropped, 99.1%. The two fixes tie — until you mistune them: α=3.0 thrashes to 39.9% dropped and 60.4% quality, while γ=0.05 merely slips to 5.2% dropped, 94.5%. The bias fails gently; the penalty fails catastrophically.

Lab notebook — what our run found

Setup. 64 experts (scaled from Inkling's 256 so it runs in seconds), embedding dim 32, top-6 routing, 2,048 tokens per batch, 100 routing steps. Tokens are drawn from a Zipf distribution over expert specialties — expert 0's home cluster is ~30× more common than expert 63's — then jittered, so affinities are realistic: lumpy with lots of near-ties. Every expert has a hard capacity of 1.25× the ideal load; overflow tokens are dropped exactly as in production MoE serving. We measure DeepSeek's MaxVio (hottest expert's demand over ideal, minus 1), the drop rate, and a layer quality score: how much true affinity the surviving weighted mixture captures versus an uncapacitated oracle.

Result 1 — doing nothing is expensive. Unbalanced, the hot experts sit at 2.2× fair share all run (MaxVio ≈ 1.2 never improves), 7.1% of expert slots go over capacity and get dropped, and quality sits at 93.5% — a persistent ~6.5% tax paid precisely on the most common token types.

Result 2 — at their best settings, the two fixes tie. Both the score penalty (α=0.6) and the bias (γ=0.01) drive MaxVio to ~0.25 and drops to zero, with quality ~99%. In a frozen-router sandbox, gate distortion from a well-tuned penalty is real but small.

Result 3 — the difference is the failure mode. At α=3.0 the penalty overshoots each step, the load EMA lags, and the system oscillates: MaxVio averages 2.7, 39.9% of slots drop, quality craters to 60%. The bias at 5× its good setting (γ=0.05) just gets slightly sloppy: 5.2% drops, 94.5% quality. A selection-only offset is structurally incapable of corrupting the mixture, so pushing it harder cannot create a new failure class.

What does NOT transfer. Our router is frozen — affinities never train. The production argument for aux-loss-free balancing has a second half we cannot show here: a true auxiliary loss injects a competing gradient into router training for the model's whole life, and that interference compounds in ways a routing sandbox cannot measure. Treat this lab as the selection-vs-gates story and the mistuning story, not the full training-dynamics story.

Things to try · ALPHA=3.0 — watch the score penalty thrash (40% drops in our run)
· GAMMA=0.05 — the bias mistuned 5×: barely worse
· CAP=1.0 — tight memory: which arm suffers first?
· K=2 — sparser routing: near-ties get rarer, does the bias still work?

02hConcept check

Chapter 2 · one question

Inkling's balancing bias $b_i$ is added to the router score. Why is it added only inside the top-6 selection, and not carried into the gate weights?

04Two optimizers, one model

Inkling's pretraining ran a hybrid: Muon for the large matrix weights, Adam for everything else. The split is the tell — it says matrices are different.

Nearly every parameter in a transformer is an entry in some matrix: attention projections, expert MLPs, embeddings. Adam — the default optimizer of the last decade — does not know this. It treats a $64 \times 32$ weight matrix as 2,048 unrelated scalars, each with its own little learning-rate adjustment based on its own gradient history. Whatever structure those 2,048 numbers have as a linear map — which directions of input space it stretches, which it ignores — is invisible to Adam by construction.

Muon (2024, Keller Jordan et al.; since scaled to frontier runs by multiple labs) is an optimizer built around the opposite premise: treat the matrix as a matrix. Its recipe per step, for each large weight matrix:

Accumulate momentum as usual
$M \leftarrow \mu M + G$, where $G$ is the gradient of that weight matrix. Nothing exotic yet — this is SGD-momentum's buffer, kept in matrix shape.
same information as SGD
Orthogonalize the buffer
Compute (approximately) $\mathrm{msign}(M) = UV^\top$, where $M = U\Sigma V^\top$ is the SVD (unpacked from zero in the next section). This keeps the directions of the update and throws away the magnitudes: every singular value becomes ~1.
the whole trick lives here
Step with the orthogonalized update
$W \leftarrow W - \eta \cdot \mathrm{msign}(M) \cdot \sqrt{\max(1, \tfrac{n_{\text{out}}}{n_{\text{in}}})}$ — the scale factor keeps updates consistent across matrix shapes.
equal push in every direction

Vectors, gains, and norm parameters — things with no meaningful "directions" to equalize — stay on Adam. Hence the hybrid.

04bWhat Adam cannot see

A momentum matrix is a to-do list of directions. The problem: a few loud directions drown out the rest.

One tool first, built from zero. A matrix is a linear map, and any linear map can be described completely by answering three questions: which input directions does it act on, which output directions does it send them to, and how strongly? The singular value decomposition (SVD), $M = U\Sigma V^\top$, is that answer written as three matrices: the columns $\mathbf{v}_i$ of $V$ are a set of perpendicular input directions, the columns $\mathbf{u}_i$ of $U$ are the perpendicular output directions they land on, and the diagonal entries $\sigma_i \ge 0$ of $\Sigma$ are the stretch factors. Picture a $2\times2$ matrix acting on the unit circle: the circle comes out as an ellipse, the ellipse's axes are $\mathbf{u}_1, \mathbf{u}_2$, their lengths are $\sigma_1, \sigma_2$, and $\mathbf{v}_1, \mathbf{v}_2$ are the circle directions that landed on them. Every matrix has an SVD — it is not a special property, just what "being a matrix" looks like once you sort its action by direction and strength.

Decompose any momentum buffer $M = U\Sigma V^\top$. Each pair $(\mathbf{u}_i, \mathbf{v}_i)$ is a direction of change — "rotate inputs like $\mathbf{v}_i$ toward outputs like $\mathbf{u}_i$" — and its singular value $\sigma_i$ says how loudly the recent gradients voted for it. In real training the spectrum is brutally skewed: a handful of directions carry large $\sigma$ (usually driven by common, high-frequency features of the data) while hundreds of directions that matter — rare words, unusual code patterns, the tails — sit at tiny $\sigma$.

Raw SGD applies the buffer as-is: the loud directions move fast, the quiet ones barely move. Adam helps only where the disparity happens to line up with individual coordinates — its per-entry rescaling cannot rotate. If the loud and quiet directions are oblique mixtures of coordinates (in real networks they always are), Adam's diagonal view sees roughly uniform per-entry noise and fixes nothing. The ⚖ lab below makes this concrete: give the data a 100× anisotropy along the axes and Adam keeps up fine; rotate that same anisotropy off-axis and Adam loses its grip while Muon does not care — orthogonalization never had a preferred basis in the first place.

Slogan for the mechanism: Muon spends the same update budget in every direction the gradients voted for at all. Loud directions get capped, quiet-but-real directions get amplified to the same strength. The bet is that a direction's presence in the momentum is more informative than its magnitude — magnitudes mostly encode feature frequency, not feature importance.

04cNewton-Schulz, by hand

Nobody computes an SVD for a 956-billion-parameter model every step. Muon fakes it with five matrix multiplies.

The Newton-Schulz iteration exploits a lovely fact: if you repeatedly apply a well-chosen odd polynomial $f(x) = ax + bx^3 + cx^5$ to a matrix (as $f(X) = aX + b\,XX^\top X + c\,(XX^\top)^2 X$), the polynomial acts independently on each singular value while leaving the singular directions $U, V$ untouched. Choose coefficients that push values toward 1, iterate a few times, and you have $\approx UV^\top$ using only matmuls — the one operation GPUs love. Muon uses Jordan's tuned coefficients $(a, b, c) = (3.4445,\, -4.7750,\, 2.0315)$ and just five iterations.

Follow two singular values through all five rounds. Take our chapter's running matrix $M = \begin{bmatrix} 3 & 0 \\ 0 & 0.1 \end{bmatrix}$: singular values 3 and 0.1, so the loud direction moves 30× faster than the quiet one. Newton-Schulz first normalizes by the Frobenius norm ($\|M\|_F = \sqrt{9.01} \approx 3.002$), giving $\sigma = (0.999,\; 0.033)$. Now apply $f(x) = 3.4445x - 4.7750x^3 + 2.0315x^5$ five times:

iteration	σ₁ (was 30× louder)	σ₂	ratio
start	0.999	0.033	30×
1	0.702	0.114	6.2×
2	1.114	0.386	2.9×
3	0.720	1.072	0.67×
4	1.089	0.686	1.6×
5	0.696	1.130	0.62×

Both values now live in a band around 1 — a 30× disparity flattened to under 2×. Note the honest fine print: Jordan's coefficients are tuned for speed on tiny values, not exact convergence, so the spectrum oscillates inside roughly $[0.65, 1.15]$ rather than pinning to 1.000. For an optimizer update, "everything is O(1)" is all that matters — and tiny values get pulled up fast, which is the expensive part done cheaply.

newton-schulz, the real thing (from the lab below)

def newton_schulz(Gm, iters=5):
    a, b, c = 3.4445, -4.7750, 2.0315
    X = Gm / (np.linalg.norm(Gm) + 1e-7)   # normalize: spectrum into (0, 1]
    tp = X.shape[0] > X.shape[1]
    if tp: X = X.T                           # work on the fat orientation
    for _ in range(iters):
        A = X @ X.T                          # the polynomial acts on the
        X = a * X + (b * A + c * A @ A) @ X  # singular values only
    return X.T if tp else X               # ≈ U Vᵀ : directions, no magnitudes

Count the cost: five iterations × three matmuls on a matrix the model already holds — a few percent overhead per step, purchasable at datacenter scale. That is the engineering reason Muon made it out of blog posts and into a 975B production run on GB300 NVL72 systems.

04dInteractive: flattening the spectrum

Twelve singular values, five Newton-Schulz rounds. Drag the slider and watch the quiet directions get their voice back.

The bars are a momentum matrix's singular spectrum — skewed the way real buffers are (a few loud, many quiet; the smallest starts 50× below the largest). The slider applies $f(x) = 3.4445x - 4.775x^3 + 2.0315x^5$ the given number of times to every value. The dashed line marks 1.0.

NS iterations 0

singular value after k iterations target σ = 1

Watch iteration 1 especially: the tiny values roughly triple (for small $x$, $f(x) \approx 3.44x$) while the big ones get knocked down (at $x=1$, $f(1) = 0.701$). The polynomial is a compressor pedal for spectra — quiet up, loud down, everything toward the band.

04e⚖ Run the experiment: the Muon Race Lab

SGD vs Adam vs Muon on the same teacher-student MLP — with the anisotropy rotated off the axes, where Adam can't follow.

⚖ The Muon Race Lab ✓ verified run

A 100× spread in feature scales, hidden behind a random rotation — then three optimizers race from the same init on the same data.

Hypothesis. On anisotropic data whose strong/weak directions do not align with coordinate axes, Muon's basis-free orthogonalization should beat Adam's per-coordinate scaling, which should beat SGD. Setting the rotation to identity should erase most of Adam's deficit — per-coordinate scaling is exactly the right tool when the anisotropy is axis-aligned.

Our verified run (loss at step 50 / 200 / 400): SGD .0057 / .0025 / .0019 · Adam .0050 / .0015 / .0006 · Muon .0069 / .0009 / .0003. Muon starts slowest — orthogonalization initially amplifies directions that haven't earned it — then overtakes by ~step 150 and finishes 2× ahead of Adam, 6× ahead of SGD.

Lab notebook — what our run found

Setup. Teacher-student regression: a fixed random 2-layer MLP (32 → 64 → 16) generates targets; a student of the same shape learns from fresh batches of 256. Inputs are anisotropic — feature scales span 1 down to 10²⁻²… that is, logspace(0, −2), a 100× condition — and then multiplied by a random rotation so the strong and weak directions are oblique mixtures of coordinates, the way real features always are. Each optimizer gets a reasonably tuned lr (SGD 0.2, Adam 0.01, Muon 0.02), identical data order, identical init.

Result. Muon's curve crosses Adam's around step 150 and keeps pulling away: .0009 vs .0015 at step 200, .0003 vs .0006 at step 400. The mechanism is visible in the crossover itself — early on, flattening the spectrum amplifies directions with little evidence (slightly hurting), but as momentum accumulates real signal about the weak-feature directions, equal-strength updates let the student learn them at full speed while Adam is still taking baby steps along them.

The control that proves the mechanism. Set ROT = np.eye(D_IN) and rerun: the anisotropy now lies along coordinate axes, Adam's per-entry variance estimates see it perfectly, and the Adam-Muon gap nearly closes. Adam's blind spot is not anisotropy — it is rotated anisotropy. Muon never cared either way, because orthogonalization has no preferred basis.

What does NOT transfer. This is a ~3,000-parameter MLP doing regression, not a 975B transformer doing language. The honest production claims for Muon are more modest and more valuable: reports from multiple labs of roughly 1.3–2× sample-efficiency gains at equal compute, robustness to learning-rate choice (try lr=0.005 and 0.08 below — the spread is startlingly small), and the fact that the per-step overhead — five matmul-only iterations — is negligible at scale. Our toy shows the mechanism; the production wins are the reason Inkling bet a frontier pretraining run on it.

Things to try · ROT = np.eye(D_IN) — axis-aligned anisotropy: watch Adam catch up
· COND = 0 — isotropic data: the race gets much closer
· iters=1 in newton_schulz — half-orthogonalized: still helps?
· lr=0.005 then lr=0.08 for muon — feel how lr-forgiving it is

04fWeight decay ∝ lr²

One sentence in the release note — "we couple weight decay strength to the square of the learning rate" — hides a tidy piece of dynamics.

Weight decay shrinks every weight slightly each step ($W \leftarrow W - \eta\lambda W$), fighting the growth that updates cause. These two forces settle into an equilibrium, and you can find it on the back of an envelope. Per step, the squared norm changes by approximately:

Norm dynamics under decay + updates $$ \Delta \|W\|^2 \approx -\,2\eta\lambda \|W\|^2 + \eta^2 \|u\|^2 \qquad \Rightarrow \qquad \|W\|_{\text{eq}} = \|u\|\sqrt{\frac{\eta}{2\lambda}} $$

$-2\eta\lambda\|W\|^2$ — decay shrinks, proportionally to how big the weights already are.
$\eta^2\|u\|^2$ — updates grow the norm; with Muon, $\|u\|$ is essentially constant (orthogonalized updates have fixed scale), which is what makes this analysis unusually clean here.
$\|W\|_{\text{eq}}$ — where shrink and growth cancel: the norm the weights drift toward. (This equilibrium view is developed in the rotational-dynamics literature the release cites — Kosson et al. 2023; see also Defazio 2025.)

Now watch what happens across a schedule. Learning rates decay during training; suppose $\eta$ falls 10×. With constant $\lambda$, the equilibrium norm $\propto \sqrt{\eta}$ falls ~3.2× — late training spends its precious low-lr steps dragging every weight matrix toward a new, smaller equilibrium instead of refining the function. Couple $\lambda = c\,\eta$ and the equilibrium freezes but the decay stays aggressive late. Inkling's choice, $\lambda = c\,\eta^2$, goes one step further: the equilibrium target loosens as $\eta$ falls, while the relaxation time toward it ($\sim 1/2\eta\lambda = 1/2c\eta^3$ steps) explodes — so late in training the norms are effectively frozen wherever they are: no dragging, no drift, stable weight magnitudes no matter how long the horizon runs. Which is precisely the phrase in the release note: "stable weight sizes across training horizons." When your training run is measured in months and your schedule in trillions of tokens, "the norms stop moving" is a gift to every downstream consumer of the checkpoint — including the RL phase coming in the next chapter.

04gConcept check

Chapter 4 · one question

In our lab, Adam nearly matches Muon when the 100× anisotropy is axis-aligned, but loses when the same anisotropy is rotated. What does this reveal about the two optimizers?

Phase	What happened	Scale
SFT bootstrap	Imitation on synthetic conversations generated by open-weights models, including Kimi K2.5	"a small fraction of compute"
Large-scale RL	Graded rollouts on synthetic and human-created environments, two continuous runs	30+ million rollouts
The receipt	Aggregate reasoning-evaluation score	0.264 → 0.356 (SFT init → released checkpoint)

Demo	What happened	The doubt it targets
One-prompt web app	A functional job-application app from a single prompt — with an embedded AI assistant that drives the browser from natural language	"Open models can't do modern agentic scaffolding."
Design Arena	Blinded human ranking of generated web apps: 1257, among the strongest open-weights entries (Claude Sonnet 5, for scale: 1333)	"Open model output looks like a programmer's idea of design."
Nine-page PDF	"Breakfast Around the World" — a food-and-travel journal with editorial layouts, accurate cultural details, cohesive design across pages	"Long-form multi-artifact coherence falls apart."
Multiplayer snake	A game refined through 40 feedback cycles: server-authoritative real-time simulation, browser clients, bot opponents	"Open models can't sustain iterative engineering." (40 rounds is an endurance claim, not a one-shot claim.)
Self-fine-tuning	Inkling retrained itself into a lipogram model — the rest of this chapter	"A model is a static artifact."

Property	How it was trained / tested	The receipt
Calibration	RL against proper scoring rules on resolved real-world questions	ForecastBench Brier index 61.1 ± 0.79 (63.7 ± 0.82 with search); Prophet Arena Brier 0.1617
Instruction following	A dual grader: rubric checklist + agentic claims verification	IFBench 79.8 (and Small: 83.4)
Censorship resistance	Evaluated on Cognition's Propaganda & Censorship Eval	“Strong patterns of censorship non-compliance” — qualitative; no score published
Safety refusals	Refusal benchmarks: FORTRESS (attack + benign prompts) and StrongREJECT	FORTRESS adversarial 78.0 vs Nemotron 3 Ultra's 77.6 (best peer); benign 95.9; StrongREJECT 98.6

Benchmark	Inkling	Reference
Humanity's Last Exam (text only)	29.7%	GLM 5.2 (best open): 40.1% · Claude Fable 5: 53.3%
HLE (with tools)	46.0%	GLM 5.2 (best open): 54.7% · Claude Fable 5: 64.5%
AIME 2026	97.1%	GPT 5.6 Sol: 99.9%
GPQA Diamond	87.2%	Gemini 3.1 Pro: 94.1%

Benchmark	Inkling	Reference
SWEBench Verified	77.6%	Claude Fable 5: 95.0%
Terminal Bench 2.1	63.8%	GPT 5.6 Sol: 89.5%
SWEBench ProPublic	54.3%	Claude Fable 5: 80.0%

Benchmark	Inkling	Reference
GDPVal-AA v2	1238	Claude Fable 5: 1760
MCP Atlas	74.1%	Claude Fable 5: 83.3%
Tau 3 Banking	23.7%	GPT 5.6 Sol: 33.0%
BrowseComp (w/ ctx management)	77.1%	GPT 5.6 Sol: 90.8%

Benchmark	Inkling	Reference
SimpleQA Verified	43.9%	Gemini 3.1 Pro: 77.3%
AA Omniscience	2.1	Claude Fable 5: 40.0

Benchmark	Inkling	Reference
IFBench	79.8%	Nemotron 3 Ultra: 81.4%
Global-MMLU-Lite	88.7%	Claude Fable 5: 93.3%

Domain	Benchmark	Score
Vision	MMMU Pro Standard 10	73.5%
Vision	Charxiv RQ	78.1% (82.0% with Python)
Audio	Audio MC	56.6%
Audio	MMAU	77.2%
Audio	VoiceBench	91.4%

Benchmark	Inkling	Reference
ForecastBench Brier index (no search)	61.1 ± 0.79	—
ForecastBench (with search)	63.7 ± 0.82	—
Prophet Arena Brier score	0.1617	—
FORTRESS (adversarial)	78.0%	Nemotron 3 Ultra: 77.6%
FORTRESS (benign)	95.9%	Nemotron 3 Ultra: 90.5%
StrongREJECT	98.6%	Kimi K2.6: 99.8%

Benchmark	Inkling (975B/41B)	Inkling-Small (276B/12B)
HLE (text only)	29.7%	29.6%
HLE (with tools)	46.0%	46.6%
AIME 2026	97.1%	95.1%
Global-MMLU-Lite	88.7%	86.8%
GPQA Diamond	87.2%	88.3%
IFBench	79.8%	83.4%

00Concept constellation

00bHow to read this lesson

00cConcept index

The sparse bet

MoE routing

Aux-loss-free balancing

The 5:1 attention weave

Relative positions

Short convolutions

Muon

Weight decay ∝ lr²

SFT bootstrap

30M-rollout RL

Thinking effort

Self-fine-tuning

dMel audio

hMLP patches

Proper scoring rules

The dual grader

Inkling-Small

Tinker & the ecosystem

01What was released

01bThe sparse bet, by hand

The cost of a dense token

Where the 41B comes from

Why capacity and compute want to be different numbers

01cWhat sparsity does not buy

01dInteractive: the expert wall

01eWhy give it away

01fConcept check

02Anatomy of one MoE layer

Why shared experts exist at all

02bSigmoid, not softmax

02cA routing decision, by hand

02dThe rich-get-richer problem

02eThe aux-loss-free bias

02fInteractive: nudge the router

02g⚖ Run the experiment: the Router Balance Lab

02hConcept check

03The bill for a million tokens

03bThe 5:1 weave

03cKV-cache arithmetic, by hand

03dInteractive: the reach explorer

03eOffsets, not addresses

03fTwo small convolutions

03gConcept check

04Two optimizers, one model

Accumulate momentum as usual

Orthogonalize the buffer

Step with the orthogonalized update

04bWhat Adam cannot see

04cNewton-Schulz, by hand

04dInteractive: flattening the spectrum

04e⚖ Run the experiment: the Muon Race Lab

04fWeight decay ∝ lr²

04gConcept check

05The shape of the recipe

05bThe bootstrap: borrowed voices

05cThirty million rollouts

05dInteractive: the long climb

05eChains that compressed themselves

05fConcept check

06A dial, not a switch

06bFrontiers, not points

06cInteractive: the frontier explorer

06dWhat log-linear buys and forbids

06eConcept check

07The demo reel, read closely

07bThe self-improvement loop

Draft a training plan

Generate evaluation datasets

Execute fine-tuning via the Tinker API

Evaluate the result

Load the updated weights

07cInteractive: the loop, animated

07d⚖ Run the experiment: the Lipogram Lab

07eConcept check

08One stream, three senses

08bVision: 40×40 through an hMLP

08cAudio: dMel spectrograms