Griffin & Hawk

Chapter 1: The Linear Recurrence

We want RNN-style inference: one state, updated per token. The simplest possible update that still carries memory is a linear recurrence:

Read it out loud: the new state h_t is a fraction a of the old state plus a fraction b of the new input x_t. The number a is the recurrence weight — how much of the past you keep. The number b is the input weight — how much of the present you let in. In the real models h and x are vectors and a, b act element-by-element (a diagonal recurrence), but a single number teaches the whole idea.

The word that matters is linear. There is no nonlinear function wrapped around the state — no tanh, no sigmoid squashing h_t-1 before it is reused. That one restriction is the secret to everything that follows. We will cash it in next chapter to train this thing in parallel. For now, let us see what it remembers.

Unrolling: what does the state actually hold?

Substitute the recurrence into itself. Start from h₀ = b·x₀:

The state is a weighted sum of all past inputs, where an input from k steps ago is weighted by a^k. Because a is between 0 and 1, older inputs fade geometrically. This is an exponential moving average with decay rate a. The value of a is a memory dial: near 1, the past lingers for hundreds of steps; near 0, the state is basically just the latest input.

Worked example by hand

Let a = 0.8, b = 1, and feed the inputs x = [5, 0, 0, 0] (one spike, then silence). Watch the state decay:

Each step keeps 80% of the previous value: 5 → 4 → 3.2 → 2.56, exactly 5·0.8^t. The single input from t=0 is still echoing three steps later. That echo is the memory. Make a bigger and the echo lasts longer; make it smaller and it dies faster.

python — linear recurrence, the recurrent (inference) form
def linear_recurrence(x, a, b):     # x: list of inputs
    h = 0.0
    out = []
    for xt in x:
        h = a * h + b * xt           # one multiply-add, O(1) memory
        out.append(h)
    return out

# linear_recurrence([5,0,0,0], a=0.8, b=1.0) -> [5.0, 4.0, 3.2, 2.56]

This loop is exactly how Hawk generates text: constant state h, one cheap step per token. But a for loop over the sequence is death for training on a GPU — every step waits for the one before it. Next chapter, the linearity pays off and we delete the loop.

Chapter 2: The Parallel Scan

Here is the problem. The recurrence loop is sequential: h₅ needs h₄ needs h₃… On a GPU with thousands of cores, that loop uses one of them and the rest sit idle. Transformers don’t have this problem — every token’s attention is computed at once. To compete, the recurrence must also be computed all at once. Linearity is what lets us.

The trick: combining is associative

Think of each step as carrying a little package (a, b·x) — a decay factor and an additive contribution. Applying step t after step s means: first scale the running state by a_s and add b_sx_s, then scale by a_t and add b_tx_t. Two such steps compose into one equivalent step:

where c = b·x. The key fact: this combine operator is associative. Grouping doesn’t matter — (A•B)•C equals A•(B•C). And anything associative can be computed by a parallel scan (also called a prefix-sum): a tournament-style tree that produces all the running states in log n parallel rounds instead of n sequential steps.

So the same math has two faces. Training: lay out the whole sequence, run the parallel scan, get all h_t at once — GPU-friendly, like attention. Inference: throw away the scan and just do the one-step loop — constant memory, like an RNN. One computation, two execution modes. (If that rings a bell, it is the same duality RetNet and Mamba exploit.)

For an 8-token sequence the loop needs 8 ticks; the scan finishes in 3 (because 2³ = 8). For 2048 tokens: 2048 versus 11. That is why a linear recurrence trains at transformer speed. DeepMind actually wrote a custom TPU kernel for this scan, because the memory-movement pattern, not the arithmetic, is the bottleneck — but the idea is exactly this tree.

Chapter 3: Stability (keeping the state from exploding)

The recurrence multiplies the state by a every step. Unroll it and a contribution from k steps back is scaled by a^k. That single fact decides whether the model lives or dies:

• If |a| > 1: a^k grows without bound. The state explodes, activations go to infinity, training diverges.
• If |a| = 1: the state neither grows nor shrinks — perfect memory, but knife-edge; any drift ruins it.
• If 0 < a < 1: a^k decays to zero. Bounded, stable, forgetful in a controlled way. This is the safe zone.

The fix: never let a leave (0, 1)

We must guarantee the recurrence weight stays in the safe zone — not hope the optimizer keeps it there. The trick is to never learn a directly. Instead learn an unconstrained number Λ and pass it through a squashing function:

Now no matter what value the optimizer pushes Λ to — minus a million or plus a million — a is mathematically pinned inside (0, 1). The model is stable by construction. (Griffin actually parameterizes it in log-space for numerical comfort, but the guarantee is this sigmoid.)

Why real and diagonal?

Earlier linear-recurrence models (S4, the LRU) used complex numbers for a, so the state could rotate as well as decay — useful for modeling oscillations. DeepMind found that for language a plain real diagonal recurrence works just as well and is simpler and faster: no complex arithmetic, just a per-channel decay. “Diagonal” means each channel of the state has its own independent a — no mixing between channels inside the recurrence (mixing happens in the linear projections around it). That keeps the scan a cheap element-wise operation.

Chapter 4: RG-LRU — gating the recurrence

A fixed decay a treats every token the same. But language isn’t uniform: the name of the protagonist on page 1 should persist; the word “the” should not. We want the model to decide, per token, how much to remember and how much to let in. That decision-making is the RG-LRU — the Real-Gated Linear Recurrent Unit, the heart of Hawk and Griffin.

Two gates

Both gates are simple: take the current input x_t, run a linear layer, squash with a sigmoid (so each is a number in (0,1)):

The input gate i_t filters the incoming token — like deciding how loudly this word gets to speak into the state. The recurrence gate r_t is cleverer: it modulates the decay itself. The effective decay becomes

Stare at this. When the gate r_t → 0, the exponent → 0, so a_t → a⁰ = 1: the state is preserved almost perfectly — “this token says: hold everything, don’t decay.” When r_t → 1, the exponent is its largest, so a_t drops toward a⁸ (a strong decay) — “flush the old state, the topic just changed.” The gate slides the decay between “remember everything” and “forget aggressively,” per token, per channel.

The full RG-LRU update

The first term is the gated memory. The second term is the gated input — but notice the input weight is √(1 − a_t²), not a free parameter. This is a variance-preserving normalizer. If you keep a fraction a_t of the old state, this exact input weight ensures the state’s variance stays roughly constant instead of blowing up or shrinking as gates change. When a_t is near 1 (holding memory), √(1−a_t²) is near 0 (let almost nothing new in); when a_t is near 0 (forgetting), it is near 1 (let the new token dominate). The two terms trade off automatically.

Worked example by hand

Base decay a = 0.9, constant c = 8, current state h_t-1 = 2.0, input x_t = 1.0. Compare a “keep” token vs. a “flush” token.

Same input, same base decay — but the “keep” token barely moves the state (2.0 → 2.04, the past dominates), while the “flush” token pulls it down hard (2.0 → 1.38, the old memory is being discarded). The model learned when to do which, from data. That selectivity — the same idea Mamba calls “input-dependent dynamics” — is what lets a fixed-size state punch above its weight.

python — one RG-LRU step (per-channel, vectorized with numpy)
import numpy as np

def rg_lru_step(h, x, Lam, Wa, Wx, c=8.0):
    a    = sigmoid(Lam)              # base decay per channel, in (0,1)
    r    = sigmoid(Wa @ x)           # recurrence gate, in (0,1)
    i    = sigmoid(Wx @ x)           # input gate, in (0,1)
    a_t  = a ** (c * r)              # data-dependent decay
    h    = a_t * h + np.sqrt(1 - a_t**2) * (i * x)
    return h                        # new state, same shape as h

Chapter 5: The Recurrent Block (data flow, end to end)

The RG-LRU is the engine; the recurrent block is the car built around it. Hawk and Griffin stack many copies of one residual block. Let us trace a real tensor through it so you could implement it from memory. Say the model width is D = 1024 and we process a sequence of T = 2048 tokens, so the input to the block is shaped [T, D] = [2048, 1024].

The outer residual wrapper

Like a transformer, the block is two residual sub-layers: a temporal-mixing sub-layer (this is where the recurrence lives) and a channel-mixing MLP. Each is preceded by RMSNorm and added back to the residual stream:

Inside the recurrent core (the temporal-mixing sub-layer)

This is where Griffin’s design echoes Mamba: the normed input splits into two branches that are multiplied back together at the end — a gating structure.

1. Split into two branches with two linear layers, each [D] → [D_rnn] (the paper uses D_rnn = D, so [2048,1024] → [2048,1024] for each branch).
2. Gate branch: linear → GeLU. This branch carries no recurrence; it is a learned, element-wise multiplicative mask applied at the end.
3. Recurrence branch: linear → a small causal 1-D convolution (kernel width 4, depthwise) → RG-LRU. The tiny conv mixes each channel over the last ~4 tokens before the recurrence — cheap, sharp, short-range pattern detection that complements the recurrence’s smooth long-range memory.
4. Merge: multiply the two branches element-wise (the GeLU branch gates the RG-LRU branch), then a final linear [D_rnn] → [D] back to model width.

Output shape: [2048, 1024], same as the input — so blocks stack cleanly. Note where the nonlinearities are: the GeLU, the conv, the MLP all sit outside the recurrence. The RG-LRU itself stays strictly linear in its state, which is exactly what keeps the parallel scan from Chapter 2 valid. That placement is not an accident — it is the whole design constraint.

Chapter 6: Local Attention (the sharp complement)

The RG-LRU gives us cheap, smooth, long-range memory — but it is lossy. A fixed-size state can only summarize the past; it cannot reach back and read an exact earlier token verbatim. For some jobs you need exactly that: “copy the variable name I defined two lines ago,” “match this closing bracket to that opening one.” That is attention’s superpower — precise, content-based recall. Griffin keeps a little of it.

Sliding-window attention

The expensive part of full attention is that every token looks at all previous tokens. Local (sliding-window) attention restricts each token to the most recent W tokens — a fixed window, say W = 1024. Token at position t attends only to positions t−W+1 … t.

Two consequences, both wonderful:

• Compute drops from n² to n·W. Each token does at most W comparisons, not n. For long sequences with fixed W, cost grows linearly, not quadratically.
• The KV cache is capped at W. Once you have seen more than W tokens, you evict the oldest key/value — the cache stops growing. This is the single most important fix for the inference wall from Chapter 0.

But doesn’t a window throw away long-range info?

On its own, yes. A pure local-attention model literally cannot see past W tokens. This is exactly why Griffin is a hybrid. The recurrence branch carries information from arbitrarily far back (compressed into the state), while local attention handles precise recent lookups. Neither alone is enough; together they cover both regimes — long-range summary from the RG-LRU, short-range detail from local attention. The window doesn’t need to be long because the recurrence already owns the long range.

Chapter 7: Griffin — the hybrid stack (showcase)

Now assemble it. Griffin interleaves the two block types we just built: mostly recurrent blocks (RG-LRU), with a local-attention block mixed in periodically — the paper’s recipe is roughly two recurrent blocks for every one local-attention block. Hawk is the special case of all recurrent blocks, no attention at all.

The simulator below is a small Griffin stack processing a sequence. A piece of information enters at one token and we watch how far it can travel through the network. In a recurrent layer it propagates through the state — reachable from any later position (long range, but compressed). In a local-attention layer it can only jump within the window (short range, but exact). Stacking them lets information route through whichever mechanism fits. Below the stack, a live tally compares the inference memory of this hybrid against a pure transformer of the same depth as the sequence grows.

Play with the ratio. Set it to 0:1 (all attention, window-limited) and information can only crawl forward one window per layer — and the memory readout climbs as the sequence grows. Set it to 4:0 (pure Hawk) and every later token is reachable through the recurrence, with flat memory — but there is no exact-copy path. The sweet spot in the middle gives you both: a token far in the past stays reachable via the recurrence, while the attention layers provide crisp local lookups, and inference memory stays nearly flat as the sequence grows.

Chapter 8: Inference Economics & Extrapolation

Architecture choices only matter if they change the bill. Let us make the inference advantage concrete with the three cost axes that decide whether you can serve a model cheaply.

1. Memory per generated token

A transformer must keep the KV cache of every past token. For a sequence of length n, with L layers, the cache scales like n · L — unbounded. A Griffin recurrent layer keeps one fixed-size state regardless of n; a Griffin local-attention layer keeps a cache capped at the window W. So Griffin’s state is roughly (recurrent layers × D) + (attention layers × W) — a constant that does not grow with the sequence. At n = 100k this is the difference between gigabytes and megabytes.

2. Latency per token

Lower memory traffic means each generated token is produced faster, and crucially the per-token cost stays flat as the sequence grows, instead of creeping upward as the KV cache balloons. Long-context generation is where Griffin pulls clearly ahead of a same-size transformer.

3. Throughput

Because the state is small, you can fit many more concurrent sequences in memory and batch them — so total tokens-per-second served goes up. For a serving system, throughput-per-dollar is often the number that matters, and constant-size state is what unlocks it.

Length extrapolation — a bonus from the structure

Transformers tie meaning to absolute or relative positions via positional encodings; push them past their training length and those encodings hit values never seen, and quality falls off a cliff. Griffin’s recurrence has no length-dependent positional scheme — it just keeps applying the same per-step update — and its attention is local, so it never sees a relative distance bigger than W. Neither component cares how long the sequence is. That structural fact is why Griffin generalizes to far longer inputs than it trained on.

Chapter 9: Cheat Sheet & Connections

Everything in one place. If you can reconstruct this table and the equations beneath it, you understand Griffin and Hawk well enough to teach them.

The whole pipeline in one breath

The family tree

The shared idea across this whole row of “efficient” models: replace the growing KV cache with a fixed-size recurrent state, trained in parallel via a linear-recurrence / scan trick. Griffin’s distinctive bets are (1) a real, gated diagonal recurrence (RG-LRU) and (2) explicitly keeping a slice of local attention for exact recall, rather than trying to make the recurrence do everything.

When to reach for Griffin/Hawk

• Long-context, latency-sensitive serving: chat over long histories, streaming transcription, on-device generation — constant state is the win.
• You need length extrapolation beyond your training window.
• Reach for a plain transformer instead when you need maximum exact-recall over the entire long context (e.g. needle-in-a-haystack retrieval far beyond the local window), where full attention’s perfect memory still has the edge.

Keep exploring

→ Linear Attention & RWKV — the recurrence=attention duality in depth
→ SSM & Mamba — the selective state-space cousin of RG-LRU
→ RetNet — one computation, three execution forms
→ xLSTM — another modernized recurrence with exponential gating
→ Attention Variants — MHA, MQA, sliding-window, FlashAttention
→ Jamba — hybrid Mamba + attention + MoE at production scale