Gated Linear Attention

In a toy memory, 40 chatter tokens bury a stored fact (0.14 times the clutter). A gate that chooses what to forget keeps it at 1.5 times. This paper shows how to train that gate fast.

Learn how a language-model layer can keep a memory of fixed size, decide what to forget, and still train quickly on a GPU.

Pick a memory, then drag the chatter. Watch the three stored facts (top rows) and the clutter (bottom rows) change. Then we build, piece by piece, the memory, its three ways of running, the fast GPU kernel, the gate, the numerical trap and its fix, and the results.

You need a little matrix arithmetic (a matrix is a grid of numbers; multiplying a row by it mixes the numbers) and a little Python for the code. We build the rest from zero.

Based on “Gated Linear Attention Transformers with Hardware-Efficient Training” by Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda and Yoon Kim (MIT and MIT-IBM Watson AI Lab), ICML 2024, arXiv 2312.06635. An assigned reading of MMAI Lecture 8, Large Multimodal Models, in the MIT MMAI course. Every idea is re-taught here in our own words and devices.

Keep the facts, forget the chatter

24

An illustrative toy, not a result from the paper. A 6 by 4 memory first stores three facts (fact j writes a 1 at row j, column j), then reads chatter tokens that each write a 1 at a random one of rows 3 to 5, with a seeded random value column. The updates are the paper’s recurrences (Eq. 1 for no gate, a single constant for RetNet style, Eq. 3 for GLA). The gate values (0.9, 0.85, and 1 for facts with 0.5 for chatter) are set by hand here; in the paper the model learns them. “Recall holds” means the facts weigh at least as much as the clutter; the matrix norm is the square root of the sum of squared cells.

Chapter 0

Why Attention Needs a Cheaper Memory

See why a language model that rereads everything it has seen gets slower and heavier with every word, and what a fixed-size memory would change

Picture a long novel. A friend asks you about a small detail from chapter 2, and you are now in chapter 40. You have two ways to answer.

The first way: keep every page you have read open on your desk. To answer, flip back through the pages and look. This never loses a detail. But the desk grows with every page, and each new question takes longer to answer.

The second way: keep one notebook of a fixed size. After each page, you rewrite a little of the notebook. To answer, you read only the notebook. It never grows and it is always fast. But it can run out of room, and what you wrote down may be wrong or missing.

A language model faces this same choice. The paper “Gated Linear Attention Transformers with Hardware-Efficient Training” (Yang, Wang, Shen, Panda and Kim, ICML 2024) is about the notebook, and about making it good enough to replace the desk.

The desk: softmax attention and its cache

A Transformer reads text as a list of tokens (pieces of words). For each token, softmax attention asks: how much does each earlier token matter right now? It turns each earlier token into two vectors, a key (a label that says what the token is about) and a value (the content it can hand over). The new token makes a query (a question). Each key scores against the query, the scores are turned into weights that sum to 1, and the output is the weighted sum of the values.

When the model writes text one token at a time, it must keep every earlier key and value so that the next token can look at them. This store is the KV cache. It is the desk. It holds one key and one value for every token so far, in every layer.

Two costs follow. When the model trains, every token is compared with every earlier one, so the work grows with the square of the length L. And when the model writes, the cache grows in a straight line with L, and so does the work to read it for each new token. These two curves drive the cost of every long-context system.

The notebook: a fixed-size state

The paper starts from linear attention, which replaces the exponential score of softmax with a plain dot product. Chapter 1 derives it. Here is the one fact you need first: linear attention can be run as a recurrent network, a model that carries a hidden state (a fixed-size block of numbers) from token to token. The state is the notebook. Its size does not depend on how many tokens the model has read.

Try the numbers. The device below uses one width that we pick for illustration (the model’s hidden size d is 2,048, with 4 heads, 16-bit numbers). Drag the length of the text and watch what one layer must keep in each design.

Desk or notebook

Both axes are logarithmic. The warm line is the KV cache of one softmax-attention layer. The purple line is the state of one linear-attention layer. Drag the length and find where the lines cross.

8,192 tokens

Illustrative widths, not a model from the paper: hidden size d = 2,048, 4 heads, key size d/2 and value size d (the shapes Chapter 7 uses), 2 bytes per number. KV cache per layer = 2 × L × d × 2 bytes. State per layer = 4 heads × 256 × 512 numbers × 2 bytes = 1 MiB. Per-token read work is counted in multiply-adds: 2 × L × d for attention, 524,288 for the state.

The lines cross at 128 tokens. Past that, the cache is bigger than the notebook, and the gap widens by a factor of 2 each time the text doubles. At a million tokens the cache of one layer holds 8 GiB and the notebook still holds 1 MiB. The work to read it for one new token follows the same two curves, because the cache must be read in full while the notebook is read once.

A worked comparison, step by step

Take the width of the device and a text of 8,192 tokens, and count what one layer does to write the next token.

  1. Softmax attention. The new query is compared with 8,192 keys, and the weights then mix 8,192 values. Each key and each value has 2,048 numbers. The work is 2 × 8,192 × 2,048 = 33,554,432 multiply-adds, and the 64 MiB cache must be read.
  2. Linear attention. The new query reads a state of 4 × 256 × 512 = 524,288 numbers. That is 524,288 multiply-adds, and the 1 MiB state is read. Updating the state costs about the same again.
  3. The ratio. 33,554,432 divided by 524,288 is 64. At 16,384 tokens it is 128. The notebook gets relatively cheaper as the text grows.

The symptom to watch for in real systems is a model that slows down and needs more memory the longer a conversation or a video gets. The measurement is the cache size above. The fix this paper pursues is the notebook. But a notebook changes the model, so the question is whether the changed model is still good, and whether it can be trained as fast. That is the rest of the lesson.

Why not just use the notebook everywhere?

The paper names two reasons, and its two contributions answer them one by one.

  1. The notebook writes worse. Linear attention “generally underperforms ordinary softmax attention”, often by a wide margin in language modelling. A notebook with no way to erase fills with clutter. Chapters 4 to 7 build the gate, a learned rule for what to forget, and the resulting layer, gated linear attention (GLA).
  2. The notebook trains slowly. Existing code for linear attention is not I/O-aware: it does not plan how many bytes move between the slow and the fast memory of a GPU. In practice it runs slower than the best softmax code (FlashAttention-2) at moderate lengths. Chapters 2 and 3 build FlashLinearAttention, which the paper reports is faster than FlashAttention-2 even at a length of 1,000 tokens.

The results (Chapter 8) are for models of 340 million and 1.3 billion parameters, trained on 15 and 100 billion tokens. GLA competes with a strong LLaMA-style Transformer (Transformer++) and with the recent linear-time models RetNet and Mamba. A model trained on 2,000-token texts keeps working on texts longer than 20,000. And GLA trains with higher throughput than a Mamba model of the same size.

Where this sits in the course

This paper is an assigned reading for Lecture 8 on large multimodal models. Images, audio and video turn into very long token lists, so the cost of reading a long past matters a great deal there. The paper itself says, in its closing section, that it would be interesting to apply GLA to other kinds of data, especially data with long-range dependencies. It runs no such experiment. We stay with what it tests: language.

The road

  1. Attention as a notebook: write with an outer product, read with a query.
  2. Three ways to run the layer: parallel, recurrent, chunkwise.
  3. Moving fewer bytes: the FlashLinearAttention kernel.
  4. The gate: forgetting that depends on the data.
  5. The gated parallel form: a cumulative product that overflows, and the log-space fix.
  6. Chunks inside chunks: secondary chunking for the tensor cores.
  7. The GLA Transformer: heads, norms, output gate, shapes.
  8. The experiments: language modelling, recall, length, speed, ablations.
  9. Field guide: numbers, checklist, core code.
A model must write 100,000 tokens one at a time. What is the main advantage of a linear-attention layer over a softmax-attention layer at that length?

Chapter 1

Attention as a Notebook: Linear Attention

Write each word into a matrix with an outer product, read it back with a query, and see why this is a recurrent network in disguise

Imagine a small filing cabinet with two drawers, labelled “topic A” and “topic B”. As you read, each fact you learn goes into the drawers. A fact that is mostly about topic A goes mostly in drawer A. A fact about both topics is split between them. To recall something, you ask about a topic and read what the drawers hold.

That cabinet is the whole idea of this chapter. The two drawers are the rows of a small matrix. We will write to it, read from it, and then show that this is exactly what a recurrent network does.

Start from softmax attention, one token at a time

Let xt be the vector for token t (a row of d numbers). Softmax attention makes three new vectors from it with three learned weight matrices: the query qt, the key kt and the value vt. The output for token t is a weighted average of all values so far. The weight of value i is exp(qtkiT), divided by the sum of these weights over all i up to t. The exponential makes every weight positive, and the division makes them sum to 1.

Two things make this expensive. The exponential must be taken for every pair (t, i). And the sum over i needs every earlier key and value, which is the KV cache from Chapter 0.

Swap the exponential for a plain dot product

Linear attention (Katharopoulos et al., 2020) replaces exp(qtkiT) with a simpler score, a kernel: the dot product of two transformed vectors, φ(qt)φ(ki)T. The gain is a regrouping trick. A dot product of two sums can be rearranged so that all the keys and values are folded into one running object before any query arrives.

The paper writes that object as St, the sum over i up to t of φ(ki)Tvi, plus a second running sum zt used for dividing. It then notes that recent work (the RetNet paper) found that you can set φ to the identity and drop the division, and it works well in practice. We follow the paper. Now each step is just two lines.

St
The state, the notebook after t tokens. A matrix with dk rows (one per key dimension, the “drawers”) and dv columns (one per value dimension).
St−1
The notebook as it was before this token. We keep all of it. Chapter 4 changes exactly this term.
ktT vt
An outer product. The key is a row of dk numbers; its transpose is a column. Column times row gives a dk by dv grid: cell (i, j) is kt,i times vt,j. This is one page of writing.
ot
The output for token t: a row of dv numbers.
qt St
Read by a row-times-matrix product. Output cell j is the sum over i of qt,i times St(i, j). The query chooses how much to read from each drawer.

This is a linear recurrence: the new state is the old state plus something. The state is a matrix, so the paper calls it a linear RNN with a 2D (matrix-valued) hidden state. It adds that this kind of model is also known as fast weights, because the matrix changes with the data while the input flows through it, as the model’s own weights do during learning.

One worked example, every step

Take drawers of size 2 by 2, and three tokens. The keys are k1 = (1, 0), k2 = (0, 1), k3 = (1, 1). The values are v1 = (2, 1), v2 = (0, 3), v3 = (1, 1). Step through them in the device below. Here is the arithmetic it shows.

  1. Token 1. The outer product of (1, 0) and (2, 1) puts the value in row 0 and nothing in row 1. So S1 = [[2, 1], [0, 0]].
  2. Token 2. The key (0, 1) writes to row 1. S2 = S1 + [[0, 0], [0, 3]] = [[2, 1], [0, 3]].
  3. Token 3. The key (1, 1) writes its value (1, 1) into both rows. S3 = [[3, 2], [1, 4]].
  4. Read with q = (1, 0). The query picks row 0 of S3, so o3 = (3, 2).

Write and read the notebook

Press Write next token to add a page to the memory. Choose a query to read with. The check line adds up the values by hand and must match.

Toy numbers chosen for easy arithmetic (illustrative, not from the paper). The state update and the read follow Eq. 1 of the paper exactly. The check line is the parallel form of the next chapter: output = sum over written tokens i of (q · ki) times vi.

The same layer in matrix form

Stack all L queries as the rows of a matrix Q (shape L by dk), and likewise K (L by dk) and V (L by dv). Unrolling the recurrence gives the parallel form: O = ((Q KT) ⊙ M) V. The product Q KT is an L by L grid of scores. The mask M holds a 1 where the row is at or after the column, and a 0 elsewhere, so no token reads the future. The sign ⊙ means “multiply cell by cell”.

Look at the check line in the device. For token 3 and q = (1, 0), the scores against the three keys are 1, 0 and 1. Weighted values: 1 × (2, 1) + 0 × (0, 3) + 1 × (1, 1) = (3, 2). Same answer as the recurrence. Two roads, one result.

One footnote in the paper deserves a pause. Without the mask, you could regroup (Q KT) V as Q (KT V) and the cost would drop from L squared times d to L times d squared. With the mask, you cannot. The mask cuts a different slice of keys for each row, so there is no single KTV to share. This is why the paper needs a third form, in Chapter 2.

What goes wrong: the notebook only fills

Watch the memory in the device once more. Nothing ever leaves. Each page adds to the sum. This has a symptom you can see in the example: reading drawer 0 after token 3 returns (3, 2), which is a blend of token 1 and token 3. If you wanted only token 1’s value (2, 1), you cannot get it back, because token 3 wrote on top of it.

To measure the problem, count the room. The state holds dk rows. If the keys of the tokens were perfectly separate (orthogonal, meaning their dot products are 0), you could store dk facts with no blending. With a longer text, there are more tokens than rows, so keys must overlap and every read mixes several writes. In the 2 by 2 example, three keys cannot all be separate.

The fix the paper builds towards is to let the notebook forget. Old pages fade, so that new ones have room. Chapter 4 adds that. First, we need to know how to run this layer fast, because the way we run it will decide how much freedom the fix gets.

In linear attention, a token with key k and value v is written into the state S. What is added to S, and what shape is it?

Chapter 2

Three Ways to Run the Same Layer

Compare the parallel, recurrent and chunkwise forms, and find the chunk size that balances work against waiting

You must add up 2,048 receipts, and you have a crowd of helpers. There are three plans.

Plan one: one person keeps a running total and reads the receipts one at a time. Almost no arithmetic is wasted, but 2,047 helpers stand idle, because each step needs the last total. Plan two: write every receipt in one huge table and have all helpers fill in every cell at once. Everyone is busy, but the table is far bigger than the job. Plan three: cut the pile into groups of 64. Inside each group, the helpers work all at once. Between groups, one running total is passed along. This is the plan that wins, and it is the one the paper uses.

The three plans are the three forms of a linear-attention layer. All three give the same output. They differ in how much arithmetic they do and how much of it can run at the same time.

What a GPU is good at

A GPU is a chip with thousands of small workers, called threads. Threads are grouped into blocks, and each block runs on one streaming multiprocessor (SM). A chip has many SMs. Occupancy is the share of them that are busy. If your job is one long serial chain, most SMs sleep. The paper points out that when the batch size is small, which is common in long-sequence training, the only way to keep the chip busy is to split work along the sequence itself.

A GPU also has special units for one job: multiplying matrices. On NVIDIA chips they are called tensor cores. The paper notes that a half-precision matrix multiply on an A100 runs roughly 16 times faster on tensor cores than on the ordinary cores. They work best on blocks whose sizes are multiples of 16. Any algorithm that wants to be fast must turn most of its work into matrix multiplies.

The parallel form: all pairs at once

From Chapter 1, O = ((Q KT) ⊙ M) V. Every step is a matrix multiply or a cell-wise mask, so it fits tensor cores. All L outputs come out together. The price is the L by L score grid. The work is about L2 times d, the quadratic cost we wanted to escape.

The recurrent form: the cheapest total, the slowest clock

The recurrence of Chapter 1 does about L times d2 multiply-adds in all, because each step touches the d by d state a few times. That is the lowest total of the three. The paper explains why it still loses on a real chip. The update is an outer product plus an addition, which are cell-wise operations. They cannot use tensor cores, and they do very little arithmetic per byte moved. And the L steps run one after another.

You might hope to run the recurrence in parallel with a parallel scan (a way to combine pairwise sums in a tree). The paper warns that this needs the d by d state of every time step to be written out, so the memory traffic cancels the gain.

The chunkwise form: a middle road with one knob

Cut the sequence into N = L / C blocks of chunk size C. Let S[i] be the state after i chunks, and let Q[i], K[i], V[i] hold the C rows of chunk i. The paper gives two equations. The first passes the state from chunk to chunk. The second computes the outputs of a chunk in parallel.

S[i+1]
The state after chunk i has been written in. Only N of these exist, not L.
S[i]
The state handed over from the previous chunk. For the first chunk it is zero (or the end of the previous text).
K[i]T V[i]
The sum of all outer products in the chunk, found in one matrix multiply of shapes (dk by C) times (C by dv). It replaces C separate writes.
O[i+1]
The C outputs of the chunk, shape C by dv.
Q[i+1] S[i]
The inter-chunk part: what every token in the chunk reads from all earlier chunks, through the state. One matrix multiply, cost about C d2.
((Q KT) ⊙ M) V
The intra-chunk part: tokens read earlier tokens of the same chunk with the parallel form of Chapter 1, cost about C2 d. M is the C by C causal mask.

Add the costs over N chunks. The inter-chunk work is N times C d2, which is L d2. The intra-chunk work is N times C2 d, which is L C d. The total is of order L C d + L d2, and the paper notes this is below L2 d when L is larger than d. Set C = L and you get the parallel form. Set C = 1 and you get the recurrent form. C is the knob.

Turn the chunk-size knob

Bars are on a log scale. Top: total multiply-adds for one head. Bottom: the steps that must run one after another. Drag the chunk size and compare it with the two ends.

64

Operation counts use the paper’s orders of growth, work = L · C · d + L · d2, for one head with L = 2,048 tokens and head size d = 64 (the paper’s speed tests use head size 64 and chunk size 64). The counts leave out constants and say nothing about wall-clock time, which depends on tensor cores and memory (Chapter 3).

Run the numbers with the device. With C = 64 and L = 2,048 there are 32 chunks. The total is 2,048 × 64 × 64 + 2,048 × 642, which is 8.4 million plus 8.4 million, or 16.8 million multiply-adds. The parallel form needs about 277 million. The recurrent form needs about 8.5 million but 2,048 steps in a row. The chunkwise form pays twice the minimum work and cuts the serial chain by a factor of 64. And nearly all of its work is matrix multiplication, which the tensor cores love. The paper says a chunk size that is a multiple of 16 lets it use them.

One chunk at a time, with shapes

  1. Input. Q, K of shape L by dk and V of shape L by dv. Take L = 8, C = 4, so N = 2 chunks.
  2. Chunk 0. The state S[0] is zero, so the inter-chunk term is zero. The intra-chunk term gives O[0], shape 4 by dv.
  3. Hand over. S[1] = K[0]T V[0], shape dk by dv.
  4. Chunk 1. The inter-chunk term Q[1] S[1] (4 by dk times dk by dv) is added to the intra-chunk term of chunk 1.

One failure to keep in mind: the form is only as fast as its code. The paper says most earlier chunkwise implementations were not I/O-aware, so they were slower than FlashAttention for moderate lengths of 2,000 to 4,000 tokens. Counting multiply-adds is not enough. The next chapter fixes the memory traffic.

What does setting the chunk size C equal to the full length L give you, and what does setting C = 1 give you?

Chapter 3

Moving Fewer Bytes: FlashLinearAttention

Follow one chunk through the GPU’s fast and slow memory, and choose whether to store the state or recompute it

A cook works at a small counter. Everything on the counter is within reach, and a knife moves in a blink. The pantry is down the hall. It is huge, but each trip takes a long time. A slow cook makes many trips and chops a little each time. A fast cook plans: carry in everything one dish needs, do all the work at the counter, and walk back once.

A GPU works the same way. This chapter is the plan for the cook.

Two memories: the pantry and the counter

The paper names the two. HBM (high-bandwidth memory) is the GPU’s main memory: large, but slower to reach. SRAM (also called shared memory) is a small block next to each processor: fast, but tiny. An I/O-aware algorithm counts the bytes it moves between the two, and plans its loops to move as few as it can. The classic tool is tiling: cut the data into blocks, load one block into SRAM, do all the work that block allows, and write the result back once. FlashAttention, which the paper takes as its model, is built this way for softmax attention.

For the chunkwise form of Chapter 2, tiling is natural. A chunk is a block. When the chunk of queries Q[n] arrives in SRAM, two products need it: the inter-chunk read Q[n] S and the intra-chunk part (Q[n] K[n]T ⊙ M) V[n]. Done together, Q[n] is loaded once, not twice. That is the saving in one sentence.

Version one: keep the state on the counter

The paper calls the first version non-materialization, because the state is never written to HBM. It is Algorithm 1 of the paper, and its data flow is short.

  1. Start. Make a state S of shape d by d, all zeros, in SRAM. Build the C by C causal mask on chip.
  2. For each chunk n, in order: load Q[n], K[n], V[n], each of shape C by d, from HBM. That is 3 × C × d numbers in.
  3. On chip: compute O′ = Q[n] S + (Q[n] K[n]T ⊙ M) V[n]. Then update S = S + K[n]T V[n].
  4. Out. Store O′ (C by d) to HBM. S stays on chip for the next chunk.

This version is thrifty. It reads each input once and writes each output once. But the chunks run one after another, because chunk n+1 needs the S that chunk n leaves behind. The work spreads over the batch, the heads and the head dimension, and that is all. The paper says plainly what follows: when the batch is small, as in long-sequence training, many streaming multiprocessors have nothing to do.

Version two: write the states down first

The second version is materialization. It splits the job in two. First, a short loop goes over the chunks in order and does only the cheap part: it loads K[n] and V[n], adds K[n]T V[n] to S, and stores S[n] in HBM. Second, once every S[n] is on the shelf, all chunks can be computed at the same time: each one loads its own Q, K, V and S[n] and writes its O. The paper marks this second loop as a parallel for.

Now the sequence itself is a source of parallel work. The cost is memory. Storing the states raises the memory footprint by about 10 to 20 percent. The paper removes most of it with recomputation: the states are thrown away after the forward pass and rebuilt during the backward pass. This costs a little time and saves a lot of memory, and the paper adopts it by default.

The device below turns this into something you can see. Eight lanes stand for eight processors. A job is one sequence in one head, cut into 8 chunks. The costs are illustrative: a chunk costs 4 time units when everything is done together, but only 1 unit for the cheap state step and 3 units for the output step when they are split.

Fill the chip with chunks

Each row is a processor. Purple blocks pass the state along; warm blocks compute outputs. Choose a version, then raise the number of independent jobs (batch times heads) and watch how many rows stay empty.

An illustrative schedule, not a measurement. Costs: 4 units per chunk fused; 1 unit per chunk for the state pass and 3 for the output step when split. Eight processors, eight chunks per job, a barrier between the two stages. The two versions, the 10 to 20 percent extra memory and the recomputation default are from Section 3.3 of the paper.

Try one job. The first version keeps one row busy for 32 units and leaves seven asleep: 12.5 percent of the chip. The second version runs the state pass in 8 units, then spreads 8 chunk tasks over all 8 rows for 3 units: 11 units in all, with 36 percent of the chip busy. Now raise the jobs to 8. The first version fills every row by itself and takes 32 units. The second takes 8 + 3 × 8 = 32 units too, and also holds extra states in memory. So the second version helps exactly when the batch is too small to keep the chip full. The paper chooses between them by the training setting.

What the paper measured

Figure 2 of the paper compares speed and memory on one H100 GPU: batch size 32, 16 heads, head size 64, chunk size 64. FlashLinearAttention, in either version, is faster than FlashAttention-2 as a standalone layer, even at a length of 1,000 tokens. A plain PyTorch version of the same chunkwise algorithm, which does not plan its memory traffic, is slower. The paper reads this gap as the benefit of I/O-awareness.

The paper also gives a backward pass (its Algorithm 2) built on the same ideas. And it tells us why the gated layer of the next chapters is harder: the gate breaks one assumption that this chapter leans on.

A team trains on very long sequences with a batch of 1. Which version of FlashLinearAttention keeps more of the GPU busy, and what does it cost?

Chapter 4

Teaching the Notebook to Forget: The Gate

Add a forget gate that depends on the data, and compare the ways to shape it by what they can express and what they cost

Back to the filing cabinet from Chapter 1. Reading a long story, you keep filing facts into the same drawers. After a while drawer A holds the hero’s plans from chapter 1, chapter 5, chapter 9, all mixed. When you ask what the hero wants right now, the old plans muddy the answer.

A smart clerk would do two things. Fade old pages a little as new ones arrive. And decide what to fade based on what is being filed: a new chapter on a new topic should wipe the drawers of the old topic, but leave unrelated drawers alone. That clerk is a forget gate.

Why plain linear attention has no way to forget

The update of Chapter 1 is St = St−1 + ktTvt. It has no decay term. The paper recalls that a forget gate has long been seen as crucial in recurrent networks (it is the core of the LSTM and the GRU), and that its absence is thought to be partly why linear attention is unstable on long contexts.

The first fix, used by RetNet and TransNormerLLM, is one fixed number γ between 0 and 1: St = γ St−1 + ktTvt. Every token multiplies the whole memory by the same γ, so old writes fade smoothly. This is a global, data-independent decay. It keeps the attention-style parallel form, which is why it is attractive. But it fades every drawer at the same speed, whatever is being filed. The paper notes that in ordinary RNNs, gates that depend on the data are known to matter a great deal.

The gated update

GLA lets the decay depend on the current token and gives it a shape. The recurrence becomes St = Gt ⊙ St−1 + ktTvt, where Gt is a dk by dv matrix of numbers between 0 and 1, computed from the token. The sign ⊙ means “multiply cell by cell”: each cell of the memory is scaled by its own gate. A gate of 1 keeps a cell, a gate near 0 erases it.

St
The state after token t, a dk by dv matrix. The paper lets dk and dv differ.
Gt
The forget gate: one number in (0, 1) for every cell. It is a function of the input xt, so it changes with every token. That is what data-dependent means.
St−1
The memory before this token. It is scaled down by Gt before anything new is written.
ktT vt
The new page, the outer product from Chapter 1. It is added at full strength.

Worked example. Take the state S = [[3, 2], [1, 4]] from Chapter 1 and a gate G = [[1, 1], [0.5, 0.5]]. The first row has gate 1, so it stays [3, 2]. The second row has gate 0.5, so it becomes [0.5, 2]. A token with k = (0, 1) and v = (1, 1) then adds [[0, 0], [1, 1]]. The new state is [[3, 2], [1.5, 3]]. Drawer 0 was kept intact, drawer 1 was half erased and then refilled.

The paper points out that this one equation covers many recent models, and Table 1 lists them. They differ only in how they compute Gt from the input.

The design problem: what shape should the gate have?

A full gate matrix computed from xt would need d × dk × dv weights. That is far too many. Every design in Table 1 is a compromise between three things the paper names: how many parameters, how large the state can be, and whether it trains efficiently. Here are the main choices.

Use the device to feel the difference. You ask each gate to erase drawer 0 and keep the others. See which one can do that without harming the rest.

Ask the gate to forget one drawer

The grid is the gate Gt for an 8 by 8 memory. Bright pink is 1, keep; dark is near 0, erase. Drag the strength, then switch the kind of gate.

80%

Illustrative: the gate values are what each shape would have to output to follow the request “scale drawer 0 by 1 minus the strength and leave the rest”; a trained model learns its own. The weight counts are the products in the paper’s Table 1 for the sizes d = 1,024, dk = 512, dv = 1,024: scalar d; GLA low-rank 16d + 16dk; outer d(dv + dk); full d · dk · dv. Mamba’s real gate adds a learned matrix A and cannot use tensor cores.

Set the strength to 80 percent. The scalar gate has no row to aim at, so it must scale all 64 cells by 0.2: drawer 0 is erased as asked, and seven drawers of good memory are erased with it. The vector, outer and full gates erase only drawer 0. The vector gate needs 24,576 weights; the outer product needs 1,572,864; the full matrix needs 536,870,912. For this request all three row-aware shapes give the same result, and the vector gate gets there with the fewest weights.

The measurement: what the gate is worth

The paper tests this in an ablation (Table 4). It trains four 340-million-parameter variants on 7 billion tokens and reports the average perplexity of the last 200 steps; lower is better. With no gate (plain linear attention): 23.21. With a fixed scalar decay (RetNet): 16.55. With a data-dependent scalar gate: 15.56. With GLA’s fine-grained data-dependent gate: 14.77. Each step down the list closes part of the gap, and the paper concludes that a finer-grained gate is still necessary after the data-dependent scalar. Chapter 8 returns to this table.

The cost of the vector gate is that the gated layer is harder to train fast. The simple parallel form of Chapter 1 no longer applies, and the next chapter shows why.

Which statement best describes the difference between RetNet’s decay and GLA’s gate?

Chapter 5

The Gated Parallel Form and Why It Blows Up

Unroll the gated recurrence into attention form, watch a cumulative product overflow, and fix it by moving to log space

A shop loses 5 percent of its value every year to inflation. After 200 years, one coin from year 1 is worth almost nothing: its value is the product of 200 factors of 0.95. Now someone asks you to compare a coin from year 150 with one from year 160. A clumsy method would first convert every coin to year-200 money by dividing by the tiny product, then compare. Those divisions create gigantic numbers. A computer with a limited number format cannot hold them.

This chapter is that story for the gated memory. It has a neat matrix form, a numerical trap, and a fix that costs speed. The next chapter wins the speed back.

Unroll the gated recurrence

Chapter 4 ended with GLA’s update St = Diag(αt) St−1 + ktTvt. Here αt is a row of dk numbers in (0, 1), one forget rate per key dimension. Write the state at time t by unrolling: the write from token i has been scaled by the gates of every token after it, up to t. The scale for key dimension m is the product of αj[m] for j from i+1 to t.

Give that product a name. Let bt be the running product of all gates up to token t, taken per dimension: bt = α1 ⊙ α2 ⊙ … ⊙ αt. The product from i+1 to t is then bt divided by bi, per dimension. So the output of token t is a sum over earlier tokens i, weighted by the ratio bt / bi.

O
All L outputs at once, shape L by dv.
Q ⊙ B
Each query row scaled cell by cell by its own running product bt. B stacks b1 to bL, shape L by dk. This supplies the bt of the ratio.
(K / B)T
Each key row divided cell by cell by its own bi. This supplies the 1 / bi of the ratio.
M
The causal mask from Chapter 1: 1 for i at or before t, 0 after.
V
The values, shape L by dv.

This is the paper’s parallel form of GLA. It has the shape of Chapter 1’s formula. The score grid is a matrix product of (Q ⊙ B) and (K / B)T. Check it on one dimension with every gate equal to 0.5. Then b = (0.5, 0.25, 0.125). The weight of token 1 in the output of token 3 is b3 / b1 = 0.125 / 0.5 = 0.25. That is right: token 1’s write was halved twice, by tokens 2 and 3.

The trap: dividing by a number that shrinks to nothing

Look at K / B. The running product bt is a chain of numbers below 1, so it shrinks with t. Its inverse 1 / bt grows without limit. The paper says it directly: the form is not numerically stable, because K / B can explode when t is large.

The size of the problem depends on the number format. In 16-bit floating point (the half precision that tensor cores like), the largest number is 65,504. In 32-bit, it is about 3.4 × 1038. If α = 0.95 on every step, then 1 / bt = (1 / 0.95)t. Take logarithms: log10(1 / bt) = t × 0.0223. It passes log10(65,504) = 4.82 at t = 217. Past that, K / B is infinity in half precision, and infinity times zero gives an invalid number that spreads through the whole network.

Watch 1 / b grow

The warm line is how big 1 / bt is, as a power of ten. The red lines are the largest numbers that 16-bit and 32-bit formats can hold. Choose a gate value, then drag the position t.

150

Exact arithmetic for one key dimension whose gate is the same number at every step. Largest finite 16-bit float 65,504; largest 32-bit float about 3.4 × 1038. Real gates vary by token and dimension, so the crossing point in a model moves, but the growth is the same: 1 / b multiplies by 1 / α at every step.

The paper adds a temperature to its gate so that forgetting is slow (Chapter 7). Even so, a gate of 0.99 puts the 16-bit limit at position 1,104, and a gate of 0.9 pulls it in to 106. Chunks of 64 tokens hit the wall for fast-forgetting dimensions, and a context of thousands of tokens hits it for slow ones. There is no safe setting.

The fix: work with ratios in log space

The cure is never to build b and 1 / b separately. Only their ratio is needed, and the ratio bt / bi is at most 1 when t is at or after i. Work with logarithms: the log of a ratio is a difference, and the difference of two logs is always zero or negative. The paper writes the score grid entry by entry as Eq. 4.

Pij = ∑k=1dk Qik Kjk exp( log Bik − log Bjk ), for i ≥ j

In this sum, every exponent is at most 0, so every factor is between 0 and 1. Nothing can overflow. For the one-dimension example, P31 = q3 k1 exp(ln 0.125 − ln 0.5) = q3 k1 × 0.25. The same answer, reached safely.

Now the price. The matrix form multiplied two matrices, and that is what tensor cores do fast. Eq. 4 sums over a third index k with an exponential of a difference inside the sum. It is not a matrix product, so it cannot use half-precision matmuls on tensor cores. It runs on the slow general cores in full precision. Done for the whole L by L grid, it would erase the speed gain of Chapters 2 and 3. The paper states this trade clearly, and it needs a second idea: use the safe slow path only where it is needed. That is the next chapter.

Why can the paper not just compute (Q ⊙ B)(K / B)T on tensor cores for a long sequence?

Chapter 6

Chunks Inside Chunks: Making the Gated Form Fast

Split the work into three kinds of tile, and count how much of it can use the GPU’s fastest matrix units

A bank clears thousands of cheques a day. A fast machine handles the ordinary ones, a hundred a second. A few cheques look odd, and a careful clerk must read them by hand. The bank cannot afford to send everything to the clerk. So it sorts: the fast machine takes everything it safely can, and the clerk sees only the small pile that needs care.

That is the plan for gated linear attention. Chapter 5 found a safe but slow path (log space, full precision). This chapter makes the slow pile small. Two ideas do it: a gated version of the chunkwise form, and a second level of chunking inside each chunk.

Decay across a chunk: three helper quantities

Take the chunkwise form of Chapter 2, and add the gates. Within chunk i, positions are numbered j = 1 to C, so the token at position j has global index iC + j. The paper defines three ratios of running products b. Each one is a product of at most C gates.

All three are numbers between 0 and 1, because each is a product of gates. None of them is a ratio with a huge denominator. This is why they are safe in half precision.

S[i+1]
The state after chunk i+1, passed to the next chunk. Shape dk by dv.
(γT 1) ⊙ S[i]
The old state faded by the whole chunk’s decay: row m is scaled by γ[m]. The factor (γT1) is the vector γ copied across all columns.
(K ⊙ Γ)T V
The new writes of the chunk, each key scaled by how much of it survives to the chunk’s end. One matrix multiply of shapes (dk by C) times (C by dv).
Ointer
What the chunk’s tokens read from everything before the chunk.
(Q ⊙ Λ) S[i]
Each query is scaled by how much old state is alive at its position, then reads the old state. One matrix multiply.

Worked example, one dimension with every gate 0.5 and C = 4. Then Λ = (0.5, 0.25, 0.125, 0.0625) for positions 1 to 4. Γ = (0.125, 0.25, 0.5, 1). And γ = 0.0625. Check against the token-by-token rule: the first write of the chunk is halved by gates 2, 3 and 4, so it survives at 0.53 = 0.125, which is Γ1. The last write is not faded at all, which is Γ4 = 1. The first token of the next chunk reads the old state after one halving: Λ1 = 0.5. Everything matches.

So the inter-chunk part is solved: it uses only matrix multiplies, which can run in half precision on tensor cores. Only the intra-chunk part is left, the chunk’s own C by C grid of token pairs. That grid still needs Eq. 4 and its slow path.

Chunks inside chunks

The paper’s answer is secondary-level chunking: cut each chunk of C tokens into sub-chunks of c tokens, and treat the C by C grid as a grid of small blocks. There are two kinds of block.

  1. Blocks between two different sub-chunks (off the diagonal). Pick a reference point inside the chunk, as the paper does with a ratio like biC / b(j+1)C. The decays then become safe, and the block is a plain matrix product. It runs on tensor cores in half precision. Figure 3 draws these tiles orange.
  2. Blocks inside one sub-chunk (on the diagonal). They are small, c by c, and they cannot be safely rewritten as a matrix product. They use Eq. 4 in log space, in full precision, on the slow cores. Figure 3 draws these tiles pink.

The paper summarises the result: with the two-level tiling, the number of non-half-precision matrix-multiply operations is greatly reduced, which gives the wall-clock gain. The device counts it for you. The left grid is a toy of 64 tokens with chunks of 16. The bar compares it with the real shape of one head at 2,048 tokens and chunks of 64.

Sort the blocks

Each square is a block of the lower-triangle attention grid. Gray: read through the state (inter-chunk). Orange: half-precision matmul between sub-chunks. Pink: the small slow blocks on the diagonal. Choose a sub-chunk size.

Counts are exact for the stated sizes: cells of the lower triangle are L(L+1)/2; each chunk holds C(C+1)/2 cells; each sub-chunk diagonal block holds c(c+1)/2. The toy uses L = 64, C = 16; the large case uses L = 2,048, C = 64 with the same c. The paper does not state the sub-chunk size in the text we use here, so c is a free choice in this device.

With c = 4 in the toy, about 7.7 percent of all cells fall in slow blocks, 18.5 percent are orange, and 73.8 percent go through the state. In the large case with c = 16, the slow blocks are under 1 percent of all cells (0.83 percent). Nearly 97 percent of cells never form a score at all, because the state carries them. Another 2.3 percent are orange matmuls. The careful clerk reads less than one cheque in a hundred.

The code, in the order it runs

The paper’s Listing 1 is a PyTorch-style outline. Its steps for each chunk are these.

  1. Take the log-gates for the chunk, shape C by dk, and cumulative-sum them to get log b inside the chunk.
  2. Make the scaled copies: q′ = q ⊙ exp(log b), k′ = k ⊙ exp(log blast − log b), and g = exp(log blast).
  3. Inter-chunk: o = q′ @ S, one matmul.
  4. Intra-chunk, per sub-chunk: the diagonal block by the slow log-space loop, then the blocks to its left by matmul, adding both into o.
  5. State update: S = gT ⊙ S + k′T @ v.

One more result from this section matters for training. Earlier work claimed that GLA-style models must store the d by d states of every step to compute the gradient of the gate. The paper instead gives a closed form for the gate gradient, built from the gradients of q and k and a sum from the right, and it needs no stored states. (The sign convention differs between the main text and the appendix; the structure is the same.)

In the chunkwise form of GLA, why does secondary-level chunking help?

Chapter 7

From Layer to Model: The GLA Transformer

Trace one token through a GLA layer, check every tensor shape, and see why the layer costs the same parameters as softmax attention

Follow one word through a small factory. It arrives as a row of numbers. At the first station it is turned into three new rows: a question, a label and a content. At the second station a gate decides how much of the shelf to clear. At the third, the content is filed and the question is answered from the shelf. At the last, the answer is cleaned, switched on or off by a valve, and sent out.

This chapter walks that factory for the GLA layer and then places it in a Transformer. The goal is simple: you could write this layer after reading it.

One head, one token

Take the layer's input for token t, a row xt of d numbers. A GLA layer has H heads. A head is an independent copy of the memory with its own smaller queries, keys and values, so the model can keep H separate notebooks. The key size per head is d′k = dk / H and the value size per head is d′v = dv / H. For head h, the paper writes:

Sht
The state of head h, shape d′k by d′v. This is the only thing carried from token to token.
αht
The forget gate of head h, a row of d′k numbers in (0, 1), one per key dimension. Diag(α) S scales row m of S by α[m], the same as the (αT1) ⊙ S form of Chapter 4.
oht
The raw read of head h, shape 1 by d′v.
o′t
Each head’s read is normalised by layer norm (LN, which rescales a row to have mean 0 and spread 1), then the H pieces are joined side by side. Shape 1 by dv. LN is applied per head, after the read, as in RetNet.
rt
An output gate computed from the layer input. Swish(z) = z times sigmoid(z), a smooth valve that is mostly shut for negative z. Shape 1 by dv.
yt
The layer output: the read, multiplied cell by cell by the valve, then mixed back to width d by the output matrix WO. Shape 1 by d.

The gate itself is made in two small steps. First a low-rank map: xt Wα1 Wα2, with Wα1 of shape d by 16 and Wα2 of shape 16 by dk. A low-rank map squeezes through a narrow middle (here 16 numbers), which saves weights. Second, a sigmoid squeezes each number into (0, 1), and the result is raised to the power 1 / τ with τ = 16. The paper says this temperature encourages a slower forgetting rate.

Try numbers. If the sigmoid gives 0.5, the 16th root is 0.958. If it gives 0.1, the root is 0.866. If it gives 0.01, the root is 0.750. Even a sigmoid output close to zero leaves a gate of about 0.75, so no single token wipes a drawer clean. The device lets you try any value.

Walk one token through, with shapes

Use illustrative sizes: d = 1,024, H = 4 heads, dk = 512 (half of d) and dv = 1,024 (equal to d). Per head that is d′k = 128 and d′v = 256. The paper makes these two choices on purpose, and the stepper below counts the weights at each stage.

Walk a token through the layer

Press Next stage to move the token along the line. Drag the gate slider to see how the sigmoid and the 16th root turn a score into a forget gate.

0.0

Sizes d = 1,024, H = 4, dk = d/2, dv = d are illustrative; the relations (dk = d/2, dv = d, low-rank 16, τ = 16, 4 heads by default) are the paper’s. The count leaves out the biases and the layer-norm scales. Shapes are for one token and one head unless a line says otherwise.

At the end, the layer has used 4,218,880 weights, against 4 × 1,0242 = 4,194,304 for a softmax attention layer of the same width (four d by d matrices WQ, WK, WV, WO). The extra output gate Wr costs a full d by dv matrix, so the paper pays for it by halving the key size: WQ and WK are d by d/2 each. The low-rank gate adds only 24,576 weights, 0.6 percent. The paper states this budget: one GLA layer needs roughly 4d2 parameters, as softmax attention does.

Why heads change the memory size

The number of heads sets how big the state is. Count it with the sizes above. With 4 heads, each state is 128 by 256, so the layer holds 4 × 32,768 = 131,072 numbers. With 8 heads, each state is 64 by 128, so the layer holds 8 × 8,192 = 65,536 numbers: half the memory. With 1 head the state is 512 by 1,024, or 524,288 numbers: four times as much.

This simple count lines up with the paper's ablation (Table 4, 340M models): 8 heads gives a perplexity of 15.29, 4 heads 14.77, and 1 head 14.61. A bigger state remembers more, and a smaller one forgets sooner. But the single head needs much more GPU memory for a small gain, so the paper keeps 4. The count is our arithmetic, not a claim from the paper; the paper only reports the three perplexities.

From layer to Transformer, and how it is trained

The block has the usual two halves. First, layer-normalise the stream, apply GLA, and add the result back (a residual connection). Second, layer-normalise, apply a feed-forward network called SwiGLU, and add that back. SwiGLU(Z) = (Swish(Z W1) ⊙ Z W2) W3, the same feed-forward net as the LLaMA family. Stack many blocks and you have the GLA Transformer.

At inference, each new token needs only the state: compute q, k, v and α for the token, scale S rows by α, add kTv, read q S. Memory and work do not grow with the text length. At training time, the whole text goes through the chunkwise algorithm of Chapters 3 to 6.

The paper’s recipe, all from Section 5.1: SlimPajama data (a 100-billion-token subset of 627 billion), the Mistral tokenizer, AdamW with a top learning rate of 3e-4, a cosine schedule with a warmup (0.5 billion tokens for the 340M models, 1 billion for the 1.3B models) that starts and ends at 3e-5, weight decay 0.01, gradient clipping at 1.0. The 340M models see 15 billion tokens with batches of 0.5 million tokens. The 1.3B models see 100 billion tokens with batches of 2 million. All baselines train on the same data for the same number of tokens.

Why does the paper set dk = d/2 and use a low-rank map (through 16 numbers) for the gate?

Chapter 8

What the Experiments Show

Read each table and figure of the paper, see where GLA wins, where it loses, and what the ablation says about the gate

You have a new engine design. Before you trust it, you would run it on a test track, against the best engine you know and against two rivals, and you would ask three questions. Is it as strong? Does it fail somewhere the others do not? Does it cost less to run? The paper answers these with language models, a recall test, a long-text test and a speed test.

One rule makes the comparison fair. All models train from scratch on the same data (a 100-billion-token subset of SlimPajama, with the Mistral tokenizer) for the same number of tokens. There are two sizes: 340 million parameters on 15 billion tokens, and 1.3 billion parameters on 100 billion tokens. The three baselines are Transformer++ (the LLaMA recipe: rotary positions, SwiGLU, RMSNorm), RetNet (fixed decay) and Mamba (the open-source code).

Table 2: language modelling and common-sense tasks

The table reports perplexity on WikiText and LAMBADA (lower is better) and zero-shot accuracy on LAMBADA, PIQA, HellaSwag, WinoGrande, ARC-easy and ARC-challenge. The last column is the average accuracy. At 1.3 billion, GLA has the best average of the four, 51.0, against 50.9 for Transformer++, 50.0 for Mamba and 48.9 for RetNet. At 340 million, Mamba leads with 41.8, then GLA 41.5, Transformer++ 41.2 and RetNet 41.0. The gaps are small. The honest reading is that GLA matches the Transformer++ and Mamba on these tasks and beats the fixed-decay RetNet.

The perplexity columns tell a slightly different story. On WikiText at 1.3 billion, Transformer++ gets 16.85, Mamba 17.06, GLA 17.22 and RetNet 18.64. GLA trails the first two by a little. The paper’s words are “competitively” and “respectably”, and the numbers support those words and no stronger ones.

Recall: where linear models are weak

A fixed-size state cannot hold everything. The paper tests that on tasks that need exact retrieval. The synthetic task MQAR (multi-query associative recall) shows the model many key-value pairs, then asks for the value of a key several times. Softmax attention scores perfectly in every setting. Among the others (Figure 4, model sizes 64 to 512, sequences of 256 and 512, with 16 and 64 pairs), models with a matrix-valued state (Mamba, RetNet, GLA) beat the vector-state models Hyena and RWKV-4, and GLA beats RetNet, which the paper takes as evidence for data-dependent gates.

Table 3 uses three real tasks: FDA and SWDE (pulling a field out of a web page or form) and SQuAD (reading comprehension). Work through the 1.3-billion column. Transformer++ scores 27.4 on FDA. Mamba scores 6.2, which is 23 percent of that. RetNet scores 14.3, 52 percent. GLA scores 19.9, 73 percent.

On SWDE the Transformer scores 66.6; GLA 50.6 (76 percent), RetNet 42.8, Mamba 41.4. On SQuAD, GLA scores 42.6, ahead of all three: Mamba 35.2, RetNet 34.7 and the Transformer 31.5. The paper explains GLA’s edge among linear models by its larger state than Mamba’s and its selective gate compared with RetNet’s. The gap to the Transformer on FDA and SWDE remains open.

Here is how the last column is built, so you can check it yourself. For GLA at 1.3 billion, the six accuracies are 46.9 (LAMBADA), 71.8 (PIQA), 49.8 (HellaSwag), 53.9 (WinoGrande), 57.2 (ARC-easy) and 26.6 (ARC-challenge). They add up to 306.2. Divide by 6 and you get 51.03, which the table rounds to 51.0. For Mamba the same steps give 299.8 divided by 6, or 49.97, which rounds to 50.0. A difference of one point on this average is small next to the spread between single tasks, which ranges from 26.6 to 71.8 for one model.

Read the paper’s tables

Choose a table, a model size and a measure. The warm bar is GLA. A green outline marks the best model on that measure. Watch where GLA wins and where the Transformer still leads.

Every number is copied from Table 2, Table 3 or Table 4 of the paper (arXiv 2312.06635v6). Table 4 trains 340M variants for 7B tokens and reports the average perplexity of the last 200 training steps; its rows other than the first four vary the number of heads (4 is the default).

Length: train short, read long

A fixed-size state does not care how long the text is, so a model trained on short texts may work on long ones. Figure 5 tests this on 1.3-billion models trained on 2,000-token texts, then scored on SlimPajama and PG19 at longer positions. A Transformer cannot extrapolate beyond its training length. Mamba struggles past 4,000 tokens. GLA and RetNet generalise to 18,000 on SlimPajama, and GLA extrapolates better than Mamba and RetNet on most position buckets of PG19. The abstract's summary: a model trained on 2,000 tokens generalises to more than 20,000 without large perplexity loss.

The paper also trains on long text in two ways: directly at 8,000 tokens, or on 24,000 tokens cut into 12 segments of 2,000, passing the state across the cut and not passing gradients (truncated backpropagation through time, TBPTT). For GLA the two give almost the same perplexity, so the cheap route works. Mamba gains a lot from 8K training and then performs like GLA.

Speed and memory

Figure 6 compares 1.3-billion models on one H100 at lengths from 2,048 (batch 8) to 16,384 (batch 1). Memory use is about the same for all three, since all have linear memory. Mamba’s training throughput trails Transformer++ and GLA. GLA has a growing advantage beyond 4,096 tokens, using the materialization kernel with recomputation from Chapter 3.

Ablations: is the gate the reason?

Table 4 answers the question of Chapter 4. Perplexity falls from 23.21 (no gate) to 16.55 (fixed scalar decay), 15.56 (data-dependent scalar gate) and 14.77 (GLA). The fine-grained gate is needed on top of the data-dependent one. For heads, 8 heads (smaller heads) gives 15.29, a clear loss. One head gives 14.61, the best, but it needs much more GPU memory for a small gain, so the paper uses 4.

Limits the authors name

The authors could not train larger than 1.3 billion for lack of compute, so how GLA scales is open. They expect its speed to look better at sizes above 7 billion, because it uses tensor cores well and, being multi-head, suits tensor parallelism, which Mamba (a single head) does not. All experiments are on language. And the recall gap to softmax attention, seen in Table 3, is reduced, not closed.

Which claim does the paper’s Table 3 (recall tasks, 1.3B models) support?

Chapter 9

Field Guide: Gated Linear Attention on One Page

Keep the numbers, the terms, a checklist and the core code, and find where to read next

Here is the whole paper in one paragraph. Softmax attention keeps a growing desk of keys and values. Linear attention swaps it for a fixed notebook, a matrix S that each token updates by adding an outer product. The notebook can be run three ways, and the chunkwise way balances work against waiting. FlashLinearAttention makes the chunkwise way fast by moving few bytes between the GPU’s slow and fast memory. Gated linear attention adds a forget gate with one value per key dimension, computed from the current token. The gate breaks the simple parallel form, so the paper rewrites it in log space and splits each chunk into sub-chunks, so that most of the work still runs as half-precision matrix multiplies. The result is a model that matches strong baselines at 340 million and 1.3 billion parameters, extrapolates to long texts, and trains faster than Mamba. It still trails softmax attention on exact recall.

The numbers to remember

WhatValueWhere in the paper
Training cost of the chunkwise formabout L·C·d + L·d2, against L2·d for the parallel formSection 2.2
Speed-test settingone H100, batch 32, 16 heads, head size 64, chunk size 64Figure 2
FlashLinearAttention speedfaster than FlashAttention-2 as a layer, even at 1K tokensAbstract, Figure 2
Extra memory when states are storedabout 10 to 20 percent, mostly removed by recomputationSection 3.3
The gateGt = αtT1, with αt = σ(xtW1W2 + b)1/16, W1 of d by 16, W2 of 16 by dkEq. 3, Section 4.4
Layer shapedk = d/2, dv = d, 4 heads, about 4d2 weightsSection 4.4, Table 4
Models and data340M on 15B tokens; 1.3B on 100B tokens; SlimPajama, Mistral tokenizerSection 5.1
Average accuracy, 1.3BTransformer++ 50.9, RetNet 48.9, Mamba 50.0, GLA 51.0Table 2
FDA recall, 1.3BTransformer++ 27.4, RetNet 14.3, Mamba 6.2, GLA 19.9Table 3
Gate ablation, 340M, 7B tokensno gate 23.21, fixed decay 16.55, scalar gate 15.56, GLA 14.77 (perplexity)Table 4
Lengthtrained on 2K tokens, works beyond 20K; reaches 18K on SlimPajamaAbstract, Figure 5

The terms

TermPlain meaning
State SThe fixed-size matrix (dk by dv) that is the model’s whole memory of the past.
Outer product kTvThe page one token writes: a grid whose cell (i, j) is ki times vj.
Forget gate αA number in (0, 1) for each key dimension that scales the old state before the new write.
Chunk, sub-chunkA block of C tokens that is handled together; a smaller block of c tokens inside it.
HBM, SRAMThe GPU’s large slow memory and its small fast memory.
Tensor coreA GPU unit that multiplies small matrices very fast, in half precision.
MaterializationWriting the chunk states to HBM so the chunks can run in parallel, at the price of memory.
RecomputationThrowing the states away after the forward pass and rebuilding them in the backward pass.
Log spaceWorking with logarithms of the gate products so that every exponent is at most 0 and nothing overflows.
MQARMulti-query associative recall: store many key-value pairs, then look up several keys.

A checklist to rebuild it

  1. Reference first. Write the recurrence St = Diag(αt) St−1 + ktTvt, ot = qt St. It is slow, and it is your test oracle.
  2. Make the gate. A low-rank map through 16 numbers, a sigmoid, the 16th root. Keep it a function of xt only.
  3. Chunk it. Per chunk, find log b by a cumulative sum of log α. Make Λ, Γ and γ from differences of log b.
  4. Split the work. Inter-chunk by (Q ⊙ Λ) S, intra-chunk by Eq. 4 in log space, then update the state with (K ⊙ Γ)T V.
  5. Check. The chunkwise output must equal the recurrent output to rounding error, for any chunk size.
  6. Speed last. Add sub-chunks so the off-diagonal tiles become half-precision matmuls; keep the diagonal tiles in full precision; write the kernel so each tile is loaded once.
  7. Wrap it. Add heads, per-head layer norm, the Swish output gate and WO; set dk = d/2 and dv = d.

The device below is the check of step 5, running live. It makes random queries, keys, values and gates, runs both forms and compares them.

Two forms, one answer

The line is one output channel over 64 tokens. The pink line is the recurrent form, the warm dots are the chunkwise form. Dotted verticals mark chunk edges. Change the chunk size or the forgetting speed, or draw fresh data.

0.90

A JavaScript port of the Python below in 64-bit arithmetic: 64 tokens, dk = 4, dv = 4, gates drawn around the typical value you pick. The difference you see is rounding error. It is not a result from the paper.

The core, as code

pythonimport torch

def gla_recurrent(q, k, v, alpha):
    # q, k: (L, dk)   v: (L, dv)   alpha: (L, dk) in (0, 1). The test oracle.
    L, dk = q.shape
    S = q.new_zeros(dk, v.shape[1])
    out = []
    for t in range(L):
        S = alpha[t][:, None] * S + k[t][:, None] * v[t][None, :]   # S_t = Diag(a_t) S_(t-1) + k_t^T v_t
        out.append(q[t] @ S)                                         # o_t = q_t S_t
    return torch.stack(out)

def gla_chunkwise(q, k, v, alpha, C=16):
    # Same output, chunk by chunk. Needs L % C == 0.
    L, dk = q.shape
    S = q.new_zeros(dk, v.shape[1])
    mask = torch.tril(torch.ones(C, C, dtype=torch.bool))
    out = []
    for s in range(0, L, C):
        qc, kc, vc = q[s:s+C], k[s:s+C], v[s:s+C]
        lb  = torch.cumsum(torch.log(alpha[s:s+C]), 0)    # log b inside the chunk, (C, dk)
        lam = torch.exp(lb)                               # Lambda: decay from chunk start
        gam = torch.exp(lb[-1] - lb)                      # Gamma: decay to chunk end
        o_inter = (qc * lam) @ S                          # read the old state
        diff = (lb[:, None, :] - lb[None, :, :]).masked_fill(~mask[:, :, None], float('-inf'))
        P = (qc[:, None, :] * kc[None, :, :] * torch.exp(diff)).sum(-1)   # Eq. 4: exponent never above 0
        out.append(o_inter + P @ vc)
        S = torch.exp(lb[-1])[:, None] * S + (kc * gam).T @ vc    # hand the state on
    return torch.cat(out)

This is our own plain reading of the paper’s equations, not the authors’ kernel. We ran it: on random data with L = 64, dk = 8, dv = 12 and C = 16, the two functions differ by at most 1e-5 in 32-bit arithmetic. It uses full precision and no sub-chunks, so it shows correctness and not speed. The authors’ Listing 1 and the open-source flash-linear-attention library add the sub-chunk tiling and the fused GPU kernels.

Where it sits in the field

Table 1 of the paper shows that many recent models share one equation, St = Gt ⊙ St−1 + ktTvt, and differ only in the shape of Gt: Mamba, Mamba-2, mLSTM, Gated Retention, DFW, GateLoop, HGRN-2, RWKV-6 and GLA. The authors say their algorithm can be used or adapted for several of them. For our course, the reading belongs next to Lecture 8 on large multimodal models, where long token sequences from images, audio and video make a fixed-size memory attractive. The paper tests none of those; it tests language.

Keep going

The takeaway. A fixed-size memory only works if it can forget, and it only trains fast if the arithmetic maps onto matrix units and moves few bytes. GLA gets both by gating each key dimension from the current token, and by keeping the delicate log-space arithmetic to the small diagonal blocks of a chunk.

Credit. The paper “Gated Linear Attention Transformers with Hardware-Efficient Training” is by Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda and Yoon Kim (MIT and MIT-IBM Watson AI Lab), Proceedings of the 41st International Conference on Machine Learning, PMLR 235, 2024. Read it at arXiv 2312.06635. Every idea above is re-taught from the paper in our own words; the figures' numbers are the authors’.

Which pair of ideas lets GLA keep a data-dependent gate and still train quickly?

Now press Teach and explain from memory, on the whiteboard, how a token is written into the state, how the gate forgets, and why the chunkwise form needs log space. Then go back to the memory at the top, stream 40 chatter tokens into each of the four gates, and say which part of the paper explains each result. Return to Lecture 8 when you are ready.

Gated Linear Attention Transformers with Hardware-Efficient Training
Read the paper · flash-linear-attention code · MIT MMAI course · Lecture 8 · Back to Gleams