In a toy memory, 40 chatter tokens bury a stored fact (0.14 times the clutter). A gate that chooses what to forget keeps it at 1.5 times. This paper shows how to train that gate fast.
Learn how a language-model layer can keep a memory of fixed size, decide what to forget, and still train quickly on a GPU.
Pick a memory, then drag the chatter. Watch the three stored facts (top rows) and the clutter (bottom rows) change. Then we build, piece by piece, the memory, its three ways of running, the fast GPU kernel, the gate, the numerical trap and its fix, and the results.
You need a little matrix arithmetic (a matrix is a grid of numbers; multiplying a row by it mixes the numbers) and a little Python for the code. We build the rest from zero.
Based on “Gated Linear Attention Transformers with Hardware-Efficient Training” by Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda and Yoon Kim (MIT and MIT-IBM Watson AI Lab), ICML 2024, arXiv 2312.06635. An assigned reading of MMAI Lecture 8, Large Multimodal Models, in the MIT MMAI course. Every idea is re-taught here in our own words and devices.
An illustrative toy, not a result from the paper. A 6 by 4 memory first stores three facts (fact j writes a 1 at row j, column j), then reads chatter tokens that each write a 1 at a random one of rows 3 to 5, with a seeded random value column. The updates are the paper’s recurrences (Eq. 1 for no gate, a single constant for RetNet style, Eq. 3 for GLA). The gate values (0.9, 0.85, and 1 for facts with 0.5 for chatter) are set by hand here; in the paper the model learns them. “Recall holds” means the facts weigh at least as much as the clutter; the matrix norm is the square root of the sum of squared cells.
Chapter 0
See why a language model that rereads everything it has seen gets slower and heavier with every word, and what a fixed-size memory would change
Picture a long novel. A friend asks you about a small detail from chapter 2, and you are now in chapter 40. You have two ways to answer.
The first way: keep every page you have read open on your desk. To answer, flip back through the pages and look. This never loses a detail. But the desk grows with every page, and each new question takes longer to answer.
The second way: keep one notebook of a fixed size. After each page, you rewrite a little of the notebook. To answer, you read only the notebook. It never grows and it is always fast. But it can run out of room, and what you wrote down may be wrong or missing.
A language model faces this same choice. The paper “Gated Linear Attention Transformers with Hardware-Efficient Training” (Yang, Wang, Shen, Panda and Kim, ICML 2024) is about the notebook, and about making it good enough to replace the desk.
A Transformer reads text as a list of tokens (pieces of words). For each token, softmax attention asks: how much does each earlier token matter right now? It turns each earlier token into two vectors, a key (a label that says what the token is about) and a value (the content it can hand over). The new token makes a query (a question). Each key scores against the query, the scores are turned into weights that sum to 1, and the output is the weighted sum of the values.
When the model writes text one token at a time, it must keep every earlier key and value so that the next token can look at them. This store is the KV cache. It is the desk. It holds one key and one value for every token so far, in every layer.
Two costs follow. When the model trains, every token is compared with every earlier one, so the work grows with the square of the length L. And when the model writes, the cache grows in a straight line with L, and so does the work to read it for each new token. These two curves drive the cost of every long-context system.
The paper starts from linear attention, which replaces the exponential score of softmax with a plain dot product. Chapter 1 derives it. Here is the one fact you need first: linear attention can be run as a recurrent network, a model that carries a hidden state (a fixed-size block of numbers) from token to token. The state is the notebook. Its size does not depend on how many tokens the model has read.
Try the numbers. The device below uses one width that we pick for illustration (the model’s hidden size d is 2,048, with 4 heads, 16-bit numbers). Drag the length of the text and watch what one layer must keep in each design.
Both axes are logarithmic. The warm line is the KV cache of one softmax-attention layer. The purple line is the state of one linear-attention layer. Drag the length and find where the lines cross.
Illustrative widths, not a model from the paper: hidden size d = 2,048, 4 heads, key size d/2 and value size d (the shapes Chapter 7 uses), 2 bytes per number. KV cache per layer = 2 × L × d × 2 bytes. State per layer = 4 heads × 256 × 512 numbers × 2 bytes = 1 MiB. Per-token read work is counted in multiply-adds: 2 × L × d for attention, 524,288 for the state.
The lines cross at 128 tokens. Past that, the cache is bigger than the notebook, and the gap widens by a factor of 2 each time the text doubles. At a million tokens the cache of one layer holds 8 GiB and the notebook still holds 1 MiB. The work to read it for one new token follows the same two curves, because the cache must be read in full while the notebook is read once.
Take the width of the device and a text of 8,192 tokens, and count what one layer does to write the next token.
The symptom to watch for in real systems is a model that slows down and needs more memory the longer a conversation or a video gets. The measurement is the cache size above. The fix this paper pursues is the notebook. But a notebook changes the model, so the question is whether the changed model is still good, and whether it can be trained as fast. That is the rest of the lesson.
The paper names two reasons, and its two contributions answer them one by one.
The results (Chapter 8) are for models of 340 million and 1.3 billion parameters, trained on 15 and 100 billion tokens. GLA competes with a strong LLaMA-style Transformer (Transformer++) and with the recent linear-time models RetNet and Mamba. A model trained on 2,000-token texts keeps working on texts longer than 20,000. And GLA trains with higher throughput than a Mamba model of the same size.
This paper is an assigned reading for Lecture 8 on large multimodal models. Images, audio and video turn into very long token lists, so the cost of reading a long past matters a great deal there. The paper itself says, in its closing section, that it would be interesting to apply GLA to other kinds of data, especially data with long-range dependencies. It runs no such experiment. We stay with what it tests: language.
Chapter 1
Write each word into a matrix with an outer product, read it back with a query, and see why this is a recurrent network in disguise
Imagine a small filing cabinet with two drawers, labelled “topic A” and “topic B”. As you read, each fact you learn goes into the drawers. A fact that is mostly about topic A goes mostly in drawer A. A fact about both topics is split between them. To recall something, you ask about a topic and read what the drawers hold.
That cabinet is the whole idea of this chapter. The two drawers are the rows of a small matrix. We will write to it, read from it, and then show that this is exactly what a recurrent network does.
Let xt be the vector for token t (a row of d numbers). Softmax attention makes three new vectors from it with three learned weight matrices: the query qt, the key kt and the value vt. The output for token t is a weighted average of all values so far. The weight of value i is exp(qtkiT), divided by the sum of these weights over all i up to t. The exponential makes every weight positive, and the division makes them sum to 1.
Two things make this expensive. The exponential must be taken for every pair (t, i). And the sum over i needs every earlier key and value, which is the KV cache from Chapter 0.
Linear attention (Katharopoulos et al., 2020) replaces exp(qtkiT) with a simpler score, a kernel: the dot product of two transformed vectors, φ(qt)φ(ki)T. The gain is a regrouping trick. A dot product of two sums can be rearranged so that all the keys and values are folded into one running object before any query arrives.
The paper writes that object as St, the sum over i up to t of φ(ki)Tvi, plus a second running sum zt used for dividing. It then notes that recent work (the RetNet paper) found that you can set φ to the identity and drop the division, and it works well in practice. We follow the paper. Now each step is just two lines.
This is a linear recurrence: the new state is the old state plus something. The state is a matrix, so the paper calls it a linear RNN with a 2D (matrix-valued) hidden state. It adds that this kind of model is also known as fast weights, because the matrix changes with the data while the input flows through it, as the model’s own weights do during learning.
Take drawers of size 2 by 2, and three tokens. The keys are k1 = (1, 0), k2 = (0, 1), k3 = (1, 1). The values are v1 = (2, 1), v2 = (0, 3), v3 = (1, 1). Step through them in the device below. Here is the arithmetic it shows.
Press Write next token to add a page to the memory. Choose a query to read with. The check line adds up the values by hand and must match.
Toy numbers chosen for easy arithmetic (illustrative, not from the paper). The state update and the read follow Eq. 1 of the paper exactly. The check line is the parallel form of the next chapter: output = sum over written tokens i of (q · ki) times vi.
Stack all L queries as the rows of a matrix Q (shape L by dk), and likewise K (L by dk) and V (L by dv). Unrolling the recurrence gives the parallel form: O = ((Q KT) ⊙ M) V. The product Q KT is an L by L grid of scores. The mask M holds a 1 where the row is at or after the column, and a 0 elsewhere, so no token reads the future. The sign ⊙ means “multiply cell by cell”.
Look at the check line in the device. For token 3 and q = (1, 0), the scores against the three keys are 1, 0 and 1. Weighted values: 1 × (2, 1) + 0 × (0, 3) + 1 × (1, 1) = (3, 2). Same answer as the recurrence. Two roads, one result.
One footnote in the paper deserves a pause. Without the mask, you could regroup (Q KT) V as Q (KT V) and the cost would drop from L squared times d to L times d squared. With the mask, you cannot. The mask cuts a different slice of keys for each row, so there is no single KTV to share. This is why the paper needs a third form, in Chapter 2.
Watch the memory in the device once more. Nothing ever leaves. Each page adds to the sum. This has a symptom you can see in the example: reading drawer 0 after token 3 returns (3, 2), which is a blend of token 1 and token 3. If you wanted only token 1’s value (2, 1), you cannot get it back, because token 3 wrote on top of it.
To measure the problem, count the room. The state holds dk rows. If the keys of the tokens were perfectly separate (orthogonal, meaning their dot products are 0), you could store dk facts with no blending. With a longer text, there are more tokens than rows, so keys must overlap and every read mixes several writes. In the 2 by 2 example, three keys cannot all be separate.
The fix the paper builds towards is to let the notebook forget. Old pages fade, so that new ones have room. Chapter 4 adds that. First, we need to know how to run this layer fast, because the way we run it will decide how much freedom the fix gets.
Chapter 2
Compare the parallel, recurrent and chunkwise forms, and find the chunk size that balances work against waiting
You must add up 2,048 receipts, and you have a crowd of helpers. There are three plans.
Plan one: one person keeps a running total and reads the receipts one at a time. Almost no arithmetic is wasted, but 2,047 helpers stand idle, because each step needs the last total. Plan two: write every receipt in one huge table and have all helpers fill in every cell at once. Everyone is busy, but the table is far bigger than the job. Plan three: cut the pile into groups of 64. Inside each group, the helpers work all at once. Between groups, one running total is passed along. This is the plan that wins, and it is the one the paper uses.
The three plans are the three forms of a linear-attention layer. All three give the same output. They differ in how much arithmetic they do and how much of it can run at the same time.
A GPU is a chip with thousands of small workers, called threads. Threads are grouped into blocks, and each block runs on one streaming multiprocessor (SM). A chip has many SMs. Occupancy is the share of them that are busy. If your job is one long serial chain, most SMs sleep. The paper points out that when the batch size is small, which is common in long-sequence training, the only way to keep the chip busy is to split work along the sequence itself.
A GPU also has special units for one job: multiplying matrices. On NVIDIA chips they are called tensor cores. The paper notes that a half-precision matrix multiply on an A100 runs roughly 16 times faster on tensor cores than on the ordinary cores. They work best on blocks whose sizes are multiples of 16. Any algorithm that wants to be fast must turn most of its work into matrix multiplies.
From Chapter 1, O = ((Q KT) ⊙ M) V. Every step is a matrix multiply or a cell-wise mask, so it fits tensor cores. All L outputs come out together. The price is the L by L score grid. The work is about L2 times d, the quadratic cost we wanted to escape.
The recurrence of Chapter 1 does about L times d2 multiply-adds in all, because each step touches the d by d state a few times. That is the lowest total of the three. The paper explains why it still loses on a real chip. The update is an outer product plus an addition, which are cell-wise operations. They cannot use tensor cores, and they do very little arithmetic per byte moved. And the L steps run one after another.
You might hope to run the recurrence in parallel with a parallel scan (a way to combine pairwise sums in a tree). The paper warns that this needs the d by d state of every time step to be written out, so the memory traffic cancels the gain.
Cut the sequence into N = L / C blocks of chunk size C. Let S[i] be the state after i chunks, and let Q[i], K[i], V[i] hold the C rows of chunk i. The paper gives two equations. The first passes the state from chunk to chunk. The second computes the outputs of a chunk in parallel.
Add the costs over N chunks. The inter-chunk work is N times C d2, which is L d2. The intra-chunk work is N times C2 d, which is L C d. The total is of order L C d + L d2, and the paper notes this is below L2 d when L is larger than d. Set C = L and you get the parallel form. Set C = 1 and you get the recurrent form. C is the knob.
Bars are on a log scale. Top: total multiply-adds for one head. Bottom: the steps that must run one after another. Drag the chunk size and compare it with the two ends.
Operation counts use the paper’s orders of growth, work = L · C · d + L · d2, for one head with L = 2,048 tokens and head size d = 64 (the paper’s speed tests use head size 64 and chunk size 64). The counts leave out constants and say nothing about wall-clock time, which depends on tensor cores and memory (Chapter 3).
Run the numbers with the device. With C = 64 and L = 2,048 there are 32 chunks. The total is 2,048 × 64 × 64 + 2,048 × 642, which is 8.4 million plus 8.4 million, or 16.8 million multiply-adds. The parallel form needs about 277 million. The recurrent form needs about 8.5 million but 2,048 steps in a row. The chunkwise form pays twice the minimum work and cuts the serial chain by a factor of 64. And nearly all of its work is matrix multiplication, which the tensor cores love. The paper says a chunk size that is a multiple of 16 lets it use them.
One failure to keep in mind: the form is only as fast as its code. The paper says most earlier chunkwise implementations were not I/O-aware, so they were slower than FlashAttention for moderate lengths of 2,000 to 4,000 tokens. Counting multiply-adds is not enough. The next chapter fixes the memory traffic.
Chapter 3
Follow one chunk through the GPU’s fast and slow memory, and choose whether to store the state or recompute it
A cook works at a small counter. Everything on the counter is within reach, and a knife moves in a blink. The pantry is down the hall. It is huge, but each trip takes a long time. A slow cook makes many trips and chops a little each time. A fast cook plans: carry in everything one dish needs, do all the work at the counter, and walk back once.
A GPU works the same way. This chapter is the plan for the cook.
The paper names the two. HBM (high-bandwidth memory) is the GPU’s main memory: large, but slower to reach. SRAM (also called shared memory) is a small block next to each processor: fast, but tiny. An I/O-aware algorithm counts the bytes it moves between the two, and plans its loops to move as few as it can. The classic tool is tiling: cut the data into blocks, load one block into SRAM, do all the work that block allows, and write the result back once. FlashAttention, which the paper takes as its model, is built this way for softmax attention.
For the chunkwise form of Chapter 2, tiling is natural. A chunk is a block. When the chunk of queries Q[n] arrives in SRAM, two products need it: the inter-chunk read Q[n] S and the intra-chunk part (Q[n] K[n]T ⊙ M) V[n]. Done together, Q[n] is loaded once, not twice. That is the saving in one sentence.
The paper calls the first version non-materialization, because the state is never written to HBM. It is Algorithm 1 of the paper, and its data flow is short.
This version is thrifty. It reads each input once and writes each output once. But the chunks run one after another, because chunk n+1 needs the S that chunk n leaves behind. The work spreads over the batch, the heads and the head dimension, and that is all. The paper says plainly what follows: when the batch is small, as in long-sequence training, many streaming multiprocessors have nothing to do.
The second version is materialization. It splits the job in two. First, a short loop goes over the chunks in order and does only the cheap part: it loads K[n] and V[n], adds K[n]T V[n] to S, and stores S[n] in HBM. Second, once every S[n] is on the shelf, all chunks can be computed at the same time: each one loads its own Q, K, V and S[n] and writes its O. The paper marks this second loop as a parallel for.
Now the sequence itself is a source of parallel work. The cost is memory. Storing the states raises the memory footprint by about 10 to 20 percent. The paper removes most of it with recomputation: the states are thrown away after the forward pass and rebuilt during the backward pass. This costs a little time and saves a lot of memory, and the paper adopts it by default.
The device below turns this into something you can see. Eight lanes stand for eight processors. A job is one sequence in one head, cut into 8 chunks. The costs are illustrative: a chunk costs 4 time units when everything is done together, but only 1 unit for the cheap state step and 3 units for the output step when they are split.
Each row is a processor. Purple blocks pass the state along; warm blocks compute outputs. Choose a version, then raise the number of independent jobs (batch times heads) and watch how many rows stay empty.
An illustrative schedule, not a measurement. Costs: 4 units per chunk fused; 1 unit per chunk for the state pass and 3 for the output step when split. Eight processors, eight chunks per job, a barrier between the two stages. The two versions, the 10 to 20 percent extra memory and the recomputation default are from Section 3.3 of the paper.
Try one job. The first version keeps one row busy for 32 units and leaves seven asleep: 12.5 percent of the chip. The second version runs the state pass in 8 units, then spreads 8 chunk tasks over all 8 rows for 3 units: 11 units in all, with 36 percent of the chip busy. Now raise the jobs to 8. The first version fills every row by itself and takes 32 units. The second takes 8 + 3 × 8 = 32 units too, and also holds extra states in memory. So the second version helps exactly when the batch is too small to keep the chip full. The paper chooses between them by the training setting.
Figure 2 of the paper compares speed and memory on one H100 GPU: batch size 32, 16 heads, head size 64, chunk size 64. FlashLinearAttention, in either version, is faster than FlashAttention-2 as a standalone layer, even at a length of 1,000 tokens. A plain PyTorch version of the same chunkwise algorithm, which does not plan its memory traffic, is slower. The paper reads this gap as the benefit of I/O-awareness.
The paper also gives a backward pass (its Algorithm 2) built on the same ideas. And it tells us why the gated layer of the next chapters is harder: the gate breaks one assumption that this chapter leans on.
Chapter 4
Add a forget gate that depends on the data, and compare the ways to shape it by what they can express and what they cost
Back to the filing cabinet from Chapter 1. Reading a long story, you keep filing facts into the same drawers. After a while drawer A holds the hero’s plans from chapter 1, chapter 5, chapter 9, all mixed. When you ask what the hero wants right now, the old plans muddy the answer.
A smart clerk would do two things. Fade old pages a little as new ones arrive. And decide what to fade based on what is being filed: a new chapter on a new topic should wipe the drawers of the old topic, but leave unrelated drawers alone. That clerk is a forget gate.
The update of Chapter 1 is St = St−1 + ktTvt. It has no decay term. The paper recalls that a forget gate has long been seen as crucial in recurrent networks (it is the core of the LSTM and the GRU), and that its absence is thought to be partly why linear attention is unstable on long contexts.
The first fix, used by RetNet and TransNormerLLM, is one fixed number γ between 0 and 1: St = γ St−1 + ktTvt. Every token multiplies the whole memory by the same γ, so old writes fade smoothly. This is a global, data-independent decay. It keeps the attention-style parallel form, which is why it is attractive. But it fades every drawer at the same speed, whatever is being filed. The paper notes that in ordinary RNNs, gates that depend on the data are known to matter a great deal.
GLA lets the decay depend on the current token and gives it a shape. The recurrence becomes St = Gt ⊙ St−1 + ktTvt, where Gt is a dk by dv matrix of numbers between 0 and 1, computed from the token. The sign ⊙ means “multiply cell by cell”: each cell of the memory is scaled by its own gate. A gate of 1 keeps a cell, a gate near 0 erases it.
Worked example. Take the state S = [[3, 2], [1, 4]] from Chapter 1 and a gate G = [[1, 1], [0.5, 0.5]]. The first row has gate 1, so it stays [3, 2]. The second row has gate 0.5, so it becomes [0.5, 2]. A token with k = (0, 1) and v = (1, 1) then adds [[0, 0], [1, 1]]. The new state is [[3, 2], [1.5, 3]]. Drawer 0 was kept intact, drawer 1 was half erased and then refilled.
The paper points out that this one equation covers many recent models, and Table 1 lists them. They differ only in how they compute Gt from the input.
A full gate matrix computed from xt would need d × dk × dv weights. That is far too many. Every design in Table 1 is a compromise between three things the paper names: how many parameters, how large the state can be, and whether it trains efficiently. Here are the main choices.
Use the device to feel the difference. You ask each gate to erase drawer 0 and keep the others. See which one can do that without harming the rest.
The grid is the gate Gt for an 8 by 8 memory. Bright pink is 1, keep; dark is near 0, erase. Drag the strength, then switch the kind of gate.
Illustrative: the gate values are what each shape would have to output to follow the request “scale drawer 0 by 1 minus the strength and leave the rest”; a trained model learns its own. The weight counts are the products in the paper’s Table 1 for the sizes d = 1,024, dk = 512, dv = 1,024: scalar d; GLA low-rank 16d + 16dk; outer d(dv + dk); full d · dk · dv. Mamba’s real gate adds a learned matrix A and cannot use tensor cores.
Set the strength to 80 percent. The scalar gate has no row to aim at, so it must scale all 64 cells by 0.2: drawer 0 is erased as asked, and seven drawers of good memory are erased with it. The vector, outer and full gates erase only drawer 0. The vector gate needs 24,576 weights; the outer product needs 1,572,864; the full matrix needs 536,870,912. For this request all three row-aware shapes give the same result, and the vector gate gets there with the fewest weights.
The paper tests this in an ablation (Table 4). It trains four 340-million-parameter variants on 7 billion tokens and reports the average perplexity of the last 200 steps; lower is better. With no gate (plain linear attention): 23.21. With a fixed scalar decay (RetNet): 16.55. With a data-dependent scalar gate: 15.56. With GLA’s fine-grained data-dependent gate: 14.77. Each step down the list closes part of the gap, and the paper concludes that a finer-grained gate is still necessary after the data-dependent scalar. Chapter 8 returns to this table.
The cost of the vector gate is that the gated layer is harder to train fast. The simple parallel form of Chapter 1 no longer applies, and the next chapter shows why.
Chapter 5
Unroll the gated recurrence into attention form, watch a cumulative product overflow, and fix it by moving to log space
A shop loses 5 percent of its value every year to inflation. After 200 years, one coin from year 1 is worth almost nothing: its value is the product of 200 factors of 0.95. Now someone asks you to compare a coin from year 150 with one from year 160. A clumsy method would first convert every coin to year-200 money by dividing by the tiny product, then compare. Those divisions create gigantic numbers. A computer with a limited number format cannot hold them.
This chapter is that story for the gated memory. It has a neat matrix form, a numerical trap, and a fix that costs speed. The next chapter wins the speed back.
Chapter 4 ended with GLA’s update St = Diag(αt) St−1 + ktTvt. Here αt is a row of dk numbers in (0, 1), one forget rate per key dimension. Write the state at time t by unrolling: the write from token i has been scaled by the gates of every token after it, up to t. The scale for key dimension m is the product of αj[m] for j from i+1 to t.
Give that product a name. Let bt be the running product of all gates up to token t, taken per dimension: bt = α1 ⊙ α2 ⊙ … ⊙ αt. The product from i+1 to t is then bt divided by bi, per dimension. So the output of token t is a sum over earlier tokens i, weighted by the ratio bt / bi.
This is the paper’s parallel form of GLA. It has the shape of Chapter 1’s formula. The score grid is a matrix product of (Q ⊙ B) and (K / B)T. Check it on one dimension with every gate equal to 0.5. Then b = (0.5, 0.25, 0.125). The weight of token 1 in the output of token 3 is b3 / b1 = 0.125 / 0.5 = 0.25. That is right: token 1’s write was halved twice, by tokens 2 and 3.
Look at K / B. The running product bt is a chain of numbers below 1, so it shrinks with t. Its inverse 1 / bt grows without limit. The paper says it directly: the form is not numerically stable, because K / B can explode when t is large.
The size of the problem depends on the number format. In 16-bit floating point (the half precision that tensor cores like), the largest number is 65,504. In 32-bit, it is about 3.4 × 1038. If α = 0.95 on every step, then 1 / bt = (1 / 0.95)t. Take logarithms: log10(1 / bt) = t × 0.0223. It passes log10(65,504) = 4.82 at t = 217. Past that, K / B is infinity in half precision, and infinity times zero gives an invalid number that spreads through the whole network.
The warm line is how big 1 / bt is, as a power of ten. The red lines are the largest numbers that 16-bit and 32-bit formats can hold. Choose a gate value, then drag the position t.
Exact arithmetic for one key dimension whose gate is the same number at every step. Largest finite 16-bit float 65,504; largest 32-bit float about 3.4 × 1038. Real gates vary by token and dimension, so the crossing point in a model moves, but the growth is the same: 1 / b multiplies by 1 / α at every step.
The paper adds a temperature to its gate so that forgetting is slow (Chapter 7). Even so, a gate of 0.99 puts the 16-bit limit at position 1,104, and a gate of 0.9 pulls it in to 106. Chunks of 64 tokens hit the wall for fast-forgetting dimensions, and a context of thousands of tokens hits it for slow ones. There is no safe setting.
The cure is never to build b and 1 / b separately. Only their ratio is needed, and the ratio bt / bi is at most 1 when t is at or after i. Work with logarithms: the log of a ratio is a difference, and the difference of two logs is always zero or negative. The paper writes the score grid entry by entry as Eq. 4.
Pij = ∑k=1dk Qik Kjk exp( log Bik − log Bjk ), for i ≥ j
In this sum, every exponent is at most 0, so every factor is between 0 and 1. Nothing can overflow. For the one-dimension example, P31 = q3 k1 exp(ln 0.125 − ln 0.5) = q3 k1 × 0.25. The same answer, reached safely.
Now the price. The matrix form multiplied two matrices, and that is what tensor cores do fast. Eq. 4 sums over a third index k with an exponential of a difference inside the sum. It is not a matrix product, so it cannot use half-precision matmuls on tensor cores. It runs on the slow general cores in full precision. Done for the whole L by L grid, it would erase the speed gain of Chapters 2 and 3. The paper states this trade clearly, and it needs a second idea: use the safe slow path only where it is needed. That is the next chapter.
Chapter 6
Split the work into three kinds of tile, and count how much of it can use the GPU’s fastest matrix units
A bank clears thousands of cheques a day. A fast machine handles the ordinary ones, a hundred a second. A few cheques look odd, and a careful clerk must read them by hand. The bank cannot afford to send everything to the clerk. So it sorts: the fast machine takes everything it safely can, and the clerk sees only the small pile that needs care.
That is the plan for gated linear attention. Chapter 5 found a safe but slow path (log space, full precision). This chapter makes the slow pile small. Two ideas do it: a gated version of the chunkwise form, and a second level of chunking inside each chunk.
Take the chunkwise form of Chapter 2, and add the gates. Within chunk i, positions are numbered j = 1 to C, so the token at position j has global index iC + j. The paper defines three ratios of running products b. Each one is a product of at most C gates.
All three are numbers between 0 and 1, because each is a product of gates. None of them is a ratio with a huge denominator. This is why they are safe in half precision.
Worked example, one dimension with every gate 0.5 and C = 4. Then Λ = (0.5, 0.25, 0.125, 0.0625) for positions 1 to 4. Γ = (0.125, 0.25, 0.5, 1). And γ = 0.0625. Check against the token-by-token rule: the first write of the chunk is halved by gates 2, 3 and 4, so it survives at 0.53 = 0.125, which is Γ1. The last write is not faded at all, which is Γ4 = 1. The first token of the next chunk reads the old state after one halving: Λ1 = 0.5. Everything matches.
So the inter-chunk part is solved: it uses only matrix multiplies, which can run in half precision on tensor cores. Only the intra-chunk part is left, the chunk’s own C by C grid of token pairs. That grid still needs Eq. 4 and its slow path.
The paper’s answer is secondary-level chunking: cut each chunk of C tokens into sub-chunks of c tokens, and treat the C by C grid as a grid of small blocks. There are two kinds of block.
The paper summarises the result: with the two-level tiling, the number of non-half-precision matrix-multiply operations is greatly reduced, which gives the wall-clock gain. The device counts it for you. The left grid is a toy of 64 tokens with chunks of 16. The bar compares it with the real shape of one head at 2,048 tokens and chunks of 64.
Each square is a block of the lower-triangle attention grid. Gray: read through the state (inter-chunk). Orange: half-precision matmul between sub-chunks. Pink: the small slow blocks on the diagonal. Choose a sub-chunk size.
Counts are exact for the stated sizes: cells of the lower triangle are L(L+1)/2; each chunk holds C(C+1)/2 cells; each sub-chunk diagonal block holds c(c+1)/2. The toy uses L = 64, C = 16; the large case uses L = 2,048, C = 64 with the same c. The paper does not state the sub-chunk size in the text we use here, so c is a free choice in this device.
With c = 4 in the toy, about 7.7 percent of all cells fall in slow blocks, 18.5 percent are orange, and 73.8 percent go through the state. In the large case with c = 16, the slow blocks are under 1 percent of all cells (0.83 percent). Nearly 97 percent of cells never form a score at all, because the state carries them. Another 2.3 percent are orange matmuls. The careful clerk reads less than one cheque in a hundred.
The paper’s Listing 1 is a PyTorch-style outline. Its steps for each chunk are these.
One more result from this section matters for training. Earlier work claimed that GLA-style models must store the d by d states of every step to compute the gradient of the gate. The paper instead gives a closed form for the gate gradient, built from the gradients of q and k and a sum from the right, and it needs no stored states. (The sign convention differs between the main text and the appendix; the structure is the same.)
Chapter 7
Trace one token through a GLA layer, check every tensor shape, and see why the layer costs the same parameters as softmax attention
Follow one word through a small factory. It arrives as a row of numbers. At the first station it is turned into three new rows: a question, a label and a content. At the second station a gate decides how much of the shelf to clear. At the third, the content is filed and the question is answered from the shelf. At the last, the answer is cleaned, switched on or off by a valve, and sent out.
This chapter walks that factory for the GLA layer and then places it in a Transformer. The goal is simple: you could write this layer after reading it.
Take the layer's input for token t, a row xt of d numbers. A GLA layer has H heads. A head is an independent copy of the memory with its own smaller queries, keys and values, so the model can keep H separate notebooks. The key size per head is d′k = dk / H and the value size per head is d′v = dv / H. For head h, the paper writes:
The gate itself is made in two small steps. First a low-rank map: xt Wα1 Wα2, with Wα1 of shape d by 16 and Wα2 of shape 16 by dk. A low-rank map squeezes through a narrow middle (here 16 numbers), which saves weights. Second, a sigmoid squeezes each number into (0, 1), and the result is raised to the power 1 / τ with τ = 16. The paper says this temperature encourages a slower forgetting rate.
Try numbers. If the sigmoid gives 0.5, the 16th root is 0.958. If it gives 0.1, the root is 0.866. If it gives 0.01, the root is 0.750. Even a sigmoid output close to zero leaves a gate of about 0.75, so no single token wipes a drawer clean. The device lets you try any value.
Use illustrative sizes: d = 1,024, H = 4 heads, dk = 512 (half of d) and dv = 1,024 (equal to d). Per head that is d′k = 128 and d′v = 256. The paper makes these two choices on purpose, and the stepper below counts the weights at each stage.
Press Next stage to move the token along the line. Drag the gate slider to see how the sigmoid and the 16th root turn a score into a forget gate.
Sizes d = 1,024, H = 4, dk = d/2, dv = d are illustrative; the relations (dk = d/2, dv = d, low-rank 16, τ = 16, 4 heads by default) are the paper’s. The count leaves out the biases and the layer-norm scales. Shapes are for one token and one head unless a line says otherwise.
At the end, the layer has used 4,218,880 weights, against 4 × 1,0242 = 4,194,304 for a softmax attention layer of the same width (four d by d matrices WQ, WK, WV, WO). The extra output gate Wr costs a full d by dv matrix, so the paper pays for it by halving the key size: WQ and WK are d by d/2 each. The low-rank gate adds only 24,576 weights, 0.6 percent. The paper states this budget: one GLA layer needs roughly 4d2 parameters, as softmax attention does.
The number of heads sets how big the state is. Count it with the sizes above. With 4 heads, each state is 128 by 256, so the layer holds 4 × 32,768 = 131,072 numbers. With 8 heads, each state is 64 by 128, so the layer holds 8 × 8,192 = 65,536 numbers: half the memory. With 1 head the state is 512 by 1,024, or 524,288 numbers: four times as much.
This simple count lines up with the paper's ablation (Table 4, 340M models): 8 heads gives a perplexity of 15.29, 4 heads 14.77, and 1 head 14.61. A bigger state remembers more, and a smaller one forgets sooner. But the single head needs much more GPU memory for a small gain, so the paper keeps 4. The count is our arithmetic, not a claim from the paper; the paper only reports the three perplexities.
The block has the usual two halves. First, layer-normalise the stream, apply GLA, and add the result back (a residual connection). Second, layer-normalise, apply a feed-forward network called SwiGLU, and add that back. SwiGLU(Z) = (Swish(Z W1) ⊙ Z W2) W3, the same feed-forward net as the LLaMA family. Stack many blocks and you have the GLA Transformer.
At inference, each new token needs only the state: compute q, k, v and α for the token, scale S rows by α, add kTv, read q S. Memory and work do not grow with the text length. At training time, the whole text goes through the chunkwise algorithm of Chapters 3 to 6.
The paper’s recipe, all from Section 5.1: SlimPajama data (a 100-billion-token subset of 627 billion), the Mistral tokenizer, AdamW with a top learning rate of 3e-4, a cosine schedule with a warmup (0.5 billion tokens for the 340M models, 1 billion for the 1.3B models) that starts and ends at 3e-5, weight decay 0.01, gradient clipping at 1.0. The 340M models see 15 billion tokens with batches of 0.5 million tokens. The 1.3B models see 100 billion tokens with batches of 2 million. All baselines train on the same data for the same number of tokens.
Chapter 8
Read each table and figure of the paper, see where GLA wins, where it loses, and what the ablation says about the gate
You have a new engine design. Before you trust it, you would run it on a test track, against the best engine you know and against two rivals, and you would ask three questions. Is it as strong? Does it fail somewhere the others do not? Does it cost less to run? The paper answers these with language models, a recall test, a long-text test and a speed test.
One rule makes the comparison fair. All models train from scratch on the same data (a 100-billion-token subset of SlimPajama, with the Mistral tokenizer) for the same number of tokens. There are two sizes: 340 million parameters on 15 billion tokens, and 1.3 billion parameters on 100 billion tokens. The three baselines are Transformer++ (the LLaMA recipe: rotary positions, SwiGLU, RMSNorm), RetNet (fixed decay) and Mamba (the open-source code).
The table reports perplexity on WikiText and LAMBADA (lower is better) and zero-shot accuracy on LAMBADA, PIQA, HellaSwag, WinoGrande, ARC-easy and ARC-challenge. The last column is the average accuracy. At 1.3 billion, GLA has the best average of the four, 51.0, against 50.9 for Transformer++, 50.0 for Mamba and 48.9 for RetNet. At 340 million, Mamba leads with 41.8, then GLA 41.5, Transformer++ 41.2 and RetNet 41.0. The gaps are small. The honest reading is that GLA matches the Transformer++ and Mamba on these tasks and beats the fixed-decay RetNet.
The perplexity columns tell a slightly different story. On WikiText at 1.3 billion, Transformer++ gets 16.85, Mamba 17.06, GLA 17.22 and RetNet 18.64. GLA trails the first two by a little. The paper’s words are “competitively” and “respectably”, and the numbers support those words and no stronger ones.
A fixed-size state cannot hold everything. The paper tests that on tasks that need exact retrieval. The synthetic task MQAR (multi-query associative recall) shows the model many key-value pairs, then asks for the value of a key several times. Softmax attention scores perfectly in every setting. Among the others (Figure 4, model sizes 64 to 512, sequences of 256 and 512, with 16 and 64 pairs), models with a matrix-valued state (Mamba, RetNet, GLA) beat the vector-state models Hyena and RWKV-4, and GLA beats RetNet, which the paper takes as evidence for data-dependent gates.
Table 3 uses three real tasks: FDA and SWDE (pulling a field out of a web page or form) and SQuAD (reading comprehension). Work through the 1.3-billion column. Transformer++ scores 27.4 on FDA. Mamba scores 6.2, which is 23 percent of that. RetNet scores 14.3, 52 percent. GLA scores 19.9, 73 percent.
On SWDE the Transformer scores 66.6; GLA 50.6 (76 percent), RetNet 42.8, Mamba 41.4. On SQuAD, GLA scores 42.6, ahead of all three: Mamba 35.2, RetNet 34.7 and the Transformer 31.5. The paper explains GLA’s edge among linear models by its larger state than Mamba’s and its selective gate compared with RetNet’s. The gap to the Transformer on FDA and SWDE remains open.
Here is how the last column is built, so you can check it yourself. For GLA at 1.3 billion, the six accuracies are 46.9 (LAMBADA), 71.8 (PIQA), 49.8 (HellaSwag), 53.9 (WinoGrande), 57.2 (ARC-easy) and 26.6 (ARC-challenge). They add up to 306.2. Divide by 6 and you get 51.03, which the table rounds to 51.0. For Mamba the same steps give 299.8 divided by 6, or 49.97, which rounds to 50.0. A difference of one point on this average is small next to the spread between single tasks, which ranges from 26.6 to 71.8 for one model.
Choose a table, a model size and a measure. The warm bar is GLA. A green outline marks the best model on that measure. Watch where GLA wins and where the Transformer still leads.
Every number is copied from Table 2, Table 3 or Table 4 of the paper (arXiv 2312.06635v6). Table 4 trains 340M variants for 7B tokens and reports the average perplexity of the last 200 training steps; its rows other than the first four vary the number of heads (4 is the default).
A fixed-size state does not care how long the text is, so a model trained on short texts may work on long ones. Figure 5 tests this on 1.3-billion models trained on 2,000-token texts, then scored on SlimPajama and PG19 at longer positions. A Transformer cannot extrapolate beyond its training length. Mamba struggles past 4,000 tokens. GLA and RetNet generalise to 18,000 on SlimPajama, and GLA extrapolates better than Mamba and RetNet on most position buckets of PG19. The abstract's summary: a model trained on 2,000 tokens generalises to more than 20,000 without large perplexity loss.
The paper also trains on long text in two ways: directly at 8,000 tokens, or on 24,000 tokens cut into 12 segments of 2,000, passing the state across the cut and not passing gradients (truncated backpropagation through time, TBPTT). For GLA the two give almost the same perplexity, so the cheap route works. Mamba gains a lot from 8K training and then performs like GLA.
Figure 6 compares 1.3-billion models on one H100 at lengths from 2,048 (batch 8) to 16,384 (batch 1). Memory use is about the same for all three, since all have linear memory. Mamba’s training throughput trails Transformer++ and GLA. GLA has a growing advantage beyond 4,096 tokens, using the materialization kernel with recomputation from Chapter 3.
Table 4 answers the question of Chapter 4. Perplexity falls from 23.21 (no gate) to 16.55 (fixed scalar decay), 15.56 (data-dependent scalar gate) and 14.77 (GLA). The fine-grained gate is needed on top of the data-dependent one. For heads, 8 heads (smaller heads) gives 15.29, a clear loss. One head gives 14.61, the best, but it needs much more GPU memory for a small gain, so the paper uses 4.
The authors could not train larger than 1.3 billion for lack of compute, so how GLA scales is open. They expect its speed to look better at sizes above 7 billion, because it uses tensor cores well and, being multi-head, suits tensor parallelism, which Mamba (a single head) does not. All experiments are on language. And the recall gap to softmax attention, seen in Table 3, is reduced, not closed.
Chapter 9
Keep the numbers, the terms, a checklist and the core code, and find where to read next
Here is the whole paper in one paragraph. Softmax attention keeps a growing desk of keys and values. Linear attention swaps it for a fixed notebook, a matrix S that each token updates by adding an outer product. The notebook can be run three ways, and the chunkwise way balances work against waiting. FlashLinearAttention makes the chunkwise way fast by moving few bytes between the GPU’s slow and fast memory. Gated linear attention adds a forget gate with one value per key dimension, computed from the current token. The gate breaks the simple parallel form, so the paper rewrites it in log space and splits each chunk into sub-chunks, so that most of the work still runs as half-precision matrix multiplies. The result is a model that matches strong baselines at 340 million and 1.3 billion parameters, extrapolates to long texts, and trains faster than Mamba. It still trails softmax attention on exact recall.
| What | Value | Where in the paper |
|---|---|---|
| Training cost of the chunkwise form | about L·C·d + L·d2, against L2·d for the parallel form | Section 2.2 |
| Speed-test setting | one H100, batch 32, 16 heads, head size 64, chunk size 64 | Figure 2 |
| FlashLinearAttention speed | faster than FlashAttention-2 as a layer, even at 1K tokens | Abstract, Figure 2 |
| Extra memory when states are stored | about 10 to 20 percent, mostly removed by recomputation | Section 3.3 |
| The gate | Gt = αtT1, with αt = σ(xtW1W2 + b)1/16, W1 of d by 16, W2 of 16 by dk | Eq. 3, Section 4.4 |
| Layer shape | dk = d/2, dv = d, 4 heads, about 4d2 weights | Section 4.4, Table 4 |
| Models and data | 340M on 15B tokens; 1.3B on 100B tokens; SlimPajama, Mistral tokenizer | Section 5.1 |
| Average accuracy, 1.3B | Transformer++ 50.9, RetNet 48.9, Mamba 50.0, GLA 51.0 | Table 2 |
| FDA recall, 1.3B | Transformer++ 27.4, RetNet 14.3, Mamba 6.2, GLA 19.9 | Table 3 |
| Gate ablation, 340M, 7B tokens | no gate 23.21, fixed decay 16.55, scalar gate 15.56, GLA 14.77 (perplexity) | Table 4 |
| Length | trained on 2K tokens, works beyond 20K; reaches 18K on SlimPajama | Abstract, Figure 5 |
| Term | Plain meaning |
|---|---|
| State S | The fixed-size matrix (dk by dv) that is the model’s whole memory of the past. |
| Outer product kTv | The page one token writes: a grid whose cell (i, j) is ki times vj. |
| Forget gate α | A number in (0, 1) for each key dimension that scales the old state before the new write. |
| Chunk, sub-chunk | A block of C tokens that is handled together; a smaller block of c tokens inside it. |
| HBM, SRAM | The GPU’s large slow memory and its small fast memory. |
| Tensor core | A GPU unit that multiplies small matrices very fast, in half precision. |
| Materialization | Writing the chunk states to HBM so the chunks can run in parallel, at the price of memory. |
| Recomputation | Throwing the states away after the forward pass and rebuilding them in the backward pass. |
| Log space | Working with logarithms of the gate products so that every exponent is at most 0 and nothing overflows. |
| MQAR | Multi-query associative recall: store many key-value pairs, then look up several keys. |
The device below is the check of step 5, running live. It makes random queries, keys, values and gates, runs both forms and compares them.
The line is one output channel over 64 tokens. The pink line is the recurrent form, the warm dots are the chunkwise form. Dotted verticals mark chunk edges. Change the chunk size or the forgetting speed, or draw fresh data.
A JavaScript port of the Python below in 64-bit arithmetic: 64 tokens, dk = 4, dv = 4, gates drawn around the typical value you pick. The difference you see is rounding error. It is not a result from the paper.
pythonimport torch def gla_recurrent(q, k, v, alpha): # q, k: (L, dk) v: (L, dv) alpha: (L, dk) in (0, 1). The test oracle. L, dk = q.shape S = q.new_zeros(dk, v.shape[1]) out = [] for t in range(L): S = alpha[t][:, None] * S + k[t][:, None] * v[t][None, :] # S_t = Diag(a_t) S_(t-1) + k_t^T v_t out.append(q[t] @ S) # o_t = q_t S_t return torch.stack(out) def gla_chunkwise(q, k, v, alpha, C=16): # Same output, chunk by chunk. Needs L % C == 0. L, dk = q.shape S = q.new_zeros(dk, v.shape[1]) mask = torch.tril(torch.ones(C, C, dtype=torch.bool)) out = [] for s in range(0, L, C): qc, kc, vc = q[s:s+C], k[s:s+C], v[s:s+C] lb = torch.cumsum(torch.log(alpha[s:s+C]), 0) # log b inside the chunk, (C, dk) lam = torch.exp(lb) # Lambda: decay from chunk start gam = torch.exp(lb[-1] - lb) # Gamma: decay to chunk end o_inter = (qc * lam) @ S # read the old state diff = (lb[:, None, :] - lb[None, :, :]).masked_fill(~mask[:, :, None], float('-inf')) P = (qc[:, None, :] * kc[None, :, :] * torch.exp(diff)).sum(-1) # Eq. 4: exponent never above 0 out.append(o_inter + P @ vc) S = torch.exp(lb[-1])[:, None] * S + (kc * gam).T @ vc # hand the state on return torch.cat(out)
This is our own plain reading of the paper’s equations, not the authors’ kernel. We ran it: on random data with L = 64, dk = 8, dv = 12 and C = 16, the two functions differ by at most 1e-5 in 32-bit arithmetic. It uses full precision and no sub-chunks, so it shows correctness and not speed. The authors’ Listing 1 and the open-source flash-linear-attention library add the sub-chunk tiling and the fused GPU kernels.
Table 1 of the paper shows that many recent models share one equation, St = Gt ⊙ St−1 + ktTvt, and differ only in the shape of Gt: Mamba, Mamba-2, mLSTM, Gated Retention, DFW, GateLoop, HGRN-2, RWKV-6 and GLA. The authors say their algorithm can be used or adapted for several of them. For our course, the reading belongs next to Lecture 8 on large multimodal models, where long token sequences from images, audio and video make a fixed-size memory attractive. The paper tests none of those; it tests language.
Credit. The paper “Gated Linear Attention Transformers with Hardware-Efficient Training” is by Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda and Yoon Kim (MIT and MIT-IBM Watson AI Lab), Proceedings of the 41st International Conference on Machine Learning, PMLR 235, 2024. Read it at arXiv 2312.06635. Every idea above is re-taught from the paper in our own words; the figures' numbers are the authors’.
Now press Teach and explain from memory, on the whiteboard, how a token is written into the state, how the gate forgets, and why the chunkwise form needs log space. Then go back to the memory at the top, stream 40 chatter tokens into each of the four gates, and say which part of the paper explains each result. Return to Lecture 8 when you are ready.