A transformer built around the symmetries of 3D space reaches about half the test loss of a plain transformer at every compute budget tested, from 10¹⁶ to 10¹⁹ FLOPs.
Drag the compute budget from small to large. Watch the two curves fall side by side, and ask whether the gap closes. Then we build, piece by piece, what a symmetry is, how the paper builds both models, how a scaling law is fitted, and what each figure shows.
You need a feel for what a neural network and a loss are and a little algebra. We build symmetry, attention and scaling laws from zero.
Paper by Johann Brehmer, Sönke Behrends, Pim de Haan and Taco Cohen (Qualcomm AI Research), arXiv 2410.23179, published in Transactions on Machine Learning Research, July 2025. A reading for MIT MMAI, Multimodal AI (Spring 2026), belonging to the lecture on alignment. Every idea is re-taught here in our own words and devices.
The two lines are the paper’s fitted compute-optimal laws, Eq. 8: loss = 1.03 / C0.268 for the plain transformer and 0.14 / C0.236 for the equivariant one, where C is nominal training FLOPs (the idealised count, Chapter 5). The paper tested 10¹⁶ to 10¹⁹ FLOPs, and the dials stop there. Loss is the mean squared error of predicted mesh vertex positions on a rigid-body benchmark.
Chapter 0
See the question the paper asks, and place your own bet before the data answers it
Picture a wooden block tumbling across a table. It spins, hits a second block, and bounces away. You film it at 48 frames each second. Your job is to build a program that looks at two frames and draws the third.
Two engineers take the job. The first says: “Space has rules. If I turn the whole scene a quarter turn, the next frame must turn a quarter turn with it. I will build that rule into the network, so it never has to learn it.”
The second says: “Rules are what data is for. Give me a big enough network and enough examples, and it will find the rule alone. Your rule only adds machinery to build and slow code to run.”
Who is right? This lesson teaches a paper that runs the experiment: “Does Equivariance Matter at Scale?” by Johann Brehmer, Sönke Behrends, Pim de Haan and Taco Cohen at Qualcomm AI Research.
A symmetry is a change that leaves the problem the same. Turn the table, slide it left, or look at it in a mirror: the physics does not change. Chapter 1 makes this exact.
An equivariant network is one that obeys the symmetry by design. Turn the input, and the output turns the same way. Think of a good map app: rotate the phone, and the arrow on the map rotates with you. Nothing is relearned.
An inductive bias is any assumption built into a model before it sees data. A convolution assumes that nearby pixels matter together. Equivariance assumes that the rules of space do not depend on where you stand or which way you face. The first engineer wants a strong bias. The second wants a weak one, and more data.
The common intuition is simple. A strong bias helps most when data is scarce. With millions of examples, the network can find the symmetry itself, so the bias stops mattering. The paper opens with evidence for that view: two famous systems chose not to be equivariant. AlphaFold 3, which predicts the shapes of proteins, and a model that generates 3D molecule shapes both used ordinary networks and data augmentation instead.
The authors give a reason to doubt the intuition. They describe learning as narrowing down a set of possible answers, called the hypothesis class, using evidence. A plain network starts from nearly every function. An equivariant network starts from only the functions that obey the symmetry. It may need fewer steps to finish the narrowing, even with huge data.
Nobody had a theory that settles this. So the paper settles it by experiment. It asks three questions, and this lesson follows them in order.
Here is the second question, made concrete. The paper trains many models of both kinds at budgets from 10¹⁶ to 10¹⁹ FLOPs (a FLOP is one addition or multiplication; Chapter 5 counts them) and measures the best loss each budget can buy. Define the gap as the plain transformer’s loss divided by the equivariant one’s. A gap of 2 means the equivariant model has half the loss.
The question for you: as the compute budget grows a thousand times, what happens to that gap? Commit to an answer. Then the device shows the paper’s fitted laws.
The bars will show the gap, plain loss divided by equivariant loss, at four compute budgets. Pick what you expect, and the bars appear.
Pick one of the three bets above.
The bars come from the paper’s printed laws, Eq. 8 (plain 1.03 / C0.268, equivariant 0.14 / C0.236), so they carry the rounding of those coefficients. The paper itself says only that the equivariant loss is lower by approximately a factor of 2 over the whole tested range.
Most people expect the gap to close, because they trust the intuition above. The fitted laws say otherwise: the equivariant model keeps roughly half the loss at the biggest budget tested. That does not prove the second engineer wrong everywhere. Chapters 7 and 8 show exactly where each engineer wins, and where the claim ends.
Chapter 1
Turn a scene, slide it, mirror it, and learn the one equation that says a network respects all three
Set a camera on a tripod and film two blocks colliding. Now unscrew the camera and turn it a quarter turn on its side. The blocks collide in exactly the same way. Only the picture is rotated. A physics engine does not care which way the camera faces.
That is a symmetry. A symmetry is a change you can make to a problem that leaves its answer the same, apart from the change itself. Space has three that matter here. You can translate (slide everything the same distance in the same direction). You can rotate (turn everything about a point). And you can reflect (look at everything in a mirror).
The paper calls the set of all such moves E(3). The “E” is for Euclidean, the geometry of ordinary flat space, and the 3 is for three dimensions. Each move in E(3) is an isometry: it keeps every distance between points unchanged. A block stays the same size and shape when you turn it.
Mathematicians call such a set a group. A group has two properties. You can chain two moves and get another move in the set. And every move can be undone by another move in the set. Turn left 30 degrees, then slide two metres: that is one move. Slide back, turn right: that undoes it.
Each move acts on data. Write one move as g. Then g · x means “apply g to the input x”. If x is a list of 3D points, applying a rotation multiplies every point by a rotation matrix. Applying a translation adds the same vector to every point.
Now take a network f that maps a scene x to a prediction f(x). Equivariance says that it does not matter whether you move the scene before or after the network runs.
Left side: move the input, then predict. Right side: predict, then move the output. Equivariance means both routes end at the same place. The paper puts it this way: when you transform the inputs, the outputs transform consistently.
A close cousin is invariance. An invariant function ignores the move: f(g · x) = f(x). “How many blocks are in the scene?” is invariant, since turning the scene does not change the count. “Where does each block go next?” is equivariant, because the answer turns with the scene.
Take one point at position (1, 0) moving with velocity (0, 1). Use a toy rule that predicts the next position as position plus velocity: f = (1, 0) + (0, 1) = (1, 1).
Now turn the scene a quarter turn counter-clockwise. The position becomes (0, 1) and the velocity becomes (−1, 0). The same rule gives f = (0, 1) + (−1, 0) = (−1, 1). Turn the first answer (1, 1) a quarter turn and you get (−1, 1) too. Both routes agree, so this rule is equivariant.
Add a second rule that pulls every point down by 0.1, with “down” fixed in the picture. The first answer becomes (1, 0.9). In the turned scene it becomes (0, 1) + (−1, 0) + (0, −0.1) = (−1, 0.9). But turning (1, 0.9) gives (−0.9, 1). The routes disagree, so this rule is not equivariant. It has a hidden assumption: which way is down.
This is a real issue for the paper. The physical world has a special direction: gravity points down. If the network is told nothing about gravity, then turning the scene sideways makes the physics look different. The paper fixes this by treating the direction of gravity as part of the input. Then gravity turns with the scene, and the full E(3) symmetry is exact. Try both versions below.
Grey dots are three points with velocity arrows. The purple arrow is gravity. Green rings are the true next positions. Turn the scene and watch which rule stays on the rings.
A toy, not the paper’s model. The true rule is next = position + velocity + gravity, with a gravity pull of 0.10. The warm rule reads the gravity arrow from its input, so it is equivariant. The blue rule has “down” hard-wired, as a network would if it had only seen upright scenes. Its error is 2 × 0.10 × sin(angle / 2).
At zero degrees both rules agree with the truth. As you turn the scene, the blue rule pulls toward the old “down” and drifts off the ring. At 180 degrees it is as wrong as it can be: it pulls the points the opposite way to gravity. The warm rule never leaves the ring, because it never assumed anything about direction.
There are two ways to fix the blue rule. Build in the symmetry, as the warm rule does. Or show the blue rule scenes at many angles, so it learns to read gravity from the input. The second is data augmentation, and Chapter 7 tests it.
One more symmetry matters. The mesh faces form a set, not a sequence. Shuffling the order of the faces must not change the physics. Both models in the paper use no position codes, so both are already permutation equivariant. The paper compares no model that lacks this. Only the E(3) part differs.
Chapter 2
Turn a pile of colliding 3D objects into a list of tokens, and see what the network must predict
Imagine a small physics lab inside a computer. Between three and ten objects, each a different lumpy shape, drop onto a flat floor. They tumble, collide, and bounce. A camera records 96 frames at 48 frames each second, which is two seconds of motion. Now you hide the future. You show the network frames 0 and 1 and ask: where is every point of every object in frame 2?
That is the paper’s benchmark. This chapter explains why the authors chose it, what the data looks like, and what goes into and comes out of the network. We need all of this before the models in Chapter 3 make sense.
The authors list what a scaling study needs. It should have a low floor and a high ceiling: a small model on little data should do badly, and a large model on lots of data should do orders of magnitude better. There must be a lot of samples, so the data has to be cheap to make. And the symmetries must be known and exact, so that the equivariant model is truly right about them.
Rigid-body physics fits. Simulators make unlimited data. The physics is exactly symmetric under E(3), provided gravity is passed in as a direction (Chapter 1). And it is hard: collisions are tough to detect, because two meshes touch along faces and edges, not only at corners. The forces during a hit are nearly discontinuous, jumping from nothing to large in a few frames.
The paper calls this “a microcosm” of problems in science and engineering: chaotic dynamics, extended 3D objects given as meshes, and the symmetries of space and time. It also states a limit: the goal is not a new state of the art. Problem-specific networks, such as the face interaction graph network of Allen et al. (2023), do better on this task. The authors deliberately use general-purpose models.
The dataset follows the “MOVi-B” recipe that Allen et al. proposed. The authors rebuilt it with the Kubric simulator, which sits on top of the PyBullet physics engine.
| Fact | Value |
|---|---|
| Training trajectories | 400,000 (validation 1,000; test 1,000) |
| Frames per trajectory | 96, at 48 frames per second |
| Objects per scene | 3 to 10, plus a floor plane at z = 0 |
| Mesh faces per object | 98 to 2,160 |
| Mesh faces per scene, on average | 5,470 |
| Gravity | negative z direction, in the data |
A mesh is a 3D surface made of triangles. Each triangle is a face with three corner points called vertices. A transformer reads a list of tokens, which are the small units it attends over. Here the paper gives every mesh face one token.
Each token carries numbers that describe its face. Counting the list in the paper’s Appendix B, a token holds:
That is 36 raw numbers per face by my count. The network then embeds them: the plain transformer with random Fourier features (128 frequencies drawn from a Gaussian with standard deviation 0.1), the equivariant one into a geometric algebra (Chapter 3). So a scene with N faces enters as a list of shape [N, 36]. The velocities need two frames, which is why the input is t0 and t1: one frame says where things are, never where they are going.
The tokens form an unordered set, so there is no position code. Move the objects around in the list and nothing changes. This is the permutation symmetry from Chapter 1.
Each coloured bar is one object, sized by its faces. The square on the right has one cell for every pair of faces that full attention compares. Change the objects and the faces.
Illustrative: every object gets the same face count so the arithmetic stays simple; the dataset mixes shapes of 98 to 2,160 faces. The floor plane is left out. The pair count is N × N, since this attention is not causal and every face looks at every face. The paper’s average of 5,470 faces per scene is the marker.
Slide to ten objects of 2,160 faces and you reach 21,600 tokens, almost 467 million face pairs. An average scene already has about 30 million pairs. That is why the paper picks an equivariant model whose attention works with FlashAttention, a method that computes exact attention without writing the whole pair table to memory.
The network must say where all vertices are at t2 = t1 + Δt. Chapter 4 explains the output in full. In short, it does not write positions directly. Each token outputs eight numbers describing a small move, these are averaged over each object, and the average move is applied to the object’s old vertices.
The loss is the mean squared error between predicted and true vertex positions. Take every vertex, measure how far the prediction is from the truth, square it, and average. Lower is better. Every number on the paper’s plots is this loss on held-out test trajectories.
Symptom. A small model predicts free flight well and then misses at the moment of a collision. A block that should bounce sails on through another block. Measurement. The loss falls fast at first, then slows: free flight is easy, so the remaining error sits in the hard collision frames. Fix. Scale. The paper’s whole point is how much model and compute it takes to push that error down, and whether symmetry knowledge makes the climb cheaper.
Chapter 3
Compare a plain transformer with the Geometric Algebra Transformer, and see why the authors chose a pair that differ in one thing
You want to know if a spice improves a soup. A fair test uses the same pot, the same stock, the same heat, and changes only the spice. If you also change the pot, you learn nothing. The authors face the same problem. They want to test equivariance, so the two networks must look as alike as possible in everything else.
A transformer reads a set of tokens, each a vector of numbers (its channels). It stacks blocks. Each block has two parts: attention, where tokens exchange information, and a small network called the MLP (multilayer perceptron), which refines each token alone. Each part adds its result back to the token. That add-back is a residual connection, a river that tributaries pour into.
Attention works like a room of people each asking a question. Every token makes a query (what I look for), a key (what I offer) and a value (what I hand over). A token compares its query with every key by a dot product. High score means a good match. The scores become weights, and the token takes a weighted mix of the values.
A head is one such attention pattern. Several heads run side by side, each looking for something different. The paper’s baseline uses multi-query attention, where all heads share one set of keys and values, which saves memory. It also uses pre-LN layout, where each part first normalises its input (rescales it to a steady size), which keeps training stable.
The plain model is exactly that: a standard pre-LN transformer with multi-query attention. Each token is a mesh face. The positions and velocities enter through random Fourier features, which the authors found improved performance in early tests. There is no positional encoding, since faces form a set. So even this baseline is “hardly free from inductive biases”, in the authors’ words: it is already permutation equivariant.
For the equivariant side the authors picked the Geometric Algebra Transformer, GATr (Brehmer et al., 2023). They say it is the only E(3)-equivariant architecture they know of that is built on dot-product attention with unlimited reach, and that otherwise follows the transformer plan closely. Many equivariant networks pass messages only between near neighbours. GATr lets every token attend to every other, like the baseline.
GATr has two defining features. First, besides ordinary scalar channels, it carries multivectors from projective geometric algebra. A multivector is a bundle of 16 numbers. Different slots of the bundle mean different geometric things: an absolute position, a direction, a translation, a rotation. Think of it as a typed number: a point and an arrow are both “three numbers”, but they transform differently when you slide the scene, and the type records that.
In this task the faces are embedded as follows (Appendix B). Vertex and centre positions become trivectors, the geometric-algebra slot for points. The vectors from centre to vertex, and the surface normals, become vectors. Velocities become bivectors. Second, GATr is a transformer: it processes tokens, with an equivariant version of dot-product attention that works with FlashAttention.
I have not opened the original GATr paper for this lesson, so the layer-by-layer details of its equivariant linear maps and attention are left to that source. What this paper relies on is one fact: every GATr layer commutes with E(3) moves. That is enough for the whole network to commute, by a short argument.
Chain as many equivariant layers as you like and the stack stays equivariant. The paper adds one more fact, from Appendix B: for GATr the equivariant linear maps are not a subset of the plain model’s linear maps. You cannot get GATr by taking a plain transformer and tying some weights together. The two are different families, which is why the comparison is a real experiment.
Both models are described by one size number n. Table 1 of the paper fixes how everything grows with n, so each model is a one-parameter family.
| Setting, for size n | Plain | Equivariant |
|---|---|---|
| Attention blocks | 2n | 2n |
| Scalar channels | 64n | 4n |
| Multivector channels | none | n |
| Attention heads | 2n | 2n |
| Scalars per key, query, value | 64 | 8 |
| Multivectors per key, query, value | none | 2 |
| Hidden scalar channels in the MLP | 128n | 8n |
| Hidden multivector channels in the MLP | none | 2n |
The equivariant model is far narrower: 4n scalar channels against 64n. Part of the width is multivector, and each multivector channel holds 16 numbers. The authors found the equivariant model prefers a narrow shape, and say this may be evidence of how expressive multivector channels are. They tuned depth, width and heads by hand at low and middle compute, then fixed the linear relation across sizes.
Each bar is the width of one token inside the model. The blue bar is the plain model’s scalar channels. The warm bar is the equivariant model: a few scalars, then multivector channels of 16 numbers each. Drag n.
The counts follow the paper’s Table 1 and the fact that a multivector has 16 components. Total numbers per token is my sum (4n + 16n = 20n for the equivariant model), not a figure the paper prints. The paper does not say which values of n it ran; the dials stop at 16 only to keep the picture legible.
Slide n to 4. Both models stack 8 attention blocks with 8 heads. A plain token holds 256 numbers. An equivariant token holds 16 scalars plus 4 multivector channels of 16 numbers each: 80 numbers in all. The equivariant model is about three times thinner at every n, yet each of its numbers does more work. Chapter 5 counts how much more.
Failure mode. Symptom: you compare the two at the same parameter count and the equivariant model looks slow and weak. Measurement: it needs more arithmetic per parameter, so equal parameters means unequal cost. Fix: compare at equal training compute, which is what the whole paper does.
Chapter 4
Let some attention heads see only their own object, and make every output move an object as one rigid piece
Two blocks lie on a table. A face on the top of block A has two kinds of neighbours. Some are on its own block: the faces that must travel with it, because a block does not bend. Others are on block B, which may be about to hit it. A good model needs both views, and it needs to keep each block in one piece.
The authors add two small changes to the transformer blueprint, and they add them to both models. So neither changes the comparison. The paper says both models benefit from them.
In a plain transformer every token can attend to every token. Here, half of the heads in each attention layer are restricted by a mask: a table of allowed pairs. In these heads a face may only attend to faces of the same object. The other half of the heads attend to all faces, as usual.
How does a mask work? Attention scores every pair. For a blocked pair, the score is set to minus infinity before the softmax, the step that turns scores into weights. A weight of zero follows, so the blocked token contributes nothing. The pair is simply not seen.
Why this helps: the model gets built-in awareness of mesh structure, which faces belong together, without giving up the efficient dot-product form of attention. It is a mask, not a new layer. This is the paper’s own “novel hierarchical attention mechanism”, and a minor modification, in its words.
Twelve faces from three objects, shown as coloured ticks on both axes. A lit cell means the row face may attend to the column face. Switch between the two kinds of head.
A toy scene of 5, 4 and 3 faces; the real scenes have thousands. The paper states the split (half the heads restricted to the same object, half unrestricted) but gives no count of faces per object for this picture.
In the toy scene, a same-object head lets 50 of the 144 pairs through, which is 34.7%. An all-faces head lets all 144 through. Mix half and half and a typical layer sees about 67% of the pairs, with the object structure built in.
Why both kinds of head? A face near a collision needs the other object, which the all-faces heads supply. It also needs to know the rest of its own object, because the whole object moves together. The same-object heads supply that without the model having to find out which faces belong together. The mask is cheap: it is a table of zeros and minus infinities added to the scores.
A rigid object never bends, stretches or squashes. All its vertices move by the same rotation and the same shift. The naive way to predict the next frame is to output a new position for every vertex. Each vertex then has its own error, and the object can come out slightly bent.
The paper does something else. Each face token outputs eight numbers h that describe one motion: in the main text, a translation vector and a rotation quaternion (a four-number code for a rotation). The appendix writes the same thing as the even-grade parts of a geometric-algebra multivector. Then three steps follow:
The paper reports that, in preliminary tests, this made performance substantially better than directly predicting vertex positions or velocities. It also tried exponentiating the Lie algebra of E(3) (a way to write a motion as a small push that you then carry out), and found that marginally worse.
The plain transformer uses the same trick, so it too is guaranteed to move objects rigidly. The paper writes the formula in geometric algebra because the equivariant model speaks that language natively; the baseline outputs the same eight numbers.
A worked example. Three faces of one block guess an x shift of 0.9, 1.1 and 1.0. The mean is 1.0, so every vertex of the block moves by exactly 1.0, and every distance between its vertices stays the same. Now let each vertex follow its own guess instead. Two neighbouring vertices could move by 0.9 and 1.1. The gap between them changes by 0.2, although a rigid block never changes its size.
Failure mode. Symptom: the predicted blocks look slightly wobbly, with edges that grow and shrink from frame to frame. Measurement: divide each edge length at t2 by its length at t1. For a rigid object that ratio must be exactly 1. Fix: the per-object average, which makes the ratio exactly 1 by construction. Try it below.
Six faces each guess the same shift, with noise. Dashed is the object now, green is the truth next, warm is the prediction. Raise the noise, then switch on averaging.
A toy: a hexagon of radius 0.5 shifted by (0.6, 0.15), with Gaussian noise added to each face’s guess. Only a shift is predicted here; the paper also averages a rotation. The noise is a stand-in for network error, not a measured value.
With averaging off, each vertex follows its own noisy guess, and the hexagon bends: some edges stretch, some shrink. With averaging on, all six guesses are merged into one shift. The shape is exact, and only the position of the whole hexagon is off, by about the noise divided by the square root of six.
Chapter 5
Derive why training costs about 6 FLOPs per parameter and token, and why the equivariant model pays ten times that
Two cooks have different stoves. To compare their soups fairly, you do not count the pots. You count the gas each cook burns. A paper that compares two neural networks faces the same choice. Counting parameters is counting pots. The fair unit is the compute spent on training.
A FLOP is one floating-point operation: one addition or one multiplication of two numbers. N is the number of parameters, the learned numbers inside the model. D is the number of training tokens: how many tokens the model processes in total, counting repeats. In this paper a token is a mesh face, so D counts faces seen. And C is the total training compute, in FLOPs.
One scene has about 5,470 faces. The paper trains on batches of 64 scenes, which is on average 3.5 × 105 tokens. Check: 64 × 5,470 = 350,080. The numbers agree.
Take a model whose cost is dominated by linear layers: layers that multiply a vector by a weight matrix. A matrix with N weights does one multiplication and one addition per weight when it processes one token. That is 2N FLOPs for the forward pass, the step that makes a prediction.
Training also needs the backward pass, which finds how to change each weight. It must compute two things: how the loss depends on each weight, and how it depends on each layer’s input, so the error can flow to earlier layers. Each costs about as much as the forward pass. The backward pass is therefore about 4N. Forward plus backward is 6N FLOPs per token. Over D tokens:
This is a count in the limit the paper states: the number of parameters is much larger than the sequence length, which is much larger than one. Then the linear layers dominate and the attention pair table (Chapter 2) is a small correction.
The paper gives two reference points for the equivariant model. A model with only scalar channels behaves like the plain one: ξ = 6. A model made only of multivector channels has more weight sharing: one learned weight is reused many times inside a layer, so each weight does more arithmetic. The paper puts that case at ξ = 6 · 16² / 9 ≈ 171.
My reading of that formula: a map between 16-component multivectors does 16 × 16 multiply-adds per token, but the equivariance rules leave only about nine independent weights to learn. The ratio 256 to 9 is 28 times, and 6 × 28.4 is 171. The paper does not spell this out, so treat it as an interpretation.
For the sizes it actually uses, the paper finds ξ ≈ 61.2. Since ξ averages over parameters, a mix of the two kinds gives 6(1 − f) + 171 f = 61.2 where f is the share of parameters in multivector layers. That solves to f ≈ 0.33, about a third. This back-of-envelope step is mine, not the paper’s.
The consequence is large. At equal N and equal compute, the equivariant model trains on about ten times fewer tokens.
A worked example. Take C = 1018 FLOPs and N = 106 parameters. The plain model gets D = C / (6N) = 1.67 × 1011 tokens, which is about 3.0 × 107 scenes (divide by 5,470). The equivariant model gets D = C / (61.2 N) = 1.63 × 1010 tokens, about 3.0 × 106 scenes. In steps of 3.5 × 105 tokens, that is about 4.8 × 105 steps against 4.7 × 104. The equivariant model needs far less training to spend the same budget, so it must make that training count.
Each curve fixes the compute and sweeps the model size. Bigger models see fewer tokens. The dip marks the best size for that budget. Choose a model, then move your own size along its curve.
Curves are the paper’s fitted law, Eq. 6 and 7, with D = C / (ξ N); the law is the paper’s, the reading of it at a budget and size is computed live. Sizes outside the range the paper trained are extrapolation. The best sizes use Eq. 4, derived in Chapter 6.
Drag your size left of the dip and the model is too small: it sees many tokens but cannot hold what it learns. Drag it right and the model is too big for the budget: it sees too few tokens. The best size sits between, and it is different for the two models.
The authors are plain about a catch. Their FLOP counts are nominal: the theoretical count, not the work a GPU actually did. Small models and the equivariant model use the chip less well. The equivariant code runs many small operations and hits CPU bottlenecks. Two models with equal nominal FLOPs could differ by as much as an order of magnitude in real training time. Appendix C and Figure 6 show that the throughput of nominal FLOPs varies by two orders of magnitude across the runs.
They still use nominal FLOPs, for two reasons. The count does not depend on the code or the hardware. And it is the best predictor they have of the achievable cost once the implementation is tuned. This matters later: the paper’s compute claim assumes both models could reach the same throughput. If the plain model runs on the chip better, the real-world race is open.
Failure mode. Symptom: the equivariant model wins a plot against nominal FLOPs but loses against wall-clock hours. Measurement: divide nominal FLOPs by seconds for each run, as in Figure 6. Fix: better kernels and batching for the equivariant code, which the paper names as the practical task.
Chapter 6
Write loss as two power laws, fit them to the runs, and derive the best split of any compute budget
You bake bread, and you learn two things. A bigger oven helps, and a longer bake helps, but each gives less than the last. After a while you can predict a loaf from the oven size and the minutes, without baking it. A scaling law is that prediction for neural networks: a formula for the loss from the model size and the training length.
The paper runs many training jobs, then fits one formula per architecture to all of them. The formula follows Kaplan et al. (2020) and Hoffmann et al. (2022), the “Chinchilla” paper. This chapter builds the formula, the fit, and the main result that follows from it.
An ansatz is a shape you assume, then test. The paper assumes loss falls as a power of model size plus a power of training tokens:
Read it as two leaks in a bucket. One leak closes when the model gets bigger. The other closes when training gets longer. Loss is the sum of both leaks. If you only grow the model, the second leak keeps the loss up, and the reverse.
The fitted laws are the paper’s Eqs. 6 and 7. For the plain transformer, L = 1.27 / N0.909 + 0.202 / D0.379. For the equivariant one, L = 2.82 × 10−4 / N0.348 + 469 / D0.734. The two are strikingly different: the equivariant model gets far less from extra size (α = 0.348 against 0.909) and far more from extra training (β = 0.734 against 0.379).
The five numbers A, B, E, α, β are fitted separately for each architecture. The recipe follows Hoffmann et al. and has four ideas.
To report uncertainty, the authors use a nonparametric bootstrap. Resample the runs with replacement, refit, and repeat 10,000 times. The middle 95% of the refitted values is the 95% confidence interval.
Now the key step. Fix the compute C. By Chapter 5, D = C / (ξ N), so there is only one free number, N. Substitute and the loss becomes
L(N) = A N−α + B (C/ξ)−β Nβ
The first term falls with N, the second rises: a bigger model has fewer tokens. The minimum is where the slope is zero. Set the derivative to zero: −αA N−α−1 + βB (C/ξ)−β Nβ−1 = 0. Collect the powers of N on one side:
Nα+β = (αA / βB) (C/ξ)β
Take the (α+β)-th root. This gives the paper’s Eq. 4, with G = (αA / βB)1/(α+β):
The exponents tell the story. In the best split, N grows as Ca and D grows as Cb, and a + b = 1, so doubling the budget is shared between the two. Put the optimum back into the loss and both terms scale as C−γ with γ = αβ / (α+β): that gives the paper’s Eq. 5, L* = F / Cγ. That is why the best loss is itself a power law in compute.
A worked example, plain model. α = 0.909, β = 0.379, so α+β = 1.288. Then a = 0.379 / 1.288 = 0.294, b = 0.706, and γ = 0.909 × 0.379 / 1.288 = 0.268. Next G = (0.909 × 1.27 / (0.379 × 0.202))1/1.288 = 15.080.776 ≈ 8.22. At C = 1018: N* = 8.22 × (1018/6)0.294 ≈ 9.6 × 105 parameters, and D* = (1018/6)0.706 / 8.22 ≈ 1.7 × 1011 tokens. Repeat with the equivariant numbers (a = 0.678, b = 0.322, ξ = 61.2) and you get N* ≈ 8.9 × 104 and D* ≈ 1.8 × 1011.
| Table 2: central value (95% interval) | Plain | Equivariant |
|---|---|---|
| α (size exponent) | 0.909 (0.832 to 1.03) | 0.348 (0.293 to 0.417) |
| β (token exponent) | 0.379 (0.256 to 0.404) | 0.734 (0.689 to 0.747) |
| a (size grows as Ca) | 0.294 (0.215 to 0.307) | 0.678 (0.619 to 0.711) |
| b (tokens grow as Cb) | 0.706 (0.693 to 0.785) | 0.322 (0.289 to 0.381) |
| γ (loss falls as C−γ) | 0.268 (0.213 to 0.284) | 0.236 (0.212 to 0.267) |
| F (prefactor of the best loss) | 1.03 (0.124 to 1.89) | 0.14 (0.0524 to 0.517) |
Two readings. The loss exponents γ are 0.268 and 0.236, and their intervals overlap: the paper calls the exponents compatible. The prefactors F differ by roughly seven times at their central values, which is what lifts the equivariant line below the plain one. The F intervals alone do overlap, because F and γ are fitted together and move together. In Figure 1, which shows the loss at each budget, the two shaded bands sit apart over the tested range.
Dots are noisy experiments drawn around the plain model’s law. The bold line is the fit. The faint lines refit on resampled dots. Add noise or drop dots and watch the faint lines spread.
Illustrative: points follow loss = 1.03 / C0.268 times random noise, at budgets from 1016 to 1019. The line is an ordinary least-squares fit in log space; the interval is the middle 95% of 300 resampled fits. The paper’s own fit uses the Huber loss, L-BFGS and 10,000 resamples.
Failure mode. Symptom: the fit looks good, yet fewer measured points fall inside the shaded bootstrap band than the 95% you would expect. The paper reports exactly this. Measurement: the bootstrap only covers the effect of sampling a finite set of runs, so fewer than 95% of points fall inside it. Fix: read it as a sign that the ansatz is not perfect. A fair reading is that the stated intervals understate the real uncertainty.
Chapter 7
Shrink the dataset by five orders of magnitude and watch where equivariance helps, and what data augmentation can repair
A student has one week to prepare for an exam. She can read a thousand different practice problems once. Or she can read ten problems a hundred times each. Given the same hours, the second student has seen far fewer ideas. The first thing the paper tests is what happens to the two models when the pool of different examples shrinks and the training time stays the same.
The authors fix the compute at 1018 nominal FLOPs. For each model they use the model size and the number of training tokens that the first experiment found compute-optimal at that budget (Chapter 6 shows them: about 105 parameters and 1.8 × 1011 tokens for the equivariant model, about 106 and 1.7 × 1011 for the plain one). Then they change only the number of unique tokens in the training set, over five orders of magnitude: from 2 × 106 to 2 × 1011.
Because the compute and the model are fixed, the training time is fixed too. A small dataset is simply cycled many times. One pass through the data is an epoch. At the right end of the plot, each sample is seen once. At the left end, the paper states it is seen 6 × 105 times.
For each dataset size, three models are trained:
One detail matters. The plain model is not told the direction of gravity. So the augmentation cannot use the full E(3). It uses only the symmetries that keep gravity pointing down: 2D translations across the floor and rotations around the gravity axis.
Figure 2 plots test loss against unique training tokens, both on log scales, three lines. Reading it from right to left:
Right end, one epoch. At the large-data end the augmented and the plain models sit together, and the equivariant line is lower than both by roughly the compute gap of Chapter 8. This is expected: with every sample seen once, a random pose of an unseen sample is just another unseen sample, so augmentation changes nothing.
Moving left. The plain model falls apart. Its loss climbs steeply as the dataset shrinks. The equivariant model degrades slowly. The gap between them widens a lot. That confirms the old expectation: equivariance improves data efficiency.
The augmented model. It follows the equivariant line closely on the left, where each sample is seen thousands of times, and it drifts a little above it in the middle. At the far left, the paper says, data augmentation makes the plain transformer as data-efficient as the equivariant one. This is the first of the paper’s three conclusions: equivariance helps with little data, but augmentation largely closes the gap, given enough epochs.
The three lines sketch Figure 2: loss as the number of unique tokens grows. Drag along the bottom axis to choose a dataset size. Switch the augmented line on and off.
A schematic drawn by eye from Figure 2, not measured values: the paper prints no table for this figure, so the vertical axis has no numbers. What it does state: the three lines meet on the right, the equivariant line sits lower there by the compute gap (about 1.95 times in the fitted laws at 1018 FLOPs), and augmentation matches equivariance only at the small-data end. Passes over the data use 1.8 × 1011 tokens trained, from the fitted laws; the paper itself states 6 × 105 epochs at 2 × 106 tokens, which is larger than my division gives.
Slide the marker to the far left. The plain line is far above the other two, and the augmented line hugs the equivariant one. Now slide it right to 1010 and beyond: the augmented line joins the plain one, and equivariance has its modest, constant lead.
Augmentation teaches a symmetry by example. Each epoch the model sees the same scene in a new random pose. It must see enough poses to cover the space of moves. In the simplest case, one rotation around the vertical axis, cut into 36 slices of 10 degrees each. After k random epochs, the chance that a given slice has been visited is 1 − (35/36)k. At k = 36 that is 64%. At k = 100, 94%. At k = 200, 99.6%.
Real symmetry is bigger than one angle: a rotation, plus a slide in two directions. Coverage needs far more epochs. That is why the paper says augmentation closes the gap “given sufficient epochs”, and why it does nothing at one epoch.
The ring has 36 slices of 10 degrees. Each epoch shows the same scene at a random rotation. A lit slice is an orientation the plain model has now seen. Add epochs, or switch augmentation off.
Illustrative: only rotation about the gravity axis, in 36 slices. The paper’s augmentation also slides the scene in two directions, which needs more epochs to cover. The expected coverage is 1 − (35/36)k after k epochs; the lit slices are real random draws in your browser.
Failure mode. Symptom: a plain model trained on 2 × 106 tokens has a low training loss and a high test loss. Measurement: loss on held-out scenes, compared with the equivariant model at the same compute. Fix: either build the symmetry in, or augment so the repeated samples differ each epoch. The paper finds both work at the small-data end.
Chapter 8
Read Figures 1, 3, 4 and 5, see how the best split differs, and test the claim against real-world speed
Now the data is plentiful: every sample is seen once. The only limit is compute. A team has a fixed amount of GPU time and one question. Should it spend that time on a plain transformer or an equivariant one? And how big should the model be? These are the paper’s second and third questions.
The authors train both architectures at budgets from 1016 to 1019 nominal FLOPs. At each budget they vary the split: a small model trained for long, or a large model trained for short, as in the device of Chapter 5. Each run needs early stopping and a tuned learning rate. Budgets below 1018 use a higher rate (10−3 or 2 × 10−3 instead of 5 × 10−4), and the better result is kept. The batch size is 64 scenes throughout.
Figure 4 shows the loss against model size, one panel per budget, with the fitted lines. Each panel is a bowl: too small is bad, too big is bad. Figure 3 shows the same data as a map of loss against model size and tokens, dots over the colours of the fit. The paper judges the fit good, with two flaws it names. The power law underestimates the loss for the largest equivariant models, and one baseline run is an outlier.
Take the best loss from the fitted laws at each budget. Both are straight lines on a log-log plot, so both are power laws. The exponents are close: the paper gives L*plain = 1.03 / C0.268 and L*equivariant = 0.14 / C0.236 (the hero at the top of this page draws exactly these). The lines are nearly parallel, so the vertical gap stays close to a constant factor, about 2, over the tested range.
That is the paper’s second conclusion, and the less expected one. Equivariance helps even with unlimited data, because at every budget the equivariant model reaches a lower loss. The paper adds a condition: this holds if both implementations are about equally efficient. We return to that below.
Chapter 6 gave the best size N* and best token count D* for each model. Plot N* against the budget, as Figure 5 does. The equivariant optimum is much smaller than the plain one at low budgets. That is expected, since it does about ten times more arithmetic per parameter. The surprise is the trend. For the plain model, a = 0.294 and b = 0.706: spend most of any extra budget on more tokens. For the equivariant model, a = 0.678 and b = 0.322: spend most of it on a bigger model.
Each line is the best choice for a given budget. Choose what to plot. Drag the budget to compare the two models at one point.
Computed from the printed coefficients of Eq. 6 and 7 with Eq. 4, so values carry the rounding of those coefficients. The paper shows this relation as Figure 5 (model size only) and states the opposing trends in words; the token view and the ratios are my arithmetic from its laws.
Start the slider at 1016. The best equivariant model has about 3.9 × 103 parameters, and the best plain model about 2.5 × 105: 63 times more. Slide to 1019: 4.2 × 105 against 1.9 × 106, a gap of 4.4 times. The sizes are closing in. Switch to tokens and the lines cross: at 1016 the equivariant model trains on about six times more tokens than the plain one, and at 1019 on about 0.44 times as many.
The paper labels these as hypotheses. It does not test them.
Chapter 5 said nominal FLOPs are not wall-clock time. Here is what that does to the claim. Suppose the equivariant code spends r times longer than the plain code per nominal FLOP. Then at the same wall-clock time, the equivariant model has only C / r FLOPs to spend. Its loss is Feq / (C / r)γ.
The break-even r is where that equals the plain loss, so r = (Lplain / Leq)1/γeq. At 1018, (1.95)1/0.236 ≈ 17. At 1019, 1.811/0.236 ≈ 12. This is my arithmetic from the paper’s laws, not a claim in the paper. Try it below.
Both lines now use wall-clock time as the budget. The warm line moves to the right when the equivariant code is slower. Find where it meets the blue line.
My calculation, not the paper’s. Budget means nominal FLOPs the plain model can run in the available time. The equivariant model runs C / r nominal FLOPs in that time. The paper says only that equal nominal FLOPs can differ by as much as an order of magnitude in training time, and that this may shrink with better code.
The authors list the limits plainly. It is one benchmark and two model families, so the result may not carry over to other problems, or to equivariant networks that pass messages between neighbours. The task is solved better by problem-specific networks, so no new state of the art is claimed. Budgets stop at 1019 FLOPs, while the largest language models train on about 1025, six orders of magnitude more. The power law did not break in the tested range, but the paper makes no claim beyond it. The conclusion it draws is careful: symmetry-aware modelling “can be a sensible choice even for large compute and data budgets”, and more efficient equivariant implementations are worth building.
Failure mode. Symptom: someone extends the lines to 1025 FLOPs and declares the winner. Measurement: nothing in the paper was measured above 1019, and the fit already misses the largest equivariant models. Fix: treat the lines as a map of the tested range only.
Chapter 9
Keep the numbers, the checklist and the core code, and see where the paper sits among scaling studies
The paper asks whether a built-in symmetry still pays off when data and compute are large. On one rigid-body benchmark, with a plain transformer against the Geometric Algebra Transformer, it finds three things. Equivariance helps most with little data, and augmentation can close that gap given enough epochs. At every tested compute budget the equivariant model keeps roughly half the loss. And the best split of a budget between model size and training length differs, with the two best sizes moving closer as the budget grows.
| What | Value and where it comes from |
|---|---|
| Benchmark | Rigid-body meshes, MOVi-B recipe via Kubric and PyBullet; 400,000 training trajectories of 96 frames at 48 fps; 3 to 10 objects; 5,470 faces per scene on average (Sec. 3.1, App. A) |
| Task and loss | Frames at t0 and t1 in, vertices at t2 out; mean squared error of vertex positions |
| Token | One mesh face; 36 raw numbers per face by my count of App. B |
| Plain model | Pre-LN transformer, multi-query attention, random Fourier features, no positional code (Table 1: 2n blocks, 64n scalar channels) |
| Equivariant model | GATr: 2n blocks, 4n scalar plus n multivector channels of 16 numbers (Table 1) |
| Shared tricks | Half the heads restricted to the same object; per-object averaged output so objects stay rigid |
| Cost per parameter and token ξ | 6 for the plain model; about 61.2 for the equivariant model; C ≈ ξ N D (Eq. 1) |
| Fitted laws | Plain: 1.27 / N0.909 + 0.202 / D0.379. Equivariant: 2.82 × 10−4 / N0.348 + 469 / D0.734 (Eq. 6, 7; E = 0, δ = 0.001) |
| Best loss against compute | Plain 1.03 / C0.268; equivariant 0.14 / C0.236; about 2 times lower at every budget from 1016 to 1019 FLOPs (Eq. 8, Fig. 1) |
| Best split | Plain: N* grows as C0.294, D* as C0.706. Equivariant: N* as C0.678, D* as C0.322 (Table 2, Fig. 5) |
| Data experiment | 1018 FLOPs, 2 × 106 to 2 × 1011 unique tokens; augmentation closes the small-data gap, does nothing at one epoch (Fig. 2) |
| Main limits | One task, two model families; nominal FLOPs, not wall-clock; budgets stop at 1019, far below about 1025 for the largest language models |
Terms to keep: symmetry (a change that leaves the answer the same), equivariant (the output moves with the input), E(3) (rotations, translations and reflections), data augmentation (training on moved copies), scaling law (loss as a power law in size and tokens), compute-optimal (the best split of a fixed budget), and nominal FLOPs (the idealised arithmetic count).
Four pieces are enough to redo the analysis in miniature: the best split from a fitted law, the Huber fit on log loss, the hierarchical masks, and the rigid output step. The last is a simplified stand-in written with quaternions in numpy. The paper does it with geometric algebra, where the sandwich product h x h̃ applies the averaged motion.
pythonimport numpy as np from scipy.optimize import minimize # 1. Compute-optimal split (paper Eq. 4): given a fitted law and a budget C def optimal_split(C, A, B, alpha, beta, xi): G = (alpha * A / (beta * B)) ** (1 / (alpha + beta)) a, b = beta / (alpha + beta), alpha / (alpha + beta) N = G * (C / xi) ** a # best parameters D = (C / xi) ** b / G # best training tokens return N, D # 2. Fit L(N, D) = A / N^alpha + B / D^beta on log loss with a Huber penalty (E = 0) def huber(r, d): return np.where(np.abs(r) <= d, 0.5 * r**2, d * (np.abs(r) - 0.5 * d)) def fit_law(N, D, L, delta=1e-3, starts=20, seed=0): rng, best = np.random.default_rng(seed), None def loss(t): # t = [log A, log B, alpha, beta] pred = np.log(np.exp(t[0]) / N**t[2] + np.exp(t[1]) / D**t[3]) return huber(pred - np.log(L), delta).sum() for _ in range(starts): # many starts, to avoid poor local minima t0 = rng.uniform([-5, -5, 0.1, 0.1], [5, 5, 1.2, 1.2]) r = minimize(loss, t0, method="L-BFGS-B") if best is None or r.fun < best.fun: best = r return np.exp(best.x[0]), np.exp(best.x[1]), best.x[2], best.x[3] # 3. Hierarchical attention masks: half the heads see only their own object def head_masks(obj_id, n_heads): # obj_id: [F] object index of each face token same = obj_id[:, None] == obj_id[None, :] # [F, F] bool return [same if h < n_heads // 2 else np.ones_like(same) for h in range(n_heads)] # 4. Rigid output: average each face's motion per object, then apply it to the object def quat_to_matrix(q): w, x, y, z = q return np.array([[1-2*(y*y+z*z), 2*(x*y-z*w), 2*(x*z+y*w)], [2*(x*y+z*w), 1-2*(x*x+z*z), 2*(y*z-x*w)], [2*(x*z-y*w), 2*(y*z+x*w), 1-2*(x*x+y*y)]]) def rigid_step(shift, quat, face_obj, verts, vert_obj): # shift [F,3], quat [F,4] per face; verts [V,3] at t1. Returns verts at t2, [V,3]. out = verts.copy() for k in np.unique(face_obj): t = shift[face_obj == k].mean(0) q = quat[face_obj == k].mean(0); q = q / np.linalg.norm(q) m = vert_obj == k c = verts[m].mean(0) # turn about the object centre out[m] = (verts[m] - c) @ quat_to_matrix(q).T + c + t return out
The paper follows the scaling-law method of Kaplan et al. (2020) and Hoffmann et al. (2022), including the fit procedure of the second. Tay et al. (2023) compared scaling across architectures, and Qiu et al. (2024) found that structured linear layers can improve scaling. This paper differs by studying a symmetric problem and an equivariant architecture. Its motivation is the choice made by AlphaFold 3 and by a model for molecule shapes, which skipped equivariance and relied on augmentation.
The authors end with a call to action, not a verdict: build more efficient implementations of equivariant networks, because the nominal-FLOP advantage only pays if the hardware can run it as fast.
Now press Present or Teach and explain the paper from memory: what a symmetry is, why the two models differ, how the best split of a budget is derived, and what Figures 1, 2 and 5 show. Then go back to the compute dial at the top, drag it across the tested range, and say why the gap between the lines is about two at every point.
Credit: “Does Equivariance Matter at Scale?” by Johann Brehmer, Sönke Behrends, Pim de Haan and Taco Cohen, Qualcomm AI Research. arXiv 2410.23179, Transactions on Machine Learning Research, July 2025. All figures and tables named here are theirs; the toy devices, the token count of 36, the 20n width sum, the f = 0.33 estimate and the break-even slowdowns are this lesson’s own arithmetic and are marked where used.