A pattern can turn up anywhere in a signal. A dense network must learn it again at every position. A convolution learns it once, because it assumes a shift changes nothing. That assumption is an inductive bias. Choose the one your data really has.
Learn what an inductive bias is and how a symmetry turns into a block: convolution for shifts, Deep Sets for sets, message passing for graphs, recurrence for time. See why attention needs more data, and test any block with a symmetry check in PyTorch.
Three models train in your browser on N examples. In training, the pattern only appears in the left half. Slide the test shift to move the pattern right, and watch which model still finds it. Then change N.
You need vectors, matrix multiplication, and the idea of a trained classifier. We build the rest from zero.
Synthetic data, illustrative numbers. Each signal is 32 samples of noise (spread 0.5). Half the signals hold an 8-sample wiggle; the other half hold the same wiggle upside down, so loudness alone cannot tell them apart. Training wiggles start at samples 0 to 8. A test shift of s moves them to start at s to s + 8. Models: a dense MLP (32→16→1, 545 weights), a convolution (4 filters of width 7, max over positions, 37 weights), and a small attention model (8 patches of 4 samples, learned position embeddings, one attention layer, 297 weights). Each trains 400 steps of Adam here and is tested on 300 new signals per shift.
Chapter 0
Train a dense detector on patterns near the start of a window, move the pattern, and see exactly why the detector goes blind
You fit a vibration sensor to a motor. A worn bearing makes a short wiggle in the signal. You cut the signal into windows of 32 samples and label each one: "wiggle" or "no wiggle". You want a model that flags the wiggle.
By chance, your recordings caught the wiggle near the start of each window. In the field it can start anywhere. The wiggle is the same wiggle. Only its position changed.
The simplest detector is one dense layer: a weighted sum of all 32 samples, s = w1x1 + … + w32x32, then a threshold. It has one weight for each position. The device trains one, and a second detector that slides one short template along the signal.
Top: one test signal; the warm band is the wiggle. Middle: the dense detector's 32 weights. Bottom: the sliding detector's 7 weights and its response at every position. Drag the wiggle along the window.
Both detectors are linear and trained here by gradient descent on 200 synthetic windows. The sliding detector takes the largest response over positions. Illustrative numbers.
Look at the dense weights. They hold a copy of the wiggle at the positions where training wiggles lived. Everywhere else they are near zero. Those weights never saw a wiggle, so they never learned one.
Worked: a 3-sample bump (1, 2, 1) at positions 3 to 5, where w = (1, 2, 1): s = 1 + 4 + 1 = 6. The same bump at positions 20 to 22, where w = (0, 0, 0): s = 0. Same bump, no detection.
Now switch training to "anywhere". The dense weights blur. A wiggle at position 3 and the same wiggle at position 4 want different weights at the same place. One weighted sum cannot hold all shifted copies at once, so its accuracy now swings from spot to spot. A dense network needs a separate unit, and separate examples, for each position.
The sliding detector has no such problem. It uses the same 7 weights at every position, and takes the best match. It learned the wiggle once. It finds it anywhere. That "same weights everywhere" choice is an assumption made before seeing any data: a shift does not change what a pattern means. Assumptions like this are the subject of this lesson.
Chapter 1
Count how many rules fit your data, see why a learner must guess, and watch four different guesses fill the same gap
Training data only tells you about inputs you have seen. Every prediction on a new input is a guess. The guess comes from what the learner assumed before it saw any data. That set of assumptions is its inductive bias.
You already know some. Linear regression assumes the answer is a straight line. Nearest neighbours assumes that close inputs have close answers. The sliding detector of Chapter 0 assumes a pattern means the same thing at every position.
Take a task with 3 yes/no inputs. There are 23 = 8 possible inputs. A rule assigns yes or no to each, so there are 28 = 256 possible rules. Now label 5 of the 8 inputs.
Worked: n = 3, m = 5 → R = 23 = 8 rules fit the data. On each of the 3 unseen inputs, 4 of them say yes and 4 say no. The data alone gives a 50/50 split.
So the data cannot choose. Something else must. This is the idea behind the no free lunch theorem (Wolpert, 1996): averaged over every possible target rule, all learners have the same error on unseen inputs. A learner beats chance only on problems where its assumptions are true. There is no bias-free learner, only biases that fit your problem and biases that do not.
Six blue points come from a hidden rule on the left half. Each bias fills the right half its own way. Pick a bias, then change the hidden rule. No bias wins on both.
Computed live. The error is the root-mean-square gap between the prediction and the hidden rule on the unseen half. The wave bias assumes the true period (0.5); the smooth curve uses a Gaussian kernel of width 0.12. Illustrative numbers.
The same six points, four sensible answers. The line bias is right about lines and wrong about waves. The wave bias is the reverse. The lookup table copies the nearest point and learns no shape at all. None of them is "unbiased". Each one is a bet about the world.
This lesson is about the most useful kind of bias: symmetry. A symmetry of your data is a change you can make to an input that does not change the answer, or changes it in a way you know. Shift a sound in time: still a dog bark. Reorder the points of a point cloud: still the same chair. Once you know a symmetry, you can build it into the block, and the model never has to learn it.
Chapter 2
Define the two ways a model can respect a symmetry, work one example by hand, and check both numerically
Let g be a change to the input: shift by 3 samples, reorder the points, rotate by 30°. Write g·x for the changed input. A model f can respect g in two ways.
Invariance: the output does not change. "Is there a dog bark in this clip?" should give the same answer if the bark moves. Equivariance: the output changes in the same way as the input. "Where is the bark?" should move when the bark moves.
Worked by hand. Take a signal on a loop of 6 positions, x = (0, 1, 3, 1, 0, 0). Apply a 3-tap filter with weights (1, 2, 1): each output is left neighbour + 2 × itself + right neighbour, wrapping round at the ends.
y = f(x) = (1, 5, 8, 5, 1, 0). Shift x right by 1: (0, 0, 1, 3, 1, 0) → f = (0, 1, 5, 8, 5, 1). That is y shifted right by 1: equivariant. Take the maximum: 8 both times: invariant.
The pattern above is the general recipe. Stack equivariant layers, so features move with the input. End with an invariant pool (max, mean, sum) when you need one answer per input. The whole model is then invariant.
The solid line is f(shifted input). The teal dashed line is the shifted f(input). If they overlap, the layer is equivariant. Slide the shift. Then switch the layer to dense.
A loop of 24 samples. The filter has 5 fixed weights; the dense layer has 24 × 24 fixed random weights. Errors are relative: |difference| / |output|.
Chapter 3 Why it works
Ask which linear layers commute with a shift, derive the answer, and count the weights it saves
Here is the claim: if you demand that a linear layer respects shifts, you get convolution, and nothing else. We prove it in three steps, on signals that wrap round a loop of n positions.
Step 1: write the shift as a matrix. A linear layer is y = Wx, with W an n × n matrix. The shift S moves every sample one place right: (Sx)i = xi−1, with indices wrapping round.
Step 2: demand equivariance. We need W(Sx) = S(Wx) for every x. Write out entry i of each side:
Step 3: read off the answer. Every diagonal of W is constant. So Wij depends only on i − j: Wij = c(i−j) mod n. Then yi = Σj ci−j xj. That is a convolution with kernel c. A matrix with constant, wrapping diagonals is called circulant.
Worked: n = 4 → W = c0 c3 c2 c1
c1 c0 c3 c2
c2 c1 c0 c3
c3 c2 c1 c0 4 free numbers instead of 16.
Add one more assumption, locality: only nearby samples interact, so cm = 0 unless m is small. The 4 numbers shrink to the kernel width k: 3, 5, 7. Two assumptions, shift symmetry and locality, give the ordinary conv layer.
Left: a 12 × 12 weight matrix (filled squares are positive, outlined are negative). Right: W(Sx) in the layer's colour against the teal S(Wx). Tie the diagonals, keep only neighbours, or both.
Tying replaces each diagonal by its average, which is the closest shift-respecting matrix to the random one. "Neighbours" keeps the main diagonal and one on each side, so k = 3. Error = |W(Sx) − S(Wx)| / |S(Wx)| on a fixed random x.
| Layer | Dense weights | Conv weights |
|---|---|---|
| 1-D, 1,000 samples in and out | 1,000,000 | 9 (k = 9) |
| 2-D, 224 × 224 RGB in, 64 maps out | 483,385,147,392 | 1,728 (3 × 3 × 3 × 64) |
Worked: dense = (3 · 224 · 224) × (64 · 224 · 224) = 150,528 × 3,211,264 = 483,385,147,392. Conv = 3 × 3 × 3 × 64 = 1,728. Ratio about 280 million. (Counts printed by the PyTorch code in Chapter 9.)
Weights are only half the story. A shared kernel also gets a gradient from every position of every example. One training image teaches the edge detector at 50,000 places. That is why the conv in the hero needs so few examples.
Chapter 4
Grow a receptive field layer by layer, see how dilation reaches far cheaply, and where pooling breaks the shift rule
Locality says that nearby samples belong together. A 3-tap filter looks at 3 samples. Stack two, and each output depends on 5 inputs. The set of inputs that can affect one output is its receptive field.
Worked: 4 layers, k = 3. Dilations 1, 1, 1, 1 → R = 1 + 2 × 4 = 9. Dilations 1, 2, 4, 8 → R = 1 + 2 × 15 = 31. Same 12 weights, more than three times the reach.
WaveNet uses this trick for raw audio. Its filters have 2 taps and dilations 1, 2, 4, …, 512, repeated. One such block sees 1,024 samples WaveNet · §2.1. Check: 1 + (1 + 2 + … + 512) = 1 + 1,023 = 1,024.
The top dot is one output. Lines trace every input it depends on. The two warm marks are clues the task needs together. Push them apart until the network cannot see both.
Stride 1, no pooling, 32 inputs. Weight count is layers × k for one channel.
Pooling summarises a small window, for example the max of every 2 samples, and keeps every second result. It halves the length, so the next layer's receptive field doubles. It also adds a little local invariance: a feature can move inside the window without changing the max.
But strided pooling breaks exact shift equivariance. Pool x = (0, 0, 4, 0, 0, 0) in pairs: (0, 4, 0). Shift x by 1: (0, 0, 0, 4, 0, 0) pools to (0, 4, 0), unchanged. Shift by 2: (0, 0, 4), moved. Only shifts by a multiple of the stride behave exactly.
Multi-scale designs (image pyramids, U-Nets, feature pyramids) run the same filters at several resolutions. The bias they add: a pattern means the same thing at different sizes, roughly. That is scale symmetry, and it is only approximate in real data.
Chapter 5
Build a model that cannot see the order of its inputs, derive its form, and train it against an MLP
A lidar scan is a list of 3-D points. The list order is an accident of the sensor. A team's players, the readings in a bag of sensors, the objects in a scene: all are sets. Reorder the list and the answer must not change.
Reordering is a permutation, written P. An MLP on the concatenated list is not invariant to it: position 1 has its own weights, as in Chapter 0. With N elements there are N! orders, and the MLP must learn each one from data. For 6 elements, that is 720 orders.
Zaheer et al. proved that a function on sets (with elements from a countable universe) is permutation invariant if and only if it can be written as below Deep Sets · Theorem 2:
Worked: X = {0.2, 0.9, 0.8}, φ(x) = (x, x2) → sum = (1.9, 0.04 + 0.81 + 0.64) = (1.9, 1.49). ρ(a, b) = b − a2/3 = 1.49 − 1.203 = 0.287: three times the variance of the set. Any order gives the same.
For the layers inside the network you often want equivariance instead: one output per element, which moves when the element moves. The paper's Lemma 3 says a linear layer x → Θx on M elements is permutation equivariant exactly when Θ = λI + γ11⊤ Deep Sets · Lemma 3. In words: each element keeps λ of itself and adds γ times the set's sum. Two weights, not M2.
Worked: λ = 2, γ = 0.5, x = (1, 2, 3), sum 6 → (2 + 3, 4 + 3, 6 + 3) = (5, 7, 9). Reorder to (3, 1, 2) → (9, 5, 7): the same outputs, reordered.
Task: does the set hold at least two numbers above 0.7? Both models train on N sets. Press Shuffle: the same six numbers, a new order. Watch which output moves.
Trained here, three seeds averaged, 500 test sets. MLP: 6→16→1 (129 weights). Deep Sets: φ = one layer of 16 ReLU units on each number, sum, then a linear ρ (49 weights). "Shuffle change" is the average |p(X) − p(PX)| over test sets. Illustrative numbers.
Self-attention computes Q = XWQ, K = XWK, V = XWV, then softmax(QK⊤/√d)V. Reorder the rows of X with P. Then Q, K and V get the same row order, QK⊤ becomes P(QK⊤)P⊤, and the row-wise softmax just reorders too. The output is P times the old output. So attention with no position information is permutation equivariant. Add a mean pool and it is invariant. The Set Transformer builds on exactly this.
Chapter 6
Relabel the nodes of a sensor network, see which models notice, and write message passing so it never does
Six temperature sensors hang in a building. Wires link some pairs. That is a graph: nodes (the sensors) with features, and edges (the links). The numbers 1 to 6 we give the sensors are just names. Rename them, and the building has not changed.
Renaming is a permutation P again. The node features become PX and the adjacency matrix A (1 where two nodes are linked) becomes PAP⊤. A model that reads the flattened A or X depends on the names. A 6-node graph has 720 namings.
The fix is to let each node talk only to its neighbours, with shared weights, and combine messages with a sum. This is message passing, the core of graph neural networks (Gilmer et al. 2017 gave it this general form):
Worked: path 1–2–3 with h = (1, 2, 3), messages ψ = hj, update φ = hi + sum → h′ = (1 + 2, 2 + 1 + 3, 3 + 2) = (3, 6, 5). Rename so the middle node is called 1: the 6 now sits at node 1. Graph total 14 either way.
Every piece is either per-node, per-edge, or a sum over neighbours. None of them reads a node's name. So the layer is permutation equivariant, and a final sum makes the graph output invariant. Each layer also adds one hop of reach: the graph version of Chapter 4's receptive field.
Each sensor shows its name and its message-passing output (green bar). Rename them: the bars stay with the sensors. The flat MLP reads the list in name order, so its answer jumps. More layers reach more hops from the ringed sensor.
Fixed weights, no training: h′ = tanh(0.8 h + 0.5 · sum of neighbours − 0.2). The flat MLP is one tanh layer of 8 units on the 6 readings in name order. Readings are made-up temperatures, centred.
Chapter 7
Separate the two assumptions behind sequence blocks, and show that a linear state-space model is a causal convolution
Time series carry two different assumptions. Keep them apart, because you may want one without the other.
Time-shift symmetry: the same rule applies at 9:00 and at 9:05. That is Chapter 3's shift, along time. It gives weight sharing over time. Causality: the output at time t may use inputs up to t, never after. A fall detector running live cannot read the next second.
Three blocks give both. A causal convolution only looks back: yt = Σk=0K−1 wk xt−k. In code, pad K − 1 zeros on the left only. A recurrent network (RNN) updates a hidden state with the same weights at every step: ht = tanh(W ht−1 + U xt). A state-space model (SSM) is the linear version of that update.
Worked: a = 0.5, b = c = 1. A single 1 at t = 3 gives y = (0, 0, 0, 1, 0.5, 0.25, 0.125, …). Nothing before t = 3: causal. Move the input to t = 7 and the same tail starts at 7: time-shift equivariant.
A single pulse (blue) enters at time t0. The block's output is below. Any output before t0 means the block read the future: red. Move the pulse; change the memory.
Kernels: causal (0.4, 0.3, 0.2, 0.1) on xt…xt−3; centred (0.1, 0.2, 0.4, 0.2, 0.1) on xt−2…xt+2; SSM with b = c = 1.
Long memory is where these blocks differ. A causal conv sees K steps back. An RNN can, in principle, remember forever, but its gradients through many steps tend to vanish or explode. A linear SSM with a near 1 remembers long and trains like a convolution, in parallel. Modern SSMs such as Mamba go one step further: they make a, b and c depend on the input, so the model can choose what to keep (Gu and Dao, 2023). That breaks the fixed-kernel convolution view; they compute it with a parallel scan instead.
Chapter 8
See what attention assumes and what it does not, why position encodings put the order back, and how much data the Vision Transformer needed
Attention compares every element with every other element. It does not assume that neighbours matter more (no locality). Without positions, it does not even know the order (Chapter 5). It assumes very little, so it must learn most structure from data. That is a weak bias.
Language and images do have order. "Dog bites man" is not "man bites dog". So Transformers add a position encoding et to each token: xt + et. Reordering the words now changes the inputs, so the outputs can differ. The position encoding is the order you removed, added back on purpose.
Worked: X = (dog, bites, man), PX = (man, bites, dog). Without E, the mean-pooled outputs are identical: distance 0. With E, slot 1 holds "dog + e1" in one sentence and "man + e1" in the other, so the pooled outputs differ.
Two sentences with the same words in a different order. The grids are the attention weights; the bars are the mean-pooled outputs (solid: sentence A, dashed: sentence B). Turn the position encodings on.
One attention layer with fixed random weights, 6-dimensional word vectors and sinusoidal position encodings. No training: the point is the symmetry, not the meaning.
The Vision Transformer (ViT) cut an image into 16 × 16 patches and fed them, with learned position embeddings, to a standard Transformer. Its authors spelled out the bias it lacks. A CNN builds locality, 2-D neighbourhoods and translation equivariance into every layer. In ViT, only the MLP layers are local and translation equivariant. Its position embeddings start with no information about 2-D positions ViT · §3.1.
Then they measured what that costs ViT · §4.3, Fig. 3, Fig. 4:
Their summary: "the convolutional inductive bias is useful for smaller datasets, but for larger ones, learning the relevant patterns directly from data is sufficient, even beneficial" ViT · §4.3. The crossover sat between tens of millions and about a hundred million pre-training images, not thousands.
The hero shows the same shape in miniature. With 8 or 16 examples, the conv is already near its best while attention is near chance. With 1,024, attention comes within a few points of the conv in the training range. It still fails on shifted tests: its position embeddings tie what it learned to where it saw it. A weak bias is not a weaker model. It is a model that pays for flexibility with data.
Chapter 9 Build it in PyTorchDid it work?
Write the symmetry checks in PyTorch, see augmentation as a soft bias, and read a data-efficiency curve
A symmetry claim is a testable statement. Never trust a diagram: measure it. The test is always the same: change the input by g, and compare.
Worked (by hand): Chapter 2's filter: f(Sx) = (0, 1, 5, 8, 5, 1) and S f(x) = (0, 1, 5, 8, 5, 1) → err = 0. A dense layer: the PyTorch run below gives err = 1.37 for a 1-sample shift.
import torch import torch.nn as nn torch.manual_seed(0) L = 32 # signal length x = torch.randn(1, 1, L) # (batch, channels, length) conv = nn.Conv1d(1, 4, kernel_size=7, padding=3, padding_mode='circular', bias=False) dense = nn.Linear(L, 4 * L, bias=False) # same output size, no weight sharing def shift(t, k): # circular shift along the last axis return torch.roll(t, shifts=k, dims=-1) def equivariance_error(f, x, k): a = f(shift(x, k)) # shift, then apply b = shift(f(x), k) # apply, then shift return ((a - b).norm() / b.norm()).item() f_conv = lambda t: conv(t) # (1, 4, 32) f_dense = lambda t: dense(t.flatten(1)).view(1, 4, L) # (1, 4, 32) with torch.no_grad(): for k in (1, 5): print(f"shift {k}: conv {equivariance_error(f_conv, x, k):.2e}" f" dense {equivariance_error(f_dense, x, k):.2f}") print("params: conv", sum(p.numel() for p in conv.parameters()), " dense", sum(p.numel() for p in dense.parameters())) # shift 1: conv 0.00e+00 dense 1.37 # shift 5: conv 0.00e+00 dense 1.42 # params: conv 28 dense 4096
Note padding_mode='circular'. With zero padding, samples shifted off one end are lost and zeros come in at the other, so the error near the borders is not zero. That is a real effect in CNNs, not a bug in the test.
import torch import torch.nn as nn class DeepSets(nn.Module): """f(X) = rho( sum_i phi(x_i) ), Zaheer et al. 2017, Theorem 2.""" def __init__(self, d_in=1, d_hid=16, d_out=1, pool="sum"): super().__init__() self.phi = nn.Sequential(nn.Linear(d_in, d_hid), nn.ReLU(), nn.Linear(d_hid, d_hid), nn.ReLU()) self.rho = nn.Sequential(nn.Linear(d_hid, d_hid), nn.ReLU(), nn.Linear(d_hid, d_out)) self.pool = pool def forward(self, x, mask=None): # x: (B, N, d_in); mask: (B, N), True = real h = self.phi(x) # (B, N, d_hid): same phi for every element if mask is not None: h = h * mask.unsqueeze(-1) # padding contributes zero z = h.sum(dim=1) # (B, d_hid): order is forgotten here if self.pool == "mean": n = mask.sum(1, keepdim=True) if mask is not None else x.shape[1] z = z / n return self.rho(z) # (B, d_out) torch.manual_seed(0) model = DeepSets() x = torch.rand(8, 6, 1) # 8 sets of 6 numbers perm = torch.randperm(6) with torch.no_grad(): err = (model(x) - model(x[:, perm])).abs().max().item() print(f"permutation test, max |f(X) - f(PX)|: {err:.1e}") # permutation test, max |f(X) - f(PX)|: 6.0e-08 (float rounding, not a leak)
import torch.nn as nn def n_params(m): return sum(p.numel() for p in m.parameters()) # one layer, 224 x 224 RGB image in, 64 feature maps of the same size out H = W = 224 dense = n_params(nn.Linear(3 * H * W, 64 * H * W, bias=False, device="meta")) # meta: no memory conv = n_params(nn.Conv2d(3, 64, kernel_size=3, padding=1, bias=False)) print(f"dense: {dense:,} conv 3x3: {conv:,} ratio: {dense / conv:,.0f}") # 1-D: length 1,000 in, length 1,000 out, one channel print("1-D dense:", n_params(nn.Linear(1000, 1000, bias=False)), " 1-D conv k=9:", n_params(nn.Conv1d(1, 1, 9, padding=4, bias=False))) # a set layer (Zaheer et al. Lemma 3): Theta = lambda*I + gamma*11^T has 2 numbers M = 100 print("set layer, M = 100: dense", M * M, " permutation-equivariant", 2) # dense: 483,385,147,392 conv 3x3: 1,728 ratio: 279,736,775 # 1-D dense: 1000000 1-D conv k=9: 9 # set layer, M = 100: dense 10000 permutation-equivariant 2
The same blocks in one line each: nn.Conv1d(c_in, c_out, k) for shifts; nn.Conv1d(c, c, k, dilation=d) with left padding (k−1)·d for causal dilated stacks; torch_geometric.nn.aggr.DeepSetsAggregation(local_nn, global_nn) for sets; torch_geometric.nn.MessagePassing or GCNConv for graphs; nn.TransformerEncoderLayer without position encodings for a permutation-equivariant set block.
You can also teach a symmetry with data. Data augmentation trains on changed copies: shifted signals, rotated images, shuffled sets, all with the old label. The model is pushed toward invariance, but only near the data it saw, and never exactly. Built-in symmetry is a hard constraint; augmentation is a soft one.
The hero's task, run at N = 8, 32, 128 and 512 for four models. Switch the test set between the training range and shifted wiggles. The right panel is the measured shift sensitivity: how much each model's output moves when the input shifts by 1 to 8 samples.
Trained here, 400 Adam steps each. Augmentation rolls each training signal by a random 0 to 31 samples (wrapping round), which sometimes cuts the wiggle in two. Shift sensitivity = mean |p(x) − p(shifted x)| on 200 in-range signals, shifting in zeros. Illustrative numbers.
Chapter 10 Where else it worksWhen not to use it
List each field's real symmetries, meet rotation-equivariant networks, and see what happens when you assume a symmetry the data does not have
The method transfers: list the changes that do not change the answer, then pick blocks that respect exactly those. What changes from field to field is the list.
| Field | Real symmetries | Not a symmetry | Blocks |
|---|---|---|---|
| Images | Translation; small scale changes; often left-right flips | Up-down flips for scenes; rotation for text and digits | Conv, pooling, pyramids; ViT at very large scale |
| Audio spectrograms | Shift in time | Shift in frequency, only roughly: it changes pitch, and low and high bands sound different | Conv along time, frequency-aware layers or positions along frequency |
| Point clouds, molecules | Permutation of points; rotation; translation | Mirror images, for chiral molecules | Deep Sets, PointNet, EGNN |
| Sensor networks | Renaming nodes | Moving a sensor to a different place | Message passing with real attributes as features |
| Multi-agent robots | Renaming the other agents; moving or rotating the whole scene | Swapping "me" with another agent | GNN or attention over agents, relative coordinates, a "self" flag |
| Tabular data | Order of rows | Order or mixing of columns: age is not income | Tree ensembles; MLPs with per-column embeddings |
Spectrograms are the subtle case. Shifting a sound in time keeps its meaning. Shifting it up the frequency axis changes its pitch, and on a mel scale the bands are not even equally spaced. A 2-D conv assumes both shifts are free. It still works well in practice, but the bias is half right. The AudioFM lesson treats the two axes differently on purpose: one view hides whole time columns, the other hides whole frequency bands.
Tabular data has no column symmetry. Grinsztajn et al. benchmarked 45 tabular datasets and found tree-based models still ahead of deep nets on medium-sized data (about 10K samples). One reason they give: good tabular models must "preserve the orientation of the data". Columns have fixed meanings, and a model that freely mixes them assumes a symmetry that is not there.
Molecules and particle systems keep their physics when you rotate, reflect or move the whole system. That is the Euclidean group E(n). Satorras, Hoogeboom and Welling's E(n)-equivariant GNN (EGNN) builds it in simply. Messages use only distances: mij = φe(hi, hj, ‖xi − xj‖2, aij). Positions move only along relative vectors: xi ← xi + C Σj(xi − xj) φx(mij) EGNN · §3, eq. 3–6. Distances do not change under rotation, and relative vectors rotate with the system. On a 3-D charged-particle task, EGNN's error was 0.0071, against 0.0107 for a plain GNN and 0.0104 for the next best, 32% lower EGNN · Table 2. It also beat the plain GNN from 100 to 50,000 training samples EGNN · Fig. 2.
A 9 is a 6 turned upside down. Make a digit classifier fully rotation invariant, and it can no longer tell them apart. The data's real symmetry is "small tilts", not "any rotation".
Test digits are tilted by up to the angle you set. Three template classifiers vote: one with no rotation invariance, one invariant to any rotation, one invariant to tilts up to ±40°. The glyph shows a 6 at your tilt.
Computed live on 160 synthetic tilted digits made of 34 points each, with a little jitter. Each classifier picks the template (upright 6 or 9) with the smaller average nearest-point distance, after trying the rotations it is invariant to. Ties are a coin flip. Illustrative numbers.
| Block | Use it when | Avoid it when | Use instead |
|---|---|---|---|
| Convolution | Patterns are local and can be anywhere | Absolute position matters (aligned faces, fixed sensor slots); long-range links | Add position channels; dilation; attention |
| Full rotation invariance | Molecules, satellite tiles, microscopy | Up and down mean something (6 and 9, text, scenes) | Small-tilt augmentation |
| Deep Sets / sum pool | Order is an accident | Order carries meaning (text, time); interactions between elements dominate | Positions; attention between elements |
| Message passing | Relations are given and names are arbitrary | Edges are guesses; far nodes must talk (over-smoothing with many layers) | Learned edges, graph Transformers |
| Causal block | Prediction or control runs live | The whole recording is available | Bidirectional or centred blocks |
| Attention | Data is huge, or structure is unknown | Data is small and the symmetry is known | The matching biased block |
Chapter 11
A three-step decision procedure, the whole idea on one screen, honest limits, and where to go next
The procedure has three steps. 1. List the changes to an input that leave the answer alone (or change it in a known way). 2. Pick blocks that respect exactly those, and no more. 3. Test: run the symmetry check, a held-out transformation split, and the wrong-symmetry check.
Choose your data. The grid marks each candidate symmetry: yes, roughly, or no. The bottom row is the block to start with and the check to run first.
A starting point from this lesson, not a law. The data-efficiency curve decides.
| Part | In one line |
|---|---|
| Inductive bias | What a learner assumes about unseen inputs. Needed: 2(2n−m) rules fit any m labels. |
| Symmetry | Invariant: f(g·x) = f(x). Equivariant: f(g·x) = g·f(x). Equivariant layers, then an invariant pool. |
| Shift → conv | WS = SW forces constant diagonals: yi = Σ ci−jxj. (B, C, L) → (B, C′, L). n2 weights become k. |
| Locality, scale | R = 1 + Σ(k−1)dl; dilation doubles reach cheaply; strided pools break exact equivariance. |
| Permutation → sets | f(X) = ρ(Σφ(x)); (B, N, D) → (B, D′). Equivariant layer λI + γ11⊤. |
| Graphs | hi′ = φ(hi, Σj∈N(i)ψ(hi, hj)); (N, D) + edges → (N, D′). |
| Time | Shift symmetry and causality are separate. SSM = causal conv with kernel c ak b. |
| Attention | Permutation equivariant, global, weak bias. Position encodings re-add order. ViT needed about 90M+ images to pass ResNets. |
| Augmentation | The soft version: approximate, costs data, good for messy symmetries. |
| Did it work? | Symmetry error near 10−7; held-out transform split; data-efficiency curve; wrong-symmetry check. |
| Failure modes | False symmetry (6/9, table columns, frequency); too little reach; leaks from padding, IDs, positions; peeking into the future. |
This lesson is part of the Builder's Toolkit series on reusable blocks and how to choose them.
Now press Present or Teach and explain, out loud and from memory, why a shift symmetry forces a convolution and why attention needs more data. Then go back to the hero and find the smallest N at which attention comes close to the conv in the training range.