Inductive Bias and Symmetry

A pattern can turn up anywhere in a signal. A dense network must learn it again at every position. A convolution learns it once, because it assumes a shift changes nothing. That assumption is an inductive bias. Choose the one your data really has.

Learn what an inductive bias is and how a symmetry turns into a block: convolution for shifts, Deep Sets for sets, message passing for graphs, recurrence for time. See why attention needs more data, and test any block with a symmetry check in PyTorch.

Three models train in your browser on N examples. In training, the pattern only appears in the left half. Slide the test shift to move the pattern right, and watch which model still finds it. Then change N.

You need vectors, matrix multiplication, and the idea of a trained classifier. We build the rest from zero.

Find the wiggle, wherever it is

Ready
32
0

Synthetic data, illustrative numbers. Each signal is 32 samples of noise (spread 0.5). Half the signals hold an 8-sample wiggle; the other half hold the same wiggle upside down, so loudness alone cannot tell them apart. Training wiggles start at samples 0 to 8. A test shift of s moves them to start at s to s + 8. Models: a dense MLP (32→16→1, 545 weights), a convolution (4 filters of width 7, max over positions, 37 weights), and a small attention model (8 patches of 4 samples, learned position embeddings, one attention layer, 297 weights). Each trains 400 steps of Adam here and is tested on 300 new signals per shift.

Chapter 0

The pattern that moved

Train a dense detector on patterns near the start of a window, move the pattern, and see exactly why the detector goes blind

You fit a vibration sensor to a motor. A worn bearing makes a short wiggle in the signal. You cut the signal into windows of 32 samples and label each one: "wiggle" or "no wiggle". You want a model that flags the wiggle.

By chance, your recordings caught the wiggle near the start of each window. In the field it can start anywhere. The wiggle is the same wiggle. Only its position changed.

The simplest detector is one dense layer: a weighted sum of all 32 samples, s = w1x1 + … + w32x32, then a threshold. It has one weight for each position. The device trains one, and a second detector that slides one short template along the signal.

What did each detector learn?

Top: one test signal; the warm band is the wiggle. Middle: the dense detector's 32 weights. Bottom: the sliding detector's 7 weights and its response at every position. Drag the wiggle along the window.

3

Both detectors are linear and trained here by gradient descent on 200 synthetic windows. The sliding detector takes the largest response over positions. Illustrative numbers.

Look at the dense weights. They hold a copy of the wiggle at the positions where training wiggles lived. Everywhere else they are near zero. Those weights never saw a wiggle, so they never learned one.

xi
The signal at position i (here, 32 positions).
wi
The weight for position i. Each position has its own weight, learned only from what appeared there.
s
The detector's score. Above a threshold means "wiggle".

Worked: a 3-sample bump (1, 2, 1) at positions 3 to 5, where w = (1, 2, 1): s = 1 + 4 + 1 = 6. The same bump at positions 20 to 22, where w = (0, 0, 0): s = 0. Same bump, no detection.

Now switch training to "anywhere". The dense weights blur. A wiggle at position 3 and the same wiggle at position 4 want different weights at the same place. One weighted sum cannot hold all shifted copies at once, so its accuracy now swings from spot to spot. A dense network needs a separate unit, and separate examples, for each position.

The sliding detector has no such problem. It uses the same 7 weights at every position, and takes the best match. It learned the wiggle once. It finds it anywhere. That "same weights everywhere" choice is an assumption made before seeing any data: a shift does not change what a pattern means. Assumptions like this are the subject of this lesson.

The lesson in one line. Every model assumes something about inputs it has not seen. When the assumption matches a real symmetry of your data, the model learns from far fewer examples. When it does not, the model is blind to the truth.
  1. What an inductive bias is: the assumptions every learner needs, and why no single one wins.
  2. Invariance and equivariance: the two ways a model can respect a symmetry.
  3. Why it works: convolution from shifts: derive convolution as the only shift-respecting linear map.
  4. Locality and scale: receptive fields, pooling, dilation.
  5. Sets: permutation symmetry and Deep Sets.
  6. Graphs: relabelling nodes, and message passing.
  7. Time: causality, recurrence and state-space models.
  8. Attention: a weak bias, position encodings, and the ViT data finding.
  9. Build it in PyTorch, and did it work?: symmetry tests, augmentation, data-efficiency curves.
  10. Where else it works, and when not: six fields, rotation, and 6 against 9.
  11. Cheat sheet: a decision procedure for your next dataset.
A dense detector was trained only on wiggles in samples 0 to 15. A test wiggle sits at samples 20 to 27. Why does it miss?

Chapter 1

What an inductive bias is

Count how many rules fit your data, see why a learner must guess, and watch four different guesses fill the same gap

Training data only tells you about inputs you have seen. Every prediction on a new input is a guess. The guess comes from what the learner assumed before it saw any data. That set of assumptions is its inductive bias.

You already know some. Linear regression assumes the answer is a straight line. Nearest neighbours assumes that close inputs have close answers. The sliding detector of Chapter 0 assumes a pattern means the same thing at every position.

Why a learner must assume something

Take a task with 3 yes/no inputs. There are 23 = 8 possible inputs. A rule assigns yes or no to each, so there are 28 = 256 possible rules. Now label 5 of the 8 inputs.

n
Number of yes/no inputs. There are 2n possible inputs.
m
Number of different inputs you have labels for.
R
Number of rules that agree with every label you have.

Worked: n = 3, m = 5 → R = 23 = 8 rules fit the data. On each of the 3 unseen inputs, 4 of them say yes and 4 say no. The data alone gives a 50/50 split.

So the data cannot choose. Something else must. This is the idea behind the no free lunch theorem (Wolpert, 1996): averaged over every possible target rule, all learners have the same error on unseen inputs. A learner beats chance only on problems where its assumptions are true. There is no bias-free learner, only biases that fit your problem and biases that do not.

Four guesses for the same gap

Six blue points come from a hidden rule on the left half. Each bias fills the right half its own way. Pick a bias, then change the hidden rule. No bias wins on both.

Computed live. The error is the root-mean-square gap between the prediction and the hidden rule on the unseen half. The wave bias assumes the true period (0.5); the smooth curve uses a Gaussian kernel of width 0.12. Illustrative numbers.

The same six points, four sensible answers. The line bias is right about lines and wrong about waves. The wave bias is the reverse. The lookup table copies the nearest point and learns no shape at all. None of them is "unbiased". Each one is a bet about the world.

This lesson is about the most useful kind of bias: symmetry. A symmetry of your data is a change you can make to an input that does not change the answer, or changes it in a way you know. Shift a sound in time: still a dog bark. Reorder the points of a point cloud: still the same chair. Once you know a symmetry, you can build it into the block, and the model never has to learn it.

You have labels for 6 of the 16 inputs of a task with 4 yes/no inputs. How many rules agree with all your labels?

Chapter 2

Invariance and equivariance

Define the two ways a model can respect a symmetry, work one example by hand, and check both numerically

Let g be a change to the input: shift by 3 samples, reorder the points, rotate by 30°. Write g·x for the changed input. A model f can respect g in two ways.

Invariance: the output does not change. "Is there a dog bark in this clip?" should give the same answer if the bark moves. Equivariance: the output changes in the same way as the input. "Where is the bark?" should move when the bark moves.

g
A change to the input, from a family of changes called a group: you can combine two changes, and undo any change.
invariant
Changing the input leaves the output alone.
equivariant
Changing the input changes the output in the matching way. "Change, then apply f" equals "apply f, then change".

Worked by hand. Take a signal on a loop of 6 positions, x = (0, 1, 3, 1, 0, 0). Apply a 3-tap filter with weights (1, 2, 1): each output is left neighbour + 2 × itself + right neighbour, wrapping round at the ends.

y = f(x) = (1, 5, 8, 5, 1, 0). Shift x right by 1: (0, 0, 1, 3, 1, 0) → f = (0, 1, 5, 8, 5, 1). That is y shifted right by 1: equivariant. Take the maximum: 8 both times: invariant.

The pattern above is the general recipe. Stack equivariant layers, so features move with the input. End with an invariant pool (max, mean, sum) when you need one answer per input. The whole model is then invariant.

Shift, then apply; or apply, then shift

The solid line is f(shifted input). The teal dashed line is the shifted f(input). If they overlap, the layer is equivariant. Slide the shift. Then switch the layer to dense.

5

A loop of 24 samples. The filter has 5 fixed weights; the dense layer has 24 × 24 fixed random weights. Errors are relative: |difference| / |output|.

A model maps an image to a heat map of where cats are. You shift the image 10 pixels left. What should a good model do?

Chapter 3 Why it works

Convolution falls out of the shift symmetry

Ask which linear layers commute with a shift, derive the answer, and count the weights it saves

Here is the claim: if you demand that a linear layer respects shifts, you get convolution, and nothing else. We prove it in three steps, on signals that wrap round a loop of n positions.

Step 1: write the shift as a matrix. A linear layer is y = Wx, with W an n × n matrix. The shift S moves every sample one place right: (Sx)i = xi−1, with indices wrapping round.

Step 2: demand equivariance. We need W(Sx) = S(Wx) for every x. Write out entry i of each side:

(WSx)i
Shift first, then apply W. Re-index the sum so each term multiplies xj.
(SWx)i
Apply W first, then shift: entry i is entry i−1 of Wx.
Wi+1,j+1 = Wi,j
For the two sums to agree for every x, the coefficients must match. Moving one step down and one step right never changes the weight.

Step 3: read off the answer. Every diagonal of W is constant. So Wij depends only on i − j: Wij = c(i−j) mod n. Then yi = Σj ci−j xj. That is a convolution with kernel c. A matrix with constant, wrapping diagonals is called circulant.

Worked: n = 4 → W = c0 c3 c2 c1
c1 c0 c3 c2
c2 c1 c0 c3
c3 c2 c1 c0
  4 free numbers instead of 16.

Add one more assumption, locality: only nearby samples interact, so cm = 0 unless m is small. The 4 numbers shrink to the kernel width k: 3, 5, 7. Two assumptions, shift symmetry and locality, give the ordinary conv layer.

Tie the weights

Left: a 12 × 12 weight matrix (filled squares are positive, outlined are negative). Right: W(Sx) in the layer's colour against the teal S(Wx). Tie the diagonals, keep only neighbours, or both.

Tying replaces each diagonal by its average, which is the closest shift-respecting matrix to the random one. "Neighbours" keeps the main diagonal and one on each side, so k = 3. Error = |W(Sx) − S(Wx)| / |S(Wx)| on a fixed random x.

What the bias buys: weights

LayerDense weightsConv weights
1-D, 1,000 samples in and out1,000,0009 (k = 9)
2-D, 224 × 224 RGB in, 64 maps out483,385,147,3921,728 (3 × 3 × 3 × 64)

Worked: dense = (3 · 224 · 224) × (64 · 224 · 224) = 150,528 × 3,211,264 = 483,385,147,392. Conv = 3 × 3 × 3 × 64 = 1,728. Ratio about 280 million. (Counts printed by the PyTorch code in Chapter 9.)

Weights are only half the story. A shared kernel also gets a gradient from every position of every example. One training image teaches the edge detector at 50,000 places. That is why the conv in the hero needs so few examples.

A linear layer on a loop of 8 samples commutes with the shift. How many free weights can it have, and what is it?

Chapter 4

Locality and scale

Grow a receptive field layer by layer, see how dilation reaches far cheaply, and where pooling breaks the shift rule

Locality says that nearby samples belong together. A 3-tap filter looks at 3 samples. Stack two, and each output depends on 5 inputs. The set of inputs that can affect one output is its receptive field.

k
Kernel width: how many taps each filter has.
dl
The dilation of layer l: the gap between taps. d = 1 is an ordinary filter; d = 4 reads every fourth sample.
R
Receptive field: how many input samples one top output can see (stride 1).

Worked: 4 layers, k = 3. Dilations 1, 1, 1, 1 → R = 1 + 2 × 4 = 9. Dilations 1, 2, 4, 8 → R = 1 + 2 × 15 = 31. Same 12 weights, more than three times the reach.

WaveNet uses this trick for raw audio. Its filters have 2 taps and dilations 1, 2, 4, …, 512, repeated. One such block sees 1,024 samples WaveNet · §2.1. Check: 1 + (1 + 2 + … + 512) = 1 + 1,023 = 1,024.

How far can one output see?

The top dot is one output. Lines trace every input it depends on. The two warm marks are clues the task needs together. Push them apart until the network cannot see both.

3
6

Stride 1, no pooling, 32 inputs. Weight count is layers × k for one channel.

Pooling and scale

Pooling summarises a small window, for example the max of every 2 samples, and keeps every second result. It halves the length, so the next layer's receptive field doubles. It also adds a little local invariance: a feature can move inside the window without changing the max.

But strided pooling breaks exact shift equivariance. Pool x = (0, 0, 4, 0, 0, 0) in pairs: (0, 4, 0). Shift x by 1: (0, 0, 0, 4, 0, 0) pools to (0, 4, 0), unchanged. Shift by 2: (0, 0, 4), moved. Only shifts by a multiple of the stride behave exactly.

Multi-scale designs (image pyramids, U-Nets, feature pyramids) run the same filters at several resolutions. The bias they add: a pattern means the same thing at different sizes, roughly. That is scale symmetry, and it is only approximate in real data.

Five layers of 2-tap filters with dilations 1, 2, 4, 8, 16. What is the receptive field?

Chapter 5

Sets: when order means nothing

Build a model that cannot see the order of its inputs, derive its form, and train it against an MLP

A lidar scan is a list of 3-D points. The list order is an accident of the sensor. A team's players, the readings in a bag of sensors, the objects in a scene: all are sets. Reorder the list and the answer must not change.

Reordering is a permutation, written P. An MLP on the concatenated list is not invariant to it: position 1 has its own weights, as in Chapter 0. With N elements there are N! orders, and the MLP must learn each one from data. For 6 elements, that is 720 orders.

The form every set function takes

Zaheer et al. proved that a function on sets (with elements from a countable universe) is permutation invariant if and only if it can be written as below Deep Sets · Theorem 2:

φ
A network applied to each element on its own, with the same weights for every element.
ρ
A network applied once, to the pooled sum.
f(X)
The output for the whole set. The sum does not care about order, so neither does f.

Worked: X = {0.2, 0.9, 0.8}, φ(x) = (x, x2) → sum = (1.9, 0.04 + 0.81 + 0.64) = (1.9, 1.49). ρ(a, b) = b − a2/3 = 1.49 − 1.203 = 0.287: three times the variance of the set. Any order gives the same.

For the layers inside the network you often want equivariance instead: one output per element, which moves when the element moves. The paper's Lemma 3 says a linear layer x → Θx on M elements is permutation equivariant exactly when Θ = λI + γ11⊤ Deep Sets · Lemma 3. In words: each element keeps λ of itself and adds γ times the set's sum. Two weights, not M2.

Worked: λ = 2, γ = 0.5, x = (1, 2, 3), sum 6 → (2 + 3, 4 + 3, 6 + 3) = (5, 7, 9). Reorder to (3, 1, 2) → (9, 5, 7): the same outputs, reordered.

Shuffle the set

Task: does the set hold at least two numbers above 0.7? Both models train on N sets. Press Shuffle: the same six numbers, a new order. Watch which output moves.

16

Trained here, three seeds averaged, 500 test sets. MLP: 6→16→1 (129 weights). Deep Sets: φ = one layer of 16 ReLU units on each number, sum, then a linear ρ (49 weights). "Shuffle change" is the average |p(X) − p(PX)| over test sets. Illustrative numbers.

Attention without positions is a set block too

Self-attention computes Q = XWQ, K = XWK, V = XWV, then softmax(QK⊤/√d)V. Reorder the rows of X with P. Then Q, K and V get the same row order, QK⊤ becomes P(QK⊤)P⊤, and the row-wise softmax just reorders too. The output is P times the old output. So attention with no position information is permutation equivariant. Add a mean pool and it is invariant. The Set Transformer builds on exactly this.

A permutation-equivariant linear layer acts on a set of 100 elements, one number each. How many weights does Zaheer et al.'s Lemma 3 allow?

Chapter 6

Graphs: same network, new labels

Relabel the nodes of a sensor network, see which models notice, and write message passing so it never does

Six temperature sensors hang in a building. Wires link some pairs. That is a graph: nodes (the sensors) with features, and edges (the links). The numbers 1 to 6 we give the sensors are just names. Rename them, and the building has not changed.

Renaming is a permutation P again. The node features become PX and the adjacency matrix A (1 where two nodes are linked) becomes PAP⊤. A model that reads the flattened A or X depends on the names. A 6-node graph has 720 namings.

Message passing

The fix is to let each node talk only to its neighbours, with shared weights, and combine messages with a sum. This is message passing, the core of graph neural networks (Gilmer et al. 2017 gave it this general form):

ψ(hi, hj)
The message from neighbour j to node i. Same small network on every edge.
φ
The update: combines the node's old state with the sum of its messages. Same network on every node.
hi′
The node's new state. Sum all nodes at the end for one answer per graph.

Worked: path 1–2–3 with h = (1, 2, 3), messages ψ = hj, update φ = hi + sum → h′ = (1 + 2, 2 + 1 + 3, 3 + 2) = (3, 6, 5). Rename so the middle node is called 1: the 6 now sits at node 1. Graph total 14 either way.

Every piece is either per-node, per-edge, or a sum over neighbours. None of them reads a node's name. So the layer is permutation equivariant, and a final sum makes the graph output invariant. Each layer also adds one hop of reach: the graph version of Chapter 4's receptive field.

Rename the sensors

Each sensor shows its name and its message-passing output (green bar). Rename them: the bars stay with the sensors. The flat MLP reads the list in name order, so its answer jumps. More layers reach more hops from the ringed sensor.

1

Fixed weights, no training: h′ = tanh(0.8 h + 0.5 · sum of neighbours − 0.2). The flat MLP is one tanh layer of 8 units on the 6 readings in name order. Readings are made-up temperatures, centred.

You rename the nodes of a graph and feed it to a message-passing network with a sum readout. What happens to the node outputs and the graph output?

Chapter 7

Time: the same rule, and no peeking

Separate the two assumptions behind sequence blocks, and show that a linear state-space model is a causal convolution

Time series carry two different assumptions. Keep them apart, because you may want one without the other.

Time-shift symmetry: the same rule applies at 9:00 and at 9:05. That is Chapter 3's shift, along time. It gives weight sharing over time. Causality: the output at time t may use inputs up to t, never after. A fall detector running live cannot read the next second.

Three blocks give both. A causal convolution only looks back: yt = Σk=0K−1 wk xt−k. In code, pad K − 1 zeros on the left only. A recurrent network (RNN) updates a hidden state with the same weights at every step: ht = tanh(W ht−1 + U xt). A state-space model (SSM) is the linear version of that update.

st
The state: a running memory of the past. In real SSMs it is a vector and a is a matrix.
a
How much memory survives one step. Between 0 and 1 here.
c ak b
Unroll the recursion: the SSM is a causal convolution whose kernel decays as ak. Infinitely long, but only 3 numbers.

Worked: a = 0.5, b = c = 1. A single 1 at t = 3 gives y = (0, 0, 0, 1, 0.5, 0.25, 0.125, …). Nothing before t = 3: causal. Move the input to t = 7 and the same tail starts at 7: time-shift equivariant.

Does the output peek into the future?

A single pulse (blue) enters at time t0. The block's output is below. Any output before t0 means the block read the future: red. Move the pulse; change the memory.

12
0.70

Kernels: causal (0.4, 0.3, 0.2, 0.1) on xt…xt−3; centred (0.1, 0.2, 0.4, 0.2, 0.1) on xt−2…xt+2; SSM with b = c = 1.

Long memory is where these blocks differ. A causal conv sees K steps back. An RNN can, in principle, remember forever, but its gradients through many steps tend to vanish or explode. A linear SSM with a near 1 remembers long and trains like a convolution, in parallel. Modern SSMs such as Mamba go one step further: they make a, b and c depend on the input, so the model can choose what to keep (Gu and Dao, 2023). That breaks the fixed-kernel convolution view; they compute it with a parallel scan instead.

A linear SSM has a = 0.8, b = 1, c = 2. A single 1 enters at t = 0. What is the output at t = 2?

Chapter 8

Attention: a weak bias that learns structure

See what attention assumes and what it does not, why position encodings put the order back, and how much data the Vision Transformer needed

Attention compares every element with every other element. It does not assume that neighbours matter more (no locality). Without positions, it does not even know the order (Chapter 5). It assumes very little, so it must learn most structure from data. That is a weak bias.

Position encodings: putting the order back

Language and images do have order. "Dog bites man" is not "man bites dog". So Transformers add a position encoding et to each token: xt + et. Reordering the words now changes the inputs, so the outputs can differ. The position encoding is the order you removed, added back on purpose.

Attn
Self-attention with shared WQ, WK, WV. Reorder in, reorder out.
E
The position encodings, one row per slot. They stay in place when the tokens move, so they tell the model which slot a token is in.

Worked: X = (dog, bites, man), PX = (man, bites, dog). Without E, the mean-pooled outputs are identical: distance 0. With E, slot 1 holds "dog + e1" in one sentence and "man + e1" in the other, so the pooled outputs differ.

Dog bites man

Two sentences with the same words in a different order. The grids are the attention weights; the bars are the mean-pooled outputs (solid: sentence A, dashed: sentence B). Turn the position encodings on.

One attention layer with fixed random weights, 6-dimensional word vectors and sinusoidal position encodings. No training: the point is the symmetry, not the meaning.

The price of a weak bias: data

The Vision Transformer (ViT) cut an image into 16 × 16 patches and fed them, with learned position embeddings, to a standard Transformer. Its authors spelled out the bias it lacks. A CNN builds locality, 2-D neighbourhoods and translation equivariance into every layer. In ViT, only the MLP layers are local and translation equivariant. Its position embeddings start with no information about 2-D positions ViT · §3.1.

Then they measured what that costs ViT · §4.3, Fig. 3, Fig. 4:

Their summary: "the convolutional inductive bias is useful for smaller datasets, but for larger ones, learning the relevant patterns directly from data is sufficient, even beneficial" ViT · §4.3. The crossover sat between tens of millions and about a hundred million pre-training images, not thousands.

The hero shows the same shape in miniature. With 8 or 16 examples, the conv is already near its best while attention is near chance. With 1,024, attention comes within a few points of the conv in the training range. It still fails on shifted tests: its position embeddings tie what it learned to where it saw it. A weak bias is not a weaker model. It is a model that pays for flexibility with data.

According to the ViT paper's JFT subset experiment, when did ViT-B/32 beat ResNet50 in linear few-shot ImageNet accuracy?

Chapter 9 Build it in PyTorchDid it work?

Build it, test the symmetry, measure the data it saves

Write the symmetry checks in PyTorch, see augmentation as a soft bias, and read a data-efficiency curve

A symmetry claim is a testable statement. Never trust a diagram: measure it. The test is always the same: change the input by g, and compare.

err
Relative equivariance error. For an invariance check, g acts on the output as "do nothing". Built-in symmetry gives about 10−7 (float rounding). Learned symmetry gives something larger.

Worked (by hand): Chapter 2's filter: f(Sx) = (0, 1, 5, 8, 5, 1) and S f(x) = (0, 1, 5, 8, 5, 1) → err = 0. A dense layer: the PyTorch run below gives err = 1.37 for a 1-sample shift.

From scratch: the translation test

import torch
import torch.nn as nn

torch.manual_seed(0)
L = 32                                     # signal length
x = torch.randn(1, 1, L)                   # (batch, channels, length)

conv = nn.Conv1d(1, 4, kernel_size=7, padding=3, padding_mode='circular', bias=False)
dense = nn.Linear(L, 4 * L, bias=False)    # same output size, no weight sharing

def shift(t, k):                           # circular shift along the last axis
    return torch.roll(t, shifts=k, dims=-1)

def equivariance_error(f, x, k):
    a = f(shift(x, k))                     # shift, then apply
    b = shift(f(x), k)                     # apply, then shift
    return ((a - b).norm() / b.norm()).item()

f_conv = lambda t: conv(t)                                   # (1, 4, 32)
f_dense = lambda t: dense(t.flatten(1)).view(1, 4, L)        # (1, 4, 32)

with torch.no_grad():
    for k in (1, 5):
        print(f"shift {k}: conv {equivariance_error(f_conv, x, k):.2e}"
              f"   dense {equivariance_error(f_dense, x, k):.2f}")

print("params: conv", sum(p.numel() for p in conv.parameters()),
      " dense", sum(p.numel() for p in dense.parameters()))
# shift 1: conv 0.00e+00   dense 1.37
# shift 5: conv 0.00e+00   dense 1.42
# params: conv 28  dense 4096

Note padding_mode='circular'. With zero padding, samples shifted off one end are lost and zeros come in at the other, so the error near the borders is not zero. That is a real effect in CNNs, not a bug in the test.

From scratch: Deep Sets and the permutation test

import torch
import torch.nn as nn

class DeepSets(nn.Module):
    """f(X) = rho( sum_i phi(x_i) ), Zaheer et al. 2017, Theorem 2."""
    def __init__(self, d_in=1, d_hid=16, d_out=1, pool="sum"):
        super().__init__()
        self.phi = nn.Sequential(nn.Linear(d_in, d_hid), nn.ReLU(),
                                 nn.Linear(d_hid, d_hid), nn.ReLU())
        self.rho = nn.Sequential(nn.Linear(d_hid, d_hid), nn.ReLU(),
                                 nn.Linear(d_hid, d_out))
        self.pool = pool

    def forward(self, x, mask=None):       # x: (B, N, d_in); mask: (B, N), True = real
        h = self.phi(x)                    # (B, N, d_hid): same phi for every element
        if mask is not None:
            h = h * mask.unsqueeze(-1)     # padding contributes zero
        z = h.sum(dim=1)                   # (B, d_hid): order is forgotten here
        if self.pool == "mean":
            n = mask.sum(1, keepdim=True) if mask is not None else x.shape[1]
            z = z / n
        return self.rho(z)                 # (B, d_out)

torch.manual_seed(0)
model = DeepSets()
x = torch.rand(8, 6, 1)                    # 8 sets of 6 numbers
perm = torch.randperm(6)
with torch.no_grad():
    err = (model(x) - model(x[:, perm])).abs().max().item()
print(f"permutation test, max |f(X) - f(PX)|: {err:.1e}")
# permutation test, max |f(X) - f(PX)|: 6.0e-08   (float rounding, not a leak)

Parameter counts, and the library one-liners

import torch.nn as nn

def n_params(m):
    return sum(p.numel() for p in m.parameters())

# one layer, 224 x 224 RGB image in, 64 feature maps of the same size out
H = W = 224
dense = n_params(nn.Linear(3 * H * W, 64 * H * W, bias=False, device="meta"))  # meta: no memory
conv = n_params(nn.Conv2d(3, 64, kernel_size=3, padding=1, bias=False))
print(f"dense: {dense:,}   conv 3x3: {conv:,}   ratio: {dense / conv:,.0f}")

# 1-D: length 1,000 in, length 1,000 out, one channel
print("1-D dense:", n_params(nn.Linear(1000, 1000, bias=False)),
      " 1-D conv k=9:", n_params(nn.Conv1d(1, 1, 9, padding=4, bias=False)))

# a set layer (Zaheer et al. Lemma 3): Theta = lambda*I + gamma*11^T has 2 numbers
M = 100
print("set layer, M = 100: dense", M * M, " permutation-equivariant", 2)
# dense: 483,385,147,392   conv 3x3: 1,728   ratio: 279,736,775
# 1-D dense: 1000000  1-D conv k=9: 9
# set layer, M = 100: dense 10000  permutation-equivariant 2

The same blocks in one line each: nn.Conv1d(c_in, c_out, k) for shifts; nn.Conv1d(c, c, k, dilation=d) with left padding (k−1)·d for causal dilated stacks; torch_geometric.nn.aggr.DeepSetsAggregation(local_nn, global_nn) for sets; torch_geometric.nn.MessagePassing or GCNConv for graphs; nn.TransformerEncoderLayer without position encodings for a permutation-equivariant set block.

Augmentation: the soft version of a bias

You can also teach a symmetry with data. Data augmentation trains on changed copies: shifted signals, rotated images, shuffled sets, all with the old label. The model is pushed toward invariance, but only near the data it saw, and never exactly. Built-in symmetry is a hard constraint; augmentation is a soft one.

The data-efficiency curve

The hero's task, run at N = 8, 32, 128 and 512 for four models. Switch the test set between the training range and shifted wiggles. The right panel is the measured shift sensitivity: how much each model's output moves when the input shifts by 1 to 8 samples.

Trained here, 400 Adam steps each. Augmentation rolls each training signal by a random 0 to 31 samples (wrapping round), which sometimes cuts the wiggle in two. Shift sensitivity = mean |p(x) − p(shifted x)| on 200 in-range signals, shifting in zeros. Illustrative numbers.

The checklist

Your conv model's permutation or shift test gives an error of 0.08, not about 10−7. What is the most likely cause?

Chapter 10 Where else it worksWhen not to use it

Six fields, rotations, and the 6 that became a 9

List each field's real symmetries, meet rotation-equivariant networks, and see what happens when you assume a symmetry the data does not have

The method transfers: list the changes that do not change the answer, then pick blocks that respect exactly those. What changes from field to field is the list.

FieldReal symmetriesNot a symmetryBlocks
ImagesTranslation; small scale changes; often left-right flipsUp-down flips for scenes; rotation for text and digitsConv, pooling, pyramids; ViT at very large scale
Audio spectrogramsShift in timeShift in frequency, only roughly: it changes pitch, and low and high bands sound differentConv along time, frequency-aware layers or positions along frequency
Point clouds, moleculesPermutation of points; rotation; translationMirror images, for chiral moleculesDeep Sets, PointNet, EGNN
Sensor networksRenaming nodesMoving a sensor to a different placeMessage passing with real attributes as features
Multi-agent robotsRenaming the other agents; moving or rotating the whole sceneSwapping "me" with another agentGNN or attention over agents, relative coordinates, a "self" flag
Tabular dataOrder of rowsOrder or mixing of columns: age is not incomeTree ensembles; MLPs with per-column embeddings

Spectrograms are the subtle case. Shifting a sound in time keeps its meaning. Shifting it up the frequency axis changes its pitch, and on a mel scale the bands are not even equally spaced. A 2-D conv assumes both shifts are free. It still works well in practice, but the bias is half right. The AudioFM lesson treats the two axes differently on purpose: one view hides whole time columns, the other hides whole frequency bands.

Tabular data has no column symmetry. Grinsztajn et al. benchmarked 45 tabular datasets and found tree-based models still ahead of deep nets on medium-sized data (about 10K samples). One reason they give: good tabular models must "preserve the orientation of the data". Columns have fixed meanings, and a model that freely mixes them assumes a symmetry that is not there.

Rotation and physics

Molecules and particle systems keep their physics when you rotate, reflect or move the whole system. That is the Euclidean group E(n). Satorras, Hoogeboom and Welling's E(n)-equivariant GNN (EGNN) builds it in simply. Messages use only distances: mij = φe(hi, hj, ‖xi − xj‖2, aij). Positions move only along relative vectors: xi ← xi + C Σj(xi − xj) φx(mij) EGNN · §3, eq. 3–6. Distances do not change under rotation, and relative vectors rotate with the system. On a 3-D charged-particle task, EGNN's error was 0.0071, against 0.0107 for a plain GNN and 0.0104 for the next best, 32% lower EGNN · Table 2. It also beat the plain GNN from 100 to 50,000 training samples EGNN · Fig. 2.

When the symmetry is false: 6 and 9

A 9 is a 6 turned upside down. Make a digit classifier fully rotation invariant, and it can no longer tell them apart. The data's real symmetry is "small tilts", not "any rotation".

A 6 is a 9, upside down

Test digits are tilted by up to the angle you set. Three template classifiers vote: one with no rotation invariance, one invariant to any rotation, one invariant to tilts up to ±40°. The glyph shows a 6 at your tilt.

30°

Computed live on 160 synthetic tilted digits made of 34 points each, with a little jitter. Each classifier picks the template (upright 6 or 9) with the smaller average nearest-point distance, after trying the rotations it is invariant to. Ties are a coin flip. Illustrative numbers.

When not to use a block: the decision table

BlockUse it whenAvoid it whenUse instead
ConvolutionPatterns are local and can be anywhereAbsolute position matters (aligned faces, fixed sensor slots); long-range linksAdd position channels; dilation; attention
Full rotation invarianceMolecules, satellite tiles, microscopyUp and down mean something (6 and 9, text, scenes)Small-tilt augmentation
Deep Sets / sum poolOrder is an accidentOrder carries meaning (text, time); interactions between elements dominatePositions; attention between elements
Message passingRelations are given and names are arbitraryEdges are guesses; far nodes must talk (over-smoothing with many layers)Learned edges, graph Transformers
Causal blockPrediction or control runs liveThe whole recording is availableBidirectional or centred blocks
AttentionData is huge, or structure is unknownData is small and the symmetry is knownThe matching biased block
You build a model for a table of patient records. A colleague suggests a permutation-invariant set model over the columns "to be robust". What is wrong?

Chapter 11

Cheat sheet: from data to block

A three-step decision procedure, the whole idea on one screen, honest limits, and where to go next

The procedure has three steps. 1. List the changes to an input that leave the answer alone (or change it in a known way). 2. Pick blocks that respect exactly those, and no more. 3. Test: run the symmetry check, a held-out transformation split, and the wrong-symmetry check.

List, pick, test

Choose your data. The grid marks each candidate symmetry: yes, roughly, or no. The bottom row is the block to start with and the check to run first.

A starting point from this lesson, not a law. The data-efficiency curve decides.

The idea on one screen

PartIn one line
Inductive biasWhat a learner assumes about unseen inputs. Needed: 2(2n−m) rules fit any m labels.
SymmetryInvariant: f(g·x) = f(x). Equivariant: f(g·x) = g·f(x). Equivariant layers, then an invariant pool.
Shift → convWS = SW forces constant diagonals: yi = Σ ci−jxj. (B, C, L) → (B, C′, L). n2 weights become k.
Locality, scaleR = 1 + Σ(k−1)dl; dilation doubles reach cheaply; strided pools break exact equivariance.
Permutation → setsf(X) = ρ(Σφ(x)); (B, N, D) → (B, D′). Equivariant layer λI + γ11⊤.
Graphshi′ = φ(hi, Σj∈N(i)ψ(hi, hj)); (N, D) + edges → (N, D′).
TimeShift symmetry and causality are separate. SSM = causal conv with kernel c ak b.
AttentionPermutation equivariant, global, weak bias. Position encodings re-add order. ViT needed about 90M+ images to pass ResNets.
AugmentationThe soft version: approximate, costs data, good for messy symmetries.
Did it work?Symmetry error near 10−7; held-out transform split; data-efficiency curve; wrong-symmetry check.
Failure modesFalse symmetry (6/9, table columns, frequency); too little reach; leaks from padding, IDs, positions; peeking into the future.

Honest limits

Where to go next

This lesson is part of the Builder's Toolkit series on reusable blocks and how to choose them.

The takeaway. Every model assumes something. Make the assumption a symmetry your data really has, and the block learns from a fraction of the data: shifts give convolution, reorderings give Deep Sets and message passing, time gives causal and recurrent blocks, rotations give equivariant networks. Attention assumes little and pays in data. Then test it: measure the symmetry, hold out the transformation, and check that a false symmetry hurts.
You must predict the energy of small molecules from atom types and 3-D positions, with 5,000 examples. Which design and which first check?

Now press Present or Teach and explain, out loud and from memory, why a shift symmetry forces a convolution and why attention needs more data. Then go back to the hero and find the smallest N at which attention comes close to the conv in the training range.

Inductive Bias and Symmetry. Sources: A. Dosovitskiy et al., "An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale" (ViT, ICLR 2021, arXiv); M. Zaheer, S. Kottur, S. Ravanbakhsh, B. Póczos, R. Salakhutdinov, A. Smola, "Deep Sets" (NeurIPS 2017, arXiv); V. G. Satorras, E. Hoogeboom, M. Welling, "E(n) Equivariant Graph Neural Networks" (ICML 2021, arXiv); A. van den Oord et al., "WaveNet: A Generative Model for Raw Audio" (2016, arXiv); D. H. Wolpert, "The Lack of A Priori Distinctions Between Learning Algorithms" (Neural Computation, 1996); E. Wagstaff et al., "On the Limitations of Representing Functions on Sets" (ICML 2019, arXiv); J. Gilmer et al., "Neural Message Passing for Quantum Chemistry" (ICML 2017, arXiv); A. Gu, T. Dao, "Mamba: Linear-Time Sequence Modeling with Selective State Spaces" (2023, arXiv); J. Lee et al., "Set Transformer" (ICML 2019, arXiv); C. R. Qi et al., "PointNet" (CVPR 2017, arXiv); L. Grinsztajn, E. Oyallon, G. Varoquaux, "Why do tree-based models still outperform deep learning on tabular data?" (NeurIPS 2022 Datasets and Benchmarks, arXiv).
Back to Gleams