A smartwatch sees motion and heart rate. A robot sees a camera frame and reads an instruction. Two inputs, one answer. How you join them decides what the model can learn. Adding two votes can never learn "it depends".
Learn the five common ways to combine two signals: late votes, concatenation, gates, FiLM and cross-attention. See what each can and cannot learn, build each in PyTorch, and test whether your model really uses both inputs.
Pick a rule for the label, then a fusion block. Five small models train in your browser on 2,000 examples. Then switch on Drop B to see what each block does when one input goes missing.
You need vectors, a small neural network, and the idea of a trained classifier. We build the rest from zero.
Synthetic data, illustrative numbers. Signal A is three numbers: a value a and a pointer to one of four directions. Signal B is four tokens of three numbers: a value and a direction each. All five blocks train here with Adam (1,500 steps of 32) and are scored on 600 new examples. "Drop B" sets B to zeros at test time only.
Chapter 0
Meet a label that neither input can predict alone, and sort labelling rules into the kinds that decide which fusion block you need
You are building an alarm for a smartwatch. It should fire when the wearer's heart rate is odd for what they are doing. The watch has two sensors: a motion sensor and a heart-rate sensor.
Think through the four cases. A high heart rate while running is normal. A high heart rate while sitting still is odd: alarm. A low heart rate while sitting is normal. A low heart rate while running is odd too, often a strap that slipped: alarm.
Now cover one sensor. Heart rate alone says nothing: high is fine half the time. Motion alone says nothing either. Each sensor, alone, is a coin flip. Together they answer perfectly. The answer lives in how the two signals relate, not in either one.
Each kind of input is a modality: a camera, a microphone, a motion sensor, a row of table data. Combining modalities into one prediction is fusion. The part of the network that does the joining is a fusion block. This lesson is about choosing and building that block.
Call the two signals A and B, and give each a number: a for A and b for B. Every labelling rule you will meet is close to one of four kinds.
Worked: a = 1.2, b = −0.4. Sum rule: 1.2 − 0.4 = 0.8 > 0 → yes. Product rule: 1.2 × (−0.4) = −0.48 < 0 → no. Same inputs, different kinds of rule, opposite answers.
Each dot is one example, placed by its value of a (across) and b (up). Green means label yes, grey means no. Pick a rule. Then look with only A or only B: the dots slide onto one axis, and the readout gives the best accuracy any threshold can reach.
400 synthetic examples, a and b drawn from a standard normal curve. Accuracies are computed here with the best single threshold on the visible signal. For Look-up, B alone (without A's pointer) sees only the average of its four tokens.
The device shows the whole lesson in miniature. When one signal decides, either a one-input model or any fusion works. When evidence adds up, each signal alone gets about 75%, and adding their votes gets nearly everything. When the signals interact, each alone is at 50%, and only a block that can multiply or bend can win. When A selects part of B, a block must let A look inside B.
Chapter 1
Follow the tensor shapes for the smartwatch, and see what each join point gains and loses
Before the maths, fix the data. Take a 10-second window from the watch. The motion sensor (an IMU, inertial measurement unit) gives 6 numbers 50 times a second: 3 of acceleration and 3 of rotation. The heart-rate sensor gives 1 number per second.
So A is a B × 500 × 6 tensor and B is a B × 10 × 1 tensor. (The first B is the batch size; we keep "B" for the second signal elsewhere.) The two signals have different rates and different widths. There are three places to join them.
| Join | What happens | Shapes |
|---|---|---|
| Early fusion | Join the raw inputs, then one network reads both. | Repeat each heart-rate value 50 times: B × 500 × 1. Concatenate on the channel axis: B × 500 × 7. One encoder, one head. |
| Middle (feature) fusion | Each input gets its own encoder. Join the two embeddings, then a shared head reads them. | IMU encoder → B × 128. Heart-rate encoder → B × 16. Join → B × 144. Head → B × 1. |
| Late fusion | Each input makes its own prediction. Combine the predictions. | Each branch → B × 1 logit. Weighted sum → B × 1. |
Worked (middle): 128 IMU features + 16 heart-rate features → a 144-number joint vector. A head of 64 hidden units on it has 144 × 64 + 64 = 9,280 weights in its first layer.
The motion stream (blue, 50 ticks a second) and the heart-rate stream (warm, 1 tick a second) flow right into one answer. Move the join. Then lose the heart-rate sensor, or let its clock drift.
A diagram computed from your settings. The offset counts how many motion samples get paired with the wrong heart-rate reading when the raw streams are joined.
Read the trade. Early fusion lets the network relate any raw motion sample to any heart-rate sample, but it needs both streams on one clock and one grid, and it cannot reuse a pretrained encoder for each. Late fusion survives a missing sensor and lets each branch be trained on its own, but its branches never see each other. Middle fusion sits between.
Transformers blur the line, because any token can attend to any other at any layer. Nagrani et al. define the fusion layer Lf as the layer where cross-modal attention starts: Lf = 0 is early fusion and Lf = L is late fusion MBT · §3.3. On AudioSet, middle fusion beat both ends; the best fusion layer was 8 of 12 for their bottleneck model MBT · §4.3.1, Fig. 3.
Chapter 2 Why it works
Derive why adding log-odds is the right way to combine independent evidence, then prove that adding can never learn an interaction
Late fusion is the simplest block: each modality predicts, and you combine the predictions. But combine how? Averaging probabilities is common. Adding log-odds is better founded. Here is why.
The log-odds of a probability p is logit(p) = ln(p / (1 − p)). It turns 0.5 into 0, 0.8 into +1.39, and 0.2 into −1.39. Now assume A and B are conditionally independent given the label: once you know y, learning A tells you nothing new about B. Bayes' rule then gives p(y | a, b) ∝ p(a | y) p(b | y) p(y). Take log-odds of both sides and rearrange:
Worked: A says 0.8 (log-odds 1.386), B says 0.7 (0.847), balanced classes (prior log-odds 0). Fused log-odds 2.234 → p = 0.903. Two mild votes agree, so the fused belief is stronger than either. Averaging probabilities would give only 0.75.
Real branches are not exactly independent and not perfectly calibrated. So late fusion learns trust weights: logit = wAℓA + wBℓB + c. Fitting wA, wB and c on held-out data is stacking. It costs three numbers.
Any late fusion computes f(a) + g(b): a function of A plus a function of B. Try XOR with it. Let f+ be A's vote when a > 0 and f− its vote when a < 0; same for g+, g−. XOR needs:
f+ + g+ > 0 and f− + g− > 0 (same signs: yes) f+ + g− < 0 and f− + g+ < 0 (opposite: no)
Add the first two: f+ + f− + g+ + g− > 0. Add the last two: the same sum < 0. Both cannot hold. So at least one of the four quadrants is always wrong: late fusion is capped at 75% on balanced XOR. Try to beat it.
Each square is a quadrant of the (a, b) plane. Its colour is the true label; the number is your fused vote f + g. A tick means the sign is right. Set the four votes. Then press Search to try every vote from −2 to 2.
Computed here. The search tries all 414 ≈ 2.8 million settings in steps of 0.1.
In the hero, late fusion trained on A × B lands near 50%, not 75%. The 75% rule has to give up one quadrant on purpose. Gradient descent does not find it, for a reason worth seeing.
Chapter 3 Why it works
Join two embeddings three ways, prove that concat plus a linear layer is still just voting, and train all three on XOR
Middle fusion works on embeddings. Encoder A gives a vector hA of dA numbers; encoder B gives hB of dB numbers. The three simplest joins are:
Worked: hA = (1, 2), hB = (3, −1). Concat = (1, 2, 3, −1). Sum = (4, 1). Product = (3, −2).
Put a linear layer w on the concat. Split w into the part that touches A and the part that touches B:
w · [hA; hB] = wA · hA + wB · hB
That is a function of A plus a function of B: Chapter 2's votes again, with the same 75% ceiling. A sum join followed by a linear layer is the same, with wA = wB. The concat itself mixes nothing. Interaction appears only when something non-linear comes after the join: a hidden layer with tanh or ReLU, which can bend the sum. Or when the join itself multiplies.
The product does it directly. If hA holds a and hB holds b, one entry of the product is exactly a·b, the XOR score. Its full form, every entry of A times every entry of B, is bilinear pooling: the outer product hAhB⊤, with dA × dB numbers.
Each map shades where a trained block says yes (green). Pick a rule and press Train: all three blocks learn from 2,000 examples. On A + B all three succeed. On A × B, watch which one cannot bend.
Trained here with Adam on scalar signals (a and b plus small noise), 1,500 steps of 32, scored on 600 new examples. Concat + MLP has 8 tanh units; the product block projects each signal to 4 numbers and multiplies. Illustrative numbers.
Chapter 4
Let a sigmoid decide, for every feature and every example, how much to take from each modality, and watch it learn to ignore a broken sensor
Sensors fail. A camera faces the sun; a microphone sits next to a fan. A fixed vote weight wB trusts B the same amount every time. A gate sets the trust for each example, from the inputs themselves.
Arevalo et al. call this the gated multimodal unit (GMU). Each modality gets a small expert, and a sigmoid gate, which sees both inputs, mixes them feature by feature GMU · §3.1:
Worked: hA = (0.8, −0.2), hB = (−0.5, 0.6), gate inputs (2, −1) → g = (0.881, 0.269). z1 = 0.881×0.8 + 0.119×(−0.5) = 0.645 (mostly A). z2 = 0.269×(−0.2) + 0.731×0.6 = 0.385 (mostly B).
This is the mixture of experts idea at small scale: a gate picks which expert speaks. In a mixture-of-experts layer the experts are sub-networks and the gate picks per token. Here the experts are the two modalities and the gate picks per feature.
The gate also multiplies, so it can express interaction. Make one gate unit g1 = σ(kb) and a second g2 = σ(−kb), both with A-expert tanh(ka) and a B-expert of 0. Subtract the two units: tanh(ka)(σ(kb) − σ(−kb)) = tanh(ka)·tanh(kb/2). Its sign is the sign of a·b. That is XOR, which is why the gated block passes A × B in the hero.
Label: a + b > 0. In 30% of examples sensor A is broken (very noisy), in 30% sensor B is broken. Each sensor also sends a "broken" flag. Press Train. The grid shows each of the 4 gate units' average value: high (yellow) means "take A".
Trained here: 3,000 examples, 2,000 Adam steps, 4 gate units, scored on 1,500 new examples. A working sensor has noise 0.2; a broken one has noise 3. Illustrative numbers.
Read the grid. With both sensors working, the gate sits near the middle. When A breaks, the gate units drop toward B; when B breaks, they rise toward A. Concat plus a linear layer cannot do this: its trust in each sensor is one fixed number for all examples, so it pays for every broken reading.
Chapter 5 Why it works
Derive feature-wise linear modulation, see why a scale and a shift per channel can express interaction, and why robots start it as the identity
Think of a mixing desk. The image network is the music: many channels, each a feature such as "red", "round" or "left". The question "how many red cubes?" does not add a new channel. It turns some faders up, mutes others, and flips a few. That is FiLM, feature-wise linear modulation.
Perez et al. define it as an affine map on each feature, with the scale and shift predicted from the conditioning input FiLM · §2.1, eq. 1-2:
Worked: F = (2, −1, 0.5), γ = (1.5, 0, −1), β = (0, 0.3, 1) → γF + β = (3.0, 0.3, 0.5). Channel 1 amplified, channel 2 erased and replaced by a constant, channel 3 flipped and lifted.
Let B set the scale: γ = k·b, β = 0, on one A channel holding a. Use two units with opposite signs: h1 = ReLU(kba) and h2 = ReLU(−kba). Then h1 − h2 = k·a·b. The product appears because γ multiplies. Concat feeds b in beside a; FiLM feeds b in as the slope of a.
Left: four FiLM units' responses (pink) and the final logit (purple) as A's value moves across. Right: each unit's γ and β, computed from b. Slide b before training: nothing moves, because FiLM starts as the identity. Press Train, then slide b through zero.
Trained here: h = leaky-ReLU(γ(b)·(wia + ci) + β(b)), γ = 1 + linear(b), 4 units, 1,500 Adam steps on 2,000 examples. Illustrative numbers.
On CLEVR, a benchmark of questions about rendered 3-D scenes, a GRU read the question and generated γ and β for four residual blocks of a CNN. The model reached 97.7% accuracy, against 96.9% for the best prior method (which used extra program labels) and 92.6% for humans FiLM · Table 1.
The scale matters more than the shift. Training with γ fixed at 1 cost 1.5% accuracy; with β fixed at 0, 0.5%. At test time, replacing β with its training-set mean cost 1.0%, but doing the same to γ cost 65.4% FiLM · §4.3. FiLM does most of its work by multiplying.
RT-1, a robot policy from Google, uses FiLM to fuse the instruction into the camera encoder. A Universal Sentence Encoder embeds the instruction; FiLM layers inside an ImageNet-pretrained EfficientNet-B3 apply it. The image-and-instruction tokenizer has 16M parameters and outputs 81 vision-language tokens RT-1 · §4.
Chapter 6
When B is a set of parts and A must pick the right one, follow the shapes of cross-attention and see why every pooled block is stuck at two thirds
Ask "what colour is the mug?" about a photo. The photo is many parts: 196 patch vectors from a vision encoder. The answer lives in a few of them. Pool all 196 first, and the mug is averaged away. The question has to reach inside the image and pick.
Cross-attention does that. A's tokens become queries; B's tokens become keys and values. Each query scores every key, the softmax turns scores into weights, and the query gets back a weighted average of the values.
Worked: one query, three keys, scores (2.0, 0.5, −1.0) → es = (7.389, 1.649, 0.368), sum 9.406 → weights (0.786, 0.175, 0.039). Values (1, 4, 10) → output 0.786 + 0.701 + 0.391 = 1.878, mostly the first token.
Four B tokens (warm) sit at four key directions, each holding a value. A's query (blue arrow) points somewhere. Turn it and sharpen it: the weights and the answer move. Then press Train to pit concat against cross-attention on Look-up.
The top panel is computed: score = length × cos(angle between query and key). Training uses the hero's Look-up data: 2,000 examples, 1,500 Adam steps, scored on 600 new ones. Illustrative numbers.
Every block except cross-attention sees B through its average. In Look-up, the label is the sign of one token's value ck. The average of four independent values has correlation 1/√4 = 0.5 with any one of them. For two normal variables with correlation ρ, their signs agree with probability 1 − arccos(ρ)/π. With ρ = 0.5 that is 1 − (π/3)/π = 2/3. The hero's pooled blocks land at 67%: exactly this.
The cost: the scores form an Nq × Nk table per head. Sixteen question tokens over 196 patches is 3,136 scores, cheap. Two long streams attending to each other at every layer is not; Chapter 7 shows a way out. Padding or missing B tokens are masked: their scores are set to minus infinity, so their weights are exactly 0.
Chapter 7
Let a video transformer and an audio transformer talk only through four shared tokens, and count what that saves
Put all video and audio tokens into one transformer, and every token attends to every other at every layer. For 8 frames of 14 × 14 patches plus 8 seconds of spectrogram, that is 1,568 + 400 tokens MBT · §4.2. The attention table per layer has about 3.9 million entries, and most video patches have nothing to say to most audio patches.
Nagrani et al. propose a tighter channel. Add a few fusion bottleneck tokens. Each modality's transformer layer runs on its own tokens plus the bottleneck tokens. Then the two updated copies of the bottleneck are averaged MBT · §3.2.3, eq. 8-9:
Worked (attention-score entries per layer): all tokens together (1,568 + 400 + 2 class tokens)2 = 3,880,900. Bottleneck with B = 4: 1,5732 + 4052 = 2,638,354. Separate (late): 1,5692 + 4012 = 2,622,562. The bottleneck adds 0.6% to late fusion; full joint attention adds 48%.
Each column is one of 12 layers. Its height is the number of attention-score entries: blue within video, warm within audio, purple across. Layers before the fusion layer are unimodal. Switch between full joint attention and bottleneck tokens.
Computed here: attention-score entries only, one class token per modality, ViT-Base depth of 12. Real cost also includes the MLP blocks; the paper reports GFLOPs in Fig. 3.
On AudioSet (mean average precision, mAP), trained on 500K clips: audio only 41.5, video only 31.3, the bottleneck model 49.6 MBT · Table 1. On VGGSound (top-1 accuracy): audio 52.3%, video 51.2%, fused 64.1% MBT · Table 3. Fusion beat the better single modality by 8 and 12 points.
The bottleneck size barely mattered: B from 4 to 1,024 stayed within 0.5 mAP, so they kept B = 4. At fusion layer 0, bottlenecks beat full cross-attention by over 2 mAP at less than half the compute MBT · §4.3.1. Four tokens were enough because they carry only what the other modality needs.
Chapter 8 How
Four fusion modules with every shape commented, a training loop with modality dropout, the library one-liners, and the parameter bill
You have done each block by hand: a vote sum (Chapter 2), a concat (3), a gate (4), a FiLM (5) and an attention (6). Now as code. All blocks below take cached embeddings from frozen encoders, so training touches only the block and a small head.
import torch import torch.nn as nn class ConcatMLP(nn.Module): """Feature fusion: concatenate two embeddings, then an MLP.""" def __init__(self, d_a, d_b, d_h=256, n_out=1, p_drop=0.1): super().__init__() self.mlp = nn.Sequential( nn.Linear(d_a + d_b, d_h), nn.GELU(), nn.Dropout(p_drop), nn.Linear(d_h, n_out)) def forward(self, a, b): # a: (B, d_a), b: (B, d_b) x = torch.cat([a, b], dim=-1) # (B, d_a + d_b) return self.mlp(x) # (B, n_out): the non-linearity mixes them class GatedFusion(nn.Module): """Gated multimodal unit: a per-dimension sigmoid gate picks A or B.""" def __init__(self, d_a, d_b, d_h=256, n_out=1): super().__init__() self.pa = nn.Linear(d_a, d_h) # A's expert self.pb = nn.Linear(d_b, d_h) # B's expert self.gate = nn.Linear(d_a + d_b, d_h) # sees both inputs self.out = nn.Linear(d_h, n_out) def forward(self, a, b): ha, hb = torch.tanh(self.pa(a)), torch.tanh(self.pb(b)) # (B, d_h) each g = torch.sigmoid(self.gate(torch.cat([a, b], -1))) # (B, d_h), in (0, 1) z = g * ha + (1 - g) * hb # mix per dimension return self.out(z), g # logits, and the gate to inspect a, b = torch.randn(8, 512), torch.randn(8, 64) # e.g. image embedding + metadata print(ConcatMLP(512, 64)(a, b).shape) # torch.Size([8, 1]) logits, g = GatedFusion(512, 64)(a, b) print(logits.shape, g.shape) # torch.Size([8, 1]) torch.Size([8, 256])
import torch import torch.nn as nn class FiLM(nn.Module): """B produces a scale and a shift for every channel of A's features.""" def __init__(self, d_cond, n_channels): super().__init__() self.to_gb = nn.Linear(d_cond, 2 * n_channels) nn.init.zeros_(self.to_gb.weight) # identity at start: gamma = 1, beta = 0 nn.init.zeros_(self.to_gb.bias) def forward(self, h, cond): # h: (B, C, H, W) or (B, C); cond: (B, d_cond) dg, beta = self.to_gb(cond).chunk(2, dim=-1) # (B, C) each gamma = 1 + dg while gamma.dim() < h.dim(): # broadcast over H, W gamma, beta = gamma.unsqueeze(-1), beta.unsqueeze(-1) return gamma * h + beta # same shape as h class FiLMBlock(nn.Module): """conv -> norm -> FiLM -> ReLU, as in a FiLM-ed residual block.""" def __init__(self, c, d_cond): super().__init__() self.conv = nn.Conv2d(c, c, 3, padding=1) self.norm = nn.BatchNorm2d(c, affine=False) # FiLM supplies the affine part self.film = FiLM(d_cond, c) def forward(self, x, cond): h = torch.relu(self.film(self.norm(self.conv(x)), cond)) return x + h # residual img = torch.randn(4, 128, 14, 14) # image feature map text = torch.randn(4, 512) # instruction embedding print(FiLMBlock(128, 512)(img, text).shape) # torch.Size([4, 128, 14, 14])
The FiLM-ed block follows Perez et al.: batch norm without its own affine parameters just before FiLM, then ReLU, inside a residual block FiLM · §2.2. They used 128 feature maps on 14 × 14 image features, the shapes in the example.
import torch import torch.nn as nn class CrossAttnFusion(nn.Module): """A's tokens ask questions; B's tokens answer.""" def __init__(self, d, n_heads=8, p_drop=0.1): super().__init__() self.norm_q, self.norm_kv = nn.LayerNorm(d), nn.LayerNorm(d) self.attn = nn.MultiheadAttention(d, n_heads, dropout=p_drop, batch_first=True) self.ff = nn.Sequential(nn.LayerNorm(d), nn.Linear(d, 4 * d), nn.GELU(), nn.Linear(4 * d, d)) def forward(self, xa, xb, b_pad=None): # xa: (B, Nq, d) queries; xb: (B, Nk, d) keys and values # b_pad: (B, Nk) bool, True = padding or a missing B token out, w = self.attn(self.norm_q(xa), self.norm_kv(xb), self.norm_kv(xb), key_padding_mask=b_pad) # out: (B, Nq, d), w: (B, Nq, Nk) xa = xa + out # residual: A keeps its own content return xa + self.ff(xa), w xa = torch.randn(2, 16, 256) # 16 text tokens xb = torch.randn(2, 196, 256) # 196 image patches pad = torch.zeros(2, 196, dtype=torch.bool) y, w = CrossAttnFusion(256)(xa, xb, pad) print(y.shape, w.shape) # torch.Size([2, 16, 256]) torch.Size([2, 16, 196])
The other library one-liner is the bilinear join: nn.Bilinear(512, 64, 256) computes every product of an A entry and a B entry, mixed into 256 outputs. It has 8,388,864 weights. That is why people use low-rank products (project each side, then ⊙) instead.
import torch import torch.nn.functional as F # ConcatMLP is the class defined above torch.manual_seed(0) N, d_a, d_b = 6000, 32, 8 a, b = torch.randn(N, d_a), torch.randn(N, d_b) # stand-ins for cached embeddings y = ((a[:, 0] + b[:, 0]) > 0).float() # the answer needs both tr, te = slice(0, 5000), slice(5000, N) model = ConcatMLP(d_a, d_b, d_h=64) opt = torch.optim.AdamW(model.parameters(), lr=1e-3, weight_decay=1e-2) p_miss = 0.3 # modality dropout rate for B for step in range(3000): idx = torch.randint(0, 5000, (64,)) xa, xb = a[idx], b[idx].clone() xb[torch.rand(64) < p_miss] = 0.0 # zero-fill B, as at test time loss = F.binary_cross_entropy_with_logits(model(xa, xb).squeeze(-1), y[idx]) opt.zero_grad(); loss.backward(); opt.step() model.eval() with torch.no_grad(): acc = lambda xb: ((model(a[te], xb).squeeze(-1) > 0).float() == y[te]).float().mean().item() print(f'both {acc(b[te]):.3f} B missing {acc(torch.zeros_like(b[te])):.3f}')
We ran every block on this page (PyTorch 2.10). The last one prints both 0.960 B missing 0.748: with B gone, it falls to the 75% that A alone allows on a sum rule (Chapter 0), not below.
The training details that matter. Normalise each embedding (a LayerNorm per modality) before the join: a 512-number image vector and 3 numbers of patient age, sex and site have very different scales, and the bigger one wins by default. Keep encoders frozen and cache their outputs when data is small. Use weight decay and dropout on the head. Start FiLM as the identity. And always train the two single-modality baselines with the same budget.
Worked: dA = 512, dB = 64, dh = 256. Concat: 576 × 256 + 512 + 1 = 147,969. Gated: 2 × 576 × 256 + 1,024 + 1 = 295,937, twice the bill.
Choose a block and set the sizes. The diagram traces the tensor shapes from the two inputs to the logit, and the readout counts the weights.
Counts include biases and one output logit. FiLM here: A projected to dh channels, B generating γ and β for each. Cross-attention: A and B projected to dh, one multi-head attention (4dh2 + 4dh), with Nq = 16 and no feed-forward block.
Chapter 9
Train a model that survives a lost sensor, then meet the stranger failure: a fused model that scores below one of its own inputs
At test time a sensor can fail, a field can be empty, a clip can be silent. You must feed the model something. Zeros are common. But a model that never saw zeros during training reads them as a real, extreme value. If heart rate is centred near 70 after scaling, zero means "no pulse", and the alarm fires.
The fix is modality dropout: during training, delete a whole modality at random, with probability p, and fill it the same way you will at test time. Add a mask flag, a 0/1 input that says "B is missing". Neverova et al. introduced this as ModDrop for gesture recognition, to make predictions robust to missing or corrupted channels ModDrop · §5.
Worked: batch of 64, p = 0.3 → about 19 rows lose B each step. Over 1,500 steps the model sees about 29,000 B-missing rows, enough to learn a fallback that uses A alone.
Label: a + b > 0. B arrives shifted (centred at 3), with a presence flag. Set the training drop rate and press Train. The curves show test accuracy as a growing share of test examples lose B (zeros, flag 0).
Trained here: concat + MLP (12 tanh units), 3,000 examples, 1,500 Adam steps, scored on 1,500 new examples. Illustrative numbers.
A fused model contains every single-modality model: set the weights on B to zero and you have A alone. So it should never score lower. Wang, Tran and Feiszli found the opposite on Kinetics: audio + RGB, trained jointly with late concatenation, reached 71.4% top-1 against 72.6% for RGB alone Wang 2020 · Table 1. Every pair and the triple they tried lost to the best single stream.
They name two causes. First, the joint model has more capacity, so it overfits more. Second, modalities overfit and generalise at different rates, so one learning rate and one stopping time cannot suit both Wang 2020 · abstract. Often the fast-fitting modality memorises the training set and the loss stops pushing the slow, useful one.
They measure it with the overfitting-to-generalisation ratio between two checkpoints, and weight each modality's loss by it Wang 2020 · eq. 3, Alg. 1:
Worked: A has G = 0.02, O = 0.10 → 0.02/0.01 = 2. B has G = 0.03, O = 0.02 → 0.03/0.0004 = 75. Normalise: wA = 2/77 = 0.026, wB = 75/77 = 0.974. The overfitting modality gets almost no weight.
On Kinetics audio + RGB this lifted top-1 from 71.7% (naive joint) and 72.6% (RGB only) to 74.7% offline and 75.8% online Wang 2020 · Table 2.
Only 120 training examples. A has 80 weak features (easy to memorise); B has one decent feature. Curves are validation accuracy over training. Lower A's gradient weight and train again: does the joint model beat B alone?
Trained here: logistic regression with plain SGD (learning rate 0.05, batch 32, 3,000 steps), scored on 2,000 new examples. "Blended" multiplies the gradient of A's weights by wA, in the spirit of Gradient-Blending, not the paper's exact method. Illustrative numbers.
Chapter 10 Did it work?
Prove that your fused model uses both inputs: unimodal baselines, ablations, the shuffle test, and the bugs they catch
A good test score does not prove fusion. The model may be ignoring B, or using B for the wrong reason. Run these checks, in this order.
Worked (the PyTorch model of Chapter 8, sum rule): both 0.960, B shuffled 0.644, B zeroed 0.748, A shuffled 0.673 → RB = 0.316. Shuffling hurts more than zeroing: a wrong B is misleading evidence, a zero B is no evidence.
import torch @torch.no_grad() def reliance(model, a, b, y): """How much does the fused model lean on each input? (a: (N, d_a), b: (N, d_b))""" model.eval() acc = lambda xa, xb: ((model(xa, xb).squeeze(-1) > 0).float() == y).float().mean().item() perm = torch.randperm(len(y)) # pair each A with someone else's B return {'both': acc(a, b), 'shuffle B': acc(a, b[perm]), # breaks the A-B pairing 'shuffle A': acc(a[perm], b), 'zero B': acc(a, torch.zeros_like(b))} # the missing-B case
Pick a rule and a block, then press Train. Three models train: the fused block, A alone and B alone. The bars are test accuracy for each, and for the fused block with B zeroed, B shuffled and A shuffled. The dashed line is chance.
The hero's data and blocks, trained here: 2,000 examples, 1,500 Adam steps each, scored on 600 new ones. Illustrative numbers.
Chapter 11 What elseWhat not
Move the blocks to six fields, meet the cases where fusion hurts, and keep the whole lesson on one screen
| Field | A + B → answer | Usual block, and what changes |
|---|---|---|
| Audio-visual events | Video frames + soundtrack → "what is happening" | Middle fusion through bottleneck tokens. On VGGSound, fusion beat the better single modality by 12 points MBT · Table 3. Audio and video can drift in time; MBT found results largely robust to sampling the two windows separately MBT · §4.3.2. |
| Visual question answering | Question + image → answer | The question selects (Look-up kind): FiLM from the question into the CNN, as on CLEVR FiLM · Table 1, or question tokens cross-attending to patches. |
| Robot policies | Camera + instruction (+ joint angles) → action | RT-1 FiLM-conditions the image encoder on the instruction, starting from the identity RT-1 · §4. Low-dimensional robot state (proprioception) is commonly concatenated with the vision features. |
| Wearables | Motion (IMU) + heart rate → alarm or activity | Different rates, so middle or late fusion. Sensors drop out, so modality dropout and a mask flag. The alarm is an interaction, so not late fusion. |
| Medical imaging | Scan + table data (age, sex, scanner) → diagnosis | B is a few numbers, A is a big image. FiLM from the table into the CNN, or concat after normalising each. Beware leaks: a scanner ID that tracks the hospital, which tracks the disease. |
| Text + image classification | Product photo + title, poster + plot → category | Gated units: Arevalo et al. classified movie genres from poster and plot on MM-IMDb, 25,959 movies GMU · §4. Often one modality (the text) carries most of it, so check the unimodal baseline first. |
What stays the same in every row: two encoders, one join, the four kinds of rule, and the checks of Chapter 10. What changes: the shapes, which signal asks and which answers, and how often each input goes missing.
| Situation | Why it hurts | Use instead |
|---|---|---|
| One modality is enough (its baseline matches the fused model) | The second input adds noise, overfitting, cost, a second sensor to fail | The unimodal model. Revisit if a slice (night, noise) needs the other input. |
| Tiny data, wide embeddings, concat or bilinear | (dA + dB)dh or dAdB weights fit noise (Chapter 9's race) | Late fusion of two strong unimodal models, low-rank projections first, or FiLM (2 numbers per channel) |
| Streams misaligned in time | Early fusion pairs the wrong samples (Chapter 1) | Align or resample first; else late fusion, or cross-attention over a time window |
| An input is often missing at test | Zeros are out of range for a model that never saw them | Late fusion, or any block with modality dropout and a mask flag |
| The rule is an interaction | Late fusion and concat + linear are capped (75%) and train to about 50% | Concat + MLP, a product, a gate or FiLM |
| One input selects part of the other | Pooling B first averages the answer away (67% in the toy) | Cross-attention, A querying B's tokens |
Choose an answer in each row. The pipeline below updates.
A rule of thumb from this lesson, not a law. Always compare with both unimodal baselines on held-out data.
| Block | Shape signature | Maths in one line | Learns |
|---|---|---|---|
| Late | (B, 1) + (B, 1) → (B, 1) | wAℓA + wBℓB + c | Adding evidence; survives a missing input. No interaction (≤ 75% on XOR). |
| Concat + MLP | (B, dA) + (B, dB) → (B, dA+dB) → (B, C) | W2φ(W1[hA; hB]) | Interaction, through the hidden layer only. The default. |
| Product | (B, d) ⊙ (B, d) → (B, d) | hA ⊙ hB | Interaction directly. Full bilinear is dAdB: use low rank. |
| Gated | two (B, d) experts + (B, d) gate → (B, d) | g⊙hA + (1−g)⊙hB, g = σ(W[xA; xB]) | Per-example trust; ignores a broken sensor. |
| FiLM | (B, C, H, W) + (B, dB) → (B, C, H, W) | γ(xB)⊙F + β(xB), γ = 1 + Δγ | B steers A's channels. Cheap; directional; zero-init. |
| Cross-attention | (B, Nq, D) × (B, Nk, D) → (B, Nq, D) | softmax(QK⊤/√d)V | Selection inside a set; mask padding. Cost NqNk. |
| Bottleneck tokens | (B, Nv, D) ‖ (B, 4, D) ‖ (B, Na, D) | per-modality layers share B tokens, averaged | Two transformers trade summaries at near-late cost. |
| Part | In one line |
|---|---|
| Knobs | Join point (early, middle, late), block type, width dh, per-modality normalisation, modality dropout rate p, per-modality loss weights. |
| Failure modes | Votes on an interaction; pooled B on a selection; zeros never seen in training; one modality overfitting; misaligned clocks; broken pairing; scale domination; leaky metadata. |
| Did it work? | Beat both unimodal baselines; ablate (zero, mean) and shuffle each input; robustness curve; slices. |
| Alternatives | A single modality; stacking of two unimodal models; one shared transformer over both token sets (early fusion). |
This lesson is part of the Builder's Toolkit series on reusable blocks for frozen backbones. Go on to:
Now press Present or Teach and explain, out loud and from memory, why late fusion cannot learn XOR and what FiLM's γ does. Then go back to the hero and predict, before training, which blocks will pass Look-up.