Fusion Blocks

A smartwatch sees motion and heart rate. A robot sees a camera frame and reads an instruction. Two inputs, one answer. How you join them decides what the model can learn. Adding two votes can never learn "it depends".

Learn the five common ways to combine two signals: late votes, concatenation, gates, FiLM and cross-attention. See what each can and cannot learn, build each in PyTorch, and test whether your model really uses both inputs.

Pick a rule for the label, then a fusion block. Five small models train in your browser on 2,000 examples. Then switch on Drop B to see what each block does when one input goes missing.

You need vectors, a small neural network, and the idea of a trained classifier. We build the rest from zero.

Which join learns which rule?

Ready

Synthetic data, illustrative numbers. Signal A is three numbers: a value a and a pointer to one of four directions. Signal B is four tokens of three numbers: a value and a direction each. All five blocks train here with Adam (1,500 steps of 32) and are scored on 600 new examples. "Drop B" sets B to zeros at test time only.

Chapter 0

Two signals, one answer

Meet a label that neither input can predict alone, and sort labelling rules into the kinds that decide which fusion block you need

You are building an alarm for a smartwatch. It should fire when the wearer's heart rate is odd for what they are doing. The watch has two sensors: a motion sensor and a heart-rate sensor.

Think through the four cases. A high heart rate while running is normal. A high heart rate while sitting still is odd: alarm. A low heart rate while sitting is normal. A low heart rate while running is odd too, often a strap that slipped: alarm.

Now cover one sensor. Heart rate alone says nothing: high is fine half the time. Motion alone says nothing either. Each sensor, alone, is a coin flip. Together they answer perfectly. The answer lives in how the two signals relate, not in either one.

Each kind of input is a modality: a camera, a microphone, a motion sensor, a row of table data. Combining modalities into one prediction is fusion. The part of the network that does the joining is a fusion block. This lesson is about choosing and building that block.

Four kinds of rule

Call the two signals A and B, and give each a number: a for A and b for B. Every labelling rule you will meet is close to one of four kinds.

yA
One signal decides. B is noise for this label. 1[…] means 1 if the condition holds, else 0.
y+
Evidence adds up. Each signal pushes the answer its own way. A strong A can outvote a weak B.
y×
The signals interact. What A means depends on B. This is the smartwatch alarm, and it is the logic gate called XOR.
ylook
One signal selects. B is a set of parts (tokens) c1…c4. A says which part matters, k(A). A question about an image works like this.

Worked: a = 1.2, b = −0.4. Sum rule: 1.2 − 0.4 = 0.8 > 0 → yes. Product rule: 1.2 × (−0.4) = −0.48 < 0 → no. Same inputs, different kinds of rule, opposite answers.

Look with one eye

Each dot is one example, placed by its value of a (across) and b (up). Green means label yes, grey means no. Pick a rule. Then look with only A or only B: the dots slide onto one axis, and the readout gives the best accuracy any threshold can reach.

400 synthetic examples, a and b drawn from a standard normal curve. Accuracies are computed here with the best single threshold on the visible signal. For Look-up, B alone (without A's pointer) sees only the average of its four tokens.

The device shows the whole lesson in miniature. When one signal decides, either a one-input model or any fusion works. When evidence adds up, each signal alone gets about 75%, and adding their votes gets nearly everything. When the signals interact, each alone is at 50%, and only a block that can multiply or bend can win. When A selects part of B, a block must let A look inside B.

The lesson in one line. A fusion block decides which functions of two inputs your model can express: adding votes captures evidence that adds up, a non-linearity or a product after the join captures interaction, and attention captures selection.
  1. Where to join: early, middle and late fusion, with the shapes at every step.
  2. Late fusion: adding votes: why adding log-odds works, and the 75% ceiling it hits on interaction.
  3. Concatenate, sum, multiply: why concat plus a linear layer is still just voting.
  4. Gating: a learned trust dial for each feature, mixture-of-experts style.
  5. FiLM: one signal sets a scale and a shift for every channel of the other.
  6. Cross-attention: one signal asks, the other answers.
  7. Bottleneck tokens: let two transformers talk through four tokens.
  8. Build it in PyTorch: four modules, the shapes, and the parameter bill.
  9. Missing inputs, and one input winning: modality dropout and gradient blending.
  10. Did it work?: unimodal baselines, ablations and the shuffle test.
  11. Where else, when not, cheat sheet: six fields, the failure cases, and one screen to keep.
The smartwatch alarm fires when heart rate and motion disagree. Which statement is true?

Chapter 1

Where to join: early, middle, late

Follow the tensor shapes for the smartwatch, and see what each join point gains and loses

Before the maths, fix the data. Take a 10-second window from the watch. The motion sensor (an IMU, inertial measurement unit) gives 6 numbers 50 times a second: 3 of acceleration and 3 of rotation. The heart-rate sensor gives 1 number per second.

So A is a B × 500 × 6 tensor and B is a B × 10 × 1 tensor. (The first B is the batch size; we keep "B" for the second signal elsewhere.) The two signals have different rates and different widths. There are three places to join them.

JoinWhat happensShapes
Early fusionJoin the raw inputs, then one network reads both.Repeat each heart-rate value 50 times: B × 500 × 1. Concatenate on the channel axis: B × 500 × 7. One encoder, one head.
Middle (feature) fusionEach input gets its own encoder. Join the two embeddings, then a shared head reads them.IMU encoder → B × 128. Heart-rate encoder → B × 16. Join → B × 144. Head → B × 1.
Late fusionEach input makes its own prediction. Combine the predictions.Each branch → B × 1 logit. Weighted sum → B × 1.
[xA; xB]
Concatenation: stack the numbers of A and B into one longer list. f is one network.
eA, eB, J, g
Two encoders (often frozen, pretrained), a join J (concat, sum, product, gate, FiLM, attention), and a head g.
ℓA, ℓB, w
Each branch's own logit (its log-odds score). wA, wB are learned trust weights; σ turns the sum into a probability.

Worked (middle): 128 IMU features + 16 heart-rate features → a 144-number joint vector. A head of 64 hidden units on it has 144 × 64 + 64 = 9,280 weights in its first layer.

Move the join

The motion stream (blue, 50 ticks a second) and the heart-rate stream (warm, 1 tick a second) flow right into one answer. Move the join. Then lose the heart-rate sensor, or let its clock drift.

0.0 s

A diagram computed from your settings. The offset counts how many motion samples get paired with the wrong heart-rate reading when the raw streams are joined.

Read the trade. Early fusion lets the network relate any raw motion sample to any heart-rate sample, but it needs both streams on one clock and one grid, and it cannot reuse a pretrained encoder for each. Late fusion survives a missing sensor and lets each branch be trained on its own, but its branches never see each other. Middle fusion sits between.

Transformers blur the line, because any token can attend to any other at any layer. Nagrani et al. define the fusion layer Lf as the layer where cross-modal attention starts: Lf = 0 is early fusion and Lf = L is late fusion MBT · §3.3. On AudioSet, middle fusion beat both ends; the best fusion layer was 8 of 12 for their bottleneck model MBT · §4.3.1, Fig. 3.

The heart-rate sensor drops out for 20% of users. Which join point degrades most gracefully without extra training tricks?

Chapter 2 Why it works

Late fusion: adding votes

Derive why adding log-odds is the right way to combine independent evidence, then prove that adding can never learn an interaction

Late fusion is the simplest block: each modality predicts, and you combine the predictions. But combine how? Averaging probabilities is common. Adding log-odds is better founded. Here is why.

The log-odds of a probability p is logit(p) = ln(p / (1 − p)). It turns 0.5 into 0, 0.8 into +1.39, and 0.2 into −1.39. Now assume A and B are conditionally independent given the label: once you know y, learning A tells you nothing new about B. Bayes' rule then gives p(y | a, b) ∝ p(a | y) p(b | y) p(y). Take log-odds of both sides and rearrange:

logit p(y|a)
What A alone says, as log-odds: the output of A's own classifier.
logit p(y|b)
What B alone says.
logit p(y)
The prior: how common the label is before seeing anything. Both branches include it, so subtract it once.
logit p(y|a,b)
The fused belief. Under independence, evidence adds in log-odds.

Worked: A says 0.8 (log-odds 1.386), B says 0.7 (0.847), balanced classes (prior log-odds 0). Fused log-odds 2.234 → p = 0.903. Two mild votes agree, so the fused belief is stronger than either. Averaging probabilities would give only 0.75.

Real branches are not exactly independent and not perfectly calibrated. So late fusion learns trust weights: logit = wAℓA + wBℓB + c. Fitting wA, wB and c on held-out data is stacking. It costs three numbers.

The ceiling: votes cannot interact

Any late fusion computes f(a) + g(b): a function of A plus a function of B. Try XOR with it. Let f+ be A's vote when a > 0 and f− its vote when a < 0; same for g+, g−. XOR needs:

f+ + g+ > 0  and  f− + g− > 0  (same signs: yes)    f+ + g− < 0  and  f− + g+ < 0  (opposite: no)

Add the first two: f+ + f− + g+ + g− > 0. Add the last two: the same sum < 0. Both cannot hold. So at least one of the four quadrants is always wrong: late fusion is capped at 75% on balanced XOR. Try to beat it.

Beat the ceiling

Each square is a quadrant of the (a, b) plane. Its colour is the true label; the number is your fused vote f + g. A tick means the sign is right. Set the four votes. Then press Search to try every vote from −2 to 2.

1.0
−1.0
1.0
−1.0

Computed here. The search tries all 414 ≈ 2.8 million settings in steps of 0.1.

In the hero, late fusion trained on A × B lands near 50%, not 75%. The 75% rule has to give up one quadrant on purpose. Gradient descent does not find it, for a reason worth seeing.

Branch A outputs p = 0.9 and branch B outputs p = 0.6, with balanced classes and independent evidence. What is the fused probability by adding log-odds?

Chapter 3 Why it works

Concatenate, sum, multiply

Join two embeddings three ways, prove that concat plus a linear layer is still just voting, and train all three on XOR

Middle fusion works on embeddings. Encoder A gives a vector hA of dA numbers; encoder B gives hB of dB numbers. The three simplest joins are:

[hA; hB]
Concatenation: dA + dB numbers. Works for any two sizes.
hA + hB
Sum: needs equal sizes, usually after a linear projection of each. Keeps d numbers.
⊙
Element-by-element product (the Hadamard product): multiply matching entries. Also needs equal sizes.

Worked: hA = (1, 2), hB = (3, −1). Concat = (1, 2, 3, −1). Sum = (4, 1). Product = (3, −2).

Concat plus linear is late fusion in disguise

Put a linear layer w on the concat. Split w into the part that touches A and the part that touches B:

w · [hA; hB] = wA · hA + wB · hB

That is a function of A plus a function of B: Chapter 2's votes again, with the same 75% ceiling. A sum join followed by a linear layer is the same, with wA = wB. The concat itself mixes nothing. Interaction appears only when something non-linear comes after the join: a hidden layer with tanh or ReLU, which can bend the sum. Or when the join itself multiplies.

The product does it directly. If hA holds a and hB holds b, one entry of the product is exactly a·b, the XOR score. Its full form, every entry of A times every entry of B, is bilinear pooling: the outer product hAhB⊤, with dA × dB numbers.

Three joins, one task

Each map shades where a trained block says yes (green). Pick a rule and press Train: all three blocks learn from 2,000 examples. On A + B all three succeed. On A × B, watch which one cannot bend.

Trained here with Adam on scalar signals (a and b plus small noise), 1,500 steps of 32, scored on 600 new examples. Concat + MLP has 8 tanh units; the product block projects each signal to 4 numbers and multiplies. Illustrative numbers.

You concatenate an image embedding and a text embedding and put one linear layer on top. Can this model learn "the caption is true only if the image is a cat AND the text says dog-free"-style interactions?

Chapter 4

Gating: a trust dial per feature

Let a sigmoid decide, for every feature and every example, how much to take from each modality, and watch it learn to ignore a broken sensor

Sensors fail. A camera faces the sun; a microphone sits next to a fan. A fixed vote weight wB trusts B the same amount every time. A gate sets the trust for each example, from the inputs themselves.

Arevalo et al. call this the gated multimodal unit (GMU). Each modality gets a small expert, and a sigmoid gate, which sees both inputs, mixes them feature by feature GMU · §3.1:

hA, hB
The two experts' outputs, d numbers each, squashed into (−1, 1).
g
The gate: d numbers between 0 and 1, one per feature. Near 1 means "take A here"; near 0, "take B here".
z
The fused vector, d numbers. A linear layer on z gives the logit.

Worked: hA = (0.8, −0.2), hB = (−0.5, 0.6), gate inputs (2, −1) → g = (0.881, 0.269). z1 = 0.881×0.8 + 0.119×(−0.5) = 0.645 (mostly A). z2 = 0.269×(−0.2) + 0.731×0.6 = 0.385 (mostly B).

This is the mixture of experts idea at small scale: a gate picks which expert speaks. In a mixture-of-experts layer the experts are sub-networks and the gate picks per token. Here the experts are the two modalities and the gate picks per feature.

The gate also multiplies, so it can express interaction. Make one gate unit g1 = σ(kb) and a second g2 = σ(−kb), both with A-expert tanh(ka) and a B-expert of 0. Subtract the two units: tanh(ka)(σ(kb) − σ(−kb)) = tanh(ka)·tanh(kb/2). Its sign is the sign of a·b. That is XOR, which is why the gated block passes A × B in the hero.

Trust the sensor that works

Label: a + b > 0. In 30% of examples sensor A is broken (very noisy), in 30% sensor B is broken. Each sensor also sends a "broken" flag. Press Train. The grid shows each of the 4 gate units' average value: high (yellow) means "take A".

Trained here: 3,000 examples, 2,000 Adam steps, 4 gate units, scored on 1,500 new examples. A working sensor has noise 0.2; a broken one has noise 3. Illustrative numbers.

Read the grid. With both sensors working, the gate sits near the middle. When A breaks, the gate units drop toward B; when B breaks, they rise toward A. Concat plus a linear layer cannot do this: its trust in each sensor is one fixed number for all examples, so it pays for every broken reading.

What can a gated fusion block do that concat plus a linear layer cannot?

Chapter 5 Why it works

FiLM: one signal turns the other's knobs

Derive feature-wise linear modulation, see why a scale and a shift per channel can express interaction, and why robots start it as the identity

Think of a mixing desk. The image network is the music: many channels, each a feature such as "red", "round" or "left". The question "how many red cubes?" does not add a new channel. It turns some faders up, mutes others, and flips a few. That is FiLM, feature-wise linear modulation.

Perez et al. define it as an affine map on each feature, with the scale and shift predicted from the conditioning input FiLM · §2.1, eq. 1-2:

xi
The conditioning input for example i: signal B, such as the question's embedding.
Fi,c
Channel c of signal A's features: a number, or a whole H × W feature map. The same γ, β apply at every position.
γi,c
The scale. Large: amplify. 0: switch off. Negative: flip.
βi,c
The shift. Before a ReLU it acts as a threshold. f and h are one small network, the FiLM generator.

Worked: F = (2, −1, 0.5), γ = (1.5, 0, −1), β = (0, 0.3, 1) → γF + β = (3.0, 0.3, 0.5). Channel 1 amplified, channel 2 erased and replaced by a constant, channel 3 flipped and lifted.

Why it can express interaction

Let B set the scale: γ = k·b, β = 0, on one A channel holding a. Use two units with opposite signs: h1 = ReLU(kba) and h2 = ReLU(−kba). Then h1 − h2 = k·a·b. The product appears because γ multiplies. Concat feeds b in beside a; FiLM feeds b in as the slope of a.

B sets A's slope

Left: four FiLM units' responses (pink) and the final logit (purple) as A's value moves across. Right: each unit's γ and β, computed from b. Slide b before training: nothing moves, because FiLM starts as the identity. Press Train, then slide b through zero.

1.00

Trained here: h = leaky-ReLU(γ(b)·(wia + ci) + β(b)), γ = 1 + linear(b), 4 units, 1,500 Adam steps on 2,000 examples. Illustrative numbers.

What the paper found

On CLEVR, a benchmark of questions about rendered 3-D scenes, a GRU read the question and generated γ and β for four residual blocks of a CNN. The model reached 97.7% accuracy, against 96.9% for the best prior method (which used extra program labels) and 92.6% for humans FiLM · Table 1.

The scale matters more than the shift. Training with γ fixed at 1 cost 1.5% accuracy; with β fixed at 0, 0.5%. At test time, replacing β with its training-set mean cost 1.0%, but doing the same to γ cost 65.4% FiLM · §4.3. FiLM does most of its work by multiplying.

RT-1, a robot policy from Google, uses FiLM to fuse the instruction into the camera encoder. A Universal Sentence Encoder embeds the instruction; FiLM layers inside an ImageNet-pretrained EfficientNet-B3 apply it. The image-and-instruction tokenizer has 16M parameters and outputs 81 vision-language tokens RT-1 · §4.

A FiLM generator outputs γ = 0 for channel 7 when the question is about colour. What happens to channel 7?

Chapter 6

Cross-attention: one signal asks, the other answers

When B is a set of parts and A must pick the right one, follow the shapes of cross-attention and see why every pooled block is stuck at two thirds

Ask "what colour is the mug?" about a photo. The photo is many parts: 196 patch vectors from a vision encoder. The answer lives in a few of them. Pool all 196 first, and the mug is averaged away. The question has to reach inside the image and pick.

Cross-attention does that. A's tokens become queries; B's tokens become keys and values. Each query scores every key, the softmax turns scores into weights, and the query gets back a weighted average of the values.

XA, Q
A's tokens, B × Nq × D (batch, queries, width), projected to queries of width d.
XB, K, V
B's tokens, B × Nk × D, projected to keys and values. The scores QK⊤ are B × Nq × Nk: one number per (query, key) pair.
√d
Scaling that keeps scores from growing with the width, so the softmax does not saturate.
O
The output, B × Nq × d: one answer per query, shaped like A. Add it back to XA (a residual), so A keeps its own content.

Worked: one query, three keys, scores (2.0, 0.5, −1.0) → es = (7.389, 1.649, 0.368), sum 9.406 → weights (0.786, 0.175, 0.039). Values (1, 4, 10) → output 0.786 + 0.701 + 0.391 = 1.878, mostly the first token.

Ask the right token

Four B tokens (warm) sit at four key directions, each holding a value. A's query (blue arrow) points somewhere. Turn it and sharpen it: the weights and the answer move. Then press Train to pit concat against cross-attention on Look-up.

20°
3.0

The top panel is computed: score = length × cos(angle between query and key). Training uses the hero's Look-up data: 2,000 examples, 1,500 Adam steps, scored on 600 new ones. Illustrative numbers.

Why pooled blocks stop at 67%

Every block except cross-attention sees B through its average. In Look-up, the label is the sign of one token's value ck. The average of four independent values has correlation 1/√4 = 0.5 with any one of them. For two normal variables with correlation ρ, their signs agree with probability 1 − arccos(ρ)/π. With ρ = 0.5 that is 1 − (π/3)/π = 2/3. The hero's pooled blocks land at 67%: exactly this.

The cost: the scores form an Nq × Nk table per head. Sixteen question tokens over 196 patches is 3,136 scores, cheap. Two long streams attending to each other at every layer is not; Chapter 7 shows a way out. Padding or missing B tokens are masked: their scores are set to minus infinity, so their weights are exactly 0.

Text has 16 tokens of width 256 and the image has 196 patches of width 256, batch 2. Text queries the image. What are the shapes of the attention weights and the output?

Chapter 7

Bottleneck tokens

Let a video transformer and an audio transformer talk only through four shared tokens, and count what that saves

Put all video and audio tokens into one transformer, and every token attends to every other at every layer. For 8 frames of 14 × 14 patches plus 8 seconds of spectrogram, that is 1,568 + 400 tokens MBT · §4.2. The attention table per layer has about 3.9 million entries, and most video patches have nothing to say to most audio patches.

Nagrani et al. propose a tighter channel. Add a few fusion bottleneck tokens. Each modality's transformer layer runs on its own tokens plus the bottleneck tokens. Then the two updated copies of the bottleneck are averaged MBT · §3.2.3, eq. 8-9:

zil
The tokens of modality i (video or audio) at layer l. ‖ means "append".
zfsn
B shared bottleneck tokens (B = 4 in the paper). The only path between the modalities.
θi
Each modality's own transformer weights.
Avgi
Average the bottleneck's video-side and audio-side updates into one.

Worked (attention-score entries per layer): all tokens together (1,568 + 400 + 2 class tokens)2 = 3,880,900. Bottleneck with B = 4: 1,5732 + 4052 = 2,638,354. Separate (late): 1,5692 + 4012 = 2,622,562. The bottleneck adds 0.6% to late fusion; full joint attention adds 48%.

Count the attention bill

Each column is one of 12 layers. Its height is the number of attention-score entries: blue within video, warm within audio, purple across. Layers before the fusion layer are unimodal. Switch between full joint attention and bottleneck tokens.

1,568
400
4
8

Computed here: attention-score entries only, one class token per modality, ViT-Base depth of 12. Real cost also includes the MLP blocks; the paper reports GFLOPs in Fig. 3.

What it bought

On AudioSet (mean average precision, mAP), trained on 500K clips: audio only 41.5, video only 31.3, the bottleneck model 49.6 MBT · Table 1. On VGGSound (top-1 accuracy): audio 52.3%, video 51.2%, fused 64.1% MBT · Table 3. Fusion beat the better single modality by 8 and 12 points.

The bottleneck size barely mattered: B from 4 to 1,024 stayed within 0.5 mAP, so they kept B = 4. At fusion layer 0, bottlenecks beat full cross-attention by over 2 mAP at less than half the compute MBT · §4.3.1. Four tokens were enough because they carry only what the other modality needs.

In a bottleneck fusion layer, how does a video token learn about the audio?

Chapter 8 How

Build it in PyTorch

Four fusion modules with every shape commented, a training loop with modality dropout, the library one-liners, and the parameter bill

You have done each block by hand: a vote sum (Chapter 2), a concat (3), a gate (4), a FiLM (5) and an attention (6). Now as code. All blocks below take cached embeddings from frozen encoders, so training touches only the block and a small head.

Concat + MLP, and the gated unit

import torch
import torch.nn as nn

class ConcatMLP(nn.Module):
    """Feature fusion: concatenate two embeddings, then an MLP."""
    def __init__(self, d_a, d_b, d_h=256, n_out=1, p_drop=0.1):
        super().__init__()
        self.mlp = nn.Sequential(
            nn.Linear(d_a + d_b, d_h), nn.GELU(), nn.Dropout(p_drop),
            nn.Linear(d_h, n_out))
    def forward(self, a, b):              # a: (B, d_a), b: (B, d_b)
        x = torch.cat([a, b], dim=-1)     # (B, d_a + d_b)
        return self.mlp(x)                # (B, n_out): the non-linearity mixes them

class GatedFusion(nn.Module):
    """Gated multimodal unit: a per-dimension sigmoid gate picks A or B."""
    def __init__(self, d_a, d_b, d_h=256, n_out=1):
        super().__init__()
        self.pa = nn.Linear(d_a, d_h)               # A's expert
        self.pb = nn.Linear(d_b, d_h)               # B's expert
        self.gate = nn.Linear(d_a + d_b, d_h)       # sees both inputs
        self.out = nn.Linear(d_h, n_out)
    def forward(self, a, b):
        ha, hb = torch.tanh(self.pa(a)), torch.tanh(self.pb(b))   # (B, d_h) each
        g = torch.sigmoid(self.gate(torch.cat([a, b], -1)))       # (B, d_h), in (0, 1)
        z = g * ha + (1 - g) * hb                                  # mix per dimension
        return self.out(z), g                       # logits, and the gate to inspect

a, b = torch.randn(8, 512), torch.randn(8, 64)     # e.g. image embedding + metadata
print(ConcatMLP(512, 64)(a, b).shape)               # torch.Size([8, 1])
logits, g = GatedFusion(512, 64)(a, b)
print(logits.shape, g.shape)                        # torch.Size([8, 1]) torch.Size([8, 256])

FiLM, identity-initialised

import torch
import torch.nn as nn

class FiLM(nn.Module):
    """B produces a scale and a shift for every channel of A's features."""
    def __init__(self, d_cond, n_channels):
        super().__init__()
        self.to_gb = nn.Linear(d_cond, 2 * n_channels)
        nn.init.zeros_(self.to_gb.weight)          # identity at start: gamma = 1, beta = 0
        nn.init.zeros_(self.to_gb.bias)
    def forward(self, h, cond):        # h: (B, C, H, W) or (B, C); cond: (B, d_cond)
        dg, beta = self.to_gb(cond).chunk(2, dim=-1)   # (B, C) each
        gamma = 1 + dg
        while gamma.dim() < h.dim():                   # broadcast over H, W
            gamma, beta = gamma.unsqueeze(-1), beta.unsqueeze(-1)
        return gamma * h + beta                        # same shape as h

class FiLMBlock(nn.Module):
    """conv -> norm -> FiLM -> ReLU, as in a FiLM-ed residual block."""
    def __init__(self, c, d_cond):
        super().__init__()
        self.conv = nn.Conv2d(c, c, 3, padding=1)
        self.norm = nn.BatchNorm2d(c, affine=False)    # FiLM supplies the affine part
        self.film = FiLM(d_cond, c)
    def forward(self, x, cond):
        h = torch.relu(self.film(self.norm(self.conv(x)), cond))
        return x + h                                   # residual

img = torch.randn(4, 128, 14, 14)                     # image feature map
text = torch.randn(4, 512)                            # instruction embedding
print(FiLMBlock(128, 512)(img, text).shape)           # torch.Size([4, 128, 14, 14])

The FiLM-ed block follows Perez et al.: batch norm without its own affine parameters just before FiLM, then ReLU, inside a residual block FiLM · §2.2. They used 128 feature maps on 14 × 14 image features, the shapes in the example.

Cross-attention, with the library one-liner inside

import torch
import torch.nn as nn

class CrossAttnFusion(nn.Module):
    """A's tokens ask questions; B's tokens answer."""
    def __init__(self, d, n_heads=8, p_drop=0.1):
        super().__init__()
        self.norm_q, self.norm_kv = nn.LayerNorm(d), nn.LayerNorm(d)
        self.attn = nn.MultiheadAttention(d, n_heads, dropout=p_drop, batch_first=True)
        self.ff = nn.Sequential(nn.LayerNorm(d), nn.Linear(d, 4 * d), nn.GELU(), nn.Linear(4 * d, d))
    def forward(self, xa, xb, b_pad=None):
        # xa: (B, Nq, d) queries; xb: (B, Nk, d) keys and values
        # b_pad: (B, Nk) bool, True = padding or a missing B token
        out, w = self.attn(self.norm_q(xa), self.norm_kv(xb), self.norm_kv(xb),
                           key_padding_mask=b_pad)     # out: (B, Nq, d), w: (B, Nq, Nk)
        xa = xa + out                                  # residual: A keeps its own content
        return xa + self.ff(xa), w

xa = torch.randn(2, 16, 256)          # 16 text tokens
xb = torch.randn(2, 196, 256)         # 196 image patches
pad = torch.zeros(2, 196, dtype=torch.bool)
y, w = CrossAttnFusion(256)(xa, xb, pad)
print(y.shape, w.shape)               # torch.Size([2, 16, 256]) torch.Size([2, 16, 196])

The other library one-liner is the bilinear join: nn.Bilinear(512, 64, 256) computes every product of an A entry and a B entry, mixed into 256 outputs. It has 8,388,864 weights. That is why people use low-rank products (project each side, then ⊙) instead.

Train it, with modality dropout

import torch
import torch.nn.functional as F
# ConcatMLP is the class defined above
torch.manual_seed(0)
N, d_a, d_b = 6000, 32, 8
a, b = torch.randn(N, d_a), torch.randn(N, d_b)        # stand-ins for cached embeddings
y = ((a[:, 0] + b[:, 0]) > 0).float()                  # the answer needs both
tr, te = slice(0, 5000), slice(5000, N)
model = ConcatMLP(d_a, d_b, d_h=64)
opt = torch.optim.AdamW(model.parameters(), lr=1e-3, weight_decay=1e-2)
p_miss = 0.3                                            # modality dropout rate for B
for step in range(3000):
    idx = torch.randint(0, 5000, (64,))
    xa, xb = a[idx], b[idx].clone()
    xb[torch.rand(64) < p_miss] = 0.0                  # zero-fill B, as at test time
    loss = F.binary_cross_entropy_with_logits(model(xa, xb).squeeze(-1), y[idx])
    opt.zero_grad(); loss.backward(); opt.step()
model.eval()
with torch.no_grad():
    acc = lambda xb: ((model(a[te], xb).squeeze(-1) > 0).float() == y[te]).float().mean().item()
    print(f'both {acc(b[te]):.3f}  B missing {acc(torch.zeros_like(b[te])):.3f}')

We ran every block on this page (PyTorch 2.10). The last one prints both 0.960 B missing 0.748: with B gone, it falls to the 75% that A alone allows on a sum rule (Chapter 0), not below.

The training details that matter. Normalise each embedding (a LayerNorm per modality) before the join: a 512-number image vector and 3 numbers of patient age, sex and site have very different scales, and the bigger one wins by default. Keep encoders frozen and cache their outputs when data is small. Use weight decay and dropout on the head. Start FiLM as the identity. And always train the two single-modality baselines with the same budget.

|θconcat|
Weights of ConcatMLP with one output: the first layer, its bias, the output weights and bias. Dropout and GELU have none.
|θgated|
GatedFusion: two experts (dAdh and dBdh), a gate that reads both ((dA + dB)dh), three biases and the output layer.

Worked: dA = 512, dB = 64, dh = 256. Concat: 576 × 256 + 512 + 1 = 147,969. Gated: 2 × 576 × 256 + 1,024 + 1 = 295,937, twice the bill.

Shapes and the bill

Choose a block and set the sizes. The diagram traces the tensor shapes from the two inputs to the logit, and the readout counts the weights.

512
64
256
196

Counts include biases and one output logit. FiLM here: A projected to dh channels, B generating γ and β for each. Cross-attention: A and B projected to dh, one multi-head attention (4dh2 + 4dh), with Nq = 16 and no feed-forward block.

Why zero-initialise the layer that produces FiLM's γ and β when you write γ = 1 + Δγ?

Chapter 9

Missing inputs, and one input winning

Train a model that survives a lost sensor, then meet the stranger failure: a fused model that scores below one of its own inputs

When B goes missing

At test time a sensor can fail, a field can be empty, a clip can be silent. You must feed the model something. Zeros are common. But a model that never saw zeros during training reads them as a real, extreme value. If heart rate is centred near 70 after scaling, zero means "no pulse", and the alarm fires.

The fix is modality dropout: during training, delete a whole modality at random, with probability p, and fill it the same way you will at test time. Add a mask flag, a 0/1 input that says "B is missing". Neverova et al. introduced this as ModDrop for gesture recognition, to make predictions robust to missing or corrupted channels ModDrop · §5.

m
1 if B is present for this example, 0 if dropped. Drawn fresh every time the example is used.
p
The drop rate. Match it to how often B is really missing, or a little higher.
x̃B
B as the model sees it: real, or zeros. Feeding m lets the model tell "missing" from "zero".

Worked: batch of 64, p = 0.3 → about 19 rows lose B each step. Over 1,500 steps the model sees about 29,000 B-missing rows, enough to learn a fallback that uses A alone.

The robustness curve

Label: a + b > 0. B arrives shifted (centred at 3), with a presence flag. Set the training drop rate and press Train. The curves show test accuracy as a growing share of test examples lose B (zeros, flag 0).

0.30

Trained here: concat + MLP (12 tanh units), 3,000 examples, 1,500 Adam steps, scored on 1,500 new examples. Illustrative numbers.

When one input wins: modality imbalance

A fused model contains every single-modality model: set the weights on B to zero and you have A alone. So it should never score lower. Wang, Tran and Feiszli found the opposite on Kinetics: audio + RGB, trained jointly with late concatenation, reached 71.4% top-1 against 72.6% for RGB alone Wang 2020 · Table 1. Every pair and the triple they tried lost to the best single stream.

They name two causes. First, the joint model has more capacity, so it overfits more. Second, modalities overfit and generalise at different rates, so one learning rate and one stopping time cannot suit both Wang 2020 · abstract. Often the fast-fitting modality memorises the training set and the loss stops pushing the slow, useful one.

They measure it with the overfitting-to-generalisation ratio between two checkpoints, and weight each modality's loss by it Wang 2020 · eq. 3, Alg. 1:

O
Overfitting: the gap between validation loss and training loss. ΔO is how much it grew between checkpoints.
G
Generalisation: how much the validation loss fell between checkpoints.
OGR
Bits learned that do not generalise, per bit that does. Low is good.
wi
Gradient-Blending's weight on modality i's loss (a per-modality head), normalised to sum to 1. Their total loss is Σ wiLi.

Worked: A has G = 0.02, O = 0.10 → 0.02/0.01 = 2. B has G = 0.03, O = 0.02 → 0.03/0.0004 = 75. Normalise: wA = 2/77 = 0.026, wB = 75/77 = 0.974. The overfitting modality gets almost no weight.

On Kinetics audio + RGB this lifted top-1 from 71.7% (naive joint) and 72.6% (RGB only) to 74.7% offline and 75.8% online Wang 2020 · Table 2.

The overfitting race

Only 120 training examples. A has 80 weak features (easy to memorise); B has one decent feature. Curves are validation accuracy over training. Lower A's gradient weight and train again: does the joint model beat B alone?

0.06

Trained here: logistic regression with plain SGD (learning rate 0.05, batch 32, 3,000 steps), scored on 2,000 new examples. "Blended" multiplies the gradient of A's weights by wA, in the spirit of Gradient-Blending, not the paper's exact method. Illustrative numbers.

You train with zeros for missing B but never drop B in training. At test, 30% of examples lack B. What is the most likely result?

Chapter 10 Did it work?

Did it work?

Prove that your fused model uses both inputs: unimodal baselines, ablations, the shuffle test, and the bugs they catch

A good test score does not prove fusion. The model may be ignoring B, or using B for the wrong reason. Run these checks, in this order.

  1. Unimodal baselines. Train A alone and B alone, with the same encoders, budget and tuning. If the fused model does not beat the better one on held-out data, ship the simpler model.
  2. Ablate each input. Zero B (or set it to its training mean) at test time. A large drop means the model leans on B.
  3. Shuffle one modality. Pair each example's A with another example's B, keeping the labels. This keeps B's distribution but breaks the pairing. If accuracy does not fall, the model is not using the A-B relationship.
  4. Robustness curve. Plot accuracy as more examples lose B (Chapter 9).
  5. Slices. Report the metric per class and per condition (night images, noisy rooms). Fusion often helps only where one modality is weak; MBT found fusion lifted AudioSet AP by over 60% absolute on classes with weak audio, such as "bicycle" MBT · §4.4.
  6. Inspect, then distrust, the gates and attention. They hint at what the model uses. The ablations measure it.
RB
Reliance on B: how much accuracy the true pairing is worth. Near 0 means B is unused.
Bπ
B shuffled across the test set by a random permutation π.

Worked (the PyTorch model of Chapter 8, sum rule): both 0.960, B shuffled 0.644, B zeroed 0.748, A shuffled 0.673 → RB = 0.316. Shuffling hurts more than zeroing: a wrong B is misleading evidence, a zero B is no evidence.

import torch

@torch.no_grad()
def reliance(model, a, b, y):
    """How much does the fused model lean on each input? (a: (N, d_a), b: (N, d_b))"""
    model.eval()
    acc = lambda xa, xb: ((model(xa, xb).squeeze(-1) > 0).float() == y).float().mean().item()
    perm = torch.randperm(len(y))                    # pair each A with someone else's B
    return {'both': acc(a, b),
            'shuffle B': acc(a, b[perm]),             # breaks the A-B pairing
            'shuffle A': acc(a[perm], b),
            'zero B': acc(a, torch.zeros_like(b))}    # the missing-B case

The ablation panel

Pick a rule and a block, then press Train. Three models train: the fused block, A alone and B alone. The bars are test accuracy for each, and for the fused block with B zeroed, B shuffled and A shuffled. The dashed line is chance.

The hero's data and blocks, trained here: 2,000 examples, 1,500 Adam steps each, scored on 600 new ones. Illustrative numbers.

Bugs these checks catch

Your image + metadata model scores 91%. With metadata shuffled across the test set it still scores 91%. What do you conclude?

Chapter 11 What elseWhat not

Where else, when not, and the cheat sheet

Move the blocks to six fields, meet the cases where fusion hurts, and keep the whole lesson on one screen

Where else it works

FieldA + B → answerUsual block, and what changes
Audio-visual eventsVideo frames + soundtrack → "what is happening"Middle fusion through bottleneck tokens. On VGGSound, fusion beat the better single modality by 12 points MBT · Table 3. Audio and video can drift in time; MBT found results largely robust to sampling the two windows separately MBT · §4.3.2.
Visual question answeringQuestion + image → answerThe question selects (Look-up kind): FiLM from the question into the CNN, as on CLEVR FiLM · Table 1, or question tokens cross-attending to patches.
Robot policiesCamera + instruction (+ joint angles) → actionRT-1 FiLM-conditions the image encoder on the instruction, starting from the identity RT-1 · §4. Low-dimensional robot state (proprioception) is commonly concatenated with the vision features.
WearablesMotion (IMU) + heart rate → alarm or activityDifferent rates, so middle or late fusion. Sensors drop out, so modality dropout and a mask flag. The alarm is an interaction, so not late fusion.
Medical imagingScan + table data (age, sex, scanner) → diagnosisB is a few numbers, A is a big image. FiLM from the table into the CNN, or concat after normalising each. Beware leaks: a scanner ID that tracks the hospital, which tracks the disease.
Text + image classificationProduct photo + title, poster + plot → categoryGated units: Arevalo et al. classified movie genres from poster and plot on MM-IMDb, 25,959 movies GMU · §4. Often one modality (the text) carries most of it, so check the unimodal baseline first.

What stays the same in every row: two encoders, one join, the four kinds of rule, and the checks of Chapter 10. What changes: the shapes, which signal asks and which answers, and how often each input goes missing.

When not to fuse, or not this way

SituationWhy it hurtsUse instead
One modality is enough (its baseline matches the fused model)The second input adds noise, overfitting, cost, a second sensor to failThe unimodal model. Revisit if a slice (night, noise) needs the other input.
Tiny data, wide embeddings, concat or bilinear(dA + dB)dh or dAdB weights fit noise (Chapter 9's race)Late fusion of two strong unimodal models, low-rank projections first, or FiLM (2 numbers per channel)
Streams misaligned in timeEarly fusion pairs the wrong samples (Chapter 1)Align or resample first; else late fusion, or cross-attention over a time window
An input is often missing at testZeros are out of range for a model that never saw themLate fusion, or any block with modality dropout and a mask flag
The rule is an interactionLate fusion and concat + linear are capped (75%) and train to about 50%Concat + MLP, a product, a gate or FiLM
One input selects part of the otherPooling B first averages the answer away (67% in the toy)Cross-attention, A querying B's tokens

Which block for my problem?

Choose an answer in each row. The pipeline below updates.

A rule of thumb from this lesson, not a law. Always compare with both unimodal baselines on held-out data.

The cheat sheet

BlockShape signatureMaths in one lineLearns
Late(B, 1) + (B, 1) → (B, 1)wAℓA + wBℓB + cAdding evidence; survives a missing input. No interaction (≤ 75% on XOR).
Concat + MLP(B, dA) + (B, dB) → (B, dA+dB) → (B, C)W2φ(W1[hA; hB])Interaction, through the hidden layer only. The default.
Product(B, d) ⊙ (B, d) → (B, d)hA ⊙ hBInteraction directly. Full bilinear is dAdB: use low rank.
Gatedtwo (B, d) experts + (B, d) gate → (B, d)g⊙hA + (1−g)⊙hB, g = σ(W[xA; xB])Per-example trust; ignores a broken sensor.
FiLM(B, C, H, W) + (B, dB) → (B, C, H, W)γ(xB)⊙F + β(xB), γ = 1 + ΔγB steers A's channels. Cheap; directional; zero-init.
Cross-attention(B, Nq, D) × (B, Nk, D) → (B, Nq, D)softmax(QK⊤/√d)VSelection inside a set; mask padding. Cost NqNk.
Bottleneck tokens(B, Nv, D) ‖ (B, 4, D) ‖ (B, Na, D)per-modality layers share B tokens, averagedTwo transformers trade summaries at near-late cost.
PartIn one line
KnobsJoin point (early, middle, late), block type, width dh, per-modality normalisation, modality dropout rate p, per-modality loss weights.
Failure modesVotes on an interaction; pooled B on a selection; zeros never seen in training; one modality overfitting; misaligned clocks; broken pairing; scale domination; leaky metadata.
Did it work?Beat both unimodal baselines; ablate (zero, mean) and shuffle each input; robustness curve; slices.
AlternativesA single modality; stacking of two unimodal models; one shared transformer over both token sets (early fusion).

Honest limits

Where this sits, and where to go next

This lesson is part of the Builder's Toolkit series on reusable blocks for frozen backbones. Go on to:

The takeaway. A fusion block decides which functions of two inputs your model can express. Adding votes (late, or concat plus linear) handles evidence that adds up and survives a missing input. A non-linearity, a product, a gate or FiLM after the join handles interaction. Cross-attention handles selection. Train with modality dropout, weigh a fast-overfitting input down, and prove fusion with unimodal baselines and the shuffle test.
A dermatology model combines a skin photo with 3 table fields (age, body site, sex). You have 2,000 cases, and the body site is often blank. Which design comes first?

Now press Present or Teach and explain, out loud and from memory, why late fusion cannot learn XOR and what FiLM's γ does. Then go back to the hero and predict, before training, which blocks will pass Look-up.

Fusion Blocks. Sources: E. Perez, F. Strub, H. de Vries, V. Dumoulin, A. Courville, "FiLM: Visual Reasoning with a General Conditioning Layer" (AAAI 2018, arXiv); A. Nagrani, S. Yang, A. Arnab, A. Jansen, C. Schmid, C. Sun, "Attention Bottlenecks for Multimodal Fusion" (NeurIPS 2021, arXiv); W. Wang, D. Tran, M. Feiszli, "What Makes Training Multi-modal Classification Networks Hard?" (CVPR 2020, arXiv); A. Brohan et al., "RT-1: Robotics Transformer for Real-World Control at Scale" (2022, arXiv); J. Arevalo, T. Solorio, M. Montes-y-Gómez, F. A. González, "Gated Multimodal Units for Information Fusion" (ICLR 2017 workshop, arXiv); N. Neverova, C. Wolf, G. Taylor, F. Nebout, "ModDrop: adaptive multi-modal gesture recognition" (2015, arXiv).
Back to Gleams