Interactions and Fusion Models

Two signals can say more together than apart. A sum of them cannot see it. A product can. Slide the pairing and watch four fusion rules score the same 200 clips.

Learn how a model combines words, voice, pictures and more into one decision, and how to test whether it truly uses the pairing.

Pick a rule, then drag the pairing slider all the way right. Watch the words-only rule and the plain sum drop to a coin flip while the product stays right. Then we build, piece by piece, early and late fusion, the product term, tensor and low-rank fusion, gates, and the test that tells you whether a model learned any interaction at all.

You need the idea of a number line and a slope, and a little Python for the code. We build the rest from zero.

Based on Lecture 4.1, Multimodal Fusion, of MIT MMAI, Multimodal AI (Spring 2026), taught by Paul Liang of the MIT Multisensory Intelligence group; some of its slides were co-developed with Louis-Philippe Morency for CMU 11-777. Lecture slides. Every idea is re-taught here in our own words and devices.

Fuse words and voice

0%

Toy data, labelled illustrative: 200 clips with a seeded words tone and voice tone between −1 and +1. At 0% the true answer is “yes” when the words are positive. At 100% it is “yes” when words and voice disagree, the sarcasm case of the lecture’s slide 7. In between, that share of clips follows the pairing rule. Five percent of the answers are flipped as label noise. The words and voice rules are the best cut on one tone, from 13 cut points; the sum rule is the best weighted sum of both tones, cut at zero, from 36 directions; the product rule is the best cut on words × voice.

Chapter 0

Why Signals Must Be Fused

See why one signal alone can mislead, and what a fusion model must learn to do about it

A friend says, “Great job.” Read on a page, the words are praise. Now hear it: flat voice, eyes rolled to the ceiling. The same two words now mean the opposite. Neither signal told you that alone. The words said “good”. The voice said “not quite”. The meaning lived in the pair.

This is the problem of the whole lecture. A modality is one kind of signal: the words of a sentence, the sound of a voice, the pixels of a face. Most real tasks give a model several at once. The model must turn them into one decision, and the hard part is that the best decision often cannot be read off any single signal.

The definition from slide 4

The lecture calls this job multimodal fusion. Its definition fits on one line: learn a joint representation that models the multimodal interactions between individual elements of different modalities. Take that apart. A representation is a list of numbers that stands for something, the way a map pin stands for a city. Joint means one list for both signals, not one list each. An interaction is information that exists only because two signals meet. Fusion is the machine that makes the joint list.

The slide draws the machine as a funnel. Two coloured bars go in, one bar comes out. How different the two inputs are decides how hard the funnel is to build. Hold that thought, because the slide splits it into two cases.

Abstract modalities and raw modalities

In the first case the lecture calls them abstract modalities. Each signal has already been through an encoder, a neural network that turns raw input into a vector (a list of numbers). A vision model such as a CNN or a ViT does this for pictures. A language model such as Word2vec or BERT does it for words. After encoding, both signals are lists of numbers of similar kind. The slide calls this closer homogeneity. It adds one practical note: the encoders can be trained together with the fusion network, or they can be pre-trained and frozen.

In the second case, raw modalities, nothing has been encoded yet. A picture is a grid of continuous colour values. A sentence is a short list of whole-number word codes. These two share almost nothing in shape, scale or meaning. The slide calls this heterogeneity, and it sits at the far end of a line that runs from homogeneous to heterogeneous. Try both cases below and watch the shapes.

Trace the shapes into the funnel

Each box is one tensor (a block of numbers), with its shape written under it. Choose what enters the fusion funnel: raw signals, or signals an encoder has already turned into vectors.

Illustrative shapes: a 224 × 224 colour picture, a 12-word sentence, 768-number vectors from a ViT-style and a BERT-style encoder, and a 512-number joint vector. The lecture draws the idea (slides 4 and 8) without fixed sizes; the arithmetic here is exact.

Raw, the picture is 3 × 224 × 224 = 150,528 real numbers on a grid. The sentence is 12 integers, each a position in a word list. A fusion network cannot add these, multiply them or line them up. After the encoders, both are 768 numbers. Now a layer can combine them element by element. This is why most of the lecture works on the abstract case first, and why the encoders matter so much.

Logistics, in one paragraph

Slide 2 holds the week’s business. The project proposal is due the day of the lecture, and Paul Liang offers feedback from 4 to 5 pm and wants to meet every group at least once that day or on Thursday. The course gives out compute credits: 40 credits of $50 for Kimi and 40 of $40 for other providers. Homework 2 was released the week before and is due Wednesday, March 4.

Today’s plan, and where the readings fit

Slide 3 lists four parts. First, the basics of multimodal interactions and fusion. Second, early, intermediate and late fusion. Third, multiplicative and dynamic fusion. Fourth, quantifying fusion, which means measuring how much of an interaction a model uses. The deck credits Louis-Philippe Morency for slides co-developed for CMU 11-777.

The assigned readings come in two groups. Two papers give tools for measuring interactions: the information decomposition framework of Liang and coauthors (arXiv 2302.12247), taught in Chapter 1, and the EMAP test of Hessel and Lee (EMNLP 2020), taught in Chapter 7. Four papers show fusion inside large models: Chameleon, Kosmos-2, MM1 and MoMa, summarised in Chapter 9 with links to their own lessons.

The road below follows the slides in order. Each chapter opens with a small scene and ends with a one-idea quiz.

  1. Interactions: redundancy, uniqueness and synergy, and how to measure them.
  2. Early and late fusion: combine the inputs first, or combine the answers last.
  3. The linear case: the interaction term in a regression, on 300 book reviews.
  4. Multiplicative and tensor fusion: the product of two vectors, and why it grows so fast.
  5. Low-rank fusion: the same product at a fraction of the cost.
  6. Gated and shifting fusion: let the inputs decide how much to trust each other.
  7. Nonlinear fusion and EMAP: test whether a network learned any interaction.
  8. Residual optimization: make the model prefer the simplest interaction that works.
  9. Fusion in large models: early-fusion tokens, and the four reading papers.
  10. Field guide: numbers, code and where to go next.
Why does the lecture treat encoded (abstract) modalities as the easier case?

Chapter 1

Redundancy, Uniqueness and Synergy

Sort what two signals tell you about an answer into the part they share, the part only one holds, and the part that appears only in the pair

You skim film reviews and want to know whether each one is positive. Review one reads, “This movie is great!”, and the reviewer’s photo is laughing. Words and face say the same thing. Review two reads, “The movie does a good job developing…”, and the face is blank. The words carry the verdict. The face adds nothing. Review three reads “wowwww” over an eye-roll. Neither signal says “negative” alone, yet together they do.

Slides 5, 6 and 7 of the lecture draw exactly these three scenes. Each one shows a different way that two modalities can combine to give information about a task, and the lecture’s word for that is interactions: how modalities combine to provide information for a task.

Three kinds, in plain words

Redundancy is information that both signals give. If you removed the face, the words would still tell you. Slide 5 draws it as the overlap of two circles that also overlaps the task. Uniqueness is information that only one signal gives. In review two the words are unique: the face stays neutral and says nothing. Slide 6 marks a region of modality 1 outside the overlap, and a matching one for modality 2. Synergy is information that exists only when you read both together. Slide 7 calls it emerging information from multimodal interaction. Sarcasm is the standard example, because the meaning comes from the clash.

Why does the split matter? Because each kind asks for a different model. If the information is redundant, one signal is enough and a second signal only adds robustness. If it is unique to one signal, the other signal is dead weight. Only synergy forces you to build a model that really looks at both at once. That last case is where fusion earns its keep.

Measuring the three

To count them you need a unit. Information theory gives one. The mutual information I(A;Y) is how many yes-or-no questions’ worth of the answer Y you learn by seeing A. One bit means you can halve the uncertainty exactly once. The reading for this lecture, Quantifying & Modeling Multimodal Interactions: An Information Decomposition Framework by Paul Pu Liang and coauthors, uses the idea of partial information decomposition, PID for short. It splits the information that both signals give about the task into the four parts above.

I(A,B;Y)
Everything the pair of signals tells you about the answer, in bits.
R
Redundancy: what each signal tells you on its own, shared. Also I(A;Y) = R + UA and I(B;Y) = R + UB.
UA
Uniqueness of A: what only A knows about the answer.
UB
Uniqueness of B: what only B knows about the answer.
S
Synergy: what you learn only when A and B are read together.

The paper calls R, UA, UB and S the PID statistics of a multimodal dataset. It introduces two new estimators that scale to high-dimensional data, checks them on synthetic data where the true answer is known, and compares them with human labels on real benchmarks. It then uses them in four ways: to describe datasets, to describe what a trained model captured, to pick a model, and in three case studies in pathology, mood prediction and robotic perception.

Worked example: the exclusive-or

Take the smallest synergy there is. Two bits, A and B, are each equally likely 0 or 1 and independent. The answer is Y = A xor B: yes when they differ. Step one: what does A alone tell you? If A is 0, Y is 0 or 1 with equal chance; if A is 1, the same. So I(A;Y) = 0 bits. Step two: the same holds for B, so I(B;Y) = 0. Step three: the pair fixes Y exactly, so I(A,B;Y) = 1 bit. The equation says 1 = R + UA + UB + S with the first three all zero, so S = 1. All the information is synergy, and no single-signal rule can beat a coin flip.

Now try it. The device holds four tiny worlds, each a table of how A, B and the answer relate. Slide the label noise and read the bars.

Split the information

The bar is everything the pair tells you about the answer. Its colours show who holds it. Pick a world, then add label noise and watch the pieces shrink.

0%

Toy tables, labelled illustrative, with exact arithmetic. Redundancy is taken as the smaller of the two single-signal informations, a simple choice that is exact for these four worlds; the paper’s estimators handle general, high-dimensional data. Noise flips the answer with the chosen probability.

Pick Synergy and leave the noise at 0%. The bar is all warm: 1.00 bit of synergy, and yet the best rule from the words alone is right on half the clips. Pick Redundant: the bar is all grey, and either signal alone gets every clip right. Add 10% noise and every bar shrinks, because a noisy answer carries fewer bits to share.

Failure mode: a clever model on a dataset with no synergy

The symptom: a team builds a large cross-attention fusion model, and it scores the same as the unimodal text baseline. The measurement: estimate PID on the data. If S is near zero and UA is large, nearly everything lives in one signal. The fix: stop paying for fusion. Choose the simplest model that matches the information, and put the effort into a better encoder for the signal that carries it. The paper’s own case studies recommend models in just this way.

In the exclusive-or world, A alone and B alone each carry 0 bits about the answer, but the pair carries 1 bit. Which kind of interaction is that?

Chapter 2

Early Fusion and Late Fusion

Choose between mixing the signals before anyone decides and mixing the decisions after, and see what each choice can and cannot learn

A clinic sees a patient with two readings: a temperature, and the sound of a cough. The manager has two ways to set up the desk. In the first, a nurse writes both readings on one sheet and hands it to a single doctor, who decides. In the second, a fever specialist reads only the temperature and gives a verdict, a lung specialist listens to only the cough and gives a verdict, and a clerk averages the two.

The lecture’s slide 9 draws exactly these two desks. The first is early fusion: combine the inputs, then predict once. The second is late fusion: predict from each input alone, then combine the predictions. The names say when the signals meet, early in the pipeline or late.

What flows where

Follow the data. In early fusion, the two inputs xA and xB are joined by a combination step, often just placing the two vectors end to end (concatenation). One model reads the joined vector and outputs one prediction. In late fusion, each input goes into its own model. Each model outputs its own prediction, written ŷA and ŷB. A small combiner, typically an average or a vote, turns the two predictions into the final ŷ.

There is a third design in between, intermediate fusion. Each input first goes through its own encoder (slide 8). The model then merges the hidden vectors in the middle, and later layers keep working on the merged result. Most modern systems are of this kind. The agenda on slide 3 names all three, early, intermediate and late, and most of this lecture is about the merge step inside the middle one.

Mix early or mix late

Top: the two pipelines. Bottom: how often each design is right on 100 held-out clips. Slide the pairing to the right, then switch off the voice and see which design survives.

100%

Toy data, labelled illustrative: 200 seeded clips with a words tone and a voice tone, labelled as in the hero; each design learns on 100 clips and is scored on the other 100 (the pairing share follows the sarcasm rule, the rest follows the words). Late fusion averages two single-signal predictors, each the share of yes answers in its half of the line. Early fusion looks up the share of yes answers in each of the four quadrants. A dropped voice is filled with a constant positive tone.

Leave the pairing at 100%. Late fusion sits near a coin flip (a little under or over 50%, since 100 clips is a small sample), and early fusion is close to perfect, held back only by the 5% label noise. The reason is the arithmetic of the clerk. Each specialist says “50 percent yes”, because words alone or voice alone cannot tell sarcasm. The average of two 50s is 50. The single doctor saw both readings on one sheet, so she could learn that a clash between them means yes.

A worked example, eight clips

Take eight clips, two in each quadrant of words and voice. Quadrant (+, +) holds two “no” clips. Quadrant (+, −) holds two “yes”. Quadrant (−, +) holds two “yes”. Quadrant (−, −) holds two “no”. Step one: the words specialist sees four clips with positive words, two yes and two no, so it says 0.5. For negative words it also says 0.5. Step two: the voice specialist says 0.5 in both halves too. Step three: the clerk averages 0.5 and 0.5 to get 0.5, a tie, so the best it can do is guess: 4 right out of 8 on average. Step four: the single doctor looks up each quadrant, and every quadrant is pure, so she gets 8 out of 8.

Now drop the voice. Late fusion loses one specialist, but the clerk still averages whatever arrives, so it keeps working as before. The early model was trained on sheets with two readings, and now one box is blank. Its lookup lands in the wrong quadrant half the time. Press the toggle and watch the early bar fall.

Choosing between them

DesignWhere signals meetCan it see synergy?If one input goes missing
EarlyAt the input, before any predictionYes, if the model is expressive enoughHurts: the model expects both
IntermediateIn the middle, between encoders and the headYes, through the merge stepDepends on how it was trained
LateAt the predictions, after each input decidedNo: it sees only each side’s verdictDegrades gracefully

The lesson is not that early beats late. It is that late fusion cannot represent a pure interaction, and early fusion can but depends on both inputs being present. When the information is redundant or unique (Chapter 1), late fusion is cheap, modular and robust. When synergy matters, the merge must happen before the verdict.

python# shapes: x_a [B, d_a], x_b [B, d_b], C classes
# late fusion: two heads, average the class probabilities
p_late = (head_a(x_a).softmax(-1) + head_b(x_b).softmax(-1)) / 2      # [B, C]
# early fusion: join the inputs, one model decides
p_early = mlp(torch.cat([x_a, x_b], dim=-1)).softmax(-1)            # [B, d_a + d_b] to [B, C]

Failure mode: the model that quietly ignores one input

The symptom: an early-fusion model scores well, but a test with the voice blanked gives almost the same score. The measurement: compare accuracy with each input removed in turn; a tiny drop means the model barely used it. The fix: pre-train each encoder alone so that both carry signal, or train with random input dropout so the model learns to use whichever inputs it has. Chapter 7 gives a sharper test.

Two specialists each see only half the evidence and each says 50 percent. A clerk averages them. Why can this design never solve a pure exclusive-or task?

Chapter 3

The Linear Case: One Product Term

Add a single multiplied term to a straight-line fit and watch it bend the answer for one group but not the other

Picture 300 book reviews. For each one you know two things from the page: how much the reviewer is smiling in their photo, and whether the reviewer is a professional critic. You also know one outcome: the audience score the review earned. You want a formula that predicts the score from the two facts.

This is the lecture’s example on slides 10 to 14. It is deliberately tiny, so that every number can be read. The lecture’s plan is to study the univariate case first: each signal is a single number, not a vector. What we learn here carries over to vectors in the next chapters.

The regression, term by term

A linear regression predicts a number as a weighted sum of inputs. The lecture writes it with an extra product on slide 10:

y
The audience score we want to predict.
w0
The intercept (the bias term): the average score when xA and xB are both zero.
w1xA
An additive term: the effect of smiling alone. Here xA is the percentage of smiling.
w2xB
An additive term: the effect of professional status alone. Here xB is 0 for a non-critic and 1 for a critic.
w3xAxB
The multiplicative term: the effect that exists only when smiling and status are both present.
ε
The error (the residual): whatever the weights w0 to w3 do not explain.

Why is a product the way to model an interaction? Think of what a product does. It is large only when both numbers are large, and it flips sign when one flips sign. A sum cannot do that: in a sum, each input’s push is the same whatever the other input is. In a product, how much xA counts depends on xB. That dependence is the interaction.

Three fits on the same 300 reviews

The lecture fits three models and prints their weights. The first, on slide 12, uses smiling alone: y = w0 + w1xA. It gives w0 = 4.63 and w1 = 1.20. The second, on slide 13, adds status: w0 = 5.29, w1 = 1.19, w2 = −1.69. Smiling has a positive effect and being a critic a negative effect. The third, on slide 14, adds the product: w0 = 5.79, w1 = 0.68, w2 = −2.94, w3 = 1.29. The slide labels the new weight multiplicative interaction!

See what the third model says. With xB = 0 (non-critics) the product vanishes, so the slope in smiling is w1 = 0.68. With xB = 1 (critics) the slope is w1 + w3 = 0.68 + 1.29 = 1.97. The two groups have different slopes. The second model could not say that: it forces both groups to share the slope 1.19 and only shifts one line below the other by 1.69.

Bend one line

Each model draws one line per group: critics in warm, non-critics in blue. Choose a model, then in the third one drag the interaction weight and watch only the critic line tilt.

1.29

The weights are the lecture’s (slides 12 to 14). The range of smiling, 0 to 1, is an assumption for the picture, since the slide does not state its scale; the lines and slopes follow from the weights exactly. The w3 slider is a what-if on the third model: the other weights stay as fitted.

A worked number

Set smiling to 0.5. Step one, additive model: a non-critic scores 5.29 + 1.19 × 0.5 = 5.885, and a critic scores 5.885 − 1.69 = 4.195. Step two, product model, non-critic: 5.79 + 0.68 × 0.5 = 6.13. Step three, product model, critic: 6.13 − 2.94 + 1.29 × 0.5 = 3.835. The gap between the groups has grown from 1.69 to 2.30, and it will keep changing as smiling changes. At smiling 1.0 the product model’s gap is 2.94 − 1.29 = 1.65, nearly closed. That is what a changing gap looks like, and a sum cannot draw it.

Notice also what happened to the other weights. When the product entered, w1 fell from 1.19 to 0.68 and w2 fell from −1.69 to −2.94. The weights changed meaning. In the third model, w1 is the smiling slope for non-critics only, and w2 is the critic offset at zero smiling only. This is the first lesson of fusion, and it returns again and again: once an interaction is in the model, a single input no longer has a single effect.

pythonimport numpy as np
# x_a: [300] smiling, x_b: [300] critic (0 or 1), y: [300] audience score
X = np.stack([np.ones(300), x_a, x_b, x_a * x_b], axis=1)   # [300, 4]: one column per weight
w, *_ = np.linalg.lstsq(X, y, rcond=None)                    # [4]: w0, w1, w2, w3
slope_noncritic = w[1]
slope_critic    = w[1] + w[3]                                  # the product term moves the slope

Failure mode: forgetting the product

The symptom: the two groups’ points fan out, but the fitted lines stay parallel, and the residuals form a pattern, positive on one side and negative on the other. The measurement: refit with the product column and compare. If w3 is large and the error drops, the pairing mattered. The fix: keep the product term, and read the other weights as conditional on the other input.

In the third model, what is the slope of audience score against smiling for critics?

Chapter 4

Multiplicative, Bilinear and Tensor Fusion

Multiply every part of one signal by every part of the other, and count how fast the result grows

A restaurant pairs wines with dishes. Each wine has a few traits: dry, sweet, strong. Each dish has a few too: salty, rich, spicy. The score of a pairing is not the wine’s score plus the dish’s score. A strong wine is wonderful with a rich dish and awful with a delicate one. The score lives in the pair of traits, and the way to hold it is a table with one cell for every wine trait against every dish trait.

That table is the idea of this chapter. In Chapter 3 two numbers had one product. Now each signal is a vector with many numbers, and we want a product for every pair.

From one number to a representation

Slide 15 restates the regression, but now the output is a representation z, a list of numbers that feeds the next layer, instead of a score. It names three ways to build it. An additive interaction is z = w1xA + w2xB. A multiplicative interaction is z = w3xAxB. The third has both. The lecture then moves to the multivariate case with vectors, which we follow.

Slide 16 gives additive fusion for vectors: z = w1xA + w2xB. With encoders, it becomes z = fA(xA) + fB(xB). The slide notes that this can be seen as an ensemble, which is late fusion in disguise. The two parts never multiply, so no pair of traits ever meets. That is exactly why Chapter 2’s clerk failed.

Slide 17 gives the first way to multiply. Multiplicative fusion is z = w(xA × xB), where the product is taken element by element: the first number of xA times the first of xB, the second times the second, and so on. This needs both vectors to have the same length, and it pairs only traits that share a position. Bilinear fusion is Z = w(xATxB), where the product is the outer product: every number of xA times every number of xB, giving a full matrix Z with one cell per pair. The vectors may now have different lengths.

Tensor fusion: keep the sums too

Slide 18 shows the tensor fusion network of Zadeh and coauthors (EMNLP 2017). A bilinear table holds only products. A good fusion should hold the additive terms too, so that the model can still learn the single-signal effects of Chapter 3. The trick is small: append a constant 1 to each vector before taking the outer product.

Work it by hand. Let xA = [2, 3] and xB = [4, 5]. Step one: append the 1s, giving [2, 3, 1] and [4, 5, 1]. Step two: take the outer product, a 3 × 3 table whose rows are [8, 10, 2], [12, 15, 3] and [4, 5, 1]. Step three: read it. The top-left 2 × 2 block, 8, 10, 12 and 15, holds the bimodal products. The last column, 2 and 3, is xA times the appended 1, so it is xA itself: the unimodal terms. The last row, 4 and 5, is xB itself. The corner, 1, is the constant. One table holds all three kinds of term, just as the regression of slide 15 did. A weight w then maps the flattened table to the output.

Count the table

Each cell is one number the next layer must weight. Grey is the constant, blue and purple are single-signal terms, warm cells are pair products. Choose the vector sizes, then add a third signal.

64
64

Exact counts for an output of 512 numbers. The grid shows at most 14 cells per side, with the real counts in the read-out. Sizes 8, 32, 64, 256 and 768 are illustrative choices; 768 is a common encoder size.

Set both sizes to 768 and the table has 769 × 769 = 591,361 cells. A weight matrix to an output of 512 numbers then holds about 303 million parameters. Switch on a third signal and the table becomes a cube of 769 × 769 × 769, about 455 million cells, and the weights reach about 233 billion. The slide says it plainly: but the weight matrix may end up quite large.

FusionFormulaPairs it can seeSize of z
Additivew1xA + w2xBNoned
Element-wise productw(xA × xB)Same position onlyd
Bilinearw(xATxB)Every pairdA × dB
Tensor fusionw([xA, 1]T[xB, 1])Every pair, plus each signal alone(dA + 1)(dB + 1)
python# x_a [B, d_a], x_b [B, d_b]; h = output size
ones = torch.ones(x_a.shape[0], 1)
a1, b1 = torch.cat([x_a, ones], 1), torch.cat([x_b, ones], 1)   # [B, d_a+1], [B, d_b+1]
Z = torch.einsum('bi,bj->bij', a1, b1)                              # [B, d_a+1, d_b+1] the table
out = Z.flatten(1) @ W                                                  # W: [(d_a+1)*(d_b+1), h]

Failure mode: the third modality that does not fit in memory

The symptom: training runs out of memory the moment a third signal is added, though the data is small. The measurement: multiply the sizes. Each new signal multiplies the table by its size plus one, so the count grows like a power. The fix: do not build the table at all. The next chapter shows how to get the same output from small pieces.

Why do tensor fusion networks append a constant 1 to each vector before the outer product?

Chapter 5

Low-Rank Fusion: The Same Answer from Small Pieces

Replace one giant weight table with a few thin ones, and compute the fused output without ever building the table

A wall of tiles in a hotel lobby looks like it needs a thousand different tiles. Look closer: it is one pattern, repeated. You could write the pattern down on a card and a short list of where to stamp it. The card is much smaller than the wall, yet it gives the same wall.

Chapter 4 ended with a problem: a table of 591,361 cells and 303 million weights for just two signals. Slides 19 to 22 give the fix, low-rank fusion from Liu and coauthors (“Efficient Low-rank Multimodal Fusion with Modality-Specific Factors”, ACL 2018). The slides also cite LoRA, the low-rank adaptation of large language models (Hu et al., ICLR 2022), because both use the same trick.

The trick: a table that is a sum of outer products

Recall tensor fusion. Call the appended vectors ẑA (length dA + 1) and ẑB (length dB + 1). The table is their outer product Z, and the weight W is a block of numbers of shape [dA + 1, dB + 1, h]. Each output number hk is a weighted sum over every cell of Z.

The low-rank idea, slide 19 step 1, is to decompose the weight. Instead of storing W cell by cell, store r pairs of thin matrices, WA(i) with shape [dA + 1, h] and WB(i) with shape [dB + 1, h], for i = 1 to r. Build W from them: the cell at row a, column b, output k is the sum over i of WA(i)[a, k] times WB(i)[b, k]. The number r is the rank: how many such pairs you keep. A rank-one piece is a single pair. A low-rank weight is a short sum of them, the pattern card for the wall.

Slides 20 and 21 then do the two moves that matter. Step 2 decomposes the input table too, as an outer product that never needs to be written. Step 3 rearranges the sum. Put the two decompositions into the formula for hk and the double sum over cells splits into two single sums, one over a and one over b:

h
The fused output, a vector of h numbers.
∑i=1r
Add up r small pieces, one per rank.
WA(i)T ẑA
Project signal A with the i-th thin matrix: a vector of h numbers.
WB(i)T ẑB
The same for signal B.

The circle ∘ means multiply element by element. So each rank projects A, projects B, and multiplies the two projections. That is all. The table Z is never built, and neither is W. This is slide 22: the final form is a short sum of products of two thin matrix-vector multiplications.

Same output, small pieces

Top: the output of a tiny fusion computed two ways, by building the full table, and by the low-rank formula. Bottom: how many weights each needs at real sizes. Change the rank, or draw new random weights.

4

Top bars: a toy with 6 + 1 numbers per signal, 4 outputs and random seeded weights, computed live in your browser in both ways; the largest difference is rounding error. Bottom bars: exact counts for 768 numbers per signal and 512 outputs, as in Chapter 4. Real systems choose r by validation accuracy; r here is a free slider.

A worked count

Take 768 numbers per signal and 512 outputs. The full table weight has 769 × 769 × 512 = 302,776,832 numbers. At rank 4, each signal needs 4 thin matrices of 769 × 512, so the total is 4 × (769 + 769) × 512 = 3,149,824 numbers. That is about 96 times fewer. The cost to compute one output falls the same way: instead of 303 million multiply-adds over the table, you do about 3.1 million for the two projections plus 2,048 for the element-wise products and the sum.

The price is expressiveness. A weight built from r pieces can capture r “patterns” of interaction, not every possible one. The lecture’s point is that real interaction structure is often simple enough that a small r is enough, so the trade is a good one. The slider shows the other side: at small r the parameter bar is tiny, and you can move toward the table only by paying for more rank.

LoRA uses the same shape of idea for a different job. To tune a big frozen weight, it learns a change that is the product of two thin matrices, so that the change has low rank. In fusion, the weight itself is the low-rank object. In both cases, thin matrices replace one fat one.

python# z_a [B, d_a+1], z_b [B, d_b+1]; factors: wa [r, d_a+1, h], wb [r, d_b+1, h]
fa = torch.einsum('bd,rdh->rbh', z_a, wa)       # [r, B, h] one projection per rank
fb = torch.einsum('bd,rdh->rbh', z_b, wb)       # [r, B, h]
h  = (fa * fb).sum(dim=0)                         # [B, h] multiply per rank, then sum over rank

Failure mode: rank too small

The symptom: low-rank fusion scores below the unimodal baseline on a task known to have synergy. The measurement: sweep r over 1, 2, 4, 8 and plot validation accuracy; if it is still climbing at the largest r you tried, the pieces are too few. The fix: raise r until the curve flattens, and stop there, since each extra rank costs another pair of thin matrices.

What does low-rank fusion avoid doing?

Chapter 6

Gated and Modality-Shifting Fusion

Let a small network look at both signals and decide how much of each to let through, or how far one should nudge the other

You are on a video call with two colleagues who both report the same number. Alice has a clear line. Ben’s microphone crackles. You do not average their words equally. Without thinking, you lean on Alice and half-ignore Ben. The next minute, Alice’s line drops and you switch. Your trust is not fixed. It changes with what you hear, moment by moment.

Slide 23 puts that behaviour into a fusion layer. In the plain additive fusion of Chapter 4, the weights of the two signals were fixed after training. A gated fusion makes the weights depend on the input.

The gate

A gate is a small neural network that outputs a number between 0 and 1 (or a vector of such numbers). You multiply a signal by its gate. If the gate is 1, the signal passes. If the gate is 0, the signal is blocked. Between the two, it is turned down. The key point is what the gate reads: both signals. The lecture writes the additive example this way, citing the Gated Multimodal Unit of Arevalo and coauthors (ICLR workshop 2017):

z
The fused representation.
gA(xA, xB)
The gate for A. It reads both signals and returns how much of A to pass.
xA
The first signal, for example the words.
gB(xA, xB)
The gate for B, also reading both signals.
xB
The second signal, for example the voice.

The slide adds that gA and gB can be seen as attention functions, and that the gate output can be one weight for the whole modality. Slide 24 plays with the two names. Seen negatively, a gate is a “neural network designed to mask unwanted signal from propagating forward”. Seen positively, it is attention: “designed to select preferable signal to move forward”. The maths is the same; the view differs.

Soft or hard

Slide 24 also asks what should it be? The gate looks at an input and chooses among: the target modality, the other modality, or all modalities. A soft attention gate returns fractions, such as 0.9 and 0.1, so every signal contributes a little. Its derivative (the gradient used in training) is easy to compute. A hard attention gate returns a pure choice: 1 for one signal and 0 for the other. A choice has no smooth slope, so the derivative is harder; the slide gives reinforcement learning as the usual route, from Chen and coauthors (ICMI 2017). The cost of softness is a blend that may carry some noise from the weaker signal. The cost of hardness is a harder training problem.

Gate the signals

Words and voice both estimate a true sentiment of +1. The voice is sometimes corrupted. Bars show the typical error, so shorter is better. Drag how often the voice is corrupted. In the second mode, a voice cue shifts a word.

50%

Toy simulation, labelled illustrative: 600 seeded trials; words have noise 0.6; the voice has noise 0.3 when clean and 2.0 when corrupted. The gate is told which case it is in, standing in for what a trained gate learns; the soft gate weights each signal by the inverse of its noise variance, the hard gate picks the cleaner signal. The shift mode is a one-line toy: z = 0.10 + gate × tone × 0.8, with the gate fixed by hand.

Try 50% corruption. Equal averaging suffers whenever the voice is bad, because it hands the broken signal half the vote. The soft gate gives the broken voice only a small weight, and its error falls below either signal alone. The hard gate does well too, but it throws away the small gain from blending when both signals are good.

A worked number

On a corrupted clip, words have noise 0.6, so variance 0.36, and the voice has noise 2.0, so variance 4. Step one: the ideal weight for a signal is its inverse variance divided by the sum of both inverse variances. Equivalent: gA = 4 / (0.36 + 4) = 0.917 and gB = 0.083. Step two: the fused variance is 0.36 × 4 / 4.36 = 0.330, so the noise is 0.57. Step three: equal weights of 0.5 give variance 0.25 × 4.36 = 1.09, so the noise is 1.04. The gate cuts the typical error by almost half on those clips, with the same two inputs.

Modality-shifting fusion

Slide 25 applies a gate for a different job. Choose one primary modality, here language, and treat the others (sound and vision) as secondary. The primary signal passes through unchanged except for a shift: a vector computed from the secondary signals and added to it, scaled by a gate. The slide’s example is the word “wow”. Said in a delighted voice, its representation moves toward the positive side. Said flatly with a frown, it moves to the negative side. The word has no single meaning, so its vector must move. The slide cites Wang and coauthors (“Words Can Shift”, AAAI 2019) and Rahman and coauthors (ACL 2020), who built the idea into a large pre-trained transformer.

What makes a gate different from the fixed weights of ordinary additive fusion?

Chapter 7

Nonlinear Fusion, and the EMAP Test

Ask a big network to fuse two signals, then take out its interactions and see whether its answers change

A student shows a model that answers questions about pictures. “How many dogs are in the photo?” The model says “two”. It is right. The student says: “it looks at the picture and the question together.” But maybe the model never looked at the picture. The word “how many” alone makes “two” a fair guess, and in this dataset it is right more often than not. How would you tell?

That is the puzzle of slides 26 to 28 and of the assigned reading, Does my multimodal model learn cross-modal interactions? It’s harder to tell than you might think! by Jack Hessel and Lillian Lee (EMNLP 2020).

Nonlinear fusion

Slide 26 states the most general design. Nonlinear fusion is ŷ = f(xA, xB), where f can be a neural network or any nonlinear model, with output in Rd (a vector of d real numbers). The slide notes that this is early fusion: join the inputs, ŷ = f([xA, xB]). A neural network with enough hidden units can represent the product terms of Chapter 3 and much more. So the question on the slide is a fair one: but will our neural network learn the nonlinear interactions? Being able to is not doing it. Training finds some solution, and the easiest one may use each signal separately.

The EMAP test

Hessel and Lee propose a diagnostic named EMAP, for empirical multimodally-additive function projection. The idea is to take the trained model f and forcibly remove its cross-modal interactions, leaving only the part that adds the two signals’ effects. If the score does not change, the model was not using interactions. The name unpacks as follows. A function is multimodally additive if it has the form fA(xA) + fB(xB) + a constant. A projection finds the additive function closest to f. Empirical means the averages come from your own data.

The slides write the projection as the average of f over one signal plus the average over the other, minus the overall mean. In symbols, f̃(xA, xB) = ExB[f(xA, xB)] + ExA[f(xA, xB)] − E[f]. Read it in words. To predict for a picture, average the model’s output over every question it could be paired with. To predict for a question, average over every picture. Add the two and subtract the grand average once so that it is not counted twice.

Do it on a 2 × 2 table. Let f(0,0) = 0, f(0,1) = 1, f(1,0) = 1, f(1,1) = 4. Step one: the row averages are 0.5 for xA = 0 and 2.5 for xA = 1. Step two: the column averages are 0.5 and 2.5 for xB = 0 and 1. Step three: the grand mean is (0 + 1 + 1 + 4) / 4 = 1.5. Step four: f̃(0,0) = 0.5 + 0.5 − 1.5 = −0.5, f̃(0,1) = 0.5 + 2.5 − 1.5 = 1.5, f̃(1,0) = 1.5, f̃(1,1) = 3.5. Step five: the leftover f − f̃ is +0.5, −0.5, −0.5, +0.5. That leftover is all the interaction the table had.

Project out the interaction

Left: the model’s output over an 8 × 8 grid of inputs (warm is positive, blue is negative). Middle: its EMAP, the additive part. Right: what EMAP removed. Slide the interaction share and watch the right panel.

30%

Toy, labelled illustrative: f(a, b) = (1 − s)(a + b)/2 + s · a · b on an 8 × 8 grid, where s is the interaction share. EMAP is computed exactly from the grid (row means plus column means minus the grand mean). Here the model is perfect, so the test shows how much of the correct answer depends on the interaction.

At 0% the right panel is empty: the function is purely additive, and EMAP changes nothing. At 100% the middle panel is blank, because a pure product has no additive part. In between, the sign of f and of its EMAP disagree on only a handful of cells: 6 of 64 at 30% and at 50%, where the product outvotes the sum. At 100% the count reads 32, because the EMAP is zero everywhere and half the cells are positive. A model whose behaviour sits near the left of the slider would not show up as better than an additive model on a test set.

What the paper found

Slide 28 tabulates seven image-and-text tasks from the paper. In many cases, removing the cross-modal interactions caused little or no loss. So a gain from a bigger model, when present, often cannot be credited to cross-modal interactions. Slide 28 shows three of the columns: for the best model against its EMAP, 91.3 against 91.1, 74.4 against 74.2, and 81.5 against 81.3. A plain linear model scores 90.4, 72.8 and 80.9 on the same columns. The slide’s own annotation is that the differences are small, and that additive fusion is always a good baseline.

The authors’ recommendation is practical: report not just the unimodal baselines but also the EMAP of your best model. You pay almost nothing, because EMAP only reruns the trained model on mixed-up pairs.

python# xa [N, d_a], xb [N, d_b]; model(...) returns logits [..., C]
F = model(xa[:, None, :], xb[None, :, :])               # [N, N, C], F[i, j] = f(xa_i, xb_j)
emap = F.mean(1) + F.mean(0) - F.mean((0, 1))       # [N, C]: row mean + column mean - grand mean
acc_full = (model(xa, xb).argmax(-1) == y).float().mean()
acc_emap = (emap.argmax(-1) == y).float().mean()      # a tiny gap means few interactions used

Failure mode: crediting a model with an interaction it never used

The symptom: a paper shows a large gain from a cross-attention model and says it “captures cross-modal relations”. The measurement: compute the EMAP accuracy. If it is within a few tenths of a point of the full model, the model works additively. The fix: say so, and look for the gain elsewhere, such as a better encoder or more data. Pair this with the PID estimate of Chapter 1: PID asks whether the data holds synergy, and EMAP asks whether the model used any.

A fancy fusion model scores 81.5, and its EMAP scores 81.3. What is the most direct reading?

Chapter 8

Beyond Additive: Simple Terms First, Then the Residual

Make a model explain what it can with single signals, then pairs, then triples, so that each level fixes only what the level below missed

A detective arrives at a crime scene with three witnesses. A careful one starts with what each witness says alone. Only then does she ask which two accounts agree or clash. Only after that does she look for something that appears only when all three stories are laid together. She does not start with the grand theory. If a simple explanation covers the facts, she stops.

Slide 29 asks a fusion model to work like her. Its heading is the idea: prioritize simpler interactions. The method is from Wortwein and coauthors, “Beyond Additive Fusion: Learning Non-Additive Multimodal Interactions” (EMNLP 2022), and the lecture calls it multimodal residual optimization.

Why a flexible model needs a push

Chapter 7 showed the trouble. A big network that can learn any interaction may learn none, because the additive route is easier. The reverse trouble exists too. A big network may explain everything with one tangled, hard-to-read function of all signals, and nobody can say which signal did what. Both problems come from letting the model choose its own route.

The fix is to give the route in advance. The slide splits the prediction into three parts, ordered by how simple they are: a unimodal part ŷuni, which sums the effect of each signal alone (it is additive); a bimodal part ŷbi, which depends on pairs of signals, such as (xA, xB); and a trimodal part ŷtri, which depends on all three together. The final prediction is the sum of the three.

ℒ(y, ŷuni)
The first loss: how well the unimodal part alone matches the true answer y.
ℒ(y − ŷuni, ŷbi)
The second loss. The target is now the residual, what the unimodal part left over. The pair terms must explain only that.
ℒ(y − ŷuni − ŷbi, ŷtri)
The third loss. The three-way part may use only what is still left after both lower levels.

A residual is the part of the answer not yet explained. Each level is trained to match the residual of the levels below it. A higher-order term can therefore earn its place only by explaining something the simpler terms could not. If pairs add nothing, ŷbi stays near zero, and you learn that the task is mostly additive.

Peel off the residual

Three signals a, b and c, and an answer that mixes single, pair and three-way effects. Bars show how much of the answer each stage explains in total, since each stage fits what the stage before left behind. Set how strong the pair and three-way effects are.

1.00
0.80

Toy, labelled illustrative: 300 seeded points of three signals between −1 and +1, and an answer y = a − 0.6b + 0.4c + s2(ab + bc) + s3abc plus small noise. Each stage is an exact least-squares fit to the residual of the stage before. The stage terms are a, b, c and a constant; then ab, ac and bc; then abc.

At the default settings, the single terms alone explain 68% of the answer. Adding the pair terms lifts it to 96%. The three-way term finishes at 99%. Now slide the pair strength to zero and the three-way strength to zero: the first bar jumps to 98% and the other two have nothing to add. The staged fit reads like a report. It says: here is what the signals do alone, here is what pairs add, here is what only the triple adds.

A worked example, two stages

Suppose the true answer for one clip is 2.0, and the unimodal part predicts 1.2. Step one: the first loss compares 1.2 to 2.0, an error of 0.8. Step two: the residual is 2.0 − 1.2 = 0.8. Step three: the bimodal part is trained against 0.8, not against 2.0, and suppose it predicts 0.6. Step four: the new residual is 0.8 − 0.6 = 0.2. Step five: the trimodal part is trained against 0.2. The total prediction is 1.2 + 0.6 + 0.2 = 2.0 when all three succeed. Each part has a small, clear job.

The lecture states the benefit as a hierarchy of interactions: unimodal (additive), bimodal (non-additive) and trimodal (non-additive). The result is a model that is easier to read, and one that uses a high-order interaction only when the data demands it. It connects directly to Chapter 7: you can compare the size of ŷbi and ŷtri with ŷuni to see how much of the answer is truly non-additive.

python# y [B]; heads see: uni -> each signal alone, bi -> pairs, tri -> all three
y_uni = uni_head(x_a, x_b, x_c)                 # [B] sum of per-signal terms
y_bi  = bi_head(x_a, x_b, x_c)                  # [B] terms over (a,b), (a,c), (b,c)
y_tri = tri_head(x_a, x_b, x_c)                 # [B] one joint term
loss = mse(y_uni, y)                                    \
     + mse(y_bi,  (y - y_uni).detach())                 \
     + mse(y_tri, (y - y_uni - y_bi).detach())          # detach: each level fits the residual, not the others
y_hat = y_uni + y_bi + y_tri                    # [B]

Failure mode: the higher level soaks up everything

Without the ordering, a flexible head can take the whole answer and leave the simple heads idle. The symptom: unimodal and bimodal outputs are near zero while the trimodal head matches y. The measurement: the share of the answer explained by each stage, as the bars above show. The fix: keep the ordered losses, and detach the targets so that a higher level cannot pull a lower one off course.

In multimodal residual optimization, what is the target for the bimodal part?

Chapter 9

Fusion Inside Large Models

See how the ideas of this lecture grow up: one sequence of tokens, attention as the interaction, and experts split by modality

Chapters 3 to 8 fused two or three signals, each a small vector. Real systems today read a page that mixes a paragraph, a photo, another paragraph and a chart. There is no neat list of modalities to wire together. Everything must go into one model, in the order it appears.

Slide 30 closes the lecture with a map. It repeats the definition of fusion and lays methods along a line that runs from homogeneous modalities to heterogeneous modalities. From left to right the names read: late fusion, additive fusion, multiplicative fusion, tensor fusion, polynomial fusion, gated fusion, modality-shift fusion, nonlinear fusion, very early fusion, dynamic early fusion, heterogeneity-aware fusion, then two goals, improving optimization and improving robustness. We have met seven of the first eight. Polynomial fusion, which adds powers of each signal as extra product terms, is only named here. The rest belong to the next lecture and to the four model papers on this week’s reading list, which push fusion all the way to the input.

Very early fusion: one sequence, one transformer

The idea is simple to say. Turn every modality into tokens (whole-number codes from a fixed list) and put them all in one sequence. A single transformer then reads the sequence. No encoder-per-modality, no separate fusion layer. The fusion happens in attention: each token computes a score against every other token as the dot product of a query and a key. That score is a product of one token’s numbers with another’s, summed. It is exactly the bilinear form of Chapter 4, applied to every pair of tokens, including every text-token and image-token pair. Attention is a learned, input-dependent version of the bilinear and gated fusion you already know.

Count what that costs. A 512 × 512 image becomes 1,024 image tokens (Chameleon’s tokenizer). Add a 20-word question and the sequence has 1,044 tokens. Causal attention means each token looks only at itself and the tokens before it. It compares each token with itself and every earlier one: 1,044 × 1,045 / 2 = 545,490 pairs in each layer, of which more than 99% involve an image token. A picture weighs as much as a thousand words in the sequence, and that is a real, countable cost.

The four readings

Chameleon (arXiv 2405.09818; our lesson: Chameleon). A family of early-fusion token-based mixed-modal models that understand and generate images and text in any order. A tokenizer cuts a 512 × 512 image into 1,024 tokens from a codebook of 8,192 codes, which join the roughly 65,000 text tokens in one vocabulary. The paper describes a stable training approach from the start, an alignment recipe, and an architecture tuned for this setting. It reports strong image-captioning results and beats Llama-2 on text-only tasks while generating images in the same model.

Kosmos-2 (arXiv 2306.14824; our lesson: Kosmos-2). A multimodal language model that grounds text in the picture. It writes a mention of an object as a Markdown link, “[text span](bounding boxes)”, where the box is a short run of location tokens. The authors built a large set of grounded image-text pairs called GrIT to train it. It handles referring to objects and describing regions, plus the usual captioning and question tasks. Here the fusion carries where, not only what.

MM1 (Springer chapter; our lesson: MM1). A study of what matters when building a multimodal language model. Its design lessons: a careful mix of image-caption, interleaved image-text and text-only data is crucial for strong few-shot results; the image encoder, with the image resolution and the number of image tokens, matters a lot; the vision-language connector matters comparatively little. The recipe scales to models up to 30 billion parameters, dense and mixture-of-experts.

MoMa (arXiv 2407.21770; our lesson: MoMa). A mixture of experts splits a layer’s feed-forward block into several experts, and a learned router sends each token to some of them. MoMa makes the groups modality-aware: some experts see only text tokens, others only image tokens, with learned routing inside each group. Try the device below.

Route the tokens

Top: ten tokens, words in blue and image patches in purple, going to experts. Bottom: the compute each design needs to reach the same loss as a dense model of equal cost. Choose a design.

The savings are the MoMa paper’s (1.4B model, 1 trillion training tokens, measured by pre-training loss): the standard expert-choice MoE with 8 mixed-modal experts saves 3x overall (3x text, 2.8x image); MoMa with 4 text and 4 image experts saves 3.7x overall (2.6x text, 5.2x image); adding mixture-of-depths saves 4.2x (3.4x text, 5.3x image) but hurts causal inference through router sensitivity. The bar is 1 divided by the savings. The routing lines are an illustrative, fixed pattern, not learned routes.

The savings come from a simple fact. Image tokens and text tokens have very different statistics, so letting each group have its own experts spares the model from using one expert for both. In the paper, the 3.7 times overall saving splits unevenly: 2.6 times on text and 5.2 times on images. Reading the bars, an image token needs about 19% of the dense compute, a text token about 38%.

Failure mode: one modality dominates the sequence

The symptom: the model captions well but its text answers get worse, or training diverges early. The measurement: count tokens and compute by modality, as with the 99% above, and watch the loss for each modality separately. The fix is at the architecture level, and each paper shows one: a stable training recipe for the shared vocabulary (Chameleon), data mixes that keep text in the diet (MM1), and separate experts per modality (MoMa).

In a single-sequence early-fusion model, where does the cross-modal interaction mainly happen?

Chapter 10

The Field Guide

Keep the numbers, the terms, a checklist and the core code on one page, and pick a fusion for your own problem

The lecture in one paragraph. Fusion builds one joint representation from several signals. What the signals tell you about a task splits into redundancy, uniqueness and synergy. Early fusion joins inputs, late fusion joins verdicts, and only early or intermediate fusion can see synergy. A product term models an interaction; tensor fusion keeps every pair of terms and costs a table that grows like a power; low-rank fusion gets the same output from thin factors. Gates and shifts make the weights depend on the input. Whether a trained network used any interaction is a question you can answer: project it onto its additive part with EMAP. And a model that must explain itself can fit singles first, then pairs, then triples.

Pick a fusion for your problem

Choose a recipe

Switch on each statement that is true for your task. The recipe below follows the chapters of this lesson.

A rule of thumb that restates this lesson, not a result from a paper. Always run the unimodal baselines and the EMAP of your best model first, whatever the recipe says.

The numbers

WhatValueSource
Book-review regression, slope in smiling0.68 for non-critics, 1.97 for critics (0.68 + 1.29)Lecture, slides 12 to 14
Exclusive-or information0 bits alone, 1 bit togetherComputed in Chapter 1
Tensor table, 768 and 768769 × 769 = 591,361 cells; 302,776,832 weights at 512 outputsComputed in Chapter 4
Low-rank at rank 43,149,824 weights, about 96 times fewerComputed in Chapter 5
Ideal gate on a corrupted clip0.917 words, 0.083 voice; error 0.57 against 1.04 for an equal mixWorked in Chapter 6, illustrative
Best model against its EMAP91.3 vs 91.1, 74.4 vs 74.2, 81.5 vs 81.3Lecture, slide 28 (Hessel and Lee)
Image as tokens512 × 512 gives 1,024 tokens from 8,192 codesChameleon
One image plus 20 words1,044 tokens; 545,490 causal attention pairs per layerComputed in Chapter 9
MoMa savings at 1.4B3.7x overall; 2.6x text; 5.2x imageMoMa paper

The terms

TermIn one line
ModalityOne kind of signal: words, sound, pixels.
Redundancy, uniqueness, synergyInformation both signals give, only one gives, or only the pair gives.
Early, intermediate, late fusionJoin the inputs, the hidden features, or the predictions.
Interaction termA product such as w3xAxB that makes one input’s effect depend on the other.
Tensor fusionThe outer product of the two vectors, each with a 1 appended.
Low-rank fusionA short sum of products of thin projections instead of the full table.
GateAn input-dependent weight between 0 and 1 on a signal.
EMAPThe additive projection of a trained model: row mean plus column mean minus grand mean.
Residual optimizationFit singles, then pairs to what is left, then triples to what is left.

The checklist

  1. Name the signals. Say whether each is raw or already encoded, and what shape it has.
  2. Run each signal alone. The unimodal scores tell you the uniqueness before any fusion does.
  3. Run late fusion. It is cheap, and it cannot see synergy, so a gap above it is the synergy you can buy.
  4. Add the product term. If a linear model with a product beats the one without, the pairing matters.
  5. Count before you build. Multiply the sizes. If the table does not fit, use low-rank factors or attention.
  6. Gate when quality varies, and train with input dropout if inputs can go missing.
  7. Run EMAP on your best model. Report it next to the unimodal baselines.

The core, as code

pythonimport torch, torch.nn as nn

class Fusion(nn.Module):
    """x_a [B, d_a], x_b [B, d_b] -> z [B, h]"""
    def __init__(self, d_a, d_b, h, kind="lowrank", rank=4):
        super().__init__(); self.kind = kind
        if kind == "additive":
            self.wa, self.wb = nn.Linear(d_a, h), nn.Linear(d_b, h)
        elif kind == "tensor":
            self.w = nn.Linear((d_a + 1) * (d_b + 1), h)            # the big table
        elif kind == "lowrank":
            self.wa = nn.Parameter(torch.randn(rank, d_a + 1, h) * 0.05)
            self.wb = nn.Parameter(torch.randn(rank, d_b + 1, h) * 0.05)
        else:                                              # gated
            self.pa, self.pb = nn.Linear(d_a, h), nn.Linear(d_b, h)
            self.gate = nn.Linear(d_a + d_b, 2)

    def forward(self, x_a, x_b):
        one = x_a.new_ones(x_a.shape[0], 1)
        a1, b1 = torch.cat([x_a, one], 1), torch.cat([x_b, one], 1)
        if self.kind == "additive": return self.wa(x_a) + self.wb(x_b)
        if self.kind == "tensor":   return self.w(torch.einsum("bi,bj->bij", a1, b1).flatten(1))
        if self.kind == "lowrank":
            fa = torch.einsum("bd,rdh->rbh", a1, self.wa)
            fb = torch.einsum("bd,rdh->rbh", b1, self.wb)
            return (fa * fb).sum(0)
        g = self.gate(torch.cat([x_a, x_b], 1)).softmax(-1)         # [B, 2], reads both signals
        return g[:, :1] * self.pa(x_a) + g[:, 1:] * self.pb(x_b)

Where this sits in the field, and where to go next

Chapters 3 to 8 are the classical toolkit for fusing a few signals. The modern systems of Chapter 9 fuse tokens inside one transformer, and the toolkit becomes the way to read them. For related lessons here: Fusion Architectures and the JDL Model compares early, intermediate and late fusion for sensors; microCLIP and microVLM build the vision-language side; CS224N Lecture 17, Multimodality and Multi-Modal Foundation Models give other course views; and the paper lessons for this week are Chameleon, Kosmos-2, MM1 and MoMa. The two measuring papers are taught in Chapters 1 and 7: Quantifying and Modeling Multimodal Interactions and Does my multimodal model learn cross-modal interactions?

The takeaway. Before you build a fusion, find out which kind of information you have. If the signals only repeat or work alone, a simple sum is enough. If the answer lives in the pair, build the product on purpose, keep its cost under control, and test with EMAP that the model really used it.

Now explain, from memory, why a clerk who averages two verdicts can never solve an exclusive-or, and how a product term, a tensor table and a low-rank factor each fix it. Then go back to the sentiment plane at the top, set the pairing to 100%, and say aloud why only the product gets the clips right.

This lesson re-teaches MIT MMAI (Multimodal AI), Spring 2026, Lecture 4.1, Multimodal Fusion, by Paul Liang of the MIT Multisensory Intelligence group, in our own words and devices; some of the course’s slides were co-developed with Louis-Philippe Morency for CMU 11-777. See the course schedule and the lecture slides. Toy numbers are marked illustrative where they appear.

Your team’s cross-attention fusion model scores 0.3 points above its own EMAP. What do you tell the team?
MIT MMAI Lecture 4.1: Interactions and Fusion Models
The course site · Lecture slides · PID paper · EMAP paper · Back to Gleams