Two signals can say more together than apart. A sum of them cannot see it. A product can. Slide the pairing and watch four fusion rules score the same 200 clips.
Learn how a model combines words, voice, pictures and more into one decision, and how to test whether it truly uses the pairing.
Pick a rule, then drag the pairing slider all the way right. Watch the words-only rule and the plain sum drop to a coin flip while the product stays right. Then we build, piece by piece, early and late fusion, the product term, tensor and low-rank fusion, gates, and the test that tells you whether a model learned any interaction at all.
You need the idea of a number line and a slope, and a little Python for the code. We build the rest from zero.
Based on Lecture 4.1, Multimodal Fusion, of MIT MMAI, Multimodal AI (Spring 2026), taught by Paul Liang of the MIT Multisensory Intelligence group; some of its slides were co-developed with Louis-Philippe Morency for CMU 11-777. Lecture slides. Every idea is re-taught here in our own words and devices.
Toy data, labelled illustrative: 200 clips with a seeded words tone and voice tone between −1 and +1. At 0% the true answer is “yes” when the words are positive. At 100% it is “yes” when words and voice disagree, the sarcasm case of the lecture’s slide 7. In between, that share of clips follows the pairing rule. Five percent of the answers are flipped as label noise. The words and voice rules are the best cut on one tone, from 13 cut points; the sum rule is the best weighted sum of both tones, cut at zero, from 36 directions; the product rule is the best cut on words × voice.
Chapter 0
See why one signal alone can mislead, and what a fusion model must learn to do about it
A friend says, “Great job.” Read on a page, the words are praise. Now hear it: flat voice, eyes rolled to the ceiling. The same two words now mean the opposite. Neither signal told you that alone. The words said “good”. The voice said “not quite”. The meaning lived in the pair.
This is the problem of the whole lecture. A modality is one kind of signal: the words of a sentence, the sound of a voice, the pixels of a face. Most real tasks give a model several at once. The model must turn them into one decision, and the hard part is that the best decision often cannot be read off any single signal.
The lecture calls this job multimodal fusion. Its definition fits on one line: learn a joint representation that models the multimodal interactions between individual elements of different modalities. Take that apart. A representation is a list of numbers that stands for something, the way a map pin stands for a city. Joint means one list for both signals, not one list each. An interaction is information that exists only because two signals meet. Fusion is the machine that makes the joint list.
The slide draws the machine as a funnel. Two coloured bars go in, one bar comes out. How different the two inputs are decides how hard the funnel is to build. Hold that thought, because the slide splits it into two cases.
In the first case the lecture calls them abstract modalities. Each signal has already been through an encoder, a neural network that turns raw input into a vector (a list of numbers). A vision model such as a CNN or a ViT does this for pictures. A language model such as Word2vec or BERT does it for words. After encoding, both signals are lists of numbers of similar kind. The slide calls this closer homogeneity. It adds one practical note: the encoders can be trained together with the fusion network, or they can be pre-trained and frozen.
In the second case, raw modalities, nothing has been encoded yet. A picture is a grid of continuous colour values. A sentence is a short list of whole-number word codes. These two share almost nothing in shape, scale or meaning. The slide calls this heterogeneity, and it sits at the far end of a line that runs from homogeneous to heterogeneous. Try both cases below and watch the shapes.
Each box is one tensor (a block of numbers), with its shape written under it. Choose what enters the fusion funnel: raw signals, or signals an encoder has already turned into vectors.
Illustrative shapes: a 224 × 224 colour picture, a 12-word sentence, 768-number vectors from a ViT-style and a BERT-style encoder, and a 512-number joint vector. The lecture draws the idea (slides 4 and 8) without fixed sizes; the arithmetic here is exact.
Raw, the picture is 3 × 224 × 224 = 150,528 real numbers on a grid. The sentence is 12 integers, each a position in a word list. A fusion network cannot add these, multiply them or line them up. After the encoders, both are 768 numbers. Now a layer can combine them element by element. This is why most of the lecture works on the abstract case first, and why the encoders matter so much.
Slide 2 holds the week’s business. The project proposal is due the day of the lecture, and Paul Liang offers feedback from 4 to 5 pm and wants to meet every group at least once that day or on Thursday. The course gives out compute credits: 40 credits of $50 for Kimi and 40 of $40 for other providers. Homework 2 was released the week before and is due Wednesday, March 4.
Slide 3 lists four parts. First, the basics of multimodal interactions and fusion. Second, early, intermediate and late fusion. Third, multiplicative and dynamic fusion. Fourth, quantifying fusion, which means measuring how much of an interaction a model uses. The deck credits Louis-Philippe Morency for slides co-developed for CMU 11-777.
The assigned readings come in two groups. Two papers give tools for measuring interactions: the information decomposition framework of Liang and coauthors (arXiv 2302.12247), taught in Chapter 1, and the EMAP test of Hessel and Lee (EMNLP 2020), taught in Chapter 7. Four papers show fusion inside large models: Chameleon, Kosmos-2, MM1 and MoMa, summarised in Chapter 9 with links to their own lessons.
The road below follows the slides in order. Each chapter opens with a small scene and ends with a one-idea quiz.
Chapter 1
Sort what two signals tell you about an answer into the part they share, the part only one holds, and the part that appears only in the pair
You skim film reviews and want to know whether each one is positive. Review one reads, “This movie is great!”, and the reviewer’s photo is laughing. Words and face say the same thing. Review two reads, “The movie does a good job developing…”, and the face is blank. The words carry the verdict. The face adds nothing. Review three reads “wowwww” over an eye-roll. Neither signal says “negative” alone, yet together they do.
Slides 5, 6 and 7 of the lecture draw exactly these three scenes. Each one shows a different way that two modalities can combine to give information about a task, and the lecture’s word for that is interactions: how modalities combine to provide information for a task.
Redundancy is information that both signals give. If you removed the face, the words would still tell you. Slide 5 draws it as the overlap of two circles that also overlaps the task. Uniqueness is information that only one signal gives. In review two the words are unique: the face stays neutral and says nothing. Slide 6 marks a region of modality 1 outside the overlap, and a matching one for modality 2. Synergy is information that exists only when you read both together. Slide 7 calls it emerging information from multimodal interaction. Sarcasm is the standard example, because the meaning comes from the clash.
Why does the split matter? Because each kind asks for a different model. If the information is redundant, one signal is enough and a second signal only adds robustness. If it is unique to one signal, the other signal is dead weight. Only synergy forces you to build a model that really looks at both at once. That last case is where fusion earns its keep.
To count them you need a unit. Information theory gives one. The mutual information I(A;Y) is how many yes-or-no questions’ worth of the answer Y you learn by seeing A. One bit means you can halve the uncertainty exactly once. The reading for this lecture, Quantifying & Modeling Multimodal Interactions: An Information Decomposition Framework by Paul Pu Liang and coauthors, uses the idea of partial information decomposition, PID for short. It splits the information that both signals give about the task into the four parts above.
The paper calls R, UA, UB and S the PID statistics of a multimodal dataset. It introduces two new estimators that scale to high-dimensional data, checks them on synthetic data where the true answer is known, and compares them with human labels on real benchmarks. It then uses them in four ways: to describe datasets, to describe what a trained model captured, to pick a model, and in three case studies in pathology, mood prediction and robotic perception.
Take the smallest synergy there is. Two bits, A and B, are each equally likely 0 or 1 and independent. The answer is Y = A xor B: yes when they differ. Step one: what does A alone tell you? If A is 0, Y is 0 or 1 with equal chance; if A is 1, the same. So I(A;Y) = 0 bits. Step two: the same holds for B, so I(B;Y) = 0. Step three: the pair fixes Y exactly, so I(A,B;Y) = 1 bit. The equation says 1 = R + UA + UB + S with the first three all zero, so S = 1. All the information is synergy, and no single-signal rule can beat a coin flip.
Now try it. The device holds four tiny worlds, each a table of how A, B and the answer relate. Slide the label noise and read the bars.
The bar is everything the pair tells you about the answer. Its colours show who holds it. Pick a world, then add label noise and watch the pieces shrink.
Toy tables, labelled illustrative, with exact arithmetic. Redundancy is taken as the smaller of the two single-signal informations, a simple choice that is exact for these four worlds; the paper’s estimators handle general, high-dimensional data. Noise flips the answer with the chosen probability.
Pick Synergy and leave the noise at 0%. The bar is all warm: 1.00 bit of synergy, and yet the best rule from the words alone is right on half the clips. Pick Redundant: the bar is all grey, and either signal alone gets every clip right. Add 10% noise and every bar shrinks, because a noisy answer carries fewer bits to share.
The symptom: a team builds a large cross-attention fusion model, and it scores the same as the unimodal text baseline. The measurement: estimate PID on the data. If S is near zero and UA is large, nearly everything lives in one signal. The fix: stop paying for fusion. Choose the simplest model that matches the information, and put the effort into a better encoder for the signal that carries it. The paper’s own case studies recommend models in just this way.
Chapter 2
Choose between mixing the signals before anyone decides and mixing the decisions after, and see what each choice can and cannot learn
A clinic sees a patient with two readings: a temperature, and the sound of a cough. The manager has two ways to set up the desk. In the first, a nurse writes both readings on one sheet and hands it to a single doctor, who decides. In the second, a fever specialist reads only the temperature and gives a verdict, a lung specialist listens to only the cough and gives a verdict, and a clerk averages the two.
The lecture’s slide 9 draws exactly these two desks. The first is early fusion: combine the inputs, then predict once. The second is late fusion: predict from each input alone, then combine the predictions. The names say when the signals meet, early in the pipeline or late.
Follow the data. In early fusion, the two inputs xA and xB are joined by a combination step, often just placing the two vectors end to end (concatenation). One model reads the joined vector and outputs one prediction. In late fusion, each input goes into its own model. Each model outputs its own prediction, written ŷA and ŷB. A small combiner, typically an average or a vote, turns the two predictions into the final ŷ.
There is a third design in between, intermediate fusion. Each input first goes through its own encoder (slide 8). The model then merges the hidden vectors in the middle, and later layers keep working on the merged result. Most modern systems are of this kind. The agenda on slide 3 names all three, early, intermediate and late, and most of this lecture is about the merge step inside the middle one.
Top: the two pipelines. Bottom: how often each design is right on 100 held-out clips. Slide the pairing to the right, then switch off the voice and see which design survives.
Toy data, labelled illustrative: 200 seeded clips with a words tone and a voice tone, labelled as in the hero; each design learns on 100 clips and is scored on the other 100 (the pairing share follows the sarcasm rule, the rest follows the words). Late fusion averages two single-signal predictors, each the share of yes answers in its half of the line. Early fusion looks up the share of yes answers in each of the four quadrants. A dropped voice is filled with a constant positive tone.
Leave the pairing at 100%. Late fusion sits near a coin flip (a little under or over 50%, since 100 clips is a small sample), and early fusion is close to perfect, held back only by the 5% label noise. The reason is the arithmetic of the clerk. Each specialist says “50 percent yes”, because words alone or voice alone cannot tell sarcasm. The average of two 50s is 50. The single doctor saw both readings on one sheet, so she could learn that a clash between them means yes.
Take eight clips, two in each quadrant of words and voice. Quadrant (+, +) holds two “no” clips. Quadrant (+, −) holds two “yes”. Quadrant (−, +) holds two “yes”. Quadrant (−, −) holds two “no”. Step one: the words specialist sees four clips with positive words, two yes and two no, so it says 0.5. For negative words it also says 0.5. Step two: the voice specialist says 0.5 in both halves too. Step three: the clerk averages 0.5 and 0.5 to get 0.5, a tie, so the best it can do is guess: 4 right out of 8 on average. Step four: the single doctor looks up each quadrant, and every quadrant is pure, so she gets 8 out of 8.
Now drop the voice. Late fusion loses one specialist, but the clerk still averages whatever arrives, so it keeps working as before. The early model was trained on sheets with two readings, and now one box is blank. Its lookup lands in the wrong quadrant half the time. Press the toggle and watch the early bar fall.
| Design | Where signals meet | Can it see synergy? | If one input goes missing |
|---|---|---|---|
| Early | At the input, before any prediction | Yes, if the model is expressive enough | Hurts: the model expects both |
| Intermediate | In the middle, between encoders and the head | Yes, through the merge step | Depends on how it was trained |
| Late | At the predictions, after each input decided | No: it sees only each side’s verdict | Degrades gracefully |
The lesson is not that early beats late. It is that late fusion cannot represent a pure interaction, and early fusion can but depends on both inputs being present. When the information is redundant or unique (Chapter 1), late fusion is cheap, modular and robust. When synergy matters, the merge must happen before the verdict.
python# shapes: x_a [B, d_a], x_b [B, d_b], C classes # late fusion: two heads, average the class probabilities p_late = (head_a(x_a).softmax(-1) + head_b(x_b).softmax(-1)) / 2 # [B, C] # early fusion: join the inputs, one model decides p_early = mlp(torch.cat([x_a, x_b], dim=-1)).softmax(-1) # [B, d_a + d_b] to [B, C]
The symptom: an early-fusion model scores well, but a test with the voice blanked gives almost the same score. The measurement: compare accuracy with each input removed in turn; a tiny drop means the model barely used it. The fix: pre-train each encoder alone so that both carry signal, or train with random input dropout so the model learns to use whichever inputs it has. Chapter 7 gives a sharper test.
Chapter 3
Add a single multiplied term to a straight-line fit and watch it bend the answer for one group but not the other
Picture 300 book reviews. For each one you know two things from the page: how much the reviewer is smiling in their photo, and whether the reviewer is a professional critic. You also know one outcome: the audience score the review earned. You want a formula that predicts the score from the two facts.
This is the lecture’s example on slides 10 to 14. It is deliberately tiny, so that every number can be read. The lecture’s plan is to study the univariate case first: each signal is a single number, not a vector. What we learn here carries over to vectors in the next chapters.
A linear regression predicts a number as a weighted sum of inputs. The lecture writes it with an extra product on slide 10:
Why is a product the way to model an interaction? Think of what a product does. It is large only when both numbers are large, and it flips sign when one flips sign. A sum cannot do that: in a sum, each input’s push is the same whatever the other input is. In a product, how much xA counts depends on xB. That dependence is the interaction.
The lecture fits three models and prints their weights. The first, on slide 12, uses smiling alone: y = w0 + w1xA. It gives w0 = 4.63 and w1 = 1.20. The second, on slide 13, adds status: w0 = 5.29, w1 = 1.19, w2 = −1.69. Smiling has a positive effect and being a critic a negative effect. The third, on slide 14, adds the product: w0 = 5.79, w1 = 0.68, w2 = −2.94, w3 = 1.29. The slide labels the new weight multiplicative interaction!
See what the third model says. With xB = 0 (non-critics) the product vanishes, so the slope in smiling is w1 = 0.68. With xB = 1 (critics) the slope is w1 + w3 = 0.68 + 1.29 = 1.97. The two groups have different slopes. The second model could not say that: it forces both groups to share the slope 1.19 and only shifts one line below the other by 1.69.
Each model draws one line per group: critics in warm, non-critics in blue. Choose a model, then in the third one drag the interaction weight and watch only the critic line tilt.
The weights are the lecture’s (slides 12 to 14). The range of smiling, 0 to 1, is an assumption for the picture, since the slide does not state its scale; the lines and slopes follow from the weights exactly. The w3 slider is a what-if on the third model: the other weights stay as fitted.
Set smiling to 0.5. Step one, additive model: a non-critic scores 5.29 + 1.19 × 0.5 = 5.885, and a critic scores 5.885 − 1.69 = 4.195. Step two, product model, non-critic: 5.79 + 0.68 × 0.5 = 6.13. Step three, product model, critic: 6.13 − 2.94 + 1.29 × 0.5 = 3.835. The gap between the groups has grown from 1.69 to 2.30, and it will keep changing as smiling changes. At smiling 1.0 the product model’s gap is 2.94 − 1.29 = 1.65, nearly closed. That is what a changing gap looks like, and a sum cannot draw it.
Notice also what happened to the other weights. When the product entered, w1 fell from 1.19 to 0.68 and w2 fell from −1.69 to −2.94. The weights changed meaning. In the third model, w1 is the smiling slope for non-critics only, and w2 is the critic offset at zero smiling only. This is the first lesson of fusion, and it returns again and again: once an interaction is in the model, a single input no longer has a single effect.
pythonimport numpy as np # x_a: [300] smiling, x_b: [300] critic (0 or 1), y: [300] audience score X = np.stack([np.ones(300), x_a, x_b, x_a * x_b], axis=1) # [300, 4]: one column per weight w, *_ = np.linalg.lstsq(X, y, rcond=None) # [4]: w0, w1, w2, w3 slope_noncritic = w[1] slope_critic = w[1] + w[3] # the product term moves the slope
The symptom: the two groups’ points fan out, but the fitted lines stay parallel, and the residuals form a pattern, positive on one side and negative on the other. The measurement: refit with the product column and compare. If w3 is large and the error drops, the pairing mattered. The fix: keep the product term, and read the other weights as conditional on the other input.
Chapter 4
Multiply every part of one signal by every part of the other, and count how fast the result grows
A restaurant pairs wines with dishes. Each wine has a few traits: dry, sweet, strong. Each dish has a few too: salty, rich, spicy. The score of a pairing is not the wine’s score plus the dish’s score. A strong wine is wonderful with a rich dish and awful with a delicate one. The score lives in the pair of traits, and the way to hold it is a table with one cell for every wine trait against every dish trait.
That table is the idea of this chapter. In Chapter 3 two numbers had one product. Now each signal is a vector with many numbers, and we want a product for every pair.
Slide 15 restates the regression, but now the output is a representation z, a list of numbers that feeds the next layer, instead of a score. It names three ways to build it. An additive interaction is z = w1xA + w2xB. A multiplicative interaction is z = w3xAxB. The third has both. The lecture then moves to the multivariate case with vectors, which we follow.
Slide 16 gives additive fusion for vectors: z = w1xA + w2xB. With encoders, it becomes z = fA(xA) + fB(xB). The slide notes that this can be seen as an ensemble, which is late fusion in disguise. The two parts never multiply, so no pair of traits ever meets. That is exactly why Chapter 2’s clerk failed.
Slide 17 gives the first way to multiply. Multiplicative fusion is z = w(xA × xB), where the product is taken element by element: the first number of xA times the first of xB, the second times the second, and so on. This needs both vectors to have the same length, and it pairs only traits that share a position. Bilinear fusion is Z = w(xATxB), where the product is the outer product: every number of xA times every number of xB, giving a full matrix Z with one cell per pair. The vectors may now have different lengths.
Slide 18 shows the tensor fusion network of Zadeh and coauthors (EMNLP 2017). A bilinear table holds only products. A good fusion should hold the additive terms too, so that the model can still learn the single-signal effects of Chapter 3. The trick is small: append a constant 1 to each vector before taking the outer product.
Work it by hand. Let xA = [2, 3] and xB = [4, 5]. Step one: append the 1s, giving [2, 3, 1] and [4, 5, 1]. Step two: take the outer product, a 3 × 3 table whose rows are [8, 10, 2], [12, 15, 3] and [4, 5, 1]. Step three: read it. The top-left 2 × 2 block, 8, 10, 12 and 15, holds the bimodal products. The last column, 2 and 3, is xA times the appended 1, so it is xA itself: the unimodal terms. The last row, 4 and 5, is xB itself. The corner, 1, is the constant. One table holds all three kinds of term, just as the regression of slide 15 did. A weight w then maps the flattened table to the output.
Each cell is one number the next layer must weight. Grey is the constant, blue and purple are single-signal terms, warm cells are pair products. Choose the vector sizes, then add a third signal.
Exact counts for an output of 512 numbers. The grid shows at most 14 cells per side, with the real counts in the read-out. Sizes 8, 32, 64, 256 and 768 are illustrative choices; 768 is a common encoder size.
Set both sizes to 768 and the table has 769 × 769 = 591,361 cells. A weight matrix to an output of 512 numbers then holds about 303 million parameters. Switch on a third signal and the table becomes a cube of 769 × 769 × 769, about 455 million cells, and the weights reach about 233 billion. The slide says it plainly: but the weight matrix may end up quite large.
| Fusion | Formula | Pairs it can see | Size of z |
|---|---|---|---|
| Additive | w1xA + w2xB | None | d |
| Element-wise product | w(xA × xB) | Same position only | d |
| Bilinear | w(xATxB) | Every pair | dA × dB |
| Tensor fusion | w([xA, 1]T[xB, 1]) | Every pair, plus each signal alone | (dA + 1)(dB + 1) |
python# x_a [B, d_a], x_b [B, d_b]; h = output size ones = torch.ones(x_a.shape[0], 1) a1, b1 = torch.cat([x_a, ones], 1), torch.cat([x_b, ones], 1) # [B, d_a+1], [B, d_b+1] Z = torch.einsum('bi,bj->bij', a1, b1) # [B, d_a+1, d_b+1] the table out = Z.flatten(1) @ W # W: [(d_a+1)*(d_b+1), h]
The symptom: training runs out of memory the moment a third signal is added, though the data is small. The measurement: multiply the sizes. Each new signal multiplies the table by its size plus one, so the count grows like a power. The fix: do not build the table at all. The next chapter shows how to get the same output from small pieces.
Chapter 5
Replace one giant weight table with a few thin ones, and compute the fused output without ever building the table
A wall of tiles in a hotel lobby looks like it needs a thousand different tiles. Look closer: it is one pattern, repeated. You could write the pattern down on a card and a short list of where to stamp it. The card is much smaller than the wall, yet it gives the same wall.
Chapter 4 ended with a problem: a table of 591,361 cells and 303 million weights for just two signals. Slides 19 to 22 give the fix, low-rank fusion from Liu and coauthors (“Efficient Low-rank Multimodal Fusion with Modality-Specific Factors”, ACL 2018). The slides also cite LoRA, the low-rank adaptation of large language models (Hu et al., ICLR 2022), because both use the same trick.
Recall tensor fusion. Call the appended vectors ẑA (length dA + 1) and ẑB (length dB + 1). The table is their outer product Z, and the weight W is a block of numbers of shape [dA + 1, dB + 1, h]. Each output number hk is a weighted sum over every cell of Z.
The low-rank idea, slide 19 step 1, is to decompose the weight. Instead of storing W cell by cell, store r pairs of thin matrices, WA(i) with shape [dA + 1, h] and WB(i) with shape [dB + 1, h], for i = 1 to r. Build W from them: the cell at row a, column b, output k is the sum over i of WA(i)[a, k] times WB(i)[b, k]. The number r is the rank: how many such pairs you keep. A rank-one piece is a single pair. A low-rank weight is a short sum of them, the pattern card for the wall.
Slides 20 and 21 then do the two moves that matter. Step 2 decomposes the input table too, as an outer product that never needs to be written. Step 3 rearranges the sum. Put the two decompositions into the formula for hk and the double sum over cells splits into two single sums, one over a and one over b:
The circle ∘ means multiply element by element. So each rank projects A, projects B, and multiplies the two projections. That is all. The table Z is never built, and neither is W. This is slide 22: the final form is a short sum of products of two thin matrix-vector multiplications.
Top: the output of a tiny fusion computed two ways, by building the full table, and by the low-rank formula. Bottom: how many weights each needs at real sizes. Change the rank, or draw new random weights.
Top bars: a toy with 6 + 1 numbers per signal, 4 outputs and random seeded weights, computed live in your browser in both ways; the largest difference is rounding error. Bottom bars: exact counts for 768 numbers per signal and 512 outputs, as in Chapter 4. Real systems choose r by validation accuracy; r here is a free slider.
Take 768 numbers per signal and 512 outputs. The full table weight has 769 × 769 × 512 = 302,776,832 numbers. At rank 4, each signal needs 4 thin matrices of 769 × 512, so the total is 4 × (769 + 769) × 512 = 3,149,824 numbers. That is about 96 times fewer. The cost to compute one output falls the same way: instead of 303 million multiply-adds over the table, you do about 3.1 million for the two projections plus 2,048 for the element-wise products and the sum.
The price is expressiveness. A weight built from r pieces can capture r “patterns” of interaction, not every possible one. The lecture’s point is that real interaction structure is often simple enough that a small r is enough, so the trade is a good one. The slider shows the other side: at small r the parameter bar is tiny, and you can move toward the table only by paying for more rank.
LoRA uses the same shape of idea for a different job. To tune a big frozen weight, it learns a change that is the product of two thin matrices, so that the change has low rank. In fusion, the weight itself is the low-rank object. In both cases, thin matrices replace one fat one.
python# z_a [B, d_a+1], z_b [B, d_b+1]; factors: wa [r, d_a+1, h], wb [r, d_b+1, h] fa = torch.einsum('bd,rdh->rbh', z_a, wa) # [r, B, h] one projection per rank fb = torch.einsum('bd,rdh->rbh', z_b, wb) # [r, B, h] h = (fa * fb).sum(dim=0) # [B, h] multiply per rank, then sum over rank
The symptom: low-rank fusion scores below the unimodal baseline on a task known to have synergy. The measurement: sweep r over 1, 2, 4, 8 and plot validation accuracy; if it is still climbing at the largest r you tried, the pieces are too few. The fix: raise r until the curve flattens, and stop there, since each extra rank costs another pair of thin matrices.
Chapter 6
Let a small network look at both signals and decide how much of each to let through, or how far one should nudge the other
You are on a video call with two colleagues who both report the same number. Alice has a clear line. Ben’s microphone crackles. You do not average their words equally. Without thinking, you lean on Alice and half-ignore Ben. The next minute, Alice’s line drops and you switch. Your trust is not fixed. It changes with what you hear, moment by moment.
Slide 23 puts that behaviour into a fusion layer. In the plain additive fusion of Chapter 4, the weights of the two signals were fixed after training. A gated fusion makes the weights depend on the input.
A gate is a small neural network that outputs a number between 0 and 1 (or a vector of such numbers). You multiply a signal by its gate. If the gate is 1, the signal passes. If the gate is 0, the signal is blocked. Between the two, it is turned down. The key point is what the gate reads: both signals. The lecture writes the additive example this way, citing the Gated Multimodal Unit of Arevalo and coauthors (ICLR workshop 2017):
The slide adds that gA and gB can be seen as attention functions, and that the gate output can be one weight for the whole modality. Slide 24 plays with the two names. Seen negatively, a gate is a “neural network designed to mask unwanted signal from propagating forward”. Seen positively, it is attention: “designed to select preferable signal to move forward”. The maths is the same; the view differs.
Slide 24 also asks what should it be? The gate looks at an input and chooses among: the target modality, the other modality, or all modalities. A soft attention gate returns fractions, such as 0.9 and 0.1, so every signal contributes a little. Its derivative (the gradient used in training) is easy to compute. A hard attention gate returns a pure choice: 1 for one signal and 0 for the other. A choice has no smooth slope, so the derivative is harder; the slide gives reinforcement learning as the usual route, from Chen and coauthors (ICMI 2017). The cost of softness is a blend that may carry some noise from the weaker signal. The cost of hardness is a harder training problem.
Words and voice both estimate a true sentiment of +1. The voice is sometimes corrupted. Bars show the typical error, so shorter is better. Drag how often the voice is corrupted. In the second mode, a voice cue shifts a word.
Toy simulation, labelled illustrative: 600 seeded trials; words have noise 0.6; the voice has noise 0.3 when clean and 2.0 when corrupted. The gate is told which case it is in, standing in for what a trained gate learns; the soft gate weights each signal by the inverse of its noise variance, the hard gate picks the cleaner signal. The shift mode is a one-line toy: z = 0.10 + gate × tone × 0.8, with the gate fixed by hand.
Try 50% corruption. Equal averaging suffers whenever the voice is bad, because it hands the broken signal half the vote. The soft gate gives the broken voice only a small weight, and its error falls below either signal alone. The hard gate does well too, but it throws away the small gain from blending when both signals are good.
On a corrupted clip, words have noise 0.6, so variance 0.36, and the voice has noise 2.0, so variance 4. Step one: the ideal weight for a signal is its inverse variance divided by the sum of both inverse variances. Equivalent: gA = 4 / (0.36 + 4) = 0.917 and gB = 0.083. Step two: the fused variance is 0.36 × 4 / 4.36 = 0.330, so the noise is 0.57. Step three: equal weights of 0.5 give variance 0.25 × 4.36 = 1.09, so the noise is 1.04. The gate cuts the typical error by almost half on those clips, with the same two inputs.
Slide 25 applies a gate for a different job. Choose one primary modality, here language, and treat the others (sound and vision) as secondary. The primary signal passes through unchanged except for a shift: a vector computed from the secondary signals and added to it, scaled by a gate. The slide’s example is the word “wow”. Said in a delighted voice, its representation moves toward the positive side. Said flatly with a frown, it moves to the negative side. The word has no single meaning, so its vector must move. The slide cites Wang and coauthors (“Words Can Shift”, AAAI 2019) and Rahman and coauthors (ACL 2020), who built the idea into a large pre-trained transformer.
Chapter 7
Ask a big network to fuse two signals, then take out its interactions and see whether its answers change
A student shows a model that answers questions about pictures. “How many dogs are in the photo?” The model says “two”. It is right. The student says: “it looks at the picture and the question together.” But maybe the model never looked at the picture. The word “how many” alone makes “two” a fair guess, and in this dataset it is right more often than not. How would you tell?
That is the puzzle of slides 26 to 28 and of the assigned reading, Does my multimodal model learn cross-modal interactions? It’s harder to tell than you might think! by Jack Hessel and Lillian Lee (EMNLP 2020).
Slide 26 states the most general design. Nonlinear fusion is ŷ = f(xA, xB), where f can be a neural network or any nonlinear model, with output in Rd (a vector of d real numbers). The slide notes that this is early fusion: join the inputs, ŷ = f([xA, xB]). A neural network with enough hidden units can represent the product terms of Chapter 3 and much more. So the question on the slide is a fair one: but will our neural network learn the nonlinear interactions? Being able to is not doing it. Training finds some solution, and the easiest one may use each signal separately.
Hessel and Lee propose a diagnostic named EMAP, for empirical multimodally-additive function projection. The idea is to take the trained model f and forcibly remove its cross-modal interactions, leaving only the part that adds the two signals’ effects. If the score does not change, the model was not using interactions. The name unpacks as follows. A function is multimodally additive if it has the form fA(xA) + fB(xB) + a constant. A projection finds the additive function closest to f. Empirical means the averages come from your own data.
The slides write the projection as the average of f over one signal plus the average over the other, minus the overall mean. In symbols, f̃(xA, xB) = ExB[f(xA, xB)] + ExA[f(xA, xB)] − E[f]. Read it in words. To predict for a picture, average the model’s output over every question it could be paired with. To predict for a question, average over every picture. Add the two and subtract the grand average once so that it is not counted twice.
Do it on a 2 × 2 table. Let f(0,0) = 0, f(0,1) = 1, f(1,0) = 1, f(1,1) = 4. Step one: the row averages are 0.5 for xA = 0 and 2.5 for xA = 1. Step two: the column averages are 0.5 and 2.5 for xB = 0 and 1. Step three: the grand mean is (0 + 1 + 1 + 4) / 4 = 1.5. Step four: f̃(0,0) = 0.5 + 0.5 − 1.5 = −0.5, f̃(0,1) = 0.5 + 2.5 − 1.5 = 1.5, f̃(1,0) = 1.5, f̃(1,1) = 3.5. Step five: the leftover f − f̃ is +0.5, −0.5, −0.5, +0.5. That leftover is all the interaction the table had.
Left: the model’s output over an 8 × 8 grid of inputs (warm is positive, blue is negative). Middle: its EMAP, the additive part. Right: what EMAP removed. Slide the interaction share and watch the right panel.
Toy, labelled illustrative: f(a, b) = (1 − s)(a + b)/2 + s · a · b on an 8 × 8 grid, where s is the interaction share. EMAP is computed exactly from the grid (row means plus column means minus the grand mean). Here the model is perfect, so the test shows how much of the correct answer depends on the interaction.
At 0% the right panel is empty: the function is purely additive, and EMAP changes nothing. At 100% the middle panel is blank, because a pure product has no additive part. In between, the sign of f and of its EMAP disagree on only a handful of cells: 6 of 64 at 30% and at 50%, where the product outvotes the sum. At 100% the count reads 32, because the EMAP is zero everywhere and half the cells are positive. A model whose behaviour sits near the left of the slider would not show up as better than an additive model on a test set.
Slide 28 tabulates seven image-and-text tasks from the paper. In many cases, removing the cross-modal interactions caused little or no loss. So a gain from a bigger model, when present, often cannot be credited to cross-modal interactions. Slide 28 shows three of the columns: for the best model against its EMAP, 91.3 against 91.1, 74.4 against 74.2, and 81.5 against 81.3. A plain linear model scores 90.4, 72.8 and 80.9 on the same columns. The slide’s own annotation is that the differences are small, and that additive fusion is always a good baseline.
The authors’ recommendation is practical: report not just the unimodal baselines but also the EMAP of your best model. You pay almost nothing, because EMAP only reruns the trained model on mixed-up pairs.
python# xa [N, d_a], xb [N, d_b]; model(...) returns logits [..., C] F = model(xa[:, None, :], xb[None, :, :]) # [N, N, C], F[i, j] = f(xa_i, xb_j) emap = F.mean(1) + F.mean(0) - F.mean((0, 1)) # [N, C]: row mean + column mean - grand mean acc_full = (model(xa, xb).argmax(-1) == y).float().mean() acc_emap = (emap.argmax(-1) == y).float().mean() # a tiny gap means few interactions used
The symptom: a paper shows a large gain from a cross-attention model and says it “captures cross-modal relations”. The measurement: compute the EMAP accuracy. If it is within a few tenths of a point of the full model, the model works additively. The fix: say so, and look for the gain elsewhere, such as a better encoder or more data. Pair this with the PID estimate of Chapter 1: PID asks whether the data holds synergy, and EMAP asks whether the model used any.
Chapter 8
Make a model explain what it can with single signals, then pairs, then triples, so that each level fixes only what the level below missed
A detective arrives at a crime scene with three witnesses. A careful one starts with what each witness says alone. Only then does she ask which two accounts agree or clash. Only after that does she look for something that appears only when all three stories are laid together. She does not start with the grand theory. If a simple explanation covers the facts, she stops.
Slide 29 asks a fusion model to work like her. Its heading is the idea: prioritize simpler interactions. The method is from Wortwein and coauthors, “Beyond Additive Fusion: Learning Non-Additive Multimodal Interactions” (EMNLP 2022), and the lecture calls it multimodal residual optimization.
Chapter 7 showed the trouble. A big network that can learn any interaction may learn none, because the additive route is easier. The reverse trouble exists too. A big network may explain everything with one tangled, hard-to-read function of all signals, and nobody can say which signal did what. Both problems come from letting the model choose its own route.
The fix is to give the route in advance. The slide splits the prediction into three parts, ordered by how simple they are: a unimodal part ŷuni, which sums the effect of each signal alone (it is additive); a bimodal part ŷbi, which depends on pairs of signals, such as (xA, xB); and a trimodal part ŷtri, which depends on all three together. The final prediction is the sum of the three.
A residual is the part of the answer not yet explained. Each level is trained to match the residual of the levels below it. A higher-order term can therefore earn its place only by explaining something the simpler terms could not. If pairs add nothing, ŷbi stays near zero, and you learn that the task is mostly additive.
Three signals a, b and c, and an answer that mixes single, pair and three-way effects. Bars show how much of the answer each stage explains in total, since each stage fits what the stage before left behind. Set how strong the pair and three-way effects are.
Toy, labelled illustrative: 300 seeded points of three signals between −1 and +1, and an answer y = a − 0.6b + 0.4c + s2(ab + bc) + s3abc plus small noise. Each stage is an exact least-squares fit to the residual of the stage before. The stage terms are a, b, c and a constant; then ab, ac and bc; then abc.
At the default settings, the single terms alone explain 68% of the answer. Adding the pair terms lifts it to 96%. The three-way term finishes at 99%. Now slide the pair strength to zero and the three-way strength to zero: the first bar jumps to 98% and the other two have nothing to add. The staged fit reads like a report. It says: here is what the signals do alone, here is what pairs add, here is what only the triple adds.
Suppose the true answer for one clip is 2.0, and the unimodal part predicts 1.2. Step one: the first loss compares 1.2 to 2.0, an error of 0.8. Step two: the residual is 2.0 − 1.2 = 0.8. Step three: the bimodal part is trained against 0.8, not against 2.0, and suppose it predicts 0.6. Step four: the new residual is 0.8 − 0.6 = 0.2. Step five: the trimodal part is trained against 0.2. The total prediction is 1.2 + 0.6 + 0.2 = 2.0 when all three succeed. Each part has a small, clear job.
The lecture states the benefit as a hierarchy of interactions: unimodal (additive), bimodal (non-additive) and trimodal (non-additive). The result is a model that is easier to read, and one that uses a high-order interaction only when the data demands it. It connects directly to Chapter 7: you can compare the size of ŷbi and ŷtri with ŷuni to see how much of the answer is truly non-additive.
python# y [B]; heads see: uni -> each signal alone, bi -> pairs, tri -> all three y_uni = uni_head(x_a, x_b, x_c) # [B] sum of per-signal terms y_bi = bi_head(x_a, x_b, x_c) # [B] terms over (a,b), (a,c), (b,c) y_tri = tri_head(x_a, x_b, x_c) # [B] one joint term loss = mse(y_uni, y) \ + mse(y_bi, (y - y_uni).detach()) \ + mse(y_tri, (y - y_uni - y_bi).detach()) # detach: each level fits the residual, not the others y_hat = y_uni + y_bi + y_tri # [B]
Without the ordering, a flexible head can take the whole answer and leave the simple heads idle. The symptom: unimodal and bimodal outputs are near zero while the trimodal head matches y. The measurement: the share of the answer explained by each stage, as the bars above show. The fix: keep the ordered losses, and detach the targets so that a higher level cannot pull a lower one off course.
Chapter 9
See how the ideas of this lecture grow up: one sequence of tokens, attention as the interaction, and experts split by modality
Chapters 3 to 8 fused two or three signals, each a small vector. Real systems today read a page that mixes a paragraph, a photo, another paragraph and a chart. There is no neat list of modalities to wire together. Everything must go into one model, in the order it appears.
Slide 30 closes the lecture with a map. It repeats the definition of fusion and lays methods along a line that runs from homogeneous modalities to heterogeneous modalities. From left to right the names read: late fusion, additive fusion, multiplicative fusion, tensor fusion, polynomial fusion, gated fusion, modality-shift fusion, nonlinear fusion, very early fusion, dynamic early fusion, heterogeneity-aware fusion, then two goals, improving optimization and improving robustness. We have met seven of the first eight. Polynomial fusion, which adds powers of each signal as extra product terms, is only named here. The rest belong to the next lecture and to the four model papers on this week’s reading list, which push fusion all the way to the input.
The idea is simple to say. Turn every modality into tokens (whole-number codes from a fixed list) and put them all in one sequence. A single transformer then reads the sequence. No encoder-per-modality, no separate fusion layer. The fusion happens in attention: each token computes a score against every other token as the dot product of a query and a key. That score is a product of one token’s numbers with another’s, summed. It is exactly the bilinear form of Chapter 4, applied to every pair of tokens, including every text-token and image-token pair. Attention is a learned, input-dependent version of the bilinear and gated fusion you already know.
Count what that costs. A 512 × 512 image becomes 1,024 image tokens (Chameleon’s tokenizer). Add a 20-word question and the sequence has 1,044 tokens. Causal attention means each token looks only at itself and the tokens before it. It compares each token with itself and every earlier one: 1,044 × 1,045 / 2 = 545,490 pairs in each layer, of which more than 99% involve an image token. A picture weighs as much as a thousand words in the sequence, and that is a real, countable cost.
Chameleon (arXiv 2405.09818; our lesson: Chameleon). A family of early-fusion token-based mixed-modal models that understand and generate images and text in any order. A tokenizer cuts a 512 × 512 image into 1,024 tokens from a codebook of 8,192 codes, which join the roughly 65,000 text tokens in one vocabulary. The paper describes a stable training approach from the start, an alignment recipe, and an architecture tuned for this setting. It reports strong image-captioning results and beats Llama-2 on text-only tasks while generating images in the same model.
Kosmos-2 (arXiv 2306.14824; our lesson: Kosmos-2). A multimodal language model that grounds text in the picture. It writes a mention of an object as a Markdown link, “[text span](bounding boxes)”, where the box is a short run of location tokens. The authors built a large set of grounded image-text pairs called GrIT to train it. It handles referring to objects and describing regions, plus the usual captioning and question tasks. Here the fusion carries where, not only what.
MM1 (Springer chapter; our lesson: MM1). A study of what matters when building a multimodal language model. Its design lessons: a careful mix of image-caption, interleaved image-text and text-only data is crucial for strong few-shot results; the image encoder, with the image resolution and the number of image tokens, matters a lot; the vision-language connector matters comparatively little. The recipe scales to models up to 30 billion parameters, dense and mixture-of-experts.
MoMa (arXiv 2407.21770; our lesson: MoMa). A mixture of experts splits a layer’s feed-forward block into several experts, and a learned router sends each token to some of them. MoMa makes the groups modality-aware: some experts see only text tokens, others only image tokens, with learned routing inside each group. Try the device below.
Top: ten tokens, words in blue and image patches in purple, going to experts. Bottom: the compute each design needs to reach the same loss as a dense model of equal cost. Choose a design.
The savings are the MoMa paper’s (1.4B model, 1 trillion training tokens, measured by pre-training loss): the standard expert-choice MoE with 8 mixed-modal experts saves 3x overall (3x text, 2.8x image); MoMa with 4 text and 4 image experts saves 3.7x overall (2.6x text, 5.2x image); adding mixture-of-depths saves 4.2x (3.4x text, 5.3x image) but hurts causal inference through router sensitivity. The bar is 1 divided by the savings. The routing lines are an illustrative, fixed pattern, not learned routes.
The savings come from a simple fact. Image tokens and text tokens have very different statistics, so letting each group have its own experts spares the model from using one expert for both. In the paper, the 3.7 times overall saving splits unevenly: 2.6 times on text and 5.2 times on images. Reading the bars, an image token needs about 19% of the dense compute, a text token about 38%.
The symptom: the model captions well but its text answers get worse, or training diverges early. The measurement: count tokens and compute by modality, as with the 99% above, and watch the loss for each modality separately. The fix is at the architecture level, and each paper shows one: a stable training recipe for the shared vocabulary (Chameleon), data mixes that keep text in the diet (MM1), and separate experts per modality (MoMa).
Chapter 10
Keep the numbers, the terms, a checklist and the core code on one page, and pick a fusion for your own problem
The lecture in one paragraph. Fusion builds one joint representation from several signals. What the signals tell you about a task splits into redundancy, uniqueness and synergy. Early fusion joins inputs, late fusion joins verdicts, and only early or intermediate fusion can see synergy. A product term models an interaction; tensor fusion keeps every pair of terms and costs a table that grows like a power; low-rank fusion gets the same output from thin factors. Gates and shifts make the weights depend on the input. Whether a trained network used any interaction is a question you can answer: project it onto its additive part with EMAP. And a model that must explain itself can fit singles first, then pairs, then triples.
Switch on each statement that is true for your task. The recipe below follows the chapters of this lesson.
A rule of thumb that restates this lesson, not a result from a paper. Always run the unimodal baselines and the EMAP of your best model first, whatever the recipe says.
| What | Value | Source |
|---|---|---|
| Book-review regression, slope in smiling | 0.68 for non-critics, 1.97 for critics (0.68 + 1.29) | Lecture, slides 12 to 14 |
| Exclusive-or information | 0 bits alone, 1 bit together | Computed in Chapter 1 |
| Tensor table, 768 and 768 | 769 × 769 = 591,361 cells; 302,776,832 weights at 512 outputs | Computed in Chapter 4 |
| Low-rank at rank 4 | 3,149,824 weights, about 96 times fewer | Computed in Chapter 5 |
| Ideal gate on a corrupted clip | 0.917 words, 0.083 voice; error 0.57 against 1.04 for an equal mix | Worked in Chapter 6, illustrative |
| Best model against its EMAP | 91.3 vs 91.1, 74.4 vs 74.2, 81.5 vs 81.3 | Lecture, slide 28 (Hessel and Lee) |
| Image as tokens | 512 × 512 gives 1,024 tokens from 8,192 codes | Chameleon |
| One image plus 20 words | 1,044 tokens; 545,490 causal attention pairs per layer | Computed in Chapter 9 |
| MoMa savings at 1.4B | 3.7x overall; 2.6x text; 5.2x image | MoMa paper |
| Term | In one line |
|---|---|
| Modality | One kind of signal: words, sound, pixels. |
| Redundancy, uniqueness, synergy | Information both signals give, only one gives, or only the pair gives. |
| Early, intermediate, late fusion | Join the inputs, the hidden features, or the predictions. |
| Interaction term | A product such as w3xAxB that makes one input’s effect depend on the other. |
| Tensor fusion | The outer product of the two vectors, each with a 1 appended. |
| Low-rank fusion | A short sum of products of thin projections instead of the full table. |
| Gate | An input-dependent weight between 0 and 1 on a signal. |
| EMAP | The additive projection of a trained model: row mean plus column mean minus grand mean. |
| Residual optimization | Fit singles, then pairs to what is left, then triples to what is left. |
pythonimport torch, torch.nn as nn class Fusion(nn.Module): """x_a [B, d_a], x_b [B, d_b] -> z [B, h]""" def __init__(self, d_a, d_b, h, kind="lowrank", rank=4): super().__init__(); self.kind = kind if kind == "additive": self.wa, self.wb = nn.Linear(d_a, h), nn.Linear(d_b, h) elif kind == "tensor": self.w = nn.Linear((d_a + 1) * (d_b + 1), h) # the big table elif kind == "lowrank": self.wa = nn.Parameter(torch.randn(rank, d_a + 1, h) * 0.05) self.wb = nn.Parameter(torch.randn(rank, d_b + 1, h) * 0.05) else: # gated self.pa, self.pb = nn.Linear(d_a, h), nn.Linear(d_b, h) self.gate = nn.Linear(d_a + d_b, 2) def forward(self, x_a, x_b): one = x_a.new_ones(x_a.shape[0], 1) a1, b1 = torch.cat([x_a, one], 1), torch.cat([x_b, one], 1) if self.kind == "additive": return self.wa(x_a) + self.wb(x_b) if self.kind == "tensor": return self.w(torch.einsum("bi,bj->bij", a1, b1).flatten(1)) if self.kind == "lowrank": fa = torch.einsum("bd,rdh->rbh", a1, self.wa) fb = torch.einsum("bd,rdh->rbh", b1, self.wb) return (fa * fb).sum(0) g = self.gate(torch.cat([x_a, x_b], 1)).softmax(-1) # [B, 2], reads both signals return g[:, :1] * self.pa(x_a) + g[:, 1:] * self.pb(x_b)
Chapters 3 to 8 are the classical toolkit for fusing a few signals. The modern systems of Chapter 9 fuse tokens inside one transformer, and the toolkit becomes the way to read them. For related lessons here: Fusion Architectures and the JDL Model compares early, intermediate and late fusion for sensors; microCLIP and microVLM build the vision-language side; CS224N Lecture 17, Multimodality and Multi-Modal Foundation Models give other course views; and the paper lessons for this week are Chameleon, Kosmos-2, MM1 and MoMa. The two measuring papers are taught in Chapters 1 and 7: Quantifying and Modeling Multimodal Interactions and Does my multimodal model learn cross-modal interactions?
Now explain, from memory, why a clerk who averages two verdicts can never solve an exclusive-or, and how a product term, a tensor table and a low-rank factor each fix it. Then go back to the sentiment plane at the top, set the pairing to 100%, and say aloud why only the product gets the clips right.
This lesson re-teaches MIT MMAI (Multimodal AI), Spring 2026, Lecture 4.1, Multimodal Fusion, by Paul Liang of the MIT Multisensory Intelligence group, in our own words and devices; some of the course’s slides were co-developed with Louis-Philippe Morency for CMU 11-777. See the course schedule and the lecture slides. Toy numbers are marked illustrative where they appear.