Cross-Modal Interactions and EMAP

A model that sees a picture and its caption together may score no better than two models that each see one. EMAP removes the combining and measures: on six of seven tasks, almost nothing is lost.

Pick a task, then press Remove interactions. Watch the best model's score fall back toward a plain sum of a text score and an image score. Then we build the test from zero, piece by piece: what an interaction is, how to project it away, why the projection is the best possible, and what the paper found.

You need an idea of what a classifier is (a program that picks a label for an input) and an average. We build the rest from zero.

Based on “Does My Multimodal Model Learn Cross-modal Interactions? It’s Harder to Tell than You Might Think!” by Jack Hessel (Allen Institute for AI) and Lillian Lee (Cornell University), arXiv 2010.06572, October 2020. It is a reading for Lecture 6 of MIT MMAI, Interactions and Fusion Models. Every idea is re-taught here in our own words and devices.

Remove the interactions

All scores are the paper’s: VQA 2.0 and GQA from Table 2 (accuracy, mean of 15 random subsamples of 500 dev questions, LXMERT), the other three from Table 4 (I-INT in AUC, T-ST2 in accuracy, T-VIS in weighted F1). Table 2 reports no linear model. The bars show the score from 0 to 100, so a small gap looks small, as it should.

Chapter 0

Why a Smart Model Can Ignore Half Its Input

See how a model that gets both a picture and a caption can score well while reading only one of them, and why that is hard to catch

Imagine a quiz show. The host holds a photo behind a card, and asks: “What colour is the banana?” The contestant never sees the photo. She says “yellow” and scores. Next question: “Is the sky in the photo blue?” She says “yes” and scores again. She is right most of the time and has looked at nothing.

This is not a joke about quiz shows. It happened to a real research benchmark. An early Visual Question Answering (VQA) challenge asks a program to answer a question about a picture. It was later “broken” by a method that ignored the image entirely (Jabri et al., 2016). The questions alone gave away the answers.

What we want a multimodal model to do

A modality is one kind of input: text, an image, a sound. A multimodal model takes more than one. We hope it does something no single modality allows. Picture a caption that says “Lovely day.” Over a sunny beach photo, it means what it says. Over a photo of a storm, it is sarcasm, and the meaning flips.

That flip is a cross-modal interaction: the right answer depends on the pair, in a way neither the text nor the image gives alone. Scholars of signs and meaning (semiotics) call it “meaning multiplication”. Text times image is worth more than text beside image.

Two old ways to check, and why each is weak

The first way is to balance the dataset. Build the questions so that each question has two images with two different answers. A text-only guesser then scores chance. VQA 2.0, NLVR2 and GQA are balanced in this spirit. It works, but it is not always possible. Take posts from a social network, collected to predict popularity or sentiment. If you rebalance them after the fact, you may erase the very trends you wanted to study.

The second way is model comparison. Train a text-only model, an image-only model, a simple multimodal model and a fancy one with cross-modal attention. If the fancy model wins, credit its extra expressiveness. The paper argues this is indirect evidence. Small gaps can come from the search over hyperparameters (the settings you tune), from random seeds, or from how many models you compared (Yogatama and Smith, 2015; Dodge et al., 2019).

The third way: take the interactions away and measure

Hessel and Lee propose a different test. Take one trained model. Replace its predictions with those of the closest possible ensemble of a text-only classifier and an image-only classifier. Then score the replacement. If the score barely moves, the model was not using the pair. If it drops, it was. The paper calls this EMAP, empirical multimodally-additive function projection. The next five chapters build it.

First, feel why the test is needed. Below is a tiny world. Each example has one text cue and one image cue, each either 0 or 1. The label usually follows the text cue. In some share of examples, the label follows the pair instead: the sarcasm rule, where the label is 1 when exactly one of the two cues is 1. Drag the share and read what each kind of model can reach.

Four models, one tiny world

Bars show the best accuracy each kind of model can reach in this world. Drag the share of “sarcasm” examples from 0% to 100% and watch the purple gap between the last two bars.

80%

Toy world, computed exactly, not from the paper. The four cue pairs are equally likely. With probability p the label is 1 when exactly one cue is 1; otherwise it equals the text cue. “Additive” means a score a(text) + b(image) compared with a threshold; the best one is found by trying every such labelling of the four cue pairs.

Work the 80% case by hand. The chance that the label is 1, for each cue pair (text, image), is 0.2 × text + 0.8 × (1 if exactly one cue is 1). So (0,0) gives 0, (0,1) gives 0.8, (1,0) gives 1.0 and (1,1) gives 0.2. A model that can pick any answer per pair takes the likelier label in each: it is right 100%, 80%, 100% and 80% of the time. The average is 90%.

An additive model cannot give (0,0) and (1,1) one label while giving (0,1) and (1,0) the other. That is the sarcasm pattern. It must break it somewhere, and the best it can do is 75%. The 15 point gap is the prize for modelling the pair. At 50% the gap is zero. Both reach 75%, so a model that sees the pair has nothing extra to find.

Here is the trap. Real data sits somewhere on that slider, and we do not know where. A fancy model that scores 3 points above a baseline might have found the pair rule. It might also have found a better way to read the text alone. Both look identical in a results table.

The road

  1. Additive or interactive: what the two words mean, with a four-number test.
  2. The closest additive twin: average out one modality to see what the other adds alone.
  3. Algorithm 1, step by step: the N×N cache of mismatched pairs, with shapes and cost.
  4. Why it is the best fit: a least-squares proof you can check by hand.
  5. Two sanity checks: a synthetic task and two balanced benchmarks, where EMAP must hurt.
  6. Seven tasks, four models: the data and the models the paper tests.
  7. What the seven tasks showed: Table 4, and the single task where interactions mattered.
  8. How much can additive models compute? The logic argument, the limits, three hypotheses.
  9. The field guide: the reporting recipe, the numbers, the code, what to read next.
A new fusion model beats a linear text+image baseline by 3 points. Why is that not yet proof that it learned cross-modal interactions?

Chapter 1

Additive or Interactive

Tell a model that adds two separate scores from a model that needs the pair, using one sum you can do on four numbers

You are at a diner. Your bill is a list: burger 9, fries 4, shake 5. The total is 18. Each item has its own price, and no item cares what else is on the table. That bill is additive: the total is a sum of separate parts.

Now the diner adds a combo deal. Burger plus fries together cost 11, not 13. The fries now cost 2 if a burger is on the table, and 4 if not. The price of one item depends on the other. That bill is interactive. You cannot split it into a burger price and a fries price that always hold.

The same split for a model

A model in this paper reads a text t and an image v and gives back a list of d scores, one per class. These raw scores, before they turn into probabilities, are called logits. Write the model as a function f(t, v). Hessel and Lee call f multimodally additive if it splits as follows.

f(t, v)
The model’s scores for the text t paired with the image v: a list of d numbers, one per class.
fT(t)
A part that reads the text only. It never sees the image. Think of one judge who gets only the caption.
fV(v)
A part that reads the image only. A second judge who gets only the picture. The model adds the two judges’ scores.

So an additive model is an ensemble (a team that votes by adding) of one text-only classifier and one image-only classifier. Some authors call this “naive.” In the multimedia field it is late fusion: process each modality alone, combine at the end. Early fusion mixes them inside the model. Any f that cannot be written as Equation 1 is interactive.

The linear model is additive: the proof in one line

The simplest multimodal model glues the two feature vectors end to end and applies one weight vector. The glued vector is written [t; v]. Its length is dt + dv, so the weights split into a text block wt and an image block wv.

f(t, v) = wT[t; v] + b = wtTt + wvTv + b

The second step is plain arithmetic: a dot product over a glued vector is the dot product of the first part plus the dot product of the second part. The first term sees only t. The second sees only v. The bias b is a constant you can give to either. With d classes, w becomes a d × (dt + dv) matrix and the same split holds for every class.

Try numbers. Let dt = 3 and dv = 2, one class. Take t = (1, 0, 2), wt = (0.5, −1, 0.25), v = (2, 1), wv = (1, −2) and b = 0.1. The text part is 0.5 + 0 + 0.5 = 1.0. The image part is 2 − 2 = 0. The score is 1.1. Swap in the image v = (0, 3). The image part is −6, and the score is −4.9. Notice what did not change: the text still adds exactly 1.0. In the paper’s words, each image, whatever text it is paired with, “contributes a fixed score,” and so does each text.

A four-number test

Suppose only two texts (A, B) and two images (A, B) exist. The model has four scores: fAA, fAB, fBA, fBB. If it is additive, write fAA = aA + bA, and the same for the others, where a is the text part and b the image part. Then fAA + fBB = aA + bA + aB + bB. And fAB + fBA = aA + bB + aB + bA. The same four letters appear. So the two sums are equal.

I
The interaction. It is 0 exactly when the four scores can be split into a text part and an image part.
fAA + fBB
The two scores on one diagonal of the grid: text A with image A, and text B with image B.
fAB + fBA
The other diagonal. We subtract it, so a grid with equal diagonals gives I = 0.

Check it on the diner. Put nothing on the table: 0. A burger alone: 9. Fries alone: 4. Both, with the combo: 11. Then I = 0 + 11 − (9 + 4) = −2. Without the combo, both would cost 13 and I would be 0. The combo is the interaction. Play with a grid of your own. The device also shows the closest additive grid, the idea of the next chapter, and what it leaves out.

Split the grid

Left: your four scores. Middle: the closest grid that is a text part plus an image part. Right: what is left over. Set the sliders, or press a preset.

1.0
2.0
3.0
6.0

Toy scores, one class, not from the paper. The closest additive grid is the row mean plus the column mean minus the grand mean, derived in Chapter 2. Presets: a bill with no combo, a combo that adds 2, and a flip where the text changes meaning with the image.

What breaks first when people read this? They label the architecture and stop. “It has cross-attention, so it is interactive.” The symptom is a model that scores well and a paper that claims fusion. The measurement is not the wiring but the function the wiring learned: is I close to 0 across real inputs? The fix is to test the trained function itself, which is the job of the next four chapters.

A grid has fAA = 2, fAB = 5, fBA = 1 and fBB = 4. Is it additive?

Chapter 2

The Closest Additive Twin

Average one modality away to learn what the other adds alone, then use two such averages to rebuild the model as a plain sum

A judge scores pasta dishes. Each dish is a sauce on a pasta, and her score depends on the pair: a garlic sauce is great on spaghetti and odd on ravioli. You want one number for what garlic is worth, ignoring which pasta carries it. How?

Serve garlic on every pasta you own, collect the scores and take the average. That average is garlic’s worth in general. Do the same for every sauce, and again for every pasta. If the judge only ever added a sauce score and a pasta score, these averages would rebuild every dish exactly. Where they fail to rebuild a dish, the dish has an interaction.

Partial dependence: average the other input away

The name for this average is partial dependence (Friedman, 2001; Hooker, 2004). For the text t, it is the model’s score for t, averaged over all the images v it could meet.

fT(t)
What the text t is worth on its own: the same garlic score on every pasta.
Ev[ f(t, v) ]
The expected (average) score when t is paired with an image v drawn at random. The subscript says which input we average away.
∫ f(t, v) p(v) dv
The same average written as an integral: each image’s score, weighted by how likely that image is, p(v).

We never have p(v), only our N test images. So we replace the integral with a plain average. The empirical partial dependence of the i-th text is f̂T(ti) = (1/N) ∑j f(ti, vj). The model scores text i with every image in the set, including images it was never paired with. The image side is the mirror: f̂V(vi) averages over all texts.

Why we subtract the grand mean

Add the two averages and you almost have the twin. One thing is wrong: the overall level is counted twice. Both averages already contain the model’s overall average score, so the sum is too high by exactly that amount. Call the overall average μ = Et,v[f(t, v)]. The projection is:

f̃(t, v)
The projected model: the additive twin of f. The paper calls it the projection of f onto the set of multimodally-additive functions.
Ev[f(t, v)]
The text part: what t is worth with the image averaged away. It depends on t only.
Et[f(t, v)]
The image part: what v is worth with the text averaged away. It depends on v only.
Et,v[f(t, v)]
The grand mean μ, subtracted once to undo the double count.

Check that it is fair. Suppose f was already additive, f = fT(t) + fV(v). The text average is fT(t) + E[fV]. The image average is E[fT] + fV(v). The grand mean is E[fT] + E[fV]. Add the first two and subtract the third: fT(t) + fV(v). You get f back. Projecting an additive model changes nothing, which is why a drop under EMAP can only come from interactions.

The projection on the combo bill

Take the combo grid from Chapter 1, scores 1, 2, 3, 6 for (text A, image A), (A, B), (B, A), (B, B). The text averages are (1+2)/2 = 1.5 and (3+6)/2 = 4.5. The image averages are (1+3)/2 = 2 and (2+6)/2 = 4. The grand mean is 12/4 = 3. Now rebuild each cell as text average + image average − 3:

(A,A): 1.5 + 2 − 3 = 0.5   (A,B): 1.5 + 4 − 3 = 2.5   (B,A): 4.5 + 2 − 3 = 3.5   (B,B): 4.5 + 4 − 3 = 5.5

Subtract from the original: 1 − 0.5 = 0.5, 2 − 2.5 = −0.5, 3 − 3.5 = −0.5, 6 − 5.5 = 0.5. The leftover is I/4 = 2/4 = 0.5 in each cell, with the sign pattern + − − +. That is the interaction, found with two averages and a subtraction. Try a larger grid next.

Average, add, subtract

Four texts down, four images across. Left: the scores f. Middle: the twin f̃. Right: the leftover. Drag the interaction strength, then pick a cell to see the arithmetic.

1.50
2
3

Toy scores, one class, not from the paper. The base is an additive grid (text parts 0, 1, 2, 3 plus image parts 0, 0.5, 1, 1.5) and the interaction is a fixed pattern with zero row and column sums, multiplied by your strength. Filled cells are positive; grey cells with a minus sign are negative.

Notice that the twin in the middle does not move as you drag the strength. The pattern was built with zero row and column sums, so the averages ignore it completely. All the interaction lands in the right-hand grid. With a real model, the two grids share the load in a way you cannot predict, and that is exactly why we compute them instead of guessing.

Why average over mismatched pairs

A reader might ask: why score text i with image j when the dataset only has text i with image i? Because a real caption and its own photo are related. If you averaged only over the true partner, the average would carry the link between the two, and part of the interaction would hide inside the “text part.” Friedman (2001) and Hooker (2004) argue for the average over the marginal p(v), every image regardless of text, and not the conditional p(v | t). The paper follows them. Mismatched pairs are the price.

What goes wrong if you skip this? Symptom: a projected model scores almost the same as the original on a task that does need the pair. Measurement: the projection was built from matched pairs only, so it still contains pair information. Fix: score all N×N combinations, as the next chapter does.

In the 2×2 combo example the grand mean is 3 and the text-A average is 1.5. Why is the twin built as text average + image average − 3, not just the sum of the two averages?

Chapter 3

Algorithm 1, Step by Step

Fill an N by N table with the model’s own scores on every text paired with every image, then read two averages off it

Picture a spreadsheet. Each row is a caption from your test set. Each column is an image from the same set. The cell where row 3 meets column 5 holds one number: what your trained model says when you show it caption 3 and image 5. Most of these pairs are mismatches. The model has never seen them, and that is fine.

The inputs and the output

The paper gives the procedure as Algorithm 1. It needs a trained model f that returns logits (d raw scores per input, one per class) and a labelled evaluation set of N pairs (ti, vi). It returns N projected score lists. Nothing is trained: f is read, not changed.

  1. Fill the cache. For every i and j from 1 to N, compute fcache[i, j] = f(ti, vj). This is N×N calls to the model.
  2. Find the grand mean. μ̂ = (1/N²) ∑i,j fcache[i, j]. One list of d numbers.
  3. For each i, take a row mean. projt = (1/N) ∑j fcache[i, j]: text i with every image.
  4. For each i, take a column mean. projv = (1/N) ∑j fcache[j, i]: every text with image i.
  5. Combine. preds[i] = projt + projv − μ̂. This is the Chapter 2 twin, for the real pair i.

That is the whole method. The paper stresses that it has no hyperparameters, takes a few lines of code and works on any classifier, from a polynomial kernel SVM to a pretrained transformer. Only the pairs on the diagonal (text i with its own image i) are scored in the end. The off-diagonal cells exist only to be averaged.

The shapes

NameShapeWhat it holds
f(t, v)dlogits for one text and one image
fcacheN × N × dlogits for every text with every image
μ̂dthe grand mean over all N² pairs
predsN × dEMAP logits, one list per real pair

Then score preds with the same metric as the original model: accuracy for a class decision, AUC (area under the ROC curve, a ranking score) for others. Compare the two numbers. The difference is the paper’s measure of how much the model gained from the pair.

The cost, and how the paper pays it

Step 1 is the only expensive one. A test set of N = 500 needs 500 × 500 = 250,000 calls to the model, not 500. The paper says to compute the projection over the evaluation set only. When even that is too much, as for the 87.2K-pair Reddit popularity set, run EMAP repeatedly on random 500-instance samples of the test set. Table 2 does this: it averages over 15 random samples of 500 dev-set questions.

A note on memory, our own arithmetic and not the paper’s. For VQA 2.0 the model has about 3,100 candidate answers, so d = 3,100. A full cache of 500 × 500 × 3,100 numbers is 775 million numbers, about 3.1 GB at four bytes each. You do not need to keep it. Add each new score into a running row sum and a running column sum as you go, and the memory falls to N × d.

Now run it by hand. The device below uses N = 3 and one score per pair, so you can see every step. Press Step and read the sentence under the table.

Run the algorithm

Each cell is the model’s score for one text with one image. Step through: fill the nine cells, find the grand mean, then read each real pair (the diagonal) from its row and its column.

Toy scores, one class, not from the paper. “Only adds” is a table of text part plus image part. “Needs the pair” scores 3 when the image matches the text and −1 when it does not, a match detector, which no sum of a text part and an image part can imitate.

The first table shows the fair case. A model that only adds comes back from the projection unchanged: the three diagonal scores 3, −1 and 3 return as 3, −1 and 3. The second table shows the cost of an interaction. The match detector gave 3 on every real pair. After the projection every real pair gets 0.33, the same number. The pair information is gone, and a classifier that needed it would now be guessing.

The failure mode to watch is a misleading setup, not a bad model. Symptom: the projected score equals the original on every task, even ones built to need the pair. Measurement: run the same code on a tiny model you wrote by hand to be a match detector, as above. If the projection does not flatten it, your code has a bug, such as averaging over matched pairs only. Fix: loop over all N×N pairs. The next two chapters give you more checks like this one.

You have N = 500 test pairs. How many times does Algorithm 1 call the model, and what does it do with the off-diagonal scores?

Chapter 4

Why It Is the Best Additive Fit

Show with a few lines of algebra that no other text-plus-image sum sits closer to the model, and see what “closest” does and does not promise

You have scattered points and you draw the straight line that passes closest to them. “Closest” has a standard meaning: add up the squared gaps between each point and the line, and pick the line with the smallest total. That is least squares. The paper makes the same kind of claim, with a grid of model scores in place of the points and the family of additive functions in place of the lines.

The claim

Among all additive functions, the EMAP twin has the smallest total squared gap to the model, on all N×N pairs. The paper states it as Equation 4 and proves it in Appendix A.

f̂
The chosen additive function: the EMAP twin, with a text part and an image part.
arg min
“The choice that makes the following number smallest.” The subscript says we only try additive functions.
∑i,j ∥ … ∥2
For each of the N×N pairs, take the d-number gap between the model and the twin, square each number, and add them all up.

The derivation for one class

The squared length of a gap splits into one squared term per class, and the classes do not mix. So we can solve one class at a time. Call the model’s score fij, and write the twin as ai + bj: a text part ai and an image part bj. The total to minimise is:

L = ∑i,j ( fij − ai − bj )²

At the smallest value, nudging ai changes nothing to first order, so the slope with respect to ai is zero. The slope is −2 ∑j ( fij − ai − bj ). Set it to zero and divide by N: the text-i average of f equals ai plus the average of the b’s. Write ri for the row average of f, and b̄ for the average of the b’s. Then ai = ri − b̄.

Do the same for bj: with cj the column average and ā the average of the a’s, bj = cj − ā. Add the two: ai + bj = ri + cj − (ā + b̄). Average the equation for ai over all i. The average of the ri is the grand mean μ, so ā = μ − b̄, which gives ā + b̄ = μ. Therefore

ai + bj = ri + cj − μ

That is Algorithm 1, step 5. It came out of the algebra, not out of a guess.

Unique, and what is not

Add 5 to every ai and subtract 5 from every bj. Every sum ai + bj stays the same. So the two parts are not unique, and no method can pick them out. The twin as a whole is unique. The total L is a bowl-shaped (convex) function of the grid of sums, and a bowl has one lowest point. The paper's claim is exactly this: the sum, mean-centered, is the unique and optimal solution.

There is also a picture with right angles. The leftover f − f̃ adds to zero along every row and every column. That makes it perpendicular to every additive grid, the way the error of a least-squares line is perpendicular to the line. So Pythagoras holds. On the combo grid of Chapter 2, the spread of f around μ is (−2)² + (−1)² + 0² + 3² = 14. The twin carries 2.5² + 0.5² + 0.5² + 2.5² = 13, and the leftover carries 4 × 0.5² = 1. And 13 + 1 = 14.

Try to beat it. The device holds a 4×4 grid with a real interaction. Start from the EMAP twin, nudge one text part and one image part, and watch the total gap L. You can also let the computer try 2,000 random additive fits.

Try to beat EMAP

The curve is the total squared gap L as you move text part 2. The pink dot is EMAP. The teal dot is your fit. Move the sliders, or press the search button.

0.0
0.0
0.0

Toy 4×4 grid, one class, not from the paper: the base of Chapter 2 plus interaction strength 1.5. Random fits start from EMAP and add Gaussian noise (spread 0.5) to every text part and image part. Everything shown is computed live in your browser.

Two things should stand out. The shift slider changes the parts and does not change L. And no nudge in any direction gets below the EMAP point, because the bowl has only one bottom. Random search finds many fits that are close, and none that are better.

What “closest” does not promise

Closest means closest in squared logit error. It does not mean best accuracy. Some other additive function, farther from f, could in principle classify better. EMAP asks a narrower question: how well does the additive function that best copies f do? If that copy loses nothing, then f’s interactions were not buying accuracy that the copy lacks. Symptom of a misreading: “EMAP is the best possible additive model.” Measurement: the paper’s own linear baselines are trained on labels, and are sometimes worse than EMAP. Fix: read EMAP as a copy of f, not a competitor to it.

You add 5 to every text part and subtract 5 from every image part of the EMAP twin. What happens to the fit, and why?

Chapter 5

Two Sanity Checks Where EMAP Must Hurt

Hold a match under the smoke detector: test EMAP on tasks built to need the pair, and see its score fall

You buy a smoke detector. Before you trust it, you hold a lit match underneath. If it stays silent, you do not need a second test. A tool that says “no interactions found” is only worth believing if it says “interactions found” when they are really there. The paper runs two such tests.

Test 1: a task that is pure interaction

Build a fake dataset where the label is impossible to get from either input alone. Take two random vectors in d dimensions, one for the “text” and one for the “image”. The label is whether their dot product is positive. A dot product multiplies the vectors position by position and adds the products: v · t = v1t1 + v2t2 + … Each term needs one number from each side. Knowing t alone says nothing about the sign, because v can be anything.

The paper's recipe has five steps. (1) Draw two random matrices V and T, with entries uniform in −0.5 to 0.5, that will stretch each vector into a larger “modality”. (2) Draw v and t as random vectors in d dimensions and scale each to length 1. (3) Keep the pair only if |v · t| is larger than a margin δ; otherwise draw again. (4) The label is 1 if v · t > 0, else 0. (5) The data point is (Vv, Tt, y). The sizes are d = 100, d1 = 2,000, d2 = 1,000 and δ = 0.25, with 5K points split 80/10/10.

Three models learn it. Table 1 gives the test accuracy before and after EMAP; chance is 50%.

ModelTest accuracyAfter EMAP
Linear (additive)52.8%52.8%
Polynomial kernel SVM (interactive)99.6%49.4%
Feed-forward network (interactive)99.0%53.8%

Read it row by row. The linear model cannot use products, so it stays near chance, and its EMAP is identical (52.8 to 52.8), just as Chapter 2 predicted for an additive model. The two interactive models solve the task almost perfectly. EMAP drops them to 49.4% and 53.8%, which is chance. The match is lit and the detector sounds.

You can reproduce the idea in the browser. The device below trains two tiny models on a small version of this task (d = 6, not the paper’s 100, and no random stretching). It then runs the Chapter 3 algorithm on 80 test pairs, which is 6,400 model calls. A second task, where the text alone decides, is the control: EMAP should do nothing there.

Run the match test

Press Train and project. Each press draws new data and trains two models: one on [t; v], and one on [t; v; t⊙v] (it also sees the products). Then EMAP is applied to the second one.

Toy, not the paper’s numbers: d = 6 unit-length vectors scaled to length √6, margin 0.25, 600 training points, 80 test points, logistic regression trained by 800 steps of gradient descent. The interactive model adds the products t⊙v (position by position) as extra features. Runs are random and different each time.

Test 2: two benchmarks balanced on purpose

Now real data. VQA 2.0 and GQA were designed to resist guessing from the question alone. The paper fine-tunes LXMERT, a pretrained transformer that processes image regions and words together, on each. It treats question answering as classification over a fixed answer list: 3.1K candidate answers for VQA 2.0 and 1.8K for GQA. Table 2 reports the mean over k = 15 random subsamples of 500 dev-set questions.

TaskLXMERTAfter EMAPConstant guess
VQA 2.070.340.523.4
GQA60.341.018.1

EMAP cuts accuracy by 29.8 points on VQA 2.0 and 19.3 on GQA, which the paper rounds to 30 and 19. To see how much of the gain over guessing is lost, divide. On VQA 2.0 the model beats the constant guess by 70.3 − 23.4 = 46.9 points. EMAP removes 29.8 of them: about 64%. On GQA the lead is 60.3 − 18.1 = 42.2 and EMAP removes 19.3: about 46%. (These two percentages are our arithmetic from the table.)

Notice that EMAP does not fall to the constant guess. At 40.5 and 41.0 it stays well above it. The paper reads this as LXMERT’s logits also holding real unimodal signal. For scale, text-only LSTM models reach 44.3 on the full VQA 2.0 test set and 41.1 on GQA, close to the EMAP scores. A model that scores 40 on a question set without using the image is not broken. Questions like “what colour is the banana?” carry a lot of the answer.

What if the checks had failed? Symptom: EMAP leaves LXMERT near 70 on VQA 2.0. Measurement: that would say the benchmark does not need the image, which contradicts how it was built. Fix: suspect your code first (Chapter 3’s match-detector test), then the model. Because both checks came out as expected, the tool earns the right to be pointed at harder cases.

On the synthetic dot-product task the linear model scores 52.8% and its EMAP also scores 52.8%. What does that tell you?

Chapter 6

Seven Tasks, Four Models

Meet the unbalanced datasets and the models the paper pits against its own test, and find where each model first lets text meet image

A good test needs a fair field. The paper picks seven real tasks that nobody balanced to force interactions, and four kinds of model that range from a plain sum to a pretrained transformer. Think of it as a tasting with seven dishes and four cooks. The question is not who wins. It is whether any cook used the one trick, combining flavours, that the dish was supposed to need.

The seven tasks

TaskWhat it predictsSourcePairsMetric, folds
I-INTInstagram post intent, 7 classesKruk et al. 20191.3KAUC, 5-fold
I-SEMInstagram semiotic relation, 3 classesKruk et al. 20191.3KAUC, 5-fold
I-CTXInstagram contextual relation, 3 classesKruk et al. 20191.3KAUC, 5-fold
T-VISTwitter “visual-ness” of the text, 4 classesVempala and Preoţiuc-Pietro 20193.9Kweighted F1, 10-fold
R-POPReddit: which of two posts is more popularHessel et al. 201787.2Kaccuracy, 15-fold
T-ST1Twitter sentiment, 2 classesBorth et al. 20130.6KAUC, 5-fold
T-ST2Twitter sentiment, 2 classesNiu et al. 20164.0Kaccuracy, 5-fold

The sizes are the number of image+text pairs the authors could recover. AUC and weighted F1 are scores where higher is better. A k-fold setup splits the data k ways and uses each part once as the test set. Several original papers say interaction should matter. Hessel et al. (2017) wonder whether “the visual and the linguistic interact, sometimes reinforcing and sometimes counteracting each other.” Kruk et al. (2019) write that integrating text and image “requires inference that creates a new meaning.” Vempala and Preoţiuc-Pietro (2019) credit the gap between their additive baseline and their neural model to the interaction. The paper’s goal is not to downplay these tasks, only to test the claim.

Features: frozen summaries

For the first three model types, the paper does not train on raw pixels or words. It first turns each image into one feature vector with a pretrained EfficientNet B4 (ResNet-18 for the three Instagram sets, for reproducibility). It turns each text into a vector by running RoBERTa, then mean pooling: averaging the per-token vectors into one. These summaries are fixed. All learning happens on top of them.

The models, one mixing point each

Linear (additive). A linear model on [t; v], as in Chapter 1, tuned by grid search over the penalty type (L1, L2 or both), its strength (from 10−7 to 10) and the loss (logistic or squared hinge). It has one place where text meets image: the final sum.

Polynomial kernel SVM. A kernel lets a model score two inputs by a formula that implicitly adds new features. A degree-2 polynomial kernel adds every product of two input coordinates, including one text coordinate times one image coordinate. The paper tries degree 2 and 3.

Neural network. Following Chen et al. (2017), an affine layer projects the text and the image to the same size h, giving t and v. The network then reads the glued vector [t; v; v − t; v ⊙ t], where ⊙ multiplies position by position. That is 4h numbers going into a multi-layer feed-forward network with 2 to 4 layers. The last two blocks are built to expose differences and products.

LXMERT. A transformer pretrained to relate pictures and words. The image becomes 36 boxes found by a Faster R-CNN (ResNet-101 trained on Visual Genome). A 5-layer transformer reads the boxes, a 9-layer one reads the words, and a 5-layer cross-modal transformer lets the two attend to each other. Pretraining covers masked region and word prediction, matching, and VQA. It is fine-tuned on each task with a learning rate between 10−6 and 10−4.

LXMERT + linear logits. The paper's own twist. The logits of the trained linear model are fixed and fed into the final classifier layer. LXMERT then has to learn only what the sum is missing: the residual. The hope was that this pushes it toward interactive structure.

Why the product block is an interaction

Take one text number t and one image number v, each 0 or 1, and a model z = wtt + wvv + wp(t × v) + b. The four scores are b, wt + b, wv + b and wt + wv + wp + b. Apply the Chapter 1 test I = z11 + z00 − z10 − z01. Everything cancels except wp. The weight on the product is the interaction. A linear model has no product feature, so its wp is 0 and I = 0. The network’s v ⊙ t block and the kernel’s cross products give the models a way to set wp above zero. Whether training does so is what EMAP checks.

Where do text and image meet?

Warm is the text stream, blue is the image stream, purple is where they can interact. Pick a model. For LXMERT, switch on the linear logits.

A schematic built from the paper’s Section 5 description. Sizes such as the hidden width h and the number of SVM features are not given there, so none are drawn. The 5, 9 and 5 layer counts are the paper’s.

The failure mode here is a comparison that confuses wiring with use. Symptom: the SVM beats the linear model by 4.8 points on T-ST2. Measurement: that gap alone cannot say whether the extra points came from the products or from a smoother function of each input. Fix: ask EMAP, which keeps the smooth unimodal part and removes only the products. Chapter 7 shows the answer.

In the one-number model z = wtt + wvv + wp(t×v) + b, which weight carries the cross-modal interaction, and what is it for the linear model on [t; v]?

Chapter 7

What the Seven Tasks Showed

Read Table 4 the way the paper does: the model wins, then EMAP says where the win came from

A runner beats her best time wearing new shoes. Was it the shoes, or the months of training? To find out, you would put her old shoes back and keep the training. EMAP is the old shoes. It keeps everything a model learned about each input on its own and takes away only the combining.

Step one: the model beats the baseline

Table 4 first lists a constant guess, the previous best published score, and the paper's own unimodal linear models. The authors beat the prior best on most tasks, which they credit mostly to using RoBERTa features. The unimodal baselines are already strong. On I-INT the text-only model scores 89.9 AUC against 90.4 for the linear text+image model. On T-VIS the image-only model scores 47.2 weighted F1 and the text-only 37.6.

Then come the interactive models. In six of seven tasks some interactive model beats the linear one:

Task (metric)Linear (A)Best interactive (I)EMAP of best (A)Gain, I over ALeft after EMAP
I-INT (AUC)90.491.391.1+0.9+0.7
I-SEM (AUC)72.874.474.2+1.6+1.4
I-CTX (AUC)80.981.581.3+0.6+0.4
T-VIS (weighted F1)51.353.451.0+2.1−0.3
R-POP (accuracy)63.764.264.1+0.5+0.4
T-ST1 (AUC)75.675.575.9−0.1+0.3
T-ST2 (accuracy)76.180.980.7+4.8+4.6

The last two columns are our subtraction from the paper’s table. The paper notes that this is where earlier analyses would stop and claim that interactions help. For T-ST2 the polynomial SVM beats the linear model by 4.8 points. That looks like a clear win for modelling the pair.

Step two: the EMAP column

Now the third column. For I-INT, I-SEM, I-CTX, T-ST1, T-ST2 and R-POP, the paper reports that EMAP “results in essentially no performance degradation.” The T-ST2 SVM goes from 80.9 to 80.7. Of its 4.8-point lead over the linear model, 4.6 points (96%) survive with every interaction removed. So the lead did not come from combining the tweet and its picture. The projection is additive but not linear: it can bend each input’s score in a curved way. The SVM’s edge came from reading each modality better, not from reading them together.

The one exception is T-VIS, with LXMERT plus linear logits. It gains 2.1 points over the linear model (53.4 against 51.3). Under EMAP it falls to 51.0, a drop of 2.4 F1, below the linear baseline. Here, and only here in these experiments, modelling the pair seems to help.

How big is a small drop?

A drop of 0.2 could be real or noise. The paper gives a yardstick. For all tasks except T-VIS, the best model and its EMAP choose the same top label in more than 95% of cases on average. Retraining the full model with a different random seed gives about 96% agreement. So the EMAP copy differs from the model about as much as the model differs from itself on a rerun. In the device, set your own threshold for how large a drop must be before you call it interaction.

How much of the win survives?

Each row is a task. The purple bar is the best interactive model’s gain over the linear model. The pink bar is the gain that is left after EMAP. Raise the threshold to call only big drops “interaction”.

1.0

Gains are differences of the paper’s Table 4 entries. The metrics differ by task (AUC, weighted F1, accuracy), so read each row on its own, in points of its own metric.

At a threshold of 1.0 point, only T-VIS is flagged. Pull it down to 0.1 and several tasks light up with drops of 0.2. The paper does not call them interactions. Its yardstick is that a rerun with another seed already changes about 4% of top labels. A threshold has to come from a noise estimate, not from the wish to find something.

What breaks if you stop here? Symptom: “no drop, so the task needs no interactions.” Measurement: Chapter 8 shows additive models cannot compute most logical functions, so some tasks surely do need more. Fix: say what the paper says: these models do not use them on these datasets.

On T-ST2 the SVM beats the linear model by 4.8 accuracy points, and its EMAP scores 80.7 against its own 80.9. What is the best reading?

Chapter 8

How Much Can Additive Models Compute?

See that a sum of two separate scores cannot copy even a two-switch staircase light, and read the three reasons the paper offers for the puzzle

A staircase has a light and two switches, one at the bottom and one at the top. Flip either switch and the light changes state. The light is on when exactly one switch is up. This is the classic XOR (exclusive or) rule. Try to explain it with two separate scores, one per switch. You cannot, and the reason is the whole point of this chapter.

The puzzle the results leave

Chapter 7 ends in tension. Additive models are weak (this chapter shows how weak), and yet on six of seven real tasks an additive twin matches the full model. The authors put it as a conundrum. They also answer a question a skeptical reader would ask: do strong EMAP results mean most functions of interest are additive? Their answer is a short no, with a proof by example.

Why a sum cannot copy XOR

Treat the task as a logic puzzle. A text has n yes/no features, an image has n yes/no features, and the target is any true/false function of all 2n. To keep the question clean, assume the model sees every possible input and has a perfect view of each modality. Ask: can some fT(t) + fV(v), compared with a threshold θ, copy the target on every input?

Start with n = 1: one text bit and one image bit, so four inputs and 16 possible true/false rules. The paper finds that 14 of 16 can be copied by a sum. Only XOR and XNOR (its mirror) cannot. Here is why XOR fails. Let a0, a1 be the text part and b0, b1 the image part. XOR needs a0 + b1 > θ and a1 + b0 > θ. It also needs a0 + b0 < θ and a1 + b1 < θ. Add the first pair: a0 + a1 + b0 + b1 > 2θ. Add the second pair: the same four letters sum to less than 2θ. A sum cannot be both. This is Chapter 1’s equal-diagonals fact, seen from the other side.

Can a sum copy this rule?

Pick a rule with the slider or a preset. The computer tries every text part, image part and threshold on a small grid and shows the best sum it can find. Red marks a wrong cell.

6

Exact for this case, not the paper’s code: parts a0, a1, b0, b1 from −2 to 2 in steps of 1, threshold from −3.5 to 3.5 in steps of 1. The rule number is the truth table read as four bits: the cell (text 0, image 0) is the lowest bit and (text 1, image 1) the highest. The tally of 14 matches the paper.

More variables, no luck

With n = 2 (four input bits) many rules still work. The paper gives a surprising one: (t2 and not v2) or (t1 and t2 and v1) or (not t1 and not v1 and not v2). It looks like it needs the pair, yet a sum copies it. But by n = 3 the picture flips. The authors drew over 5 million random logic circuits on three bits per modality, and none was copied perfectly by an additive model.

Figure 2 pushes further. For 10K random circuits at each problem size, it plots the ranking score (AUC) on the training set. Four models compete. The first is a full AdaBoost (a method that adds up many weak rules). The second is AdaBoost restricted so each weak rule sees only one modality. The third is the EMAP of the target function itself, and the fourth is a random guess. The full one is always at 100%. The additive ones drop fast as variables increase. We read this only as a shape, since the paper gives no table of its values. The moral, in the paper’s words: additive models and EMAPs “have very limited capacity to compute or approximate logical functions of multimodal variables.”

Three reasons for the puzzle

If additive models are so weak, why do they match the full models on real tasks? The paper offers three hypotheses and tests none of them. First, the unbalanced tasks may not need complex cross-modal reasoning. A sum can also pass a benchmark that misses the spirit of the task, as shown when cleverly added examples later break text-only models (Kiela et al., 2020). Second, interactions may be data-hungry: feed-forward networks can need very many examples to learn them (Jayakumar et al., 2020). Third, paradoxically, unimodal models may be too weak: if each modality is poorly read, there is nothing good to combine.

What EMAP cannot say

The paper’s FAQ and conclusion mark out what EMAP can and cannot say. EMAP says nothing about the task itself, only about a specific model on a specific dataset, like model comparison does. It does not mean a task paper should be rejected for lacking interactions, since datasets have other merits such as human studies. At the level of single examples, EMAP lets you compare f(t, v) with its projection to find items where the model computes something non-additive, with caveats. The paper shows such a case on T-VIS in Appendix F. Last, the method extends to video, audio, three or more modalities, or two groups of features in one modality, such as the premise and the hypothesis in language inference.

Add one practical caution of our own, which the paper does not test. The N×N table feeds the model texts and images that were never paired. A model trained only on matched pairs may behave oddly on them. If its logits on such pairs are poor, the averages are, too. Symptom: an EMAP that looks too good or too bad. Measurement: compare with the agreement across seeds. Fix: report the number with its seed noise.

The staircase light is on when exactly one of two switches is up. Why can no text-part-plus-image-part sum with a threshold copy it?

Chapter 9

The Field Guide

Keep the numbers, the terms, the reporting recipe and the code of the paper on one page, and see where it sits in the lecture

The paper makes one move and one request. The move: take a trained image+text classifier, score it on every text paired with every image, and rebuild its predictions as the closest sum of a text part and an image part. The request: report that rebuilt score, called EMAP, next to the score of your best model. If the two match, the model’s extra power did not come from combining the inputs. On six of seven real tasks they matched.

The reporting recipe

The paper's answer to its first FAQ is a list of five things to report. Use the audit below to see what each one lets you say.

Audit a claim

Imagine a paper that says its model learns cross-modal interactions. Tick what the paper reports. The ladder shows how strong the evidence is.

The five items are the paper’s answer to its FAQ 1. The verdict sentences are our reading of what each combination supports, not text from the paper.

  1. A constant or random predictor. It shows how the label mix interacts with the metric. On T-VIS it scores 17.2 weighted F1; on T-ST2, 66.2% accuracy.
  2. The strongest text-only and image-only models. They show how far one modality goes alone.
  3. The strongest additive model. A linear model on [t; v], as in Chapter 1. It shows what two separate readers can do.
  4. The strongest interactive model. LXMERT, a kernel SVM, a network. It pushes the score as high as it goes.
  5. The EMAP of that model. The only item that speaks to whether item 4 used the pair.

The numbers to remember

QuantityValueWhere
Synthetic dot-product task, polynomial SVM99.6% → 49.4%Table 1; chance is 50%
Same task, feed-forward network99.0% → 53.8%Table 1
Same task, linear model52.8% → 52.8%Table 1; additive, so unchanged
VQA 2.0, LXMERT70.3 → 40.5Table 2; constant guess 23.4
GQA, LXMERT60.3 → 41.0Table 2; constant guess 18.1
T-ST2, polynomial SVM80.9 → 80.7Table 4; linear model 76.1
T-VIS, LXMERT + linear logits53.4 → 51.0Table 4; the one task with a real drop
Agreement of model and EMAP on the top labelabove 95%Section 5.1; reruns with another seed give 96%
Model calls for N test pairsN×NAlgorithm 1; 250,000 for N = 500
Rules a sum can copy, one bit per modality14 of 16Section 6, FAQ 5; XOR and XNOR fail
Random 3-bit circuits copied exactly by a sum0 of 5 millionSection 6, FAQ 5

The terms

TermMeaning in one line
Multimodally additivef(t, v) = fT(t) + fV(v): a text part plus an image part.
InteractiveAny f that cannot be written that way.
Partial dependenceA modality’s score with the other modality averaged away.
EMAPThe empirical projection of f onto additive functions: row mean + column mean − grand mean.
LogitsThe d raw class scores before they become probabilities. EMAP works on these.
Late and early fusionCombine at the end (an additive ensemble) or mix inside the model.

A checklist before you claim interactions

  1. Does an additive baseline, trained on the labels, sit in the results table?
  2. Did you run the match test (Chapter 5) on your EMAP code, with a model that must fail?
  3. Did you score all N×N pairs, including mismatched ones, or a random 500-pair sample of a large test set?
  4. Did you project the logits, not the probabilities?
  5. Do you know the seed noise (for the paper, about 96% label agreement between reruns)?
  6. Do you say “this model on this dataset”, not “this task needs no interactions”?

The core as code

python
import numpy as np

def emap(f, texts, images):
    """f(t, v) returns logits of shape (d,).  Returns EMAP logits, shape (N, d)."""
    N = len(texts)
    # Algorithm 1, step 1: N x N forward passes, even on mismatched pairs
    cache = np.stack([np.stack([f(t, v) for v in images]) for t in texts])  # (N, N, d)
    mu     = cache.mean(axis=(0, 1))   # (d,)   grand mean over all N*N pairs
    proj_t = cache.mean(axis=1)         # (N, d) text i with every image
    proj_v = cache.mean(axis=0)         # (N, d) every text with image j
    return proj_t + proj_v - mu          # (N, d) text part + image part - grand mean

# The report: compare the model with its EMAP on the same metric
orig = np.stack([f(t, v) for t, v in zip(texts, images)])   # (N, d)
proj = emap(f, texts, images)
acc  = lambda z: (z.argmax(-1) == labels).mean()
print("model", acc(orig), "EMAP", acc(proj))
print("agree", (orig.argmax(-1) == proj.argmax(-1)).mean())

# Memory: keep running row sums and column sums instead of the whole cache.
# The result is the same and the memory falls from N*N*d to N*d.

For a large test set, call it on random samples of about 500 pairs and average, as the paper does for R-POP and in Table 2.

Where it sits in the field

This paper is a reading for the lecture on interactions and fusion, Interactions and Fusion Models. The lecture asks what kinds of interaction exist. This paper asks the earlier, humbler question: for a given trained model, is there any interaction at all? Another assigned reading sorts interactions into redundancy, uniqueness and synergy, with a precise measure: the partial information decomposition reading. The fusion basics sit in Lecture 5, Fusion. For a model that learns to relate images and text by contrast, see microCLIP, and for large vision-language models see microVLM.

The takeaway. A model that could combine the picture and the caption is not a model that did. To find out, strip the combining out of its predictions and measure what is lost.

Credit

“Does My Multimodal Model Learn Cross-modal Interactions? It’s Harder to Tell than You Might Think!” was written by Jack Hessel (Allen Institute for AI; the work was done at Cornell University) and Lillian Lee (Cornell University), arXiv 2010.06572, October 2020. Read the original at arxiv.org/abs/2010.06572. The projection method builds on Hooker (2004), “Discovering additive structure in black box functions.” Every table value here is from the paper, and every toy is labelled as ours.

Press Present to teach it back, or Teach to draw the derivation on a whiteboard. Then go back to the hero and remove the interactions again, now knowing what the machine just did.

Does My Multimodal Model Learn Cross-modal Interactions? Hessel and Lee, 2020
Read the paper · Back to Lecture 6 · Back to Gleams