Quantifying and Modeling Multimodal Interactions

Two inputs can tell you the same thing, different things, or something only the two of them tell you together. This paper measures all three in bits, then uses the answer to pick a model.

Learn how partial information decomposition (PID) splits what two modalities know about a task into redundancy, uniqueness and synergy, and how two estimators compute it on real data.

Pick a task, then slide the label noise. Watch the old one-number measure blur what PID pulls apart. Then we build it piece by piece: bits, the four pieces, the definition, two estimators and the results.

You need no information theory. We build bits, entropy and mutual information from zero in Chapter 1. A little Python helps for the code.

A paper reading in the MIT MMAI follow-along (Multimodal AI, Spring 2026, Paul Liang). The paper: Paul Pu Liang, Yun Cheng, Xiang Fan, Chun Kai Ling, Suzanne Nie, Richard J. Chen, Zihao Deng, Nicholas Allen, Randy Auerbach, Faisal Mahmood, Ruslan Salakhutdinov and Louis-Philippe Morency, NeurIPS 2023, arXiv 2302.12247. It belongs to the lecture Interactions and Fusion Models.

Split what two inputs know

0%

Every number is computed live in your browser by an exact PID solver (the paper's CVX idea, Chapter 5) on a 2 by 2 by 2 table of bits. The four tasks and their values at zero noise match the paper's Table 1 (AND: 0.31 redundancy, 0.50 synergy; XOR: 1.00 synergy). The noise levels are illustrative.

Chapter 0

Two Inputs, Three Different Stories

See why a builder needs to know what kind of information two modalities share before picking a way to fuse them

A doctor reads a tissue slide and a gene report for the same patient. A robot arm watches a puck with a camera and feels it with a force sensor. A voice assistant hears a sentence and reads its words. In each case a model gets two modalities: two different kinds of input, such as pictures, sound, text or numbers. It must predict one label, the answer we want, such as a risk score or a mood.

Now you must choose how the model joins them. You could glue the two inputs together. You could let each one vote and average the votes. You could multiply every feature of one by every feature of the other. You could let the two attend to each other. Papers justify each choice with a story about the data. This paper asks whether we can measure that story first.

Three stories two inputs can tell

Picture two witnesses to the same event. The first story is redundancy: both witnesses saw the same thing, so you learn nothing new from the second. Two cameras on the same road are redundant. The second story is uniqueness: one witness saw something the other missed. A camera that sees the puck and a sensor that feels nothing useful are like this, and the camera is the unique one.

The third story is the strange one. Synergy means neither witness tells you the answer, but the two together do. Think of sarcasm. The words “Great job” alone sound happy. A flat voice alone sounds neutral. Together they say “this was not great.” The meaning exists only in the pair.

The distinction matters because each story calls for a different model. If the inputs are redundant, averaging them is safe. If one is unique, you must not throw it away. If the answer is synergy, a model that only looks at each input on its own can never get it right. The device below makes this concrete.

What each input can predict

Each scene has two inputs, each a single bit, and a label. Bars show how often the best possible guess is right when it sees only X1, only X2, or both. Pick a scene and compare the bars.

Illustrative toy tables, not paper data. Each input bit is fair, the label is a fixed function of them, and “best guess” means picking the likeliest label for each input value, computed exactly from the table.

Read the third scene twice. Each key alone gives a coin flip: 50% right. Both keys give 100%. No amount of staring at one input helps. Yet a model that scores each input separately and averages the scores would stay at 50% forever.

Why the old tools fall short

People have tried to measure these stories before. The oldest idea comes from co-training (training two classifiers that teach each other). It calls two inputs redundant if they become independent once you know the label. It then reads redundancy off a single number, the three-way mutual information I(X1; X2; Y). Chapter 1 defines this term. For now note one fact from the paper: this number can be positive or negative, so it is hard to read.

Other tools measure distances between the predictions of trained models, or run statistical tests on the inputs, or ask people. Others inspect a model that is already trained, using Shapley values or integrated gradients. Those tell you what one model did. They do not tell you what the data demands. The paper wants a number that belongs to the data, before any model exists, in a unit that is the same for every task.

What the paper contributes

The authors bring in partial information decomposition (PID), an older idea from information theory, and turn it into a working toolkit for machine learning. The toolkit has four parts.

  1. Four numbers in bits. Redundancy R, uniqueness U1 and U2, and synergy S. They add up to all the information the inputs hold about the label.
  2. Two estimators. CVX is exact and works when each variable has about 100 values. BATCH uses neural networks and works on images, audio and video.
  3. Checks. Synthetic data with known answers, and human annotators on real benchmarks.
  4. Three uses. Measure datasets, measure what models learn, and pick a model before training. Three case studies cover pathology, mood prediction and robotics. The authors also release trained models from 10 families on 30 datasets.

What a task looks like in this paper

Every task in the paper has the same shape. There are two inputs, X1 and X2, and one label Y. The inputs can be big, such as a video clip and a voice track. The label is a small set of choices, such as “sarcastic” or “not sarcastic”. The dataset is a long list of examples. From that list we can count how often each combination appears, and that count is the distribution p(x1, x2, y): the chance of seeing this pair of inputs with this label.

Everything in the paper is a number computed from that distribution. It is not about a particular neural network. This is why the authors can measure a dataset first and a model second. Two datasets with the same distribution have the same interactions, whichever model you later train on them.

Two failure modes are worth naming now. A builder who guesses wrong may choose a late-fusion average for a synergistic task. The symptom is accuracy stuck near chance on the validation set. The measurement that exposes it is a PID with a large S. The fix is a model that can learn interactions. The rest of the lesson shows how to get that measurement.

The road

The page at the top is the toolkit in miniature. The grey bar is the old measure. The four coloured parts are PID. Each chapter below explains one piece of that picture.

  1. Bits: entropy and mutual information, built from a coin flip.
  2. The four pieces: R, U1, U2 and S, and the rules they must obey.
  3. What the old measure hides: the grey bar, and why it cancels.
  4. The definition: marginal-matching distributions, the trick that pins down the four numbers.
  5. CVX: the exact estimator, a convex program.
  6. BATCH: the neural estimator, with Sinkhorn’s algorithm.
  7. Do the numbers mean anything? Synthetic data, human judges, real benchmarks.
  8. Models and model selection: what models capture, and how to choose one.
  9. Field guide: numbers, a checklist, the core code, and where to go next.
A model scores X1 and X2 separately and averages the scores. On which kind of task can it never beat a coin flip?

Chapter 1

Information, Counted in Bits

Build entropy, mutual information and conditional mutual information from a guessing game, so every later number has a meaning

Your friend picks a card from a deck of eight and you must find it by asking yes or no questions. “Is it in the top half?” cuts the deck to four. Another question cuts it to two. A third finds the card. Three questions, no matter which card. We say the card held 3 bits of uncertainty. A bit is the answer to one fair yes or no question.

If your friend always picks the ace of spades, you need zero questions. If the card comes from a lopsided deck where one card shows up most of the time, you need fewer questions on average. This average is the idea we need.

Entropy: how many questions you still owe

Entropy measures how unsure you are about a variable, in bits. For a variable Y that takes value y with probability p(y), the surprise of one outcome is log2(1/p(y)). A rare event surprises you a lot. Entropy is the average surprise.

H(Y)
The entropy of the label Y, in bits. The yes or no questions you owe on average.
∑y
Add up over every value y the label can take.
p(y)
How likely the value y is. It weights its own surprise.
log2(1 / p(y))
The surprise of y in bits. A fair coin face has p = 1/2, so the surprise is 1 bit.

Fair coin: H = 1/2 × 1 + 1/2 × 1 = 1 bit   (a coin that always lands heads: H = 0 bits)

Mutual information: questions that a clue removes

Now someone whispers a clue X. Maybe X is a photo, a word or a sensor reading. The clue may leave you with fewer questions to ask. The conditional entropy H(Y | X) is the questions you still owe once you have seen X. The drop is the mutual information I(X; Y): how many bits the clue gave you about Y.

I(X; Y)
Mutual information, in bits. It is zero if X tells you nothing. It equals H(Y) if X tells you everything.
H(Y)
Your uncertainty before the clue.
H(Y | X)
Your uncertainty after the clue. The bar reads “given”.

A worked case. Y is a fair bit (H = 1). X copies Y but flips it 10% of the time. After seeing X you are 90% sure. H(Y | X) is about 0.47 bits, so I(X; Y) is about 1 − 0.47 = 0.53 bits. The clue removed a little over half your uncertainty.

The twist: a second clue changes the first

Our task has two clues, X1 and X2. We need one more tool. Conditional mutual information I(X1; Y | X2) asks: how much does X1 tell you about Y once you already know X2? Same drop in uncertainty, but measured after the second clue is in your hand.

This can differ wildly from the plain version. Take XOR, where Y = X1 xor X2. Alone, X1 tells you nothing: I(X1; Y) = 0. But once you know X2, X1 settles Y completely: I(X1; Y | X2) = 1 bit. The device lets you see both numbers side by side.

Before and after the second clue

Pick a task, then add label noise. The top bar is what X1 tells you about Y alone. The bottom bar is what X1 tells you once X2 is known. Compare them.

0%

Illustrative 2 by 2 by 2 tables of fair bits, with the label flipped at the noise rate. Both bars are exact (computed from the table), in bits.

Why log base 2? The logarithm turns multiplication into addition, so independent clues add their bits. Base 2 sets the unit to one fair coin flip. The paper uses bits everywhere. That shared unit is what lets us compare a medical dataset to a sarcasm dataset.

One more fact before we move on. Information about Y can never be negative for one variable: I(X; Y) ≥ 0. Learning something cannot make you less informed. The trouble starts when we ask about three variables at once. Chapter 3 shows that the natural three-way version can go below zero.

One more worked case, this time for entropy itself. A lopsided coin lands heads 90% of the time. Heads has surprise log2(1/0.9) = 0.152 bits and tails has log2(1/0.1) = 3.32 bits. Weight each by its chance: 0.9 × 0.152 + 0.1 × 3.32 = 0.137 + 0.332 = 0.469 bits. A fair coin has 1 bit. The lopsided one is more predictable, so it holds less than half the uncertainty.

For the XOR task, X1 alone gives I(X1; Y) = 0, but I(X1; Y | X2) = 1 bit. What does that mean?

Chapter 2

Four Pieces of One Pie

Cut the information two inputs hold about a label into redundancy, two uniquenesses and synergy, and learn the rules the pieces must obey

Two friends, Ana and Ben, each hold one clue about a hidden number. You want to know how much the pair knows. A pie chart helps. The whole pie is the total information the two clues give about the label, I(X1, X2; Y), written with a comma because X1 and X2 are read together as one input. PID cuts that pie into four slices.

The paper writes these as one picture (its Figure 1): the total I(X1, X2; Y) splits into R, U1, U2 and S, measured in bits.

The rules: five equations

We already know how to measure some things. We can measure what Ana alone tells you, I(X1; Y). We can measure what Ben alone tells you. We can measure what Ana adds once Ben has spoken, I(X1; Y | X2). Any sensible slicing of the pie must match these numbers. The paper lists the matches (its Appendix equations 7 to 11).

R + U1
What X1 tells you alone is the shared slice plus the slice only X1 has.
R + U2
The same for X2.
U1 + S
What X1 adds once X2 is known: its own slice plus the synergy that X2 unlocks.
U2 + S
The same for X2 given X1.
R − S
The old three-way mutual information. Redundancy counts for it and synergy counts against it. Chapter 3 shows why that hurts.

A worked example: AND

Let X1 and X2 be fair independent bits and Y = X1 AND X2. Y is 1 only one time in four, so H(Y) = 0.811 bits. X1 alone tells you I(X1; Y) = 0.311 bits: if X1 = 0 you know Y = 0. The same holds for X2. Both together give the full 0.811 bits, since Y is a function of them.

Now the conditional numbers. Given X2 = 0, Y is always 0, so X1 adds nothing. Given X2 = 1, Y equals X1, so X1 adds 1 bit. Half the time each: I(X1; Y | X2) = 0.5 bits. The same for X2.

R + U1 = 0.311   U1 + S = 0.500   R − S = 0.311 − 0.500 = −0.189

Try U1 = U2 = u. Then R = 0.311 − u and S = 0.500 − u, and R − S is −0.189 for every u. The five equations cannot pick u. They leave one number free. The paper’s definition (Chapter 4) closes that gap. Its answer for AND is u = 0, so R = 0.31 and S = 0.50, matching the paper’s Table 1. The device below applies that definition to any table you build.

Build a task, read its four pieces

The four buttons set the label Y for each input pair (X1, X2). Tap one to flip its label. The slider makes the two inputs agree more often. Watch the pie and the checks.

50%

Toy tables, no noise. “Inputs agree” is the chance that X1 equals X2 (50% means independent). The pieces come from the exact solver of Chapter 5. The checks use values computed from the table, so they hold to rounding.

Try the second preset, OR. You get the same numbers as AND, R = 0.31 and S = 0.50. That is not a coincidence. The two tasks are mirror images with the bit values swapped. Now try XOR and raise “Inputs agree” to 0.95. The synergy shrinks, because inputs that nearly always agree rarely show the pair that makes XOR informative.

Note what the pie is not. It is not a score for how good a model is. It is a property of the data: how the label relates to the two inputs. A model may or may not capture each slice. Chapter 8 measures exactly that.

Here is the same bookkeeping for XOR with fair bits. H(Y) = 1 bit. X1 alone gives I(X1; Y) = 0, and so does X2. Given X2, X1 gives I(X1; Y | X2) = 1. The first pair of rules says R + U1 = 0, so R = U1 = 0, and likewise U2 = 0. The second pair says U1 + S = 1, so S = 1. Here the rules do pin everything down, because both unique parts are forced to zero. AND is the case where they do not.

On the AND task, R + U1 = 0.311 and U1 + S = 0.500. Why can these five rules not give you the four numbers on their own?

Chapter 3

What the Old Measure Hides

See how one signed number, the three-way mutual information, lets redundancy and synergy cancel, and why PID needs four numbers

A bank keeps one number for each customer: income minus debt. Two customers both show zero. One earns nothing and owes nothing. The other earns a million and owes a million. The single number is the same, and the two lives are not. A number that subtracts two things can hide both.

The classic measure of what two inputs share about a label does the same. It is the three-way mutual information, also called interaction information.

I(X1; X2; Y)
The three-way number. Positive means “shared” and negative means the opposite.
I(X1; X2)
How much the two inputs tell each other, ignoring the label.
I(X1; X2 | Y)
How much they tell each other once the label is known.

Co-training and multi-view learning use it as their measure of redundancy. Two inputs are “redundant” if they become independent given the label. Then the second term is zero and the whole number is positive.

The cancel

The paper points out the catch: this number equals R − S, the last of the five rules in Chapter 2. A positive value could mean “a lot of redundancy” or “a little redundancy and even less synergy”. A negative value means synergy beats redundancy. A zero could be a task with nothing in common between the inputs, or a task where redundancy and synergy are both large and cancel.

The paper ties this to an old statistics distinction. Mediation means one feature carries the information already in another feature (R larger than S). Moderation means one feature changes how another relates to the label (S larger than R). The old number can only say which side wins. It cannot say by how much each side is present.

Build a task that shows the problem. Give Y two bits. The first bit is a copy that both inputs see: a pure redundancy task. The second bit is the XOR of the two inputs’ hidden bits: a pure synergy task. Put them side by side in one dataset. The device computes the old number and the PID pie for each.

Same old number, different tasks

Pick a task. The grey bar is the old three-way number (can go left of zero). The coloured bar is the PID pie. Choose the third task and see the grey bar sit at zero while the pie is full.

Toy tables with fair bits and no noise, solved exactly. The two-bit task has X1 = (c, a), X2 = (c, b) and Y = (c, a xor b) for fair independent bits a, b, c. The paper’s Appendix Table 10 shows the same effect on its synthetic data.

For the shared bit: R = 1 and S = 0, so the old number says +1. For the XOR bit: R = 0 and S = 1, so the old number says −1. For both bits at once: R = 1 and S = 1, and the old number says 0. A zero for a task whose inputs carry two bits, all of them either redundant or synergistic. A model builder who read the zero would conclude “nothing interesting here” and be wrong on both counts.

Other measures fail in other ways

The paper tests three more definitions on its synthetic data (Appendix Table 10). I-min can overestimate synergy and uniqueness. WMS is actually synergy minus redundancy, so it can be negative, and it cancels to 0 when the two are equal. CI can be negative and sometimes reports the highest uniqueness for data that is pure synergy. PID is built to avoid these problems: its four numbers are never negative.

This is the key move of the paper. It does not invent a better single number. It splits the one confusing number into pieces that cannot hide each other.

A worked check, step by step

Take the shared bit first: X1 = X2 = Y, a fair bit. The inputs tell each other everything, so I(X1; X2) = 1. Once Y is known, each input is fixed and tells the other nothing new, so I(X1; X2 | Y) = 0. The three-way number is 1 − 0 = +1.

Now XOR. Two fair independent bits tell each other nothing: I(X1; X2) = 0. Once Y is known, knowing X1 fixes X2, so I(X1; X2 | Y) = 1. The number is 0 − 1 = −1. Knowing the label created a link between inputs that had none.

The failure mode has three steps. The symptom is a dataset with a three-way number near zero. The measurement is to ask PID for R and S separately: both could be large. The fix is to stop treating the zero as “no interaction” and to look at all four numbers before choosing a model.

A dataset has I(X1; X2; Y) = 0. What can you conclude?

Chapter 4

The Definition: Search the Worlds That Agree With Your Ledgers

Pin down the one free number with a single optimisation over distributions that match both input-and-label ledgers

Two accountants keep separate ledgers. Ana records every (X1, Y) pair she has seen. Ben records every (X2, Y) pair. A third book would record which X1 went with which X2 in the same example, but neither accountant has it.

What can Ana and Ben work out from their own ledgers? How much X1 tells about Y, how much X2 tells, and how much of that is shared or unique. They cannot work out how the two clues combine, because that lives in the third book. So the paper makes a rule: redundancy and uniqueness may depend only on the two ledgers. Synergy is the one slice that may depend on the third book.

The set of consistent worlds

Write p for the true distribution of (X1, X2, Y), the real world. Call q another world: another joint distribution over the same three variables. Define Δp as the set of all worlds that agree with p on both ledgers.

Δp
The marginal-matching distributions: every world that Ana and Ben could not tell apart from the real one.
q
One candidate world, a full table of probabilities q(x1, x2, y).
q(xi, y) = p(xi, y)
The ledger for input i is copied exactly. The coupling q(x1, x2) is free to differ from p’s.

Now the four slices follow in one line each. Here Iq means mutual information computed in world q, and Ip in the real world.

R
The most the shared slice can be, over all worlds the ledgers allow.
U1, U2
The least that one input adds once the other is known, over those worlds.
S
Total information in the real world, minus the least total information any consistent world has.

Read the synergy line like a story. Among all worlds that match the ledgers, find the one in which the two inputs together know the least about Y. That world has no synergy beyond what the ledgers force. Whatever the real world knows beyond it is synergy. The paper’s Appendix Figure 7 draws exactly this gap.

One search solves all four

Bertschinger and co-authors proved that the same world q* solves all four optimisations. So you search once. A neat way to say it: choose the world in the set where Y is hardest to predict from the pair.

q*
The least informative consistent world. You plug it into the four lines above.
Hq(Y | X1, X2)
How unsure Y stays once you know both inputs, in world q. Maximising it is the same as minimising Iq(X1, X2; Y).

Walk the set yourself

For tasks with two fair-bit inputs and a bit label, the set has just two free dials. Add a number a to every label-0 cell and b to every label-1 cell, with a sign that alternates around the four input pairs. The ledgers do not change, because each sum over X1 or X2 cancels. The map below shows Iq over the whole set. The dot marked p is the real world, at a = b = 0.

Walk the set of consistent worlds

Each coloured cell is a world that obeys both ledgers. Warmer means the pair knows more about Y there. Tap or drag a cell to try a world. The ring marks the real world p. The star marks q*.

Toy 2 by 2 by 2 tables of fair independent input bits. Axes: a (right) shifts the label-0 cells, b (up) the label-1 cells, each from −0.5 to +0.5. Only cells where every probability stays at or above zero are coloured. The star is the lowest coloured cell, checked against the exact solver of Chapter 5.

Pick XOR. The coloured region is a small box, and the star sits at its centre where every one of the eight cells is 1/8: the uniform world. There X1 and X2 tell you nothing about Y, Iq = 0, so S = 1 − 0 = 1 bit. Pick AND. The star is the world where the two inputs always agree: the pair then knows no more than one input does, Iq* = 0.311 bits. The real world holds 0.811, so S = 0.500. This matches the paper’s Table 1.

Real data has more than two values per input, so there are many more dials and no map. You need a solver. The next two chapters give two.

Which part of the real data may redundancy and uniqueness depend on, and which part may synergy also depend on?

Chapter 5

CVX: The Exact Estimator

Turn the search for q* into a convex program on a table of counts, and see what it needs and where it breaks

You have a spreadsheet. Each row is one example, with a cluster number for the image, a cluster number for the text and a label. You count how often each triple appears and divide by the number of rows. Now you have a tensor of probabilities and a search to run over all the tables that share the same two ledgers. With 100 values per input that is about a million dials. You cannot walk it by hand. A convex optimisation program can.

A problem is convex if its landscape is one bowl. A solver that rolls downhill always ends at the single lowest point, never at a false valley. That is why the paper’s first estimator, CVX, is exact: it returns the true q*, not a guess.

The setup, with shapes

Let X1 take n1 values, X2 take n2 values, and Y take ny values. The unknown is a tensor Q of shape (n1, n2, ny) where Q[i, j, k] is the probability of the world q at (X1 = i, X2 = j, Y = k). The data gives two fixed tables:

Add that Q is non-negative and sums to 1. These are all linear constraints. Only the goal is non-linear.

Making the goal convex

We want to maximise Hq(Y | X1, X2), the unsureness of Y given the pair. Bertschinger and co-authors report that handing this to a general optimiser is numerically hard, because such routines do not use convexity. The paper’s fix is to rewrite entropy as a distance. Define an auxiliary table Q̃ that spreads each input pair evenly over the labels: Q̃(x1, x2, y) = q(x1, x2) / |Y|. Then:

Hq(Y | X1, X2)
The quantity we want to make as large as possible.
log |Y|
A constant: the entropy if Y were uniform over its |Y| values.
KL(q ‖ q̃)
The KL divergence: how far q is from the version that spreads Y evenly. It is convex in q, and it is always zero or more.

Check it in one line. KL(q ‖ q̃) = Σ q log(q(x, y) · |Y| / q(x)) = log |Y| − Hq(Y | X). So maximising entropy is minimising KL. The paper’s printed Equation 4 writes “max” over the KL term; since the KL is convex and the solvers need a convex minimisation, the working form is to minimise it. Keep that in mind when you read the paper’s equation.

The linear constraint q̃(x1, x2, y) = q(x1, x2) / |Y| plus the two ledger constraints define a program that the conic solvers SCS, ECOS and MOSEK, driven through CVXPY, accept. The authors found MOSEK the fastest and most stable and ECOS and SCS fine for small problems. Plugging the answer q* into the four formulas of Chapter 4 gives R, U1, U2 and S.

Watch a solver run

The device uses a small solver written for this page. It starts at the real world p, where no synergy is credited. Each step makes the inputs and the label independent, then repairs the two ledgers. The paper uses conic solvers, not this loop. Both aim at the same q*.

Run the exact solver

The line shows the information of the pair about Y in the current world, step by step. The bars show the four PID values at the step you choose. Press Run, or drag the step.

0

Illustrative small tables, solved in your browser by this page’s own loop, not by CVXPY. The final values match the paper’s Table 1 for AND (R 0.31, S 0.50). The two-bit task is the one from Chapter 3 (4 by 4 by 4 table).

Step 0 shows S = 0. The real world is itself consistent with the ledgers, so on its own it earns no synergy. Then the line falls as the solver finds worlds where the pair knows less. Each drop turns into synergy, because S is the gap between the start and where the line ends. On the two-bit task the line falls to 1 bit, and the bars show R = 1 and S = 1.

A quick check on the scale rule. Suppose you have 106 samples. The cube root is 100, so Rice’s rule gives about 100 bins per variable. That is what the paper’s synthetic runs used (at least 106 samples, no more than 100 bins). With 100 bins on each input and 2 labels the tensor Q has 100 × 100 × 2 = 20,000 cells, which a conic solver handles. With 108 samples, many cells hold one sample and the probabilities of order 10−8 make the solvers numerically unstable.

Where CVX breaks

CVX works on discrete data with about 100 values per variable. Real inputs are continuous or huge, so the paper bins them. For a continuous variable, histogramming cuts its range into bins and counts. For images and text, the authors reduce the features with PCA, cluster them into 20 groups and use the cluster ids. Two warnings from the appendix. The number of bins should grow more slowly than the number of samples; Rice’s rule suggests the cube root of the sample count. And with 108 or more samples, a single bin can hold one sample, probabilities near 1/n get tiny, and all three solvers crash.

The real cost is the information lost in binning. In the robotics case study of Chapter 8, the authors report that CVX underestimates two numbers because a high-dimensional time series cannot be described by clusters without losing information. For that kind of data, they built the second estimator.

Why does the paper rewrite H(Y | X1, X2) as log|Y| minus a KL divergence?

Chapter 6

BATCH: Let a Network Hold the Table

Replace the giant table with a small neural network on mini-batches, and keep the ledgers true with Sinkhorn’s algorithm

A pathologist’s slide is a gigapixel image. A video has thousands of frames. You cannot cluster these into 100 neat values without throwing away what matters, and a table of every possible pair of inputs would never fit in memory. The second estimator, BATCH, never builds the table. It learns a small network that can produce the part of the table it needs, a mini-batch at a time. This is the same trick that lets us train on a dataset too big to load at once.

Step 1: a network for the world q

Take a mini-batch of m examples. Write x1[i] for the i-th example’s first input and x2[j] for the j-th example’s second input. Pair every first input with every second input, not only the ones that came together. That gives an m by m grid. For each label y, the grid cell (i, j) holds the weight q̃(x1[i], x2[j], y): a candidate world where input i of one example meets input j of another.

Two encoders f(1) and f(2) turn each input into a feature vector for every label y. Write k for the feature size (our notation). The grid for label y is an outer product, passed through exp so every entry is positive:

Ay
An m by m matrix of positive numbers: the unnormalised weight of each (input 1, input 2) pair under label y. Stacked over all labels it has shape (m, m, |Y|).
f(1)φ(X1, y)
Encoder for the first modality, with weights φ. Output shape (m, k) for each label.
f(2)φ(X2, y)
The same for the second modality. The dot product of row i and row j is a score for the pair.

This looks like attention or a contrastive similarity matrix, and the paper says so. The weights φ are the only thing being learned. Moving φ moves the world.

Step 2: keep the ledgers true

A random A is not a legal world. It must match the two ledgers: q̃(x1, y) = p(x1, y) and q̃(x2, y) = p(x2, y). In a batch, each example has chance 1/m. The ledger for example i and label y says the row of Ay at i must sum to p̂(y | x1[i]) / m. The same holds for each column and the second input.

Where does p̂(y | x) come from? From unimodal classifiers, one per modality, trained first and then frozen. The paper expands each ledger with Bayes’ rule, p(xi, y) = p(xi) p(y | xi). Sampling xi from the dataset already supplies p(xi). Only p(y | xi) is missing, so a classifier fills it in.

Sinkhorn’s algorithm does the repair. It alternates two moves. First, scale each row so it sums to its target. Then scale each column to its target. The first move breaks the second, and vice versa, but each pass leaves the error smaller, and the two converge to a matrix that meets both. Every move is a multiply and a divide, so it is differentiable. The paper unrolls a fixed number of passes and backpropagates through them.

Repair the ledgers with Sinkhorn

A 5 by 5 grid for one label. Bars on the right are row sums, bars below are column sums. The tick on each bar is the ledger target. Drag the passes and watch every bar reach its tick.

0

Illustrative: random 2-number features, a batch of 5, one label, targets chosen so rows and columns have the same total. The paper’s batch is 256 and it unrolls a fixed number of passes; the passes and the maths are the same.

Here is one pass by hand on a 2 by 2 grid. Let A = [[2, 1], [1, 1]], row targets 0.6 and 0.4, and column targets 0.5 and 0.5. Scale the rows: row 1 sums to 3, so multiply by 0.2 to get [0.4, 0.2]. Row 2 sums to 2, so multiply by 0.2 to get [0.2, 0.2]. Now the column sums are 0.6 and 0.4, not 0.5 and 0.5. Scale column 1 by 0.5/0.6 and column 2 by 0.5/0.4. You get [[0.333, 0.25], [0.167, 0.25]]. The row sums are now 0.583 and 0.417, close to 0.6 and 0.4. One more row pass gives column sums of 0.503 and 0.497. The matrix settles.

The failure mode here is a grid that cannot meet its targets. If the row targets add to a different total than the column targets, no scaling can satisfy both. The symptom is bars that never reach their ticks. The measurement is to compare the two totals. The fix is to use calibrated classifiers, so that p̂(y | x1) and p̂(y | x2) agree on how common each label is.

Step 3: the objective, and reading off the numbers

Now ask the network for the world with the least synergy, as Chapter 4 prescribed. The main text maximises the three-way information of the learned world, Iq̃(X1; X2; Y) = Iq̃(X1; X2) − Iq̃(X1; X2 | Y). This equals R in the best world. Because q̃ is just the matrix A, every probability this needs (q̃(xi), q̃(x1, x2), q̃(xi | y), q̃(x1, x2 | y)) comes from summing A over some axes. No sampling is needed in the loss.

After training converges, the paper’s appendix extracts the answers. Let Iq̃ be the information of the pair about Y under the learned world, and Ip the same quantities under the data, from the frozen unimodal classifiers and one multimodal classifier:

U1 = Iq̃ − Ip(Y; X2)   U2 = Iq̃ − Ip(Y; X1)   S = Ip(Y; X1, X2) − Iq̃

Redundancy follows from the Chapter 2 rules: R = Ip(Y; X1) + Ip(Y; X2) − Iq̃. (The appendix prints R as Iq̃ alone, which looks like a slip given its own U and S lines; the check on AND, where Iq̃ = 0.311 = I(X1; Y), gives R = 0.311 only from the sum form.) Note the need for a multimodal classifier for Ip(Y; X1, X2). If it is too weak, S is underestimated, while R and U do not depend on it. The paper says to try stronger models until S stops growing.

One training step, with shapes

Step the BATCH loop

Press Next to move one stage. Each box names a stage, the tensor it produces and its shape. m is the batch size (256 in the paper), |Y| the number of labels, k a feature size.

Following the paper’s Algorithm 1 and Figure 9. Shapes are given with m, |Y| and k; the paper does not fix k, and the encoders are small feed-forward networks.

The full loop: sample a batch, compute A, run Sinkhorn, compute the loss from sums of q̃, take a gradient step on φ, repeat. When it converges, extract R, U and S. The authors note that PID values are not sensitive to hyperparameters as long as training converges. The limit is expressiveness: the learned world is only as rich as the encoders, and the pairing is limited to examples within a batch, which they ease with a large batch.

Why does BATCH run Sinkhorn’s algorithm on the matrix A?

Chapter 7

Do the Numbers Mean Anything?

Check the estimators on data with known answers, then against human judges on eight real benchmarks

You buy a new bathroom scale. Before you trust it, you stand on it holding a 5 kg bag, then a 10 kg bag. If it says 5 and 10, you start to believe it. Then you try it on yourself and compare to the doctor’s scale. The paper tests PID the same way: first on data where the answer is known, then on real data where the best available reference is a person.

Known answers: bits and Gaussians

The first test uses the three bit tasks from Chapter 2. The paper’s Table 1 shows that both estimators recover the exact answers. OR and AND each give R = 0.31, S = 0.50. XOR gives S = 1.00 and zero everywhere else. The device in the hero uses the same tasks.

The second test is a mixture of two Gaussians. The label Y is −1 or +1 with equal chance. The inputs (X1, X2) are drawn around a centre at +μ or −μ. The authors turn the angle of μ and compute PID (CVX, with 106 samples in 50 bins). Two findings stand out. When the centres share a coordinate (angle 0 or π/2), one input tells nothing and the other tells all: uniqueness dominates. Redundancy and synergy peak at the diagonal, π/4.

The second finding matters more. Describe the same data in polar coordinates (r, θ) and PID changes: redundancy drops to 0, and θ alone has no unique information, because its meaning depends on the sign of r. The total information is the same, but the slices differ. PID depends on how you split the data into inputs. It measures the relation between two chosen views, not a fact about the world apart from them.

The third test is a generator with hidden causes. Latent vectors z1, z2 (unique to each input) and zc (shared) are pushed through fixed maps to make X1 and X2 with 100 dimensions each (the latents have 50). The label depends on a chosen mix of the latents, so the true interaction is known by construction. Table 2 holds 10 such datasets. Both estimators put the largest value on the right interaction. They do not match the size of the truth: for the redundancy-only dataset the ground truth is R = 0.58, CVX says 0.16 and BATCH says 0.29. Trust the ranking and the pattern more than the absolute size.

Real data: eight benchmarks

Real datasets have no reference answer. So the authors ask people. Each annotator sees both inputs and the label and rates redundancy, uniqueness and synergy from 0 to 5, with a confidence rating. They use 50 examples per dataset and 3 annotators each, skipping MIMIC and ENRICO that need expert knowledge. They rescale the human numbers to sum to the estimator’s total, so the two can sit side by side. Agreement between annotators is high (Krippendorff’s alpha 0.72 for R, 0.68 for U1, 0.70 for U2, 0.72 for S) and so is their confidence (about 4.3 to 4.5 out of 5).

Four datasets use CVX on clusters (images and text). Four use BATCH (video, audio, medical time series). Explore the paper’s Table 3 below.

Explore the paper’s Table 3

Pick a benchmark. The top bar is the estimator’s split (CVX or BATCH). The bottom bar is the human split, where the paper has one. Both use the same scale, in bits.

Values copied from the paper’s Table 3 (CVX for the first four, BATCH for the last four; human ratings rescaled to the estimator’s total). X1 is the first modality the paper lists for each dataset.

What the table shows:

One limit the authors state: ENRICO shows every interaction at once, so it is hard to tell R from S or U from S. And BATCH and CVX can disagree, because clustering loses information. The numbers are estimates with an error, not readings from a perfect instrument.

Why trust the ranking more than the size? Look at what each estimator can lose. CVX loses detail when it bins or clusters. BATCH loses detail when its networks or its unimodal classifiers are weak. In the paper's Table 2 both read below the known truth, and both still put the largest value on the right interaction. That is why the ranking is the safer reading (our summary, not a claim of the paper). The symptom of a bad estimate is a small total next to a model that clearly predicts well. The check is to compare the sum of the four values with the accuracy of the best model. The fix is finer clusters, or a stronger classifier.

In the Gaussian test, the same data described in polar coordinates gives different PID values than in Cartesian coordinates. What does that teach?

Chapter 8

Which Model for Which Interaction?

Measure what ten fusion models capture, define agreement between data and model, and use it to pick a model before you train

A tailor measures you before cutting cloth. The measurements do not make the suit, but they stop you from cutting a coat for someone half your size. So far we measured the data. Now the paper asks two more questions. Do models capture the interactions the data holds? And can the measurement tell us, before any training, which model to use?

Measuring a model

To measure a model f, train it on the training set, then predict labels for the validation set. That gives a new dataset: the same inputs, but with the model’s predicted labels ŷ. Run PID on that. The result is the interactions the model learned. Because the total of the four slices equals I(X1, X2; Y), their sum also tracks test accuracy. The paper’s suite has ten families, grouped by the story each was designed to tell:

They test each on three specialised synthetic datasets, DR (only redundancy), DU (only uniqueness) and DS (only synergy). Table 4 holds the averages:

One more result ties PID to robustness. When one input is removed, the accuracy drop correlates with that model’s uniqueness for the removed input (ρ = 0.8). It works only one way: a large Ui means a large drop, but a small Ui can still hurt, because the model also lost synergy. Among cases with Ui at most 0.05, the drop correlates with S at ρ = 0.73 and with R at only ρ = 0.01. Lesson: what a model loses when an input goes missing follows its uniqueness and its synergy, and does not follow its redundancy.

Agreement: does the model match the data?

Write the dataset’s interactions as ID for each I in {R, U1, U2, S}, and the interactions the model f captured on it as If(D). The paper defines agreement as the amount the model captures, weighted by how much the dataset needs it.

αI(f, D)
Agreement for one interaction type I. The total agreement α(f, D) adds the four.
ÎD
The share of that interaction in the dataset: how much the dataset needs it, between 0 and 1.
If(D)
How much of it the model captured, in bits.

Across all ten synthetic datasets, total agreement correlates with accuracy at ρ = 0.81 (the paper’s Figure 5). For datasets with mostly synergy, the synergy agreement correlates at 0.97.

Model selection, without training

Now the payoff. You have a new dataset. Compute its PID. Compare its normalised shares to those of each of the 10 synthetic datasets with s(D, D′) = Σ |ÎD − ÎD′|. Take the closest synthetic one and recommend the top 3 models on it. The next device does the same step, using Table 4’s averages as a stand-in for what each model captures.

Pick models by agreement

Set how much of your dataset is redundancy, uniqueness and synergy. Each bar is a model’s agreement, using its average captured R, U and S from Table 4. The three best are marked.

10
10
80

Illustrative. The paper computes I_f(D) for each model on each dataset; here each model’s R, U and S are its averages on the specialised datasets (Table 4), and the three sliders are normalised to shares. The real recipe matches against 10 synthetic datasets.

The paper tests this on unseen data. On 5 new synthetic datasets and 6 real ones, the top three recommended models reach 95% to 100% of the best model’s accuracy: 99.91% (synthetic), 99.78% (MIMIC), 100% (ENRICO), 98.58% (UR-FUNNY), 99.35% (MOSEI), 95.15% (MUStARD), and 100% on the mood dataset MAPS. MIMIC has the largest U1 (0.17), so unimodal models almost suffice. UR-FUNNY and MUStARD carry synergy, and the models that help are the transformer (MULT) and the higher-order ones (MI, TENSOR).

Three case studies

Limits the authors state

The estimators are approximations: clustering and unimodal classifiers add optimisation and generalisation error. Some datasets, such as ENRICO, show every interaction, so R, U and S are hard to separate. And for real data the generating process is never known, so validation leans on human judgement and on downstream results like model selection. Everything in the paper treats two modalities at a time.

A new dataset has PID shares of about 80% synergy. According to the paper’s results, which choice is wise?

Chapter 9

Field Guide: The Paper on One Page

Keep the numbers, the checklist and the core code, and see where the paper sits in the field

Two inputs hold information about a label in four ways: redundancy, uniqueness in each input, and synergy. The old three-way mutual information is R minus S, so it hides both. PID defines the four pieces by searching all worlds that match the two (input, label) ledgers for the least informative one. CVX solves that exactly on a table of up to about 100 values per variable. BATCH solves it with neural networks and Sinkhorn’s algorithm on mini-batches. The numbers describe datasets and models, and they let you choose a model before training.

The numbers worth keeping

WhatValueWhere
AND or OR, fair bitsR 0.31, S 0.50 bitsTable 1
XOR, fair bitsS 1.00 bit, all else 0Table 1
CVX scaleabout 100 values per variableSection 3
BATCH batch size256Appendix B.3
Question answering synergyVQA 2.0: 75% (4.92 of 6.58 bits); CLEVR: 83% (5.16 of 6.19)Table 3
Sarcasm (MUStARD)S 0.30, the largest sliceTable 3
Human agreementKrippendorff alpha 0.72, 0.68, 0.70, 0.72 for R, U1, U2, SSection 4.2
What models capture (average)R 0.41, U 0.37, S 0.21 bitsTable 4
Missing-input drop vs Urho = 0.8Figure 4
Agreement vs accuracyrho = 0.81 over 10 synthetic datasetsFigure 5
Model selection reachtop 3 reach 95% to 100% of the best modelTable 5
Robot pushingBATCH U1 = 1.79, image dominatesSection 4.5

Terms

A checklist for using PID

  1. Fix the two views first. Different splits of the same data give different PID (Chapter 7).
  2. Small discrete data, or features you can cluster to about 100 values? Use CVX. Otherwise use BATCH.
  3. For BATCH, train and freeze one classifier per modality before the estimator runs. Use a strong multimodal classifier for the total, and try stronger ones until S stops growing.
  4. Report values in bits, with the estimator named. Treat sizes as estimates and trust rankings more than absolute values.
  5. Sanity check against a known case: XOR should give S = 1, a copy should give R = 1.
  6. Match your dataset’s shares to the synthetic suite and try the top 3 models first.

The core as code

This is the whole CVX idea for a small table, in about 40 lines of NumPy. It is not the paper’s code (the paper uses CVXPY with conic solvers, and its code is public). It uses the alternating loop from Chapter 5, and it reproduces the paper’s Table 1: AND and OR give R 0.31 and S 0.50, XOR gives S 1.00.

python
import numpy as np

def mi(q, a):                                  # I(A;Y) in bits; q has shape (n1, n2, ny)
    qa, qay = {(0,): (q.sum((1,2))[:, None], q.sum(1)),
               (1,): (q.sum((0,2))[:, None], q.sum(0)),
               (): (q.sum(2).reshape(-1, 1), q.reshape(-1, q.shape[2]))}[a]
    qy = q.sum((0, 1))[None, :]
    m = qay > 1e-15
    return float((qay[m] * np.log2(qay[m] / (qa * qy)[m])).sum())

def cmi(q, given):                             # I(A;Y | B), B at axis `given`
    s = 0.0
    for b in range(q.shape[given]):
        sub = q.take(b, axis=given); pb = sub.sum()      # shape (n_other, ny)
        if pb <= 0: continue
        pa, py = sub.sum(1, keepdims=True), sub.sum(0, keepdims=True)
        m = sub > 1e-15
        s += float((sub[m] * np.log2(sub[m] * pb / (pa * py)[m])).sum())
    return s

def ipf(q, t1, t2, sweeps=25):                # make q(x1,y)=t1 and q(x2,y)=t2 again
    for _ in range(sweeps):
        q = q * (t1 / np.maximum(q.sum(1), 1e-300))[:, None, :]
        q = q * (t2 / np.maximum(q.sum(0), 1e-300))[None, :, :]
    return q

def pid(p, steps=60):                           # p[x1, x2, y] -> R, U1, U2, S in bits
    t1, t2 = p.sum(1), p.sum(0)                  # the two ledgers
    q = ipf(p + 1e-9, t1, t2, 60)
    for _ in range(steps):                        # search for q*
        q = ipf(q.sum(2, keepdims=True) * q.sum((0, 1), keepdims=True), t1, t2)
    U1, U2 = cmi(q, 1), cmi(q, 0)
    return dict(R=mi(p, (0,)) - U1, U1=U1, U2=U2, S=mi(p, ()) - mi(q, ()))

The paper’s BATCH estimator replaces the table q with the encoders and Sinkhorn passes of Chapter 6, and extracts U and S from Iq̃ and the frozen classifiers.

Where it sits in the field

PID joins two lines of work: partial information decomposition from information theory (Williams and Beer, then Bertschinger and co-authors, whose definition this paper uses) and multimodal machine learning. It formalises what contrastive learning (redundancy), modality-specific encoders (uniqueness) and tensor or attention fusion (synergy) were each designed for. It sits in the lecture on interactions and fusion models.

Keep going

The takeaway. Do not ask whether two modalities are useful. Ask in what way. Redundant, unique or synergistic inputs need different models, and you can measure which you have, in bits, before you train.

Credit: Paul Pu Liang, Yun Cheng, Xiang Fan, Chun Kai Ling, Suzanne Nie, Richard J. Chen, Zihao Deng, Nicholas Allen, Randy Auerbach, Faisal Mahmood, Ruslan Salakhutdinov and Louis-Philippe Morency, “Quantifying & Modeling Multimodal Interactions: An Information Decomposition Framework”, NeurIPS 2023, arXiv 2302.12247. The authors release their trained models and code at github.com/pliang279/PID. Now try Teach mode: draw the four pieces for XOR from memory. Then go back to the instrument and predict each bar before you move the slider.

You must fuse a camera and a microphone for a new task. The PID of your data shows S about 0.5 bits and R, U1, U2 all near 0.05. What do you try first?
Quantifying and Modeling Multimodal Interactions
Read the paper · Back to the lecture