One survey sorts all of multimodal machine learning into 3 principles and 6 challenges with 19 subchallenges. Pick a challenge and see what it asks and where we teach it.
Press each of the six challenges in turn. Watch the ring and the list of subchallenges change. Then we build, piece by piece, the idea behind every one of them.
You need a feel for vectors (lists of numbers) and a little probability. We teach the rest from zero.
Based on the survey by Paul Pu Liang, Amir Zadeh and Louis-Philippe Morency (Carnegie Mellon University), arXiv 2209.03430. It is a reading for Lecture 1 of MIT MMAI, Multimodal AI.
The six challenges and their 19 subchallenges are the survey’s taxonomy (Figure 1 and Table 1). The chapter numbers are this lesson’s.
Chapter 0
See why a model that sees, hears and reads needs its own way of thinking, and meet the six questions this survey sorts the field into
Picture a hospital bed. A camera watches the patient’s face. A microphone hears the voice. A sensor on the finger reports the heart rate. A nurse’s note says “patient says they are fine”. A program must decide: is this person in pain?
The face looks calm. The voice sounds tight. The heart rate is high. The note says “fine”. No single signal gives the answer. The answer lives in how they fit together. This scene is our own illustration, but it is the kind of problem the survey in this lesson is about.
The survey, by Paul Pu Liang, Amir Zadeh and Louis-Philippe Morency, starts with one definition. A modality is a way in which something is perceived or expressed. Think of a modality as a channel, the way a radio station and a television station carry the same event in different forms.
The paper’s examples are concrete. Speech and audio come from microphones. Images and video come from cameras. Force and vibration come from touch sensors. Multimodal simply means that more than one of these channels is involved.
The paper adds a useful idea: modalities sit on a line from raw to abstract. A raw modality is close to the sensor, like a speech recording or a camera image. An abstract modality is far from the sensor. Examples are the words extracted from a recording, the objects found in an image, or a concept such as “sentiment intensity”. Walk along that line yourself.
Two lanes start at a sensor. Bar height shows how many numbers describe the signal at each stop, on a log scale. Drag the slider to move each lane one stop at a time toward the abstract end.
The four stop names follow the paper’s examples (speech recordings, images, words from speech, objects from images, sentiment intensity, object categories). The counts are illustrative: one second of 16 kHz audio, one 224 × 224 colour image, 3 words, 5 object labels.
Look at the bars. At the sensor end a one-second recording holds 16,000 numbers and one small image holds 150,528. At the abstract end each lane holds one number. You gain meaning and you lose detail. Every multimodal system must pick where on this line to join the senses.
Over the last years, models learned to understand images and video, to generate pictures from text, and to drive cars and robots. The paper lists embodied agents, self-driving cars, image and video understanding, image and video generation, robotics and healthcare. Each field built its own methods and its own words.
The result was a pile of progress with no map. The authors say the rate of progress made it hard to see the common themes and the open questions. A survey is a paper that reads many papers and arranges them. The arrangement is the contribution here.
This paper is a survey, not a new model. It has no training run and no results table to beat. It runs 36 pages, holds 20 figures, and cites 325 papers. Its single table (Table 1) is the taxonomy, a taxonomy being a tree that sorts methods into named groups. The authors also taught it as tutorials at CVPR 2022 and NAACL 2022, and in the CMU courses 11-777 and 11-877.
The survey rests on three principles. Modalities are heterogeneous: they differ in kind. Modalities are connected: they share information. Modalities interact: together they can produce something neither holds alone. Chapters 1 and 2 teach these.
From the principles the authors derive six core challenges. They say these are understudied in ordinary single-modality machine learning. The hero at the top of this page lets you step through them. In one line each:
Together the six hold 19 subchallenges, counted from the paper’s section headings and Table 1. You do not need to memorise them. You need to see that each has a clear question, and that each question is about a different part of the pipeline.
A survey has no one mechanism to derive, so we do something different. Each chapter takes one idea, builds the smallest working version of it, and lets you run it. Where the paper only names a family of methods, we say so, and we build one honest member of the family. Every toy is labelled as a toy.
This paper is a reading for Lecture 1 of MIT MMAI, the lecture that introduces the course. The survey is its map. Here is the road.
Chapter 1
Measure how far apart two modalities are along six dimensions, and see why one encoder cannot serve them all
You hold a photo of a dog and the sentence “a dog runs on the beach”. Both describe one moment. Now ask a hard question: how would you feed both to one program? The photo is a grid of 150,528 colour values. The sentence is six words. They share a subject and almost nothing else.
The survey calls this the first principle: modalities are heterogeneous. The word means “made of different kinds of stuff”. Think of apples and oranges, but with a ruler. The authors give you six measures, so that you can say how different two modalities are, not only that they are.
The paper stresses that this is a scale, not a switch. Two photos from the same camera of the same view are close to homogeneous, which means made of the same kind of stuff. Two languages that say the same thing are slightly heterogeneous. Language and vision are far more so.
Every multimodal problem mixes several of the six dimensions at once. The authors say the dimensions are complementary and may overlap. Here they are, each with a picture in the mind.
The device below turns the spectrum into numbers. Pick a pair and read the six bars.
Each bar is how different the pair is on one dimension, from 0 (the same) to 1 (nothing in common). Choose a pair and compare how the total grows along the spectrum.
Illustrative scores that I set by hand. Only the order comes from the paper: same camera is close to homogeneous, two languages are slightly heterogeneous, and language with vision is even more so.
The total rises along the spectrum. That is the point of the paper’s ruler. It lets you compare problems before you build anything. A problem with a high total will need more work to bring its modalities into one model.
Heterogeneity is why we cannot push every input into one network as it comes. The standard answer is a unimodal encoder for each modality: a small network that knows that modality’s structure and writes its output as vectors. A common setup looks like this.
The text encoder cuts “a dog runs on the beach” into 6 tokens and looks each up in a table of size d. Its output has shape [6, d]. The image encoder cuts a 224 × 224 photo into 196 patches of 16 × 16 pixels and writes each as a vector. Its output has shape [196, d]. With d = 768, both are now lists of 768-number vectors.
The encoders are different because the elements, the structure and the noise are different. Their outputs are alike in kind, so that later steps can compare them. Later chapters do exactly that.
One dimension is easy to compute: noise. Engineers write the signal-to-noise ratio on a log scale called decibels (dB), because signal strengths vary over many orders of magnitude.
Take a clean typed transcript with signal power 4 and noise power 0.04. The ratio is 4 / 0.04 = 100. The log base 10 of 100 is 2. Ten times 2 is 20 dB. Now take a noisy audio track with signal power 4 and noise power 2. The ratio is 2, the log is about 0.301, and the result is about 3 dB.
transcript: 10 × log10(4 / 0.04) = 20 dB · audio: 10 × log10(4 / 2) ≈ 3 dB
These powers are made-up numbers for the example. They show the failure mode. A model that trusts both inputs equally treats a 3 dB stream like a 20 dB one. The symptom is that adding the audio makes the answers worse than using the transcript alone. You find it by measuring each input’s SNR and by testing the model with one input removed. The fix is to weigh or gate the inputs, which Chapters 3 and 8 return to.
Chapter 2
Separate what two senses share from what they create together, and test each case with a model that uses one sense or both
A friend sends you a photo of a ruined cake with the message “great job”. Read the words alone and they are praise. Look at the photo alone and you see a cake. Put them together and you get a joke about a disaster. The meaning appears only when both are present.
That small story holds the survey’s second and third principles. Modalities are connected: they share things. Modalities interact: they can create things. The paper keeps the two apart on purpose.
A connection is a fact about the data itself. It exists before any model looks at it. The paper says to think about connections from the bottom up (what the numbers show) and from the top down (what we know the things mean). That gives four kinds.
Two of these come from statistics and two from meaning. Chapter 5 uses correspondence to build alignment. Chapter 8 uses association and dependence to measure what a model has learned.
An interaction is different. It does not exist in the data alone. It arises when a system integrates the modalities to answer a task. The paper gives three views of it.
First, the information in an interaction. It is redundant when both modalities carry the same fact. It is non-redundant when it depends on what only one carries (uniqueness) or on something new that appears only when both are present (synergy). The paper cites the decomposition of Williams and Beer.
Take one fair bit for each input and three tasks. If Y copies both inputs, one input already tells you everything: R = 1 bit. If Y copies only input 2, the answer is unique to input 2: U2 = 1 bit. If Y is 1 exactly when the two bits differ (XOR, short for “exclusive or”), neither bit alone says anything about Y, yet both together say everything: S = 1 bit.
copy: R = 1 · only input 2: U2 = 1 · XOR: S = 1 (bits, with fair independent inputs)
Second, the mechanics: how the operator combines elements. It may be additive, non-additive or non-linear, or logical, causal or temporal. Third, the response: how the answer changes with more than one modality. The paper names four. Equivalence: the combined answer matches either alone. Enhancement: it is more confident. Modulation and emergence: it differs from every single-modality answer.
Now test these ideas on a machine. The device below builds three tasks from two bits and trains the best possible lookup rule on text only, image only, and both.
Choose a task. The bars show how often the best rule is right with text only, with the image only, and with both. Then drag the noise on the image bit.
Toy data: 400 samples of two fair bits per task, seeded so the page is the same on every visit. Noise flips the image bit with the chosen chance. The rule is the best lookup table fitted and scored on the same samples, so small gaps from 50% or 100% are chance.
With no noise, the redundant task gives 100% for text alone, image alone and both. Add noise to the image and the image-alone bar falls while text and both stay high. That is the robustness bonus of redundancy. In the unique task the text bar sits near 50%, a coin flip, and only the image bar rises. In the synergy task both single bars sit near 50%, and only the joint bar reaches the top.
This is why a model built from two separate single-modality parts can never solve the synergy task. Each part sees only half of the story. Chapter 3 shows the first tools that can: operators that multiply the inputs instead of adding them.
Chapter 3
Combine two inputs into one vector, and see why adding is not enough when the answer lives in the product
Someone says “great job” in a flat voice. A program has two numbers. One scores the words: +1 if they sound positive. The other scores the tone: +1 if the voice sounds warm. Here the words are +1 and the tone is −1. The program must output one answer, say “does the tone match the words?”, and it should say no.
This is the job of representation fusion, the survey’s first subchallenge. Fusion takes two or more modality inputs and writes one joint representation. It reduces the number of separate representations, as the paper’s Figure 5 shows: more inputs in, fewer out. A representation is just a vector that stands for the input.
The paper sorts fusion by where it happens. In fusion with abstract modalities, each input first goes through its own encoder, so fusion works on tidy, similar vectors. In fusion with raw modalities, you fuse at the start with little preparation, such as joining the raw inputs side by side. This is called early fusion, and it is harder because raw modalities differ more (Chapter 1).
Barnum and colleagues found early fusion can make a model more robust. Gadzicki and colleagues found complex early fusion can beat abstract fusion. The authors then ask the key question, which we will keep: do such models really learn anything that is not just a sum?
Each operator is a small, differentiable block. You can drop it into almost any network. Take two inputs, vectors x1 and x2.
Beyond order two, the paper turns to tensors. A tensor fusion uses the outer product z = x1 ⊗ x2: every entry of x1 times every entry of x2. If x1 has d1 numbers and x2 has d2, the result has shape [d1, d2]. Higher products capture higher-order polynomial interactions.
The price is size. With three modalities of d = 64 numbers each, concatenation gives 3 × 64 = 192 numbers. The outer product gives 64 × 64 × 64 = 262,144. The tensor fusion network of Zadeh and colleagues first appends a constant 1 to each vector so single-modality terms survive, which makes it 653 = 274,625. Size grows exponentially with the number of modalities, so people use low-rank approximations, which store the big table as a sum of a few small pieces.
Multiplicative Interactions (MI), from Jayakumar and colleagues, unify the ladder with a learned bilinear form.
Last comes the gated unit. One input decides how much of the other to let through: z = x1 ⊙ h(x2), where ⊙ multiplies entry by entry and h ends in a sigmoid, so every entry of h is between 0 and 1. The paper notes h is often called “attention weights”. Worked example: x1 = [2, 4] and h(x2) = sigmoid([0, 2]) = [0.50, 0.88]. Then z = [1.00, 3.52]. The second input turned the first entry down by half and left the second almost whole.
Back to the flat voice. With inputs in {−1, +1}, the target “match?” is y = x1 × x2: +1 when the signs agree, −1 when they differ. This is the synergy from Chapter 2. The device fits the best weights by least squares on the four possible input pairs.
Each pair of bars is one of the four input pairs. Grey is the target, pink is what the fused model outputs. Choose a target, then switch the cross term on and off.
Toy: inputs are only −1 or +1, so four points cover every case. On this grid the four basis terms (1, x1, x2, x1 × x2) do not overlap, so each best weight is just the average of y times that term. Error is the root mean square gap.
For the first two targets the cross term is useless: the error is already 0.00. For the match target, additive fusion fits a flat zero and the error stays at 1.00, however long you train. Switch the cross term on and the weight w3 becomes 1.00 and the error falls to 0.00. No amount of data helps an additive model here, because the problem is in the form of the model.
The symptom: your new fusion layer scores the same as two single-modality models added together. The measurement is the one the paper points to. Hessel and Lee showed that complex non-additive models do not necessarily learn non-additive interactions. Build the best purely additive version of your model and compare. If the gap is small, the data may hold no synergy, or the model did not find it. The fix is to check the data first (Chapter 2) and then give the model a real cross term, a gate, or a tensor.
Chapter 4
Pull matching pairs together in a shared space, then split a representation into parts you can read and control
You type “dog” into a photo app and it shows dog photos. How? The word is a short string. The photos are grids of colour. Somewhere the app must have put the word and the photos on one common map, where the word sits close to its photos and far from photos of cars.
Chapter 3 squeezed the inputs into one vector. The survey’s next two subchallenges do the opposite kind of work. Coordination keeps one representation per modality, but links them. Fission goes further and makes more representations than it was given. The paper’s Figure 5 puts the three side by side by counting. Fusion: more modalities than representations. Coordination: equal. Fission: fewer modalities than representations.
Strong coordination demands equivalence. The vector for the word “dog” and the vector for a dog photo should be close. The vector for “dog” and a car photo should be far. A matching pair is a positive pair. A mismatched pair is a negative pair. The paper says the distance is usually cosine distance, or a max-margin loss.
Cosine similarity measures the angle between two vectors and ignores their length. It is 1 when they point the same way, 0 when they are at right angles, and −1 when opposite. Cosine distance is one minus that. Here is a loss that uses it.
Worked example. Place the text “dog” at angle 0°. A dog photo at 30° has cosine 0.866, so d = 0.134: a cheap positive pair. A car photo at 60° has cosine 0.5, so d = 0.5. With margin m = 1 this negative pair costs max(0, 1 − 0.5) = 0.5. Move the car photo to 120°. Now cosine is −0.5, d = 1.5, and the cost is max(0, 1 − 1.5) = 0. Training keeps pushing until every wrong pair is farther than the margin.
Scaled up, this is the recipe behind large image and text models: contrastive learning on image and text pairs, which the paper cites to Radford and colleagues. Tosh and colleagues showed it provably captures the redundant information shared by the two views, but not the non-redundant parts (Chapter 2). A second route skips the contrast and maps one modality into the other’s space, as Socher and colleagues did for zero-shot image classification.
Train a tiny version yourself. Three words and three photos sit on a circle, each as an angle.
Squares are words, circles are photos. Matching pairs have the same letter. Press a train button and watch the pink lines shrink. Change the margin to change how far apart wrong pairs must end up.
Toy: every point is a free angle on a circle, with no encoder behind it. The loss is the one above with cosine distance. Each step moves every angle a small amount downhill (learning rate 0.05, slope found numerically). Start angles are fixed.
Watch the numbers. The average matching distance falls towards 0.00. The closest wrong pair settles at or beyond the margin. The retrieval count rises to 3 of 3: each word has its own photo as the nearest photo.
Strong coordination forces everything to be equal. Partial coordination keeps only the links that matter. Canonical correlation analysis (CCA) finds a straight-line projection of each modality so that the projected values are as correlated as possible, with each new dimension kept at right angles to the others. Order embeddings keep a partial order, such as “animal” above “dog”. Other methods keep hierarchies or relations beyond similarity.
A common failure, standard in contrastive learning and not discussed in the paper, is collapse. If a shared encoder only sees positive pairs, the cheapest way to get zero loss is to map every input to the same point. The symptom: training loss hits zero fast and retrieval drops to chance (1 in N). The measurement: the distance between random wrong pairs goes to about 0. The fix is the second sum in the loss: negative pairs and a margin.
Sometimes you want to separate what a representation holds. Modality-level fission splits a pair into what is shared by both and what is private to each. Disentangled representation learning forces these parts to be independent, each explaining one cause of variation. Shi and colleagues use a mixture-of-experts layer to do this in a multimodal variational autoencoder, and Wu and Goodman use a product of experts.
The shapes make it concrete (sizes are illustrative). Two inputs come in: a text vector of 256 numbers and an image vector of 256 numbers. Fission writes three outputs: a shared vector of 128, a text-private vector of 64, and an image-private vector of 64. Two representations in, three out. You can now read, edit or switch off each part alone.
Fine-grained fission splits further, into clusters or subspaces, using k-means, subspace clustering, dictionary learning or matrix factorization. A related post-hoc tool, EMAP, separates the purely additive unimodal effects from the cross-modal part of any trained model, without retraining it.
Chapter 5
Find which element of one modality matches which element of another, when the pairs are given, hidden or blurred in time
A cooking video shows a cook adding salt at second 44. The spoken recipe says “add the salt” at second 42. A program that reads both streams must decide that the spoken phrase and the shaken salt belong together. Nobody labelled the pair. The two streams also drift: the cook talks fast, then slow.
This is the survey’s second challenge, alignment: finding the connections between the elements of different modalities. The paper warns that it is hard for three reasons. It can depend on long-range links. The pieces (words, utterances) are often ambiguously cut. And a match can be one-to-one, many-to-many, or absent.
When the pieces are clear, such as words in text or object boxes in an image, we have discrete alignment. If we also know which pieces match, that is local alignment, and contrastive learning from Chapter 4 does the job. The paper lists uses: matching books to their movie scripts, matching a referring phrase to an object, and matching image regions to their descriptions.
If we do not know the matches, we need global alignment. The paper points to optimal transport (OT). Picture two piles of sand, the words and the image regions, and a cost for moving sand between each pair. OT finds the cheapest way to move one pile onto the other. The cheapest plan is the alignment.
A tiny worked case. Two words (A, B) and two regions (1, 2), each holding half the sand. Moving costs: A to 1 is 0.1, A to 2 is 0.9, B to 1 is 0.8, B to 2 is 0.2. Plan one sends A to 1 and B to 2: cost 0.5 × 0.1 + 0.5 × 0.2 = 0.15. Plan two sends A to 2 and B to 1: 0.5 × 0.9 + 0.5 × 0.8 = 0.85. OT picks plan one. The paper adds that exact OT is costly, so recent work blends it with neural networks, uses an entropy term to smooth it, and relaxes it to a convex problem.
Many signals have no pieces at all: medical time series, satellite images, MRI scans. Two tools appear in the paper. Warping bends one signal to fit the other. Segmentation cuts a signal into pieces so that discrete methods apply.
The classic warping tool is dynamic time warping (DTW). It compares two sequences while letting one stretch or squeeze in time. It builds a table D. Cell (i, j) holds the cheapest total cost of matching the first i steps of A with the first j steps of B.
Run DTW yourself on two toy signals. B is A played back at an uneven speed.
The top shows signal A (the speech) and signal B (the video), with lines for the matched steps. The square is the cost table, dark where the two match. Drag the stretch, then switch between the two ways to match.
Toy signals of 24 steps: A is three bumps, B is A resampled by a smooth uneven time warp whose strength is the stretch. The cost is the sum of absolute differences along the matched steps. DTW lets a step match once or many times, never backwards.
At stretch 0.00 the two signals are identical, and both ways cost the same. As you raise the stretch, step-by-step matching pairs bumps with the wrong partners and the cost climbs. DTW follows the bend and stays low. The failure is a slow drift: pairs that start together end up further apart. You see it as a cost that grows with the stretch. The fix is to let the match bend.
For segmentation, forced alignment lines up speech sounds with the words of a known transcript. For signals with no natural pieces, clustering turns them into a small set of symbols. VideoBERT clusters video features and trains a masked model on the symbols, as a language model does on words. DALL·E, VQ-VAE and CMCM also use these discrete symbols, made by vector quantization, which snaps each vector to the nearest entry in a codebook.
The last subchallenge learns better representations by modelling the links directly. The paper names three styles.
Joint undirected alignment joins the sequences into one and runs full self-attention over all of it. Every element connects to every other, all with the same dot-product similarity. Cross-modal directed alignment lets one modality ask and another answer: the queries come from the target, and the keys and values come from the source. Graphical alignment uses a graph neural network, so only chosen elements connect, with different edge functions.
The directed style shows clean shapes. Text queries Q [Ttext, d] meet video keys K [Tvid, d]. The scores QKT/√d have shape [Ttext, Tvid]. Softmax makes each row sum to 1, and the output A V has shape [Ttext, d].
Worked example with d = 2: one query q = (1, 0) and three frames with keys (1, 0), (0, 1), (1, 1). The dot products are 1, 0, 1. Divide by √2 to get 0.707, 0, 0.707. The softmax weights are 0.401, 0.198, 0.401. With values (10, 0), (0, 10) and (5, 5), the output is 0.401 × (10, 0) + 0.198 × (0, 10) + 0.401 × (5, 5) = (6.02, 3.98). The word “salt” has pulled mostly from frames 1 and 3. That row of weights is a soft alignment.
Chapter 6
Break a question into small steps over a scene, and see which kinds of step the survey says a model needs
A photo shows a table with orange cubes, blue spheres and one orange sphere. Someone asks: “Are there more orange cubes than blue spheres?” You do not answer in one glance. You find the orange things. You keep the cubes. You count them. You do the same for the blue spheres. Then you compare the two counts.
That chain of small steps is what the survey calls reasoning: combining knowledge, usually in several inferential steps, using the structure of the problem. It rests on the alignment of Chapter 5. The paper splits it into four subchallenges, and each one answers a plain question.
The paper names four shapes. Hierarchical structure builds abstract ideas from simpler ones, as a sentence parse builds a tree. Neural module networks use the grammar of the question to wire together small visual modules. Temporal structure spreads reasoning across time and needs a memory to store and retrieve what came before.
Interactive structure means each decision changes the situation, as in reinforcement learning. The paper separates multimodal-conditional work (the task itself needs language, such as following instructions) from language-assisted work (language helps but is optional, such as reading a manual). Last, discovered structure: when we do not know the shape, search for it with architecture search, memory-attention-composition cells, or policy gradients over a set of modules.
For our table question the structure is a tree. It reads greater( count( and(orange, cube) ), count( and(blue, sphere) ) ). Each node is a small module and each leaf is an attribute. The same few modules answer a different question when you rewire them. That reuse is the point of a hierarchical structure.
Inside the chain, each step holds a value. In most networks it is a dense vector that no person can read. The paper lists three readable options. An attention map is a set of weights over the input, such as which image regions matter. It is partly human-readable and still differentiable. Discrete symbols are things like “cube” or “3”, as in neuro-symbolic work. They are harder to train because a hard choice has no slope. Language is the most readable of all: Zeng and colleagues let separate pretrained models talk to each other in plain words, with no further training.
To keep symbols trainable, many methods use differentiable fuzzy logic. A fact gets a probability between 0 and 1, and a logical AND becomes a product. If an object is orange with probability 0.9 and a cube with probability 0.8, it is an orange cube with probability 0.9 × 0.8 = 0.72. Because multiplication has a slope, the whole chain can be trained from the final answer. The paper notes the gain: interpretable, compositional multi-step reasoning.
Try it. The device runs a small program, one module at a time, on a toy scene where a perception step has already given each of eight objects a soft score for each attribute.
Each column is an object. The bars are the current attention map, from 0 to 1. Choose a question, then press Next step and watch one module act at a time.
Toy scene with made-up perception scores. Square means cube, circle means sphere. Modules: filter (keep an attribute), AND (multiply two maps), count (add up a map), exists (take the largest value), greater (compare two counts).
Run the second question to the end. The soft count of orange cubes comes out near 3, and the soft count of blue spheres near 2.
Notice object 8. It is half orange, half blue, and half cube, so it adds a little to every count. That is the failure mode: a soft count is rarely a whole number. The symptom is an answer like 2.78 where the true count is 3. The measurement is to compare the soft count with a hard count after a 0.5 threshold. The fix is to sharpen the maps, or to train with a loss on the count itself.
A second kind of inference is causal. It climbs from what goes together, to what happens if we act, to what would have happened otherwise. The CLEVRER benchmark of Yi and colleagues asks four kinds of question about videos of colliding objects: descriptive (“what colour”), explanatory (“what is responsible”), predictive (“what happens next”) and counterfactual (“what if”). Causal VQA and Counterfactual VQA edit the question or the image under control and check that the model reacts as a causal model should.
Some answers need facts that no image holds. Multimodal knowledge graphs store concepts as nodes and relations as edges, with images and numbers attached to the nodes. Visual Genome is a dense example, with objects, attributes and relations annotated in images and text. Commonsense benchmarks such as VCR and VisualCOMET ask what people might be doing or thinking. TVQA, MovieQA and MovieFIB test reasoning over time in video and text.
Chapter 7
Tell summarizing, translating and creating apart, then move knowledge from a rich modality to a poor one
A news site has a 20-minute video of a council meeting. An editor wants a 3-line text summary. A blind reader wants a spoken description of a chart. A designer types a sentence and wants an image. In each case the output is a new piece of data, not a label. That is the survey’s fourth challenge, generation: learning a process that produces raw modalities that stay consistent with each other.
The paper sorts generation by one question: how does the amount of information change from input to output? That gives three kinds, which the table sets side by side.
| Kind | Information out vs in | Example from the paper |
|---|---|---|
| Summarization | Less: keep the most salient parts | Text summary of a news video; a short video of a long one |
| Translation | Same: change the form, keep the content | Image to caption; text to image; speech to gesture |
| Creation | More: invent new content from a small seed | Photorealistic images; talking heads; speech and music |
Summarization is either extractive (pick the best pieces of the input) or abstractive (write new text that covers them). Translation is either exemplar-based (retrieve from training data, so the result is faithful but limited) or generative (make anything, with more risk in quality, variety and checking). Creation is the hardest because the output must be conditional on the seed, synchronized across modalities, stochastic (many futures are valid), and auto-regressive over long spans. Producing realistic video, text and audio together, in sync, is still open.
The list of translation pairs in the paper shows how wide the field is: text to image, text to video, audio to image, text to speech, speech to gesture, speaker to listener, and language to pose. Large models now give impressive quality on many of these. Each pair has its own notion of “faithful”, so each needs its own test. Photorealistic images, high-resolution speech and talking heads are examples of creation that now work.
The paper stresses a second difficulty: how to judge the output. User studies are the best test, and they are slow, costly and subjective. Automatic proxy metrics exist, but none is robust across all tasks. The authors also raise misuse: fake news, hate speech, deepfakes and lip-synced fakes.
The fifth challenge starts from a different problem. Your primary modality is poor: few labels, noisy inputs, or unreliable labels. A secondary modality is rich. How can the second help the first? The paper calls this transference, because the transfer gives rise to behaviours the primary modality did not have. It names three routes.
Cross-modal transfer adapts a model trained on the rich modality to the poor one. Prefix tuning is the cleanest case. Keep a language model frozen, and train only a small encoder that turns an image into a few extra vectors placed in front of the text. Illustrative shapes: 2 image-prefix vectors plus 12 word vectors, each of d = 4096 numbers, make a sequence of shape [14, 4096]. Only the encoder learns. Tsimpoukelli and colleagues used this idea to caption images with a frozen language model. Representation tuning goes deeper, with extra attention layers or a gate inside the language model. Multitask models (Perceiver, PolyViT, HighMMT, Gato) share one architecture over many tasks.
Co-learning shares a representation space between the modalities during training. One way uses a shared or coordinated space, as in DeViSE and in image classifiers trained with noisy captions. The other way trains a translation from the primary to the secondary modality, so the model can “hallucinate” the missing one.
Model induction keeps one model per modality and lets them teach each other through their predictions. Its classic form is co-training of Blum and Mitchell: each model labels new examples to enlarge the other’s training set. Dunnmon and colleagues use text to make weak labels for images.
The image model has only a few labeled examples of its own. A text model labels 200 more unlabeled examples for it. Drag the number of own labels, then make the text model noisier.
Toy: each example has one number per view, equal to the class (−1 or +1) times 0.6 plus Gaussian noise, so the two views are independent given the class. The classifier is a midpoint between the two class averages. Every bar is a mean over 300 seeded trials, scored on 300 fresh examples. The text noise here is random, not systematic.
With 4 labels of its own the image model gets about 80%. The text-made labels lift it to about 84%, which is the best this toy view allows. The text model is the teacher here; it scores 84% at noise 0.6. Make the text noisy and the teacher falls toward chance, yet the student stays near the top. The labels only need to be better than a coin flip on average, because the noise pushes both class averages together and leaves the midpoint where it was.
Do not trust that for real systems. The failure mode in practice is a systematic teacher error, such as always mislabeling one kind of image. The student then copies it. You catch it by scoring the student on a small, carefully labeled test set from the primary modality. The fix is to keep a few trusted labels and to check the teacher on slices of the data.
Chapter 8
Measure when an extra modality helps or hurts, and read the open questions the taxonomy leaves behind
A hospital tests a pain model in the lab. It reads face video and voice, and scores well. On the ward, a camera lens gets smudged, and the model gets worse than a model that never used the camera. Nobody measured how much the model leaned on each input, or what happens when one input degrades. The sixth challenge exists to prevent this.
Quantification is the empirical and theoretical study of what multimodal models do and why. The paper wants better robustness, interpretability and reliability. It splits the task in three: measure the heterogeneity, measure the connections and interactions, and measure the learning process.
Modality information: which parts of which modality did the model use? Some methods are interpretable by design, such as concept bottleneck models, sparse linear layers, or decision trees on top of learned features. Others explain a black box afterwards, with gradient maps or with feature attributions such as LIME and Shapley values. A Shapley value shares credit for a prediction among the inputs by averaging each input’s effect over all the orders in which inputs could be added.
Modality bias is an unintended link between input and output, from data collection, modelling or annotation. A known case: in visual question answering, models often answered from the words of the question and ignored the image. Curated benchmarks such as VQA 2.0, GQA and NLVR2 were built to reduce this. The paper also notes social biases that compound across modalities.
Noise topologies describe the typical imperfections of each modality. Images blur and shift. Typed text has typos that follow keyboard layout. Synchronized time series have errors that correlate across steps. Liang and colleagues collected such noise types for many modalities to benchmark robustness. For missing or noisy inputs, people impute them with probabilistic models, autoencoders, translation models or low-rank approximations. The risk is that errors compound, and you must know which input is bad beforehand.
To study connections, researchers visualize joint spaces and attention maps, perturb inputs and watch the output, or build diagnostic sets. Winoground tests whether a model can tell two captions with the same words in a different order which one fits an image. For interactions, they estimate redundancy, uniqueness and synergy (Chapter 2) with mutual information or contrastive estimators. They also edit questions and images in controlled ways, as in Causal VQA, and build tools such as EMAP, DIME, M2Lens and MultiViz to expose interactions in real datasets and models.
Three questions remain. Generalization: does the model work on new modality combinations, or on tasks that use only a few of many sensors? Optimization: multimodal networks overfit easily, and different modalities overfit and generalize at different speeds, so one training schedule for all is a poor fit. Weighting schemes help. Tradeoffs: an extra modality brings more data collection, more model complexity and more ways to fail. The paper asks how to weigh the benefit of each modality against these costs.
The device puts a number on that last question. A text input and an image input each point at the right answer with noise. You fuse them two ways.
The text input has fixed noise 0.6. Drag the image noise upward and watch the two fused bars. One fuses with equal weights. The other weighs each input by how reliable it is.
Exact for this toy, no sampling. Each input is the class (−1 or +1) times 0.6 plus Gaussian noise. Equal weights add the two inputs. Reliability weights multiply each input by 1 over its noise squared, which is the best combination for Gaussian noise. Accuracy is the normal curve evaluated at the signal-to-noise ratio of the fused score.
At equal noise, both fused bars reach 92%, well above 84% for either input alone. Now raise the image noise. Past about 1.04, the equal-weight bar falls below the text-alone bar. At image noise 2.00 it is 72%, while text alone is 84% and the reliability-weighted bar is 85%. The extra modality helped nobody until we weighed it.
This is the survey’s tradeoff in one picture. The failure mode: a fused model that is worse than its best single input. You find it with a simple ablation, scoring the model with each input removed in turn. The fix is to weigh inputs by reliability, learn a gate (Chapter 3), or drop the modality.
The paper closes with open problems, one or two per challenge. In its words, they are unsolved from theoretical, computational and application views.
Chapter 9
Keep the numbers, the terms, the checklist and the core code, then follow the links onward
You can now place any multimodal paper on the survey’s map. The scene from Chapter 0, a bed with a camera, a microphone and a heart-rate sensor, asked for a program that decides whether a patient is in pain. Here is how each chapter spoke to it.
A modality is a way something is perceived or expressed. Modalities are heterogeneous (six dimensions of difference), connected (statistically and semantically), and they interact (redundantly, uniquely or synergistically). From these principles the survey draws six challenges. Representation fuses, coordinates or splits. Alignment links parts, with given pairs, transport, warping or attention. Reasoning composes evidence by structure, concepts, inference and outside knowledge. Generation summarizes, translates or creates. Transference moves knowledge from a rich modality to a poor one. Quantification measures what each model really does.
| What | Number | Where it comes from |
|---|---|---|
| Core principles | 3 | Section 2: heterogeneity, connections, interactions |
| Core challenges | 6 | Figure 1 and Table 1 |
| Subchallenges | 19 | 3 + 3 + 4 + 3 + 3 + 3, counted from Table 1 |
| Dimensions of heterogeneity | 6 | Section 2.1 and Figure 2: element, distribution, structure, information, noise, relevance |
| Kinds of connection | 4 | Section 2.2: association, dependence, correspondence, relation |
| Interaction responses | 4 | Section 2.3: equivalence, enhancement, modulation, emergence |
| The paper | 36 pages | 20 figures, 325 references, arXiv 2209.03430 (version 2, February 2023) |
| Outer product, 3 inputs of 64 | 274,625 | Lesson arithmetic: 65 × 65 × 65 with a 1 appended to each input, against 192 for concatenation |
| Synergy task, one input alone | 50% | Chapter 2 device: a coin flip, while both inputs reach 100% |
| Equal-weight fusion starts to hurt | noise 1.04 | Chapter 8 toy: image noise above 1.04, with text noise at 0.60 |
| Term | Plain meaning | Chapter |
|---|---|---|
| Heterogeneity | Modalities differ in kind, along six dimensions. | 1 |
| Synergy | Information that appears only when both inputs are present. | 2 |
| Fusion | Many inputs become fewer representations. | 3 |
| Coordination | One representation per modality, shaped to be comparable. | 4 |
| Fission | One representation becomes several readable parts. | 4 |
| Optimal transport | The cheapest plan to move one pile of matches onto another. | 5 |
| Dynamic time warping | Match two sequences while one stretches in time. | 5 |
| Neural module | A small step in a question program, such as filter or count. | 6 |
| Prefix tuning | Train only extra input vectors in front of a frozen model. | 7 |
| Co-training | Two models label data for each other. | 7 |
| Noise topology | The typical imperfections of one modality. | 8 |
These functions hold the operators from this lesson. They are short, and they run with NumPy. The comments name the chapter. Each worked example in the text comes out of them: the gate gives [1.00, 3.52], the attention gives [6.02, 3.98], and the outer product of three inputs of 64 has 274,625 entries.
pythonimport numpy as np # Ch 3: fusion operators on two inputs (x1, x2 are vectors or scalars) def additive(x1, x2, w): # z = w0 + w1*x1 + w2*x2 return w[0] + w[1] * x1 + w[2] * x2 def multiplicative(x1, x2, w): # adds the cross term w3 * x1 * x2 return additive(x1, x2, w) + w[3] * x1 * x2 def tensor_fusion(*xs): # outer product, a constant 1 appended to each input z = np.ones(1) for x in xs: z = np.outer(z, np.append(x, 1.0)).ravel() return z # three inputs of 64 numbers gives 65**3 = 274,625 def gated(x1, x2, W): # z = x1 * sigmoid(W @ x2): x2 decides how much of x1 passes return x1 / (1.0 + np.exp(-(W @ x2))) # Ch 4: contrastive coordination; row i of T (text) matches row i of V (image), unit-length rows def contrastive_loss(T, V, margin=1.0): D = 1.0 - T @ V.T # cosine distance for every text/image pair, shape [n, n] pos = np.trace(D) # pull matching pairs together off = ~np.eye(len(D), dtype=bool) neg = np.maximum(0.0, margin - D)[off].sum() # push wrong pairs past the margin return pos + neg # Ch 5: dynamic time warping, D(i,j) = c(i,j) + min(D(i-1,j), D(i,j-1), D(i-1,j-1)) def dtw(a, b): n, m = len(a), len(b) D = np.full((n + 1, m + 1), np.inf); D[0, 0] = 0.0 for i in range(1, n + 1): for j in range(1, m + 1): c = abs(a[i - 1] - b[j - 1]) D[i, j] = c + min(D[i - 1, j], D[i, j - 1], D[i - 1, j - 1]) return D[n, m] # Ch 5: cross-modal directed attention; text asks, video answers def cross_attention(Q, K, V): # Q [T_text, d], K and V [T_vid, d] S = Q @ K.T / np.sqrt(Q.shape[1]) # scores [T_text, T_vid] A = np.exp(S - S.max(1, keepdims=True)); A /= A.sum(1, keepdims=True) return A @ V # output [T_text, d] # Ch 8: weigh inputs by reliability (best for Gaussian noise): w = 1 / sigma**2 def fuse_by_reliability(xs, sigmas): w = np.array([1.0 / s ** 2 for s in sigmas]) return sum(wi * x for wi, x in zip(w, xs)) / w.sum()
This paper is a reading for Lecture 1 of MIT MMAI, the course of the MIT Multisensory Intelligence group. The lecture frames the course. The survey gives the map that later lectures fill in: representation, alignment, reasoning, generation, transference and quantification. See the course schedule for the order.
Read it with its age in mind. The first version appeared in September 2022 and the second in February 2023. It cites work up to 2022, so it predates most of the large multimodal language models of the last years. It reports no experiments of its own, so its claims are about structure, not scores. Its categories overlap, as the authors say, and a modern paper often belongs to several. Use the map to ask questions. Do not use it to force a paper into one box.
Credit: Paul Pu Liang, Amir Zadeh and Louis-Philippe Morency, Carnegie Mellon University. “Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions.” arXiv 2209.03430. This lesson re-teaches their survey. The devices are our own toys, and each says so.
Now press Present or Teach and explain the map from memory: the three principles, the six challenges, and one device for each. Then go back to the six-challenge ring at the top and walk it again.