Five challenges sort every multimodal system: representation, translation, alignment, fusion and co-learning. Pick any of 12 tasks and see which of the five it needs.
Learn what a modality is, why mixing sight, sound and words is hard, and how a 2017 survey gave the whole field one map.
Tap a task, then tap a challenge. Watch which dots light up. Then we build, piece by piece, what each of the five challenges asks and how researchers answered it.
You need a rough idea of what a neural network does (it turns numbers into other numbers) and nothing else. We build the rest from zero.
Survey by Tadas Baltrušaitis, Chaitanya Ahuja and Louis-Philippe Morency, Carnegie Mellon University (arXiv 1705.09406, version 2, August 2017). This is a reading for MIT MMAI, Multimodal AI, and belongs to the introduction lecture. Every idea is re-taught here in our own words and devices.
The 12 tasks and the dots are Table 1 of the survey, read cell by cell. Short names are ours. The count under each column is the number of dots in it.
Chapter 0
See why a model that reads words, sounds and faces together beats one that reads each alone, and meet the five problems that make it hard
A friend drops a full coffee on the floor. They look at you, shrug, and say: “Oh, that’s just great.” Read on paper, those words are happy. You still know at once that they are not. Their voice is flat. Their face is tight. You took the answer from three signals at the same time.
That small scene is the whole subject of this lesson. A machine that reads only the words gets the wrong answer. A machine that listens to the voice and watches the face as well can get it right. This lesson teaches one paper, a survey (a paper that reads many other papers and sorts them), called “Multimodal Machine Learning: A Survey and Taxonomy”. It was written by Tadas Baltrušaitis, Chaitanya Ahuja and Louis-Philippe Morency at Carnegie Mellon University.
The paper defines a modality as “the way in which something happens or is experienced”. Seeing is a modality. Hearing is one. So is touch. A research problem is multimodal when it uses more than one of them. Think of each modality as one witness to the same event. Each witness has its own senses and its own blind spots.
The paper looks mostly at three witnesses. The first is language, written or spoken. The second is visual signals, such as images and video. The third is vocal signals: the sound of the voice and its prosody (the rise, fall and speed of speech that carry mood). The goal of the field is a model that can process and relate information from several of these at once.
The sentence is “Oh, that’s just great.” Each bar shows how positive (right) or negative (left) one signal sounds. Switch signals on and watch the yellow average move.
Illustrative scores, not data from the paper: words +0.80, voice −0.70, face −0.60. The marker is the plain average of the signals that are switched on. Chapter 6 shows better ways to combine them.
With the words alone, the average is a happy +0.80. Add the flat voice and it falls to +0.05, which is a coin flip. Add the tight face and it reaches −0.17, and the guess flips to sarcasm. No single signal was wrong about what it saw. Each one was only incomplete.
Now try to build this. The words are symbolic: a list of separate tokens such as “oh” and “great”. The voice is a signal: thousands of numbers per second that vary smoothly. A face in video is a grid of pixels that changes 30 times a second. The paper calls this mismatch heterogeneity, meaning the data come in unlike forms. One witness speaks in words, one in sound waves, one in pictures. You must find a way to let them testify together.
Other trouble follows. Each modality has its own kind of noise. A room may be loud while the lighting is perfect. One modality may be missing: the person may say nothing. The links between modalities are often open-ended. Many good sentences can describe one photo, so no single “correct” translation exists.
The paper’s main contribution is a map. Earlier surveys sorted multimodal work into early fusion and late fusion (join the signals early, or join the answers late). The authors say this split is too narrow. They name five core technical challenges. Every system you will meet is working on at least one of them.
The hero at the top of this page shows how these five spread across 12 real tasks. Representation is needed by all 12. Co-learning by 9. Fusion by 7. Translation and alignment by 6 each. Only video description needs all five. That is the paper’s Table 1, and the rest of this lesson takes the challenges one at a time.
This paper is a reading for the MMAI introduction lecture, which uses the same five words to plan the whole course.
Chapter 1
Follow four decades of tasks, from lip reading to image captions, and see why each one forced a new kind of model
Picture a noisy train station. You lean in to hear a friend, and without noticing you start to watch their lips. Your ears get help from your eyes. Researchers noticed the same thing in the 1990s, and it started the field.
The paper opens with a short history. It sorts early multimodal work into four families of applications (jobs the machine must do). Each family has its own story, and each story explains a challenge from Chapter 0.
The oldest example is audio-visual speech recognition (AVSR): turn speech into text using both the sound and the speaker’s lips. It began with the McGurk effect. People hear the syllable /ba-ba/ while they watch a mouth say /ga-ga/. They report a third sound, /da-da/. Hearing and sight blend inside the brain.
Speech researchers took the hint. At the time they used hidden Markov models (HMMs, simple models that move through hidden states, one per sound). So the early AVSR systems were extensions of HMMs.
The goal was better word accuracy everywhere. The experiments showed something narrower. The lips helped most when the audio was noisy. In quiet rooms they added almost nothing. The paper puts it this way: the interactions were supplementary rather than complementary. The two signals held the same information. Two copies made the system tougher, but not smarter.
A listener must tell /ba/ from /ga/. Drag the noise. See where the lips help and where they do not.
A toy model, not data from the paper. Each signal has a clue strength d: sound is 4 × (1 − noise)², lips are a fixed 1. Two independent clues add in quadrature, and accuracy is Φ(d/2) for a two-way choice. The shape matches the survey’s finding that vision helps most when the audio is noisy.
Drag the noise to 0%. Sound alone is right 98% of the time, and adding the lips gives less than one point more. Drag it to 80%. Sound alone falls to 53%, close to a coin flip, while sound plus lips stays at 69%. That is the survey’s point about AVSR in one picture.
The second family grew with the internet. Personal computers and the web filled up with video. Old search worked on keywords typed by people. New problems appeared once researchers tried to search the pictures and sounds themselves: cutting a video into shots, summarising it, and finding an event in it. A US standards agency ran a competition called TrecVid. In 2011 it started multimedia event detection, where the machine must spot an event, such as a birthday party, in a pile of clips. Here a video has frames, sound and often text, and the system must use all of them.
The third family studies how people behave together. In the early 2000s teams recorded meetings and conversations. The AMI Meeting Corpus holds more than 100 hours of video, fully transcribed and annotated. The SEMAINE corpus records talk between speakers and listeners. SEMAINE became the base of the first audio-visual emotion challenge in 2011. Cheap, strong face detection let affective computing (machines that read emotion) grow in the early 2010s. Later rounds added health uses, such as rating depression and anxiety.
A meta-analysis by D’Mello and colleagues, quoted in the survey, found that most work improved when it used more than one modality. The gain shrank when the emotions were natural, not acted. Real feelings are subtler than posed ones.
The newest family joins language and vision. In image captioning the machine writes a sentence about a picture. One reason to want it is to help people who cannot see. The main difficulty is evaluation: how do you grade a sentence when many sentences are right? The authors say this led to visual question answering (VQA). In VQA you ask a question about an image and the machine answers it. A short answer is easier to check. Chapter 4 returns to this problem.
Table 1 of the survey puts all of this in one grid, and it is the hero of this page. Each row is a task. Each column is a challenge. A tick means the task must solve that challenge. Look at the pattern. AVSR needs fusion and alignment: the lips must line up in time with the sound. Speech synthesis needs translation, because it makes one modality from another. Image description needs translation and alignment. Cross-modal hashing, which squeezes data into short codes for fast search, needs only representation and co-learning.
So the table is not decoration. It tells you which parts of this lesson you need for the task you care about. Go back to the hero, pick “Visual QA” and read its row.
Chapter 2
Turn a photo, a word and a sound into vectors, then choose between mixing them into one vector or keeping them apart but tied together
You have a photo of a dog and the word “dog”. A computer sees a block of pixel numbers and a short string of letters. These two things look nothing alike. Yet you want the machine to know they mean the same. The first step is to turn both into the same kind of object, and that is the job of representation.
A representation (also called a feature) is a list of numbers, a vector, that stands for a thing: a photo, a word, a sentence, a clip of sound. The paper uses “feature” and “representation” to mean the same. Think of it as an address. Two things with close addresses should be close in meaning.
For one modality, the story is old and simple. Engineers once designed features by hand. For images, a famous hand-made feature was SIFT. For sound, it was MFCC (numbers that summarise the shape of a sound’s spectrum). For text, it was a count of how often each word appears. Then learning replaced design. Convolutional networks learn image features. Deep networks learn sound features. Word embeddings learn text features from the words around each word. Until recently, though, the usual way to get a multimodal feature was to glue the single-modality features together end to end.
The survey lists what a good multimodal representation needs. Similar concepts should land close together. It should be easy to compute even if one modality is missing. And it should be possible to fill in a missing modality from the others.
The paper splits all methods in two. A joint representation feeds all modalities into one function and gets one vector. A coordinated representation keeps one vector per modality, each made on its own, and adds a rule that ties them together.
The simplest joint representation is concatenation: place the vectors side by side. The paper calls it early fusion. A step up is a neural network. Each modality first passes through its own layers. Then one hidden layer takes all of them and maps them into a shared space. The whole stack can train end to end, which means the error from the final task flows back into every layer. Here is the data flow, with illustrative sizes:
Because such networks need much labelled data, the paper notes a common trick: pre-train each part with an autoencoder (a network that learns to rebuild its own input through a narrow middle). Ngiam and colleagues stacked such autoencoders for each modality and fused them with another autoencoder layer. Silberer and Lapata added a second term to the loss that asks the representation to predict object labels. The drawback is that a plain network handles a missing modality badly.
Graphical models give a second joint route. The most popular are deep Boltzmann machines (DBMs): stacked layers of hidden units in an undirected graph, trained without labels. Srivastava and Salakhutdinov merged an image DBM and a text DBM with one top layer. Because a DBM is generative (it models how the data arise), it copes with a missing modality, and it can even sample the missing one. The cost is that DBMs are hard to train. For sequences, the hidden state of a recurrent network (RNN, a network that reads one step at a time) serves as the joint summary.
Now the other way. Each modality keeps its own space. A constraint pulls matching pairs close. The oldest example, WSABIE, built a linear map for images and one for tags so that a photo and its tag had a higher inner product than a photo and a wrong tag. DeViSE used the same idea with deep image and word embeddings. Kiros and colleagues did it for whole sentences with an LSTM, using a pairwise ranking loss.
Each square is a word vector, each circle a photo vector, all on a circle. Drag the training steps. Matching pairs should pull together, and wrong pairs should part.
A toy of the survey’s ranking idea, not a paper result: four unit vectors in two dimensions, margin α = 0.8, loss summed over word to photo and photo to word, 0.1 learning rate, gradient by finite differences. Start angles are illustrative.
At step 0 the word “car” is closer to the dog photo than to the car photo, so retrieval is wrong. By step 30 the loss reaches zero. Each word sits on top of its own photo (cosine 1.00 and 0.99), and the wrong pairs score only 0.19 and −0.02. The two modalities never shared one vector. They only learned to agree about distances.
The paper then covers structured coordination. Cross-modal hashing squeezes both modalities into short binary codes so that matching items get similar codes. Order embeddings make the space asymmetric, so a photo of “a woman walking her dog” sits below the text “woman walking her dog”, which sits below “woman walking”. And canonical correlation analysis (CCA) finds linear projections that make two views as correlated as possible. Its deep and kernel versions (DCCA, KCCA) bend the projection. CCA mostly captures what is shared, so some later models add a reconstruction term to keep what is private to one modality.
The discussion gives a rule. Use a joint representation when every modality is present at test time. It has done well on AVSR, affect and gesture recognition. Use a coordinated one when only one modality is present at test time, as in retrieval, translation and zero-shot learning. A caveat: joint spaces have been used with more than two modalities, while coordinated spaces have mostly been limited to two.
Chapter 3
Compare a model that looks answers up in a dictionary with one that writes new answers, and see where each breaks
You show a friend who speaks only French a photo and ask: “What is this?” They answer in French. You turned a picture into French words. A computer doing the same job faces a harder question: should it find a ready-made French sentence that fits, or write a new one?
That choice splits the whole challenge of translation. The paper defines it as mapping data from one modality to another. Given an entity in one modality, make the same entity in a different modality. Give it an image, get a sentence. Give it a sentence, get an image. It is a long-studied problem, from speech synthesis to video description. The survey sorts the methods into two kinds.
The authors compare this to the difference between non-parametric and parametric machine learning. One keeps the data. The other keeps a model fitted to the data. Their Figure 2 shows the two flows side by side.
Retrieval-based translation is the simplest. Find the closest sample in the dictionary and return it unchanged. There are two flavours. In unimodal retrieval you search in the space of the source. To caption a photo, you find the most similar photos by visual features, and borrow their captions. Ordonez and colleagues did this with global image features. Devlin and colleagues showed that a plain k-nearest-neighbour search with a vote among the captions competes with fancier generative models. The weakness: two photos can look alike without sharing a good caption, so results often need a re-ranking step.
The fix is to search in a semantic space, a space where both modalities live and nearness means shared meaning. Farhadi and colleagues built one by hand from triples of object, action and scene. Socher and colleagues learned one: a coordinated space of sentences and CNN image features, as in Chapter 2. This is where the representation challenge feeds the translation challenge. A semantic space also works both ways, from images to text and from text to images.
Combination-based translation goes one step further. It retrieves several examples and combines them with rules. Kuznetsova and colleagues retrieved phrases from captions of similar images and joined them with integer linear programming. Gupta and colleagues found the k closest images and built a sentence from phrases in their captions. The rules are usually hand-made, and these methods only work in one direction.
The paper names two limits of all example-based methods. First, the model is the whole dictionary: it is big and slow at test time, although tricks such as hashing help. Second, you cannot expect a good answer to always sit in the dictionary, unless the task is simple or the dictionary is huge.
There are 12 photos: 2 animals, 3 actions, 2 places. Pick a method. For lookup, drag the number of captions in the dictionary. Green means the exact right caption.
A toy, not a paper result. A caption has three details: animal, action, place. Lookup returns the dictionary caption that matches the most details (first one on ties). The template assumes three correct detectors, one for each detail; in real systems each detector needs its own model and training set.
Start with 5 captions in the dictionary. Lookup gets 5 of 12 photos exactly right. The other 7 are each off by one detail, such as “dog runs in the park” for a dog that jumps. Drag the dictionary to 12 and every photo is right, because every photo now has its own caption. Lookup never writes anything new. The template names all 12 at once, but every answer has the same stiff shape.
The paper splits generative methods into three families.
Grammar-based models fix the shape of the answer in advance. They detect high-level concepts first (objects in an image, actions in a video), then fill a template such as subject, verb, object. Barbu and colleagues built sentences of the form “who did what to whom, where and how” for video. Mitchell and colleagues used a tree-based language model to get more varied syntax than a template gives. Strength: the output is almost always grammatical and logical. Weakness: the output is formulaic, and every concept needs its own detector and data.
Encoder-decoder models are the most popular family in the survey. An encoder turns the source into a vector. A decoder turns that vector into the target, one piece at a time. The data flow for captioning looks like this, with illustrative sizes:
The idea first appeared in machine translation, and it carried over to image captioning, video description and even image generation. Encoders are modality-specific: RNNs for sound, CNNs for images, distributional word vectors and RNNs for text. The paper adds that using a coordinated space for the encoder gives better results.
Two problems show up. A single vector must carry the whole image, so long outputs forget the input. The survey says attention, covered in Chapter 5, partly cures this. And Devlin and colleagues noticed that k-nearest-neighbour lookup scored nearly the same as generation. That hints the network may be memorising its training data, not understanding the scene.
Continuous generation models are for streams that run in time, such as text to speech, where the output must stay consistent from one moment to the next. Earlier systems used graphical models such as hidden Markov models and Gaussian process latent variable models. Newer ones use encoder-decoder networks. Owens and colleagues predicted the sound of drumsticks from video, and found that predicting a sound feature and then retrieving the nearest real sound worked best. The survey says even that result was a blend of both kinds. WaveNet, from van den Oord and colleagues, models raw audio directly and improved speech synthesis a lot.
Chapter 4
Score captions by word overlap, by people and by ranking, and see why the survey calls evaluation the field's weak spot
Ask five people to describe one photo of a man on a horse by the sea. You will get five good sentences, and no two the same. “A man rides a horse on the beach.” “A rider gallops along the shore.” “Someone on horseback near the waves.” Which one is the right answer? All of them. So how do you grade a machine that writes a sixth?
The survey spends a full section on this because it hurts translation research more than any other issue. With speech recognition there is one right transcript. With speech synthesis or captioning there are many right answers. The authors say that fixing evaluation “will be crucial for further success”. They describe three ways to grade, each with a flaw.
People are the gold standard. Show a group of them the translations and collect scores. For speech synthesis, raters judge naturalness and give a mean opinion score. For visual speech, they judge realism. For captions, they judge grammar, meaning, relevance, order and detail. Scores usually use a Likert scale (for example, 1 for bad up to 5 for excellent). A different design is a preference study: show two translations and ask which is better.
The flaw is cost. User studies take time and money. They also need care to avoid bias from the fluency, age, gender and culture of the raters.
To get a score without people, researchers borrowed metrics from machine translation: BLEU, ROUGE, Meteor and CIDEr. Each one compares the machine’s sentence with one or more human reference sentences and rewards overlap. The simplest piece, used inside BLEU, is unigram precision: of the words the machine wrote, what fraction also appear in the reference? A word counts only as many times as it occurs in the reference. This is called clipping.
Here is a worked example. The reference is “a man rides a horse on the beach”, eight words. Take the candidate “a horse rides a man on the beach”. It has the same eight words, so unigram precision is 8 of 8, a perfect 100%. Yet it says the horse rides the man. Now take “a rider on horseback near the sea”, which is correct. It has seven words. Only “a”, “on” and “the” match, so its precision is 3 of 7, or 43%. The metric prefers the nonsense.
The reference is “a man rides a horse on the beach”. Switch the grader and watch the captions change places.
Illustrative. The overlap scores are computed live as clipped unigram precision against the single reference. The human ratings are our own, from reading each caption against the photo idea (a man on a horse at a beach). The survey’s evidence is in the cited studies, not in this toy.
Sorted by overlap, the horse-rides-man caption comes first at 100% and the correct “rider on horseback” caption sits last. Sorted by people, the order flips. Across these four captions the two scores have a correlation of −0.58, which is a negative link. In real data the match is not that bad, but the survey cites several studies that find it weak.
The studies are specific. Elliott and Keller found sentence-level unigram BLEU only weakly correlated with human judgement. Huang and colleagues found the correlation of BLEU and Meteor with people was very low for visual story telling. On the MS COCO captioning challenge, the ranking by human judgement did not match the ranking by the automatic scores, and a number of algorithms beat humans on every metric. Last, the metrics only work well when there are many reference sentences, and video description datasets rarely have them.
Hodosh and colleagues proposed a different test. Do not generate. Give the system the photo and a pool of captions, and ask it to rank them. If the correct captions rank high, the system understands the pair. This retrieval proxy, they argue, reflects human judgement better. Models that are generative can also report the likelihood of a given caption, which supports the same test. Video captioning later adopted it.
The survey also credits visual question answering with a role here. It was proposed partly to escape this problem, because a question about an image usually has a checkable answer. It still has ambiguity and question bias: some questions can be answered well from the words alone.
Chapter 5
Line up the pieces of two streams, a recipe with its video or a word with its image region, and see how attention does it without labels
You have a cooking video and a written recipe. Step 3 says “fold in the flour”. Somewhere in the video, a hand folds in flour. Which seconds of video belong to step 3? Finding that match is the challenge of alignment.
The paper defines multimodal alignment as finding relationships and correspondences between the sub-components of instances from two or more modalities. A sub-component is a small piece: a word, a video frame, a region of an image, a recipe step. Other examples in the survey: match the words of a caption to areas of an image, or match a movie to the book chapters it was based on.
The survey sorts alignment into two kinds, by purpose.
The paper splits explicit methods by whether they have alignment labels (examples of correct matches). The unsupervised ones have none. Many borrow ideas from alignment in machine translation and from DNA sequences. To make the job easier, they assume something: the two streams keep their order in time, and some way exists to score how similar two pieces are.
The classic method is dynamic time warping (DTW). Imagine two people saying the same word, one slowly and one fast. A frame-by-frame match would pair the fast speaker’s “ooh” with the slow speaker’s “ah”. DTW lets one stream stretch so that like meets like. It uses dynamic programming (solve a big problem by saving the answers to small ones). It builds a table where each cell holds the cheapest total cost of lining up the two streams up to that point.
A tiny worked case. Stream A is 1, 3, 3 and stream B is 1, 1, 3. Local costs are the absolute gaps. D(1,1) = |1−1| = 0. D(1,2) = 0 + 0 = 0, because B’s second 1 also matches A’s 1. D(2,3) = |3−3| + min(D(1,3), D(2,2), D(1,2)). D(2,2) is |3−1| + min(D(1,2), D(2,1), D(1,1)) = 2. D(1,3) is |1−3| + D(1,2) = 2. D(1,2) = 0 is smallest, so D(2,3) = 0, and D(3,3) = 0. The streams line up at cost 0 even though they differ in length of each plateau. Matching frame by frame would cost 2.
Two streams show the same pattern, but stream B drifts in speed. Drag the drift. The grid shows the cost of pairing each frame of A (across) with each frame of B (down). Darker is cheaper.
A toy, computed live. Each stream has 24 frames of one two-bump pattern; B is the same pattern read through a bending clock. Cost is the sum of absolute gaps along the path, and DTW uses the recurrence above with steps right, down and diagonal.
At zero drift the two ways tie. At drift 0.50 the frame-by-frame pairing pays a cost of 5.1 and DTW pays 1.3. The warped path bends away from the diagonal to follow the speed change. Two limits remain. DTW needs a hand-made similarity measure between modalities, and it is linear, so it cannot model nonlinear links. The survey describes the fix: join DTW with CCA (Chapter 2) so that the method learns the shared space and the alignment together, and the deep version, deep canonical time warping, for nonlinear links.
Other unsupervised work used graphical models. Malmaud and colleagues used a factored HMM to align recipes with cooking videos. Cour and colleagues aligned movie shots to screenplays. Both DTW and graphical models can enforce rules such as no big jumps in time and keep the order, but graphical models need an expert to build them. With labelled data, supervised alignment became possible: Zhu and colleagues trained a CNN to score scene against text, to match books to movies, and Mao and colleagues scored whether a phrase fits an object in an image.
Now the hidden kind. Think of a captioning decoder that writes about a photo. If the encoder squeezes the whole photo into one vector, that vector must hold everything. The decoder will struggle with details. Attention removes the squeeze. The decoder keeps all the image regions and, before each word, looks at the ones that matter.
A worked case with three regions. The scores for “horse” are 2.0, 0.5 and 0.1. Exponentials are 7.39, 1.65 and 1.11, which sum to 10.14. The weights are 0.73, 0.16 and 0.11. The context is mostly region 1, the horse. No one told the model which region shows a horse. The alignment was learned as a side effect of caption training, which is why it is called implicit. The same idea works for words in a sentence, segments of audio, frames of video and parts of an instruction. It also helps in question answering, where it aligns the words of a question with pieces of a text, an image or a video, and gives a view of what the model used.
Karpathy and colleagues took another route: a dot product between a word vector and an image-region vector, learned only from a retrieval task, which still gave a latent alignment.
The survey closes the chapter with three difficulties. Few datasets carry explicit alignment labels. It is hard to design similarity measures across modalities. And one stream may have several valid alignments, or pieces with no match at all.
Chapter 6
Join several noisy witnesses into one answer, and learn when to join early, when to join late and when to use a model built for it
Two witnesses saw a car leave the scene. One stood close and saw clearly. The other stood far away in fog. A good detective does not give them an equal vote. She trusts the clear one more. And if the foggy one is missing, she still has an answer. That is the logic of fusion.
The paper defines multimodal fusion as joining information from several modalities in order to predict an outcome: a class (happy or sad) by classification, or a number (how positive a sentiment is) by regression. It is one of the oldest topics in the field, with work reaching back about 25 years at the time of the survey. The authors give three benefits:
The survey keeps the word “fusion” for methods where the joining happens at the late prediction stage. It admits the border with representation is blurry for deep networks, where learning features and learning the answer happen together. For graphical models and kernel methods the border is clearer. It then sorts fusion into model-agnostic and model-based approaches (Table 5).
Model-agnostic means the recipe does not depend on any machine learning method. You can reuse almost any single-modality classifier or regressor.
To feel the difference between a fair vote and a smart vote, work one case. Each modality gives a score that is the true class (+1 or −1) plus random noise. Noise here means the spread of the random part. The audio has spread 0.6, the video has spread 2.5. A model that uses only the audio is right with probability Φ(1 ÷ 0.6), where Φ is the standard normal cumulative curve. That is Φ(1.67) = 95%. The video alone gives Φ(1 ÷ 2.5) = Φ(0.4) = 66%. A plain average adds the two scores. Its signal is 2 and its noise is the square root of 0.36 + 6.25, which is 2.57. So it is right with probability Φ(2 ÷ 2.57) = Φ(0.78) = 78%. The fused answer is worse than the audio alone, because the foggy witness got an equal vote.
Now weight each score by its signal divided by its noise variance, the weighting the survey mentions under “channel noise”. The audio weight is 1 ÷ 0.36 = 2.78. The video weight is 1 ÷ 6.25 = 0.16. The combined signal-to-noise ratio is the square root of (1 ÷ 0.36 + 1 ÷ 6.25) = 1.71. So Φ(1.71) = 96%: a little better than the audio alone. The extra witness helps once it is believed in proportion to its reliability.
Each modality votes +1 or −1 with random noise added. Drag the noise of each. See when an equal vote helps, and when it hurts.
A toy with two Gaussian clues, computed in closed form, not a paper result. Accuracy of a weighted vote with weights w is Φ((Σ w·1) ÷ √(Σ w²σ²)); the best weights are 1 ÷ σ². It shows the survey’s late-fusion options: averaging, and weighting by channel noise.
Defaults give the numbers above. Slide the video noise down to 0.60 and the equal average becomes the best choice, because both witnesses are now equally good. Tick “Video missing” and both fused bars drop to the audio-only value, which is the late-fusion advantage the paper lists: it still works.
Model-agnostic recipes were not built for multimodal data. The paper lists three families that were.
Multiple kernel learning (MKL) extends the kernel support vector machine. A kernel is a function that scores how similar two data points are. MKL lets each modality have its own kernel, and learns how to blend them. Its loss is convex, so training can find the global best. It was long popular for fusing visual descriptors, and for affect, sentiment and event detection. The drawback: it keeps the training data (the support vectors) at test time, so it is slow and large.
Graphical models are either generative (they model the joint probability, such as coupled and factorial HMMs and dynamic Bayesian networks) or discriminative (they model the conditional probability, such as conditional random fields). Discriminative ones gained ground, trading the joint model for predictive power. Their strength is that they use the time and space structure of the data, let experts build in knowledge, and give models a person can read. They suit AVSR and affect recognition.
Neural networks fuse in a joint hidden layer, as in Chapter 2. For time series, recurrent networks and LSTMs fuse streams step by step: a bidirectional LSTM for audio-visual emotion, an LSTM that fuses the results of audio and face LSTMs, and the multi-view LSTM that models both modality-specific and cross-modality interactions. Their strengths are learning from large data, end-to-end training of representation and fusion, and complex decision boundaries. Their weaknesses are poor interpretability, since you cannot tell which modality a prediction relied on, and a need for large datasets.
The survey ends with three open problems. Signals may not be aligned in time, such as a dense continuous signal beside a sparse event. It is hard to exploit supplementary information, not just complementary information (recall Chapter 1). And each modality has different noise types and levels at different moments, which is the weighting problem the device just showed.
Chapter 7
Use a rich modality to teach a poor one, and recognise a thing you never saw by its name alone
A child has seen horses many times. Then she sees a zebra for the first time, with a caption that says “a horse with black and white stripes”. The next day she points at a zebra and says “zebra”. She never saw a labelled zebra picture. She used what she knew about words to name a picture. This is the heart of co-learning.
The paper defines co-learning as aiding the modelling of a resource-poor modality by using knowledge from a resource-rich one. A modality is poor when it lacks labelled data, has noisy input or has unreliable labels. The survey notes one unusual feature: most often the helper modality is used only during training and is gone at test time. The challenge is task independent. It can improve fusion, translation and alignment models alike.
The survey sorts co-learning by the data you have. Figure 3 shows three cases.
Co-training creates more labelled data when you have few labels. You train a weak classifier in each modality. Each one then labels the unlabelled examples it is most sure about, and those new labels train the other. The seminal case, by Blum and Mitchell, classified web pages from the page text and from the words of the links that point to the page. It needs parallel data by definition. It has since helped statistical parsing, visual detectors and AVSR. The risk: the self-made labels can be biased and cause overfitting.
Transfer learning with parallel data moves the knowledge of one modality’s representation into the other. Multimodal autoencoders and deep Boltzmann machines (Chapter 2) give better single-modality representations, even though only one modality is present at test time. Moon and colleagues moved knowledge from an audio speech recogniser into a lip-reading network, so the lip reader runs with no audio. Arora and Livescu used CCA on acoustic and articulatory data (the position of lips, tongue and jaw) to build better sound features. They used the articulatory data only to build the features, and only the sound features at test time.
Now the two datasets share no instances, only categories. Transfer through a coordinated space still works. Frome and colleagues coordinated CNN image features with word2vec word vectors trained on a separate large text set. The visual model made “more meaningful errors”, confusing a thing with a similar category. Mahasseni and Todorovic regularised a colour-video LSTM using an autoencoder LSTM on 3D skeleton data.
Conceptual grounding means learning the meaning of a concept not only from language but also from sight, sound or even smell. People ground meaning in perception, and the survey argues that learning meaning from text alone may be a poor fit. Grounded vectors helped classify metaphors, and measure how related two words are. They are not always better. They help when the grounding matters to the task, such as pictures for visual words.
Zero-shot learning (ZSL) means recognising a concept without having seen a single example of it. Labelling every possible object is too costly, so this matters. The unimodal version uses attributes (colour, size, shape; or phonemes to recognise a word never heard). The multimodal version uses a second modality where the thing has been seen. Socher and colleagues mapped image features into a word space learned from a different dataset and told seen from unseen classes. Frome and colleagues did it with their coordinated space. Palatucci and colleagues predicted words that a person was thinking, from brain scans, including words never seen in training. Lazaridou and colleagues mapped visual vectors to text vectors with a small network.
Here is the data flow. Train: for each seen class, take photos, map them with a matrix W so that each lands near the class’s word vector. Test: map a photo of an unseen class, then return the word nearest to the result, searching over all words. The unseen word is reachable only because its vector was learned from text.
A map is trained on photos of 6 animals and things. Then it sees 40 zebra photos it has never seen. Each dot is where a photo lands in word space. Green: the nearest word is “zebra”. Drag the photo noise.
A toy of the survey’s zero-shot idea, computed in the page. Words are fixed 2-D vectors we chose (zebra sits near horse, as text would place it). A photo is a 4-number vector made from the word vector plus noise. The map W is least squares from photos of 6 seen classes to their word vectors. A plain classifier trained on those 6 labels can never answer “zebra”.
At noise 0.50, 35 of the 40 unseen zebra photos land nearest “zebra”. A plain classifier that knows only the six training labels gets 0 of 40, since it has no “zebra” to say. A random guess among seven words gets about 6 of 40. Raise the noise to 1.00 and the score falls to 12 of 40, because noisy photos drift toward the neighbouring “horse”. The word space made the new class reachable, and the quality of the photos decides how well you reach it.
In the hybrid case, two modalities with no shared data are linked through a pivot. The best-known example is the Bridge Correlational Neural Network of Rajendran and colleagues. In multilingual image captioning, every image has at least one caption in some language, so the image bridges the languages, even when no sentence pair exists. The same idea serves machine translation and transliteration. A related approach uses a large dataset from a similar task to help one with little data, for example text corpora to guide image segmentation.
Chapter 8
Check the numbers, test yourself on six systems, read the core code and find where to go next
The survey says one thing in five words. Multimodal machine learning has five challenges: representation, translation, alignment, fusion and co-learning. Every system you will meet works on at least one. Representation is needed by all 12 tasks in Table 1. Co-learning by 9, fusion by 7, translation and alignment by 6 each. Only video description needs all five.
| Idea | What it means | Where the survey puts it |
|---|---|---|
| Modality | The way something happens or is experienced: language, visual, vocal. | Section 1, Chapter 0 |
| Joint representation | One vector from all modalities: xm = f(x1, …, xn). Needs all inputs at test time. | Section 3.1, Chapter 2 |
| Coordinated representation | One space per modality, tied by a constraint: f(x1) ∼ g(x2). Works with one input at test time. | Section 3.2, Chapter 2 |
| Example-based translation | Retrieval or combination from a dictionary. The model is the dictionary. | Section 4.1, Chapter 3 |
| Generative translation | Grammar-based, encoder-decoder or continuous. Writes new output. | Section 4.2, Chapter 3 |
| Evaluation | People are the gold standard; BLEU, ROUGE, Meteor and CIDEr correlate weakly; ranking is a proxy. | Section 4.3, Chapter 4 |
| Explicit alignment | The match is the goal. Unsupervised (DTW, graphical models) or supervised. | Section 5.1, Chapter 5 |
| Implicit alignment | A hidden step inside another task. Attention is the main tool. | Section 5.2, Chapter 5 |
| Early, late, hybrid fusion | Join features, join decisions, or both. Work with any unimodal model. | Section 6.1, Chapter 6 |
| Model-based fusion | Multiple kernel learning, graphical models, neural networks. | Section 6.2, Chapter 6 |
| Co-learning | A rich modality helps a poor one. Parallel, non-parallel or hybrid data. | Section 7, Chapter 7 |
| Tasks and dots in Table 1 | 12 tasks. Representation 12, co-learning 9, fusion 7, translation 6, alignment 6. | Section 1, the hero |
Six short descriptions of real systems follow. Pick the challenge each one mainly works on. The answers use the survey’s own sorting.
Read the card, then press the challenge it works on.
The six systems are described from the survey’s text: example-based captioning by nearest neighbour, recipe to video alignment, coordinated ranking loss, late fusion for emotion, lip reading helped by audio, and attention for captioning.
Four short functions hold the survey’s main mechanisms: the ranking loss of a coordinated space, dynamic time warping, reliability-weighted late fusion and the zero-shot map.
pythonimport numpy as np def ranking_loss(t, i_pos, i_neg, margin=0.8): # coordinated space (Ch 2) # t, i_pos, i_neg: unit vectors, shape (d,). Cosine = dot product. return max(0.0, margin - t @ i_pos + t @ i_neg) def dtw(a, b): # explicit alignment (Ch 5) # a: (n,), b: (m,) feature streams. Returns the cheapest total cost. D = np.full((len(a), len(b)), np.inf) for i in range(len(a)): for j in range(len(b)): d = abs(a[i] - b[j]) prev = 0.0 if i == j == 0 else min(D[i-1, j] if i else np.inf, D[i, j-1] if j else np.inf, D[i-1, j-1] if i and j else np.inf) D[i, j] = d + prev return D[-1, -1] def late_fusion(scores, sigmas): # fusion by channel noise (Ch 6) # scores: one number per modality; sigmas: its noise spread. w = 1.0 / np.square(sigmas) # trust = 1 / variance return np.sign(w @ scores) # a missing modality: drop its entry def zero_shot(X_seen, W_seen, x_new, vocab, ridge=1.0): # co-learning (Ch 7) # X_seen: (N, 4) photos; W_seen: (N, 2) their word vectors; vocab: {word: (2,)} M = np.linalg.solve(X_seen.T @ X_seen + ridge * np.eye(4), X_seen.T @ W_seen) y = x_new @ M # photo to word space, (2,) return min(vocab, key=lambda w: np.linalg.norm(vocab[w] - y))
Read each function and name the chapter behind each line. The ranking loss is Chapter 2. The three-way minimum inside dtw is the recurrence of Chapter 5. The weight 1 divided by variance is the fix for the equal vote in Chapter 6. And zero_shot answers with a word that no photo in X_seen ever had.
This is a survey from 2017, and its “recent” work is from before the transformer. Its encoders are CNNs, LSTMs and Boltzmann machines. Its attention is a small add-on to a recurrent decoder. Today’s big systems still fit its map. A model such as CLIP learns a coordinated space for images and text with a contrastive loss, which is the ranking idea of Chapter 2 at large scale (see contrastive CLIP). Self-attention across modalities blends alignment and fusion into one block (see understanding attention). The map has limits that the authors state: the line between representation and fusion blurs for deep networks, and co-learning is not a task but a way of using data.
The survey also runs no experiments of its own. Every claim about what works is a claim about the papers it cites. Treat the toys in this lesson the same way: they show the shape of an idea, not the paper’s numbers.
Now press Present or Teach and explain the five challenges out loud, from memory, with one example of each. Then go back to the map at the top, pick “Visual QA” and explain why it needs the four challenges it has and not translation.
Credit: “Multimodal Machine Learning: A Survey and Taxonomy” by Tadas Baltrušaitis, Chaitanya Ahuja and Louis-Philippe Morency, Carnegie Mellon University, arXiv 1705.09406. This lesson re-teaches its ideas in our own words and devices. The course is MIT MMAI, Multimodal AI, by Paul Liang and the MIT Multisensory Intelligence group.