Introduction to Multimodal AI

Words alone say “wow, great.” A face alone says “annoyed.” Together they say sarcasm, a meaning neither one holds. That gap is what multimodal AI is built to close.

Learn what a modality is, why data from different senses is hard to combine, and the six problems that organise the whole field.

Pick a scene, then switch the Words and the Face on and off. Watch what each one tells a model alone, and what changes when both are on. Then we build, piece by piece, the ideas under it: what a modality is, how data from two senses can be connected, and the six challenges the course is built around.

You need no machine learning for the first chapters. A little Python helps for the code in the last one. We build the rest from zero.

Based on Lecture 1 of MIT MMAI: Multimodal AI (Spring 2026), taught by Paul Liang with Dimitris Bertsimas, Jinhua Zhao and Sang-Gook Kim at the MIT Multisensory Intelligence group. The lecture slides (PDF). We explain every idea in our own words and our own devices; the course’s slides and images are not copied here.

Read the room

The three scenes and their readings (positive, positive, then neutral face with positive words, then positive words with a negative face that means sarcasm) are the lecture’s examples of redundancy, uniqueness and synergy. The +1, 0 and −1 scores are our own simple scale, to make the point that adding them can fail.

Chapter 0

Why One Sense Is Never Enough

See why a model that reads only one kind of data misses most of the world, and where the course points you next

A nurse walks up to a hospital bed. She hears the patient’s voice. She sees a grey face and a hand pressed to the chest. She reads the chart and checks the number on the pulse monitor. No single one of these tells her what is wrong. Together they do.

Now picture an AI system that gets only the chart. It is not stupid. It is half blind. Most of what matters never reaches it.

The lecture opens on this idea. It shows one busy picture of the world, with a robot in the middle and everything around it: spoken words, voice, gestures, vision, tables, sensors, wearables, mobile phones, networks, movement, activities, emotional states, and medical images such as radiology and histology. Each item is a different way that information arrives. The lecture calls the whole picture the multimodal world.

Words first: model, input and modality

Three words carry this whole lesson, so we fix them now. An AI model is a function: numbers go in, numbers come out, like a recipe that turns ingredients into a dish. The numbers going in are the input. The numbers coming out are the output, such as a label, a score or a sentence. Running the model on an input is called inference.

A modality is one way that information is written down or sensed: text is one, a photograph is another, a recording of a voice is a third. Chapter 4 gives the lecture’s exact definition. For now, think of a modality as a channel, the way a radio and a television are different channels to the same news.

Multimodal AI is a model whose input, or output, or both, spans more than one channel. That is all it means. The hard part is the word “together”, and the rest of the course is about it.

Four places where it already matters

The lecture picks four frontiers. Each one is a bundle of modalities that a single-channel model cannot handle.

AI for physical sensing means models inside machines: factories, smart cities, robots. The slide shows two studies. One teaches a robot arm to push a part into a slot by combining vision with touch (Lee et al., ICRA 2019). The other is SmellNet, a large database for recognising real smells from sensor readings (Feng et al., ICLR 2026).

Multimodal generative AI means making media: video, music, art. The example is VideoPoet (Kondratyuk et al., ICML 2024), one language model that takes text, images or sound and produces video, and can also produce sound for a video.

Holistic health asks for the nurse’s view, but at home. The slide’s claim is plain: the majority of medical indicators will not be taken in the doctor’s office. They will come from wearables, speech, movement and daily behaviour, covering physical health, emotional wellbeing and social wellness.

Interactive multimodal agents are programs that act for you on the web or in the world. The slide’s task is “Purchase a set of earphones with at least 4.5 stars in rating and ship it to me.” To do it, an agent must look at web pages (vision), read the instruction (language), and click and type (actions). WebArena and VideoWebArena are the benchmarks the slide cites.

Try the next device. Pick one of the four jobs, then switch senses on one at a time. A sense the job needs, and the model lacks, shows as a red gap.

Give the model its senses

Choose a job. Then switch senses on. A dashed red card is a sense the job needs and the model does not have.

Which senses each job needs is our reading of the lecture’s four example slides (the pictures and the studies they cite), not a measured requirement. The percentage is simply the senses switched on that the job needs, divided by the senses it needs.

Notice what the device cannot show: how the senses are combined. Switching a sense on only gets its data to the door. Making sense of two data types at once is the research problem, and it is why the field exists.

The course in one paragraph

The course has two aims. The first is AI for new modalities: collecting data, modelling it, testing it and deploying it, for signals like gestures, radiology, sensors, wearables and mobile data. The second is multimodal AI proper: connecting language, gestures, sensing and actuation. It runs as three modules over fifteen weeks: an introduction to multimodal AI (data, fusion, representations), multimodal foundation models (alignment, generation, reasoning, agents), and applications (design, manufacturing, cities, transportation). Classes are Tuesday and Thursday, 2:30 to 4 pm. Five homeworks build up from processing your own multimodal data to training fusion models, adapting large multimodal language models, reasoning with reinforcement learning, and building agents. Five paper readings, a midterm and a team research project (groups of two or three) finish the grade: 35% homework, 15% readings, 15% midterm, 35% project. The course site is mit-mi.github.io/mmai-course/spring2026. That is all the logistics you need.

The road

The rest of this lesson follows the lecture in order: a short history, then a definition, then the six challenges. The lecture’s three assigned readings come in where they fit.

  1. Gestures and the McGurk effect: the behavioural era, where two senses disagree and the mind invents a third.
  2. Put that there: the computational and interaction eras, from voice plus pointing to meeting rooms.
  3. Deep learning: the 2010s and the tasks that followed.
  4. What a modality is, and why modalities differ.
  5. Connected modalities: shared information, counted in bits.
  6. Interacting modalities: redundancy, uniqueness, synergy.
  7. Representation and alignment: challenges one and two.
  8. Reasoning and generation: challenges three and four.
  9. Transference and quantification: challenges five and six.
  10. The field guide: the numbers, the terms, the code, the readings.
An agent must buy earphones with at least 4.5 stars from a shop website. Which set of modalities does the lecture’s example point to?

Chapter 1

When Eyes Overrule Ears

Watch two senses disagree, and see how the mind combines them into a third thing neither one said

Imagine a film with a dubbing mistake. On screen, a man’s lips close and open as if saying “ga”. The sound track plays “ba”. You do not hear “ba”, and you do not hear “ga”. You hear “da”. Close your eyes and the sound turns back to “ba”.

This is the McGurk effect, named after Harry McGurk and John MacDonald, who reported it in 1976 in a short paper called “Hearing lips and seeing voices” (Nature 264). It is the lecture’s first example of multimodal research, and the lecture labels this stretch of history the “behavioral” era: the 1970s until the late 1980s. In this era, researchers studied how people use several senses at once. Computers came later.

Gestures are thought, in action

The other thread on the slide is the psychologist David McNeill, who studied how people move their hands as they speak. The slide quotes a summary of his view: gestures are “in effect the speaker’s thought in action, and integral components of speech, not merely accompaniments or additions.” In plain words: the hand movement is not decoration for the sentence. It carries part of the meaning.

The two examples make the same point from two sides. A listener uses the speaker’s lips. A speaker uses the hands. Language, in real life, is not just a string of sounds or letters. It is a package, and the package spans several senses. Any model that sees only the transcript is reading the package with half the contents missing.

A model of the McGurk effect, built from scratch

Why “da”? A small calculation explains it. We need one idea first: a probability is a number from 0 to 1 that says how strongly we believe something. A likelihood is a score that answers a different question: “if the sound really were ‘ga’, how often would my ears report what they just reported?” Each sense gives one likelihood for each possible syllable.

Take three syllables: ba, da and ga. The ears hear “ba”, but ears are noisy, so they say: ba 0.70, da 0.20, ga 0.10. The eyes see a mouth shaped for “ga”. A mouth that is open cannot be saying “ba” (the lips must close for that), and “da” and “ga” look alike on the lips. So the eyes say: ba 0.02, da 0.40, ga 0.58.

To combine two independent witnesses, multiply their scores, syllable by syllable, then rescale so the three add to 1. The multiplication is the key step. A syllable must be plausible to both witnesses to survive.

P(s | ears, eyes)
How strongly you believe the syllable is s, once both senses have reported. This is what you perceive.
A(s)
The ears’ score for syllable s: how well s explains the sound.
V(s)
The eyes’ score for syllable s: how well s explains the lips.
Σs′ A(s′) · V(s′)
The sum of the three products. Dividing by it makes the three beliefs add up to 1.

ba: 0.70 × 0.02 = 0.014   da: 0.20 × 0.40 = 0.080   ga: 0.10 × 0.58 = 0.058   sum = 0.152

Divide each product by 0.152. Belief in “ba” falls to 9%. Belief in “ga” is 38%. Belief in “da” is 53%, the winner. The ears alone said “ba” (70%). The eyes alone said “ga” (58%). The pair said “da”, a syllable that neither sense put first. You can reproduce this below, and break it.

Combine ears and eyes

Choose the sound and the mouth shape. Each column is a belief over ba, da and ga. Then lower the picture quality and watch the eyes lose their vote.

Sound
Mouth
100%

The likelihood tables are our own illustration of the effect, with the same shape as the example above (ears are noisy; a closed-lip “ba” is easy to see; “da” and “ga” look alike). They are not measured data. Picture quality blends the eyes’ scores toward an equal 1 in 3 for every syllable.

Set the mouth to “ba” and the sound to “ga”. The lips are so clear about “ba” that they win (67%). That is the seed of a real engineering rule: a combined model should trust each sense in proportion to how reliable it is right now. Slide the picture quality to 0% and the eyes stop contributing, so the answer follows the ears. When quality is 100% and the lips are clear, the answer follows the lips.

The failure mode, as a symptom, a measurement and a fix

The symptom: a speech model that reads lips gets worse, not better, when the video is blurry. The measurement: the eyes’ scores flatten toward 1 in 3, but a model that trusts the picture at a fixed weight keeps multiplying them in, and the noisy vote drags the answer. The fix is the slider you just used: estimate how reliable each input is for this example, and weight it. Modern models learn this weighting. Chapters 7 and 9 return to it.

Five eras, one timeline

Slide 26 lays out the lecture’s map of the field. The slide’s heading says four eras, but its list names five, and the last two are still going. The “behavioral” era (1970s to the late 1980s) is where we are now. The “computational” era runs from the late 1980s to 2000. The “interaction” era covers 2000 to 2010. The “deep learning” era starts in the 2010s, and the “foundation model” era starts in the 2020s. The next two chapters take the middle eras one at a time. The slides credit this history to a deck co-developed with Louis-Philippe Morency for CMU course 11-777.

In the model above, the ears say “ba” (70%) and the eyes say “ga” (58%), but you perceive “da”. Why?

Chapter 2

“Put That There” and the Meeting Room

Follow computers as they learn to listen and look at the same time, from one pointing finger to rooms full of sensors

It is 1982, in a dark room at MIT. A man sits in a chair facing a wall-sized screen with a few coloured shapes on it. He says “put that”, points at a blue circle, says “there”, and points at an empty corner. The circle slides to the corner. The system is called Put That There, built by the Architecture Machine Group at MIT. The lecture uses it to open the second era of multimodal research.

The lecture calls this the computational era, from the late 1980s to 2000. The first work was about multimodal interfaces: ways to operate a computer by using more than one channel at once, such as voice plus pointing. This field is part of human-computer interaction (HCI), the study of how people operate machines.

Why neither channel is enough alone

Look at what the man said. “Put that there” has a verb (put) and two blanks. The word “that” does not say which object. The word “there” does not say where. Words like these, whose meaning depends on a point in the room, are called deictic words. The voice cannot fill the blanks. The finger cannot fill the verb: pointing at a circle could mean move it, copy it or delete it. Each channel is incomplete, in a different place. Together they are complete.

Count it. Suppose the screen has three shapes, three spots, and the system knows three verbs: move, copy and delete. That makes 3 × 3 × 3 = 27 possible commands. Use the device below to see how many each channel leaves alive.

Count the commands left

Each dot is one possible command: a verb (row), a shape and a spot (column). A lit dot is still possible. Switch voice and pointing on and off, then change what you point at.

That
There

A toy scene we made to show the lecture’s point: three shapes, three spots and three verbs. Voice is assumed to give the verb (“put” means move). Pointing is assumed to give the shape and the spot. The counts are exact for this toy, not for the 1982 system.

Voice alone leaves 9 commands: the verb is fixed, but 3 shapes × 3 spots are open. Pointing alone leaves 3: the shape and spot are fixed, but the verb is open. Both together leave 1. The channels do not add. They multiply away each other’s doubt, 27 to 9 to 3 to 1. This is the first hint of a word the lecture uses later: complementary channels, each filling the other’s gap.

Cameras start to watch bodies and films

Two more slides show the era widening. pFinder (Wren, Azarbayejani, Darrell and Pentland, 1995) tracks a person’s body in real time from a camera. Once a computer can see where your head and hands are, it can use them as an input channel, without a pointing device.

The other thread is multimedia computing. The flagship was the Informedia Digital Video Library at Carnegie Mellon (1994 to 2010). The slide quotes its goal: to combine speech, image and natural language understanding “to create a full-content searchable digital video library.” The problem it faced is simple to feel. A 90-minute film at 25 frames per second holds 90 × 60 × 25 = 135,000 frames. Nobody can search that by hand. The lecture lists four building blocks, with the year each began.

Task (since)What it does
Shot-boundary detection (1991)Cuts a video into continuous camera shots.
Video abstracts (1992)Makes video browsable with representative frames (keyframes), or short clips that carry its essence.
High-level parsing (1997)Splits video into segments that mean something, such as scenes.
Automatic annotation (1999)Finds pre-chosen events, scenes or objects and tags them, so the video can be indexed.

Chain them and the film shrinks. If the cut detector finds 600 shots (an illustrative number), one keyframe per shot turns 135,000 frames into 600 pictures, a 225-fold cut, and each one can carry a tag from annotation. Search then runs over tags and speech text, not over pixels.

The interaction era: watching people talk to each other

From 2000 to 2010 the focus moves from one person using a machine to modelling human interaction: how people behave with each other, with a computer listening in. The lecture names four projects.

ProjectYearsWhat it built
AMI (IDIAP)2001 to 2006More than 100 hours of meeting recordings, with synchronised audio and video, transcribed and annotated.
CHIL (Alex Waibel)Not dated on the slideComputers in the Human Interaction Loop: many sensors watching face-to-face talk.
CALO (SRI)2003 to 2008A Cognitive Assistant that Learns and Organizes, also called PAL (Personalized Assistant that Learns). Siri was a spinoff.
SSP (IDIAP)2008 to 2011Social Signal Processing, a term Sandy Pentland coined in 2007.

The word that matters in AMI is synchronised. A meeting gives you several streams that tick at very different rates. In one second you might get 16,000 audio samples (shape (16000,)), 25 video frames ((25, H, W, 3)) and about 3 spoken words. These rates are illustrative round numbers, but the shape of the problem is real. If the streams are not on one shared clock, a model cannot know that a smile and a laugh belong together.

The symptom of bad synchronisation is a model that learns lip shapes from the wrong sound. The measurement is the lag between streams, in milliseconds. The fix is to stamp every sample with a time and resample every stream onto one clock before any learning begins.

The era closes with shared contests. The Audio-Visual Emotion Challenge (AVEC, from 2011) asks teams to estimate emotional dimensions from video and audio of the SEMAINE dataset. The Emotion Recognition in the Wild challenge (EmotiW, from 2013) uses harder, real-world clips. Both give standard training and test sets. That sounds dull and matters enormously: a benchmark lets every lab score on the same data, so progress can be measured.

Voice alone leaves 9 commands and pointing alone leaves 3, out of 27. Together they leave 1. What does that show?

Chapter 3

Learning the Features Instead of Writing Them

See how deep learning let models describe pictures, speech and words with numbers they learned, and how the field was mapped into six challenges

It is 2015. A model looks at a photograph of a dog catching a frisbee in a park. It writes: “a dog is jumping to catch a frisbee.” While it writes “frisbee”, it looks at the frisbee. While it writes “dog”, it looks at the dog. This is the model from Show, Attend and Tell (Xu et al., ICML 2015), one of three papers the lecture lists for the deep learning era, which starts in the 2010s and carries on to today.

The lecture’s other two examples are Multimodal deep learning (ICML 2011) and Multimodal Learning with Deep Boltzmann Machines (NIPS 2012). The slide gives the era a second name: representation learning, also called deep learning.

What a representation is, and why learning it was the breakthrough

A representation is the list of numbers a model uses to stand for an input. A colour photograph is already a list of numbers: 224 × 224 pixels, each with three colour values, which is 150,528 numbers (shape (3, 224, 224)). But those numbers are a poor description. Move the dog one pixel to the left and almost every number changes, though nothing important has.

Before deep learning, engineers wrote rules to turn pixels or sound into better numbers, called features. A feature might count edges in a patch, or measure the pitch of a voice. Writing them by hand was slow and fragile. Deep learning flips this: the model learns its own features from examples, layer by layer, with the first layers finding simple shapes and later layers finding whole objects.

The lecture’s third reading, Representation Learning: A Review and New Perspectives (Bengio, Courville and Vincent), is the classic argument for this. Its claim, in the abstract: how well machine learning works depends on the representation, because a poor one can entangle and hide the different factors that explain the data. A good representation pulls them apart. The paper surveys four routes to getting one: probabilistic models, autoencoders, manifold learning and deep networks. It ends with open questions about the right training goals. In our terms: the quality of the numbers handed to the model decides almost everything, so learn them.

Here is what that looks like for the two inputs of a captioning model. The photo goes in as (3, 224, 224) and comes out of a vision network as a grid of region vectors, 14 × 14 = 196 regions, each a list of numbers. The caption comes in as words, and each word is mapped to its own list of numbers, a word vector. The lecture calls these “dimensional” linguistic features. Now both inputs are numbers of a similar kind, and attention can link a word to a region.

What made it possible

The slide lists four enablers. New large-scale multimodal datasets: you need millions of examples to learn features. Faster compute and GPUs: the arithmetic is huge. High-level visual features: vision networks that already summarise a picture well. “Dimensional” linguistic features: word vectors that place similar words close together. None of the four is a new idea about multimodality. All four together made it practical.

The tasks the field picked up

With features that could be learned, the field set many tasks. Slide 35 shows the older ones and the birth of “Language and Vision” research: audio-visual speech recognition, affect and emotion recognition, content-based video retrieval, video event recognition (TRECVID), multimodal sentiment analysis, and image captioning “revisited”. Slide 36, spanning 2014 to 2024, adds visual question answering, video captioning and grounding, multimodal dialogue, video question answering and referring expressions, language, vision and navigation, large-scale video event retrieval (such as YouTube-8M), text-to-image generation, and multimodal interactive agents. It ends with “and many many more!”.

The 2025 slide gives three pictures of where it is now: video generation and world models, virtual agents for health and wellbeing, and intelligent tactile and olfactory interfaces. These match the four frontiers of Chapter 0.

Two surveys that organise the mess

By 2017 there were so many tasks that someone had to sort them. Multimodal Machine Learning: A Survey and Taxonomy (arXiv 1705.09406; Baltrusaitis, Ahuja and Morency) did that. It looked past the usual split of early and late fusion and sorted the field into five challenges: representation, translation, alignment, fusion and co-learning.

The lecture’s own survey is Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions (arXiv 2209.03430; Liang, Zadeh and Morency). The slide gives its size: 6 core challenges, 50+ taxonomic classes and 700+ referenced papers, with tutorials at ICML 2023, CVPR 2022 and NAACL 2022. It first states three principles (the data is heterogeneous, connected and interacting, our Chapters 4 to 6), then six challenges: representation, alignment, reasoning, generation, transference and quantification. We cover them in Chapters 7 to 9. Compare the two lists with the device below.

From five challenges to six

The left column is the 2017 survey. The right column is the lecture’s 2022 survey. Pick a 2017 challenge to follow its line.

The mapping reads straight from the sub-challenges on the lecture’s own slides: fusion sits under representation, translation under generation, co-learning under transference. It is our summary of the two surveys, not a claim made in either paper.

Five became four, and two were added. Fusion, translation and co-learning stopped being top-level problems and became kinds of the larger ones. Reasoning and quantification appear as full challenges for the first time. The 2022 list also places each older problem inside a bigger frame.

According to the lecture, what changed with the deep learning era?

Chapter 4

What Is a Modality, and Why Do They Differ?

Define the word the whole field rests on, then measure how different two kinds of data really are

You hold two things. One is a photograph of a desk. The other is a sentence: “A teacup on the right of a laptop in a clean room.” Both describe the same desk. Yet a computer must treat them in totally different ways. The photograph is a grid of 150,528 numbers. The sentence is a row of nine words. How different are they, exactly? And how different are two photographs from two cameras?

The definition

The lecture defines it in one line. Modality refers to the way in which something is expressed or perceived. Text is a way to express something. A photograph is another. A recorded voice is a third. A wearable’s heart-rate trace is a fourth.

The slide then draws a line from raw modalities, the ones closest to a sensor, to abstract modalities, the ones farthest from it. The speech signal and the image are raw: they are just what the microphone and the camera recorded. Language sits in the middle. Detected objects in an image are further along, and so are sentiment intensity (how positive a text is) and object categories (a list of names such as “laptop” and “cup”). The further to the right, the more a person or a model has already interpreted the data.

Two definitions of “multimodal”

The next slide gives two. The dictionary one: multimodal means with multiple modalities. That is the sense we used in Chapter 0. The research one is sharper: multimodal is the science of heterogeneous and interconnected data. The rest of the lecture hangs on those two adjectives. This chapter takes heterogeneous (the data differs in form). Chapters 5 and 6 take interconnected, which the lecture splits into connected (the data shares information) and interacting (the data combines to give new information).

Heterogeneity, one dimension at a time

The slide says information in different modalities shows “diverse qualities, structures and representations”. It gives four pairs, on a line from homogeneous to heterogeneous: images from two cameras, text from two languages, language and vision, and language and sensors. The first two are homogeneous, meaning similar in kind. The last two are heterogeneous, meaning different in kind. One more line: abstract modalities are more likely to be homogeneous.

The lecture’s own survey breaks “different in kind” into dimensions you can check one by one. We use five of its six. Element representation: is each piece a continuous number or a discrete symbol? Distribution: are values spread smoothly, or do a few very common ones dominate? Structure: how are the pieces arranged (a grid, a time sequence, an unordered set, a table)? Information: how raw or how abstract is it? Noise: what typically goes wrong with it? (The paper’s sixth is relevance to the task, which depends on the job, so we leave it out.)

Work the opening example with all five. The photograph’s elements are continuous (a pixel is a number from 0 to 255). The sentence’s are discrete (each word is one of tens of thousands of symbols). That is difference one. A photo changes smoothly from pixel to pixel, while words follow a long-tailed pattern where “the” is everywhere and “teacup” is rare. Difference two. A photo is a spatial grid and a sentence is a sequence. Difference three. A photo is raw, a sentence is already meaning. Difference four. A photo goes wrong with blur and bad light, a sentence with typos and ambiguity. Difference five. Five of five: heterogeneous. Now try your own pairs.

Measure how different two modalities are

Pick two kinds of data. A pink row means they differ on that dimension. Try the two cameras, then a photo and a sentence.

The labels in each cell are our own simplification of how these data usually look, to make the five dimensions concrete. They are not measurements, and a particular dataset can differ (a noisy sensor, a clean photo).

Two cameras: 0 of 5. Photo and sentence: 5 of 5. A wrist sensor and speech audio: only 1 of 5, because both are continuous time series. That surprising pair is why the lecture says heterogeneity is a spectrum, not a yes or no.

Why this blocks the simplest idea

The simplest plan is to glue the inputs into one long list of numbers and feed that to a model. It fails, and the failure has a symptom, a measurement and a fix. Symptom: the model ignores the sentence and answers from the photo alone. Measurement: the photo contributes 150,528 numbers and the sentence nine word ids, a ratio of about 16,700 to 1, and the word ids are not even on the same scale: id 31,002 is not “bigger” than id 5. Fix: give each modality its own encoder (a small model that turns that modality into a vector of fixed size, such as 512 numbers), so that both arrive in the same form. Making that step principled is challenge one in Chapter 7.

Heterogeneity also explains something about your phone. Abstract data such as object labels from two different detectors are both short word lists, so they are easy to join. Raw data from two different sensors may differ in rate, units and noise. That is why joining abstract modalities was the early habit, and joining raw ones is the modern frontier.

Which pair is the most homogeneous in the lecture’s sense?

Chapter 5

Connected Modalities: What They Share

Count the shared information between two kinds of data in bits, and learn to tell a strong link from a weak one from none

Back to the desk. The photograph shows a teacup, a laptop and a tidy room. The caption says “A teacup on the right of a laptop in a clean room.” Some facts appear in both: there is a cup, there is a laptop, the room is clean. Some appear in only one. The photograph shows the cup is white with a blue rim. The caption says nothing about colour. The caption names a relation, “on the right of”, that you must work out from the photograph.

The lecture’s slide draws this as two overlapping circles, one per modality. The overlap is the shared information that relates the two. The part of each circle outside the overlap is unique to that modality. The lecture calls modalities with an overlap connected. It then shows three cases: a stronger connection (a big overlap), a weaker one (a small overlap) and unconnected modalities (no overlap at all).

Counting information

To count an overlap we need a unit. The unit is the bit: the information in one fair yes-or-no question. A fair coin flip holds 1 bit, because one question (“heads?”) settles it. A roll of a fair 8-sided die holds 3 bits, because three well-chosen questions settle it.

The amount of uncertainty a variable has is its entropy, in bits. The overlap between two variables is their mutual information: how many bits of the questions about one are already answered by the other. If learning B tells you nothing about A, the mutual information is 0 bits. If learning B settles A completely, it equals all of A’s bits.

The smallest example: a coin and its noisy copy

Let A be a fair coin flip, 1 bit. Let B be a copy of A that gets flipped by mistake with probability p. If p = 0, B is a perfect copy. If p = 0.5, B is another coin flip that has nothing to do with A. Everything in between is a connection of some strength.

The mutual information of this pair has a short formula: 1 − H(p) bits, where H(p) is the entropy of a biased coin that lands “flipped” with probability p.

I(A;B)
The mutual information of A and B in bits: the size of the overlap between the two circles.
H(p)
The uncertainty about whether B was flipped. It is 0 bits when p = 0 (never flipped) and 1 bit when p = 0.5 (a coin flip).
log2
The logarithm base 2, which counts how many yes-or-no questions a probability is worth.

p = 0.10: H = 0.10 × 3.322 + 0.90 × 0.152 = 0.332 + 0.137 = 0.469 bits  →  I = 1 − 0.469 = 0.531 bits

So a copy with a 10% error rate still shares more than half of A’s one bit. Move the slider yourself and watch the circles pull apart.

Pull two modalities apart

Each circle holds 1 bit. The overlap is what they share. The grid below counts 1,000 imaginary pairs. Raise the error rate and see the overlap vanish.

10%

Illustrative toy: A is one fair bit and B is A flipped with the chance you set. The shared information is exact (1 minus the coin entropy). The overlap area of the circles is drawn in proportion to it. “Stronger” and “weaker” use our own cut at 0.5 bits.

Look at the grid. With p = 0.10, about 450 pairs are (0,0), 450 are (1,1), and about 50 each are mismatches. The table is a joint distribution, the raw material for mutual information. At p = 0.5 the four cells are about 250 each: knowing A tells you nothing about B.

pythonimport numpy as np
def mutual_info(joint):                 # joint: shape (nA, nB), sums to 1
    pa = joint.sum(1, keepdims=True)    # (nA, 1)  marginal of A
    pb = joint.sum(0, keepdims=True)    # (1, nB)  marginal of B
    nz = joint > 0
    return float((joint[nz] * np.log2(joint[nz] / (pa @ pb)[nz])).sum())

p = 0.10
joint = np.array([[.5*(1-p), .5*p], [.5*p, .5*(1-p)]])
print(mutual_info(joint))               # 0.531 bits

Two kinds of connection, and one failure

The lecture’s survey separates two flavours. A statistical connection is a pattern in the numbers, such as correlation: when A goes up, B goes up. A semantic connection is a meaning link, such as a caption that names what is in a photograph. The coin toy shows only the statistical kind. Real captions show both.

The failure: assuming a connection that is not there. The symptom is a fused model that scores worse than the better of its two inputs alone. The measurement is a shuffle test. Scramble which A goes with which B and recompute the score. If it hardly moves, the pairs were not connected, which is the p = 0.5 end of the slider. The fix is to drop the weak modality, or use only the part of it that is unique and useful.

In the coin toy, B is A flipped by mistake 50% of the time. How much do they share, and what does the lecture call that?

Chapter 6

Interacting Modalities: Redundancy, Uniqueness, Synergy

Learn the three ways two modalities can help a task, and see why one of them cannot be found by looking at each modality alone

Go back to the hero at the top of the page. In its three scenes, the words and the face helped a model in three different ways. In the first, they said the same thing. In the second, only the words said anything. In the third, they said something new together. The lecture names these three ways, and the names are worth learning by heart.

First, a definition. An interaction is how modalities combine to provide information for a task. The task is what the model must predict: here, how the speaker feels. We will call that answer Y, and the two inputs X1 (the words) and X2 (the face).

The three interactions, from the lecture’s own examples

Redundancy is information shared by both modalities and the task. The lecture’s sentence is “This movie is great!” with a laughing face. Each input alone says “positive”. Together they say “very positive”. A second modality here is a back-up copy. It makes the answer firmer, and it lets the system survive when one input is lost.

Uniqueness is information that only one modality holds. The sentence is “The movie does a good job developing …” and the face is blank. The words say “positive”. The face says nothing. Information unique to modality 1 is the words’. A second modality here adds nothing, though in another example the face could hold the unique part.

Synergy is the new information that appears only when the two are combined. The sentence is “wowwww” with a rolling eye. The words alone read as praise. The face alone reads as annoyance. Together they read as sarcasm, an answer that neither input contains. The lecture’s phrase is “emerging information from multimodal interaction”.

The lecture draws it as the overlap picture of Chapter 5, now with three circles: X1, X2 and the task Y. The parts of the picture are the pieces of what the pair can tell you about Y: the redundant part, the part unique to X1, the part unique to X2, and the synergy.

I(X1,X2; Y)
All that the two inputs together tell you about the task, in bits.
R
Redundancy: what either input alone already tells you.
U1, U2
Uniqueness: what only X1 tells you, and what only X2 tells you.
S
Synergy: what you learn only by seeing both at once.

Each input alone covers only part of the sum: X1 alone gives R + U1, and X2 alone gives R + U2. Synergy is in neither. That is why it is the hard one.

The smallest example of each, with exact numbers

Take two bits, A and B, and a task answer Y that is also one bit. Build three tasks. In the redundant task, B is a copy of A, and Y equals A. In the unique task, B is a random bit unrelated to A, and Y equals A. In the synergy task, A and B are random, and Y is 1 exactly when A and B differ (the logic called XOR). The device runs all three, scoring a predictor that sees A only, B only, both together, and one more that averages two single-input predictors.

Find the interaction

Choose a task. The small grid shows the four possible (A, B) pairs, coloured by the answer Y. The bars show each predictor’s exact accuracy. Then choose which input the model sees.

Model sees

Exact, by listing every (A, B, Y) case and its probability. The best predictor for each input set picks the commonest answer for every input value. The averaged predictor mixes the two single-input answer probabilities, and a tie scores half. These tasks are toys built to isolate one interaction each.

In the synergy task, A alone gets 50%, B alone gets 50%, and the two together get 100%. Neither input shows any hint of the answer on its own. In the redundant task, both single predictors already get 100%, so the pair adds nothing. In the unique task, A is enough.

Why it matters: the failure of “add up the parts”

The fourth bar shows the failure. Symptom: a system built by training one model per modality, then averaging their answers, scores 50% on the synergy task, exactly chance, while a model that sees both together scores 100%. Measurement: each single-modality model is at chance, so no check of a single modality can warn you. Only a joint test shows it. Fix: let the model look at the inputs together before it decides, so that it can learn a rule like XOR. Chapter 7 starts that story with fusion.

Real data mixes all four pieces. Sarcasm, the lecture’s example, is one. A better-known one is the question “what is the man holding?” asked about a photograph. The question alone is not enough, the photo alone is not enough, and the answer exists only in the pair.

A model trained on the face alone scores at chance. A model trained on the words alone scores at chance. A joint model on both scores 95%. What is the most likely interaction?

Chapter 7

Challenges One and Two: Represent and Align

See how a model turns several streams into numbers it can use, and how it finds which word goes with which part of a picture

Think of a cooking video. You get three streams at once: the pictures, the voice reading the recipe, and the subtitles. Each is a row of small pieces, the way a sentence is a row of words. The lecture draws them as rows of triangles, circles and squares, feeds them into one box marked “Multimodal AI”, and shows what comes out: a single vector, or a prediction. The next four chapters walk through the six jobs that box must do. The first two come now.

Challenge 1: representation

The lecture’s definition: learning representations that reflect cross-modal interactions between individual elements, across different modalities. In plain words: turn each piece into numbers in a way that already knows how the pieces of the other modalities relate to it. The lecture calls this a core building block for most multimodal modelling problems. Chapter 3 explained what a representation is (a vector the model uses as a stand-in for an input). A vector is just a list of numbers, such as 512 of them.

The lecture sorts the ways to do this by a simple count: how many modalities go in, and how many representations come out.

Fusion: more modalities than representations. Several streams go in, one vector comes out. This is the common case: join the picture and the words to answer one question. Coordination: as many representations as modalities. Each stream keeps its own vector, but the vectors are tied together, for example by a training goal that pulls matching pairs close. CLIP, which we cover in our contrastive learning lesson, is the standard example. Fission: fewer modalities than representations. One stream goes in, several vectors come out, for example one part for what is shared and one for what is unique. Try each below.

Count what goes in and what comes out

Choose a way to build a representation. Boxes on the left are inputs. Boxes on the right are the vectors that come out. Slide the vector size and watch the numbers change.

512

Sizes are illustrative: a photo of shape (3, 224, 224) and a caption of 12 word ids, each encoded to a vector of the size you pick. The fusion layer is one linear layer from the two joined vectors (2d numbers) down to d numbers, so it holds 2d × d + d weights. Fission is drawn as three vectors (shared, unique to A, unique to B), one possible design.

With d = 512, the fusion layer holds 2 × 512 × 512 + 512 = 524,800 weights, and its one output vector has 512 numbers. Coordination has no mixing layer, but it keeps two vectors, 1,024 numbers in all. Fission makes three vectors, 1,536 numbers, from a single input.

Challenge 2: alignment

The lecture’s definition: identifying and modelling cross-modal connections between all elements of multiple modalities, building from the data structure. Representation asks “what is each piece?”. Alignment asks “which piece in one stream goes with which piece in the other?”

It exists because most modalities have internal structure with many elements. A sentence and a video are temporal: the elements come in order, and the word “pour” should match the moment the milk moves, not the moment the pan is empty. A photograph is spatial: words match regions. A scene description may be hierarchical: a room contains a table that contains a cup.

The lecture names three sub-challenges. Discrete connections are explicit links between a few elements, such as grounding, where the word “teacup” is tied to a box around the cup. Continuous alignment is for signals with no natural pieces, where the question is how finely to cut them. Contextualized representation is implicit: the model never draws links, but each element’s vector is built using the others, so the links live inside the numbers. The usual tool is cross-modal attention. Each word scores every region, the scores become percentages (by the softmax function, which turns any list of scores into percentages that add to 100), and the word’s vector becomes the weighted mix of the regions.

Take the lecture’s caption, “A teacup on the right of a laptop in a clean room.”, and a photo cut into six regions. Choose a word and see where it looks.

Let a word look at the photo

Pick a word from the caption. The brightness of each region is the share of that word’s attention. Raise the sharpness to focus it.

1.0

A hand-made toy, not a trained model: each region has a three-number content vector (cup-ness, laptop-ness, plain surface), each word has a query vector of the same size, and attention is the softmax of their dot products times the sharpness. The six-region photo (wall, window, shelf above; laptop, table, teacup below) is our own illustration of the caption.

With sharpness 1, “teacup” sends about 80% of its attention to the bottom-right region. “clean” spreads over the plain surfaces, because nothing in the photo is specifically “clean”. “right” is the interesting one: it is a relation, so it splits its attention, about 44% on the teacup and 16% on the laptop, the two things it links. No single region owns a word like that. This is the hard part of alignment, and a reason the lecture lists it separately from representation.

The failure is easy to name. Symptom: a model describes a photo with the cup on the left. Measurement: the attention of the word “right” is spread flat across regions, so the model has no spatial anchor. Fix: add structure the data already has, such as region positions, and train on examples where the relation is what changes.

A model reads a photo and a caption and returns one vector. A different model returns one vector for the photo and one for the caption, trained to be close when they match. Which is which?

Chapter 8

Challenges Three and Four: Reason and Generate

Chain several steps of inference across photo and text to reach an answer, then go the other way and make a new picture, sentence or sound

You park on a city street and see a sign: “No parking, 4 to 6 pm, Monday to Friday.” Your phone asks a question: “Can I park here at 5 pm on Tuesday?” To answer, you do four small things. You read the sign. You read the question. You check that 5 pm falls inside 4 to 6 and that Tuesday is between Monday and Friday. Then you apply what you know about the world: “no parking” means a parked car is not allowed. No single glance did it. A chain of steps did.

Challenge 3: reasoning

The lecture’s definition: combining knowledge, usually through multiple inferential steps, exploiting multimodal alignment and problem structure. An inferential step is one small move from facts you have to a new fact. The slide draws three ingredients. First, the aligned modalities from Chapter 7 feed the process: the sign’s words are tied to the question’s words. Second, the process has many stacked steps, drawn as a row of boxes with arrows. Third, external knowledge flows in from a store outside the inputs, drawn as a database: the rule that “no parking” forbids parking is in neither the photo nor the question. The structure of the problem, such as a tree of cases, shapes how the steps connect.

Try the chain. Each step needs the ones before it. Switch a step off and see what the answer becomes.

Break the chain of steps

The question: “Can I park here at 5 pm on Tuesday?” about a sign reading “No parking, 4 to 6 pm, Monday to Friday.” Switch steps off one at a time.

A hand-built toy to show dependence between steps, not a trained model. Each step is needed for the next: with one removed, the chain stops and the answer box shows what is missing.

With all four steps on, the answer is “No, do not park”. Remove “match time and day” and the model has a sign and a question but cannot say whether the sign applies now. Remove external knowledge and it knows the sign applies but not what “no parking” means. The symptom of a weak reasoner is a confident wrong answer: “Yes, you can park”. The measurement is to switch each step off, or to ask the model to show its intermediate facts, and see where the chain bends. The fix is to make the steps explicit and check each one. The course returns to this in Week 9, with reinforcement learning to train multi-step thinking.

Challenge 4: generation

So far the model has read. Now it writes. The lecture’s definition: learning a generative process to produce raw modalities that reflects cross-modal interactions, structure, and coherence. The output is no longer a label or a number. It is a sentence, an image, a sound or a video. Three words in that definition matter. The output must reflect the interactions between inputs (the picture should show a teacup on the right, if the prompt says so). It must have structure (a face has two eyes). And it must be coherent (the sound must fit the picture).

The lecture splits generation by how much information moves from input to output. In summarization the output carries less content than the input: this is reduction. In translation the content stays the same and only the form changes: maintenance. In creation the output has more content than the input: expansion, such as a short prompt becoming an image. Compare the sizes of what goes in and what comes out.

Compare the size going in and out

Pick a kind of generation. The bars use a scale where each step is ten times bigger. The longer bar is the bigger side.

We count raw numbers, which is not the same as information: a million pixels can hold less meaning than twelve words. The lecture’s signs (greater than, equal to, less than) are about information content. Examples: a 2-second clip at 24 frames per second and 224 × 224 pixels in 3 colours becomes a 12-word caption; a 12-word sentence becomes a 13-word translation; a 12-word prompt becomes a 512 × 512 colour image.

Look at creation. A 12-word prompt is 12 numbers. A 512 × 512 colour image is 512 × 512 × 3 = 786,432 numbers, a factor of 65,536 more. The model must invent every number the prompt did not fix: the exact grain of the wood, the angle of the light. This is why creation is the hardest and why it has no single right answer. Two good teacup photos can share no pixels at all. Methods that learn this, such as diffusion, are in our diffusion lesson and the video diffusion lesson.

The failure is a mismatch between the prompt and the result. Symptom: the picture shows the cup on the left. Measurement: use a separate model (as in Chapter 7) to check that the output still matches the prompt. Fix: train the generator with that check as part of its goal. Judging a generated result at all is a hard open problem, and it returns in Chapter 9 as quantification.

In the parking example, what plays the role of external knowledge?

Chapter 9

Challenges Five and Six: Transfer and Measure

Use a strong modality to help a weak one, then learn why the field insists on measuring what multimodal models really do

A hospital has plenty of clear chest X-rays with doctors’ reports attached. It has very few recordings from a cheap home stethoscope, noisy and unlabelled. The stethoscope is the modality we want to use, because everyone could own one. The X-ray is the modality we have good data for. Can the rich data help the poor one? This is the fifth challenge.

Challenge 5: transference

The lecture’s definition: transfer knowledge between modalities, usually to help the target modality, which may be noisy or with limited resources. The target modality is the one we care about at test time. The helper is often called the enriched modality. The slide shows the target as a row of noisy circles and the helper as an enriched row of triangles above it, with knowledge flowing down.

There are three sub-challenges, drawn as three small diagrams. Transfer: take what a model learned on modality A, and use it to start or guide a model on modality B. Co-learning: train on both modalities together, so each shapes the other’s representation, even when only one is available later. Model induction: keep two separate models, one per modality, and make them teach each other until they agree.

The experiment below is a small case of the third idea. The class is a hidden yes or no. A clean modality A and a noisy target B both reveal it. We have n labelled examples of B, which is few. We also have 200 pairs with both A and B but no labels. A strong model of A labels those pairs for us (it is right 93% of the time), and B’s model trains on the extras too. Slide the number of real labels and see when the help matters.

Borrow labels from a strong modality

Each bar is the average test accuracy of a classifier on the noisy modality B, over 300 random trials. Slide the number of real B labels you have.

4

A toy with two classes. A feature is a number: class means ±1.5 for A and ±0.7 for B, both with spread 1. Each classifier puts a threshold halfway between its class averages. Transfer adds 200 unlabelled (A, B) pairs, labelled by A’s own threshold at 0. Results are Monte Carlo averages with a fixed seed, so the same slider position always shows the same bars. “Best possible” and “A alone” are exact.

With 4 real labels, the noisy modality alone reaches about 70.2% on average. With borrowed labels it reaches 75.7%, almost its best possible (75.8%). With 64 real labels the gap shrinks to 0.2 points: once B has enough data of its own, A has little left to add. This pattern, big help when the target is data-poor and little help when it is not, is why transference is described as being for modalities that are “noisy or with limited resources”.

The failure: transferring a mistake. Symptom: the weak modality’s model gets worse than it was alone. Measurement: check how accurate the helper’s labels are. Here A is right 93% of the time. If A were right only 55%, the borrowed labels would mostly be noise. Fix: only borrow labels where the helper is confident, and keep some real labels to check.

Challenge 6: quantification

The last challenge is not about building anything. The lecture’s definition: empirical and theoretical study to better understand heterogeneity, cross-modal interactions, and the multimodal learning process. Its three sub-challenges map to what we have already done. Heterogeneity: measure how the modalities differ (Chapter 4’s five-dimension table). Interactions: measure how much is redundant, unique and synergistic (Chapter 6’s bits). Learning: study how training behaves, shown on the slide as a curve of loss falling over epochs, an epoch being one pass over the whole training set.

Why a whole challenge? Because multimodal models are easy to build and hard to understand. A model that takes a photo and a caption and scores well might be using only the photo. Without a measurement, you would never know. The synergy test of Chapter 6, scoring each input alone and then both, is the simplest quantification there is. The lecture promises more, in later weeks, on how to measure interactions properly and on why some multimodal models learn one modality well and neglect the other.

The whole map, and where it goes next

Slide 58 puts all six challenges in one picture. Representation and alignment sit on the left and feed reasoning in the middle. Reasoning feeds generation and transference on the right. Quantification stands to the side and watches the rest. Slide 59 then answers the three questions the lecture started with. What is multimodal? Heterogeneous, connected and interacting data. Why is it hard? The six challenges. What is next? Five fronts: multimodal LLMs, generative AI, physical sensing, holistic health, and agents.

The last slide gives the week’s assignment. Thursday’s lecture covers multimodal datasets and tasks. In the meantime: meet classmates, start forming project teams, and think about the modality and problem you want to work on. A formal project preference form was promised for that week.

In the experiment, why does borrowing labels from A help much more with 4 real labels than with 64?

Chapter 10

The Field Guide

Check yourself on the six challenges, then keep the numbers, the terms and the core code in one place

The lecture asks one question and answers it in three parts. What is multimodal? The science of heterogeneous and interconnected data: different in form, sharing information, and interacting to say new things. Why is it hard? Six challenges: representation, alignment, reasoning, generation, transference and quantification. What is next? Multimodal LLMs, generative AI, physical sensing, holistic health and agents. Test the middle part now. Each card describes a real job. Pick the challenge it mostly is.

Name the challenge

Read the job, then choose the challenge it mostly is. Six jobs, one for each challenge.

The jobs are our own examples written to match the lecture’s definitions. Real systems often touch several challenges; each card names the one it mostly is.

The numbers worth remembering

ItemValueWhere it came from
Core challenges6Foundations and Trends survey, slide 38
Taxonomic classes, referenced papers50+, 700+Same survey, slide 38
Eras of researchbehavioral 1970s to late 1980s, computational late 1980s to 2000, interaction 2000 to 2010, deep learning 2010s, foundation model 2020sSlide 26
Put That There1982Slide 27
AMI meeting corpus2001 to 2006, 100+ hoursSlide 31
Heterogeneity dimensions used here5 of the survey’s 6Chapter 4
Interaction piecesredundancy, uniqueness (one per input), synergySlides 43 to 45
Course grading35% homework, 15% readings, 15% midterm, 35% projectSlide 16

The terms, in plain words

TermPlain meaning
ModalityOne way something is expressed or perceived: text, image, speech, touch, a sensor trace.
HeterogeneousDifferent in kind: elements, spread, structure, level and noise.
ConnectedSharing information. The overlap of the circles.
InteractingCombining to give information for a task: redundancy, uniqueness or synergy.
Fusion, coordination, fissionMore inputs than outputs, equal numbers, fewer inputs than outputs.
AlignmentFinding which element of one modality matches which element of another.
TransferenceUsing one modality to help another that is noisy or has little data.
QuantificationMeasuring heterogeneity, interactions and learning, so you know what the model really does.

A checklist for a new multimodal problem

  1. List the modalities and place each on the raw-to-abstract line.
  2. Score their heterogeneity on the five dimensions, so you know what each needs from an encoder.
  3. Measure the connection between them on paired data, with a shuffle test.
  4. Test the interaction: score each input alone, then together. A big jump means synergy.
  5. Choose the representation: fusion, coordination or fission, by counting inputs and outputs.
  6. Plan alignment if the streams have internal structure (time, space, hierarchy).
  7. Ask whether it needs several steps or outside knowledge (reasoning), or must produce a new modality (generation).
  8. Check for a rich helper modality if your target is noisy or short on data (transference).
  9. Quantify what the model uses, not just its score.

The core, as code

This is Chapter 6 in nine lines: a task whose answer is in neither input alone. The average of two single-input models scores 50%. A model that sees both together scores 100%.

pythonimport numpy as np

A = np.array([0, 0, 1, 1]); B = np.array([0, 1, 0, 1]); Y = A ^ B   # synergy: XOR

def single_model(x, y):                      # best guess from ONE input
    return {v: y[x == v].mean() for v in (0, 1)}   # P(Y=1 | x=v)

pa, pb = single_model(A, Y), single_model(B, Y)    # both say 0.5 for every value
late = np.array([(pa[a] + pb[b]) / 2 for a, b in zip(A, B)])   # average two models
print("late fusion accuracy:", ((late > .5) == Y).mean())     # 0.5, chance

table = {(a, b): y for a, b, y in zip(A, B, Y)}    # a joint model sees both
print("joint accuracy:", np.mean([table[(a, b)] == y for a, b, y in zip(A, B, Y)]))  # 1.0

Where it sits, and where to go next

The three readings for this session each have their own lesson in this series. Foundations and Trends in Multimodal Machine Learning (the lecture’s survey, arXiv 2209.03430). Multimodal Machine Learning: A Survey and Taxonomy (the 2017 predecessor, arXiv 1705.09406). Representation Learning: A Review and New Perspectives (arXiv 1206.5538).

For the topics this lecture opens: contrastive learning and CLIP (coordination), multimodal foundation models, CS224N on multimodal models, the information theory workbook (the bits of Chapters 5 and 6), and diffusion (generation). The next lecture in the course, Week 1.2, covers multimodal datasets and research tasks.

The takeaway. Multimodal AI is not “more inputs”. It is the science of data that differs in form, shares information, and says new things only in combination. Six challenges follow from that: represent it, align it, reason over it, generate it, transfer between it, and measure it. When you can say which one your problem is, you know where to look.

Now press Present or Teach and explain, from memory, why “wowwww” plus an eye-roll means sarcasm, what the 27-to-1 count showed about voice and pointing, and which of the six challenges each of your own projects is. Then go back to the scenes at the top and say, for each, which interaction it was: redundancy, uniqueness or synergy.

Credit: this lesson follows Lecture 1 of MIT MMAI: Multimodal AI (Spring 2026), taught by Paul Liang with Dimitris Bertsimas, Jinhua Zhao and Sang-Gook Kim at the MIT Multisensory Intelligence group, with teaching assistants Edgar Morfin, Valdemar Danry and David Dai. The history slides were co-developed with Louis-Philippe Morency for CMU course 11-777, as the slides state. The slides (PDF). All explanations, toy devices and numbers labelled “illustrative” are our own; the course’s slides are not reproduced.

You have a cheap wrist sensor with little labelled data, and a rich, well-labelled camera dataset of the same activities. Which challenge is this mostly?
MIT MMAI Lecture 1: Introduction to Multimodal AI
The course site · Lecture slides · Foundations and Trends in MML · Survey and Taxonomy · Representation Learning · Back to Gleams