A multimodal task is a recipe: choose the signals that go in and the thing that comes out. Build one yourself and see which real dataset from the lecture you just invented.
Learn how research works, then tour the tasks and datasets that define multimodal AI: seeing, hearing, touching, smelling, browsing, acting and reading people.
Switch on two or three signals, pick what the model should produce, and read which tasks match. Then we build, piece by piece, the research loop, the questions, and every dataset family the lecture walks through.
You need a rough idea of what a dataset is (a pile of examples with answers attached) and curiosity about how research starts. We build the rest from zero.
Based on Lecture 1.2 of MIT MMAI: Multimodal AI (Spring 2026), taught by Paul Liang of the MIT Media Lab and EECS, MIT Multisensory Intelligence group. Lecture slides. Every idea is re-taught here in our own words and devices.
The task list is a sample drawn from the lecture’s own task timeline and dataset slides (about 17 tasks); a task matches when its inputs are exactly the signals you switched on and its output is the one you chose. Dataset sizes are the slides’ numbers. A task the slides name without a dataset says so.
Chapter 0
See why a multimodal researcher meets datasets first, and what today’s lecture asks you to do with them
Picture your first week on a research team. Someone hands you a folder and says, “This is our data. Find something worth studying.” Inside are photos, audio clips, short videos and a spreadsheet of labels. You have no question yet, no model and no plan.
That is where most research begins, and it is where this lecture begins. Paul Liang opens the second lecture of MIT’s Multimodal AI course by asking how to do AI research at all, and then walks through the tasks and datasets that multimodal researchers work on today.
A modality is one kind of signal: the way a piece of information reaches a computer. Text is one modality. A photo is another. A voice recording, a video, a touch sensor and a chemical sensor are each their own. A multimodal system uses two or more of them at once.
A dataset is a collection of examples. Each example holds the input signals and, often, the answer the system should give, called the label. A task is the job you ask of the system: given these inputs, produce that output. Pick a task and a dataset and you can run an experiment. Pick neither and you only have a folder.
So the lecture does two things at once. It teaches you how research is done, and it shows you the shelf of tasks and datasets you can pull from. Both are needed. A great method with no data cannot be tested. A great dataset with no question just sits there.
Slide 14 of the lecture draws one picture for every multimodal task. Several signals, called Modality A, B and C, flow into one box labelled Multimodal AI. Out of the box come three possible results: a predicted answer, written ŷ (read “y hat”), a newly generated output, or a transfer, where what the model learned helps with another task or modality. Chapter 3 returns to that picture. For now, the hero at the top of this page is that picture made playable.
Try one right now. Switch on image and text and choose predict. You have just described visual question answering: show a model a picture and a question, and it answers. Switch to generate and you have described a different family, such as a robot that reads an instruction and writes out its next move. The same two inputs, two very different jobs.
Count the possible recipes. With one signal you have one input set. With two signals you have three: the first alone, the second alone, or both. With three signals you have seven. Each extra signal doubles the count and adds one. In general, k signals give 2k − 1 non-empty input sets. Multiply by two outputs (predict or generate) and you have the number of task shapes.
Move the slider below and watch two things grow at different speeds: the number of possible task shapes, and the number the lecture’s sample of tasks actually fills.
Each dot is one task shape: a set of signals plus an output. Teal-outlined dots are shapes with a task from the lecture’s list; faint dots are still open. Drag to add signals.
The signals are, in order, image, video, audio, text, touch, smell. The filled dots come from the 17-task sample behind the hero, which is a sample, not the lecture’s complete list, so the empty count is an upper bound on what is truly unexplored. Two tasks with the same signals and output share one dot.
With four signals there are 15 input sets and 30 task shapes. With six there are 63 input sets and 126 shapes. The list of named tasks covers only a small corner. Many empty shapes are empty for a good reason: nobody has a use for them. Some are empty because nobody has built the dataset yet. Finding which is which is exactly the work of research.
Slide 2 lists four parts. First, how to do AI research: the loop, where ideas come from, and how to write questions that can fail. Second, multimodal tasks and datasets, which takes the most time. Third, advice on research projects. Fourth, a mingle to find teammates. This lesson follows the first three in order and leaves the fourth to the room. Logistics sit in Chapter 10 in one short paragraph.
The lecture also shapes the course. Students form project teams, and the datasets here are the starting points for those projects. That is why each task area on the slides ends with a short list of “models and datasets to start with”. We keep those lists as we go.
The readings for this session sit around the same ideas. Jordan and Mitchell’s Machine Learning: Trends, Perspectives, and Prospects (Science, 2015) says machine learning has grown because of better algorithms and theory, more online data and cheaper computing. Data is one of the three legs. That is our reason to start with datasets. Three paper lessons in this series cover the other readings: Representation Learning: A Review and New Perspectives, Foundations and Trends in Multimodal Machine Learning and Geometric Deep Learning. Each gets a short summary in the chapter where it fits.
Chapter 1
Walk the scientific method once around, then find an idea from failures or from a big question
You are baking bread for the first time. The loaf comes out dense. You do not throw away the oven. You read a recipe, guess that the dough needed more water, bake again with one change, and compare the two loaves. Then you write down what you learned. That small loop is the whole of science.
Slide 3 draws it as a circle with six stops. This chapter walks once around it, then asks the harder question: where does the first idea come from?
Start at the top of the circle. Initial observations and ideas: you notice something odd, such as a model that answers colour questions well and counting questions badly. Next comes the literature review: you read what others have already tried, so you do not repeat their work. Then the research question and hypothesis: you say exactly what you want to know and what you expect. Chapter 2 covers that stop in depth.
After that, you test with an experiment: you change one thing and measure. Then you analyze the data: you check whether the change helped, and by how much. Last, you report conclusions. The report is also a new observation for someone else, so the circle closes. Your answer starts another person’s loop.
Notice what the circle does not say. It does not say the loop runs once. A typical project goes round many times, and each pass makes the question sharper. The lecture states it directly: your research ideas will evolve during the semester.
Slide 4 names two directions of travel. In bottom-up discovery you start from something concrete and low-level. You look closely at what existing research gets wrong, and you turn those failures into a higher-level experimental question. In top-down design you start from a big question and break it into a smaller, concrete test.
Each has a strength and a risk. Bottom-up is a great tool for steady, incremental progress, but it may keep you from larger leaps, because you only ever fix what is already in front of you. Top-down favours bigger ideas, but a big idea can drift away from reality: you design a beautiful test for something no data can show.
An analogy helps. Bottom-up is a mechanic listening to an engine, hearing a rattle, and tracing it to a loose part. Top-down is an engineer asking, “Could a car run on half the fuel?” and then working out which small experiments would tell. Both are useful. Good researchers use both in turn.
Slide 5 turns bottom-up discovery into three steps. One: experiment with state-of-the-art models. Two: analyze their successes and failures. Three: find ways you could improve on the failures. The homeworks and the midterm assignment train this, and the proposal report must name existing methods. The final report must propose new ideas that do better.
Slide 6 does the same for top-down design. Brainstorm with teammates, TAs and instructors; office hours and project hours exist for that. Do the literature review in the first assignment, tied to your dataset and your first ideas. While you explore the dataset, expand your ideas. The midterm and final reports then aim at gaining the proposed abilities step by step.
Here is the bottom-up step made concrete. Imagine a model that answers questions about pictures. You run it on a test set and sort the questions into five kinds. The scores differ by kind. The weakest kind is where an idea hides.
The catch is noise. If each kind has only a handful of examples, a low score may be luck. Pick a kind below, then move the slider to change how many examples each kind has, and watch the margin of doubt shrink or grow.
Bars show a toy model’s accuracy on five kinds of question. The whisker is the 95 percent margin for that bar. The dashed line is the overall score. Choose a kind, then change the examples per kind.
Illustrative: the accuracies (94, 91, 78, 52 and 61 percent) are made up to show the method. The margin is the usual normal estimate, 1.96 times the square root of p(1 − p) / n, and treats each kind as a fresh sample.
At 40 examples per kind, the counting bar is 52 percent with a margin of about 15 points. The overall score is 75 percent, so counting sits 23 points below it. That gap is larger than the margin, so it is probably real. Now drag the slider down to 10: the margin grows to about 31 points and the gap no longer stands out. A weak slice is only an idea if you have enough examples to trust it.
Once the gap is real, you write it as a question. “Counting is weak” is an observation. “Does the model fail at counting because its training pictures rarely show more than three objects?” is a question, and you can test it: build counting questions from pictures with more objects, and see whether the score drops as the count rises. That is a hypothesis you could prove wrong, which is the subject of the next chapter.
Notice that you did not need a new model to get here. You needed an existing model, a test set, and the habit of splitting the score into slices. That habit is the most useful tool in the bottom-up kit.
The assigned Foundations and Trends in Multimodal Machine Learning (Liang, Zadeh and Morency, arXiv 2209.03430) gives top-down researchers a ready-made map. It proposes six core technical challenges: representation, alignment, reasoning, generation, transference and quantification. Each challenge is a big question you can narrow into a concrete test. Our paper lesson summarises it in full.
Chapter 2
Write a question with a yes or no answer, and a guess that an experiment can prove wrong
A friend says, “I think more sleep makes me happier.” You ask, “How much happier?” They shrug. You ask, “If you sleep nine hours for a week, will your mood score go up?” Now there is something to check. The second question can be answered, and it can come out wrong.
Slides 7 to 10 teach that difference. A good research question is one you can answer with data, and a good guess is one the data can contradict.
A research question is one or several explicit questions about the thing you want to know. Slide 7 gives one practical tip: hypotheses are easier to draft from yes-no questions than from how-to questions. “Does adding audio change the sentiment score?” invites a guess. “How do we use audio?” invites a long essay.
A hypothesis is what you think the answer may be, before you run anything. The word to hold on to is falsifiable: if you get a certain result the hypothesis is confirmed, and otherwise it is disproved. A hypothesis that no result could disprove is not a hypothesis. It is a slogan.
Compare two guesses. “Our model will understand people better” cannot fail, because “understand” has no number. “Our model will name the speaker’s emotion correctly on at least 5 points more of the test clips” can fail. Only the second one teaches you something when it does.
Slide 8 shows two paragraphs from real papers. One is about aligning representations across modalities, the other about AI chatbots and misinformation. Each ends by stating its research questions in sharp, narrow terms. Slide 9 shows a list of questions from a paper on a method called DRPO, which the lecture labels “not the best”. In our paraphrase they read “How does the method compare with other methods?”, “How well does it handle mixed inputs?” and “How good are the reasoning traces?”. They are not wrong. They are open-ended. They do not say what answer you expect, so no single result can prove the author wrong.
A quick repair: turn each “how” into a yes-no claim with a number. “How does it compare?” becomes “Does it beat the strongest baseline (the best existing method, run on the same data) on all three datasets?” Now the answer is yes, no, or “two of three”, and each outcome tells you something.
Slide 10 warns about the most common research question in machine learning. “Does adding X make Y better?” is natural, and the hoped-for answer is yes. The lecture calls it indirect, and gives the reason in one sentence: if the answer is “no” after your experiments, how do you tell what went wrong?
The question hides your real reason. Usually you have an intuition about why X should help Y, not just a random hunch. Write that reason down, and ask what else would be true if the reason were right. Then test those other things. If they hold, you learn something even when the headline number does not move. If they fail, you learn your reason was wrong.
Here is the idea in a toy. A model reads sentiment from short clips. A clip with sarcasm says a positive sentence in a negative voice. The text-only model gets most normal clips right (90 percent) but only 40 percent of sarcastic ones, because the words alone mislead it. Add the audio, and sarcastic clips jump to 75 percent while normal clips stay at 90.
The reason for adding audio is exactly this: tone carries what words hide. Now compare two ways to ask. One asks “Does audio improve overall accuracy?” The other asks “Does audio fix sarcastic clips?” The test set has 400 clips. Slide the share of sarcastic clips and switch the question.
Two bars: the text-only model and the model with audio. Whiskers show 95 percent margins. Choose which question you ask, then drag the share of sarcastic clips in the test set.
Illustrative: every number comes from the made-up accuracies in the text and a test set of 400 clips. The margin on each bar is 1.96 times the square root of p(1 − p) / n; the margin on the difference adds the two in quadrature, which treats the two models as tested on separate clips and is more cautious than a paired test.
At 8 percent sarcastic clips the overall question sees 86 versus 89 percent, a gap of about 3 points that sits inside its margin of about 5. The honest answer is “no clear gain”, and a researcher who only asked this question might drop the idea. The sharper question looks at the 32 sarcastic clips: 40 versus 75 percent, a gap of 35 points that clears its margin of about 23. Same data, opposite stories.
Now raise the sarcasm share to 25 percent. The overall gap grows to about 9 points and the headline question finally says yes. The idea was always sound. The test set only had to contain enough of the case it was built for.
Jordan and Mitchell’s Machine Learning: Trends, Perspectives, and Prospects surveys machine learning as a field driven by algorithms, data and computing. A field built on three legs needs questions that say which leg you are testing. A question about audio is a question about data, not about the algorithm. Writing the reason into the hypothesis tells you which leg to pull.
Chapter 3
Follow fifty years of tasks, then see the one box diagram that fits them all
Open a museum guide on your phone and point it at a painting. You say, “Who made this?” Two signals went in: a picture and your voice. One answer comes out: a name. Ten years ago that was a research demo. Fifty years ago it was a daydream. This chapter is the history of how the daydream became a list of tasks.
Slide 11 draws a timeline from 1970 to 2020. The first tasks use sound and sight together: audio-visual speech recognition, where a system reads lips as well as listening, so noise matters less. Then come content-based video retrieval (find the clip that matches a query), video event recognition (the TRECVID evaluations) and affect and emotion recognition, which reads feelings from face and voice. Around the 2000s, multimodal sentiment analysis joins them. Near the 2010s the slide marks the birth of “Language & Vision” research, and image captioning returns as the first big task of that kind.
Slide 12 zooms into 2014 to 2024, and the pace picks up. Image captioning and video captioning come first. Visual question answering (an image and a question in, an answer out) follows, then video question answering and referring expressions (point at the object a phrase names). After them come multimodal dialogue, language-and-vision navigation, large-scale video retrieval, text-to-image generation and multimodal interactive agents. A red arrow on the slide says “and many many more!”.
Drag through the years below. The step line counts how many named tasks exist by that year.
The line steps up each time a task from the lecture’s two timeline slides appears. The teal marker is the year you pick. Drag it across the decades.
The slides place tasks on an axis without printing years, so each year here is our rounding of where the box sits, good to a few years. Only the order and the rough decade come from the slides. The slides say the list is far from complete.
By 2000 the list holds four tasks. By 2016 it holds eight, three of them added in the four years before. By 2022 it holds fourteen. The curve does not just rise; it bends upward. Each new task opened a new family of datasets, and each dataset invited new methods.
Slide 13 sorts the modern work into seven areas. The rest of the lecture visits each one in order, so keep this list in mind.
Slide 14 draws the whole field in a single picture: Modality A, Modality B and Modality C flow into a box labelled Multimodal AI. Out of the box come a prediction ŷ, or a generation, or a transfer. This is the picture behind the hero at the top of the page. Here is the same box with real data flow, using made-up but typical sizes.
| Signal | What the computer receives | After its encoder |
|---|---|---|
| Image | 3 × 224 × 224 numbers (colour, height, width) | 1 × 512 vector |
| Text | L token numbers (the text cut into L small pieces, each a whole-number id), say 16 | 1 × 512 vector |
| Audio | 16,000 × seconds samples at 16 kHz | 1 × 512 vector |
An encoder is a small network that turns one signal into a list of numbers, called a representation. The model then fuses the three representations, for example by joining them side by side into one 1 × 1536 vector and passing that through a final layer. The last layer decides what comes out: one score for each possible label (predict), or a stream of new tokens (generate). The sizes are illustrative; the shape of the flow is the lecture’s box.
The third output, transfer, means the learned representation is reused. A model trained to caption images can be fine-tuned (trained a little more on the new task) to answer questions about them. Nothing new comes out of the box in that case; the box itself is the product.
The paper Foundations and Trends in Multimodal Machine Learning (arXiv 2209.03430) names three principles that make multimodal data special: heterogeneity (the signals differ in form), connections (they share information) and interactions (together they create new information). It then sorts the field into six challenges: representation, alignment, reasoning, generation, transference and quantification. Slide 14’s box is where all six live.
The second reading, Representation Learning: A Review and New Perspectives (Bengio, Courville and Vincent, arXiv 1206.5538), explains why the encoders matter. Its starting claim is that the success of machine learning depends on how data is represented, because different representations can hide or reveal the factors behind the data. Choose the encoder well and the final layer has an easy job.
Chapter 4
Meet datasets built from sensors, then cut a sensor stream into windows without leaking the test set
A cook lifts a cashew to a small box covered in wires. A chemical sensor inside the box reads the air above the nut every second and writes numbers into a file. After ten minutes the file holds 600 rows. The cook swaps in a peanut, then a clove, then a pineapple. Weeks later, the file collection can tell the nuts apart by smell. That is the setup behind one of the lecture’s datasets, and it shows what a “new modality” looks like in practice.
Slide 15 opens the first research area. The motivation: there are many ways to sense the world, each with complementary information, privacy concerns, and trade-offs in how invasive and efficient it is. The challenges are broad: physiological sensing, city and climate sensors, smell, taste, art, music, tangible and embodied systems, and learning when little data exists in any one modality.
The slide gives five places to begin, each a model or a dataset: brain EEG signals (arXiv 2306.16934), speech (arXiv 2310.02050), facial motion (arXiv 2308.10897), tactile sensing (arXiv 2204.00117) and Google’s SensorLM, which learns the language of wearable sensors.
CLIMB (Dai and colleagues, ICML 2025) gathers 4.57 million annotated samples from 44 datasets, with a total size the slide writes as 19.01T. The slide’s figure shows a wheel of clinical areas, chest X-ray images and sample records. The point is breadth: one place to train and test on many kinds of hospital data.
One of the best-known sources of such data is MIMIC-IV, which slide 17 draws as a tree. Three branches hang from a hospital record. Notes hold radiology reports, discharge summaries. The hospital branch holds demographics, lab tests and medications. The ICU branch holds observations and treatments. Below them sit the linked pieces: a 10-second, 12-lead electrocardiogram (a record of the heart’s electrical signal), the emergency department record (vital signs, triage, diagnosis) and a chest X-ray with annotations. One patient, six kinds of data, each in a different format.
SmellNet (Feng and colleagues, ICLR 2026) is the lecture’s smell dataset. Slide 19 states the headline: data is key. It holds more than 50 hours of recordings across 50 substances and over 300,000 datapoints. The collection pipeline in the slide has six boxes. Select a substance (nuts, spices, herbs, fruits, vegetables). Take a 10-minute sensor reading of 600 datapoints. Log the data, such as CO2 and volatile organic compounds. Reset the setup. Align the sensor data with gas chromatography and mass spectrometry results (a lab method that lists which chemicals are present) and with language descriptions. Finally, feed the downstream task, here food classification. A bracket on the slide marks the reading, logging and reset steps as repeated five times for each substance.
OpenTouch (Song and colleagues, 2025) pairs three streams: a glove that senses touch over the whole hand, a 3D hand pose, and first-person (egocentric) video. The slide names its uses: robotics, world models, and touch and haptics. Touch is rare in AI datasets, so each such dataset is a door that was shut before.
Slides 21 and 22 show a manufacturing project in two forms. A simulation, written in Python with SimPy (a discrete-event simulation library), models a chip factory with stations for diffusion and ion implantation, lithography and other machines, with buffers between them. It reports factory performance (wafers per hour), quality rate and costs. Its control levers are material flow and maintenance, and it focuses on contingencies: machine breakdowns, supply problems, power or workforce problems and quality problems.
The second form is real machine data from a partner: hundreds (the slide puts a question mark) of hours of grinding data, with acoustic emission and force signals, and of turning and milling data from an accelerometer, a microphone and a dynamometer (a force sensor). The goal is a foundation model for turning, milling and grinding.
A sensor reading is a long table: one row per time step, one column per channel. Models do not read the whole table at once. They read short pieces called windows. A window of w rows with C channels is an array of shape w × C. The window slides forward by a stride s rows each time. A reading of T rows gives ⌊(T − w) / s⌋ + 1 windows.
Overlapping windows are normal and useful, because they give the model many views of the same event. They also set a trap. If two windows share rows, and one lands in the training set and the other in the test set, the model has already seen part of the test. The test score looks better than the truth. This is leakage. The fix is to split by reading, not by window: all windows from one 10-minute reading go to the same side. Try it.
The purple line is one 600-row sensor reading. Each bar below it is a window: blue for training, yellow for testing, and red outline when a test window shares rows with training. Choose the split, then change the window and stride.
Illustrative: the curve is a seeded toy signal. The 600 rows and the 10-minute reading follow the SmellNet pipeline slide; the 4 channels, the 20 percent test share and the seeded shuffle are our choices. Split by reading puts the last 20 percent of the reading in the test set and drops windows that straddle the cut.
With a window of 60 and a stride of 30, one reading gives 19 windows, each a 60 × 4 array. In a shuffled split, most test windows share half their rows with a neighbour in training. Switch to split by reading and the leaks vanish. The price is fewer windows, because the cut drops those that straddle it.
Sensor streams, images and graphs all have a shape: a grid, a sequence, a network. The paper Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges (Bronstein, Bruna, Cohen and Veličković, arXiv 2104.13478) argues that most tasks have built-in regularities from the structure of the physical world, and that network designs should respect them. A window of a sensor stream is a tiny grid; a hand with touch sensors is closer to a graph. Our paper lesson covers the framework.
Chapter 5
See why a benchmark scores many things at once, and how the ranking moves with what you value
Two friends compare phones. One says, “Mine has the best camera.” The other says, “Mine lasts two days on a charge.” Both are right, and neither has answered the question “which phone is better?” The answer depends on what you value. Benchmarks for multimodal models face the same problem, and the lecture’s fusion slides show how researchers deal with it.
Slide 23 sets the motivation: design new ways to join and connect highly mixed signals, bridging space and time and bridging discrete and continuous data. Discrete data comes in separate items, like words. Continuous data flows, like a voice or a temperature curve. The listed challenges are the hard parts of that bridge. Mix deep learning with time-series analysis and tabular models. Fuse discrete and continuous data. Cut continuous data into tokens so it can align with text, and, on the other side, fuse and align without tokens at all. The suggested starting points are omni-modal models (one model for every input) and holistic benchmarks.
Fusion means combining signals into one decision. Slide 25 shows the two oldest options side by side. In feature-level fusion (also called early fusion) you join the representations of each signal first, then make one decision from the joined vector. In decision-level fusion (late fusion) each signal makes its own decision, and you combine the decisions.
Here is a worked example with two classes, “happy” and “sad”. The image model says [0.70, 0.30]. The audio model says [0.40, 0.60]. Decision-level fusion by averaging gives [0.55, 0.45], so the answer is “happy”. Feature-level fusion would instead join the two vectors, say two 512-number vectors into one 1,024-number vector, and let one final layer weigh them. That layer can learn that audio matters more when the image is dark, which averaging cannot do. The cost is that it needs aligned training data for both signals at once.
The slide’s other challenge is tokenizing continuous data. A language model reads a list of whole-number token ids. A temperature curve is a stream of decimals. To align the two, you can cut the curve into bins and give each bin an id, as you would a word. The cost is detail: values inside a bin look the same. The alternative the lecture also names is to fuse and align with no tokens at all, keeping the decimals as they are. Neither choice is settled, which is why it is a project area.
Our fusion lesson from the course’s later lectures builds both from zero. Here the point is narrower: a benchmark must compare these choices fairly.
MultiBench (slide 24) is a benchmark that spans domains (affective computing, healthcare, robotics, finance, human-computer interaction, multimedia), modalities (language, image, video, audio, time series, force sensors, proprioception, sets, tables, optical flow) and, crucially, three evaluation axes: performance, complexity and robustness. Slide 25’s MultiBench++ broadens it: more specialised domains, over 15 modalities, more advanced fusion paradigms and automated hyper-parameter tuning, so a method is not blamed for a bad setting.
HEMM, the Holistic Evaluation of Multimodal Models (slide 26), turns the idea around. It lists the factors that differ among models: size, how they process each modality, architecture, pretraining objective, fine-tuning objective and training data. It tests basic skills (interactions, alignment, reasoning, external knowledge), the flow of information (translation, editing, querying, fusion) and real-world use cases across multimedia, affect, science, healthcare and human-computer interaction. The aim is to find which factor explains which score.
Why three axes for MultiBench? Because accuracy alone hides two things a user cares about. Complexity measures cost: parameters, training time, memory. Robustness measures what happens when the input is noisy, partly missing or from a new distribution. A model that wins on accuracy but breaks when one sensor drops out may be the wrong pick for a real product.
Three toy models are scored below on all three axes. Each slider is how much you care about that axis. The total is a weighted sum of three scores on a 0 to 1 scale. Watch the winner change as you change what you value.
Each bar is a model’s weighted total. Start with all weights equal, then push “Performance” to the top and the others to zero.
Illustrative: the three models are invented. Large: 82 percent accuracy, 300M parameters, 60 percent accuracy under noise. Mid: 80, 40M, 75. Small: 78, 10M, 70. Cost score is (9 − log10 parameters) / 2, so 10M scores 1.0 and 300M scores 0.26. Totals are weighted averages. MultiBench defines the three axes; these numbers are not its results.
With equal weights, the Small model wins at 0.83, followed by Mid at 0.75 and Large at 0.56. Put every weight on performance and Large wins at 0.82. Put the weight on performance and robustness only and Mid wins at 0.78. The models did not change. Your question did. That is why a benchmark should report each axis, not only a single rank.
The paper Foundations and Trends in Multimodal Machine Learning (arXiv 2209.03430) lists alignment and representation among its six challenges, and fusion is where they meet. It also stresses quantification, meaning that we need ways to measure heterogeneity, interactions and robustness, which is the job these benchmarks do.
Chapter 6
Read the classic image and video datasets, then the tests built to catch models that only look smart
Show a friend a photo of a baseball game and ask for a caption. Easy. Ask, “How many players are in the picture?” Easy again. Now show two photos and ask, “Is the sentence ‘the left image has twice as many dogs as the right’ true?” Harder. Last, show a puzzle made of flags and ask what single word it spells. Now your friend needs several steps. This chapter climbs that ladder, one dataset per rung.
MS COCO (Microsoft Common Objects in COntext, slide 28) holds 120,000 images. Each comes with five free-form sentences that describe it, at least 8 words each, written by crowd workers on Mechanical Turk. It also carries object detections, boundaries and keypoints. Five captions per image matter: two people describe the same scene in different words, so a model that writes a caption can be compared with five references, not just one. One example caption on the slide reads, “The man at bat readies to swing at the pitch while the umpire looks on.”
In visual question answering (VQA, slide 29), the task is: given an image and a question, answer the question. The slide shows a person with a banana held like a mustache, a pizza, a cartoon scene. The questions range from colour to counting to what something is made of.
TVQA (slide 30) moves to video: questions on clips from six popular TV shows. The dataset holds 152.5K question-answer pairs from 21.8K clips (the paper puts it at 152,545 pairs from 21,793 clips, over 460 hours). The questions are compositional: to answer, a system must find the right moment, understand the spoken dialogue in the subtitles and recognise what is on screen.
NLVR (Cornell Natural Language for Visual Reasoning, slide 31) gives 92,244 pairs of sentences and synthetic pictures of coloured blocks, and asks whether the sentence is true or false of the picture. NLVR2 (slide 32) keeps the task and swaps in over 100,000 real photographs, such as two pictures of dogs. The shift from drawings to photos is the point: real images are messy.
The next device shows what one record of each looks like, so you can see what goes in and what comes out.
Choose a dataset. The panel shows what one example holds, the inputs in blue and the answer in warm orange, and the size of the pile.
The field names and shapes are our sketch of a typical loader, not the datasets’ own file formats; H × W is the image size. Sizes and example sentences come from the lecture slides, and the TVQA numbers also from its paper. Answers marked illustrative are ours.
Every record has the same shape: some signals in, one target out. A training loop reads a batch of records, runs the model, compares its output with the target, and updates the weights. That loop does not care whether the target is a sentence, a choice or a true-or-false bit; only the final layer and the loss change.
Slide 27 opens the reasoning research area. The motivation is robust, reliable and interpretable reasoning in multimodal language models. The challenges: fine-grained and compositional reasoning, neuro-symbolic reasoning (mixing neural networks with explicit rules), and emergent reasoning in large foundation models. Four starting points: a survey of LLM reasoning, the paper Can LLMs actually reason and plan?, Emotion-o1 and Social Genome. Our multimodal reasoning lesson follows this area in depth.
A model can score well on VQA by learning shortcuts. It may notice that “What colour is the banana?” is usually answered “yellow” without looking. Winoground (slide 33) closes that door. It gives two images and two captions. Both captions contain exactly the same words, in a different order, and each fits only one image: “some plants surrounding a lightbulb” against “a lightbulb surrounding some plants”. To match them correctly, the model must understand which thing surrounds which. That skill is called compositionality: the meaning of a whole depends on how its parts are combined, not just which parts there are.
The paper by Thrush and colleagues reports that none of the state-of-the-art models it probed did much better than chance. Their scoring asks for three things. The text score asks that each image picks its own caption. The image score asks that each caption picks its own image. The group score asks for all four matches at once. Chance is 25 percent, 25 percent and 16.7 percent. Try a toy model that can weigh word order.
The grid scores each caption against each image. At zero the model ignores word order, a bag of words. Raise the slider to let order count. Green rings mark the better caption in each row and column.
Illustrative: the scores are invented to show the mechanism. At zero, every caption-image pair scores 0.30. Each step adds 0.01 to the two right pairs and subtracts 0.01 from the two wrong ones. Real models are not this clean. The three-score method and the chance levels are from the Winoground paper.
At zero weight all four scores tie at 0.30, so the model has no reason to prefer a pairing. It would pass the group test by luck about one time in six. Raise the weight to 1 and every right pair beats its wrong neighbour by 0.02, so all three scores read 100 percent. In a real model the gap is rarely that clean, and that is the finding: the models behave closer to the tied case.
PuzzleWorld (slides 34 and 35; Li and colleagues, ICLR 2026) takes reasoning into puzzle hunts. It holds about 500 puzzles, each annotated with the steps a human takes. The slide’s example is “Top-Down Processing”: a picture of eight flags. Step one: recognise the images as country flags (visual reasoning). Step two: identify the country names of the flags, here Barbados, Oman, Tajikistan, Seychelles, Wales, Algeria, Namibia and Argentina (external knowledge). Step three: arrange the first letters in order, B, O, T, S, W, A, N, A, which spell BOTSWANA (text reasoning). The slide shows a GPT o1 answer that gets 3 of 3 steps right. Puzzles need many inputs (text, visual, structured), many skills (logic, knowledge, spatial) and long searches.
ScienceQA (slide 36; Lu and colleagues, NeurIPS 2022) pairs a question and answer options with a context, often an image, and also a lecture and an explanation. In the slide’s example the question is which type of force from a baby’s hand opens a cabinet door, the options are “push” and “pull”, and the answer is A. The lecture says what a force is, and the explanation ties it to the picture. The extra text lets a model learn to reason in steps, and lets you check the reasoning, not only the answer.
Two more lessons in our library cover this ground from the model side: contrastive learning with CLIP, the trick behind many image-text matchers, and multimodal deep learning from Stanford CS224N.
Chapter 7
Watch an agent act on a web page, and see why raw HTML fills its memory before the job is done
You ask an assistant: “Buy me a set of earphones rated at least 4.5 stars and ship them to my house.” To do it, the assistant must open a shop, search, read ratings, compare, click, type an address and confirm. No single answer exists. The result is a sequence of actions in a changing world, and a score that says whether the shop really received the order.
That is the interactive-agents area on slide 37. The motivation: ground AI models in the web, in a computer or in other virtual worlds, so that they help people with digital tasks. The challenges include grounding instructions in web images, tools and APIs; searching an environment and planning long action sequences; asking a human for clarification; and the risks of acting in the real world. The starting points are WebArena, AgentBench, Toolformer, SeeAct and a paper on socially intelligent AI agents.
A warm-up task comes from WebQA (slide 38). Given a question Q and a list of sources S = {s1, s2, …}, the system must do two things. First, identify which sources hold the answer. Second, write the answer as a complete sentence. The slide’s example asks at which festival you can see a castle in the background; the sources are photos and captions, and only some of them help. It is search and answering in one job.
WebArena (Zhou and colleagues, ICLR 2024; slides 39 and 40) builds a world, not a test set. It runs self-hosted copies of open-source web sites in four categories: shopping, content management, Reddit-style forums and GitLab, filled with data taken from real sites (Amazon, Reddit, GitHub). An agent receives an instruction, such as buying the earphones, and sends actions. The environment returns the new page as HTML plus feedback. A checker then decides whether the task really succeeded, by looking at the site’s final state.
The headline: tasks are easy for humans, who succeed 78 percent of the time, and hard for language-model agents, which succeeded 14 percent of the time in the paper (14.41 percent for the best GPT-4 agent against 78.24 percent for people). Slide 40 adds the catch: these tasks were designed to use only text and HTML source.
Slide 41 lists the reasons. Raw HTML and JavaScript are usually minified, which makes them hard to read. Some elements do not show correctly in the HTML: code that moves things after the page loads, or the spatial layout, which HTML does not convey well. A page is large: complex pages easily fill more than 100,000 tokens of context. Most of all, the web is built for human eyes. A general agent will have to process it as people do, and one day agents will move beyond the web to the world.
Here is the real data flow. At each step the agent receives an observation (the page), and it replies with one action from a small set. The WebArena family uses commands like these:
agent loop# one step of a web agent (sketch) obs = browser.observe() # HTML text, or a screenshot, or both prompt = [TASK, history, obs] # everything goes into the model's context action = llm(prompt) # one line, for example: # click [1234] hover [1234] type [1234] [earphones] [1] # press [Ctrl+f] scroll [down] goto [http://shop.local] # new_tab tab_focus [1] tab_close go_back go_forward # stop [answer] (ends the task, with an optional answer) browser.step(action) # the page changes; loop until stop
The command list is the one shown on the VisualWebArena slide. Each number in square brackets names an element on the page. The loop repeats until the agent says stop or runs out of room. Room is the issue, as the next device shows.
Each block is what one step costs: the page the agent sees plus a short action. The bar is the model’s whole context. Switch what the agent sees, then change the page size and the window.
Illustrative: the 100,000-token page is the slide’s “more than 100k” figure. A screenshot is assumed to cost 1,500 tokens and each action reply 300 tokens; real costs depend on the model and the image size. This toy keeps every earlier page in context and never drops anything, which real agents often avoid.
At the default setting the window holds 128,000 tokens and a raw HTML page takes 100,000. After the first step only 27,700 tokens remain, less than one more page. The agent gets 1 step before the next observation overflows. As screenshots, the same window holds about 71 steps. The lesson is not “always use screenshots”; it is that observation size sets how long a task an agent can attempt. Real agents trim, summarise or drop old pages for exactly this reason.
VisualWebArena (Koh and colleagues, ACL 2024; slides 42 to 44) adds tasks that need the picture. The slide’s example: “Buy the cheapest color photo printer and send it to Emily’s place (as shown in the image)”. The address is in an image, not in text. The benchmark spans shopping, forum and classifieds sites; its pie chart puts roughly half of the tasks on shopping sites. It also grades each task by action difficulty and visual difficulty.
OS-Atlas (Wu and colleagues, ICLR 2025; slide 45) goes beyond the browser to web, desktop and mobile apps. It is a foundation action model with three modes: grounding (given a description, find the on-screen element), action and agent. VideoWebArena (Jang and colleagues, ICLR 2025; slide 46) adds video: a task may need the agent to watch a tutorial or a product video first. Our computer-use agents lesson, the GUI grounding lesson and agent environments lesson cover these ideas in depth.
Chapter 8
Read emotion, conversation and unspoken meaning, and see how much video each kind of question needs
A friend says, “Great, just great,” and sighs. You know at once they are upset. The words are positive. The sigh, the face and the moment say the opposite. People read this without effort. Teaching a machine to do it is the whole subject of the socially intelligent AI area.
Slide 47 gives the motivation: build AI that can understand and interact with humans in social situations. The challenges: social interaction, reasoning and commonsense; building relationships over months and years; theory of mind (reasoning about what other people believe and want) and conversations with several people. The starting points: the Multimodal WereWolf game data, Ego4D, MMToM-QA and the CMU course Artificial Social Intelligence.
Affect recognition (slide 48) is the oldest task here. The lecture shows the AVEC challenge datasets (Audio-Visual Emotion Challenge), which ran in 2011 and 2012, 2013 and 2014, 2015, 2016, 2017 and 2018. Participants get audio and video of a person and predict dimensional emotion: instead of picking one word like “sad”, the system predicts numbers on continuous scales, labelled for every frame. Different years add more: transcripts (2011 and 2012), depression labels for each person (2013, 2014 and 2016), a reading task (2013 and 2014), physiological data (2015 and 2016) and depression or bipolar labels (2017 and 2018).
Think about the shape of one example. A clip with T frames has a label of shape T × 2, one pair of numbers per frame (the two usual scales are how pleasant the feeling is and how intense it is). The model slides along the clip and writes a trace, not a single answer.
CMU-MOSEI (slide 49) is a multimodal sentiment and emotion set with 23,453 annotated video segments from 1,000 distinct speakers on 250 topics. Sentiment is the overall positive or negative feeling in a segment. It is the classic case for adding audio and video to text, as Chapter 2’s toy sarcasm example showed.
MELD (slide 50) moves from one speaker to many. It labels emotion in conversations among several people, so each line must be read in light of who said what before it. Social-IQ (slide 51) goes further: 1.2k videos, 7.5k questions and 50k answers centred on social behaviour. The slide shows a conversation in a room with questions about how people feel and why. Answering needs both the dialogue and the body language.
Almost all datasets above lean on speech. MimeQA (Li and colleagues, NeurIPS 2025; slides 52 to 54) takes the words away. It uses videos of mime performers, who act with gestures and body movement alone. The questions come in three levels, and each needs a longer stretch of video than the one before.
The level changes how much video the answer needs. That has a practical effect. Video models usually read a few frames sampled from the clip, not every frame. If those few frames miss the moment that matters, the model cannot succeed, however clever it is. Test it with a toy clip.
The bar is a 60-second clip in three scenes. Yellow marks are the frames your model sees, spaced evenly. The teal span is the evidence the question needs. Pick a level, then change the number of frames.
Illustrative: the clip is 60 seconds in three 20-second scenes; a grounding question needs a 6-second moment (22 to 28 s); a scene-level question needs 2 frames inside scene 2; a global-level question needs 1 frame in every scene. These rules are our toy, chosen to match the three levels in spirit. MimeQA’s real clips and thresholds differ.
Try 3 frames. They fall at 10, 30 and 50 seconds, one in each scene. The global question passes, because it needs one frame per scene. The scene-level question fails, because only one frame lands in scene 2 and it needs two. The grounding question fails too, because the single frame at 30 seconds misses the 6-second moment at 22 to 28. Raise the count to 4 (frames at 7.5, 22.5, 37.5 and 52.5 seconds) and all three levels pass. The lesson for a researcher is that a poor result may reflect what the model saw, not what it can reason.
The last slide of this area is the Human Behavior Atlas (Ong and colleagues, ICLR 2026). Its figure shows a large collection that brings many datasets of human behaviour under one taxonomy, with charts that break the data down by task and modality. The idea echoes CLIMB for health in Chapter 4: when many small datasets exist, pool them with shared labels so one model can learn from all of them.
Chapter 9
Follow an instruction through a house, a robot arm and a question to a human, and weigh the risks
You tell a home robot, “Walk past the piano, turn left at the hallway, and wait by the grey door.” The robot has a camera, a map it has never seen and a body that can bump into things. Every sentence has to turn into movement. That is the embodied AI area, and it ends this tour with two questions that matter for every system: can people trust it, and can it cause harm?
Slide 56 states the motivation: build tangible, embodied AI systems that help people with physical tasks. The challenges: perception, reasoning and interaction; connecting sensing to actuation (the motors and grippers that act); efficient models that run on hardware; and understanding how actions change the world, which is a world model. The starting points are four simulators and one paper: VirtualHome, Habitat 3.0, RoboThor, LangSuite-E and Language Models Meet World Models.
Room-to-Room (R2R, slide 57) is the classic navigation task: 21,567 open-vocabulary navigation instructions, written by crowd workers, for walking through real buildings. The slide’s example reads, in part, “Head upstairs and walk past the piano through an archway directly in front”. Room-Across-Room (slide 58) is larger, multilingual and uses longer paths. The model sees what the camera sees, reads the sentence and picks a move. The test is whether it reaches the goal.
Two datasets record real life from the wearer’s own eyes. EPIC-Kitchens (slide 59) is a large first-person (egocentric) set, audio-visual and non-scripted, filmed in the wearers’ own homes. Ego4D (slide 60) spreads across the world; its slide is a map of scenarios, each a cluster of videos, from social interaction to geographic variety. First-person data matters because it shows hands acting and attention moving, which a camera on a wall never shows.
Slide 61 puts the whole idea in one sentence: generate precise robot control directly from trained vision-language models. OpenVLA (Kim and colleagues, 2024) is a vision-language-action model. It is a 7-billion-parameter open model trained on 970,000 real robot demonstrations. It builds on a Llama 2 language model with a visual encoder that fuses DINOv2 and SigLIP features. The paper reports that it beats the closed 55-billion-parameter RT-2-X by 16.5 points of absolute task success across 29 tasks, with seven times fewer parameters. Slide 62 shows Google RT-X, a related effort to scale learning across many different robot types, shown as a grid of robots and labs.
The data flow is simple to state. In: the camera image, say 3 × 224 × 224, and a sentence such as “put the carrot in the bowl”. Out: an action, typically a short list of numbers for the arm: how far to move in three directions, how far to rotate, and whether to open or close the gripper. The model repeats this several times a second. It reads, acts, and reads the changed scene. Our vision-language-action lesson and the world models lesson build both halves from scratch.
Slide 63 asks three questions. What is the right medium for human-AI interaction? How can we really trust AI? How do we enable collaboration? The challenges include modelling and conveying uncertainty: not only for text, but for pictures and for the interaction between signals. They include asking for human clarification, and learning from human feedback in every modality. The starting points are the MMHal-Bench paper on aligning multimodal models and HACL, on hallucination.
Uncertainty is a practical tool, not only a topic. If a model can say how sure it is, you can let it answer when it is sure and ask a person when it is not. Each question costs a human’s time. So there is a trade: more questions buy more accuracy. The device below lets you choose where to draw the line.
The curve shows accuracy against the share of items sent to a person. The teal dot is your threshold: the model asks a human whenever its confidence falls below it.
Illustrative: 400 made-up items with confidences spread between about 30 and 100 percent, a model that is right exactly as often as it says (perfectly calibrated), and a human who is right 97 percent of the time. Real models are often over-confident, which would make the curve flatter. The seeded draw is the same on every visit.
With no threshold the model answers everything and scores about 68 percent in this toy. Ask a person whenever confidence falls under 50 percent and 73 of the 400 items (18 percent) go to the human, while accuracy rises to about 78 percent. Raise the line to 80 percent and 245 items (61 percent) go to a person, for about 93 percent. The curve bends: the first questions buy the most, because they cover the items where the model was weakest. Chapter 7’s agents face the same choice when they ask for clarification.
Slide 64 gives the last area its motivation: large AI models can produce unsafe text, and can generate or retrieve biased images. The challenges are to taxonomise biases across text, vision, audio and generation; to trace biases back to the pretraining data and see how they grow during training and fine-tuning; and to find new ways of reducing bias and aligning with human preferences. A good project question starts from the slide’s own prompt: many works study fairness in language models, so how do you extend their methods to multimodal models, to image captioning and image generation?
One link to earlier chapters: the datasets you picked are where bias enters. A caption set that shows certain jobs only with certain people will teach that pattern. That is why the lecture lists “tracing biases to pretraining data” as a research task, not a footnote.
Chapter 10
Search the literature, test ideas one change at a time, and plan for datasets too big for a laptop
You have a question, a hypothesis and a dataset. Now come three practical jobs: find what others have done, test your idea without fooling yourself, and fit the work onto the machine you own. The last stretch of the lecture is advice for exactly these jobs.
Slide 65 lists five places to look, in order. First, Google Scholar, to search papers and follow who cited whom. Second, Papers with Code, GitHub and Hugging Face, where a paper meets its code, models and datasets. Third, recent conference proceedings, for the newest work. Fourth, blog posts, which often explain a paper in plain words. Fifth, survey papers, tutorials and courses, which map a whole area at once.
A tip that is ours, not the slide’s: when a field is new to you, start with the fifth source and work backwards. A survey tells you the names and the open questions. Then read the abstract, the figures and the results table of three papers before you read any paper in full. The readings for this session are such surveys: Foundations and Trends in Multimodal Machine Learning maps the field, Representation Learning: A Review and New Perspectives maps how data becomes features, and Geometric Deep Learning maps how structure guides design. Machine Learning: Trends, Perspectives, and Prospects gives the broad view of why data, algorithms and computing grew together.
Slide 66 is the heart of the practical advice. Read it as a recipe.
Why “the simplest pipeline first”? Because a bug in a complex system is almost impossible to find. A simple pipeline that runs end to end gives you a baseline number and a working loop. Every later change is then measured against it.
Why “one component at a time”? Because then each score change has one cause. Take a worked example. Your baseline sentiment model reads text and scores 71.0 percent. You then add audio features and score 74.5. Next you try a bigger text encoder alone and score 74.0. Then you do both and score 75.1.
| Run | Score | Change from baseline |
|---|---|---|
| Baseline, text only | 71.0 | 0 |
| + audio | 74.5 | +3.5 |
| + bigger encoder | 74.0 | +3.0 |
| + both | 75.1 | +4.1 |
The numbers are illustrative. If you had run only the last row you would see +4.1 and not know why. With one change at a time you can see that audio gives 3.5, the encoder gives 3.0, and together they give 4.1, which is less than the sum of 6.5. They overlap: both fix the same errors. That overlap is a finding. A single big change could never have shown it. This is also the idea behind an ablation study: remove one part, see what the score loses.
Slide 67 gives four resources for the longer road: awesome-phd-advice, awesome-tips, Harvard’s CS197 course on research and the essay The Hexagon of Ideas. Slide 68 is about compute: use the tools in your research group or pair up with someone who has them, find some GPUs, and the course will provide AWS and Google Cloud credit.
Slide 69 warns that image and video datasets get big fast. Some run to hundreds of gigabytes even compressed. Space becomes a problem, and so does memory, because you cannot hold it all at once. Time to extract features and train becomes a problem as well. The advice: plan, and expect to experiment on a subset of the data on a laptop.
Try the arithmetic. Set the size of a dataset and the memory of your machine, and see how much of it you can test at once.
The tall bar is the dataset; the box is your memory. The yellow bar is the time for one pass of feature extraction. Choose your memory, then change the dataset size.
Illustrative: 5,000 clips per gigabyte (about 200 KB each) and a feature extractor that handles 50 clips per second on a laptop. The subset rule keeps a test set at half of memory and one pass under one hour. Real numbers depend on clip length, resolution and hardware.
At 316 GB, the set holds about 1.58 million clips. On a 16 GB machine it is about 20 times too big for memory, so you must read it in batches from disk. One pass of feature extraction takes about 8.8 hours. A subset you can run in under an hour is 11 percent of the data, about 36 GB of the 316; memory caps the subset at 2.5 percent, so choose 2.5 percent, 8 GB, for a first test. The point is not the exact numbers. The point is that you can do this sum on day one, before the weekend is lost.
Slide 70 closes the lecture with the logistics, in short. Fill in the project preference form, which helps match teams. Homework 1 is multimodal data processing and visualization: find, process, visualize and label a multimodal dataset of your own choice. Next Tuesday there is a tutorial on data processing and basic machine learning code.
Chapter 11
Review the numbers, the checklist and the core code, then look ahead to the rest of the course
You can now retell the second lecture of MIT’s Multimodal AI course from memory: how research moves around its loop, how to write a question that can fail, the map of tasks, and the datasets behind each research area. This page keeps the numbers, a checklist, the core code and the way onward.
Research is a loop: observe, read, ask, test, analyze, report. Ideas come bottom-up from failures or top-down from big questions. A good question has a yes or no answer, and a good hypothesis can be proved wrong; beware “does X make Y better?” without a reason. Every multimodal task fits one box: signals in, a prediction, a generation or a transfer out. The lecture visits seven areas, each with datasets to start from: new modalities (CLIMB, MIMIC, SmellNet, OpenTouch, IntelMini Fab), fusion and alignment (MultiBench, HEMM), reasoning (COCO, VQA, TVQA, NLVR2, Winoground, PuzzleWorld, ScienceQA), interactive agents (WebArena and its visual and video cousins, OS-Atlas), social intelligence (AVEC, CMU-MOSEI, MELD, Social-IQ, MimeQA), embodied AI (R2R, EPIC-Kitchens, Ego4D, OpenVLA, RT-X) and human-AI interaction with ethics and safety. Test ideas with the simplest pipeline, one change at a time, on a subset.
| What | Number | Why it matters |
|---|---|---|
| CLIMB | 4.57M samples, 44 datasets, 19.01T | Pooling many health datasets |
| SmellNet | over 300,000 datapoints, 50 substances, 50+ hours | A new sense, a new dataset |
| SmellNet reading | 10 minutes, 600 datapoints | Split by reading, not by window |
| MultiBench axes | performance, complexity, robustness | A rank depends on the weights |
| MS COCO | 120,000 images, 5 captions each | Many references per image |
| TVQA | 152.5K questions, 21.8K clips | Needs moment, dialogue and picture |
| NLVR | 92,244 sentence-image pairs; NLVR2 over 100k real images | True or false about pictures |
| Winoground chance | 25%, 25%, 16.7% | Same words, different order |
| PuzzleWorld | about 500 puzzles, 3-step flag example | Reasoning in many steps |
| WebArena | 78% human, 14% best LLM agent | Acting is harder than answering |
| HTML page size | over 100k tokens | Fills the context in one or two steps |
| CMU-MOSEI | 23,453 segments, 1,000 speakers, 250 topics | Sentiment from face, voice and words |
| Social-IQ | 1.2k videos, 7.5k questions, 50k answers | Questions about social behaviour |
| Room-to-Room | 21,567 instructions | Language to movement |
| OpenVLA | 7B parameters, 970k demonstrations | Words to robot control |
This sketch is the lecture’s eight steps for a small sentiment project. load_clips, encode and fit stand for whatever loader, encoder and trainer you use.
pythondef split_by_reading(clips, test_frac=0.2): # Ch 4: never split inside a reading readings = sorted({c.reading_id for c in clips}) cut = int(len(readings) * (1 - test_frac)) train_ids = set(readings[:cut]) return [c for c in clips if c.reading_id in train_ids], [c for c in clips if c.reading_id not in train_ids] def run(use_audio=False, encoder="small", subset=0.05): # Ch 10: one switch per change clips = load_clips(fraction=subset) # a subset on a laptop train, test = split_by_reading(clips) X = [encode(c.text, c.audio if use_audio else None, encoder) for c in train] # (N, d) model = fit(X, [c.label for c in train]) # a basic model preds = model.predict([encode(c.text, c.audio if use_audio else None, encoder) for c in test]) return score_by_slice(test, preds) # Ch 1, 2: slices and margins table = {name: run(**cfg) for name, cfg in { "baseline": {}, "+ audio": {"use_audio": True}, "+ encoder": {"encoder": "big"}, "+ both": {"use_audio": True, "encoder": "big"}}.items()}
Read it and name the chapter behind each line. The split keeps each reading whole. The run function has one switch per change, so the table shows which change did what. The data is a subset, so the loop finishes in minutes. The score is reported by slice, so a gap like Chapter 2’s sarcasm slice cannot hide inside an average.
This lecture is the second of the course. It sets the vocabulary of tasks and datasets that every later lecture uses. The lecture on modality heterogeneity takes the box of Chapter 3 and asks how signals differ. Then fusion and interactions between signals build the Chapter 5 ideas. Multimodal reasoning follows the Chapter 6 area.
Credit: this lesson re-teaches Lecture 1.2 of MIT MMAI: Multimodal AI (Spring 2026), by Paul Liang and the MIT Multisensory Intelligence group. The slides and all datasets belong to their authors. The explanations, devices and toy numbers are ours.
Now press Present or Teach and explain the lecture out loud, from memory: the research loop, a question that can fail, the box that fits every task, and one dataset from each area. Then go back to the task builder at the top and find a combination of signals with no match in the list. That empty shape may be your project.