A language model that only reads text can learn to see by training one small adapter while its own weights stay frozen. Switch the adapter on and watch a blind model answer.
Learn how one transformer reads words, pictures and sound together, how a frozen LLM is taught to see, and how it learns to draw.
Pick a question, then switch between the text only model and the frozen model with an adapter. See what the adapter changes, and what it leaves alone. Then we build every piece from zero: the attention that mixes modalities, the training tasks, the adapters, the data, and the tricks that let a language model produce images.
You need a rough idea of what a neural network is (a stack of adjustable number tables) and what a word vector is (a list of numbers that stands for a word). We build the rest from zero.
Based on Lecture 5.1 of MIT Multimodal AI (Spring 2026), taught by Paul Liang of the MIT Media Lab and MIT EECS, Multisensory Intelligence group. Course site and slides PDF. Every idea is re-taught here in our own words and our own devices.
Question: What color is the car?
The three questions and the answers on the lecture’s slides 18 to 20 (the small red boat, blue, the Wright brothers and Steve Jobs) come from the Frozen paper’s examples. The plane question below is a toy stand in. The patch counts, the 2 prefix vectors and the text only answers are illustrative toys, not the paper’s measurements.
Chapter 0
See why words alone miss half of a scene, and meet the three jobs a large multimodal model must do
Picture a sitcom scene. A man looks at his friend and says, flatly, “It’s just a privilege to watch your mind at work.” Read those words alone and they sound like a compliment. Now add the clip: his voice is dull, his eyes roll, his friend winces. Everyone watching knows the man is being sarcastic (saying the opposite of what he means).
The meaning did not sit in the words. It sat in the words, the voice and the face, taken together. A system that reads only the text gets the scene wrong. This is the whole reason the lecture exists.
The lecture opens with a short recap. A large language model (LLM) is a neural network trained on huge amounts of text to predict the next word. That one skill turns out to cover many jobs: answering questions, open dialog, translating between languages, retrieving news, solving math problems and writing code. The slide credits Brown et al. (NeurIPS 2020), the GPT-3 paper, which showed that a big enough model can pick up a new job from a few examples placed in its prompt.
So the question for this lecture is simple. Can we keep everything that makes LLMs useful, and let them take in pictures, video and sound as well?
Slide 5 uses the sitcom still and asks five kinds of question, from the survey of Liang, Zadeh and Morency (ACM Computing Surveys, 2024). Each one needs a stronger system than the last.
Slide 6 draws one diagram, and slides 7 to 9 reveal it in three steps. On the left, the inputs: the words, and the video frames, and the sound. All of them flow into one box, the multimodal representation. This is a bundle of numbers (vectors) that holds what the model understood from all inputs at once. Think of it as the model’s private notes about the scene.
From the notes, the model can do three jobs, and the lecture builds them in order.
Slide 3 adds a fourth item, the latest directions: natively multimodal models, multimodal mixtures of experts, and real world sensors. We reach it in Chapter 9. Try the three jobs on the sitcom scene below.
Words, face and voice enter on the left and meet in the shared notes. Choose a job and see what leaves on the right and which parts must exist to do it.
Illustrative sizes: three inputs, a 4 number summary, an 11 word answer. The sentences are the ones drawn on slides 6 to 9; no real model produced the numbers.
Notice what changes between the jobs. Understanding needs only a small head on top of the notes. Explaining needs a language model that can write. Creating a picture needs a third part, an image maker. The rest of the lecture is about how to build each part cheaply and join it to the others.
Slides 2 and 36 are housekeeping. Homework 2 was due on Wednesday 4 March, Thursday’s class held the homework 2 presentations plus an in-depth tutorial on implementing multimodal LLMs, and homework 3, on multimodal LLMs, came out later that week. Everything in this lesson is a direct preview of that homework.
We follow the slides in order, and we add the lecture’s reading list at the right place. Each chapter opens on a scene, shows the idea in plain words, then lets you operate it. The last chapter is a field guide: a table of numbers, a checklist, the core code, and the links onward.
Chapter 1
Let words, frames and sound attend to each other, and see how one word borrows evidence from a face
You are watching the sitcom clip. Words arrive about three per second. Video frames arrive thirty times per second. Audio features arrive a hundred times per second. The man’s eye roll lasts one second, but it falls between two words. Nothing lines up on a clock, and yet your brain links “privilege” to the eye roll at once.
Slide 10 poses exactly this question: how do we learn temporal representations from multiple modalities? A modality is one kind of signal, such as language, visual or acoustic. The slide’s answer is to put all three streams into one transformer and let self-attention sort out the links. The slide calls this implicit alignment, because we never tell the model which frame matches which word.
A transformer works on a list of tokens, small pieces of input, each stored as a vector (a list of d numbers). Self-attention lets every token look at every other token and copy useful information from it. Think of a meeting. Each person asks a question (“who here knows about eyes?”), each person wears a name tag, and each person holds some knowledge to share.
Those three roles have names. The question is the query. The name tag is the key. The knowledge is the value. A token makes all three by multiplying its vector with three learned tables, written Wq, Wk and Wv.
To decide whom to listen to, a token compares its query with every key. The comparison is a dot product: multiply the two vectors position by position and add. A big result means a good match. A softmax then turns the list of match scores into weights that are positive and add up to 1. The token finally adds up the values, each multiplied by its weight.
Slide 10 stacks the language tokens (hL), the visual tokens (hV) and the acoustic tokens (hA) side by side and runs one block of self-attention over the lot. The output is a contextualized representation: each token’s vector now carries a bit of every token it listened to. A word like “privilege” leaves the block holding a little of the eye roll and a little of the flat voice.
Slide 11 zooms into one piece of this, vision-to-language attention. Only the language tokens ask questions. Only the visual and acoustic tokens hold keys and values. So “privilege” and “mind” query the cues “vocal emphasis” and “eye-roll”. The slide ends with a new language representation, contextualized with vision, which a classifier reads to decide: sarcasm. The slide credits two papers: Liang et al., recurrent multistage fusion (EMNLP 2018), and Tsai et al., the Multimodal Transformer (ACL 2019).
The slide marks the shapes 3 × 4, 3 × d, d × 4 and 4 × d, and says to normalize each row to 0 to 1. Here is one row worked by hand. Say “privilege” scores the four visual cues [2.0, 1.0, 0.0, −1.0], and d = 4, so √d = 2. Divide: [1.0, 0.5, 0.0, −0.5]. Exponentiate: [2.718, 1.649, 1.000, 0.607], which sums to 5.974. Divide by the sum: [0.455, 0.276, 0.167, 0.102]. The weights add up to 1, so the best cue gets the biggest share but no cue is ignored.
Below, three language tokens ask about four visual and acoustic cues. The vectors are random but fixed, so the mechanism is real even though the numbers are toys. Pick a word, then drag d, the vector length. Switch the shrink factor off and see what happens.
Bars show one row of the 3 × 4 weight grid. The top bar group is the weights; the strip below shows the new vector, a blend of the four value vectors.
Illustrative: the vectors are seeded random numbers, one set per length d. Without the divide, scores grow with √d, so the softmax saturates toward one cue. The slide’s own 0.7 and 0.3 weights are a drawing, not a measurement.
With d = 4 the weights stay spread out. With d = 256 and no shrink factor, one cue often takes almost all the weight, and the others vanish. That hurts learning, because a softmax that has saturated passes almost no gradient (a signal telling the weights how to change). Dividing by √d keeps the scores at a steady size whatever d is.
The slide cites Tsai et al. (ACL 2019), “Multimodal Transformer for Unaligned Multimodal Language Sequences”, which this series teaches in its own paper lesson. Its idea is the one on slide 11, applied to every pair. Each modality gets its own cross-modal block in which it asks questions of another modality. Because the keys and values may be longer or shorter than the queries, the method needs no word-by-word alignment between 30 frames per second and 3 words per second. The attention finds the links itself.
As data flow: in, a language tensor of shape [TL, d] and a visual or acoustic tensor of shape [TV, d]. The lengths TL and TV can differ. Out, a tensor of shape [TL, d]: the language sequence, enriched. The cost is one TL × TV grid of scores, so a long video costs more.
Chapter 2
Wire a text transformer to an image transformer with cross attention, then train it by hiding things and asking it to find them
Imagine two experts at a table. One reads captions. The other studies photographs. Neither speaks the other’s language, but they can pass notes. The caption expert writes, “I see the word dog, does the photo have one?” The photo expert answers with what it sees at that spot. After a few rounds of notes, both understand the scene better than either could alone.
That is the design on slide 12, multimodal cross-attention transformers. The slide shows ViLBERT (Lu et al., NeurIPS 2019). Chapter 1 gave us the tool, cross-attention. Now we plug it into a whole network.
ViLBERT has two towers. The unimodal transformer in each tower is an ordinary transformer that sees one modality only. The text tower reads word tokens, starting with a special [CLS] token (a slot that collects a summary) and ending with [SEP] (a separator). The image tower reads a list of image region vectors, starting with its own special token, [IMG].
In between sit the cross-modal transformer modules, drawn in red on the slide. In such a module, the text tower makes its queries from text, but takes its keys and values from the image tower. Then the image tower does the reverse. Each side asks questions of the other, which is why this is called co-attention.
The slide labels the repeated blocks L − k and k. The tower runs some number of purely unimodal layers, then k layers that swap notes. Why not swap in every layer? Words and image regions work at different speeds. Text needs several layers of its own thinking first, and sharing too early mixes up two half-formed ideas.
ViLBERT trained on Conceptual Captions, a set of about 3.3 million image and caption pairs scraped from the web. No human labelled any region. The captions themselves are the teacher.
The slide also cites UNITER (Chen et al., ECCV 2020), which takes the opposite path. UNITER puts the words and the image regions into one single list and runs one transformer over it. Every token can attend to every other token, in every layer. This is the same idea as slide 10 in Chapter 1: self-attention over a mixed list. It needs no special cross-modal block, at the price of a longer list (the cost grows with the square of the length).
Nobody can write labels for 3.3 million photographs. The trick is self-supervision: build the question and its answer out of the data itself. Slide 13 lists UNITER’s training tasks. We use the slide’s example sentence, “man with his dog on a couch”.
The ITM task is a yes or no question, so its loss (a score for how wrong the model was) is simple. If the model says “match” with probability 0.9 and the pair truly matches, the loss is −ln 0.9 = 0.105. If the pair does not match, the same answer costs −ln 0.1 = 2.303, twenty-two times more. A confident wrong answer hurts the most.
Choose a task, then drag how much is hidden. Teal frames show what the model must work out; grey cells are hidden.
The sentence is slide 13’s example. The 14 × 14 grid is 196 patches, the size a 224 pixel image has with 16 pixel patches (as in the MAE paper). UNITER masks region features from a detector, not pixel patches, so the grid is a stand in. Which words or patches are hidden is seeded, not learned.
Try 15% on masked words: one of the seven words hides, the usual BERT style rate. Now try the same on image patches, then drag to 75%. Words carry a lot of meaning per token, so we hide few. Image patches repeat their neighbours, so a model can hide most of them and still recover the scene.
He et al., “Masked Autoencoders Are Scalable Vision Learners” (arXiv 2111.06377), push the image side to the limit and have their own lesson in this series. They hide 75% of the patches. The encoder reads only the 25% that stay visible, which is 49 of 196 patches. A small decoder then paints the missing patches back. Because the big encoder never touches the hidden patches, pre-training gets much cheaper. This is the same hide and find game, tuned for speed.
For a training example, in: word tokens [7] and a visual list [196, d], with some entries replaced by a mask marker. Out: for MLM, a probability over the vocabulary at each masked word. For MRM, a vector (or a class) for each masked region. For ITM, one probability from the [CLS] slot. No human label is needed in any of the three.
Chapter 3
Drop the slow image detector with ViLT, then see why ALBEF lines up pictures and words before it mixes them
Suppose you build a photo search app in 2019. To find “a dog on a couch” the pipeline first runs an object detector, a big network that draws boxes around things in the picture. Only then does a language model see the boxes. Profile the app and you find a surprise: the detector takes more time than the part that understands language. The slow piece is not the clever piece.
Slides 14 and 15 show how ViLT removes it. Slide 16 shows a second fix, ALBEF, for a different problem: the mixing step works better when the two sides already agree on a common space.
ViLT (Kim et al., ICML 2021, the full title is Vision-and-Language Transformer Without Convolution or Region Supervision) is drawn on the slide as “BERT plus ViT”. BERT is the text transformer of Chapter 2. ViT, the Vision Transformer, is a transformer that reads an image as a list of patches (see our Vision Transformer lesson).
Here is the recipe. Cut the picture into square patches, 32 by 32 pixels in ViLT. Flatten each patch into one long vector of pixel values. Multiply by one learned table, a linear projection, to get a vector of d numbers. Add a position embedding (a learned vector that says where the patch sits). Do the same for the words. Put both lists end to end and give them to one transformer encoder. That is all. No detector, no convolution network.
The slide marks the input stack: a word embedding on one side, a linear projection of flattened patches on the other, with a modality type embedding added so the model knows which is which. On top sit three heads: image-text matching (is this caption right?), masked language modeling (fill in a hidden word), and a word patch alignment score computed by optimal transport.
Slide 14 labels the alignment loss “optimal transport”. The idea is easier than the name. Imagine each word holds one bucket of sand and each patch is an empty hole. Moving sand from word i to patch j costs more the farther apart their vectors are. Optimal transport finds the cheapest plan that moves all the sand. The plan is a table: how much of each word goes into each patch.
Training makes that cheapest plan even cheaper, which pulls each word’s vector toward the patches it names. Slide 15 shows the result: for a photo of cottages and a wall, the word “wall” lights up the wall patches and “flowers” the flower patches, with nobody ever drawing a box.
ALBEF (Li et al., NeurIPS 2021, “Align before Fuse”) is drawn as “BERT plus ViT plus CLIP-ish”. Its picture has three blocks. An image encoder (a 12 layer ViT) reads the picture alone. A text encoder (6 layers) reads the words alone. A fusion encoder (another 6 layers with cross-attention) mixes them. The point is the order: before mixing, the two single-modality encoders are trained so that a matching picture and caption land close together. The slide names this “close alignment”.
Why bother? Cross-attention works best when the two input lists already live in a shared space. If the image vectors and word vectors point in unrelated directions, the fusion layers must learn translation and reasoning at the same time. Aligning first splits that job in two.
The tool is contrastive learning. Take a batch of N image and caption pairs. Compute the similarity (cosine of the angle between vectors, divided by a temperature τ) of every image with every caption. That is an N × N grid. For image i, a softmax over its row gives the probability of each caption. The loss is −ln of the probability on the true caption, averaged over the batch. A model that cannot tell captions apart gives each one 1/N, so its loss is ln N. For N = 4 that is ln 4 = 1.386.
The slide notes that this is related to mutual information. Indeed, ln N minus the loss is a lower bound on the mutual information between image and text, the amount one tells you about the other. A lower loss proves the pairs share at least that much. Try it with 4 pairs.
Rows are images, columns are captions. Each cell shows the row’s softmax probability. The diagonal (green frame) is the true pair. Drag alignment to train.
Toy: four caption vectors sit at 0, 90, 180 and 270 degrees. Each image vector starts at a fixed wrong angle and rotates toward its caption as alignment grows. The temperature is 0.2. The loss is the real softmax cross entropy over rows; the 1.386 guess line is ln 4.
At low alignment the grid is blurry and the loss sits above the guess line, or near it. At full alignment each row puts almost all its probability on the diagonal, and the loss falls toward zero. The mutual information bound rises to its ceiling of ln 4.
Besides the contrastive loss, ALBEF keeps the image-text matching loss and the masked language loss, and it makes the matching loss harder with hard negatives: wrong captions that the contrastive step found most similar to the picture. Easy mismatches teach little; near misses teach a lot. It also trains with momentum distillation. Web captions are noisy, so a slowly moving copy of the model (an average of its past selves) writes soft targets, and the live model learns from both the noisy label and that steadier teacher.
As data flow: in, an image [3, 224, 224] and tokens [T]. Unimodal out, one summary vector each, used for the contrastive grid. Fusion out, a mixed sequence read by the matching and masking heads.
Chapter 4
Keep the language model exactly as it is, train a small adapter that speaks its language, and watch it learn from pictures it was never trained on
You have a brilliant friend who has read most of the internet but has been blind since birth. You cannot operate on their brain. It would cost millions, and you might damage what already works. But you can sit beside them and whisper what the camera sees, in words they understand. The friend then answers as well as ever.
Part 2 of the lecture (slide 17) is built on this idea. A big pretrained language model is already good at language. We leave it frozen, which means its weights (the numbers it learned) never change. We add a small adapter that turns a picture into something the language model can read. Only the adapter trains.
Look at the first step inside any language model. Each word becomes a vector from a table, called an embedding. From that point on, the model works only with vectors. It does not know, or care, that they came from words. If we can produce vectors that sit in the same kind of space, the model will read them like words.
This is prefix tuning, the name on slides 18 to 21: “conditioning via prefix tuning”. A prefix is a few extra vectors placed at the front of the input. The adapter’s whole job is to turn a picture into a prefix.
The paper on slide 18 is Frozen (Tsimpoukelli et al., NeurIPS 2021). The slide shows a stack of two parts: an adapter, and the pretrained language model with a snowflake on it. A picture goes into the adapter. The adapter outputs a few vectors, which sit at the front of the sequence. The language model, frozen, then continues the sequence with words: “A small red boat on the water.”
The slide writes the goal as modeling p(text | image): the probability of the words, given the picture. The frozen model alone gives p(text). The adapter shifts it into p(text | image). In Frozen the adapter is a vision network (a ResNet) followed by a linear map that makes 2 prefix vectors, as the paper reports. Training uses image and caption pairs, and the only loss is the usual next word loss on the caption. Errors flow backward through the frozen model into the adapter, but only the adapter’s numbers change.
Slides 19 to 21 show what falls out. Because the language model is frozen, it keeps its habit of learning from examples in its prompt. Show it a picture of a car and ask “What color is the car?” and it says “Blue”. Slide 20 shows a prompt with an aeroplane, “Who invented this?” and the answer “The Wright brothers”. Add a new picture of a phone with the same question, and the model answers “Steve Jobs”. Neither task was in training. The words of the answers come from the language model. The adapter only passed along what the picture contained.
Slide 21 goes further. The prompt holds two pictures, each followed by a made-up name: “This is a dax.” and “This is a blicket.” Then comes a new picture and a question. The model can answer with the right made-up word. It has bound a new word to a visual category after seeing a single example, using nothing but its prompt. Try it.
The row is the prompt: pictures (blue) as prefix vectors, words (orange). Switch examples on or off, choose the new picture, and see what the frozen model says.
Toy rule built from slide 21’s example: the frozen model repeats the made-up word if the prompt bound it to this kind of picture, and otherwise falls back on its usual word. The real model is not a lookup; its behaviour is reported in the Frozen paper.
With no examples the model says “an apple” because that is its habit. Show the dax example and the same model, with identical weights, now says “a dax”. Nothing was trained. The adapter turned the pictures into vectors that the language model could compare, and its in-context learning (copying a pattern seen earlier in the prompt) did the rest.
Slide 22 gives one picture, Flamingo (Alayrac et al., NeurIPS 2022), shown as a long chat with pictures woven into the text. Flamingo keeps the same rule, freeze the big models, and adds two new parts.
Flamingo trains on web pages where images and text are mixed, so it learns to read prompts like “image, text, image, text, question”. The result, in the paper, is strong few-shot performance: with a handful of examples in its prompt it beat models fine tuned on thousands of times more task data on 6 of the 16 tasks tested. The largest Flamingo is built on the 70 billion parameter Chinchilla language model, and reaches 80 billion parameters in total.
Three reasons. Cost: a few million trainable parameters are cheap to train, unlike tens of billions. Safety of skills: the language ability does not erode while we teach vision, a problem called catastrophic forgetting. Reuse: one language model can serve many adapters.
The price is a bottleneck. All visual information must squeeze through the few vectors the adapter writes. If the adapter is too small, fine detail is lost.
Chapter 5
Follow the two stage recipe that turns a frozen vision encoder and a frozen LLM into a chat assistant that can see
A new hire joins a museum. In week one she learns the names of everything in the halls, by looking at labels next to exhibits. After week one she can say “vase, painting, statue” but she rambles and repeats herself when you ask her a real question. In week two a mentor gives her a few hundred model tours, each one a good, detailed answer to a visitor’s question. After week two she can hold a conversation.
Slide 23 shows exactly this two week plan for machines: MiniGPT-4 (Zhu et al., 2023). Stage 1 is alignment. Stage 2 is instruction tuning.
The slide draws four blocks, stacked bottom to top. At the bottom, a ViT (vision transformer) turns the image into patch features. Above it, a Q-Former, a small transformer borrowed from the BLIP-2 paper, squeezes those patch features into a short fixed list, 32 vectors, by using learned queries that attend to the patches (the same trick as Flamingo’s resampler in Chapter 4). Above that, one linear layer (a single table of numbers) maps the 32 vectors into the same size as the language model’s word vectors. At the top sits Vicuna, an open chat language model built from LLaMA.
MiniGPT-4 freezes the ViT, the Q-Former and Vicuna. The one thing it trains is the linear layer. That layer is the whole bridge between seeing and speaking, and it is tiny compared with the models it connects.
Stage 1 uses paired image and text data: millions of pictures, each with a short caption. For each pair, the model reads the picture and must write the caption. Only the linear layer learns. After stage 1 the layer has learned to translate “picture features” into “something Vicuna can read”. This is why the slide calls it alignment: the two worlds now line up.
But the MiniGPT-4 authors found the result was poor to talk to. Captions are short and clipped, so the model produced unnatural text with repetition and broken sentences. It had learned to see, but not to answer.
Stage 2 fixes that with image plus text instructions and example completions, as the slide says. Each example has a picture, a request such as “Describe this image in detail”, and a good, fluent answer. The MiniGPT-4 team wrote and filtered about 3,500 such pairs, far fewer than stage 1 used. The linear layer is trained again, now inside a chat template (the fixed layout of a conversation, with a slot for the request and a slot for the reply), so the model learns the shape of a good reply.
Two lessons hide here. First, quality beats quantity for the second stage: a few thousand excellent examples can change the style of a model trained on millions of ordinary ones. Second, the cost is tiny. You train one layer, on small data, for a short time, and you reuse two huge models.
Choose a stage. Pink shows the part that trains; grey dashed boxes are frozen. The text shows the kind of answer you get to the request “Describe this image.”
Data sizes are the approximate figures reported for MiniGPT-4: about 5 million short caption pairs for stage 1 and about 3,500 curated detailed descriptions for stage 2. The sample answers are illustrative, written to show the style the authors describe, not model outputs.
Step through the three states. Before any training the model has no bridge at all. After stage 1 it can name things but writes broken text. After stage 2 it answers like an assistant.
The lecture’s reading “Improved Baselines with Visual Instruction Tuning” (Liu et al., arXiv 2310.03744) is taught in our LLaVA-1.5 lesson. It shows how far a simple recipe can go. LLaVA-1.5 uses a CLIP ViT-L image encoder at 336 pixel resolution, and replaces the single linear layer with a small MLP projector (two linear layers with a nonlinearity between them), which carries more detail across. It adds academic question data with a short response format prompt (“answer with a single word or phrase”). The final 13 billion parameter model trains on 1.2 million public examples, in about one day on a single machine with 8 A100 GPUs, and reports state-of-the-art results on 11 benchmarks. A better bridge and better data beat sheer size.
Two more readings scale this recipe up. SPHINX-X (Gao et al., arXiv 2402.05935, series lesson) is a family of models on several language backbones, from TinyLlama at 1.1 billion parameters through InternLM2 and LLaMA2 to Mixtral 8×7B. It merges the multi-stage training into one all-in-one stage and removes redundant parts, and it finds that performance tracks both data and parameter scale. InternVL 1.5 (arXiv 2404.16821, series lesson) keeps training a 6 billion parameter vision encoder, InternViT-6B, splits each image into 1 to 40 tiles of 448 × 448 pixels to handle high resolution up to 4K, and adds a bilingual English and Chinese dataset. It reaches state-of-the-art on 8 of 18 benchmarks, as its authors report.
As data flow, for an image of 224 × 224: ViT out [256, 1024] patch features (toy sizes), Q-Former out [32, 768], linear out [32, 4096] matched to the LLM’s width, then 32 visual tokens join the text tokens and Vicuna writes the answer.
Chapter 6
Count what it costs to change a 7 billion parameter model, and meet the four tricks that make it affordable
You own a 7 billion parameter language model, and you want to teach it to see. The model weighs 14 gigabytes on disk. To train all of it with the usual optimizer, you would hold the weights, their gradients and two running averages in memory, several times that size. A single graphics card cannot do it. Even if it could, you would end up with a second 14 gigabyte copy for each new skill.
The cure is the same idea as Chapter 4, taken further: change as few numbers as you can. This chapter holds four tricks. Slide 24 shows the first. The reading list gives the other three.
LLaMA-Adapter (Zhang et al., ICLR 2024) freezes LLaMA and adds a set of learnable adaption prompts: a few extra vectors for each of the top layers. In those layers, the text tokens attend to the adaption prompts as if they were more words. The slide’s picture shows the same model answering in two languages, taking an image, and even a 3D point cloud: all through small add-ons.
The paper’s clever part is zero-init attention. A fresh adaption prompt is random noise, and noise mixed into a finished model can wreck it on the first training step. So the attention to the adaption prompts is multiplied by a gate, a number that starts at exactly zero. At step 0 the adapted model is the pretrained model. Training then opens the gate slowly, only as far as the new signal helps. The paper reports 1.2 million trainable parameters, 52,000 instruction examples, and about an hour of training on 8 A100 GPUs. A series lesson covers it: LLaMA-Adapter.
Flamingo’s gated cross-attention (Chapter 4) uses the same trick, with the gate written as tanh(g) and g starting at 0. Try it.
Output = pretrained output + gate × adapter signal. Drag g from 0. Grey bars are the pretrained output; pink is what the adapter adds; the outline is the result.
Toy vectors with 8 numbers, seeded. The gate is tanh(g), as in Flamingo; LLaMA-Adapter multiplies by a gate that also starts at 0. The drift is the length of the added part relative to the length of the pretrained output.
At g = 0 the drift is exactly 0%. The new part cannot hurt what the model already knows. As g grows, the adapter’s influence rises smoothly, and training can stop wherever the model is best.
LoRA (Hu et al., 2021, in our LoRA lesson) edits the weights themselves, but never directly. A weight table W has d rows and k columns, so d × k numbers. Instead of training W, LoRA freezes it and learns an update written as the product of two thin tables: B (d × r) and A (r × k). The number r is the rank, small, like 8. The update has only r × (d + k) numbers. The layer computes W x + B (A x). B starts at zero, so, again, step 0 is the pretrained model.
For d = k = 4096 and r = 8, one layer’s update holds 8 × 8,192 = 65,536 numbers, against 16,777,216 for W: 0.39%. For GPT-3 175B, the LoRA paper reports about 10,000 times fewer trainable parameters and 3 times less GPU memory than full fine-tuning, with equal or better quality. After training you can fold B A into W, so inference is not slower.
A weight is normally stored in 16 bits (2 bytes). Quantization stores it as a small integer plus one shared scale. The simplest form, absmax: find the biggest absolute weight, divide it by 127 to get a scale, and store each weight as round(weight / scale) in a single byte. For the weights [0.12, −0.50, 0.31, 0.90], the scale is 0.90 / 127 = 0.00709. The stored integers are [17, −71, 44, 127]. To use them, multiply back: 17 × 0.00709 = 0.1205, close to 0.12. The memory drops from 2 bytes to 1, and to half a byte at 4 bits. The reading is Maarten Grootendorst’s Visual Guide to Quantization; our quantization article covers the same ground. Combine 4-bit weights with LoRA and a 7 billion parameter model fits on one consumer card.
A mixture of experts (MoE) layer holds several parallel feed-forward blocks, the experts, and a small router that picks which experts each token visits. If a layer has 8 experts and each token uses 2, only a quarter of that layer’s weights work per token. The model can be big in storage but cheap per token. The reading is Grootendorst’s Visual Guide to Mixture of Experts; see also our MoE lesson. SPHINX-X, in Chapter 5, uses Mixtral 8×7B as one of its backbones. We return to MoE in Chapter 9.
Put the first three side by side.
Choose how to adapt a 7 billion parameter model. Bars are on a log scale: each gridline is 10 times more. Drag the LoRA rank, and switch on 4-bit weights.
The model is a LLaMA-7B shape (32 layers, width 4,096, 7 billion weights). LoRA adapts the query and value tables in every layer, so it trains 32 × 4 × 4,096 × r numbers. LLaMA-Adapter’s 1.2 million is the paper’s figure. Weight storage is parameters × bytes per weight and ignores optimizer state.
At rank 8 LoRA trains about 4.2 million numbers, which is 0.06% of the model. Full fine-tuning trains all 7,000 million. Storage is a separate bill: 4-bit weights shrink 14 GB to 3.5 GB, whichever method you chose.
Chapter 7
See what goes into pre-training and instruction tuning, and how to split a compute budget between model size and data
A cook with a huge kitchen and almost no ingredients makes a poor dinner. A cook with a tiny stove and a mountain of ingredients wastes most of them. A good meal needs both in balance. Large models are the same: parameters are the kitchen, data is the food, and compute is the time on the stove.
Slides 25 and 26 list the data. The readings ask how to balance it. We take the data first.
Slide 25 shows a timeline of image and text collections. YFCC-100M came in 2018. LAION-400M followed in 2021, LAION-5B in 2022, and DataComp in 2023. DataComp is the largest, with 12.8 billion image and text pairs. The slide notes a shift: recent work cares less about the raw count and more about filtering, keeping only pairs whose caption truly fits the picture. Examples are DFN (2 billion pairs), COYO (600 million) and Obelics (141 million documents).
The slide also gives one sample pre-training mix, the share of each source that the model sees. Image and caption pairs are 27% from DFN, 11.25% from COYO and 6.75% from HQITP (400 million pairs). Interleaved documents, where images and text alternate as on a web page, are 45%, from Obelics. Text only data from DCLM (6.6 trillion tokens) makes up the last 10%. The shares add up to 100%. Text stays in the mix so that the model does not forget how to write while it learns to see.
Each bar shows how many examples of a batch come from a source. The tick marks the slide’s sampling share. Press draw, and draw more.
The five sampling shares are slide 25’s. The draws are real random draws in your browser. Small batches stray from the shares; large ones sit on them.
Draw 8 examples and a source may get none, or three. Draw 2,048 and every source lands within a few percent of its share. Each training step sees one batch, so over millions of steps the mix is exact on average, though any single step is noisy.
Slide 26 contrasts this with the data for instruction tuning: requests paired with good answers. It is “more scattered, smaller in nature”. General domain sets include Vision-Flan (187 thousand examples), LLaVA-Instruct (150 thousand), InstructBLIP (about 1.6 million) and M3IT (2.4 million). Clinical sets include CLIMB-QA (4.51 million), BioMed-VITAL (210 thousand) and LLaVA-Med (60 thousand). Compare 12.8 billion for the biggest pre-training set with the millions here: instruction data is expensive to write, so it is small.
The text only world learned the same lesson. The FLAN work (Chung et al., “Scaling Instruction-Finetuned Language Models”, our lesson) fine-tuned models on 1,836 tasks written as instructions, with chain of thought examples among them, and found that more tasks and larger models both help. Super-NaturalInstructions (Wang et al., arXiv 2204.07705, series lesson) is a benchmark of 1,616 tasks across 76 task types; its model Tk-Instruct, trained on it, beats InstructGPT by more than 9% on the benchmark while being about ten times smaller, as the paper reports. Variety of tasks matters more than the number of examples for each.
Compute is counted in FLOPs, floating point operations. Training a transformer with N parameters on D tokens costs about C = 6 × N × D FLOPs. For a fixed C, you can pick a big N and small D, or a small N and big D. Which is better?
Chinchilla (Hoffmann et al., arXiv 2203.15556, series lesson) trained more than 400 language models, from about 70 million to over 16 billion parameters, on 5 to 500 billion tokens, and answered: grow both in equal proportion. Each doubling of parameters calls for a doubling of tokens. The rule of thumb that falls out is about 20 tokens per parameter. The paper’s test: Chinchilla has 70 billion parameters and trained on 1.4 trillion tokens, with the same compute as Gopher, a 280 billion parameter model trained on 300 billion tokens. The smaller, better fed model won.
Worked example. Use C = 5.88 × 1023 FLOPs. With D = 20 N, C = 6 × N × 20 N = 120 N2. So N = √(C / 120) = √(4.9 × 1021) = 70 billion, and D = 1.4 trillion. A fixed 300 billion token recipe would put N at C / (6 × 300 billion) = 327 billion parameters instead.
Drag the budget. Warm bars show the compute optimal split (20 tokens per parameter). Grey bars show a recipe stuck at 300 billion tokens.
Uses C = 6 N D and D = 20 N. The 20 is the rule of thumb that follows from the Chinchilla results; the paper fits curves rather than one exact ratio. At 5.9e23 FLOPs the optimal split is the 70 billion and 1.4 trillion of Chinchilla itself.
Chapters 4 to 6 built models by joining a frozen vision encoder to a frozen LLM, called late fusion. “Scaling Laws for Native Multimodal Models” (arXiv 2504.07951, series lesson) asks whether that is the best plan. The authors trained 457 models of different designs and data mixes. They found no inherent advantage for late fusion. Early fusion models, which feed image patches straight into the transformer with no separate image encoder, did better at small sizes and cost less to train and serve. Adding mixture of experts layers helped further. Chapter 9 picks this thread up again.
Chapter 8
Let a language model ask for a picture: retrieve one that exists, or hand a plan to an image maker
You ask a friend for “a bowl of soup that is a portal to another dimension”. No photograph of this exists. A search engine would return soup. To satisfy you, someone must make the picture.
Part 3 of the lecture (slide 27) adds a new box to the diagram from Chapter 0. The model must not only write about the scene. It must also retrieve the next episode, or generate a new image. The lecture has three ways to get there.
Slide 28 shows two images from DALL-E 2 (Ramesh et al. 2022, a method also called unCLIP): “an astronaut riding a horse in the style of Andy Warhol” and the portal soup. Slide 29 shows how the system works in three steps.
The slide calls this “directly training diffusion models with conditional information”. The diffusion model is trained from scratch to follow the CLIP embedding. This works well, but there is no language model that can talk, reason or hold a dialog.
Slides 30 and 31 ask: can a language model that already chats also choose pictures? The method (Koh et al., ICML 2023, “Grounding Language Models to Images for Multimodal Inputs and Outputs”; the slide names it LIMBeR, and the paper itself is known as FROMAGe) keeps the language model frozen and uses CLIP as the picture side. It is trainable in 1 day on 1 GPU, says the slide.
It does two things with small trained layers. For reading, it maps a CLIP image feature into the language model’s input space as a few vectors, exactly as in Chapter 4. For writing, it adds a special [RET] token to the vocabulary. When the model wants a picture, it emits [RET]. The model’s hidden vector at that spot is mapped by a linear layer into CLIP’s space, and the system returns the closest image from a library by cosine similarity. Training uses captions and a contrastive loss between the [RET] vector and the true picture. The slide shows a chat where each turn mixes text and retrieved images.
This is cheap and gives real photos, but it can only return what the library holds.
Slides 32 to 34 show the third way, GILL (Koh et al., NeurIPS 2023, “Generating Images with Multimodal Language Models”). It keeps the frozen language model and the input adapter (“part 2” on the slide). It adds a new output path, “part 3”: a few learned [IMG] tokens (the paper uses 8) that the model emits when it wants a picture. The model’s hidden vectors at those tokens go into a small network, the GILLMapper. The mapper turns them into the kind of conditioning vectors that a text to image diffusion model, Stable Diffusion, expects. A decision model chooses between the two paths: retrieve an existing image, or generate a new one. The slide’s final example is a conversation about cookies that ends with the model’s advice and a picture it made.
The two main ideas: only small mappers are trained, and the language model decides when and what to draw. Step through both pipelines.
Choose a pipeline and press Next step. The lit box is where your request is now.
Shapes follow the methods as described in the lecture: one CLIP vector for DALL-E 2’s prior; 8 learned [IMG] tokens for GILL, mapped to the 77 by 768 conditioning that Stable Diffusion 1.5 reads. Box timings are not measured.
When should the system fetch a real photo, and when should it draw? If the request is common (“a golden retriever on a beach”), the library likely has a match, and a real photo is faster and more faithful. If the request is odd (“soup that is a portal”), nothing in the library fits, so it must generate. GILL’s decision model learns this choice. The device uses a plain threshold instead, so you can see the logic.
Blue dots are library images on a ring; the teal dot is your request. Drag how unusual the request is. The line goes to the best match.
Toy: five library images at fixed angles, one request that rotates away from the closest. The score is the cosine between vectors. The 0.75 line is arbitrary; GILL trains a small decision network instead.
Four readings generalise this from images to every output. MetaQueries (arXiv 2504.06256, series lesson) puts a set of learnable queries between a multimodal LLM and a diffusion decoder. The LLM can stay frozen, so its understanding stays intact while it gains the power to guide image generation. NExT-GPT (arXiv 2309.05519, series lesson) joins an LLM to pretrained encoders and several diffusion decoders so it can take and produce text, images, video and audio in any mix, and tunes only about 1% of its parameters, the projection layers. ModaVerse (arXiv 2401.06395, series lesson) uses language itself as the bridge: the LLM writes instructions that feed straight into off the shelf generators, an I/O alignment that avoids training latent feature projections, and the authors report comparable results with less data and training time. Spider (arXiv 2411.09439, series lesson) goes further, from any to many: one answer that holds text plus several other modalities, controlled by a Decoders-Controller and trained on a new text-formatted many-modal dataset.
As data flow, for GILL: in, image and text tokens. Out, text tokens, and when [IMG] appears, 8 hidden vectors that become the diffusion model’s condition.
Chapter 9
Look at where the field is heading: native models, experts, cheaper sequences, balanced training, and fairer alignment
Everything so far followed one recipe: take a good language model, take a good image encoder, and join them with a small bridge. It works, and it is cheap. But a house built from two kits has seams. The slide 35 summary names the next steps: natively multimodal models, multimodal mixtures of experts, and real world modalities beyond pictures and text. The slide gives only titles; this chapter fills them with the readings.
A native multimodal model trains from the start on all modalities in one network. Image patches enter the same transformer as the words, with no separate pretrained image encoder. The scaling law paper from Chapter 7 (457 models trained) found no inherent edge for the bridge design. Early fusion did better at small sizes, and it was cheaper to train and to serve. The reason is simple: one network to run, no second tower to load, and the model sees the pixels itself instead of a summary made by someone else’s encoder.
Chapter 6 introduced the mixture of experts: a router sends each token to a few expert blocks. In a multimodal MoE, the experts can specialise. Some drift toward text tokens, others toward image tokens, and the model has room for both skills without making every token pay for both. The scaling law paper found that adding MoE layers improved the native models, because the weights can become modality specific. Try the router below.
Twelve tokens, six text (orange) and six image (blue), each go to their top choices among 8 experts. Pick how many experts each token uses, and let the router learn to specialise.
Toy router: scores are seeded random numbers. With specialisation on, experts 1 to 4 get a bonus for text tokens and experts 5 to 8 for image tokens. A real router learns its scores during training, and also needs a load balancing term so no expert is overloaded.
With 2 of 8 experts, each token uses a quarter of the layer’s expert weights. The model can store four times more knowledge than it spends per token. Switch specialisation on and the lines sort themselves: text goes left, image goes right.
Images are expensive to read. At 336 pixels with 14 pixel patches, one picture is 24 × 24 = 576 tokens. Attention compares every token with every other, so the work grows with the square of the length (Chapter 1’s 3 × 4 grid, grown to n × n). Five pictures make 2,880 tokens, and 4.1 million pairs per layer.
A state space model such as Mamba reads the sequence once, left to right, with a fixed size memory, so the work grows in a straight line. Cobra (arXiv 2403.14520, series lesson) plugs Mamba into the multimodal setting as a “linear computational complexity” model. Its authors report that it runs faster than comparable efficient transformer models and reaches results comparable to LLaVA with roughly 43% of LLaVA’s parameters. Gated Linear Attention (Yang et al., arXiv 2312.06635, series lesson) is a different route to the same goal. It adds data dependent gates to linear attention (the memory can forget, by amounts that depend on the input), and a hardware aware training algorithm. In the paper, GLA competes with LLaMA style transformers and Mamba at moderate scale, generalises from 2,000 token training to sequences over 20,000, and trains faster than a Mamba model of similar size.
Each image is 576 tokens. Add images and compare how fast the two curves climb.
576 tokens per image is the LLaVA 336 pixel setting (24 × 24 patches). The prompt also holds 100 text tokens. Attention counts n(n + 1)/2 pairs per layer; the linear model counts n steps. Real systems also differ in constant factors, so this shows the growth, not the wall clock time.
Two readings are about what goes wrong when we train. Modality imbalance (arXiv 2404.08347, series lesson): when a model learns from several inputs, it leans on the easiest one and neglects the others, which caps its overall quality. The paper’s fix, adaptive mask subnetworks, estimates how important each modality is using mutual information rates, then picks “foreground” sets of weights to update for each, masking out the rest, so no single modality dominates the learning.
The other, “Unintended Impacts of LLM Alignment on Global Representation” (arXiv 2402.15018, series lesson), studies alignment methods (the training that makes a chat model follow human preferences) and asks who benefits. It found that today’s alignment improves ability in several languages, but creates gaps between English dialects and between the opinions of different countries. For a multimodal model, which will be asked about people of many cultures (remember slide 5’s counterfactual question), this is not a side issue.
The last item on slide 35 points past video and audio to sensors: touch, motion, depth, point clouds (slide 24 already shows one), and the signals of robots and bodies. The pattern stays the same: an encoder turns the signal into tokens, an adapter fits them to the model, and the transformer reads them all.
Chapter 10
Keep the numbers, the terms, a checklist, the core code, and the readings, then go back and play the instrument
The lecture has one spine. Meaning lives across modalities, so a model needs a shared representation (Chapters 0 to 3). The cheapest way to get a large multimodal model is to take a strong language model, keep it frozen, and train a small bridge into it (Chapters 4 to 6). Data and compute must be balanced (Chapter 7). Pictures come out through retrieval or an image maker steered by the model (Chapter 8). The frontier removes the seams (Chapter 9).
| Term | What it is | Number |
|---|---|---|
| Cross-attention | Queries from one modality, keys and values from another | 3 × 4 grid on slide 11 |
| ViLT patch | Linear projection of a flattened image square, no detector | 32 × 32 pixels |
| ALBEF | Align with a contrastive loss before the fusion layers | 12 + 6 + 6 layers |
| Contrastive guess loss | Loss of a model that cannot tell N pairs apart | ln N (1.386 for N = 4) |
| Frozen | Adapter writes a prefix; the LLM stays frozen | 2 prefix vectors |
| Flamingo | Resampler plus gated cross-attention in a frozen LLM | 64 visual tokens, 80B total |
| MiniGPT-4 | Train one linear layer in two stages | about 5M, then about 3,500 examples |
| LLaVA-1.5 | MLP projector, better data, 336 pixel CLIP | 1.2M examples, about 1 day, 8 A100 |
| LLaMA-Adapter | Adaption prompts with a gate that starts at 0 | 1.2M trained numbers |
| LoRA | Frozen W plus a low rank update B A | r × (d + k) numbers per table |
| Quantization | Store weights in fewer bits with a shared scale | 7B: 14 GB, 7 GB, 3.5 GB |
| Pre-training mix (slide 25) | Sampling share per source | 27, 11.25, 6.75, 45, 10 percent |
| Chinchilla | Grow parameters and tokens together | 70B and 1.4T; about 20 tokens per parameter |
| GILL | [IMG] tokens, a mapper, and a diffusion model | 8 [IMG] tokens |
| Image tokens | One 336 pixel picture at 14 pixel patches | 576 tokens |
pythonimport torch, torch.nn as nn, torch.nn.functional as F class PrefixAdapter(nn.Module): # the only trained part (Chapters 4 and 5) def __init__(self, d_vis, d_llm, n=2): super().__init__(); self.n, self.d = n, d_llm self.proj = nn.Linear(d_vis, n * d_llm) def forward(self, v): # [B, d_vis] -> [B, n, d_llm] return self.proj(v).view(-1, self.n, self.d) def train_step(vision, adapter, llm, image, ids, opt): with torch.no_grad(): v = vision(image) # frozen encoder [B, d_vis] prefix = adapter(v) # trained [B, n, d_llm] x = torch.cat([prefix, llm.embed(ids)], 1) # [B, n + T, d_llm] logits = llm.forward_embeds(x) # frozen LLM [B, n + T, V] n, T = adapter.n, ids.size(1) # the last prefix vector predicts word 0, and so on loss = F.cross_entropy(logits[:, n-1:n-1+T].reshape(-1, logits.size(-1)), ids.reshape(-1)) opt.zero_grad(); loss.backward(); opt.step() # opt holds adapter params only return loss.item() class LoRALinear(nn.Module): # Chapter 6 def __init__(self, base, r=8, alpha=16): super().__init__(); self.base, self.scale = base, alpha / r for p in base.parameters(): p.requires_grad = False self.A = nn.Parameter(torch.randn(r, base.in_features) * 0.01) self.B = nn.Parameter(torch.zeros(base.out_features, r)) # zero: step 0 = pretrained def forward(self, x): return self.base(x) + self.scale * (x @ self.A.T @ self.B.T) def itc_loss(img_emb, txt_emb, tau=0.2): # Chapter 3: align before fusing a, b = F.normalize(img_emb, dim=-1), F.normalize(txt_emb, dim=-1) logits = a @ b.T / tau; y = torch.arange(len(a)) return (F.cross_entropy(logits, y) + F.cross_entropy(logits.T, y)) / 2
This lecture is the third in a run on multimodal AI. Read it beside our lessons on vision language models, CLIP, the Vision Transformer, the transformer, diffusion, LoRA and parameter efficient fine-tuning and mixtures of experts.
The readings, in the order the lecture lists them. Chinchilla: lesson. Super-NaturalInstructions: lesson. LoRA: lesson. A Visual Guide to Mixture of Experts: lesson. A Visual Guide to Quantization: lesson. LLaVA-1.5: lesson. Gated Linear Attention: lesson. Unintended Impacts of LLM Alignment: lesson. Scaling Instruction-Finetuned Language Models: lesson. Multimodal Transformer: lesson. Masked Autoencoders: lesson. Scaling Laws for Native Multimodal Models: lesson. MetaQueries: lesson. LLaMA-Adapter: lesson. Cobra: lesson. ModaVerse: lesson. Spider: lesson. SPHINX-X: lesson. InternVL 1.5: lesson. NExT-GPT: lesson. Adaptive mask rebalancing: lesson. (The lecture’s link for the FLAN paper points to the LoRA arXiv number; our lesson covers the Chung et al. FLAN paper.)
Now press Present or Teach and explain one picture question from memory: how the picture becomes prefix vectors, which weights change, why a gate that starts at zero is safe, and how the model could ask for a new image. Then go back to the instrument at the top, switch the adapter on and off for each question, and explain which part did the seeing and which part did the answering.
Credit: this lesson re-teaches Lecture 5.1 of MIT Multimodal AI (Spring 2026) by Paul Liang, MIT Media Lab and MIT EECS, Multisensory Intelligence group. The slides and the reading list belong to the course. Find them on the course site. The examples, devices and words here are our own.