Type a sentence and a model hands back a picture. One recipe writes the picture as a row of tokens, one at a time. The other starts from noise and clears it.
Learn how a model that reads many senses learns to write words, fetch images and draw them, and the one idea underneath: a model of what real data looks like.
Press Play, then drag the progress bar back and forth in each mode. See how the token recipe fills the picture cell by cell while the noise recipe sharpens the whole picture at once. Then we build, piece by piece, every part of the lecture: representations, adapters, retrieval, DALL-E, diffusion, mixtures of Gaussians and variational autoencoders.
You need a rough idea of what a probability is (a number from 0 to 1 that says how likely something is) and what a neural network does (it turns numbers into numbers). We build the rest from zero.
Based on Lecture 6.1 of MIT MMAI, Multimodal AI (Spring 2026), taught by Paul Liang of the MIT Multisensory Intelligence group; the slides (PDF) are on the course schedule. Every idea is re-taught here in our own words and devices.
An armchair in the shape of an avocado.
A toy: a 16 by 16 picture made of 256 cells and a codebook of 5 colours, so each cell is one token numbered 0 to 4. Real DALL-E tokens come from a codebook of 8,192 (slide 17) and a real diffusion run uses its own number of steps; the 50 steps here are illustrative. The picture is fixed, so a real model would draw a different one each run. The cell order and the step counts are exact for this toy.
Chapter 0
See the three jobs a multimodal model can do with the same input, and why making a new picture is the hardest of the three
Picture a friend at dinner. She leans back, rolls her eyes, and says in a bright voice: “It’s just a privilege to watch your mind at work.” Read on paper, that is a compliment. Heard and seen, it is a joke at your expense.
You knew this at once because you used three senses together: the words, the tone of her voice, and her face. A modality is one such channel of information: text, speech, a video of a face, a photo. A multimodal model is one that takes in more than one modality at once. This lecture of the MIT course Multimodal AI asks what happens when we want such a model to make something, and not only to understand.
The lecture keeps one running example, the sarcastic dinner remark, and gives the model three different jobs in turn. The first job is to understand. The model reads the clip and builds a representation: a list of numbers that holds what matters about the clip, the way a hotel key card holds just enough to open one door. A small classifier on top reads that list and answers “sarcasm”. That is Part 1 of the lecture.
The second job is to describe. The model writes words: “This person is being sarcastic. They seem to be close friends.” Now the output is not one label but a whole new text, written one word after another. That is Part 2, where a large language model learns to read pictures.
The third job is to draw. The model must produce a new image, a thing that did not exist a second ago. The lecture splits this job in two: find an existing picture that fits, or generate a new one. That is Part 3. And once we want to generate, we need a theory of what a good picture even is. So the second half of the lecture builds that theory, from mixtures of Gaussians to variational autoencoders.
Understanding squeezes a lot of input into a small answer. The clip holds millions of numbers; the answer holds one word. Generating runs the other way. The prompt “An armchair in the shape of an avocado.” holds eight words, and the answer must hold hundreds of thousands of pixel values, every one of them chosen.
Nothing in the prompt says which pixel is green. So the model must invent the missing detail, and it must invent it in a way that is consistent: the legs match the seat, the light falls from one side. Many different pictures fit the same eight words. A good generator can make any of them, and it should not always make the same one.
That last point is the key. A generator is not a function from prompt to picture. It is a machine that draws from a spread of possible pictures, which is why the second half of the lecture talks about probability so much. Chapters 8 to 10 build that idea.
Try the three jobs yourself below. The input stays the same. Only the output changes shape, and watch how much larger the output becomes.
The left box is the dinner clip: words, voice and face. Choose a job and watch what the model must hand back. Notice how the size of the answer changes.
The label and the two sentences are the lecture’s own (slides 5 and 6). The picture is the toy 16 by 16 grid from the top of this page, so its 256 cells are illustrative; a real image holds far more numbers.
The lecture lists three topics: text-to-image generation, an introduction to generative models, and then mixtures of Gaussians and variational autoencoders (VAEs). Its outline slides show the same story as a ladder, each rung adding one output. Chapters 1 to 7 climb it: representations, text out, an image fetched, an image drawn, and then the two big recipes for drawing. Chapters 8 to 10 then step back and ask what a generative model is. Chapter 11 is the field guide.
Two readings go with the session. One argues that we should build big generative systems by composing small models, and sits in Chapter 9 beside the mixture of Gaussians. The other describes Meta’s Movie Gen video models, and sits in Chapter 7 beside diffusion.
The slides also carry the week’s course business, and here it is in one line: homework 3 is due on 3/17, the midterm is on Thursday 3/19 with a review on Tuesday 3/17, and project mentors were assigned that week.
Chapter 1
Let each word look at the voice and the face, and copy in the parts that change its meaning
Back at the dinner table. The word “privilege” arrives. On its own it means something warm. But your friend says it with stress on the first syllable, and her eyes roll. By the time the word reaches you, its meaning has already shifted. You did not look up a new definition. You mixed in what you saw and heard.
The lecture’s first part asks how a model can do the same mixing. The plan is simple to say. Turn every channel into lists of numbers. Then let each word look at the other channels and take in whatever helps. The result is a multimodal representation: one list of numbers per word that already carries the voice and the face inside it.
A neural network only reads numbers. So each word becomes a vector, a list of d numbers called its embedding. Three words make a table with 3 rows and d columns, which the slide calls X1. The voice and the face become a table too: the slide names vocal emphasis and eye-roll among its signals, and stacks them into X2, a table of 4 rows and d columns. A number in a row says how much of a feature is present at that moment.
Now the question is how a word row should look at the signal rows. The tool is attention, a way of letting one list of items read from another list. It works like a library desk. Each word writes a query, a short note saying what it needs. Each signal row carries a key, a label saying what it offers, and a value, the content to hand over. A word compares its query with every key. The better the match, the more of that signal’s value it takes.
The slide packs this into one line. Read it from the inside out. First the words make queries by multiplying X1 by a learned matrix Wq. The signals make keys by multiplying X2 by Wk. Multiplying queries by keys gives a table of match scores, one per word and signal pair. The slide draws it as a 3 by 4 grid and calls it a weighted outer product.
Why divide by √d? A score adds up d small products. The more terms you add, the wider the spread of the sum, and it grows about as fast as √d. A wide spread makes the softmax pick one winner and ignore the rest. Dividing by √d undoes that growth.
Take the word “privilege” and two signals, vocal emphasis and eye-roll. Say the scores are 0.85 for the vocal signal and 0 for the eye-roll. Step 1: raise e to each score. e0.85 is 2.34 and e0 is 1. Step 2: add them, 3.34. Step 3: divide each by the sum. The weights are 2.34 / 3.34 = 0.70 and 1 / 3.34 = 0.30.
Those are the numbers on the slide’s small grid for “privilege”: 0.7 and 0.3. The word’s new representation is then 0.70 of the vocal value plus 0.30 of the eye-roll value. The slide’s “mind” row puts the whole weight, 1.0, on a single signal. Try other scores in the device and watch the weights follow.
Each row is a word. Each column is a signal. The sliders set how much better the vocal signal matches the word than the eye-roll does. Slide to the right and the word leans on the voice; slide left and it leans on the eye-roll.
The 0.70 and 0.30 for “privilege” match the slide’s grid (a gap of 0.85 gives them exactly). The gap for “mind” is set so its weight is almost 1.0, as on the slide. Real scores come from learned matrices; these two are set by hand.
As data flow: X1 is 3 × d and X2 is 4 × d. The queries are 3 × d, the keys 4 × d, so the score table is 3 × 4. After the softmax, each of the 3 rows adds up to 1. Multiply by the values (4 × d) and the output h is 3 × d again. The model has the same number of words as before, each now richer.
The slide cites two papers for this idea. Liang and colleagues (EMNLP 2018) fused language, voice and face step by step with a recurrent network. Tsai and colleagues (ACL 2019) built the Multimodal Transformer, where one modality attends to another even when the two are unaligned: the voice may be sampled far faster than the words, and attention does not need them to line up in time.
Suppose the model calls every loud speaker sarcastic. That is the symptom. Measure it: print the 3 by 4 weight grid. If every row puts about 0.9 on the vocal column no matter which word it is, the model has learned a shortcut and ignores the face. The usual fix is data, not code: add clips where voice and face disagree, so the shortcut stops paying. (This failure is illustrative; the lecture returns to such failures in its fusion weeks.)
Chapter 2
Turn a picture into a few word-like vectors, then train one small layer so a frozen language model can read them
You hold a photo of a logo up to a very well-read friend who has been blind from birth. She has read every book. She writes well. But you cannot show her the photo. What can you do? You could describe the photo in words, one detail at a time. Or you could build a translator that turns the picture into something she can read.
A large language model (LLM) is that friend. It has read a huge amount of text, and it reads nothing but token embeddings: lists of numbers, one per word piece. Part 2 of the lecture asks how to give such a model eyes without rebuilding it. The answer is the translator.
The slide shows MiniGPT-4 (Zhu and colleagues, 2023) as four boxes in a row. First, a vision encoder, a ViT, cuts the image into patches and turns each into a vector. Second, the Q-Former compresses those many patch vectors into a small, fixed number of vectors, by letting a few learned queries look at the patches (it is cross-modal attention again, as in Chapter 1). Third, one linear layer multiplies each of those vectors by a matrix so they match the size of the language model’s own embeddings. Fourth, the language model, Vicuna, reads them like words.
Look at the snowflake icons on the slide. They mark a part that is frozen: its numbers are held fixed during training, like a printed page. Vicuna, the ViT and the Q-Former are frozen. Only the linear layer is not. The paper’s abstract says the same: it aligns a frozen visual encoder with a frozen LLM using one projection layer.
Why freeze? Training a whole LLM costs far more compute and can make it forget the language skills it already has. Training only the glue is cheap and leaves the knowledge intact. Here is the arithmetic, with sizes labelled illustrative. Say the Q-Former outputs vectors of 768 numbers and the language model uses 4,096. The linear layer then holds 768 × 4,096 weights plus 4,096 biases, which is 3,149,824 numbers. A 7-billion-number language model is about 2,200 times bigger.
The four boxes run left to right. Frozen boxes are grey and dashed. The trained box is pink. Switch the stage to see which data feeds it, and drag the size of the language model to see how small the trained share is.
Sizes are illustrative: Q-Former vectors of 768 numbers, language-model embeddings of 4,096, and a language model of the size you set. Which boxes are frozen is from the slide and the paper’s abstract. The two sample outputs are made-up examples of the two behaviours the paper describes.
The slide gives each stage one line. Stage 1 is alignment using paired image-text data. Each example is a picture with a short caption. The training signal is the usual one for a language model: given the picture vectors, predict the next word of the caption, and nudge the linear layer to make that prediction better. After enough pairs, the vectors the layer produces sit in the language model’s space where a real description of that picture would sit. The translator works.
But there is a flaw, and the paper reports it plainly. A model trained only on short caption pairs produced unnatural language: repetition and fragmentation. Short captions are all it saw, so short and choppy is all it knew how to write.
Stage 2 is instruction tuning: training on image plus text instructions, each with an example of a good completion. In the slide’s picture, the prompt reads “### Human: <Img>…</Img> What do you think of this logo design? ### Assistant:” and the model answers with a long, fluent paragraph about the flamingo logo. The paper builds a detailed image-description dataset for this step, and it fixes the unnatural output.
Look at the data flow when you ask a question. The image becomes a few × 4,096 block of vectors. The text “What do you think of this logo design?” becomes one embedding per word piece. The two are laid end to end in one sequence, and the language model reads that sequence as it would read any text. It never knows that some of its “words” came from pixels.
Notice what the recipe does not do. The model can now write text about an image, but it cannot draw one. Its output side is still a vocabulary of word pieces. The next chapters change that.
Suppose the model describes a photo and keeps repeating one phrase. That is the symptom. Measure it: count how often the most common three-word phrase appears in a long answer. A healthy answer repeats it once or not at all. A stage-1-only model repeats it many times. The fix is the one the paper uses: add the second stage with long, carefully written examples, which teaches the model what a full answer looks like.
Chapter 3
Let the model say “a picture goes here” with one special token, then find the closest match in a photo library
You are chatting with an assistant about cooking. You ask: “What does a farmer’s market stall look like?” A good answer is a sentence, and then a photo. The assistant does not need to draw the stall. A million photos of stalls already exist. It only needs to point at the best one.
That is the first half of Part 3 in the lecture: text-to-image retrieval, which means finding an existing image that matches a request. The second half, generation, comes in Chapter 4. Retrieval is easier because the hard part, making a good picture, is already done. The model only has to pick.
How can a computer say that a photo matches a sentence? The lecture’s answer is CLIP, a pair of encoders trained together so that a photo and its caption land close to each other. (Our CLIP lesson builds it from zero.) Every photo becomes a point in a shared space, a vector of numbers. Close points mean similar content.
“Close” needs a measure. The usual one is the cosine similarity: the cosine of the angle between two vectors. Two arrows pointing the same way score 1. Arrows at a right angle score 0. Opposite arrows score −1. It ignores how long the arrows are and looks only at direction, which is what we want, because direction carries the meaning.
The slide calls the method “LIMBeR + CLIP” and says it is trainable in one day on one GPU. It is the paper “Grounding Language Models to Images for Multimodal Inputs and Outputs” (Koh and colleagues, ICML 2023). The language model stays frozen. Only input and output linear layers are trained, as the paper’s abstract says. That is why it is so cheap.
The trick is a special token, written [RET]. When the model wants a picture, it writes [RET] as its next piece of text. At that position, the language model has a hidden vector of 4,096 numbers (illustrative size) that sums up what it wants. A trained output layer maps that vector into CLIP space, say 512 numbers. Now we have a point in the same space as every photo in the library.
Retrieval is then plain arithmetic. Compute the cosine between the [RET] point and every library photo’s CLIP vector. Return the photo with the highest score. No new image is made. The slide’s picture shows the same loop on the input side. A photo goes through a frozen CLIP image encoder. An input linear layer maps it to a few word-like vectors. The language model then reads them next to the text, just as in Chapter 2.
Each dot is a library photo placed in a shared space (shown in two dimensions). The arrow is the vector the model wrote for [RET]. Choose a request, then add noise to the arrow and see when the wrong photo wins.
Illustrative: the library and its six angles are made up, and the space has two dimensions where CLIP’s has hundreds. The method is the real one: the cosine of the angle between the [RET] vector and each photo, with the highest score returned.
Training uses image-caption pairs and two losses at once. The first is the ordinary cross-entropy loss for text: predict the caption word by word. The second is a contrastive loss on the [RET] vector: after the caption, the [RET] vector should sit close to its own photo and far from the other photos in the batch. The slide’s diagram labels both losses.
A worked case with τ = 0.1 and a batch of three photos. If the similarities are 0.8 for the right photo and 0.2 and 0.1 for the others, the exponentials are e8 = 2,981, e2 = 7.4 and e1 = 2.7. The right photo gets 2,981 / 2,991 = 0.997 of the share, so the loss is −ln(0.997) = 0.003. If instead the similarities are 0.3, 0.4 and 0.2 (the right photo is the first), the shares are 20.1, 54.6 and 7.4. The right photo gets 0.245 and the loss is 1.41. The loss tells the layer to push that 0.3 up.
The lecture’s next slide shows what this buys: a chat in which the model answers in text and shows retrieved photos, and can even take photos as part of the conversation. It uses what the language model already knows, such as in-context learning, because the language model itself was never changed.
Suppose you ask for “a red couch” and the model returns a red car. The symptom is a wrong but related photo. Measure it: list the top five cosines. If the car and the couch score within 0.02 of each other, the [RET] vector was too vague to separate them. Training with more varied captions, so the output layer learns finer differences, is the fix. And remember the hard limit: retrieval can only return what is already in the library. A request for something that was never photographed finds only its nearest neighbour. Chapter 4 fixes that.
Chapter 4
Borrow an image generator that already draws well, and train a small adapter that tells it what the conversation wants
Back in the cooking chat. This time you ask for something nobody has photographed: “Show me an armchair shaped like an avocado, with a cushion for the pit.” Retrieval fails. The library has no such chair, and the closest photo is just a green sofa.
Training a new model to draw from scratch would cost enormous compute. But good image generators already exist. They take a sentence and make a picture. The trouble is that they understand only short sentences, in the shape their own text encoder expects. Our chat model understands the whole conversation. How do we connect them?
The lecture’s paper is “Generating Images with Multimodal Language Models” (Koh and colleagues, NeurIPS 2023), known as GILL. It keeps both big models frozen. The language model stays frozen. The image generator, a Stable Diffusion decoder on the slide, stays frozen. What is new is a small network between them that the abstract calls a mapping network and the slide calls the adapter layer.
An adapter is a small trained piece that translates between two models that were never built to talk. Think of two experts who speak different languages and a human interpreter in between. Neither expert changes. Only the interpreter learns.
Here is the data flow, step by step. First, when the language model wants to draw, it writes a few special tokens, [IMG1] up to [IMG(r)]. The slide labels the stack of these “Model Output Embeddings”. Second, take the hidden vectors at those r positions: a block of r × 4096 numbers (illustrative size). Third, the mapper turns that block into something shaped like the image generator’s own text embeddings, for Stable Diffusion a block like 77 × 768 (illustrative). Fourth, the frozen decoder draws the picture from it, as if a normal prompt had been typed.
How is the mapper trained? With the thing it must imitate. Take a caption and run it through the image generator’s own text encoder. That gives the target block. Train the mapper so the language model’s [IMG] vectors, passed through it, land near that target. After training, the language model can ask for any picture in the generator’s own language, even though it never learned to speak it.
One benefit follows from the design. The language model reads the whole chat, so the picture can depend on earlier turns: “make it purple” works because the model remembers what “it” was. The paper’s abstract notes that this approach does best on tasks with longer and more complex language, which a short-prompt generator handles badly. The slide shows this as a story told in pictures and chat, with interleaved images and text going in and coming out.
Now the model has two tools: retrieve (Chapter 3) and generate. Retrieval is cheap and exact, and the photo is real. Generation is slower and can invent a flawed picture, but it can make things that do not exist. Which one should the model use?
The paper adds a learned decision module, which reads the language model’s hidden state and decides at inference time. The slide shows a decision box with the retrieved candidates and a similarity score beside it. The idea is simple. If a library photo already matches well, return it. If not, draw.
Each bar is the best cosine similarity any library photo reaches for a request. The line is your threshold. Bars above it are answered by retrieving. Bars below it are answered by drawing. Drag the line.
Toy numbers: the three similarities (0.82, 0.64 and 0.31) are made up to show the idea. In the paper the decision is made by a learned module reading the language model’s hidden state, not by a fixed threshold on one number.
Drag the line to 0.70. The farmer’s-market photo (0.82) is still retrieved, but the baker’s portrait (0.64) now gets a fresh drawing. A high bar makes the system draw more, which costs more and risks odd pictures. A low bar returns library photos even when they fit badly. A good threshold is found by checking real requests, not by guessing.
One more design choice deserves a word: why [IMG] tokens, and why more than one? A single vector is a thin way to describe a rich scene. Several tokens give the mapper room to carry the object, its style and its layout, a little like using a whole sentence instead of one word. The cost is small, because the mapper only has to learn a translation, not a new skill. This is also why the same recipe works with other frozen generators: swap the decoder, retrain the mapper, and leave the language model alone.
Suppose the drawn avocado chair looks nothing like the request. The symptom is a clean picture of the wrong thing. Measure it: ask the frozen generator the same request typed as plain text, and compare. If the typed prompt works and the mapped one does not, the mapper is the weak link, and more varied caption data is the fix. If both fail, the generator itself cannot draw that idea, and no adapter can add what the frozen generator lacks.
Chapter 5
Split drawing into two steps, first decide what the picture is about, then fill in every detail
The lecture now turns from chat to the pure task: words in, picture out. Two prompts on the slide set the tone. “An astronaut riding a horse in the style of Andy Warhol.” “A bowl of soup that is a portal to another dimension as digital art.” Neither scene exists. A system that makes them must combine ideas it has only seen apart.
Think of how a human illustrator works on such a brief. She does not start by choosing pixel colours. She first settles what the picture is: an astronaut, a horse, bold flat colour. Only then does she draw. The recipe on slide 16, from the DALL-E 2 paper (Ramesh and colleagues, 2022), copies that order. It has three parts.
Part 1, the CLIP encoder. CLIP (Chapter 3) turns an image into one vector, its image embedding. That vector is a short summary of what the image shows: its objects, style and mood. It holds very little pixel detail. The slide calls these embeddings the “conditional latent variables”: hidden summaries that the rest of the system conditions on. They are pretrained, so this part is not trained again.
Part 2, the prior. The text “An armchair in the shape of an avocado” goes through a text encoder and becomes a text embedding, one vector of, say, 768 numbers (illustrative). The slide’s middle box, labelled “autoregressive model” (one that writes its output piece by piece, each piece after the last), takes that vector and predicts a CLIP image embedding: a second vector of the same size that stands for a picture that fits. This box is called the prior, because it gives the probable image embeddings before we draw.
Part 3, the decoder. A diffusion model reads the predicted image embedding and generates the picture. Diffusion (Chapter 7) starts from noise and clears it step by step, using the embedding as a guide. The paper then adds upsampling models that sharpen the result to full size. Input: one vector. Output: pixels.
It is a fair question. Why not feed the text embedding straight into the decoder? Because a sentence and a picture are not the same kind of thing. The sentence is vague about most of the picture. A CLIP image embedding holds what an actual picture holds: this chair, this shade of green. The prior’s job is to choose one of the many image embeddings that fit the words.
That choice is a random draw, and this is where variety comes from. Ask twice and the prior picks two different embeddings, so the decoder draws two different chairs. The same sentence gives a different avocado chair each time. Below, play with how wide the prior spreads.
The cross is what the prompt means in embedding space. The ring is “on topic”. Each dot is one image embedding the prior drew, and each card is the picture the decoder would make from it. Widen the spread, then press Draw 6.
A toy: the map has two dimensions where a CLIP embedding has hundreds, the draws are random in your browser, and each card is a stylised glyph whose colour and tilt follow its dot. The idea is the paper’s: a prior that samples image embeddings for the decoder.
Read the result. At a tiny spread every card looks the same and sits on the ring’s centre: safe, but dull. At a huge spread the cards differ wildly and several fall outside the ring: lively, but they stop matching the words. Every real system sets this dial somewhere between, and it is a trade between variety and fidelity.
The pieces are trained in separate jobs, and each job is simple. CLIP is trained once on image-caption pairs. The prior learns from pairs of (text embedding, true image embedding) made by running the CLIP encoders over a caption dataset. The decoder learns from (image embedding, image) pairs: given the true embedding of a picture, regenerate that picture. At generation time we chain them: text to text embedding, prior to image embedding, decoder to pixels. The chain is trained in pieces and run as one.
A worked example shows the trade. Say the prompt’s meaning sits at the centre of a map, and the on-topic ring has radius 0.35. At a spread of 0.10, a typical draw lands about 0.10 from the centre, well inside the ring, so almost every picture matches but looks alike. At a spread of 0.50, a typical draw lands near 0.50, outside the ring, so many pictures drift from the words. The pictures are more varied, and fewer of them are what you asked for.
Suppose you ask for “a red cube on top of a blue cube” and the picture shows a blue cube on top. The symptom is the right objects in the wrong relation. Measure it: draw 20 images and count how many put red on top. A summary vector holds which things are in the picture better than which is above which, so the count is often near half. The fix is not a bigger dial. It is a better summary, or a model that reads relations more carefully. This kind of failure is a known weakness of compressed summaries, and it is why later systems moved to richer conditioning.
Chapter 6
Turn an image into a short list of whole numbers, then write new images the way a language model writes new sentences
Look at the slide of avocado armchairs. A grid of twenty-five small pictures, each a different chair, each green and round, some with a pit-shaped cushion. The prompt was the same eight words for every one. This is the DALL-E result of 2021 (Ramesh and colleagues, ICML), and it stunned people because it came from a model that, at heart, works like a text predictor.
You already know how a language model writes: it picks one token, adds it to the text, and repeats. Could a model write a picture the same way? Only if a picture is a list of tokens. It is not. A picture is a grid of continuous colour values. So the first job is to change the picture into tokens.
An autoencoder is a pair of networks. The encoder squeezes an input into a small summary. The decoder rebuilds the input from the summary. It is trained so the rebuilt copy is close to the original. The word “discrete” adds one rule: the summary may only hold whole numbers picked from a fixed list.
The slide shows exactly this. An image goes into the image encoder. Out comes a row of digits, for example [98] [3] [990]. Each digit is a number from 0 to 8,192, and the slide calls each one a visual token. A second picture, the shop front, gives [46] [390] [6]. The image decoder turns each row back into a picture that looks like the original.
How does the encoder pick a whole number? Through a codebook: a table of 8,192 vectors, each with a number. The encoder looks at one small patch of the image and writes a vector for it. The system then finds the codebook entry closest to that vector, and records the entry’s number. This snap-to-nearest step is called quantization. It works like rounding a price to the nearest dollar: you lose the cents, and you gain a number you can store.
The cost of rounding is detail. Fewer codes means coarser rounding. Try it on a small picture below.
Left: the original picture, 256 cells. Right: the same picture after every cell snaps to its nearest code. Drag the number of codes. Watch the error fall as the codebook grows.
A toy: each cell is a colour and each code a colour found by k-means clustering, run live in your browser. Real DALL-E codes describe whole patches, not single colours, and its codebook holds 8,192 entries (slide 17). The error counts the average distance in red, green and blue steps out of 255.
Now a picture is a row of numbers, and we can use a language model on it. Slide 18 shows how. Put the prompt text first. Then ask for one visual token. The model gives a probability to each of the 8,192 codes. Sample one: say [56]. Add it to the end, so the model now reads “An armchair in the shape of an avocado. [56]”, and ask again. It gives [73]. Add it. Then [67]. And so on, until the whole grid is filled.
In the DALL-E paper the text is 256 tokens, and the image is a 32 by 32 grid of 1,024 tokens, so the model writes a sequence 1,280 long. Finally the image decoder from Step 1 turns the 1,024 digits back into pixels. The top of this page shows the same loop on a tiny grid: press Play in Token by token mode and each cell appears in order.
Look at the data flow of one step. Input: a list of integers, the prompt tokens followed by the image tokens so far. Output: a vector of 8,192 probabilities. Sampling from it adds exactly one integer. Nothing is ever edited after it is written; each token is a commitment. That is the price of this recipe, and Chapter 7 shows how it compares with the alternative.
The slide also points to Esser and colleagues (CVPR 2021), who made a closely related system called “Taming Transformers” for high-resolution images. It uses the same two-step idea: learn a codebook, then model the sequence of codes with a transformer.
Look at the shop-front image on the slide. Its sign reads as a French pastry shop. A model with too small a codebook turns letters into mush. The symptom: the sign is a smear. Measure it before any generation: encode a real picture, decode it, and look at the reconstruction. If the sign is already a smear there, the generator can never do better, because it can only write codes the decoder understands. The fix is more codes, finer patches, or both. Another symptom is repeated patterns, a sign that the sampler has latched onto a loop.
Chapter 7
Draw a picture by clearing noise in a small hidden space, and see why that beats writing it one cell at a time
Chapter 6 left us with a worry. The token recipe writes a picture cell by cell, and it cannot go back. If it picks a bad token early, every later token must live with it. A 32 by 32 grid also means 1,024 passes through the model, one after another. It is like a painter who must paint the canvas in reading order and may never touch up a stroke.
Slide 19 gives the modern answer in one line: “Nowadays: replacing the VAE with a diffusion model.” The source is Rombach and colleagues (CVPR 2022), High-Resolution Image Synthesis with Latent Diffusion Models, the paper behind Stable Diffusion. To see why it helps, we need diffusion from zero.
Take a clear photo and add a little random noise. Then a little more. After enough rounds the photo is pure static. That forward path is easy, and we can run it as far as we like. Diffusion trains a network to walk the path backwards: given a noisy image, remove a little of the noise. Start from pure static, run the denoiser many times, and a clean picture appears.
What does the network predict? Usually the noise itself. The training example is easy to make. Take a real picture x0, draw random noise ε, and mix them: xt = √α · x0 + √(1 − α) · ε. Here α is a number between 0 and 1 that says how much of the clean picture survives (1 means all). Show the network xt and ask for ε. If it knows ε, it can undo the mix.
Work one number by hand. Let one pixel of the clean picture be x0 = 0.8, the noise ε = 0.5, and α = 0.36, so √α = 0.6 and √(1 − α) = 0.8. Then xt = 0.6 × 0.8 + 0.8 × 0.5 = 0.88. Suppose the network guesses ε = 0.5 correctly. Undo the mix: x0 = (0.88 − 0.8 × 0.5) / 0.6 = 0.48 / 0.6 = 0.8. We get the pixel back exactly. Real networks guess imperfectly, so the walk back is done in many small steps, each removing a little noise.
Running that denoiser on full pixels is costly. A 512 by 512 colour picture holds 786,432 numbers, and the denoiser contains attention (Chapter 1), whose cost grows with the square of the number of positions. The slide’s picture shows the fix. A pixel-space encoder E compresses the image into a small latent space, the same idea as the autoencoder of Chapter 6 but with continuous numbers, not whole-number codes. The diffusion process runs on the latent, and a decoder D turns the final clean latent into pixels.
The denoiser itself, a U-Net (a network that shrinks the picture to see the big shapes, then grows it back, passing detail across on skip connections), is shown in the middle of the slide with skip connections and a “denoising step” loop. How does it hear the prompt? Through cross-attention: queries come from the image latent and keys and values come from the conditioning, exactly the Q, K, V of Chapter 1. The slide’s right-hand box lists what can be plugged in as conditioning: a semantic map, text, other representations, and images. Stable Diffusion shrinks a 512 by 512 picture by a factor of 8 per side, to a 64 by 64 latent with 4 channels.
The big square is a 512 by 512 colour picture. The small square is the latent the denoiser works on. Choose how far the autoencoder shrinks each side and compare the two bars.
Exact counts for a 512 by 512 colour picture: 3 numbers per pixel, and 4 latent channels at each shrink factor above 1 (the Stable Diffusion setting). Attention pairs are positions squared, the cost of comparing every position with every other once. The speed-up is real arithmetic; the picture quality at each factor is not shown here.
With the shrink at 8, the denoiser holds 16,384 numbers instead of 786,432, which is 48 times fewer. It compares 4,096 positions instead of 262,144, and the number of position pairs falls from about 68.7 billion to about 16.8 million. That is the whole reason diffusion became practical for large pictures.
Diffusion changes the whole picture at every step. Early steps set the big shapes and later ones add the detail. A mistake in an early step can still be corrected later, because nothing is final until the last step. The count of passes is the number of denoising steps, say 50, not the number of cells. The top of this page shows the contrast: in Noise to picture mode the whole grid sharpens together. The price is that each pass is bigger, and that we must choose the step count.
The session’s second reading, Movie Gen: A Cast of Media Foundation Models (Meta, 2024), takes this recipe to video. Its abstract describes a family of models that make 1080p video with synchronized audio, edit video from instructions, and make a personalised video from a user’s photo. The largest video model is a 30 billion parameter transformer with a context of up to 73 thousand video tokens, which stands for a 16 second video at 16 frames per second.
Do the arithmetic: 16 seconds at 16 frames per second is 256 frames, and 73,000 tokens over 256 frames leaves under 300 tokens per frame. Far less than one token per pixel, so video also has to live in a compressed latent space. That is the same idea as this chapter, with time added. Read the full lesson on the paper for how it does it, and the paper itself.
Suppose a 16× shrink gives pictures with the right layout but smeared fine detail, such as hair or text. That is the symptom. Measure it the same way as Chapter 6: encode a real photo, decode it, and compare before running any diffusion. If detail is already gone, the denoiser can never bring it back. The fix is a smaller shrink, more latent channels, or a better autoencoder. If you want the full mathematics of diffusion, our diffusion lesson and flow and diffusion models lesson build it from zero.
Chapter 8
Learn how likely any picture or sentence is, then use that one skill to score, to sample and to learn features
Spread a thousand photos of dogs on a table. Pick up a new photo and ask: does this look like it belongs in the pile? A photo of a puppy on grass does. A photo of static does not. A model that can answer that question for any photo has learned something deep about what dog photos are.
From here the lecture steps back from the recipes of Chapters 1 to 7. They were all answers to one question, and slide 20 states it: learn to model p(x), where x is text, an image, a video, or any mix. The symbol p(x) reads “the probability of x”. For a continuous thing like an image it is a density: a score that is high for realistic x and low for unrealistic x.
The slide lists three, and each is a skill you already met. First, evaluate. Given x, compute p(x). Realistic data should score high. This is how Chapter 6’s token model scored a whole sequence: multiply the probability of each token.
Second, sample. Draw a new x according to p(x), so that likely things come up often and unlikely things come up rarely. Every system in this lecture does this when it makes a picture.
Third, learn representations without labels. If the model can tell what makes dog photos likely, it has found what they share: ears, a tail, fur. Those shared features are useful for other tasks, which is the thread from Chapter 1. The slide calls this unsupervised representation learning.
Where does p(x) come from? We pick a family of formulas with adjustable numbers, called parameters and written θ, and we adjust them until the model rates our real data as likely as possible. The slide says “by maximizing likelihood”. The likelihood of the data is the product of the model’s probability for each example. A product of many small numbers gets tiny fast, and a computer rounds it to exactly zero, so in practice we add logarithms instead. A log turns a product into a sum.
Try the whole idea on the smallest model there is: a single bell curve, the Gaussian, with two parameters. Its centre μ says where the data sits. Its spread σ says how wide it is. Twelve data points sit on the line below.
The dots are real data. The curve is your model. Slide its centre and spread, or press Best fit. Then slide the test point to evaluate, and press Draw 5 to sample.
Toy data: twelve made-up numbers, the same on every visit. The best fit is exact: the mean and the standard deviation of the data are the parameters that maximise the sum of log-densities for a Gaussian. The draws are random in your browser.
Slide to the best fit. The total log-likelihood climbs to its peak, and the curve hugs the data. Move the test point to the edge and its density drops. Press Draw 5 and the new circles land where the dots already are. One model, three skills: evaluate, sample, and (through μ and σ) a summary of what the data shares.
One worked number. A Gaussian with μ = 3 and σ = 1 gives density 1 / (σ√(2π)) × e−(x−μ)² / 2σ². At x = 3 that is 0.399. At x = 5, two spreads away, it is 0.399 × e−2 = 0.054. The model finds x = 3 about 7 times more natural than x = 5.
Often we want more than any realistic sample. Slide 21 adds two extensions. The first is p(x | c), conditional generation: draw x given a condition c. Here c can be a category, such as faces or outdoor scenes. In this lecture c is usually a prompt, and that is every text-to-image system of Chapters 4 to 7.
The second is p(x2 | x1, c), style transfer: given an example x1, make a new x2 in a different style c. The slide shows photos turned into paintings, and also sentences turned from negative to positive: “my goodness it was so gross” becomes “my goodness it was so awesome”. The meaning stays and one style attribute changes.
Suppose the samples look perfect, but every one is a near copy of a training image. The model has memorised its data, which fits the likelihood well and teaches nothing. Measure it: for each sample, find the closest training example and compute the distance. If the distances are close to zero, that is copying. The fix is a simpler model, more varied data, or a penalty that discourages memorising. Another trap: the density of a narrow curve can be above 1 (a spread of 0.1 gives a peak of 3.99). A density is not a probability, so it may exceed 1 without any error.
Chapter 9
Explain messy data with a few hidden choices, and build a rich model by blending simple ones
Look at a wall of face photos, the one on slide 22. No two are alike. But you can say why they differ: gender, eye colour, hair colour, pose, and so on. Nobody wrote those labels next to the photos. The factors are real, and they are hidden: they shaped each photo, and the photo does not list them.
The lecture’s idea is to put those hidden factors into the model on purpose. A latent variable, written z, is a quantity the model uses to explain the data but that the data never shows. We write the photo as x. The model then says: first some z is chosen, and then x is made from it. Slide 23 draws the graph with z above x, and shades only x, because only x is observed.
Why is this hard? We would like each z to mean something readable, such as hair colour. But we cannot write the rules from z to x by hand, and z is never observed. So we do the only thing left. We let the model learn them, as the slide says.
Start with the smallest version. Let z be a single choice among K groups, such as “which of three kinds of face is this”. Inside each group the data follows a Gaussian bell (Chapter 8) with its own centre and spread. This is the mixture of Gaussians (also called a GMM). The slide writes it as two lines: z follows a categorical distribution over 1 to K, and p(x | z = k) is a Gaussian with centre μk and spread Σk.
The slide gives the generative process in two steps. Step 1: pick a mixture component by sampling z. Step 2: generate a data point by sampling from that Gaussian. Like rolling a die to choose a coin, then flipping the coin.
This is the slide’s point: combining simple models into more expressive ones. One bell curve can only be a single bump. Three bells, weighted and added, can be a landscape with three peaks, even though each piece stays simple.
We do not know the groups, so how do we learn them? There is a loop that works, and the slide names it: expectation maximization, EM for short. It alternates two steps. Expectation: use the current centres and spreads to estimate p(z = k) for every point, which means “how much does each group claim this point?” Maximization: use those claims to update each group’s centre and spread, so each group moves toward the points that claimed it. Repeat.
A small example makes this concrete. A point at x = 3.0 sits between two groups with centres 2 and 4, both with spread 1 and equal weight. Both bells give it the same density, so each group claims it by 0.50. In the update, that point pulls each centre toward 3.0, but only half as hard as a point that belongs to one group alone. As the loop runs, groups settle onto the clumps, and the claims become decisive.
Dots are data. Each curve is one group; the bold line is their sum, the model’s p(x). Dot colour shows which group claims it. Press E step and M step by hand, or Run 10.
Toy data: 45 one-dimensional numbers drawn from three made-up clumps, the same on every visit. The loop is exact EM for a one-dimensional mixture. The starting centres are spread evenly across the line, so a run is the same every time.
Try two groups on the three-clump data, and then four. With too few groups one bell must cover two clumps and the fit is poor. With too many, a group may split a clump or sit on one lone point. The model needs a sensible K; the data does not tell you for free.
The session’s first reading, Compositional Generative Modeling: A Single Model is Not All You Need (Du and Kaelbling, 2024), turns this slide into an argument. Its abstract says we should build large generative systems by composing smaller generative models, not by training one giant model. It claims three gains: learning distributions in a more data-efficient way, generalising to parts of the data distribution unseen in training, and building new generative models for tasks never seen in training. It also shows that separate components can often be discovered from data.
The mixture is the smallest example of composing: the whole model is a weighted choice among simple parts, and each part can be learned from fewer data than the whole. The word “or” lives in the sum. Multiplying densities makes an “and” instead: the product of two bells N(0, 1) and N(2, 1), renormalised, is a narrower bell centred at 1 with spread 0.71, which prefers points that both parts like. The paper develops ways to compose in this spirit; see our lesson on the paper, and the paper itself.
The classic EM failure: one group collapses onto a single point. The symptom is a log-likelihood that suddenly shoots up. Measure it: check each group’s spread and weight. A spread near zero with a weight near 1/N means it is wrapped around one dot. The model assigns that dot enormous density, and the likelihood looks great while the model is useless. The fix is a floor on the spread, or a prior that rules out collapse.
Chapter 10
Replace a handful of groups with a smooth hidden space, and train an encoder and a decoder that agree on it
A mixture of Gaussians has K groups and no more. Real faces do not fall into three tidy kinds. They vary smoothly: a little more smile, a little more tilt. What if the hidden choice z were not one of K labels, but a whole point in a smooth space, with a few numbers per photo?
That is slide 27, and it is the jump from mixture to variational autoencoder (VAE). We put a prior on z: a rule for which z values are normal before we see any data. The slide picks the simplest, the standard Gaussian, z ~ N(0, I): a bell around 0 in every direction. The decoder is a neural network that maps z to the mean and spread of a Gaussian over x. Given z, x is simple. But z can take infinitely many values, so the model is an infinite blend of Gaussians. That is how a simple p(x | z) gives a rich p(x).
We want to train by maximum likelihood (Chapter 8). The likelihood of one picture x is the sum, over every possible z, of p(x, z). Slide 28 shows why that is hopeless. If z is a list of 30 binary switches, the sum has 230, about a billion, terms for each data point. If z is continuous, the sum becomes an integral with no closed form. The slide marks it “intractable”. We need a cheaper trick.
The trick of slide 29 is variational inference. What we really want is p(z | x): given a photo, which z could have made it? That is hard to get exactly. So we pick a simple family, here a Gaussian q(z; φ), and tune its parameters φ until it is as close as possible to the true p(z | x), while staying cheap to compute. The slide’s figure shows a hard-shaped true curve and two Gaussian guesses: the orange one is closer than the green one.
In a VAE the guess is made by a neural network. Give it a photo x, and it outputs the Gaussian’s centre μ and spread σ. This network is the encoder. It reads x and answers “here is the cloud of z values that could have made this photo”.
Slide 30 lists them. Take a photo xi, say 64 × 64 × 3 numbers. One: the encoder maps it to μ and σ, each a vector of, say, 128 numbers (illustrative). Two: sample a random ε from N(0, I), also 128 numbers. Three: compute z = μ + σ ⊙ ε. Four: the decoder maps z back to a reconstruction x̂, again 64 × 64 × 3.
Step three is the reparameterization trick. Why not just draw z straight from N(μ, σ)? Because a random draw blocks the flow of gradients, the signals that tell the encoder how to improve. Writing z = μ + σε moves the randomness into ε, which has no parameters. Now z is a plain formula in μ and σ, and gradients pass through it. A worked case: μ = 0.4, σ = 0.5, ε = −1.2 gives z = 0.4 + 0.5 × (−1.2) = −0.2.
For Gaussians the second term has a closed form: ½(μ² + σ² − 1 − ln σ²) for each latent number. For μ = 0 and σ = 1 it is exactly 0. For μ = 1 and σ = 0.5 it is ½(1 + 0.25 − 1 + 1.386) = 0.82. A cloud that sits far from 0, or is much narrower than 1, costs more.
Why keep the second term? Without it the encoder cheats. It can place each photo at its own far-off, razor-thin spot in z space, like giving every guest a private island. Reconstruction is then perfect, but a random z from the prior lands in empty ocean, where the decoder has never been. The prior term pulls all the clouds toward the centre and makes them overlap, so the space between photos is meaningful. Slide 31 shows the real result with and without the term: a packed, smooth latent map against a scattered one. Test the idea below.
Each coloured dot is a photo’s latent point, two kinds of photo. The grey rings are the prior N(0, I). Small dots are random draws from the prior: green ones land near a real photo’s cloud, red ones land in empty space. Drag the weight of the KL term.
A toy geometry, not a trained network: as the KL weight falls the clusters spread outward and shrink, and as it rises they pack toward 0 and widen. The mapping from weight to geometry is illustrative; the draws and the green or red count are computed live from 400 fixed draws. The idea matches the lecture’s slide 31 (with and without the KL term).
Slide 32 sums the VAE into five ideas. One: an encoder and a decoder. Two: a simple latent variable z. Three: a complex decoder p(x | z) built from neural networks. Four: a reconstruction objective. Five: a prior over z, which makes the latent space smoother so we can sample from it. The slide adds that the prior pulls the dimensions of z apart, so each one stands for something different. Its picture shows a face, encoded into distributions over attributes (smile, skin tone, gender, beard, glasses, hair colour), sampled, and decoded: the same attributes give a faithful copy, and nudging one gives a variant.
The usual symptom is blurry samples. Measure it with the two terms: log them separately during training. If the KL term is near 0 and the reconstruction is poor, the decoder has learned to ignore z, a failure called posterior collapse. The fixes are to raise the decoder’s reliance on z (a weaker decoder, or ramping up the KL weight slowly from 0). VAEs are fast and smooth. Their samples are often blurrier than those of the token and diffusion recipes of Chapters 6 and 7, and that is one reason those recipes are popular for pictures. The VAE idea lives on inside them as the autoencoder.
Chapter 11
Keep the numbers, the terms, the checklist and the core code, and find where to go next
The lecture’s closing slide lists its three topics again: text-to-image generation, an introduction to generative models, and mixtures of Gaussians with VAEs. Here is the story in one paragraph. A multimodal model fuses senses by letting one channel attend to another (Chapter 1). A frozen language model learns to see through one trained layer (Chapter 2), to fetch pictures through a retrieval token (Chapter 3), and to draw through an adapter into a frozen image generator (Chapter 4). Image generators work in two ways: predict a CLIP summary, then decode it (Chapter 5), or turn pictures into tokens and write them (Chapter 6), with diffusion in a latent space now the common choice (Chapter 7). Underneath sits one object, a model of p(x) (Chapter 8), and one idea for building it richly, hidden variables: first a mixture (Chapter 9), then a VAE (Chapter 10).
| What | Number | Where it comes from |
|---|---|---|
| Attention score table, words by signals | 3 × 4 | Slide 5; X1 is 3 × d, X2 is 4 × d |
| Weights for “privilege” | 0.7 and 0.3 | Slide 5; a score gap of 0.85 reproduces them |
| Trained layer in MiniGPT-4 | 1 linear layer | Slide 7 and the paper’s abstract; sizes in Chapter 2 are illustrative |
| Codebook size for visual tokens | 8,192 | Slide 17 |
| Image tokens per picture (DALL-E) | 1,024 | The DALL-E paper: a 32 × 32 grid, after 256 text tokens |
| Latent for a 512 × 512 colour picture | 16,384 vs 786,432 | Exact arithmetic for the Stable Diffusion setting, 48 times fewer |
| Terms in the intractable sum | 230 | Slide 28; thirty binary latent switches |
| KL cost, μ = 1, σ = 0.5 | 0.82 | Closed form, worked in Chapter 10 |
| Movie Gen largest video model | 30B, 73K tokens | The paper’s abstract: 16 s at 16 frames per second |
| Term | Plain meaning |
|---|---|
| Modality | One channel of information: text, voice, a face, a photo. |
| Frozen | A part whose numbers are held fixed during training. |
| Adapter | A small trained piece that translates between two frozen models. |
| Cosine similarity | How well two vectors point the same way: 1 same, 0 unrelated, −1 opposite. |
| Prior (CLIP recipe) | The model that draws an image embedding for a text embedding. |
| Visual token | A whole number that names one codebook vector for an image patch. |
| Quantization | Snapping a vector to the nearest codebook entry. |
| Latent variable | A hidden quantity the model uses to explain data that the data never shows. |
| Maximum likelihood | Choose the parameters that make the real data most likely. |
| Reparameterization | Write z = μ + σε so gradients flow through the random draw. |
pythonimport numpy as np
# Chapter 3: pick the library image closest to the [RET] vector
def retrieve(ret_vec, library): # ret_vec (512,), library (N, 512)
a = ret_vec / np.linalg.norm(ret_vec)
B = library / np.linalg.norm(library, axis=1, keepdims=True)
scores = B @ a # (N,) cosine similarities
return int(scores.argmax()), scores
# Chapter 6: write an image as visual tokens, one at a time
def draw_tokens(model, text_ids, n_tokens=1024, rng=np.random.default_rng(0)):
seq = list(text_ids) # the prompt comes first
for _ in range(n_tokens):
probs = model(seq) # (8192,) probabilities over the codebook
seq.append(int(rng.choice(len(probs), p=probs)))
return seq[len(text_ids):] # (1024,) ids, then the image decoder
# Chapter 7: one diffusion training step (alpha = how much of x0 survives)
def diffusion_loss(x0, alpha, eps_net):
eps = np.random.randn(*x0.shape)
xt = np.sqrt(alpha) * x0 + np.sqrt(1 - alpha) * eps
return ((eps_net(xt, alpha) - eps) ** 2).mean()
# Chapter 10: one VAE step, minimise recon + KL (the negative of the bound L)
def vae_loss(x, enc, dec):
mu, log_var = enc(x) # each (128,)
eps = np.random.randn(*mu.shape) # noise with no parameters
z = mu + np.exp(0.5 * log_var) * eps # reparameterize
recon = ((x - dec(z)) ** 2).sum() # Gaussian log-likelihood, up to a constant
kl = 0.5 * (mu**2 + np.exp(log_var) - 1 - log_var).sum()
return recon + kl
This lecture is the front half of generative multimodal AI. The next session, Week 6.2, goes on to diffusion, flow models and controllable generation. For the pieces here, our other lessons build them from zero: VAEs and VQ-VAEs, diffusion, flow and diffusion models, CLIP, vision-language models, GANs, attention and transformers and latent variables and adversarial models.
The two readings have lessons of their own in this series: Compositional Generative Modeling and Movie Gen. The papers taught inside chapters are MiniGPT-4 (arXiv 2304.10592), the grounding paper (2301.13823), the image-generation paper (2305.17216), DALL-E (Ramesh and colleagues, ICML 2021) and DALL-E 2 (Ramesh and colleagues, 2022), and latent diffusion (Rombach and colleagues, CVPR 2022).
Now press Teach and explain, from memory, how a model that reads a clip learns to make a picture. Say where the picture comes from in the token recipe, in the noise recipe and in a VAE. Then say what p(x) has to do with all three. Then go back to the picture at the top, switch between the two modes, and say which one could fix an early mistake.
Credit: this lesson follows MIT MMAI (Multimodal AI), Spring 2026, Lecture 6.1, Multimodal Generation, by Paul Liang and the MIT Multisensory Intelligence group. Course site: mit-mi.github.io/mmai-course/spring2026/schedule; slides PDF. Every idea is re-taught in our own words and devices; no slide images are used.