Architectures & Latent Space — Flow Matching & Diffusion (MIT 6.S184)

Chapter 0: Why — the Toy Is Over, Build the Real Thing

The whole lecture in one sentence. For the last three lectures the vector field was a hand-written formula or a tiny MLP pushing dots around a 2-D plane. Today we replace it with a real neural network that can transport a 196,608-number image from Gaussian static into a dog — and, crucially, we make that practical by first compressing the image into a small latent space. By the end you'll understand every box in the Stable Diffusion 3 and Movie Gen diagrams.

Recall the spine of the whole course. A flow or diffusion model is an ODE (or SDE) whose velocity arrows — the vector field u^θ_t(x) — are produced by a neural network with weights θ. To sample, you start at a Gaussian point and Euler-step along those arrows from time t = 0 to t = 1. Lectures 2 and 3 told us how to train the arrows (flow matching, score matching) and how to steer them with a prompt (guidance).

But all of that was demonstrated on toy 2-D Gaussians, where the vector field was a three-layer MLP. The elephant in the room: a real image is not a 2-D point. A 256×256 RGB photo is a vector with d = 3·256·256 ≈ 196,608 numbers; a 1024×1024 photo has d ≈ 3 million. An MLP that maps a 3-million-dimensional input to a 3-million-dimensional output is hopeless — the first weight matrix alone would have trillions of entries.

What must this network actually compute? It takes three things — a noisy image x (a tensor of pixels), a time t (a single scalar), and an optional condition y (a class label or text prompt) — and returns a velocity, one number per pixel, telling each pixel which way and how fast to move. The output must have exactly the same shape as the image input. The art of this lecture is designing a network that does this efficiently, exploiting the structure of images (locality, multiple scales, redundancy).

Two big ideas carry the lecture, mapping onto the two sections of the original MIT slides:

The two jobs of Lecture 4.

Architectures (Section 7). How do we shape the network so the input and output are both images? Two answers dominate: the U-Net (a convolutional encoder–decoder) and the Diffusion Transformer (DiT) (chop the image into patch tokens and run a transformer). And how do we feed in a scalar time so the network can actually use it? — time embeddings.
Latent space (Section 6). Diffusing in pixel space is wasteful because nearby pixels are highly redundant. So we first compress every image with a (variational) autoencoder into a much smaller latent, run the entire flow/diffusion process there, and decode once at the very end. This is latent diffusion, and it is how essentially every image and video generator you've ever seen actually works.

We have a neural network now (finally), but we are not training it from scratch here — that's the loss from Lectures 2–3. Today is pure plumbing: what goes in, what comes out, what shape, and why each design choice was made. You'll patchify a toy image, build a Fourier time embedding, and round-trip an image through a compressing autoencoder — all in your browser.

Why can't we just reuse the tiny MLP vector field from the earlier lectures for real images?

Because MLPs can't handle a time input. The vector field's output must be the same shape as the input image (hundreds of thousands to millions of numbers); a dense MLP over that space is computationally hopeless and ignores image structure. Because images are not vectors. Because MLPs only work for classification, never generation.

Chapter 1: The Interface — What Goes In, What Comes Out

Before designing any architecture, pin down the contract the network must satisfy. Get the types and shapes exactly right and the rest is just choosing how to fill the box in the middle.

The vector-field contract. The network u^θ_t(x | y) takes three inputs and produces one output:

x ∈ R^d — the current (noisy) image, a tensor of shape C×H×W (channels × height × width).
t ∈ [0, 1] — the time, a single scalar saying how far along the noise-to-data journey we are.
y — the optional condition: a class label (an integer) or a text prompt (a string).
output ∈ R^d — the velocity, the same shape as x: one arrow component per pixel.

The "same shape in, same shape out" requirement is the single most important structural fact. The velocity is a vector in the same space as the image, because Euler's update is x ← x + h·velocity — you add the velocity to the image, so they must align coordinate-for-coordinate.

Why an MLP works for toys but not images. For the 2-D Gaussian toys, x had d = 2, so concatenating [x, embed(y), t] and passing it through a small fully-connected MLP that outputs a 2-vector was perfectly adequate. That is exactly what the earlier labs did. For images, d explodes, and a fully-connected layer connecting every pixel to every pixel would have d² weights — about 4×10¹⁰ for a 256×256 image, before you even add depth. We need an architecture that shares weights and exploits the fact that images have local, multi-scale structure.

There's a subtle asymmetry worth naming, straight from the slides. In ordinary supervised learning — say image classification — the input is a big image but the output is tiny (one class number). So a classifier can keep narrowing: big image in, small label out, and never pay the cost of producing a full-sized output. Our problem is different.

Why high dimension hurts diffusion more than classification. Two reasons compound. First, the output of u^θ_t is just as big as the input — a full image-shaped velocity — so we cannot narrow down to a scalar; the network must carry the full resolution through to the end. Second, we call the network many times: every sample requires simulating the ODE for, say, 50 to 1000 Euler steps, each one a full forward pass. A classifier runs once; a sampler runs the net dozens of times. So a fat, slow network is dozens of times worse for us than for a classifier.

Two more inputs need embedding before the network can use them, because they don't arrive in image shape:

Time t is a single scalar in [0, 1] — one lonely number against millions of pixel numbers. If we just paste it on raw, the network barely notices it. Chapter 2 fixes this with a time embedding.
The prompt y, if it's text, is a string. We turn it into vectors with a frozen pre-trained text encoder — CLIP, T5, or an LLM's embeddings — producing a sequence of vectors PromptEmbed(y) ∈ R^S×k for a prompt of S tokens. (A class label is even simpler: a learned lookup-table embedding, one vector per class.) Because these encoders are pre-trained and frozen, the diffusion model gets language understanding for free.

What is the defining shape constraint on the vector-field network for images, and why does it hold?

The output must be a single scalar (the image's quality score). The output must be smaller than the input, like a classifier. The output (the velocity) must have the same shape as the input image, because the Euler update adds velocity to the image coordinate-by-coordinate. The output must be a probability distribution over classes.

Chapter 2: Time Conditioning — the Fourier Embedding

The vector field changes with time: at t near 0 the input is almost pure noise and the network should predict the rough overall structure; at t near 1 it is almost a finished image and the network should refine fine details. So the network must know what t is. The problem: t is a single number, drowned out by the millions of pixel numbers.

Why a raw scalar t is a bad input. Imagine appending t = 0.37 as one extra feature next to a hundred-thousand-element pixel vector. A linear layer would give it one weight column out of a hundred thousand — its influence is a rounding error. Worse, a single scalar passed through linear layers can only produce effects that are smooth and roughly linear in t. But the right behavior can change sharply between early and late times. We need to turn one number into a rich, high-dimensional, expressive signal that the network cannot ignore.

The fix: sinusoidal (Fourier) features

The trick — the same one used for positional encodings in transformers — is to map the scalar t to a vector of sines and cosines at many different frequencies. With embedding dimension d (even), pick d/2 frequencies w₁, …, w_d/2 and form:

TimeEmb(t) = √(2/d) · [ cos(2πw₁t), …, cos(2πw_d/2t), sin(2πw₁t), …, sin(2πw_d/2t) ]

Read it carefully. Each component is a wave; a high-frequency wave wiggles many times as t sweeps 0→1, a low-frequency wave barely moves. Together they form a fingerprint of t that is easy for the network to read. Two nearby times produce nearly identical fingerprints (smoothness); two distant times produce very different fingerprints (distinguishability).

How are the frequencies chosen?

We want both slow waves (to track the coarse phase of generation) and fast waves (to resolve nearby times finely). So we space the frequencies geometrically — equally spaced in log-scale — between a minimum and a maximum:

w_i = w_min · (w_max / w_min)^{(i−1)/(d/2−1)}, i = 1, …, d/2

So w₁ = w_min (slowest), w_d/2 = w_max (fastest), and each step multiplies the frequency by the same ratio. A geometric ladder covers a huge dynamic range — from waves that complete a fraction of a cycle over [0,1] to waves that complete hundreds — using only d/2 entries.

The √(2/d) is a normalizer. Because sin²(x) + cos²(x) = 1 for every angle x, each frequency contributes exactly 1 to the squared length when you sum its sine and cosine. Summing over d/2 frequencies gives squared length d/2; multiplying every entry by √(2/d) rescales the squared length to (d/2)·(2/d) = 1. So every time embedding is a unit vector — same magnitude regardless of t. That keeps the signal well-scaled going into the network, no matter the time. The slides stress: the exact form matters less than the fact that it's a normed, d-dimensional embedding.

Worked number. Take a tiny embedding with d = 4, so d/2 = 2 frequencies. Let w_min = 1 and w_max = 4. Then w₁ = 1 and w₂ = 1·(4/1)^{(2−1)/(2−1)} = 4. At t = 0.25: the angles are 2π·1·0.25 = π/2 and 2π·4·0.25 = 2π. So cos(π/2) = 0, cos(2π) = 1, sin(π/2) = 1, sin(2π) = 0. The raw embedding is [0, 1, 1, 0]; its length is √(0+1+1+0) = √2, so after the √(2/4) = √(1/2) factor it becomes [0, √(1/2), √(1/2), 0], a unit vector. One number, t = 0.25, became a four-dimensional unit signal the network can act on.

The time embedding — one scalar t becomes a wall of sine/cosine waves

A heatmap: each row is one embedding dimension (a fixed frequency); the horizontal axis is time t from 0 to 1; warm/blue is the +1/−1 value of that wave. Low rows are slow waves, high rows are fast waves — the geometric frequency ladder. Drag a slider to add more frequencies or stretch the max frequency; the vertical line marks the current t and the side strip shows the embedding vector at that t (this is what the network actually receives).

freqs d/216

max freq16

time t0.30

Now build the embedding yourself in pure numpy — and prove that distinct times map to distinct, nearly-orthogonal fingerprints while the embedding stays smooth in t.

Why embed the scalar time with sines and cosines at geometrically-spaced frequencies instead of feeding the raw number t?

Because neural networks cannot read decimal numbers. A lone scalar is swamped by the millions of pixel features and only allows smooth, near-linear time dependence; the multi-frequency embedding turns t into a rich, normed, high-dimensional fingerprint that is smooth in t yet separates distant times. To make the embedding non-differentiable. Because time must be an integer for the network.

Chapter 3: The U-Net — an Image-Shaped Network

Now the architecture itself. The earliest and longest-reigning choice for image diffusion is the U-Net — a convolutional neural network whose input and output both have the shape of an image. That property is exactly what we need: feed in a noisy image, get out a velocity image of the same size.

What is a convolution, in one line? A convolution slides a small filter (say 3×3) across the image, computing a weighted sum of each local neighborhood. The same filter weights are reused at every location — that's weight sharing. So a convolution has only a handful of weights regardless of image size, and it builds in two facts about images: locality (nearby pixels matter most) and translation equivariance (a feature looks the same wherever it appears).

The U shape: contract, then expand

A U-Net is an encoder–decoder with a twist. The encoder repeatedly shrinks the spatial size while growing the number of channels; the decoder does the reverse. Follow one 256×256 RGB image (the slides' example) through it:

Input

3 × 256 × 256 noisy image x_t

↓ encoders (downsample, add channels)

Bottleneck (latent)

512 × 32 × 32 — small & spatial, many channels

↓ midcoder (process at lowest resolution)

Decoders (upsample)

grow H, W back — drop channels

↓ with skip connections from matching encoder

Output

3 × 256 × 256 velocity u^θ_t(x_t)

Notice: as resolution drops (256 → 32), the channel count rises (3 → 512). Coarse spatial information gets traded for richer per-location feature descriptions. The lowest-resolution block in the middle — the slides whimsically call it the midcoder — processes the most abstract, global representation.

The crucial feature: skip connections. The downsampling path throws away fine spatial detail (exact edge positions, texture). The upsampling path needs that detail back to reconstruct a sharp output. So a U-Net adds skip connections: each decoder block receives, alongside its upsampled input, a direct copy of the same-resolution feature map from the encoder. Picture a "U" — the two arms are encoder (down) and decoder (up), and the skip connections are horizontal wires bridging matching levels. Without them the output is blurry; with them the network keeps the high-frequency detail intact. (The name "U-Net" is literally the shape of the diagram.)

Where do t and y enter?

The time embedding from Chapter 2 and the condition embedding are injected into every block — typically by adding them (after a small projection) to the feature maps, so each convolutional stage knows the current time and prompt. This is the same idea we'll see formalized as AdaLN in the DiT next chapter.

Why the U-Net dominated early diffusion (and why it makes sense). Three properties line up perfectly with the task. (1) Image in, image out — the shape contract is satisfied by construction; the U-Net was originally built for image segmentation, where output is also image-shaped. (2) Locality via convolutions — cheap, weight-shared, well-suited to pixel data. (3) Multiscale via the encoder/decoder pyramid — it reasons about both coarse layout (in the bottleneck) and fine texture (via skips), which is exactly what denoising at different noise levels needs. In practice modern U-Nets also sprinkle in a few attention layers at low resolution, blending convolution with the global mixing of the next chapter.

U-Net dataflow — watch resolution shrink and channels grow, then reverse

The encoder arm (left) shrinks spatial size and grows channels; the bottleneck sits at the bottom; the decoder arm (right) mirrors it back to image size. Dashed teal wires are the skip connections carrying fine detail across the U. Drag the depth slider to add levels — deeper U-Nets reach a smaller, more abstract bottleneck. Each box shows its (channels) × (height × width).

depth3

What problem do the U-Net's skip connections solve?

They speed up training by skipping layers. They inject the time embedding. Downsampling discards fine spatial detail; skip connections feed that detail directly from each encoder level to the matching decoder level, so the output stays sharp instead of blurry. They convert the image into patches.

Chapter 4: The Diffusion Transformer (DiT)

The U-Net's competitor — now the dominant architecture for state-of-the-art models — is the Diffusion Transformer (DiT) (Peebles & Xie, 2023). Its bet: instead of convolutions, treat an image like a sentence of patches and run a standard transformer over it. Transformers scale astonishingly well with data and compute, and DiTs inherit that.

Step 1 — Patchify: image → sequence of tokens

A transformer eats a sequence of vectors. An image is a grid of pixels. The bridge, borrowed from the Vision Transformer (ViT), is patchification: chop the image into a grid of small P×P patches and flatten each patch into one vector ("token").

Patchify(x) ∈ R^N×C′, C′ = C·P², N = (H/P)·(W/P)

Here N is the number of patches (the sequence length, often written L), C is the input channels, P is the patch size, and C′ = C·P² is the number of numbers in one flattened patch. Then a learned linear map (a weight matrix W) projects each C′-vector to the transformer's hidden dimension d, giving the patch embeddings PatchEmb(x) ∈ R^N×d.

Worked count. A 32×32 single-channel image (C = 1) with patch size P = 4 gives N = (32/4)·(32/4) = 8·8 = 64 patches, each flattened to C′ = 1·4·4 = 16 numbers. So patchify turns a 1×32×32 image into a 64×16 matrix — a sequence of 64 tokens. Halve the patch to P = 2 and you get 256 tokens of 4 numbers each: more tokens, finer granularity, more compute. The patch size is the knob trading resolution against sequence length.

Step 2 — Transformer blocks process the tokens

Now run L transformer layers (DiT blocks) over the token sequence. The engine is scaled dot-product attention: every token forms a query, key, and value; the queries and keys decide how much each token attends to every other; the output mixes the values accordingly. Crucially, attention is global — any patch can directly talk to any other patch in one layer, unlike a convolution's small local window.

The three things a DiT block fuses (from the slides).

Image via self-attention. Queries, keys, and values all come from the image patches — the patches reason about each other.
Text via cross-attention. Queries come from the image patches; keys and values come from the prompt embeddings — the image "looks at" the text to honor the prompt.
Time via adaptive layer norm (AdaLN). The time embedding produces per-channel scale and shift parameters that modulate the block's normalization. So time doesn't add a token; it reshapes the activations — a clean, parameter-light way to condition on t.

The block's contributions are summed via residual connections, exactly like a language transformer.

A class-conditional DiT (like the one in the course lab) is simpler: it drops the cross-attention and folds the class label into the same AdaLN conditioning as time, by adding the class embedding to the time embedding. Text-conditional models keep the cross-attention to attend to the full prompt sequence.

Step 3 — Depatchify: tokens → image

After the L blocks, the token sequence is still N×d. We need a velocity image of shape C×H×W. So a final linear map sends each token back to C·P² numbers, and we reassemble the patches into the full grid — the exact inverse of patchify. This depatchify step restores the image shape, satisfying our contract.

u = Depatchify( PatchEmb output through L DiT blocks ) ∈ R^C×H×W

DiT pipeline — patchify → tokens + time/text → transformer → depatchify

An image is sliced into a grid of patches (left); each becomes a token in a sequence; the time embedding and prompt tokens join in; L transformer blocks mix everything via attention; then depatchify reassembles a velocity image (right). Drag patch P to see the token count change live (N = (H/P)²). Smaller patches → more, finer tokens → more compute.

patch P

Build the patchify/depatchify pair yourself and prove the round-trip is exactly lossless — the foundation the whole DiT rests on.

DiT vs U-Net — why the field moved. The U-Net bakes in locality through convolutions, which is a strong, useful prior for images. The DiT bakes in almost nothing — attention lets any patch talk to any patch from layer one — and instead leans on scale: with enough data and compute, transformers learn the right structure and keep improving as you make them bigger, more reliably than U-Nets. That is why Stable Diffusion 3, FLUX, and Movie Gen are all DiT-based. The trade is cost: attention is quadratic in the number of tokens N, which is the very reason the next idea — shrinking the image into a small latent before patchifying — matters so much.

In a DiT, what does patchification accomplish, and what undoes it?

It blurs the image; sharpening undoes it. It turns the image grid into a sequence of patch tokens so a transformer can process it; depatchify reassembles the tokens back into an image-shaped velocity at the end. It compresses the image into a latent; the decoder undoes it. It computes the loss; backprop undoes it.

Chapter 5: The Cost of Pixels — Why We Compress

We have a network that takes images and returns images. But running diffusion directly in pixel space is brutally expensive, and the reason is worth feeling in your bones.

The number that breaks everything. A high-resolution image — 3 color channels, height 600, width 1000 — is a vector of 3·600·1000 = 1.8 million numbers. A 1024×1024 RGB image is about 3 million. The vector field's output is just as large. And for video, multiply by the number of frames. We are asking a network to read and write multi-million-dimensional tensors, dozens of times, per generated sample.

The slides name three concrete problems with diffusing in raw pixel space:

GPU memory blows up. Every activation map is image-sized; stacking many layers and a batch of images quickly exhausts even a large GPU.
The learning problem is very hard. A 3-million-dimensional distribution is a vast space; learning to navigate all of it is far harder than a low-dimensional one.
Redundancy. Nearby pixels are highly correlated — a patch of blue sky is thousands of near-identical numbers. Modeling each pixel independently spends enormous capacity on perceptually irrelevant detail.

Why this hurts diffusion more than ordinary supervised learning. A classifier maps a big image to a tiny label and runs once. A diffusion model produces a full image-sized vector field and runs it many times (one network call per Euler step, 50–1000 steps). So both the per-call cost and the number of calls scale with the punishing dimension. The slides put it bluntly: "We learn a vector field — output is very high-dimensional! And we apply the vector field many times!" The cost compounds.

The compression idea

Here is the escape. The set of realistic images is a tiny, structured sliver of the full pixel space — almost every random pixel array is meaningless static. So real images plausibly lie near a much lower-dimensional manifold. If we can find a faithful map from the 3-million-dimensional pixel space down to, say, a few thousand numbers and back, then we can run the entire expensive diffusion process in the small space and only pay the pixel cost once to decode at the end.

The payoff in numbers (preview). Stable Diffusion compresses a 3×256×256 image (196,608 numbers) into a 4×32×32 latent (4,096 numbers) — a 48× reduction. FLUX 2.0 compresses 3×1024×1024 (about 3.1 million) into 32×64×64 (about 131,000) — roughly a 24× reduction. The diffusion model then operates entirely on those few thousand latent numbers. Same recipe as before — just on a transformed, much smaller dataset.

The next two chapters build the compressor (an autoencoder) and then plug it into the pipeline (latent diffusion).

The cost wall — dimension and network-calls multiply

The orange bar is the per-sample work for pixel-space diffusion (dimension × number of Euler steps); the teal bar is the same after compressing into a latent. Drag the resolution and the compression factor to feel how the latent bar stays manageable while the pixel bar explodes. (Bars are on a log scale.)

side px256

compress×48

Why is diffusing directly in pixel space especially costly compared to a supervised classifier on the same images?

Because pixels are integers and classifiers use floats. Because diffusion needs color and classifiers don't. The vector-field output is as high-dimensional as the input, and the network is evaluated many times per sample (once per Euler step); both the per-call cost and the call count scale with the huge pixel dimension, while a classifier outputs a tiny label and runs once. Because classifiers don't use GPUs.

Chapter 6: (Variational) Autoencoders — the Compressor

The compression tool is an autoencoder: a pair of networks that squeeze an image down and then rebuild it.

Autoencoder = encoder + decoder. An encoder μ_φ : R^d → R^k maps a high-dimensional image x to a small latent z (with k much smaller than d). A decoder μ_θ : R^k → R^d maps the latent back to an image. The latent space R^k is the small room where we'll later run diffusion.

How do we train them? Demand that decoding an encoding gives back the original. That's the reconstruction loss — the squared error between the input and its round-trip:

L_recon(φ, θ) = E_x [ ‖ μ_θ(μ_φ(x)) − x ‖² ]

Minimize this and the autoencoder learns to throw away only the information it can afford to lose — the redundant, perceptually unimportant detail — while keeping enough to rebuild a faithful image.

The catch: a "bad" latent space

A plain autoencoder compresses, but it has a fatal flaw for our purposes. Remember the plan: after compressing, we want to learn the distribution of latents and sample new ones with a diffusion model. But the reconstruction loss says nothing about what the distribution of latents looks like.

The subtle failure (straight from the slides). Question: what happens to the data distribution when we map it into latent space? Answer: we don't know — the reconstruction loss never constrains it. So we might compress beautifully yet produce a latent distribution that is wild, clumpy, full of holes — a distribution that is harder to learn and sample than the images we started with. We'd have traded a big easy-ish problem for a small impossible one. We need to also force the latent distribution to be nice.

The fix: a variational autoencoder (VAE)

A variational autoencoder (VAE) upgrades the autoencoder so the latent distribution is regularized toward a simple, friendly shape — a standard Gaussian. Two changes:

The encoder becomes probabilistic. Instead of outputting one latent, it outputs a small Gaussian: a mean μ_φ(x) and a variance σ²_φ(x). To encode, you sample z from that Gaussian. (Set the variance to zero and you recover the plain autoencoder.)
A second loss pulls the latent toward a Gaussian. We add a regularizer that penalizes the encoder's output distribution for differing from a standard Gaussian prior N(0, I), measured by the KL divergence — a standard measure of how different two distributions are (it is zero exactly when they match, and positive otherwise).

The full VAE loss balances the two with a weight β ≥ 0:

L_VAE(φ, θ) = L_recon(φ, θ) + β · L_prior(φ)

Read the four terms (for a Gaussian encoder). Expanded, the loss has intuitive pieces: a reconstruction error (rebuild x accurately), a decoder-confidence term (how sharply to trust the reconstruction), a term pushing the latent mean toward 0, and a term pushing the latent variance toward 1. The last two are what make the encoded data look like a clean standard Gaussian — exactly the easy-to-learn distribution we want to diffuse in. The weight β tunes compression-niceness vs. reconstruction-quality; in practice modern autoencoders use a very small β, so reconstructions stay sharp and the latent is only gently regularized.

One practical wrinkle: the reparameterization trick. The loss averages over z sampled from the encoder, whose distribution depends on φ — you can't differentiate through a random sample directly. The fix: write z = μ_φ(x) + σ_φ(x)·ε with ε ~ N(0, I). Now the only randomness is ε, which doesn't depend on φ, so ordinary backprop flows through μ and σ. This is the same noise-injection idea you've seen all course, used here to make a stochastic encoder trainable.

You'll build the simplest faithful version of this idea next — a linear encoder/decoder using the top principal directions of the data — and prove it both compresses (small latent) and reconstructs (low error vs. a trivial baseline).

Why does a plain autoencoder (reconstruction loss only) fall short for latent diffusion, and what does the VAE add?

It compresses too much; the VAE compresses less. It is too slow; the VAE is faster. Reconstruction loss never constrains the latent distribution, which can come out wild and hard to sample; the VAE adds a KL term that pulls the latent toward a clean standard Gaussian so a diffusion model can learn it. It cannot decode; the VAE adds a decoder.

Chapter 7: Latent Diffusion — the Whole Pipeline

Now snap the pieces together. A latent diffusion model (LDM) — the architecture of Stable Diffusion and essentially every modern image/video generator — runs the entire flow/diffusion process inside the VAE's latent space, and only touches pixels at the very start and end.

The latent diffusion recipe (from the slides).

Data. Take all training images x₁, …, x_N (e.g. the whole internet).
Encode. Push every image through the (pre-trained, frozen) VAE encoder to get its latent (for a VAE, just take the mean prediction).
Latent dataset. This gives a dataset of latents z₁, …, z_N — a much smaller version of every image.
Latent diffusion model. Train a flow/diffusion model on the latents — the model now generates latent vectors, using the U-Net or DiT from this lecture.
Decode. After sampling a latent from the diffusion model, push it through the VAE decoder back to a full image.

Return the decoded image as your sample.

Train time

image → VAE encoder → latent → (diffusion model learns latents)

↓ once, offline (VAE is frozen)

Sample time

Gaussian noise → Euler-step the diffusion model in latent space → latent

↓ decode ONCE at the end

Output

VAE decoder(latent) → full-resolution image

The most important sentence in the lecture. "Exact same recipe as before — just with a transformed dataset." Nothing about flow matching, score matching, or guidance from Lectures 2–3 changes. You simply apply that recipe to the dataset of latents instead of pixels. The diffusion model never sees a raw pixel during its many Euler steps; it works in the tiny latent space, where each step is dozens of times cheaper. The expensive pixel-shaped decode happens exactly once, after sampling is complete.

Two practical notes the lecture flags. First, you must train the autoencoder before training the diffusion model — the latent space has to exist first. Second, at decode time you take the decoder's mean output rather than a random sample, to avoid injecting noise artifacts into the final image. A well-trained autoencoder acts like a perceptual filter: it strips away high-frequency, semantically-meaningless detail so the diffusion model can spend all its capacity on the structure that matters.

Data-flow check (Concept + Realization). Shapes for Stable Diffusion: an image x is 3×256×256 (196,608 numbers). The frozen VAE encoder maps it to a latent z of shape 4×32×32 (4,096 numbers). The DiT diffusion model lives entirely in that 4×32×32 space: at sample time it starts from a 4×32×32 Gaussian and runs ~50 Euler steps, each a forward pass producing a 4×32×32 velocity. The final 4×32×32 latent is decoded once to a 3×256×256 image. The 48× smaller working space is why this is feasible at all.

"Virtually all AI-generated images or videos that you see," the slides say, "are generated in latent spaces." Memory would blow up otherwise, and the latent lets the model focus on the parts that matter.

In latent diffusion, where does the diffusion model do its many Euler steps, and how often do we touch full-resolution pixels?

In pixel space; we touch pixels every step. Entirely in the small VAE latent space; we decode to pixels only once, at the very end (and encode once, at training time). In the prompt's text space; pixels are never used. It alternates between latent and pixel space at every step.

Chapter 8: Case Study — Stable Diffusion 3 & Meta Movie Gen

Let's see every piece assemble at scale in two state-of-the-art systems. Notice how little is new — they are this lecture's components, made enormous.

Stable Diffusion 3 (images)

SD3 ingredient list.

Flow matching with straight-line (CondOT) schedules — they tested many flow/diffusion variants and found flow matching best (Lecture 2).
Classifier-free guidance, weight 2.0–5.0 (Lecture 3).
Latent space: flow matching runs in the latent of a pre-trained VAE (this lecture).
Architecture: an MM-DiT (multi-modal DiT) — a DiT extended from class-conditioning to text-conditioning, processing text and image jointly through the whole network via cross-attention.
Text conditioning: combines CLIP (coarse, overall meaning) and T5-XXL (granular, sequence-level detail) embeddings, fed via cross-attention.
Scale: 8 billion parameters, 50 sampling (Euler) steps, trained on LAION.

Read that list against the course: a flow-matching objective (L2), guidance (L3), a DiT in a VAE latent with a time embedding (L4). The MM-DiT's only real novelty is extending the conditioning from a single class label to a full sequence of text tokens via cross-attention — exactly the "text via cross-attention" branch of the DiT block from Chapter 4.

Meta Movie Gen (video)

Movie Gen ingredient list.

Flow matching with straight-line schedules — same objective as SD3.
Classifier-free guidance for prompts.
Latent space — even more critical here, because video adds a time-of-the-movie dimension on top of height and width, so raw pixel cost explodes. A temporal autoencoder (TAE) compresses along the frame axis too.
Architecture: a DiT adapted to video — the image is patchified across both space and the temporal frame dimension; self-attention among patches, cross-attention with text.
Text conditioning: three embeddings (UL2 for reasoning, ByT5 for character-level detail, MetaCLIP for shared text-image space).
Scale: 30 billion parameters, trained on 6,144 H100 GPUs.

The one question the slides pose: what changes for video? Almost nothing conceptual. A video is just an image with an extra frame dimension — a bigger tensor. So you (1) compress the extra dimension with a temporal autoencoder (memory matters even more), and (2) patchify across time as well as space so the transformer's tokens span frames. Everything else — flow matching, latent space, DiT, guidance — is reused unchanged. This is the whole lesson of the case studies: the components compose, and going from images to video is mostly bookkeeping for one more axis.

Stepping back: both flagship systems are the same five Lego bricks — flow matching (L2), guidance (L3), time embedding + DiT + VAE latent (L4) — scaled to billions of parameters and thousands of GPUs. You now understand every box in their architecture diagrams.

SD3 vs Movie Gen — the same bricks, scaled

A side-by-side of the two systems' shared pipeline (encode → latent diffusion via DiT → decode) and where they differ (image vs. video; text encoders; parameter count). Tap a stage to highlight it across both columns — the shared bricks light up in teal, the video-only additions in orange.

Going from Stable Diffusion 3 (images) to Movie Gen (video), what is the main conceptual change?

Switching from flow matching to a completely different objective. Mostly handling one extra dimension (frames): compress it with a temporal autoencoder and patchify across time too; flow matching, latent space, DiT, and guidance are reused unchanged. Abandoning the latent space because video is too complex. Replacing the DiT with a U-Net.

Chapter 9: Showcase — Patchify and Compress, Live

The payoff: a live image studio combining the two structural moves of the lecture. Pick a toy image, then watch it get patchified into tokens (the DiT's first step) and, separately, compressed into a small latent and reconstructed (the VAE's job). Play with the knobs and feel the trade-offs you've learned.

The image studio — patches & latent compression side by side

Left: the original image overlaid with the patch grid (the DiT's tokens); the count updates with the patch slider. Middle: the same image compressed to a low-rank latent of size k and decoded back — this is the VAE's compress-and-reconstruct, done here with the top-k principal directions of a simple image family. Right: the per-pixel reconstruction error. Drag latent k: small k = heavy compression = blurrier reconstruction (more error); large k = faithful but larger latent. Drag patch P to change the token count. Change the image to see structure-dependent behavior.

image

patch P

latent k6

What to notice. (1) Patches vs. tokens: halving the patch size quadruples the token count — and a DiT's attention cost grows with the square of that, so patch size is a real budget knob. (2) Compression trade-off: as you shrink the latent k, the reconstruction loses high-frequency detail first (sharp edges blur) while the coarse structure survives — precisely the "throw away perceptually-cheap detail" behavior we want, and why diffusing in latent space loses little. (3) Image dependence: smooth images (gradient, blob) compress beautifully with tiny k; busy images (fine checker) need more latent to look right — real autoencoders learn far better, nonlinear bases than our linear toy.

This is the literal substrate every modern generator runs on: an image becomes a small latent (left/middle), a DiT (operating on patch tokens of that latent) runs the flow in the small space, and a single decode at the end returns a full image. The simulation is the test — no quiz here.

Chapter 10: Connections & Cheat Sheet

You assembled the real machine. The vector field is no longer a formula or a toy MLP — it is a U-Net or a Diffusion Transformer, conditioned on an embedded time and prompt, running inside a VAE's compressed latent space. That is the architecture behind every image and video generator in production.

The spine of Lecture 4. The network u^θ_t(x | y) must map an image-shaped input to an image-shaped velocity. Time enters via a sinusoidal (Fourier) embedding (a normed, multi-frequency fingerprint). Architecture is a U-Net (convolutional encoder–decoder with skip connections) or a DiT (patchify → transformer → depatchify). Because pixel space is huge and redundant, we first compress with a VAE (encoder + decoder + KL regularizer for a nice latent) and run the entire flow/diffusion in latent space, decoding once. SD3 and Movie Gen are exactly these bricks at scale.

Cheat sheet

Interface. u^θ_t(x | y): inputs x ∈ R^d (image C×H×W), time t ∈ [0,1], condition y; output a velocity of the SAME shape as x.

Time embedding. TimeEmb(t) = √(2/d)·[cos(2πw_it), sin(2πw_it)], frequencies w_i geometric from w_min to w_max; the result is a unit vector (sin²+cos²=1).

U-Net. Convolutional encoder (shrink H,W, grow channels) → midcoder → decoder (reverse), with skip connections carrying fine detail across. Image in, image out. Locality + multiscale.

DiT. Patchify (N = (H/P)(W/P) tokens, each C·P² numbers) → L transformer blocks (self-attn on patches, cross-attn to text, AdaLN for time) → depatchify. Scales better than U-Nets; attention cost is quadratic in N.

VAE. Encoder μ_φ: R^d→R^k, decoder μ_θ: R^k→R^d, k « d. Loss = reconstruction + β·KL(latent ‖ N(0,I)). KL makes the latent Gaussian-like and learnable.

Latent diffusion. Encode all images to latents; train the flow/diffusion model on latents; sample a latent; decode once. SD3: 196,608 → 4,096 latent (48×). Same recipe, smaller dataset.

At scale. SD3 = MM-DiT, flow matching, CLIP+T5 text, 8B params, 50 steps. Movie Gen = video DiT, temporal autoencoder, 30B params, patchify across time.

The series so far — and what's next

Lecture 1 — Flow & Diffusion Models. The engine: ODEs, SDEs, Euler, Brownian motion — the simulator this network plugs into.
Lecture 2 — Flow Matching. The loss that trains the vector field you just learned to build.
Lecture 3 — Score Matching & Guidance. The score function and classifier-free guidance — how the prompt y (Chapter 1's condition) actually steers generation.
Lecture 4 — Architectures & Latent Space (this one). The network and the compressed space it lives in.
Lecture 5 — Discrete Diffusion. Carrying these ideas to text and language — where there is no Euclidean space, so ODEs/SDEs become continuous-time Markov chains.

Related lessons on this site

Lecture 1 — Flow & Diffusion Models (Gleam) — the ODE/SDE engine.
Diffusion Transformer (Gleam) — a focused deep-dive on the DiT block.
Vision Transformer (Gleam) — where patchify comes from.
VAE & VQ-VAE (Gleam) — the autoencoder in full detail.
Diffusion (Architecture Atlas) — bird's-eye overview.

The Feynman test. Open the original Lecture 4 slides. Can you now explain every line — the input/output contract, why a raw scalar time is a bad input and how the Fourier embedding fixes it, the U-Net's U-shape and skip connections, the DiT's patchify–transformer–depatchify with self/cross-attention and AdaLN, why pixel space is too expensive, what a VAE adds over a plain autoencoder, the five-step latent-diffusion recipe, and how SD3 and Movie Gen are just these bricks at scale — to a friend, from memory? If yes, you can read any modern image-generation paper.

One sentence: what is the architecture of a modern image generator like Stable Diffusion 3?

A single feed-forward network that outputs the image in one pass. A database lookup over training images. A flow/diffusion model whose vector field is a Diffusion Transformer (with a Fourier time embedding and text cross-attention) running inside a frozen VAE's latent space, decoding to pixels once at the end. A U-Net that classifies the prompt into one of a thousand fixed images.