Video Diffusion Models

A video model starts from noise and refines every frame of a clip at once, so each frame can see all the others. Frame by frame, the same toy ball jumps between two paths.

Learn how a diffusion or flow model turns noise into an image, and then into a whole clip: the sampler, guidance, the denoiser, the space it works in, and the space-time transformer.

Press Make 12 clips with Frame by frame, then switch to Whole clip together. Each clip is a ball crossing a strip: time runs down, position runs across. Then we build, piece by piece, everything underneath: the noise, the sampler, the guidance, the network and the space it works in.

You need a rough idea of what a probability is and what a matrix is. We build the rest from zero.

Based on Lecture 12 of Penn CIS 6280: World Models (Fall 2026), taught by Jiatao Gu. Lecture slides. Every idea is re-taught here in our own words and devices; it follows Lecture 11, Dreamer, Diffusion and Flow Matching.

Make clips from noise

20

Toy world, ours: a clip is 5 frames of 11 pixels. The ball goes left to right or right to left, and both are equally likely. The denoiser is exact for these two clips, so any jump comes from the sampling method, not from a weak network. Both methods start from the same noise.

Chapter 0

Why a Video Is a Hard Thing to Draw

Count the numbers in a short clip, and see why one noisy guess per frame is not enough

Ask a model for a short clip of a ball rolling across a table. It sends back 80 pictures, called frames, to play one after another. Look at them one at a time and each is fine: a sharp ball, a clean table. Play them and something is wrong. In frame 40 the ball sits at the left edge. In frame 41 it sits at the right edge.

No single frame is at fault. The fault lives between the frames. A clip is not a pile of pictures. It is one story, and every picture must agree on it. This lesson shows how a model can make all the frames agree, and it builds each tool the model needs on the way.

The setting is Lecture 12 of Penn CIS 6280, World Models. A world model is a learned model of an environment that can say what happens next. The lecture teaches the generator that sits inside many of them: a model that makes video. One short note on the course itself: the final project information comes out on October 16, Assignment 2 is due October 19, and office hours are on Thursday from 4:30 to 5:30 PM in AGH 423.

How big is a clip?

A pixel is one dot of a picture. A dot has three channels, the amounts of red, green and blue, and each amount is one number. So a picture with 480 rows and 832 columns holds 480 × 832 × 3 = 1,198,080 numbers. A clip stacks frames, so it adds one more axis: time.

We call a block of numbers with named axes a tensor. A clip is a tensor with the shape T × H × W × 3: T frames, H rows, W columns, three channels. Try the sizes yourself.

Count the numbers in a clip

The deck shows your frames. The bars use a log scale, so each gridline is ten times the one before. Change the length, the frame rate and the picture size.

5
16

Sizes are illustrative, chosen for the arithmetic. The count is exact: frames × rows × columns × 3 channels. The photo bar is a 256 × 256 picture, 196,608 numbers.

Work the default by hand, one step at a time. First, 5 seconds × 16 frames per second = 80 frames. Second, 480 × 832 = 399,360 pixels in one frame. Third, 399,360 × 3 channels = 1,198,080 numbers in one frame. Fourth, 1,198,080 × 80 frames = 95,846,400 numbers in the clip.

Now compare. The 256 by 256 photo holds 256 × 256 × 3 = 196,608 numbers. Divide: 95,846,400 ÷ 196,608 ≈ 487.5. The clip is nearly 500 photos. Diffusion touches every number many times, once for each step of its sampler. So a video model must be careful about every number it makes.

Two things follow. First, any trick that shrinks the tensor before the sampler runs pays off hugely, and Chapters 6 and 9 are about such tricks. Second, any network layer that compares every number with every other number is out of reach, because the pair count grows with the square. Chapter 8 returns to that sum, and it is the reason video models factor their attention.

Why not draw each frame alone?

An obvious plan: take a good image generator and run it 80 times. This fails for a plain reason. A generator draws a sample, one new example chosen by chance from everything it has learned. Run it 80 times and you get 80 independent chances. Each draw is free to pick a different ball path, a different table, a different light.

The hero at the top of this page is a tiny version of this. The ball starts at one side and ends at the other, and both directions are equally likely. A good clip picks one direction and keeps it. When each frame decides alone, a frame may pick “left to right” while the next picks “right to left”. The ball jumps. Go back up and count the red outlines.

The fix is the lecture’s central idea. The model does not draw frames. It draws one whole clip, as a single object, and it lets every frame look at every other frame while it decides. Slide 43 asks it directly: how can the same generative framework produce a whole video?

The words you need first

A diffusion model is a generator that learns to remove noise. To make something new, you start from pure random noise and remove a little of it, again and again, until a clean picture is left. Each removal is one step. The program that decides how to take the steps is the sampler. A condition is extra information you hand the model, such as a text prompt or a first frame.

The lecture builds this in an order we will follow. Slides 3 to 28 give the tools for images: the noise ladder, the DDIM sampler, flow matching and guidance. Slides 29 to 42 give the network, the space it works in, and text to image. Slides 43 to 61 stretch all of it to video. The tools do not change. The tensor just gains a time axis.

The road

  1. The noise ladder: how a clean picture is corrupted, and the one step that walks back.
  2. DDIM: skip rungs, and choose how much fresh noise each step adds.
  3. Flow matching: learn a velocity, then walk it in Euler steps.
  4. Guidance: run the network twice and push toward the condition.
  5. The denoiser: a U-Net or a patch transformer.
  6. Where to generate: pixels, latents or features.
  7. Text to image: cross-attention and the full pipeline.
  8. A whole clip: joint corruption and frames that attend to each other.
  9. Space-time tokens: the video autoencoder, patches and the cost of attention.
  10. Generate video: train, sample, and condition on text or an image.
  11. The field guide: numbers, code, evaluation and what comes next.
Every frame of a generated clip looks sharp, yet the ball jumps from one side to the other. What does this show?

Chapter 1

Diffusion Is a Ladder of Noise

Climb a fixed ladder of corruption, then learn the one step that walks back down

Photocopy a photo. Copy the copy. Do this 1,000 times, and each copy adds a little grain. The first copies still show your face. By copy 300 the face is a smudge. By copy 1,000 you hold pure grain, and nothing of the photo remains.

Now run the film backward. If someone shows you copy 300 and asks for copy 299, you can often guess: take away a little grain and the face comes back a bit. A diffusion model is a network trained to do that one small repair. Chain 1,000 repairs and you walk from pure grain to a new picture.

The clever part is in the training. We build the ladder ourselves, so we always know the clean picture and the exact grain. That gives us a perfect answer key for every rung. Slide 3 gives this a formal name: diffusion as a hierarchical VAE. A VAE is the hidden-cause model from Lecture 7. Here the hidden causes are the noisy pictures on the ladder, yJ down to y1.

Climbing up: the forward process

Call the clean picture y0. A rung j is one level of corruption, from 1 to J. The lecture’s forward process makes each rung from the one below by shrinking the picture a little and adding fresh Gaussian noise: random numbers that cluster around zero in a bell shape. The recipe has a small number βj per rung, and αj = 1 − βj.

q(yj | yj−1)
The chance of rung j, given the rung just below it. The bar reads “given”. Only the one rung below matters.
N( mean, spread )
A Gaussian: a bell-shaped cloud of values around a centre.
√αj yj−1
The centre: the old picture, a little dimmer. Because αj is just under 1, the dimming is slight.
βj I
The spread: a small amount of independent noise on every number. I is the identity, so each number gets its own draw.

The paper behind this slide, Ho and colleagues’ DDPM (2020), picks 1,000 rungs, with β rising in a straight line from 0.0001 to 0.02. These are the numbers we use below. The step is small, so many of them are needed to bury the picture.

You never have to climb rung by rung. Gaussians stack neatly, so the lecture gives a shortcut straight from y0 to any rung. Let ᾱj be the product α1 × α2 × … × αj. It tracks how much signal survives. Then rung j is a mix of the clean picture and one noise draw ε:

yj = aj y0 + bj ε,   aj = √ᾱj,   bj = √(1 − ᾱj)

Read aj as the signal weight and bj as the noise weight. The device shows both, and the picture they build.

Climb the noise ladder

Left to right: the clean picture, one fixed noise draw, and their mix at rung j. Below, the two weights across all 1,000 rungs. Drag the rung.

250

The 20 by 20 picture is a toy of ours. The schedule is the DDPM one: β from 0.0001 to 0.02 over 1,000 rungs. The noise draw is seeded, so it is the same on every visit.

At rung 250, ᾱ is 0.524, so a = 0.724 and b = 0.690. The picture is still easy to read. At rung 500 the signal weight has fallen to a = 0.280 and the noise weight has risen to b = 0.960. By rung 1,000, ᾱ is 0.00004, so a = 0.006. Nothing is left, and y1000 is almost exactly a plain Gaussian draw. That is the starting point for generation.

Walking down: the reverse target

Now the hard question on slide 3: how does our choice of the forward recipe q fix the reverse step? During training we know y0. So we can ask for the exact chance of rung j−1 given rung j and y0. Bayes’ rule says it is proportional to two Gaussians multiplied: the forward step into rung j, and the shortcut down to rung j−1.

Two Gaussians multiplied give another Gaussian. Slide 5 shows the algebra: write u for yj−1, collect the squared terms into a number A and the linear terms into a vector B, then complete the square. The result is a Gaussian with mean B/A and variance 1/A. In plain words, precisions add (precision is one over variance) and the mean is a precision-weighted blend.

μ̃j
Where the best guess for rung j−1 sits, given what we know.
weight on y0
A pull toward the clean picture.
weight on yj
A pull toward the noisy picture we hold now.
β̃j
The leftover spread. It is the 1/A from the algebra.

Work a tiny case by hand. Use two rungs: β1 = 0.1 and β2 = 0.2. Then α1 = 0.9, α2 = 0.8, ᾱ1 = 0.9 and ᾱ2 = 0.72. Take one pixel with y0 = 1.0 and y2 = 0.5, and find the target for y1.

A = 0.8/0.2 + 1/(1 − 0.9) = 4 + 10 = 14  ·  B = (√0.8/0.2)(0.5) + (√0.9/0.1)(1.0) = 2.236 + 9.487 = 11.723  ·  mean = B/A = 0.837, variance = 1/A = 0.071

The target pulls y1 to 0.837. That sits between the noisy 0.5 and the clean 1.0, and nearer the clean one. At generation time we do not have y0, so the network must supply it.

The learned step

The network takes the noisy picture and the rung, and returns its guess of the noise: ε̂ = εθ(yj, j). From the mixing recipe, a guess of the noise is also a guess of the clean picture: ŷ0 = (yj − bj ε̂) / aj. Put ŷ0 into the mean above. Then add a little fresh noise with the variance β̃j:

yj−1 = (1/√αj) ( yj − (βj/bj) ε̂ ) + √β̃j ξ,   ξ ~ N(0, I)

Training the network is plain regression. Pick a clean picture, a rung and one noise draw. Build yj with the shortcut. Ask the network for the noise, and score it with the squared gap to the true ε. The tensors keep one shape: a picture in, a same-shaped noise guess out.

The failure mode lives in the schedule. Symptom: samples come out with a wrong overall brightness and washed-out colour. Measurement: if you cut the ladder to 100 rungs, ᾱ100 is 0.897, so a = 0.947. The top rung still holds 95% signal. Yet the sampler starts from a pure Gaussian draw with none, an input the network never met in training. Fix: end the schedule where the signal weight is near zero, as the 1,000-rung one does.

During training you need the noisy picture at rung 500 for one clean image. What is the cheapest correct way to get it?

Chapter 2

DDIM: Same Noise, Different Path Down

Choose how much fresh noise each step adds, and skip rungs of the ladder

Two hikers start on a foggy mountain top and both want the base. The first follows the road, with 1,000 switchbacks. The second cuts straight down the slope and reaches the same elevation marks, only far sooner. Both hit marker 750, then 500, then 250. They never share a footpath, but they share the markers.

That is the whole idea of DDIM, the denoising diffusion implicit model of Song and colleagues (2020). The markers are the noise levels. Chapter 1’s sampler walks every switchback. DDIM keeps the markers and changes the path.

Why the path is free

Look at what training uses. Slide 8 says it in one line: denoising training samples a clean picture, one noise level and one noise draw. It never asks how rung 500 connects to rung 499. It only needs, for each level alone, the mix yj = aj y0 + bj ε.

So many different “connections between levels” produce the same single-level mixes. DDIM picks a new connection and keeps every level’s mix. The network already trained for Chapter 1 works unchanged.

Building a new connection

Take a clean picture y0 and its rung-j mix yj. Both are known in training, so we can read off the noise that made it: εj = (yj − aj y0) / bj. Slide 9 then builds the picture one rung down from three parts:

yj−1
The picture one rung down.
aj−1 y0
The clean signal, at its weight for the lower rung.
cj εj
Retained noise: the same noise draw we already have, kept at a reduced weight.
σj ξ
Fresh noise: a new independent draw ξ, weighted by σj.
cj
Chosen so the total noise variance at the lower rung is exactly bj−1², which keeps the level’s mix correct.

The knob is σj. It says how much noise to retain and how much to add fresh. Set σj large and the step forgets the old noise. Set it to zero and the step keeps all of it.

Put numbers on the knob. Step from rung 500 down to rung 250. The target noise weight at rung 250 is b = 0.690, so b² = 0.476. The DDPM choice for this jump is σ = 0.663, which uses 0.439 of that variance on fresh noise. That leaves c = √(0.476 − 0.439) = 0.192 for retained noise. With σ = 0, all of the weight, 0.690, stays on the old noise. The total variance is 0.476 in both cases. Only the split differs.

One side effect has a long name: the forward process is now non-Markovian. A Markov chain forgets everything but its last step. This chain does not, because the step down still looks at y0 (slide 10). That is fine: this is only the answer key q, the inference side. It is not yet the generator.

The generator, with no clean picture

At generation time y0 does not exist. Slide 11 fills the gap the same way as Chapter 1: the network guesses the noise, and the noise gives a clean estimate ŷ0 = (yj − bj ε̂) / aj. Replace y0 and εj with those guesses. Slide 12 gives the update, with the three labelled parts: clean signal, retained noise, fresh noise.

yj−1 = aj−1 ŷ0 + √(bj−1² − σj²) ε̂ + σj ξ

Variance choiceResulting sampler
σj² = β̃jThe posterior-variance DDPM of Chapter 1. Fresh noise at every step.
σj = 0Deterministic DDIM. The only random thing is the starting draw.

Slide 13 checks that putting σj² = β̃j into this update returns exactly the DDPM mean and variance of Chapter 1. So DDPM is one point on a dial, and DDIM is the other end.

Skipping rungs

With σ = 0 the update is yj−1 = aj−1 ŷ0 + bj−1 ε̂. Nothing in it needs the rung below to be j−1. Slide 14 says so: connect any level k below j with yk = ak ŷ0 + bk ε̂. Re-run the network after each update. Fewer chosen levels mean fewer network calls.

Try it. The device uses a world small enough to solve exactly: the clean “picture” is a single number that is either +1 or −1. For this world the best possible noise guess has a closed form, so the network is perfect and any error comes from the sampler alone. That closed form uses tanh, a smooth S-shaped function that squeezes any number into the range from −1 to 1.

Walk down with fewer steps

Each line is one draw, from noise on the left to a result on the right. Blue lines mark the two true data values, +1 and −1. Choose the step count, then slide η from 0 (DDIM) to 1 (the DDPM choice).

0.00

Toy, ours: the data is +1 or −1 with equal chance, and the noise guess is the exact one, ŷ0 = tanh(aj y / bj²). The schedule is Chapter 1’s. σ is η times the DDPM value, as in the DDIM paper. A hit is within 0.15 of a data value.

Now work one draw by hand. Start the deterministic sampler at y = 0.3 and use only 4 steps, so the levels are 1000, 750, 500, 250 and 0.

Rung j to kClean estimate ŷ0Noise guess ε̂New y
1000 to 7500.0020.3000.300
750 to 5000.0170.2990.292
500 to 2500.0890.2780.256
250 to 00.371−0.0180.371

The sampler ends at 0.371, between the two true values. That is a miss. The failure has a symptom (a blurry, in-between result), a measurement (0.371 is 0.63 from +1), and a fix. The cause is that each jump assumes the noise guess stays the same over the whole stretch, and over 250 rungs it does not. The fix is more rungs: the same start ends at 0.562 with 5 steps, 0.978 with 8, and 1.000 with 10 or more.

Keep this picture. A long straight jump on a curved path undershoots. Chapter 3 gives the picture a name, and a cleaner way to see it.

You run deterministic DDIM (σ = 0) twice from the same starting noise, once with 20 steps and once with 50. What do you expect?

Chapter 3

Flow Matching: Learn the Velocity

Draw a straight line from noise to data, regress its direction, and walk the learned field in Euler steps

Drop a leaf into a river. You do not need to know the river’s whole history. At every spot on the water there is a current, a direction and a speed. Follow the current for a second, look again, follow it for another second, and the leaf reaches the sea.

Chapter 2 ended on a hint: a long straight jump along a curved path undershoots. Slide 15 shows why the hint is exact. DDIM is already this leaf and this river. It just did not say so.

DDIM is an Euler step

Take the deterministic update from Chapter 2 and divide it by ak. The pictures become “rescaled” ones, u = y / a. Let s = b / a be the noise-to-signal ratio. The update turns into one line:

uk = uj + (sk − sj) ε̂

Read it plainly: the new value is the old value plus a short step in the direction ε̂, and the step length is the change in s. That is the Euler method: stand still, measure the direction, take a short straight step, then measure again. The continuous version says du/ds = εθ(…). So the sampler walks an ordinary differential equation (ODE): a rule that gives a velocity at every place and time.

Name the river directly

Flow matching drops the noise-prediction detour and learns the velocity itself. Slide 16 sets the notation. A new coordinate τ runs from 0 at pure Gaussian noise to 1 at data. This is the opposite direction from Chapter 1’s rung j, which counted up toward noise. The network vθ(r, τ) returns a velocity, and Euler sampling is:

r0 ~ N(0, I)  ·  rτ+Δτ ≈ rτ + Δτ · vθ(rτ, τ)

Different starting draws follow different paths, but each path is fixed once it starts. The lecture notes that a stochastic differential equation (SDE) can instead add random kicks along the way. We stay with the ODE.

The answer key is a straight line

How do we train vθ? Slide 17 gives the simplest possible recipe. Draw a noise sample n and a clean sample y0. Connect them with a straight line:

rτ
The point on the line at time τ. It has the same shape as a picture.
(1 − τ) n
The noise part, fading out as τ grows.
τ y0
The data part, fading in.
y0 − n
The velocity of the straight line. It does not change with τ: one direction, one speed.

The loss on slide 18 is a squared error: ‖vθ(rτ, τ) − (y0 − n)‖². The input is a point rτ and its time. The target is the known pair velocity. Training never runs the sampler. It picks one point on one line and does one regression, just like Chapter 1 did with noise.

The learned field is an average

Here is the subtle part, on slide 19. Two different pairs, (n, y0) and (n′, y0′), can have lines that cross at the same point at the same time. At that point the network sees one input but two different targets. It cannot give both. It learns the average, the conditional expectation: v*(r, τ) = E[y0 − n | rτ = r].

So a straight training path does not make a straight generated path. Each training line is straight. The field is a blend of many, and a sample that follows it bends.

The toy world makes this exact. The data is +1 or −1. Given the data value s, the point r at time τ is a Gaussian centred at τ s with spread (1 − τ). Bayes’ rule then gives the chance of +1 in closed form, and the average clean value is E[y0 | r] = tanh(τ r / (1 − τ)²). The average noise follows from r = (1 − τ) n + τ y0: E[n | r] = (r − τ E[y0 | r]) / (1 − τ). The velocity is the difference.

Work one point. Take τ = 0.5 and r = 0.2. Then E[y0 | r] = tanh(0.5 × 0.2 / 0.25) = tanh(0.4) = 0.380. Next, E[n | r] = (0.2 − 0.5 × 0.380) / 0.5 = 0.020. So v = 0.380 − 0.020 = 0.360. An Euler step of Δτ = 0.25 moves r from 0.200 to 0.200 + 0.25 × 0.360 = 0.290.

Follow the field in Euler steps

Time τ runs left to right, position runs up. Warm lines are Euler paths from sixteen noise draws. Switch on the training lines to see the straight pairs the field was averaged from.

Toy, ours: data is +1 or −1 with equal chance, and v is the exact field E[y0 − n | r], not a trained network. The sixteen starting draws are seeded. A hit is within 0.15 of a data value.

Start with one step. At τ = 0 the point r says nothing about which data value will come, so E[y0 | r] = tanh(0) = 0 and E[n | r] = r. The velocity is −r, and a single Euler step lands every draw at r + (−r) = 0. All sixteen draws end at the average of the two data values: 0 hits, average distance 1.000. With 2 steps, 4 draws hit. With 4 steps, 13 hit and the average distance is 0.066. With 8 or more steps, all 16 hit.

That is the failure mode, in the usual three parts. The symptom is a blurry average. The measurement is the average distance to the nearest data value. The fix is more steps, or, in real systems, a better integrator, which is beyond this lecture. Notice the match with Chapter 2: DDIM with 4 steps missed for the same reason.

In flow matching, each training pair defines a straight line from noise to data. Why can the generated path still bend?

Chapter 4

Guidance: Turn Up the Condition

Run the network twice, subtract, and push the sample toward what you asked for

You ask a painter for “a husky in snow”. A painter who works loosely gives you a husky, a rough one, sometimes in grass. You say, “More husky. More snow.” The next one is clearly a husky in clear snow, but all the huskies now look alike. You traded variety for fidelity to the request. Guidance is that request, as a dial.

Giving the model a condition

Until now the model drew any picture. Slide 21 adds a condition c: extra information the picture must match. It changes the target from pθ(y0) to pθ(y0 | c), and the network takes one more input: εθ(yj, j, c). Text reaches the network as embeddings that it reads through attention, which Chapter 7 opens up. A reference image arrives as encoded visual evidence. Training uses examples paired with their condition.

Even so, a conditional model spreads its draws over everything that fits the condition. Slide 22 gives the reason for guidance: it can emphasise samples that express the condition more strongly. The price is variety.

Two directions, from Bayes’ rule

Each noise level j has its own density of noisy pictures, pj(y). The score is the gradient of the log density: an arrow at every point that shows which way the density rises fastest. Think of it as an uphill compass. Bayes’ rule splits the conditional compass in two (slide 23):

∇y log pj(y | c)
The uphill direction for pictures that match the condition.
∇y log pj(y)
The noisy-data score: move toward more probable pictures of any kind.
∇y log pj(c | y)
The change that makes the condition more likely. It is the pull of the request.

Here y is a noisy picture at a fixed level j. The first term is what an unconditional model knows. The second is the extra pull from the condition. The lecture asks: what if we make the second pull stronger?

Classifier guidance and classifier-free guidance

The first answer, classifier guidance (slide 24, Dhariwal and Nichol 2021), trains a separate classifier pφ(c | yj, j) on noisy inputs and their noise levels. Its gradient gj is the second term. The network gives the first, since sθ ≈ −εθ/bj. Add the classifier gradient with strength w and you get sguided = sθ + w gj, or in noise form, εguided = εθ − w bj gj.

A second model is a burden. Classifier-free guidance (CFG, Ho and Salimans 2022, slides 25 and 26) removes it. During training, the condition is replaced with a null value ∅ some of the time, which the lecture calls condition dropout. One network now learns both εc = εθ(yj, j, c) and ε∅ = εθ(yj, j, ∅). Their gap points along the condition’s pull, since sc − s∅ ≈ ∇y log pj(c | yj). The combination:

εguided
The noise guess the sampler uses.
ε∅
The guess with no condition: the anchor.
w
The total scale. w = 0 gives the unconditional guess, w = 1 the ordinary conditional guess, and w above 1 extrapolates past it.
εc − ε∅
The direction the condition adds.

Some papers write this as εc + γ(εc − ε∅) with γ = w − 1. Same thing, shifted by one. For a flow sampler, the same recipe applies to velocities: vguided = v∅ + w(vc − v∅), as slide 27 shows. Video models such as Wan apply classifier-free guidance to the velocity in this way.

Inside the sampler

At each step, run the network twice on the same noisy sample and level: once with the condition and once without. Combine the two outputs into εguided. Then use that one prediction in both places the DDIM update needs it: the clean estimate ŷ0 = (yj − bj εguided) / aj, and the retained direction in yk = ak ŷ0 + bk εguided. Two network calls per step means guidance doubles the cost of every step.

Work one point in the toy below. At rung 250 the noisy value is y = 1.0, where a = 0.724 and b = 0.690. The unconditional guess is ε∅ = 0.046 and the conditional one is εc = −0.242. So ŷ0 is 1.338 at w = 0, 1.612 at w = 1, and at w = 3 the guided noise is 0.046 + 3 × (−0.288) = −0.819, giving ŷ0 = 2.162. The estimate has moved past the data’s own mode at 2.

Turn up the guidance

Bars show where 400 draws end. Without a condition the data has three clumps, at −2, 0 and 2. The condition “right side” keeps the clumps at 0 (30% of the time) and 2 (70%). Drag the guidance scale w.

1.0

Toy, ours: three Gaussian clumps of spread 0.4, with exact scores for the unconditional and the conditional distribution, and a 50-step deterministic sampler. The 400 starting draws are seeded. A real model learns the two noise guesses; the combination rule is the lecture’s.

Read the device in order. At w = 0 the draws spread evenly over three clumps. At w = 1, 66% of draws sit on the right-hand side and almost none on the left, close to the 70/30 split the condition asks for. At w = 2, 95% sit on the right-hand clump at 2. At w = 3, all of them do, their centre is 2.28 and the spread has dropped from 1.01 at w = 1 to 0.28.

That is the failure mode of too much guidance, in three parts. The symptom: every sample looks the same, and details look strained. The measurement: spread falls and the centre drifts beyond the real data, 2.81 at w = 8, against a true clump at 2. The fix: pick the smallest w that gives enough fidelity. Slide 28 says it plainly: stronger guidance favours typical, condition-aligned samples and higher apparent fidelity, and it can lose variety, coverage, and natural colours or other details when the scale is excessive.

One more limit matters for video. Guidance strengthens a learned preference. It does not make the physics right. A model that learned to draw a glass breaking too slowly will draw it more strongly too slowly.

In classifier-free guidance, what is the extra cost at each sampling step, and what does it buy?

Chapter 5

The Denoiser: U-Net or Patch Transformer

Compare two networks that both map a noisy tensor to a tensor of the same shape

Two editors fix the same blurry photo. The first zooms out to see the layout, a face on the left and a wall behind it, then zooms back in step by step, pasting fine detail back in from the original. The second cuts the photo into a grid of tiles and lets every tile ask every other tile what it sees.

The first editor is a U-Net. The second is a transformer on patches. Both play the same part in every sampler of the last four chapters: take a noisy tensor, the noise level and the condition, and return a tensor of the same shape. That tensor is the noise guess ε̂ or the velocity v̂.

The U-Net: zoom out, zoom in, keep the details

Slide 29 describes it in three moves. The U-Net was first built for medical image segmentation (Ronneberger and colleagues, 2015), and Ho and colleagues adapted it for DDPM. Its three parts are:

A diffusion U-Net also receives two extra inputs, the noise level and the condition, and predicts a denoising quantity. The shape of the whole drawing is a letter U, which gives the network its name.

The transformer: cut into patches

The diffusion transformer, DiT (Peebles and Xie, 2023), uses the attention machine that Lecture 2 of the course and the DiT lesson cover. A transformer wants a list of vectors, not a grid. So slide 30 starts by patchifying: cut the noisy tensor into small square tiles and turn each tile into one vector. Call the list (u1, …, uN). Each ui is a continuous feature vector, and it carries its spatial position so the network knows where the tile sat.

Inside the network the tokens only meet through attention, and every token can attend to every other. In the DiT paper’s setting, a 256 × 256 picture is first squeezed to a 32 × 32 × 4 latent (Chapter 6 shows how). A patch size p of 2 cuts it into (32 / 2)² = 256 tiles, and each tile holds 2 × 2 × 4 = 16 numbers.

latent [32, 32, 4] noisy tensor from the sampler patchify p=2 [256, 16] 256 tiles of 2 × 2 × 4 numbers linear to d [256, d] one vector per tile (d is the model width) N blocks [256, d] attention, feed-forward, modulation linear back [256, 16] unpatchify [32, 32, 4] the noise guess or the velocity, same shape as the input

One DiT block

Slide 31 splits a block into three jobs. Self-attention connects the visual tiles across the candidate picture. Feed-forward layers transform each token on its own. Condition modulation supplies the noise level and class information throughout the network, and it uses a trick, adaptive layer normalization (adaLN). Layer normalization rescales a token’s numbers to a standard size. In adaLN, a small network turns the noise level and condition into a new scale and shift, so the same weights act differently at rung 900 than at rung 50.

The DiT paper uses a variant called adaLN-Zero: each block’s output is multiplied by a gate that starts at zero, so a fresh block does nothing at first. The picture still passes through unchanged, and training can then turn each block on gently.

Slide 32 closes the loop. The transformer returns (û1, …, ûN), and unpatchify reshapes it back into a tensor with the same spatial organisation as the one the sampler updates. Then comes the line to remember: the backbone and the generation space are separate choices. A U-Net or a transformer can model the same latent.

Two ways to denoise a latent

Choose a network, then press Send a tensor through. The U-Net shrinks and regrows the tensor with skips. The transformer cuts it into patches, and each token talks to all the others.

Illustrative drawing of ours. The latent size and patch sizes follow the DiT paper’s 256-pixel setting. The U-Net levels (32, 16, 8, 4 per side) are a typical shape, not one specific model.

Now count what patch size buys. At p = 8 there are 16 tokens, and attention compares 16 × 16 = 256 pairs. At p = 4 there are 64 tokens and 4,096 pairs. At p = 2 there are 256 tokens and 65,536 pairs. Halving the patch size quadruples the tokens and multiplies the pairs by 16.

The symptom of too big a patch is blocky or smoothed detail. The measurement is the numbers each token must hold: at p = 8 each token squeezes 8 × 8 × 4 = 256 numbers into one vector. The fix is a smaller patch, and the DiT paper reports that smaller patches, which cost more compute, give better quality. The price is the 16-fold jump in pairs. Chapter 8 shows that video makes the same sum much worse.

A U-Net and a DiT both map a noisy latent to a same-shaped tensor. What does a U-Net’s skip connection do?

Chapter 6

Where to Generate: Pixels, Latents or Features

Trade the size of the tensor the sampler updates against the detail the decoder must restore

You must send a sketch of a city to a friend across the sea, and the post charges by the page. You could mail a full-size painting each time, or send a small outline and let your friend paint it up from memory. The outline is cheap and the painting step is where detail comes back, or does not.

An image generator faces the same choice. Slide 33 puts it in one sentence: a denoiser can process raw pixels or an encoded representation. This chapter compares three answers the lecture gives. In each, the sampler of Chapters 1 to 4 does the same job on a different tensor.

Latent diffusion: generate the outline, paint once

Rombach and colleagues (2022) split the work in two. A perceptual autoencoder learns to compress a picture into a small grid of numbers and to restore it. The encoder Eφ makes the grid, z0 = Eφ(y0). The decoder Dψ paints it back, ỹ0 = Dψ(z0). The grid is the latent: a code that keeps what the decoder needs. Slide 34 says to train the reconstruction system first, then freeze it while learning the generator. The paper uses mild KL or vector-quantisation regularisation on the code, ideas from Lecture 7.

Slide 35 then runs diffusion on the latent. Training encodes an observed picture, corrupts its latent with Chapter 1’s mix zj = aj z0 + bj ε, and updates only the latent network, with the loss ‖εθ(zj, j) − ε‖². Generation starts from Gaussian latent noise, updates the latent again and again, and decodes once at the end: ŷ0 = Dψ(ẑ0). A U-Net or a transformer can model the same latent.

The saving is large. In the DiT setting, a 256 × 256 × 3 picture holds 196,608 numbers. Its latent, 32 × 32 × 4, holds 4,096. That is 48 times fewer numbers for every one of the sampler’s many steps. The lecture says it directly: compression reduces repeated spatial computation, but the decoder must reconstruct what the representation preserves.

What the outline keeps

Left: a toy picture with a smooth disk and a fine checker patch. Middle: the compressed grid. Right: the picture restored from it. Choose how much to shrink each side.

Toy, ours: a 64 × 64 picture. Block averaging stands in for the encoder, and smooth interpolation for a decoder. A trained decoder does better than interpolation, but it cannot restore detail the code never held. Latent channels are 4, as in the DiT paper.

Read the sliding scale. At 8 per side the picture shrinks from 12,288 numbers to 8 × 8 × 4 = 256, which is 48 times fewer. The smooth disk mostly survives, with an error of 0.09. The checker patch does not, with an error of 0.45, because its pattern changes every pixel and the code only holds one average per 8 × 8 block.

That gives the failure mode. Symptom: fine texture, small text and thin lines look smeared, even when the sampler is perfect. Measurement: the error of the restored picture on its own, before any sampling. Fix: a better autoencoder, or less compression. The generator can never be sharper than the decoder allows.

Representation autoencoders: borrow a smart encoder

The next idea, slides 36 and 37, is the representation autoencoder (RAE) of Zheng and colleagues (2025). Do not train an encoder at all. Take a frozen, pretrained encoder such as DINO, SigLIP or MAE, whose features already describe what is in a picture. Train only a decoder that paints a picture back from those features. Then learn the distribution of the features with diffusion or flow matching.

The features keep rich information without squeezing channels hard. The price: the generator must handle their high dimension and noise scale, and the decoder must put up with imperfect generated features, since the generator will never hit them exactly.

JiT: no tokenizer, just pixels

The third answer, on slide 38, drops the encoder and the decoder. JiT (“just image transformers”, Li and He, 2025) runs a transformer on noisy pixel patches. Its network predicts the clean picture, ŷ0 = netθ(rτ, τ, c). The paper argues that natural pictures lie on a low-dimensional surface while noise does not, so predicting clean data is the easier target. It uses patch sizes of 16 and 32 on ImageNet at 256 and 512 pixels.

To use the flow sampler of Chapter 3, convert the clean prediction into a velocity. From rτ = (1 − τ) n + τ y0, the implied noise is (rτ − τ ŷ0) / (1 − τ), so:

vθ(rτ, τ, c) = (ŷ0 − rτ) / (1 − τ),   τ < 1

Work it with τ = 0.5, r = 0.2 and a prediction ŷ0 = 0.9. The implied noise is (0.2 − 0.45) / 0.5 = −0.5, and v = 0.9 − (−0.5) = 1.4. The shortcut gives (0.9 − 0.2) / 0.5 = 1.4. The two agree. The paper trains this velocity with the flow-matching loss of Chapter 3.

The division by (1 − τ) is a warning sign. Near the end a small error in ŷ0 becomes a large error in v: at τ = 0.99, an error of 0.01 in the clean prediction becomes 0.01 / 0.01 = 1.0 in the velocity. This is why the slide says τ < 1. A sampler must stop one step short, or put a floor under (1 − τ).

Three spaces, one table

Generation spaceWhat the sampler updatesHow pixels appear
Pixels (JiT)A noisy image tensorThe final clean prediction is already an image
Compressed latents (LDM)An autoencoder latent tensorA trained decoder reconstructs pixels
Representation features (RAE)A pretrained encoder feature tensorA trained feature decoder reconstructs pixels

Slide 39 ends with the sentence this chapter exists for: the sampling method, guidance and backbone can be chosen separately. A complete generator combines all of those decisions with a conditioning interface, which Chapter 7 adds.

A latent diffusion model produces smeared fine text even though its sampler is accurate. Where should you look first?

Chapter 7

Text to Image: Patches Ask, Words Answer

Follow a prompt through cross-attention and the whole sampling loop

You tell a painter, “a tree on a hill under the sun.” She does not read the sentence once and put it away. Every few strokes she glances back at it. When she paints the top right corner she checks the word “sun”. When she shades the ground she checks “hill”. The sentence stays on the table for the whole job.

A text-to-image model works the same way. Slides 40 to 42 put Chapters 1 to 6 together with one new part: a way for each piece of the picture to glance at the prompt. That part is cross-attention.

Training with text

Training examples are pairs: an image and its description. Two frozen encoders turn them into tensors. The image encoder gives the latent z0 = E(y0), as in Chapter 6. A text encoder turns the description into a list of vectors, one per word piece: C = TextEncoder(description), with shape [L, dc] for L pieces. Then the flow recipe of Chapter 3 runs on the latent:

zτ = (1 − τ) n + τ z0  ·  L = E ‖ vθ(zτ, τ, C) − (z0 − n) ‖²

Slide 40 adds one detail: condition dropout also trains the null-text mode. A small fraction of the time the description is blanked out, so the same network can give the v∅ that Chapter 4’s guidance needs.

How picture features read the words

Self-attention, from Lecture 2 of this course and the DiT lesson, makes queries, keys and values from the same tokens. Cross-attention splits them. Slide 41 says it in one line: image features supply the queries, and text features supply the keys and values.

Q = U WQ
One query per picture patch. U holds the current image features, shape [N, d]. A query asks: what should I look for?
K = C WK
One key per word piece. C holds the encoded text, shape [L, dc]. A key says: this is what I am about.
V = C WV
One value per word piece: the content that gets passed on.
softmax( Q K⊤ / √dk )
For each patch, a set of weights over the words that adds up to 1. The √dk keeps the scores from growing with the key size dk.

The score table QK⊤ has shape [N, L]: every patch against every word. It is small, since L is a few dozen at most. Work one patch by hand with toy numbers. Let dk = 2, the patch query q = (2, 0.5), and three words with keys sun = (1, 0), hill = (0, 1) and sky = (−1, −1).

raw scores 2.0, 0.5, −2.5  →  divide by √2 = 1.414, 0.354, −1.768  →  exponentials 4.113, 1.424, 0.171  →  weights 0.721, 0.249, 0.030

Give the words the values 3, −1 and 0.5. The patch receives 0.721 × 3 + 0.249 × (−1) + 0.030 × 0.5 = 1.93. It is a blend that leans toward “sun”, and the network adds it back into the patch’s own features.

Which patches read which word

Left: the image candidate, a grid of 64 patches. Right: how much each patch attends to the word you pick. Drag the sampler progress from pure noise to a finished picture.

0.50

Toy, ours: 3-number features, four words, and keys that point in four different directions. A patch feature is τ times its true word’s direction plus (1 − τ) times a seeded random draw. Real models learn WQ, WK and WV. The attention formula is the lecture’s.

Read the device at five points. At τ = 0 every patch is random, so only 7 of 64 patches pick the right word, and the sun patches give “sun” just 12% of their attention. At 0.5 the count is 36 of 64. At 0.75 it is 62. At 1 all 64 pick the right word, and the sun patches give “sun” 88%. Even then the other three words still get a small share, because softmax never gives exactly zero.

Why do queries come from the image and keys from the text? The picture is the thing being edited, so it must do the asking. The words are the fixed reference, so they are what gets looked up. This also keeps the cost low. Self-attention over the patches builds an [N, N] table, and N is 256 in the DiT setting. Cross-attention builds [N, L], and L is the number of word pieces, often a few dozen. The text costs little to read, however many steps the sampler takes.

Slide the progress back to 0 and watch the right panel scatter. Then slide to 1 and watch it organise. This is why the text stays on the table: the sampler changes the image candidate at every step, so each step must read the prompt again against the newest features. At 1 each patch gives its own word the most weight.

The complete pipeline

Slide 42 shows three stages: initialise, generate, decode. Notice what it does not need: an image. Generation needs the prompt and noise only. The data flow, with shapes:

prompt "a tree on a hill under the sun" C = TextEncoder(prompt) [L, d_c] computed once z ~ N(0, I) [32, 32, 4] the starting latent for each step of the sampler (tau from 0 to 1): v_c = v_theta(z, tau, C) [32, 32, 4] reads C by cross-attention v_0 = v_theta(z, tau, null) [32, 32, 4] the dropout mode v = v_0 + w * (v_c - v_0) guidance, Chapter 4 z = z + dtau * v Euler step, Chapter 3 image = Decoder(z) [256, 256, 3] once, at the end

Count the cost for 30 steps with guidance: 60 network calls on a 4,096-number latent, plus 1 decode. The same job in pixel space would update 196,608 numbers per call.

The failure mode here is a prompt that is half ignored: an image with a hill but no sun. The symptom is a missing object. The measurement is the attention weight on that word, which sits near the uniform share 1 / L for every patch. The fix is a larger guidance scale (Chapter 4), a stronger text encoder, or training data whose descriptions name the object.

In cross-attention for text-to-image, where do the queries come from, and where do the keys and values come from?

Chapter 8

One Distribution for a Whole Clip

Corrupt all frames together, let them attend to each other, and keep the cost in check

A film editor does not cut one frame, put it down, and then hope the next one matches. She lays the whole strip out on the table, looks along it, and fixes frames that clash. A video model has to do the same. Slide 44 states the job: learn a distribution over a whole clip.

The object: a clip, not a frame

Write the clip as V = (y(1), …, y(T)): T frames, each one the kind of picture Chapters 1 to 7 handled. The model is pθ(V | C), where C is the condition, a text prompt or a reference image. One sample from this is one whole clip. The lecture’s one-line reason for the whole design: the model must capture relationships between frames.

Slide 45 sorts out two clocks that are easy to confuse. Video time t says which frame, left to right across the clip. Corruption level j says how noisy the whole clip is, from j = J (pure noise) down to j = 0 (clean). Picture a table: frames across the top, levels down the side. A sampling step moves down one row, and it moves every frame in that row at once. Frames are never denoised one after another.

The training objective

Chapter 1’s recipe carries over with no change except the shape. Slide 46 writes it: Vj = aj V0 + bj ε, with the loss ‖εθ(Vj, j, C) − ε‖². Three facts about the tensors matter.

QuantityShape or role
Clean video V0T × H × W × 3
Noise εIndependent entries, same tensor shape
Level jOne value, shared by the jointly corrupted clip

The noise is independent for every number, so no frame shares noise with another. The level is shared, so every frame is equally noisy at the same moment. The loss covers the full video tensor. The relationships between frames are not in the loss. The network has to discover them in order to score well.

Why joint beats frame by frame

Here is why pooling frames helps, in the toy world of the hero. A clip is the ball going left to right (path A) or right to left (path B). Given a noisy clip, ask how likely path A is. Each frame gives some evidence, and the evidence is independent from frame to frame, because the noise is. In log odds (the logarithm of the ratio of the chance of A to the chance of B), independent evidence simply adds up. So the whole clip knows more than any one frame.

Frames pool their evidence

Left: the noisy clip at rung j. Right: how likely the true path is, judged by each frame alone, and by the whole clip. Drag j down from noise toward the clean clip.

400

Toy, ours: 5 frames of 11 pixels, two possible clips, and the exact chance of each given the noisy clip. The noise draw is seeded. Frame 3 shows the ball in the middle on both paths, so it carries no evidence and stays at 50%.

Start at rung 500 with the true path left to right. The five frames say 70%, 41%, 50%, 74% and 41%. Two frames lean the wrong way, one is blank. The whole clip says 76%. At rung 400 the frames say 86%, 45%, 50%, 90% and 44%, and the whole clip says 97%. The clip is right even though two frames still believe the wrong path. This is exactly what the hero showed: a denoiser that pools frames keeps one story.

Frames that see each other

To pool, the frames must exchange information inside the network. Slide 47 shows the mask for this as a grid of ticks. At a fixed corruption level a frame can attend to earlier and later noisy frames: every cell is open. The network refines one candidate clip as a whole. This is bidirectional attention, unlike the left-to-right attention of a language model, where a word may only look back.

Open attention has a side effect. Attention alone does not know order: swap two frames and the score table just swaps rows. So slide 48 adds position. Each token ut,h,w carries its video time t, its row h and its column w. A noise-level embedding tells the network how corrupted the whole candidate is, as in Chapter 5. Access is open in both directions, and the order of the video is still kept.

The first video network, and a cheaper attention

The original video diffusion model (Ho and colleagues, 2022) is a U-Net, the Chapter 5 shape, with two kinds of work (slide 49). Spatial computation handles structure inside a frame, with the down and up sampling of a U-Net. Temporal computation exchanges information across frames and keeps the sequence dimension intact. Slide 50 names the two attentions. Spatial attention mixes positions inside one frame. Temporal attention mixes positions across frames at one spatial location. Stack them, spatial then temporal, and information spreads across both space and time.

The saving is the point of the design. Full space-time attention is one big table over all tokens. Factorising it into two small tables reduces the cost of the full space-time attention matrix.

Who can this token hear?

Columns are 5 frames. Each column holds 9 spatial tokens (a 3 by 3 patch grid). Tap a token to choose the listener, then choose a kind of attention.

Illustrative sizes: 5 latent frames, 9 spatial tokens each, so N = 45. Pair counts are exact for these sizes: full is N², spatial is T × S², temporal is S × T². The tokens are an idea of ours; the three costs match the lecture’s formulas.

Choose “Spatial, then temporal”. After two layers the listener hears all 45 tokens, the same reach as full attention. It paid 5 × 81 + 9 × 25 = 630 pairs instead of 45² = 2,025, which is 3.2 times fewer. Chapter 9 puts real clip sizes in, where the gap is far larger.

The failure mode shows up as flicker. The symptom: objects change colour or shape from frame to frame, or the ball jumps. The measurement: does the object keep its identity across frames, which Chapter 11 turns into a checklist. The fixes are the ones in this chapter: corrupt and denoise the clip as one object, and give frames a path to hear one another.

In joint video denoising, what is shared across all frames of a clip at one step, and what is independent?

Chapter 9

Compress, Patch, and Count the Tokens

Shrink a clip with a video autoencoder, cut it into space-time patches, and price the attention

You plan to move a house across town. You cannot carry it as it stands. You take it apart, pack the parts into boxes, label the boxes, and count them, because the number of boxes decides whether one truck is enough. A video model does the same with a clip. This chapter packs a clip into tokens and counts them.

Step one: shrink the clip in space and time

In Chapter 0 a 5-second clip held about 96 million numbers. No transformer can attend over that. Chapter 6 showed the fix for images: an autoencoder. Slide 51 gives the video version, the video autoencoder of the Wan model (Wan Team, 2025). The encoder maps the clip to a latent, Z0 = Eφ(V0), and the decoder restores it, Ṽ0 = Dψ(Z0).

The slide’s figure gives the shapes. A clip of shape [1 + T, H, W, 3] becomes a latent of shape [1 + T/4, H/8, W/8, C]. Read this as three savings. Space shrinks 8 times in each direction. Time shrinks 4 times. And the first frame is kept separate (the “1 +”), so a single image fits the same format. The lecture notes the key consequence: a latent time position can represent information from several video frames.

Count what that buys on a small case. A 17-frame clip becomes 1 + 16/4 = 5 latent time positions. Each position after the first stands in for 4 frames. If a hand moves fast across those 4 frames, the code must remember the whole path with one position’s worth of numbers, and the decoder must paint all 4 frames back from it. That is the same bargain as Chapter 6: fewer numbers for the sampler, more work for the decoder. The autoencoder is trained first, then frozen, while the generator learns.

Step two: cut the latent into space-time patches

As in Chapter 5, a transformer needs tokens. Slide 52 patchifies the latent Zj of shape T′ × H′ × W′ × d into N tokens. The patch now has three sizes, pt, ph and pw: how many latent frames, rows and columns one token covers. The token count is:

N
The number of tokens the transformer sees.
T′ H′ W′
The size of the latent grid: latent frames, rows and columns.
pt ph pw
How many latent positions one token covers, in time, height and width.

In code, patchify is one reshape. The numbers inside each token are the pt × ph × pw × C values of its patch.

python# z: [B, T', H', W', C]   latent from the video autoencoder
from einops import rearrange
tokens = rearrange(z, 'b (t pt) (h ph) (w pw) c -> b (t h w) (pt ph pw c)', pt=1, ph=2, pw=2)
# tokens: [B, N, pt*ph*pw*C]  ->  a linear layer lifts each token to the model width d

Step three: count the pairs

Now the lecture’s hard sum. Let S be the number of spatial tokens in one latent frame, so N = T′ S. Attention over all tokens compares every token with every other, which costs about T′² S² pairs. Factorised attention (Chapter 8) does spatial attention inside each frame, T′ × S², and temporal attention across frames at each position, S × T′². Slide 53 gives the totals. The symbol O( ) means “grows roughly like”:

ArchitectureApproximate attention work
Full space-time attentionO(T′² S²)
Factorized spatial and temporalO(T′ S² + S T′²)

Try real-sized clips. The device uses a Wan-style plan: 16 frames per second, the 4 × 8 × 8 compression from slide 51, and patches of 1 × 2 × 2 latent positions.

Count the tokens in a clip

Choose a picture size and a clip length. The bars use a log scale, with one gridline per factor of 10. Compare the full attention bar with the factorized one.

5

Illustrative plan of ours. The 4 × 8 × 8 compression and the “1 +” first frame are from the lecture’s figure. We assume 16 latent channels, 16 frames per second and a 1 × 2 × 2 patch. The counts are exact arithmetic for these assumptions.

Take the middle setting: 5 seconds, 480 × 832. The clip has 81 frames, so the latent has 1 + 80/4 = 21 time positions, 60 rows and 104 columns. A 2 × 2 patch makes S = 30 × 52 = 1,560 spatial tokens, so N = 21 × 1,560 = 32,760 tokens. Full attention needs 32,760² = 1.07 billion pairs. The factorised form needs 21 × 1,560² + 1,560 × 21² = 51.1 million + 0.69 million = 51.8 million pairs, which is 20.7 times fewer.

Now double the clip to 10 seconds. The tokens double to 63,960. Full attention pairs grow 3.8 times, to 4.09 billion. Factorised pairs only double, to 102 million. This is the meaning of the lecture’s line: more frames and higher resolution increase the token count and the cost of every sampling step, and compression and attention design decide which clip sizes are practical.

The failure mode is a clip that is too big to run. The symptom is an out-of-memory error, or a sampler that takes minutes per step. The measurement is N and the pair count above. The fixes, in the order they help most: compress more in the autoencoder, enlarge the patch, factorise the attention. Each has a price. More compression costs fine detail, since one latent position now stands for several frames. A bigger patch costs the same, as in Chapter 5. Factorised attention costs one layer of delay before distant tokens meet.

One block of the video transformer

Slide 54 shows Wan’s block. Three jobs repeat in a stack of N blocks. Video self-attention connects the noisy space-time tokens. Text cross-attention reads the prompt, exactly as Chapter 7 built it. Noise conditioning modulates the block with the generation coordinate, the adaLN idea of Chapter 5, driven by an MLP (a small stack of layers that turns the timestep into scale and shift numbers). Nothing here is new. A video DiT is the image DiT with a longer token list and a time-aware position signal.

A video latent is 21 × 60 × 104 and the patch is 1 × 2 × 2 latent positions. How many tokens does the transformer see?

Chapter 10

Train, Sample, and Condition a Video Model

Run the flow loop on a whole latent clip, then steer it with a prompt or a first frame

You have a camera, a script and a storyboard. The camera is the video autoencoder of Chapter 9. The script is the text prompt of Chapter 7. The storyboard is the first frame you may hand in. Now put them to work. This chapter trains the generator, runs it, and steers it with a prompt or a picture. It covers slides 55 to 59.

Training: one noise level for the whole clip

The recipe is Chapter 3’s flow matching, on the latent of a whole clip. Slide 55 lists three moves. Encode a training video and its condition: Z0 = Eφ(V0). Corrupt the entire clean latent by mixing it with noise: Zτ = (1 − τ) η + τ Z0. Predict the velocity tensor for the whole clip, with the loss:

L = E ‖ vθ(Zτ, τ, C) − (Z0 − η) ‖²

Here η is Gaussian noise with the shape of the complete video latent. The autoencoder stays fixed while this loss updates the transformer. That is all of the training. Work one number from it. Take one entry of the latent with η = −0.8, Z0 = 0.6 and τ = 0.25. The corrupted value is 0.75 × (−0.8) + 0.25 × 0.6 = −0.60 + 0.15 = −0.45. The target velocity is Z0 − η = 0.6 − (−0.8) = 1.4. If the network says 1.1, the squared error is (1.1 − 1.4)² = 0.09. The real loss averages this over every entry of the 21 × 60 × 104 × 16 latent.

pythondef video_flow_step(vae, text_enc, dit, video, prompt, p_drop=0.1):
    with torch.no_grad():                          # frozen, as in Chapter 9
        z0 = vae.encode(video)                   # [B, T', H', W', C]
        c  = text_enc(prompt)                     # [B, L, d_c]
    c = null_text() if torch.rand(()) < p_drop else c   # condition dropout, Chapter 4
    eta = torch.randn_like(z0)                   # independent entries, same shape as z0
    tau = torch.rand(z0.shape[0], 1, 1, 1, 1)    # ONE level per clip, shared by every frame
    z_tau = (1 - tau) * eta + tau * z0
    v = dit(z_tau, tau, c)                    # [B, T', H', W', C], same shape as z0
    return ((v - (z0 - eta)) ** 2).mean()

Sampling: evaluate the clip, update every frame

At sampling time (slide 56) start from Zτ=0 ~ N(0, I) with the shape of the whole latent. Each Euler step does two things. It evaluates the transformer on the current complete clip. Then it updates all the latent time positions together: Zτ+Δτ ≈ Zτ + Δτ vθ(Zτ, τ, C). At the end the decoder turns the final latent into frames: V̂ = Dψ(Zτ=1). Continue the number above: from −0.45 with v = 1.1 and Δτ = 0.25, the next value is −0.45 + 0.25 × 1.1 = −0.175.

Because every step touches every frame, a frame is never finished before its neighbours. Everything sharpens together. The toy in the next device runs exactly this loop, with the exact velocity field of Chapter 3 extended to a whole clip.

Steering with text

For text-to-video (slide 57), the prompt becomes C = TextEncoder(prompt), and the model draws from pθ(V | C). The lecture lists what a prompt can specify: scene and appearance (subjects, objects, environment, style), events and motion (what should happen), and camera information (viewpoint and intended camera movement). Then it adds one sentence to keep in mind: text conditions a learned distribution; it does not certify the dynamics.

Steering with a first picture

For image-to-video (slide 58), the inputs are a reference image and optional text. The reference image goes in as image features plus a visual-condition mask, which tells the network which latent positions are given. The output is still a complete candidate video, with the latents jointly refined. The supplied first image allows joint generation of the remaining frames: the generator does not paste your frame and then guess forward. It makes the whole clip, with your frame pinned.

Slide 59 draws the whole machine in one picture. During training, encode observed videos and regress known targets, with the VAE and text encoder frozen. During sampling, initialise latent noise, evaluate the transformer repeatedly, and decode at the end. A request and its result, in our own illustrative form, show the data flow:

// our illustration of what this pipeline needs, not a specific product's API { "prompt": "a red ball rolls left to right across a table", "image": null, // or a reference image for image-to-video "num_frames": 81, "height": 480, "width": 832, "steps": 40, "guidance_scale": 5.0, "seed": 7 } → latent noise [21, 60, 104, 16] from the seed → 40 steps × 2 calls (with and without the prompt) on 32,760 tokens → decoded video [81, 480, 832, 3]

Steer the clip

Pick a condition, then press Make 12 clips. Green means the clip kept one path and followed the condition. Red means it jumped, or went the other way.

Toy, ours: the clip world of the hero and the exact flow field, with 20 Euler steps over the whole clip. The text-conditioned model is one trained so that 90% of its clips go left to right, since text conditions a learned distribution. With a first frame, the model is told which path starts there.

Run it with no condition and about half the clips go each way, and every clip keeps one path. Switch to the text prompt: most clips go left to right, but a few still go the other way, because the model learned a 90% preference, not a rule. Switch to the first frame: every clip starts with the ball at the left, and the remaining four frames follow the one path that fits it. A picture is a much stronger condition than a sentence, because it leaves less room.

The failure mode is the lecture’s own warning. The symptom: a clip that matches the prompt in words and looks wrong in motion, such as a ball that slows for no reason. The measurement is a motion check on the clip itself, which Chapter 11 sets out. The fix is not a higher guidance scale. Guidance strengthens a learned preference, and the preference itself may be wrong about the physics.

During training of the video flow model, how many noise levels τ does one clip get, and why?

Chapter 11

The Field Guide

Review the lecture on one page: the numbers, a checklist, the core code, how to judge a clip, and what comes next

You can now say, from memory, what Lecture 12 argued. A diffusion or flow model learns to remove noise one small step at a time. Many samplers share one trained network, so DDIM and Euler steps cut the cost. Guidance pushes samples toward a condition. The network can be a U-Net or a transformer on patches, and it can work in pixels, latents or features. A video model is the same machine run on one tensor with a time axis, so every frame is refined together.

The numbers to keep

IdeaWhat it saysThe number
Noise ladder (Ch 1)Rung j mixes the clean picture and noise with weights aj and bj.1,000 rungs; a = 0.280, b = 0.960 at rung 500
DDIM (Ch 2)Same noise levels, a new path: retained noise plus fresh noise σ.σ = 0 is deterministic; 10 steps landed 15 of 16 draws
Flow matching (Ch 3)Regress the straight-line velocity y0 − n; Euler-step the learned field.1 step lands at the mean; 8 steps hit 16 of 16
Guidance (Ch 4)ε∅ + w(εc − ε∅): two calls per step.w = 1 ordinary; spread 1.01 at w = 1, 0.28 at w = 3
DiT tokens (Ch 5)Cut the latent into patches; attention compares every pair.32 × 32 × 4, patch 2: 256 tokens, 65,536 pairs
Latent space (Ch 6)Generate in a code, decode once.196,608 numbers to 4,096: 48 times fewer
Video autoencoder (Ch 9)Shrink space and time before the transformer.4 × 8 × 8; 81 frames to 21 positions
Space-time attention (Ch 9)Factorise to avoid the full pair table.1.07 billion pairs to 51.8 million

Judging a generated clip (slide 60)

How do you know a clip is good? Slide 60 gives four questions, and each needs its own look. Do frames look plausible? Inspect visual detail and artifacts. Does the clip form a coherent event? Inspect identity, motion and persistence. Does it follow the condition? Inspect adherence to the prompt and the reference image. Do the dynamics make sense? Inspect contacts, occlusion and object-state changes. The slide’s sentence for it: one perceptual score cannot answer every question about the video.

The toy shows why. Three generators, all with sharp frames, are scored four ways, plus one score that only looks at frames.

One score cannot judge a clip

Pick a generator. Each makes 100 clips from the prompt “ball moves left to right”. Compare the four honest checks with the single frame-only score at the bottom.

Toy, ours: the clip world of the hero, 100 clips per generator, seeded, 20 Euler steps. A frame is plausible if it is within 0.35 root-mean-square of a valid frame. The 90% text preference is a toy choice. Real evaluation uses people and learned judges; the four questions are the lecture’s.

Read the three runs in turn. Frame by frame has perfect frames, but only 74% of clips are one event and 74% follow the prompt, and 93% of motion steps are steady. The whole clip passes all four, with 97% following the prompt. The third generator keeps every frame, event and step perfect, and follows the prompt in 52% of clips. Every generator scores 100% on the frame-only line, and so that one number would rank them all equal.

A checklist for any video generator

The recipe on one slide (slide 61)

ComponentRoleChapter
Video encoderRepresent observed clips in a latent space9
Space-time transformerPredict updates using the complete noisy candidate8, 9
ConditioningSupply text or observed visual evidence4, 7, 10
Diffusion or flow samplerRefine the complete latent clip1, 2, 3
Video decoderRecover pixel frames from the final latent6, 9

The core, as code

This sketch gathers the sampler of the lecture: a guided Euler loop over a whole latent clip, and the DDIM step it replaces when you use a noise-prediction network. vae, text_enc and dit stand for the trained parts of Chapters 5 to 10.

python# 1. DDIM step, Ch 2: z [B,...], eps = network(z, j, c). sigma = 0 is deterministic.
def ddim_step(z, eps, a_j, b_j, a_k, b_k, sigma=0.0):
    z0 = (z - b_j * eps) / a_j                         # clean estimate
    return a_k * z0 + (b_k**2 - sigma**2).sqrt() * eps + sigma * torch.randn_like(z)

# 2. Sample a whole clip, Chs 3, 4, 10: one latent, one level, every frame together
@torch.no_grad()
def sample_video(vae, text_enc, dit, prompt, shape, steps=40, w=5.0, image=None):
    c, c0 = text_enc(prompt), null_text()
    z = torch.randn(shape)                           # [B, T', H', W', C] = [1, 21, 60, 104, 16]
    for i in range(steps):
        tau, dtau = i / steps, 1 / steps
        v_c = dit(z, tau, c, image)                     # sees all N = 32,760 tokens
        v_0 = dit(z, tau, c0, image)                    # same call, no condition
        v = v_0 + w * (v_c - v_0)                      # classifier-free guidance
        z = z + dtau * v                               # Euler step
    return vae.decode(z)                             # [1, 81, 480, 832, 3], decoded once

Where this sits, and what comes next

The closing slide asks the next question: how can this generator predict futures conditioned on experience and interaction? A text prompt or one first frame says what a clip should be. A world model must go further and say what happens next after an action. Lecture 13 takes that step, with video world models and fast sampling. The images and sampling tools of this lecture also build on Lecture 11, on Dreamer, diffusion and flow matching, and on the latent-variable ideas of Lecture 7.

Keep going

Course credit. This lesson re-teaches Lecture 12 of Penn CIS 6280 World Models (Fall 2026), taught by Jiatao Gu. The slides are linked at the top of this page and in the footer. The lecture’s sources are Ho and colleagues (DDPM 2020 and Video Diffusion Models 2022), Song and colleagues (DDIM), Lipman and colleagues (Flow Matching), Dhariwal and Nichol, Ho and Salimans, Rombach and colleagues, Peebles and Xie, Zheng and colleagues, Li and He, and the Wan and Sora reports. The devices and toy worlds are ours, and each one says so in its fine print.

The takeaway. A video model is not a pile of image models. It draws one clip, with one noise level shared by every frame, and a network that lets every frame see the others. Then check the clip on four questions, because no single score will.
A video generator scores very well on a frame-quality metric, yet viewers say the clips look wrong. Which statement is the best next step?

Now press Teach and explain, from memory, how DDIM and flow matching sample, what guidance does, why a video latent has 32,760 tokens, and why a clip needs joint denoising. Then go back to the hero and make twelve clips both ways again.

CIS 6280 Lecture 12: Video Diffusion Models
The course site · Lecture slides · Back to Gleams