Video World Models and Fast Sampling

A video model refines a whole clip with dozens of network passes, which is too slow for a world you steer. One learned jump can reach the same clip in a single pass.

Learn how one network makes a coherent clip, how it predicts what happens after your action, and how a few-step sampler replaces fifty slow steps with a handful of jumps.

Pick a sampler, then drag the number of network passes down to 1. Watch where the 150 dots land. Then we build, piece by piece, everything underneath: the clip as one sample, the attention that reads it, the chunks that make it interactive, and the jumps that make it fast.

You need a rough idea of what a neural network is (a function that learns from examples) and a little calculus (a slope is a rate of change). We build the rest from zero.

Based on Lecture 13 of Penn CIS 6280: World Models (Fall 2026), taught by Jiatao Gu. Lecture slides. Every idea is re-taught here in our own words and devices; it follows Lecture 12, Video Diffusion Models.

Land the samples

2

A toy world, not a trained video model: the data are three points on a circle and the true velocity field is computed exactly from them. The average-velocity sampler stands in for a trained flow-map network: it follows the exact path, plus a small random miss that grows with the length of each jump (illustrative: 0.3 per unit of time). A dot is green when it lands within 0.45 of a data point. Noise is redrawn with the button; the dots are the same 150 for both samplers.

Chapter 0

A Clip Is One Sample

See why a video must be drawn as one object, and how the image generator you already know grows into a clip generator

Ask an image model for a woman in a red coat, walking down a street full of neon signs. It answers with a fine picture. Ask five times and you get five fine pictures. Now play them in order, one after the other, as a film.

The coat changes shade in every frame. A sign flips from left to right. The street bends. Each frame is good, and the clip is broken. Nothing in the five draws knew about the other four.

That is the problem this lecture starts from. Lecture 13 of CIS 6280 has two halves. The first half builds a video generator and turns it into a world model, a model that predicts what you will see next after you act. The second half makes any such generator fast, because a world you steer must answer in a blink.

One short note on the course first. The lecture ran on Tuesday 20 October 2026. Assignment 3 moved to Friday 30 October. The final project instructions live on Canvas. Office hours are on Thursday from 4:30 to 5:30 PM in AGH 423. That is all the logistics there are.

The image generator we already have

The lecture begins with a model from earlier weeks, so here it is again from zero. A flow-matching image model learns to turn random noise into a picture by moving along a path. You need three small ideas.

First, make a blend. Take a clean image y and a sheet of random noise n, the same shape as the image, with every number drawn from a bell curve. A dial τ (tau, the generation coordinate) mixes them: at τ = 0 you hold pure noise, at τ = 1 you hold the clean image, and in between you hold a blend.

rτ
The blend at dial position τ. It is the noisy picture the network gets to see.
(1 − τ)
The share of noise. It starts at 1 and falls to 0.
n
Gaussian noise, drawn fresh each time. “Gaussian” means a bell-curve draw for every number.
τ
The share of clean image. It rises from 0 to 1.
y
A clean training image.

Second, give the network a job. A velocity is the direction and speed that carries a point along the path. Along this blend the velocity is constant: it equals y − n, since rτ moves from n to y in one unit of τ. The network vθ sees the blend rτ, the dial τ, and a condition C (a text prompt or other evidence), and it must output y − n. The loss is the squared gap between its guess and that target, averaged over many examples.

Here is one pixel as a worked example. Take y = 0.8, n = −0.5 and τ = 0.25. The blend is 0.75 × (−0.5) + 0.25 × 0.8 = −0.375 + 0.2 = −0.175. The target velocity is 0.8 − (−0.5) = 1.3. The network sees −0.175 at τ = 0.25 and should answer 1.3. If it answers 1.0, the loss for this pixel is (1.0 − 1.3)2 = 0.09.

Third, generate. Start at pure noise (τ = 0). Ask the network for a velocity. Move a small step along it. Repeat until τ = 1. Every pass through the network is one network evaluation. Fifty small steps cost fifty evaluations. Keep that count in mind: the whole second half of this lecture is about shrinking it.

The slide then draws the full text-to-image system as three boxes. A text encoder turns the prompt into features C. The generator refines a noisy image, or a noisy latent (a small coded version of the image, built below), using C and guidance, a dial that pushes each step harder toward the prompt. An image output step decodes a latent into pixels, or returns the pixels directly. The slide adds one warning. The text conditioning, the space you generate in, the network architecture and the sampler are four separate choices. We will change them one at a time.

From one image to one clip

Now ask the same machine for a video. A video V is a list of T frames, written (y(1), …, y(T)). The lecture asks for one distribution over the whole list, pθ(V | C): given the prompt, a probability for every possible clip. We want one draw from it. The slide states the cost of doing otherwise in one line: sampling independent images discards the relationships between frames.

Test that claim in a toy. A ball drifts across eight frames. Its start and speed vary from world to world. The first strip draws each frame by itself, from the same spread of possibilities. The second strip draws one start and one speed and keeps them for all eight frames. Slide the spread up and watch the trace.

Draw frames alone, or draw one clip

Each box is one frame, and the dot is a ball. The line joins the ball from frame to frame. Raise the spread of the world, then press Draw again a few times.

0.80

Illustrative toy. The ball’s start and speed are random; with the spread at 0 the world has one clip and both strips agree. “Jump” is the distance the ball moves between neighbouring frames, as a share of the frame width, averaged over the seven gaps. Redrawn values come from a seeded random stream in your browser.

The independent strip zig-zags, because each frame picked its own start and speed. The joint strip is a straight line, because one draw fixed them for every frame. Every frame alone is plausible. Only the joint draw is a clip. A joint sample is one draw of the whole object, so the pieces agree with each other.

Two clocks

A video generator has two clocks, and mixing them up is the commonest mistake. Video time says where a frame sits in the clip: frame 1, frame 2, frame 3. Generation time is the dial τ: how far the whole clip has travelled from noise to a finished video. The slide draws them as a table. Across a row is video position. Down the table is τ.

The key sentence is short: a joint sampling step updates every video position together. The network does not finish frame 1 and then start frame 2. All frames stand at the same τ, and one evaluation moves them all one step closer to clean.

Run generation time

Five frames of a bright blob moving right. Drag τ or press Play. Every frame clears at the same time, because one step updates them all.

0.40

Illustrative toy: five frames of 9 × 9 pixels, a 2 × 2 blob that moves 1.6 pixels per frame, and fixed Gaussian noise. Each pixel is shown as (1 − τ) × noise + τ × clip, exactly the blend above.

A smaller clip to work on

A clip is large. Sixteen frames of 256 × 256 colour pixels hold 16 × 256 × 256 × 3 = 3.1 million numbers, and the network must process that tensor (a grid of numbers with several axes) at every step. The lecture’s answer comes from the Wan system: a video autoencoder, a pair of networks that squeeze a clip into a small code and expand it back. The encoder E makes Zdata = E(V). The decoder D returns V̂ = D(Zdata), close to V.

Wan’s figure shows the squeeze: a clip of shape [1 + T, H, W, 3] becomes a code of shape [1 + T/4, H/8, W/8, C]. Time shrinks by 4, height and width by 8 each, and the colour channels become C feature channels. Take T = 16 and H = W = 256 as a worked example. The input holds 17 × 256 × 256 × 3 = 3,342,336 numbers. With an illustrative C = 16, the code holds 5 × 32 × 32 × 16 = 81,920 numbers, about 41 times fewer.

The recipe has two stages. Train the autoencoder to reconstruct clips, then freeze it. Only then learn the generative model, on the codes. The slide’s remark about time is the one to keep: temporal compression shrinks the tensor processed at every sampling step. A smaller tensor saves work at every one of the fifty steps, so the saving adds up.

The generator now works on space-time tokens. Its tensor Zτ has shape Tz × Hz × Wz × dz: latent time, height, width, and channels. The transformer receives three things. Patch embeddings carry continuous features from the noisy latent. Space-time coordinates say where each token sits in the clip. The dial τ says how far generation has come. Without the coordinates the model could not tell frame 2 from frame 9.

The road

The lecture walks in a clear order, and so does this lesson: build the clip generator, make it predict futures you steer, then make every part of it fast.

  1. How attention reads a clip: spatial, temporal and joint patterns, and what each costs.
  2. Training and generating a whole clip: the loss, the sampler, an image to start from.
  3. What the frontier systems publish: Seedance, MiniMax H3, and what is public.
  4. From clips to interactive prediction: history, actions, chunks and two loops.
  5. Which history training sees: teacher forcing, diffusion forcing, and the rollout gap.
  6. Why sampling is slow: evaluations times cost, and the large step that misses.
  7. Teaching a student to jump: progressive distillation, reflow and flow maps.
  8. BOOT and π-Flow: distilling from a teacher’s local advice.
  9. Consistency models: one endpoint from anywhere on a path.
  10. MeanFlow: average velocity, its identity, and its training target.
  11. The Field Guide: the numbers, the code, and distribution matching next.
You draw eight frames from the same image model, one after another, with the same prompt. Why do they not play as a smooth clip?

Chapter 1

How Attention Reads a Clip

Choose which tokens may read which, and count what each choice costs

Picture one wheel of a car, a patch of pixels in frame 3. In frame 4 the car has rolled forward, so the wheel sits a few patches to the right. To draw frame 3 and frame 4 as the same wheel, the network must let the wheel in frame 4 look at the wheel in frame 3, even though they sit in different places.

Reading other tokens is the whole job of attention. Here it is from zero. Each token (one patch of the clip, as Chapter 0 built them) writes three short lists of numbers. A query says what this token wants to know. A key says what this token is about. A value says what this token will hand over. A token compares its query with every key it is allowed to read, turns the match scores into weights that sum to 1, and takes a weighted mix of the values.

Two numbers matter for cost. Every query meets every allowed key, so the work is the count of query and key pairs. And the answer for each token is a mix, so the output has the same shape as the input: one new feature list per token.

One block, three jobs

The lecture opens the video transformer using Wan’s block diagram. Each block does three jobs, and each job is a form of reading. Video self-attention connects locations within the candidate clip: tokens read other tokens of the same noisy clip. Text cross-attention reads the supplied prompt: the queries come from the video tokens, and the keys and values come from the text features C, so the clip asks the prompt what to draw. Time modulation injects the generation coordinate τ into the blocks, a bit like a dial that tells every layer how noisy its input still is.

The slide ends with a data-flow fact worth keeping. One network evaluation predicts an update tensor for the whole candidate: the input is the whole noisy latent clip, the output is a velocity of the same shape. Nothing is predicted frame by frame.

Three patterns of looking

The lecture draws three patterns on a grid with time down and space across. In every one, Q marks one query token and the coloured cells are the keys it may read.

Spatial attention lets a token read the other locations in its own frame. It sees the whole street but only this instant. Temporal attention lets a token read the same location across frames. It sees how one patch changes over time but nothing beside it. Full space-time (joint) attention lets a token read every location in every frame.

The slide adds a warning that explains why this choice matters: a moving object can occupy different spatial locations in successive frames. Temporal attention at a fixed location sees the wheel in frame 3 and a patch of road in frame 4. It cannot follow the wheel on its own.

The price of each pattern

Let the latent clip have L temporal positions and P spatial tokens per position, so there are N = LP tokens in all. Count the pairs in each pattern. Spatial attention: each of L frames holds P tokens that read P tokens, so L × P2 pairs. Temporal attention: each of P locations holds L tokens that read L tokens, so P × L2 pairs. Full attention: N tokens read N tokens, so (LP)2 = L2P2 pairs.

L P2
Spatial: every frame, every pair of locations inside it.
P L2
Temporal: every location, every pair of frames at it.
L2P2 = N2
Full space-time: every pair of tokens in the clip.

Here is a worked example with illustrative sizes. Take L = 20 latent frames and P = 256 tokens per frame (a 16 × 16 grid), so N = 5,120. Spatial: 20 × 2562 = 1,310,720 pairs. Temporal: 256 × 202 = 102,400 pairs. Doing both in turn costs 1,413,120. Full attention: 5,1202 = 26,214,400 pairs, about 18.5 times more.

That saving is why many video models factorize attention: they alternate a spatial layer and a temporal layer. Two layers in a row also reach everywhere, because a token can hop within its frame and then across frames. The slide states the trade in one line: alternating spatial and temporal attention factorizes the computation, while joint attention connects space and time in one operation. Try both below.

Pick a query, pick a pattern

Each column is a frame and each row is a location. Tap any cell to choose the query (teal). Pink cells are the tokens it may read. Then change the clip size and compare the bars.

6
6

Illustrative sizes, exact counts. Pair counts follow the slide: spatial L P2, temporal P L2, joint L2P2. “Spatial, then temporal” shows two layers in a row: dark pink is what the first layer reads, light pink what the second layer adds. Cells show permitted access, not learned attention weights.

Look at what the joint pattern shows for any query: every cell is pink. In the lecture’s five-frame table, query 1 and query 5 both read all five keys. This is the slide on temporal access. At one generation coordinate τ, a candidate frame may use earlier and later candidate frames. Nothing is hidden, because none of the frames is final yet. Chapter 4 will add a mask that hides the future, and the slide is careful to say that masking is a separate choice from factorizing.

A failure and its fix

The symptom: a model with spatial layers only draws sharp frames, and the wheel changes colour from frame to frame. The measurement: in the device, set the pattern to Spatial and count the keys from other frames that the query can read. The count is zero, so no information moves in time. The fix is any pattern that reaches across frames: temporal layers, or joint attention.

The second failure is cost. In the device, double L with the joint pattern: the bar for joint attention grows 4 times, while the spatial bar grows 2 times. The slide closes with the remedy: compression and sparse attention make longer or higher-resolution clips more affordable. The temporal squeeze from Chapter 0 shrinks L. Sparse attention skips pairs that matter little. The lecture notes that the MiniMax H3 release documents full-attention inference, with native sparse attention not part of the initial release.

A clip has L = 20 frames and P = 256 tokens per frame. You double it to 40 frames. How do the pair counts of spatial attention and full space-time attention change?

Chapter 2

Training and Generating a Whole Clip

Follow one training example and one generation run, tensor by tensor, from caption to decoded video

Imagine you teach a student to restore damaged film. You take a clean reel, damage it by a known amount, and ask: which way must each frame move to be repaired? The student answers, you compare with the truth, and you repeat a few million times. That is the whole training recipe of a flow-matching video model.

The lecture writes it in four lines, and each line is a data-flow step. Here they are with their tensor shapes, in the order a training script runs them.

  1. Encode. A clip V goes through the frozen encoder: Zdata = E(V). The shape is Tz × Hz × Wz × dz.
  2. Corrupt. Draw noise η of exactly that shape, and one dial position τ between 0 and 1. Blend: Zτ = (1 − τ)η + τ Zdata.
  3. Condition. Encode the paired caption into features C. They go in through cross-attention.
  4. Predict and score. The network gives vθ(Zτ, τ, C), a tensor of the same shape. The target is Zdata − η. The loss is the mean squared gap.
L
The training loss. E means “average over many random examples, noises and τ values”.
vθ(Zτ, τ, C)
The network’s answer: a velocity for every cell of the latent clip, given the noisy clip, the dial and the prompt.
Zdata − η
The true velocity of the straight path from noise to clip. It is the same for every τ on that path.

Work one cell by hand. Take the cell at frame 2, channel 1, with Zdata = 1.2 and η = −0.8, and let τ = 0.4. The network sees Zτ = 0.6 × (−0.8) + 0.4 × 1.2 = −0.48 + 0.48 = 0.0. The target is 1.2 − (−0.8) = 2.0. If the network answers 1.5, this cell adds (1.5 − 2.0)2 = 0.25 to the loss. The real loss averages every cell of every frame.

Notice one thing that is easy to miss: a single η and a single τ cover the whole clip. All frames stand at the same point on the dial. That is what makes this a joint generator. Chapter 5 gives each frame its own dial position, and then the picture changes.

Build one training example

Four latent clips, 4 frames (rows) by 4 channels (columns). Filled cells are positive and hollow cells are negative; the shade is the size. Drag τ, then tap any cell to see its arithmetic.

0.40

Illustrative toy: a 4 × 4 latent clip with smooth made-up values, seeded Gaussian noise, and exact arithmetic. With “First frame given” on, frame 1 is held clean and is left out of the loss in this toy, as a stand-in for a supplied reference; the lecture does not specify which part of a real system’s loss a reference removes.

Generating the complete clip

At generation time there is no clean clip. You draw Zτ=0 from pure Gaussian noise and walk the dial from 0 to 1, in steps of Δτ. At each step, one network evaluation gives a velocity for the whole latent clip, and the step is Zτ+Δτ ≈ Zτ + Δτ × vguidedθ(Zτ, τ, C). At τ = 1 the decoder turns the finished latent into frames: V̂ = D(Zτ=1).

The slide writes the velocity as “guided”. Guidance is a dial for how hard each step follows the prompt. A common form, called classifier-free guidance, runs the network with and without the prompt and pushes the step along the difference. It costs two evaluations per step in that form. The slide only names the guided result, so treat the two-run recipe as one standard way to get it.

The slide ends on the sentence that opens Chapter 6: repeated network evaluations refine the full latent clip, and decoding comes afterward. With 50 steps and guidance in its two-run form, one clip needs 100 passes through a large transformer. Remember that number.

Starting from a picture

Often you do not want a free clip. You hold a photograph and want it to move. This is image-to-video generation. Wan’s figure shows the extra inputs: image features, and a masked visual condition that rides beside the noisy video latent. The mask marks which frames are given and which must be made up.

The lecture states the effect in two sentences. A supplied reference narrows appearance and layout, so the street, the coat and the light are settled. The model still generates the possible subsequent motion, and the remaining frames are still generated jointly. The reference removes uncertainty about what things look like, not about what they do next. That second question, what happens next, is exactly what Chapter 4 turns into a world model.

The whole pipeline

Slide 17 puts both halves on one diagram. Learning needs the observed-video encoder: it encodes real clips, and the network learns a conditional field from them. Videos supply the temporal evidence. Generation never touches a real clip. It draws noise at the Gaussian inlet, refines a latent clip, and decodes. The sampler decides how many network evaluations that takes. Training teaches, and sampling spends. Everything in the second half of this lecture attacks the spending.

At τ = 0.25, a cell has noise η = −1.0 and clean value Zdata = 2.0. What are the network’s input value and the velocity target for that cell?

Chapter 3

What the Frontier Systems Publish

Read two production video systems the way an engineer does: what is in the architecture, what is in the data, and what is only claimed

You watch a six-second clip of a figure skater. She spins, the camera swings round her, and the film cuts to a close shot of her face. Her coat, her hair and the rink all stay the same through the spin, the swing and the cut. It looks easy. It is the hardest thing a video model does, and the lecture uses it to open its tour of two production systems.

The first is ByteDance’s Seedance. The lecture shows an official Seedance 2.0 text-to-video example, the skater, and states the demand in one line: a rotating subject, a moving camera and changes of shot all have to remain coherent. A second official example shows generation from references: instead of only a prompt, the condition can specify a storyboard, a character, a scene and props. You hand the model the cast and the set, and it films them.

Three things, three different sources

When you see a stunning demo, ask which of three things made it. Architecture decides what can interact: which tokens may read which. Data decides what the interaction can learn: what motion and what sounds the model ever saw. Training decides how the result is polished. The lecture says it in two sentences: architecture determines what can interact; data and training determine what the interaction learns.

The slide on the published Seedance recipe sorts the facts into three rows, and it is careful about its source. The recipe is documented in the Seedance 1.5 Pro technical report. The cited Seedance 2.5 product page does not specify a full training recipe. So the rows describe 1.5 Pro and say nothing about 2.5.

Treat that slide as a model for your own note-taking. Every claim about a system gets a source and a date. A sentence like “the model was trained with human feedback” is only as good as the report it came from, and a newer version may differ.

MiniMax H3: one sequence for everything

The second system, MiniMax H3, is described from its official release, checked by the lecture on 19 October 2026. Its picture is simple, and it shows exactly the idea of a shared backbone. Two kinds of tokens enter one sequence.

First come the condition tokens, built from the given references: the meaning of the text and of any visual input, VAE latents of reference images or video (a VAE, a variational autoencoder, is the kind of learned squeeze into a small code that Chapter 0 described), and VAE latents of reference audio. These are clean. They are evidence, not something to generate.

Second come the output candidates: video tokens and audio tokens that start as new Gaussian noise, not as the reference inputs. These are what the network refines. The two groups are joined into one combined token sequence. That sequence goes to a 33B dense single-stream Transformer. It has 33 billion weights, and dense means all of them work on every token, unlike a mixture that wakes only some. Single-stream means every token sits in the same sequence and shares attention and feed-forward layers.

The slide notes what stays separate: modality-specific inputs, outputs and modulation. Video and audio each get their own way in and out, and their own dial-injection, but they share the thinking in the middle. At the end, a video VAE decoder returns the video and an audio VAE decoder returns stereo sound.

Why share one stream? Because attention connects every token to every other. A burst of sound at frame 40 can read the video token showing a door slam. The common token sequence lets reference content, video and audio all inform the generation.

Assemble the shared sequence

Switch references on and off. Blue tokens are given and clean. Purple tokens start as noise and are generated. The bottom strip shows which stage of the system you can run yourself.

The structure is the lecture’s (MiniMax H3 official release, checked 19 October 2026): condition tokens plus noisy output tokens in one sequence, a 33B dense single-stream Transformer, a video VAE decoder and an audio VAE decoder. The token counts are illustrative, chosen for easy arithmetic; the release does not give them in the lecture.

The model you can download is not the product you can use

The last slide splits H3 into three stages and marks which are released. Context-IR interprets and organizes multimodal instructions: it turns a messy request into clean conditions. It is hosted. H3-Base generates 768p video and stereo audio. Its weights are released. Regenerate-2K refines the result using the original context. It is hosted. The release documentation describes full-attention inference; native sparse attention is not part of the initial release.

The slide draws a firm conclusion: a released base model and a hosted product can differ in their conditioning and refinement stages. Try the switch in the device. If you run the weights, you run the middle box only. The clean prompts and the polishing pass belong to the hosted service. A demo from the product shows all three, so it is not evidence about the weights alone.

Here is the failure mode, as symptom, measurement and fix. The symptom: you download the base model and your clips look worse than the launch video. The measurement: list the stages that made the launch video, and tick the ones you actually run. The fix: add your own conditioning step and your own refinement pass, or compare against the base model’s own outputs.

You read that a lab’s product “uses human-preference rewards”, but the only cited source is an older model’s report. What does the lecture’s habit say to write down?

Chapter 4

From Clips to Interactive Prediction

Turn a clip generator into a world you can steer, one chunk at a time, with two loops and a cache

You play a driving game, and every frame on your screen is drawn by a neural network. You press left. The car must begin to turn within a moment, and the road must stay the same road. A film cannot do this. A film plays the same future every time. A world model draws the future that follows from what has happened and from what you do.

The lecture opens the second half with a careful claim. Video learning gives us two things: a representation of how things look and change, and a distribution over possible visual events. Interactive prediction asks two further questions. What happens from this observed history? And how does that future depend on the interaction we supply? The lecture adds that the predicted world can live in observations, in pixels or latents, without an explicit physical state vector. No one writes down positions and velocities. If you want the long view of world models, see our Lecture 1 overview and the world models lesson. For a famous interactive example, see Genie.

The question, written down

Give each moment a name. The history ht is everything known up to time t: the observed frames o≤t and the earlier actions a<t. The world model then answers one question over a horizon of H steps: given this history and these candidate actions, what is the distribution of the next H observations?

ht
The evidence: observations so far, and the actions already taken.
pθ(…)
A distribution over the next H frames, given the history and the actions we are considering. We can draw samples from it. It does not name one future.

The slide ends with a sober line: action conditioning requires suitable data and tests of the predicted consequences. A model that looks good is not yet a model that obeys. You must have recordings that pair actions with what followed, and you must check that pressing left makes the car go left.

Two ways to ask

There are two ways to use such a model, and the lecture sets them side by side. In a joint future block, you condition on the known history and a whole supplied action sequence, and the model draws the entire unknown future at once. It may use bidirectional attention inside that block, so a late frame can inform an early one, and it may regenerate the block to revise candidate frames. In online interaction, you receive one action, predict one segment, show it, and then accept another action. Frames already shown constrain what comes next, and a long block can delay the effect of a new control.

The slide joins them in one sentence: future chunks can use bidirectional attention while successive chunks follow causal order. Inside a chunk the frames discuss with each other. Between chunks, the past speaks to the future and never the reverse.

The chain rule, one chunk at a time

Cut the future into K chunks B1, …, BK, each a few latent frames long. Apply the chain rule you met for language models: the probability of the whole future is a product, one factor per chunk, each conditioned on everything before it.

pθ(B1:K | …)
The probability of the next K chunks, given the history ht, the controls A and the other conditions C (for example, a prompt).
∏
Multiply the K factors, one per chunk, in order.
pθ(Bk | …)
One chunk, given the history, all earlier chunks B<k, and the controls up to and including its own, A≤k.

Across chunks, the model is autoregressive: it generates the next block from the available prefix and conditions on the controls available for that block. Within a chunk, it is a diffusion or flow-matching model over a continuous tensor, which refines all tokens of the current block together. That gives two loops. The lecture draws them as three words. The inner loop denoises the current chunk: Z(k)τ=0 is Gaussian noise, and the velocity vθ sees the history, the already generated chunks B̂<k, the controls and the condition. Commit decodes or emits the finished segment. The outer loop keeps the observed history, appends the new chunk, and receives new controls.

Who may read whom

Chunk k reads the history, every earlier chunk, and itself. It never reads a later chunk, because those do not exist yet. As a matrix with query chunks down and key chunks across, the allowed cells form a lower triangle. The slide adds the speed-up that makes this practical: KV caching reuses the prefix features across the denoising steps of the current chunk. The prefix does not change while the chunk is cleaned up, so its keys and values are computed once and stored.

A worked example shows what the cache saves. Let every chunk hold c = 100 tokens, and let there be K = 5 chunks with S = 4 denoising steps each. At chunk k, the chunk’s tokens read k chunks of keys, so one step costs c × kc = kc2 pairs. With the cache, the whole rollout costs S × c2 × (1 + 2 + 3 + 4 + 5) = 4 × 10,000 × 15 = 600,000 pairs, plus one pass per finished chunk to store its keys and values (150,000). Without the cache, every step would re-read the whole prefix as queries too, (kc)2 pairs per step: 4 × 10,000 × (1 + 4 + 9 + 16 + 25) = 2,200,000. The cache saves about 2.9 times here, and more as chunks accumulate.

Now the toy. A game runs 24 frames. It makes one chunk at a time, with S network evaluations per chunk, and it accepts a new action only between chunks. Press Turn left while it plays and see how long the world takes to obey.

Steer a streaming world

Each cell is one frame. Press Play, then press Turn left at any moment. The small triangle below is the chunk mask: the current chunk reads the cells in its row.

4

Illustrative toy of the slide’s online mode: 24 frames, one chunk generated at a time, an action accepted only between chunks. Evaluations are counted exactly (chunks × S); the playback speed is 8 frames per second of toy time, not a measured latency.

With 8-frame chunks the world can wait up to 7 frames to hear you. With 1-frame chunks it listens at every frame, and it also pays S evaluations for every single frame instead of S for every eight. Chunk length is a trade between response and cost. The failure shows up as a symptom you can feel: the controls seem mushy. You measure it as the delay between the press and the first frame that obeys. The fixes are shorter chunks or fewer steps per chunk, and Chapter 6 begins the second half of the lecture with the second fix.

Why does KV caching help when a streaming model denoises its current chunk over several steps?

Chapter 5

Which History Training Sees

Compare teacher forcing, diffusion forcing and self-rollout training by the context each one gives the model

A child learns to ride a bike with a parent holding the saddle. Every moment starts from a good position, because the parent fixes each wobble. Then the parent lets go. Now a tiny wobble is the starting point for the next wobble, and the next. Nothing in the lessons showed the child what a bad start feels like.

Chapter 4 gave us a model that predicts chunk k from the history and the chunks before it. The question for this chapter is simple. During training, which history do we feed it? The lecture compares three answers.

Teacher forcing: the real past

Teacher forcing trains each chunk with the true earlier chunks as context, as if a teacher whispered the right history. For chunk k, take the real chunk Bk, blend it with fresh noise ηk at dial position τ, and ask the network for the velocity Bk − ηk. The context is the observed history and the clean real prefix B<k.

Zτ(k)
Chunk k mixed with noise. This is the noisy input; the clean chunk Bk is the thing to be found.
LTF
The teacher-forced loss. Its context holds real chunks B<k, not generated ones. The target is still a velocity.

The slide stresses what this loss is not: it trains a conditional generator, not a deterministic next-frame regressor. The network never has to guess one next chunk. It learns a velocity field that turns noise into any plausible next chunk, so many futures stay possible.

The gap between training and rollout

Now take the bike’s training wheels off. In a real rollout the context is B̂<k, the model’s own earlier chunks. They may hold errors. The lecture’s two sentences are the whole problem: a small error can alter the context of every later prediction, and causal factorization does not remove the train-to-rollout mismatch. People call this exposure bias: the model was never exposed to its own mistakes.

Work a toy number. A point should circle at 0.5 radians per chunk, and our model errs by 0.02 radians each chunk. Judged with the real past, every prediction is off by 0.02 radians, about 1.1 degrees, a small flaw. Rolled out on its own past, the errors add up. After 24 chunks the point is off by 24 × 0.02 = 0.48 radians, about 27 degrees. Small and tolerable became large and visible.

Roll out on your own mistakes

The lines show how far the generated point drifts from the true one as the rollout grows. Raise the model’s steady bias or its random jitter, and compare the three trainings.

0.020
0.020

An illustrative model of error, not an experiment. Each chunk adds a steady bias and a random jitter, in radians on a unit circle. We assume: a teacher-forced model carries every earlier error forward; a diffusion-forced model has learned to halve random corruption of its context but not steady bias; a model trained on its own rollouts corrects 30% of its earlier error each chunk. Lines are averages of 60 seeded runs. The slide only says these robustness questions differ.

Diffusion forcing: give every chunk its own dial

Diffusion forcing tries a different cure. During training, each position in the sequence gets its own noise level: Z(i)τi = (1 − τi)ηi + τiBi, with the τi sampled independently, and each level supplied to the causal model. So the context a chunk sees is partly corrupted, which teaches the model to cope with imperfect history. A note on the slide settles notation: the paper says a larger corruption index means more noise, and in our lesson a larger τ means cleaner.

Here is that idea in four lines of code. Compare it with Chapter 2, where one τ covered the whole clip.

pythontau = torch.rand(B, K)                       # one dial position per chunk, drawn independently
z   = (1 - tau[..., None]) * eta + tau[..., None] * chunks   # each chunk blended at its own level
v   = model(z, tau, history, causal=True)       # the causal model is told every level
loss = ((v - (chunks - eta)) ** 2).mean()       # same velocity target as before

The same trick gives a new kind of sampling. The lecture’s figure sets two schedules side by side. In an autoregressive rollout, you finish the next position before you move on, so earlier positions turn into clean context. In staggered refinement, you refine several future positions at once, at different noise levels, and the more distant positions stay more uncertain. Count the passes in the toy below.

Schedule the noise across a sequence

Columns are 8 video positions. Rows are network passes, top to bottom. Purple is noise and warm is clean. Pick a schedule and drag the pass.

5

Illustrative schedules, exact counts. Each position needs 4 denoising steps. All together: every position at the same τ, 4 passes, no causal order. One at a time: 8 positions × 4 steps = 32 passes. Staggered: position i starts one pass after position i − 1, so 4 + 7 = 11 passes. This follows the shape of the lecture’s figure, not its numbers.

Three contexts, three questions

The lecture ends the comparison with a table of what each method shows the model as context. Teacher forcing: a clean real prefix. Diffusion forcing: a corrupted real prefix. Self-rollout training, the idea behind Self Forcing: the model’s own generated prefix. The caution on the slide is worth a margin note: robustness to Gaussian corruption and robustness to generated-history errors are different questions. A model can shrug off random noise in its context and still fail on its own steady bias, which is what the device assumes.

Teacher forcing is not obsolete. Slide 34 gives three reasons it stays useful. It has a clear conditional target. It trains in parallel offline: you pack clean context and noisy targets together and use causal masks, so one pass trains many chunks. And later rollout training can refine the result. The lecture names Causal-rCM (2026), which combines offline learning on real prefixes with refinement on generated prefixes. Teacher forcing and rollout training serve different stages of one system.

The chapter’s last slide is a warning about demos. A published autoregressive rollout, Self Forcing, streams its output. Streaming alone does not establish action accuracy. Three checks stay separate: does the history stay consistent, does the video respond to controls, and how long does each response take?

A model is trained with teacher forcing and then rolled out for 24 chunks on its own output. Its error is far larger than in training. What is the cause?

Chapter 6

Why Sampling Is Slow

Count the network passes behind one chunk of video, and see why one big step lands in the wrong place

Back to the driving game. You press left, and the model must draw the next chunk. Suppose one pass through the network takes 30 milliseconds. A chunk drawn in 50 steps with the two-run guidance of Chapter 2 needs 100 passes, so 3 seconds. The game answers every press three seconds late. The numbers are illustrative, but the shape of the problem is real: the world responds only as fast as the sampler runs.

The lecture states the cost in one line. Sampling cost is roughly the number of network evaluations times the cost of each evaluation. A streaming video model pays this bill again for every new chunk. You can cut the second factor with a smaller or faster network. This lecture cuts the first factor, and for that it returns to images, where the idea is easiest to see.

Two routes to fewer evaluations

The lecture splits the field in two. On the first route you keep the trained model as it is. Its learned velocity field, or its denoising field, stays fixed, and you try to follow its dynamics more cleverly. The tools are DDIM (a deterministic sampler for diffusion models) and other ODE solvers. An ODE, an ordinary differential equation, is a rule that gives the direction of motion at every point and time, and an ODE solver is a recipe for following that rule in finite steps. Our flow model is exactly such a rule: dx/dτ = v(x, τ).

On the second route you change what the network learns. You train it to make a finite jump, or to return a compact rule that is cheap to reuse. The lecture lists flow maps, consistency, π-Flow and distribution matching. It closes the slide with the sentence that sorts the two: both routes reduce sampling work, but only the second changes what the network learns.

The next slide sorts the second route by its training constraint. Trajectory-based methods learn relationships along a generative trajectory: they constrain maps, derivatives, consistency or policy velocities. Flow maps, consistency models, MeanFlow and π-Flow belong here. Distribution matching instead matches generated samples to the target distribution and does not need a fixed sample-by-sample correspondence. DMD, IMM and Drifting belong there; the lecture ends on that idea. The slide adds that these labels name the main constraint, that hybrids exist, and that distilling from a teacher versus training from scratch is a separate choice.

One big step goes wrong

Why not simply take one giant step? Look at what a step is. The network reports a local velocity: the direction to move at the point where you stand, right now. The step rule, called an Euler step after the mathematician who first wrote it down, is xτ+Δτ ≈ xτ + Δτ vθ(xτ, τ). It assumes the direction stays the same for the whole step. But as the point moves, the true direction changes, and the path curves. A long straight step cuts across the curve and ends in a different place. The lecture says it plainly: taking a larger step alone does not make a reliable one-step generator.

Our toy world lets us see this exactly. Three data points sit on a circle. At τ = 0 every start point is pure noise, and the best single guess of what the clean sample will be is the average of the three data points, which is the centre. A full step of Δτ = 1 moves the sample from x to x + (centre − x), so it lands at the centre, where no data lives. Now try smaller steps.

Take N steps along the slope

The blue curve is the true path from the start (purple). The warm line takes N straight steps. Tap the plane to move the start, or pick a preset. Bars below show the miss for N = 1 to 32.

1

The toy world of the hero: the exact velocity field for three data points. The true path is computed by a fine solver. “Miss” is the distance between the N-step landing point and the true endpoint of this path. Cost uses an illustrative 30 ms per network pass. Euler is the simple step rule on the slide; better solvers take bigger steps but spend more passes per step.

At N = 1 the warm line is a single straight segment into the middle, and the miss equals the circle’s radius, 1.6. At N = 2 it still lands between data points. From N = 4 the line follows the bend and reaches the endpoint. This toy is tame, with three clean groups. Real image and video fields bend far more, so real samplers spend many more steps. The trade is the same at any scale: accuracy costs evaluations.

An average is not a sample

A reasonable objection: the network could just output the clean sample directly. Some models do. The slide calls this denoising regression: the network learns ŷ(rτ, τ, C) ≈ E[y | rτ, τ, C], the average clean image given the noisy input. One noisy input fits many clean images, and the average of many is a blur, the star in the middle of the plane. The JiT work cited on the slide builds a network whose direct output is a clean-image prediction. The lecture is precise about what that does and does not change: changing the output parameterization does not train a one-step sampler.

What a one-step sampler needs is different. It needs a trajectory endpoint map: given this particular start point, return the particular endpoint that the flow reaches from it. The average over all endpoints is the wrong object, and the endpoint of this path is the right one. Everything from here on, flow maps, consistency models and MeanFlow, is a way to learn that map.

In the toy world, why does one Euler step with Δτ = 1 from pure noise land in the centre of the three data points?

Chapter 7

Teaching a Student to Jump

Train a network to cross a long stretch of the path in one evaluation, and learn the one rule such a jump must obey

A chess teacher plays a long combination: knight takes, bishop retreats, rook slides in, check. A beginner needs every move shown. After years, a strong player glances at the first position and sees where the combination ends. The moves have not vanished. The player has learned to jump over them.

This chapter teaches a sampler to do the same. The lecture offers three tools, which build on each other: progressive distillation, reflow, and the flow map.

Progressive distillation: halve the steps, again and again

Distillation trains a fast student to copy a slow teacher. In progressive distillation the teacher takes two deterministic DDIM steps. The student must reach the same place in one step. In symbols, the student Sθ(xs, s, t) is a complete update from generation coordinate s to t, and its loss is the squared gap to the teacher’s two-step result xTt.

Lendpoint
The gap between the student’s one step and the teacher’s two steps, squared and averaged.
Sθ(xs, s, t)
The student: from the point xs at coordinate s, produce the point at coordinate t in one network evaluation.
xtT
The teacher’s answer, from two of its own steps. It is frozen. It is the target.

Then the trained student becomes the next teacher, and you halve the step count again. Take a worked example with illustrative numbers. A 1,024-step sampler becomes 512 steps after round 1, then 256, 128, 64, 32, 16, 8 and 4 after round 8. Each round trains one student, so going from 1,024 to 4 steps costs 8 rounds of training. Every round is easy, since the student only has to match two steps, but the rounds add up.

Reflow: straighten the roads first

A second idea starts from a different complaint. Paths bend because the training pairs are random: each noise sample was paired with an unrelated image. Two lines that cross in the middle must share one velocity there, and the network learns their average, which bends every path near a crossing.

Reflow repairs the pairing. Use the current generator to turn a noise sample n into its own image yT. Keep that pair together. Then train a new velocity network vφ on straight blends of exactly those pairs: rτ = (1 − τ)n + τyT, with target yT − n. The lecture’s rule: initial flow matching pairs noise and data independently; reflow pairs each noise with its own generated image. The new pairing encourages straighter trajectories, so coarse sampling, with few steps, becomes more accurate. A straight path is the one case where a single slope step is exactly right.

The flow map: the jump as an object

Now name the thing we have been circling. Follow the ODE dxτ/dτ = v(xτ, τ) from coordinate s to coordinate t. The flow map Fs,t returns where you arrive: Fs,t(xs) = xt. It is the finite-time destination along the generative flow. A note on the slide matters here: s and t are generation coordinates, not video positions. They are the τ clock of Chapter 0.

A flow-map network Fθ takes three inputs: the current point x, the start coordinate s, and the destination coordinate t. For an image batch that is a tensor of shape [B, 3, H, W] plus two scalars per example. It returns a tensor of the same image shape. Sampling is then short. Draw x0 from Gaussian noise. With one evaluation ask for the full jump, x̂1 = Fθ(x0, 0, 1), and the random input supplies the diversity. With several evaluations, chain shorter jumps, xti+1 = Fθ(xti, ti, ti+1), and the approximation can improve with refinement.

The one rule a jump must obey

Because Fs,t follows a flow, it obeys two identities, and neither mentions any teacher. First, no time passing means no movement: Fs,s(x) = x. Second, a long jump agrees with two shorter ones: Fs,t(x) = Fr,t(Fs,r(x)) for any midpoint s ≤ r ≤ t. This second rule is the heart of the whole second half of the lecture. A map that breaks it is not a flow map. Check it yourself: the exact map passes, and a jump made from one local slope fails.

Long jump, or two short ones?

The blue curve is one true path. Move s, r and t along it. The dashed arrow is one jump from s to t. The solid arrows are two jumps, s to r, then r to t. If the rule holds, both end at the same place.

0.00
0.30
0.90

The toy world of the hero, computed exactly. “Exact flow map” stands in for a perfectly trained Fθ. “One-slope jump” is x + (t − s) v(x, s), the Euler step of Chapter 6 used as a map. The reflow pair is the straight chord from this start’s noise to its own endpoint; the distance is the largest gap between that chord and the path. Time stops at 0.98 so the field stays well defined.

Learning a map from a teacher

How do we train Fθ? The slide gives the plain recipe, which merges the ideas above. The teacher is a velocity field vT you already trained. Take a start point xs, and let the teacher’s ODE solver compute the destination: xTt = ODESolve(vT, xs, s, t). The student Fθ must predict that destination in one evaluation, and the loss is the squared gap, Lmap = E ‖ Fθ(xs, s, t) − xTt ‖2. The pairing keeps the starting point and the teacher’s destination matched. The slide notes that this target preserves a specific input-to-output correspondence: noise n always goes to the image the teacher would make from n.

The failure mode shows up in the device. The symptom: a student trained only on short hops gives different images for one long jump and for two short ones. The measurement: the gap between the dashed and the solid endpoints, which is zero for the exact map and about 0.35 for the one-slope jump at the starting settings. The fix is to train on many (s, r, t) triples and, as the next chapters show, to add the composition rule itself to the loss.

Which identity must any flow map Fs,t satisfy, whatever teacher produced it?

Chapter 8

BOOT and π-Flow: Learning From a Teacher’s Local Advice

Distill a fast student without running the teacher’s whole path, and let one network call buy many cheap steps

Chapter 7’s recipe has a cost hidden in plain sight. To label one training example for the student, the teacher must walk its entire path. A long path is many network passes, for every example, before the student learns anything. Two methods in the lecture avoid that bill in two different ways.

BOOT: predict a signal, not an image

BOOT fixes the source at noise. For a given noise n, the teacher’s flow defines one path to one image. The student is asked to predict any destination along that path: you give it the noise n and a requested time t, and it returns a destination. The slide states the key fact in two sentences: the same noise identifies the trajectory, and the requested time selects the destination.

The student does not output the noisy state directly. It outputs a signal yθ(n, t), a guess of the clean content, and the state is rebuilt from it: x̂t = αt yθ(n, t) + σt n. The weights αt and σt say how much signal and how much noise the state holds at time t; in our flow convention they are t and 1 − t. Then x̂t ≈ F0,t(n), the flow map from noise. At t = 1, yθ(n, 1) is the finished image, so one evaluation generates one image.

The clever part is the training target. A full teacher rollout would be expensive. BOOT bootstraps instead: it builds the target for a time from the student’s own prediction at the neighbouring time, plus one short teacher step. The lecture lists the pieces. Local target: predict a signal, rebuild its noisy state, and apply a short teacher update. Student update: from the same noise, predict the next signal and match the stopped target, where “stopped” means no gradient flows through the target, so it is a fixed label for this update. Training needs only noise and optional prompts, not real images, and a noise-end boundary loss anchors the trajectory.

Here is one bootstrapping step with illustrative numbers, one pixel, flow weights t and 1 − t. Take n = −0.5 and t = 0.5, and suppose the student predicts the signal y = 0.8. The state is x̂ = 0.5 × 0.8 + 0.5 × (−0.5) = 0.15. The frozen teacher advances it a short step of 0.1 in time, using its velocity there, say 1.5: the new state is 0.15 + 0.1 × 1.5 = 0.30 at t = 0.6. Which signal would make that state? Solve 0.6 × y + 0.4 × (−0.5) = 0.30, so y = 0.50 / 0.6 = 0.833. That is the stopped target for the student’s signal at t = 0.6. The loss is (yθ(n, 0.6) − 0.833)2. One teacher call stood in for a whole rollout, and the lecture’s summary line says exactly this: local teacher corrections replace complete teacher rollouts during training.

π-Flow: one call, many cheap steps

The second idea attacks the other end. What if the expensive network is called rarely, but each call hands back something that stays useful for many small steps? π-Flow does this. The expensive network Gφ looks at the current point xs, the time s and the condition C, and returns a policy πs: a compact set of numbers that defines a velocity rule. Between s and t, a cheap solver steps along it: dxτ/dτ = πs(xτ, τ). The word comes from reinforcement learning, where a policy maps a state to an action. The slide is careful: here the policy returns a generation velocity, not an environment action. A few network calls can support many inexpensive integration substeps.

The lecture shows two kinds of policy. The DX policy carries a grid of clean-image estimates, a tensor of shape [G, 3, H, W] for G grid times in an image model, and interpolates between them. Its velocity at each substep is (ŷs(τ) − xτ) / (1 − τ), the same formula our toy field uses with the interpolated estimate in place of the fixed average. The GMFlow policy caches the weights, means and variance of a mixture of Gaussians and, at each substep, computes a clean estimate from the cache and the current point (xτ, τ). That lets the estimate respond to where the sample is now. In both cases, fixed policy numbers give velocities that change with state and time.

Try the DX policy in the toy. One network call is idealized to return clean estimates read off the true path at G grid times. The cheap solver then takes M substeps.

Spend one call, take many steps

Yellow diamonds are the clean estimates one network call hands over. The warm line is the cheap solver following them. Move the start with a preset, then raise G and M.

3
32

Idealized: a perfectly trained policy would hand over the clean estimates that the true path implies at the G grid times 0, 1/G, 2/G and so on, held constant after the last one. A real network must predict them from the start point alone; that is what training teaches. Substeps use the DX rule of the slide, (estimate − x) / (1 − τ), with the estimate interpolated between grid times. Teacher Euler uses the exact toy field.

Set G = 1: the policy carries one estimate, the average of the data, and every substep heads for the centre, so the result matches the failed one-step Euler of Chapter 6. Raise G to 3 and the estimates bend toward the true endpoint, and 32 cheap substeps land on it, all from one call. The cost is paid in a bigger output, not in more calls.

How π-Flow trains: visit, then imitate

A policy needs a teacher too, and the lecture’s choice is the same pretrained flow-matching model as before, frozen. Its velocity vT(x̄τ, τ, C) is queried at the state the student has reached, so no full teacher rollout is needed. Training then runs in two moves. Visit: integrate the student’s own detached policy to get its own intermediate states. Imitate: at each visited state, match the policy’s velocity to the teacher’s velocity there. The loss is Lπ-ID = E ‖ πs(x̄τ, τ) − sg[vT(x̄τ, τ, C)] ‖2, with sg stopping gradients through the rollout and the teacher target. Both velocities are read at the same student-visited state, and the live policy is updated.

You may notice the family resemblance to Chapter 5. Training on the states you will actually visit is the same cure as self-rollout training: the student is judged where it really stands, not where a teacher would have stood. This is our link, not a claim of the slide.

The failure mode, as symptom, measurement and fix. The symptom: the policy’s samples look right at the start and drift wrong near the end. The measurement: compare the policy’s velocity with the teacher’s at the states the policy actually visits, late in the path. The fix: more grid points in the policy near the end, or training on visited states, which is what π-ID does.

What does BOOT use instead of a complete teacher rollout to build a training target?

Chapter 9

Consistency: One Endpoint From Anywhere

Ask the network for the end of the path from any point on it, and make every point agree

A sculptor photographs a block of stone at ten stages of carving: a rough block, a head, a face, a polished statue. Hand a friend any one of the ten photos and ask: what will the finished statue look like? A good friend gives the same answer for all ten photos, because all ten show the same carving.

That is the whole idea of a consistency model. Chapter 7 learned a map between any two times, Fs,t. A consistency model learns only the most useful row of it: the map to the data end, t = 1. Call it f(xs, s) = Fs,1(xs). The slide asks for a change of mindset. Instead of reusing a velocity rule step after step, predict the endpoint straight from any point on the trajectory. The lecture’s figure comes from the original Consistency Models paper, and it flips the clock: the paper runs data at 0 and noise at T, while ours runs noise at 0 and data at 1.

Three conditions on f

f(xs, s) = Fs,1(xs)
The definition: jump from here all the way to the data end of the path.
f(xs, s) = f(xt, t)
The self-consistency rule. Two points on the same path must predict the same endpoint.
f(x, 1) = x
The boundary. At the data end, the map returns its input unchanged.

The slide ends its first half with a plain sentence: a consistency model is an endpoint-map construction. Generation needs one evaluation: draw noise n, return x̂1 = fθ(n, 0).

Building the boundary into the network

The boundary f(x, 1) = x is a hard rule, and the lecture shows a way to make a network obey it without being taught. Write the network as fθ(x, s) = x + (1 − s) gθ(x, s). It is a skip connection: the output is the input plus a correction, and the correction is multiplied by (1 − s). At s = 1 the factor is zero, so fθ(x, 1) = x for any gθ at all. This is the “boundary-aware parameterization” of the slide. The network g only has to learn the rest.

The distillation target

If you have a teacher, consistency has a training signal. Take a point xs on a path. Use one teacher step to move to a nearby point on the same path: x̃s+h = ODEStep(vT, xs, s, s+h). Both points share one endpoint, so the two predictions should match. The loss compares them: LCD = E d(fθ(xs, s), sg[fθ̄(x̃s+h, s+h)]). The distance d is some prediction-distance loss. The sg stops gradients, so the right-hand side acts as a label. And θ̄ is a target copy of the weights, a slowly changing version of θ that keeps the label from jumping around while the student learns.

A worked example, with illustrative numbers and squared distance for d. Let s = 0.5, h = 0.1 and xs = 0.2. The teacher steps with velocity 1.2, so x̃ = 0.2 + 0.1 × 1.2 = 0.32 at time 0.6. The target copy says the endpoint from there is fθ̄(0.32, 0.6) = 0.90. The student says fθ(0.2, 0.5) = 0.70. The loss is (0.70 − 0.90)2 = 0.04. Training pulls the earlier prediction toward the later one, and the later one is closer to the boundary, where f is easy to get right.

Do two points agree on the ending?

Two points sit on one path, at times s and s + h. Each predicts an ending. Choose a predictor and see whether the two predictions agree, and whether they are right.

0.30
0.30

The toy world of the hero, exact. The endpoint map is Fs,1 computed by a fine solver. The clean estimate is the average of the data given the point, which is what a denoising regression learns (Chapter 6). The constant predictor ignores its input. “Right” means within 0.45 of the data point the path actually reaches.

Read the three results. The endpoint map gives the same ending from both points, and it is right: gap 0, and it lands on the true endpoint. The clean estimate drifts as s moves, because at an early time it still averages over several data points, and later it sharpens; it is not consistent along the path. The constant predictor is perfectly consistent, gap 0 everywhere, and useless: it never reaches any data. This last case is the reason the boundary exists. Self-consistency alone has a trivial solution, so the rule f(x, 1) = x pins the answer to the data end.

Without a teacher: a displacement that learns a slope

Slide 56 asks how to learn such a map when no teacher exists. It writes the jump as an interval length times a learned vector: Fθ(x, s, t) = x + (t − s) uθ(x, s, t). When the interval shrinks to nothing, u must become the local velocity: limt→s u(x, s, t) = v(x, s). On a vanishing interval, the network learns the local field from flow-matching data, which gives a known regression signal that anchors the dynamics. On a finite interval, it must enforce compatible map relations or derivative identities, and the network extends the signal to larger jumps. The slide’s closing line explains why the anchor matters: self-consistency needs a data-connected anchor. The next chapter shows the most direct such construction.

A predictor outputs a fixed point for every input and every time. Its self-consistency loss is zero. What does that show?

Chapter 10

MeanFlow: Learn the Average Velocity

Learn the speed of the whole trip instead of the speed at the start, and train it from data alone with one derivative

You drive 120 kilometres in 2 hours. Your average speed is 60 kilometres per hour. At the moment you pulled out of the driveway, the speedometer read 20. If someone had asked where you would be in 2 hours and you had answered with the speedometer, 20 kilometres per hour for 2 hours, you would be off by 80 kilometres. The speedometer reports now. The average reports the trip.

That is the whole idea of MeanFlow, and it fixes Chapter 6’s failure: a big step along the starting slope misses because the path bends. A big step along the average slope of that interval lands exactly. The slide gives the definitions side by side. The instantaneous velocity v is the tangent direction at the current point and describes an infinitesimal change. The average velocity u is the net displacement divided by the interval length and describes a finite jump.

u(xs, s, t)
The average velocity: from the point xs at time s, over the interval to time t.
(xt − xs) / (t − s)
Net displacement divided by interval length. It is the speed of the whole trip.
1/(t − s) ∫ v dτ
The same thing as an average of the instantaneous velocity along the path. The path is the generative ODE trajectory, not an arbitrary straight line between a training pair.

With u in hand, the jump is exact: Fs,t(xs) = xs + (t − s) u(xs, s, t). The slide adds the point that makes the learning problem different. A long displacement need not point along the initial tangent, so the network must know both where the interval starts and where it ends. It takes (x, s, t), exactly like the flow map of Chapter 7. A finite jump can follow a curved flow without tracing every intermediate point. Drag the interval below and compare the two arrows.

Speedometer, or the whole trip?

The blue curve is a true path. The thin arrow is the starting slope, stretched across the interval. The thick arrow is the average velocity. Drag s and t, and read the check in the sentence below.

0.10
0.80

The toy world of the hero, exact. u is computed from the true path; v is the exact field. The identity check differentiates u along the path with a small finite step (0.001), keeping t fixed, the way the derivation below does. Time stops at 0.98 so the field stays well defined.

Deriving the identity

We cannot train u from its definition, because that needs the whole path. The lecture finds a different equation that holds at one point. It takes four steps, and each is short.

  1. Start from the definition, multiplied out: (t − s) u(xs, s, t) = ∫st v(xτ, τ) dτ.
  2. Hold the destination t fixed. Slide the starting point xs along the ODE, so s changes and xs moves with it. Differentiate both sides with respect to s.
  3. Left side, product rule: d/ds of (t − s) u gives −u + (t − s) du/ds, since d(t − s)/ds = −1.
  4. Right side: only the lower limit of the integral moves, so the derivative is −v(xs, s). Set both sides equal, cancel the minus, and rearrange.
u
The average velocity over the whole interval.
v
The instantaneous velocity at the start. When the interval is tiny, u and v are the same.
(t − s)
The length of the interval. The correction grows with it.
du/ds
How the average velocity changes as the starting point slides along the path. It is a total derivative: both s and xs move.

The device above checks it. Its read-out reports u, then v + (t − s) du/ds with du/ds found by finite differences along the path, and the two agree to within a few thousandths, the size of the finite-difference error. The identity is true on the real flow.

The total derivative has two parts. The slide spells them out: du/ds = ∂su + Jxu · v. The first is the change from the start-time argument. The second is the change from the input point moving with velocity v, where Jxu = ∂u/∂x is the Jacobian, the table of all the ways u responds to each input number. A Jacobian-vector product (JVP) computes Jxu times a vector without ever building that table: it runs one extra forward pass in a chosen direction. This is why training stays affordable.

A target you can build from data

One problem remains: the identity needs v(x, s), the marginal velocity, the average over all data that could have produced x. A training sample gives only its own pair velocity. Take a data image y and noise n, blend them as rs = (1 − s)n + sy, and the pair velocity is δ = y − n. The true field is the average of those: v(x, s) = E[δ | rs = x, s]. The slide’s resolution: the training pair supplies a conditional regression signal for the unknown marginal field. Plug δ in where v belongs, and averaging over many pairs does the rest.

Dsuθ
The total derivative of the network’s own output along the pair velocity δ. One JVP in direction (δ, 1, 0): move x by δ, s by 1, and leave t alone.
utarget
The label: the identity u = v + (t − s) du/ds with δ in place of v.
LMF
The MeanFlow loss. The sg means “stop gradient”: the target is treated as a fixed label.

Look at the target when t = s. The correction vanishes and utarget = δ, so the loss becomes ‖uθ(rs, s, s) − δ‖2: plain flow matching. The training loop therefore samples intervals with s ≤ t, including diagonal ones where t = s. Diagonal samples keep the network anchored to the data. Longer intervals teach larger jumps. The lecture’s four steps: sample y, n and s ≤ t; build rs and δ; evaluate uθ and its JVP; update to match the detached target.

Build one MeanFlow target

One pair, y = 1.2 and n = −0.8, so δ = 2.0. The small made-up network uθ is linear. Drag s and t, then watch the label and the network’s guess. Turn on the improved target to swap it for the one on slide 65.

0.20
0.80

Illustrative toy with exact arithmetic: uθ(x, s, t) = 0.5 x − 0.8 s + 0.3 t + 0.4, so ∂su = −0.8 and Jxu = 0.5. The JVP in direction (δ, 1, 0) is −0.8 + 0.5 δ; a finite-difference check agrees. In the improved target, vθ(x, s) = uθ(x, s, s) replaces δ inside the JVP and δ appears only as the regression target, as on slide 65.

Sampling, the improved target, and guidance

At inference, nothing of the derivative remains. Draw x0 from Gaussian noise and take one jump: x̂1 = x0 + uθ(x0, 0, 1). For a few steps use xt ≈ xs + (t − s) uθ(xs, s, t). Sampling needs only a forward pass; the JVP is a training tool, as the slide says.

One weakness is in the target above: it uses δ, the velocity of the one pair that made rs, inside the derivative. That pair is a noisy stand-in for the marginal velocity. Improved MeanFlow (iMF, CVPR 2026) swaps it. It defines vθ(x, s) = uθ(x, s, s), the network’s own prediction of the marginal velocity, and uses that inside the JVP: Dpredsuθ = ∂suθ + Jxuθ vθ. It then forms ṽθ = uθ − (t − s) sg[Dpredsuθ] and regresses it on δ: Lv = E ‖ṽθ − δ‖2. The pair velocity now appears only as the regression target, so the prediction depends on the noisy input and not on its paired direction.

Last comes guidance. A few-step generator must also follow a prompt, with a chosen strength. The slide writes the map as Fθ(xs, s, t, C, w): the condition C names the content, the guidance scale w sets how hard to follow it, and the learned jump represents the corresponding guided transformation. iMF treats guidance settings as explicit conditioning variables. The slide ends with a warning: a guided endpoint map is not generally a linear combination of unguided endpoint maps. You cannot blend two jumps to get a stronger guidance. The network must be trained with w as an input.

The failure mode, as symptom, measurement and fix. The symptom: a network that agrees with itself across intervals yet makes samples that do not look like the data, the Chapter 9 constant predictor in a new coat. The measurement: check the diagonal. If uθ(x, s, s) does not match the flow-matching velocity, nothing ties the network to the data. The fix is the lecture’s own: sample intervals including t = s, so every batch carries the plain flow-matching signal, the data-connected anchor of slide 56.

In the MeanFlow target utarget = δ + (t − s) Dsuθ, what happens when the interval has zero length (t = s)?

Chapter 11

The Field Guide

Sort the five objects a fast model can learn, keep the numbers and the code, and see where the next lecture goes

You can now follow Lecture 13 from end to end. A video model is one joint draw. A world model draws it chunk by chunk, using a history and your actions. Training can show it the real past, a noisy past or its own past. And fast sampling means changing what the network is asked to learn, from a slope to a jump.

The one-paragraph summary

A video generator refines one latent clip jointly: one τ for all frames, space-time tokens, spatial, temporal or joint attention, and a loss on the velocity Zdata − η (Chapters 0 to 2). Production systems such as Seedance and MiniMax H3 add references and audio in one shared token sequence, and a released base model is not the same as a hosted product (Chapter 3). An interactive world model factorizes the future into chunks with an inner denoising loop and an outer rollout loop, using a causal mask over chunks and a KV cache (Chapter 4). Teacher forcing trains on a real prefix, diffusion forcing on a corrupted one, and self-rollout training on the model’s own, and the gap between training and rollout is real (Chapter 5). Sampling cost is evaluations times cost per evaluation, and a big step along a local slope misses a curved path (Chapter 6). The fix is to learn a jump: a flow map Fs,t obeying Fs,s = x and the composition rule (Chapter 7). You can train it by distillation, by bootstrapping (BOOT) or by a reusable velocity policy (π-Flow) (Chapter 8). You can train an endpoint map with a boundary (consistency, Chapter 9). Or you can learn the average velocity u, with the identity u = v + (t − s) du/ds, whose training target comes from one JVP (MeanFlow and iMF, Chapter 10).

What a fast model learns

The lecture closes its fast-sampling half with a table of five objects. Learn to tell them apart by their inputs and by what they return. Then sort the examples below.

ObjectTakesReturns
Velocity v(xs, s)a point and a timelocal motion along the generative flow
Flow map Fs,t(xs)a point, a start time, a destination timea finite-time destination on that flow
Endpoint map f(xs, s)a point and a timethe data-end destination
Average velocity u(xs, s, t)a point, a start time, a destination timedisplacement divided by interval length
Velocity policy πs(xτ, τ)a point and a time, with parameters from one network calla compact rule reused across integration substeps

Which object is it?

Read the description and pick the object it describes. A wrong pick shows the right one.

The five objects and their one-line meanings are from slide 67. The descriptions are our own wording of the properties that Chapters 6 to 10 derived.

The numbers that matter

WhatNumber or ruleWhy it matters
Wan video autoencoder[1 + T, H, W, 3] to [1 + T/4, H/8, W/8, C]Time shrinks 4 times, space 8 times per side, at every sampling step
Attention pairs, L frames, P tokens eachspatial L P2; temporal P L2; joint L2P2 = N2Doubling the frames makes joint attention cost 4 times as much
MiniMax H333B dense single-stream Transformer; 768p video plus stereo audioOne shared sequence lets audio read video; only H3-Base has released weights
Chunked rollouttwo loops: denoise a chunk, then append itResponse delay is set by chunk length; the cache reuses the prefix
Training contextclean real, corrupted real, or own generated prefixExposure bias is the gap between the first and the last
Sampling costevaluations × cost per evaluationTwo routes: a better solver, or a model that learns a jump
Progressive distillationteacher takes two steps, student one; steps halve each round1,024 to 4 steps takes 8 rounds (illustrative)
Flow map rulesFs,s(x) = x; Fs,t = Fr,t after Fs,rThe composition gap is zero for a true map
Consistencyf(xs, s) = f(xt, t); f(x, 1) = x; f = x + (1 − s) gThe boundary rules out the constant answer
MeanFlow identityu = v + (t − s) du/dsTurns a path integral into a pointwise equation
MeanFlow targetutarget = δ + (t − s) Dsuθ, with one JVPAt t = s it is flow matching; sampling needs no JVP
This lesson’s toy world1 Euler step lands 1.60 from the endpoint; 4 steps land on itIllustrative only: real fields bend far more

The core as code

Four short functions hold most of the lecture. They are a sketch: model, vae, u_net and the cache helpers stand for whatever networks you use.

pythondef clip_loss(model, vae, video, text):               # Ch 2: one joint draw per clip
    z   = vae.encode(video)                          # [B, Tz, Hz, Wz, d]
    eta = torch.randn_like(z)
    tau = torch.rand(z.shape[0], 1, 1, 1, 1)         # one dial for every frame
    z_tau = (1 - tau) * eta + tau * z
    v = model(z_tau, tau, text)                       # same shape as z
    return ((v - (z - eta)) ** 2).mean()

def rollout(model, history, actions, steps=4):           # Ch 4: two loops and a cache
    cache, out = model.prefill(history), []
    for a in actions:                                 # outer loop: one chunk per action
        z = torch.randn(CHUNK_SHAPE)
        for i in range(steps):                         # inner loop: denoise this chunk
            z = z + model(z, i / steps, cache, a) / steps
        out.append(z)
        cache = model.extend(cache, z)               # commit: keys and values of the new chunk
    return out

def meanflow_loss(u_net, y, n, s, t):                  # Ch 10: s <= t, with some t == s
    r, d = (1 - s) * n + s * y, y - n
    u, jvp = torch.func.jvp(u_net, (r, s, t), (d, torch.ones_like(s), torch.zeros_like(t)))
    target = (d + (t - s) * jvp).detach()         # stop gradient
    return ((u - target) ** 2).mean()

def sample_one_step(u_net, shape):                     # Ch 10: no JVP at inference
    x0 = torch.randn(shape)
    return x0 + u_net(x0, torch.zeros(1), torch.ones(1))  # x1 = x0 + u(x0, 0, 1)

Read it line by line and name the chapter behind each choice. The clip loss uses one τ for the whole clip. The rollout spends steps evaluations for every chunk and reuses the cache. The MeanFlow loss needs the jvp in direction (δ, 1, 0) only at training time. The sampler is one line.

A checklist for the next paper you read

Where this sits, and what comes next

Chapters 0 to 3 extend Lecture 12, Video Diffusion Models. Chapter 4 is the step from generation to prediction that Lecture 1 promised, and the world models lesson and the Genie paper lesson show how far it goes. For the flow-matching base, see flow matching and the MIT flow-matching lesson. For the generator family, see Lecture 7 and diffusion transformers.

The last slide of the lecture asks one question for next time: must a fast generator preserve the same noise-to-image correspondence? Today’s methods all learn a trajectory’s destination or its velocity policy, comparing paired predictions, flow relations or local velocities. Distribution matching drops the pairing. It makes the distribution of generated samples match the target, so it compares distributions without fixing every correspondence. The lecture names Distribution Matching Distillation, Inductive Moment Matching and Drifting Models, and says this route can complement trajectory-based training. Lecture 14 picks it up together with interactive world models.

Credits and sources

This lesson follows Lecture 13 of Penn CIS 6280 World Models (Fall 2026), taught by Jiatao Gu. See the course site and the lecture slides. All explanations, devices and toy numbers here are our own; slide numbers and published figures are the course’s. Papers the lecture cites: Flow Matching, Rectified Flow (reflow), Video Diffusion Models, Wan, Diffusion Forcing, Self Forcing, Back to Basics (JiT), Progressive Distillation, Flow Maps via Self-Distillation, BOOT, π-Flow, Consistency Models and MeanFlow.

The takeaway. A fast sampler is a different question put to the network. Ask for a slope, and you must take many small steps. Ask for a jump, and the network must obey the rule that a long jump equals two short ones, and it needs an anchor in the data. Video world models need that speed because a world you steer has to answer before you notice the wait.
You want a world model that answers your control within a few frames. Which pair of choices does the lecture point to?

Now press Teach and explain, from memory, how a video model draws one clip, how it becomes a world you steer, and why a jump needs the composition rule. Then go back to the dots at the top, pick the average-velocity jumps, and drag the passes down to 1 to see the whole lecture in one picture.

CIS 6280 Lecture 13: Video World Models and Fast Sampling
The course site · Lecture slides · Back to Gleams