Latent World Models

A robot plans a move by imagining it, one step at a time, and each imagined step starts from the last guess. Each guess is a little off, so the errors pile up. A model only 8% wrong per step ends 5.8 units off after 12 steps.

Learn how a world model encodes what it sees, keeps the past, predicts the next code, and rolls forward on its own.

Press Play, then drag the model gain and the number of steps. Compare the green line, where the model gets the true state each step, with the warm line, where it reads its own guesses. Then we build, piece by piece, the codes, the memory, the distributions and the famous designs that do this.

You need a rough idea of what a probability is and a little algebra. We build the rest from zero.

Based on Lecture 8 of Penn CIS 6280: World Models (Fall 2026), taught by Jiatao Gu, and the papers it cites. Lecture slides. Every idea is re-taught here in our own words and devices; it follows Lecture 7, Latent-Variable and Adversarial Models.

Roll the model forward

12
1.08

The system is the lecture’s own (slide 63): the true position follows xt+1 = xt + at from x0 = 1, with a = +1 for six steps and then −1 for six. The model believes xt+1 = g xt + at. At g = 1.08 the lecture reports a one-step error of 0.160 and a rollout error of 5.823 at step 12; this page computes the same numbers live.

Chapter 0

Observations Become Codes

Turn a camera frame into a short code the model can predict, and see the four choices hidden inside that step

A robot looks at a table through a camera. Each picture has 64 × 64 pixels, three colour numbers per pixel. That is 12,288 numbers, and 12,288 more arrive a moment later. Now we ask the robot: what will the next picture look like if I push the cup left?

Predicting 12,288 numbers one by one is slow, and most of them carry no news. The table stays the table. The cup moves. A smarter plan is to squeeze each picture into a short list of numbers that keeps what matters, and to predict that list instead. This lecture is about doing exactly that, and about the choices that make it work or fail.

One short paragraph on the course first. Assignment 1 is due Thursday, September 24, and Canvas holds the exact deadline. The September 22 class was tentatively on Zoom, and the September 24 class may be on Zoom too, so check Canvas for both. Office hours this week moved to Friday, September 25, at 5:00 PM.

The big words, in plain terms

An observation, written ot, is whatever the agent senses at time t, such as one camera frame. An action, at, is what the agent does, such as “push left”. A world model is a learned model of how the world changes when the agent acts. Lecture 7 told you why one answer is not enough, and how latent variables let a model draw many answers.

A latent world model is a world model that does its predicting in a hidden space. The lecture states it in one sentence: it predicts how an environment evolves under supplied actions, using a learned internal representation. A representation is a rewritten form of the input, such as a short list of numbers. A code, written zt, is the representation of one observation. The code is the thing we predict.

Two earlier ideas meet here. A state-space model says a hidden state explains how observations evolve. Representation learning says an objective, a goal for training, decides what the code keeps. Neither gives you the other for free. The lecture’s warning is plain: an observation code is not automatically a sufficient predictive state. A code can keep the colour of the cup and lose its speed. Chapter 1 returns to this.

The encoder and the decoder

The encoder is a network that turns an observation into a code. The decoder turns a code back into an observation. In a variational autoencoder (VAE), the encoder does not output one code. It outputs a distribution over codes, written qφ(z | x), and we sample one code from it. Lecture 7 wrote the observation as x. This lecture writes ot, because observations now come in time order.

The prior p(z) plus the decoder pθ(x | z) define a generative model of single observations. Nothing in that picture says how one observation leads to the next. The lecture is blunt about it: a temporal transition is an additional modeling choice. Most of this lesson is about how to make that choice.

Slide 5 adds a second option. You can train the decoder against a discriminator, a judge network that tries to tell real images from decoded ones. The decoder then learns from reconstruction losses, perceptual losses (which compare how two pictures look to a trained network, not pixel by pixel) plus the judge’s feedback, which tends to give sharper images. The discriminator exists only during training. VQGAN is the lecture’s example of a latent autoencoder with an adversarially trained decoder. Lecture 7 covered the game itself.

One vector, or many tokens

What does a code look like? Slide 6 shows two shapes. It can be one continuous vector, such as [0.2, −0.4, 1.1, …]. Or it can be a sequence of patch tokens, [v1, v2, …, vK], where each piece describes one patch of the picture and is itself a vector. The lecture’s rule is: a token is a unit of representation, and it can be a continuous vector. So a “token” here is wider than the word-pieces of a language model.

From continuous features to a codebook

Some designs go one step further and make every piece a whole number. They keep a codebook: a fixed list of V learned vectors, each with an index. Think of a paint shop that sells only four shades. You bring a colour you like, and the clerk hands you the closest shade on the shelf.

That is vector quantization. The encoder first makes a continuous feature v = E(o). Then we compare v with every codebook entry and pick the nearest one.

k* = arg mink
The index k of the entry that makes the distance smallest. “arg min” means: the argument that gives the minimum.
‖v − ek‖2
The squared distance between the feature v and entry ek: subtract coordinate by coordinate, square, and add.
zq = ek*
The quantized code: the winning entry itself. The decoder receives this vector, and the index k* is what a sequence model would predict.

The lecture works a toy case with four entries in two dimensions: e1 = (−1.0, 0.0), e2 = (0.0, 1.0), e3 = (0.7, −0.2) and e4 = (1.0, 1.0). The feature is v = (0.6, −0.1). Work it by hand. The squared distance to e3 is (0.6 − 0.7)2 + (−0.1 + 0.2)2 = 0.01 + 0.01 = 0.02. To e1 it is 2.56 + 0.01 = 2.57. To e2 it is 0.36 + 1.21 = 1.57. To e4 it is 0.16 + 1.21 = 1.37. Entry 3 wins, so k* = 3. Now move the feature yourself.

Snap to the nearest code

The four pink dots are the codebook. The teal dot is your continuous feature v. The solid green line goes to the winner. Drag the two sliders and find the borders where the winner changes.

0.60
−0.10

The codebook and the starting feature are the lecture’s toy values (slide 7). Real codebooks hold hundreds or thousands of learned entries in many more dimensions; the lecture marks these numbers as toy values too.

Notice what the decoder gets: e3 = (0.7, −0.2), not your v. The gap between them is the price of a code you can count. The payoff is that a sequence of indices is exactly what a language-style model can predict. Repeat the lookup at every feature position and you get a sequence or a grid of code indices. The encoder, the codebook and the decoder learn together from reconstruction and quantization objectives.

Two choices that people mix up

Slide 8 sorts the encoder into a small table. The code can be continuous or discrete. The encoding can be deterministic or stochastic. Deterministic and continuous is z = E(o). Deterministic and discrete is the nearest codebook index. Stochastic and continuous draws z ~ qφ(z | o), a distribution over vectors. Stochastic and discrete samples a categorical code, a draw among V labelled options.

The lecture then adds a sentence worth keeping: the encoder distribution and the future-prediction distribution are separate choices. A model can have a noisy encoder and a one-answer predictor, or a one-answer encoder and a noisy predictor. We will meet both.

Course note. Assignment 1 is due Thursday, September 24, and the class on September 22 may be on Zoom, so check Canvas for the exact time and link. Office hours this week moved to Friday, September 25 at 5:00 PM.

The road

The lecture has four parts, and this lesson follows them in order: the design space, autoregressive prediction, seven concrete models, and the failures of rollouts.

  1. State, belief and history: why one frame is not enough.
  2. Rolling the model forward: teacher forcing versus free-running rollout.
  3. Predicting a distribution: categorical codes, Gaussians and squared error.
  4. Mixtures for many futures: the mixture density network.
  5. Predicting in feature space: TD-MPC, V-JEPA 2-AC and LeWorldModel.
  6. The 2018 World Models system: vision, memory, dreaming.
  7. RSSM and the Dreamer family.
  8. Images as tokens: IRIS and Photon-1.
  9. Comparing designs and rollout failures.
  10. The field guide: numbers, code and an evaluation checklist.
A feature v sits between two codebook entries. Which code does vector quantization give the decoder?

Chapter 1

State, Belief and History

See why one frame cannot tell you which way a ball is moving, and compare the two ways a model can keep the past

You watch a video of a ball, and someone pauses it. The ball sits in the middle of the screen. Where will it be in one second? You cannot say. It might be rolling right. It might be rolling left. One frame holds the position, and the motion lives in the difference between frames.

This is the heart of the lecture’s first section, and it is the reason a world model needs memory. Before we add memory, let us see the one case where we already know exactly what to do.

The linear case, solved (slide 9)

In Lecture 4 we met the linear Gaussian state-space model (LGSSM). A hidden state st is the full truth about the world at time t, such as the ball’s position and speed. The state changes by a fixed linear rule plus noise, and a sensor sees a noisy linear view of it.

st+1 = A st + B at + ε
The next state is the old state pushed through the matrix A, plus the action pushed through B, plus process noise with covariance Q.
ot = C st + ε
The sensor reads a linear view of the state, C st, plus measurement noise with covariance R.

Because everything is Gaussian, the agent’s belief about the state, its best guess with an error bar, stays Gaussian forever. Two numbers describe it: a mean μ and a covariance P. The belief update has two steps, and the lecture lists the formulas.

Step 1, predict. Push the belief through the world model: μ− = Aμ + B a, and P− = A P A⊤ + Q. The error bar grows, because the world adds noise. Step 2, update. A new observation arrives. The innovation ν = o − Cμ− measures how wrong the prediction was. The Kalman gain K = P−C⊤S−1, with S = C P− C⊤ + R, says how much to trust the sensor. Then μ = μ− + Kν and P = P− − K S K⊤.

Here is one pass with plain numbers. Take A = B = C = 1, Q = 0.1, R = 0.5, a start of μ = 0 and P = 1, an action a = 1, and an observation o = 1.4. Predict: μ− = 1 and P− = 1.1. The sensor term is S = 1.1 + 0.5 = 1.6. The innovation is ν = 1.4 − 1 = 0.4. The gain is K = 1.1 / 1.6 = 0.6875. So the new mean is 1 + 0.6875 × 0.4 = 1.275, and the new covariance is 1.1 − 0.6875 × 1.6 × 0.6875 = 0.344. The error bar shrank from 1.1 to 0.344, because the sensor helped. The loop then repeats for the next action and observation.

μ = 1 + 0.6875 × 0.4 = 1.275,   P = 1.1 − 0.756 = 0.344 (toy numbers, ours)

Two challenges when the world is not linear (slide 10)

Real worlds have nonlinear dynamics and image observations. The lecture splits the new problem into two challenges. The first is the world model itself: how to represent the environment and predict it. In the general form, the next state is drawn from pθ(st+1 | st, at) and the observation from pθ(ot+1 | st+1). We must choose the state, the representation and the predictive distributions.

The second is the belief update: inferring the state from the history of observations. The history Ht = (o0:t, a0:t−1) is every observation so far and every earlier action. The belief now needs an integral over all earlier states, and a Bayes rule for the new observation. Those integrals may have no closed form, and a mean plus a covariance may no longer describe the belief. A note on symbols: the slide writes Ht for the history, and reserves ht for a recurrent memory, a hidden vector a network carries from step to step.

The lecture sums up the job in one line. We must design both the latent world model and how observation history informs predictions.

The same frame can hide different motion (slide 11)

The ball example is slide 11. History A shows a dot moving right across three frames. History B shows it moving left. The last frame in each history is the same image, a dot in the middle. Their futures differ. A predictor that sees only the last frame cannot tell them apart.

One frame or three

Each row is a history of three frames, then the true next frame (blue ring) and the predictor’s answer (warm dot). Frames the predictor cannot see are dimmed. Slide the number of frames it may use.

1

Toy numbers, ours: the dot moves 0.15 of the frame width per step, so history A ends at 0.50 and moves to 0.65, and history B ends at 0.50 and moves to 0.35. With two or more frames the predictor adds the last step again; with one frame the best single answer is the middle, 0.50.

Look at what the one-frame predictor does. It sees the dot at 0.50 in both rows, so it must give one answer for two different futures. The safest answer is the middle, 0.50. That is 0.15 wrong in both rows, and it matches neither future. With two frames the speed is visible, and the error drops to zero. The failure is not a flaw in the predictor’s arithmetic. The needed fact was never in its input.

Two ways to keep the past (slide 12)

So the model must keep information from the history. The lecture names two views of how. In the first, you describe state evolution. You infer a model state s̃t from the observed history Ht, apply modeled dynamics with the action, and get the next model state s̃t+1. The state is the memory, and it follows a rule you wrote down.

In the second you condition predictions on retained history. You encode the history of codes and actions, z≤t, a<t, feed it to a predictor network that holds an internal state ht, and read out the next code ẑt+1. A hat marks a prediction. Here the network’s own memory carries the past, with no hand-written dynamics.

Both retain information from history, and the lecture notes that the two views can overlap in one learned model. A recurrent network with a latent variable, as in chapters 6 and 7, is both at once. The practical test for any design is the one the device just ran: which facts about the past can the predictor reach?

Two histories end in the same last frame but lead to different futures. What is the cleanest fix?

Chapter 2

Rolling the Model Forward

Chain one-step guesses into a trajectory, and learn why training with the true past differs from running on your own

A weather forecaster says: it rains tomorrow. Asked about the day after, she does not wait for tomorrow to happen. She starts from her own forecast and predicts again. A third day follows from the second forecast. Each day leans on a guess, and each guess may be a little wrong.

That is a rollout: a trajectory made by running the model forward for many steps without new observations. It is how a world model is used for planning, so it is the thing we must get right. Slides 13 to 17 set up how rollouts work and where training differs from using.

Two ways to roll forward (slide 13)

Chapter 1 gave two ways to keep the past, and each has its own rollout. With an explicit model state, you infer a state s̃t from what you observed, then propagate it: the action at takes it to a predicted state s̃t+1, and the next action takes that to s̃t+2. The state carries the memory.

With history-conditioned prediction, you keep a growing list of codes. You start with the observed prefix …, zt. The model predicts one more code, and you append it: …, zt, ẑt+1. Then you predict again from the longer list. The slide adds a clear rule: actions are supplied at each step, earlier actions also stay in the history, and the hat marks a generated code. In both views the instruction is the same. At each step, use the model’s output to continue predicting the future.

What the model carries, and what teaches it (slide 14)

A subtle point hides here. The thing the model passes from step to step is its internal state. The thing that supervises the model is a target, such as the next observation, the next code, or a posterior target. The lecture says these need not be the same variable. A model can carry a rich hidden state and be graded only on a small readout of it. Keep that in mind: later models grade different things.

Frozen eyes or joint learning (slide 15)

Now the design question of where the code comes from. Suppose an encoder Eφ makes codes z and a predictor Fψ turns them into predictions ẑ, with a prediction loss against a target. There are two ways to train.

In pretrain, then freeze, you train the encoder first, for example as an autoencoder, then lock it. The prediction loss then updates the predictor only. The codes stay fixed, which is stable and cheap. In joint learning, both parts are trainable, and the prediction loss updates the encoder and the predictor together. The codes can bend toward whatever helps prediction, which is powerful and dangerous.

Why dangerous? The slide asks the question and leaves it open: a constant representation is easy to predict. If the encoder outputs the same code for every input, the predictor guesses it perfectly and the loss is zero. Chapter 5 gives the answer, and shows what each model uses to prevent it.

The product of one-step guesses (slide 16)

The lecture now defines the core idea. A rollout starts from the observed seed ẑ0 = z0. Every prediction uses earlier outputs and the supplied actions. Prediction can be deterministic, a direct output, or probabilistic, a draw from a predicted distribution.

ẑt+1 = fθ(…)
The deterministic case. A function f with weights θ computes the next code directly from the earlier outputs and the actions. No sampling.
∏ pθ(zt+1 | …)
The probabilistic case. The chance of a whole trajectory is the product of one-step chances, each conditioned on everything before it. This is the chain rule from Lecture 2.

The lecture names this pattern: it is autoregressive. “Auto” means self, and “regressive” means predicting from earlier values. The model reuses its earlier outputs to predict later ones. Language models do exactly the same with words.

Teacher forcing against free running (slide 17)

Here is the catch. During training you own the recording. You know the true code at every step, so you can hand the model the recorded past. That is teacher forcing: the next step always receives the recorded zt+1, the teacher’s answer, whatever the model just said. The loss compares each prediction with the recorded next code.

During a rollout you own no recording of the future. The first step reads the observed prefix, but the second step must read the model’s own generated code. The lecture’s summary is exact: teacher forcing feeds recorded history, rollout feeds generated history, and future observations are unavailable.

Step through a toy of this. The true codes are 0, 2, 4, 6, 8, 10, which grow by 2. Our model has learned a slightly wrong rule, “add 2.3 to the input”.

What feeds the next step

Each column is one prediction. Rows show the input the model reads, its answer, the recorded target, and the error. Switch between training and rollout, then add steps.

5

Toy numbers, ours: the true code at step t is 2t, and the model answers input + 2.3. This stands in for any model with a small, steady error; the lecture’s slide 17 shows the same two input flows without numbers.

In training mode every step is 0.3 off, and the error never grows, because each step starts from the truth. In rollout mode the first step is also 0.3 off. But the second step starts from a code already 0.3 too high, adds its own 0.3, and lands 0.6 off. By step 5 the rollout is 1.5 off, five times worse, although the model and its rule are identical. The hero at the top of this page shows the same effect at a different scale.

This gap is the training-rollout mismatch. The model is graded on one task, predicting from truth, and used on another, predicting from itself. Chapter 9 gives the remedies.

A world model has a small, steady error on every one-step prediction. What happens to its free-running rollout error as the horizon grows?

Chapter 3

Predicting a Distribution

Score a prediction with the negative log likelihood, for a discrete code and for a Gaussian, and see where squared error comes from

A model has to say what comes next, and it can answer in two ways. It can name one thing: the next code is 7. Or it can give odds: 7 is likely, 3 is possible, 11 is a long shot. The lecture calls the first answer deterministic and the second probabilistic, and it shows how to train each. We start with the case where the codes are whole numbers, as in Chapter 0’s codebook.

A distribution over discrete codes (slide 18)

Suppose the codebook holds V entries, and the model must pick the index of the next one. The network reads the context, which is the code history plus the action, and outputs V raw scores called logits. One score per code, any real number. A softmax turns scores into probabilities: exponentiate each score so it is positive, then divide by the total so they add to 1.

π = softmax(…)
A list of V probabilities, one per code, summing to 1. The function f with weights θ reads the context and makes the logits.
ℒ = −log πk*
The loss. Here k* is the code that really came next in the recording. A probability of 1 gives loss 0. A tiny probability gives a large loss.

This loss is the negative log likelihood (NLL). Likelihood means: how much probability did the model give to what actually happened? We take the logarithm so that products of many steps become sums, and we flip the sign so that smaller is better. Work it for four codes with logits 2.0, 1.0, 0.1 and −1.0. The exponentials are 7.39, 2.72, 1.11 and 0.37, which sum to 11.58. The probabilities are 0.64, 0.23, 0.10 and 0.03. If the true next code is the second one, the loss is about −ln 0.23 = 1.45. If it was the first, the loss is −ln 0.64 = 0.45. A model that backs the right answer pays little.

Now use the distribution. You can sample: draw k̂ from the categorical distribution, so code 1 comes up 64% of the time and code 4 comes up 3% of the time. Or you can take the argmax: always pick the largest probability. Sampling is stochastic and the argmax is deterministic. The lecture says it directly: prediction can be stochastic or deterministic. Sampling keeps the other futures alive in a rollout. The argmax always follows the single most likely path.

A distribution over continuous codes (slide 19)

What if the code is a list of real numbers? There is no finite list of options to put odds on. The standard move is to predict a Gaussian, the bell curve. The network outputs two things for the next code y = zt+1: a location μθ(c), the middle of the bell, and a spread Σθ(c), its width. The slide’s sentence is: predict location and spread from context.

The loss keeps the same idea. It is the NLL of the recorded next code y* under the bell: ℒ = −log pθ(y* | c). As before, you can sample ŷ ~ N(μ, Σ) for a stochastic step, or return the mean ŷ = μ for a deterministic one.

Score a prediction

Pick the kind of code. Move the sliders so the model puts its probability on the blue truth, and watch the loss fall. Then press Sample to draw one answer and Best guess for the deterministic answer.

1.0
2

Toy numbers, ours: six logits 2.2, 1.4, 0.7, 0.1, −0.5 and −1.2, multiplied by the sharpness; the Gaussian truth is y* = 1.0. The formulas are the lecture’s (slides 18 and 19); with sharpness 0 all six codes get 1/6.

Try the Gaussian by hand first. Take y* = 1.0, μ = 0.4 and σ = 0.5. The NLL of a one-number Gaussian is (y − μ)2 / (2σ2) + ½ ln(2πσ2). The first term is 0.36 / 0.5 = 0.72. The second is ½ ln(1.571) = 0.226. The total is 0.946. Now slide σ to 1.0: the first term drops to 0.18 and the second rises to 0.919, so the total is 1.099. The loss has two forces. A narrow bell punishes any miss hard. A wide bell pays a fixed fee for being vague.

Where squared error comes from (slide 20)

Now hold the spread fixed, σ2I. Then the second term is a constant C, and the NLL becomes ‖y − μθ(c)‖2 / (2σ2) + C. That is a scaled mean squared error (MSE), the plain “square the miss” loss. So training a deterministic network with MSE is the same as training a Gaussian with a fixed spread and keeping only its mean. If the spread is learned, the lecture notes that residual weighting and log-scale terms also matter.

What does MSE aim at? Over all the futures that could follow the same context, the squared error is smallest at their average. The slide states it: the best mean is μ*(c) = E[y | c], the conditional expectation. The lecture adds the warning that this mean need not be a typical future. If a ball can go to −1 or to +1 with equal chance, the MSE answer is 0, a place the ball never goes. Lecture 7’s blurry point predictor was exactly this.

Training scores the odds, rollout must choose

Notice a split in how the output is used. During training you never sample. You only ask how much probability the model gave to the recorded value, and you push that number up. During a rollout there is no recording, so you must choose a value to feed back in. The model gives odds, and something outside the model turns odds into one code.

The choice has a cost either way. Argmax always follows the single most likely path, so it is repeatable but can lock into a loop. Sampling keeps other futures alive, but it may draw an unlikely code, and that code then becomes the next input. Take a model that is right about its odds and gives its best code 0.64 at each of three steps. A sampled rollout follows the argmax path only 0.64 × 0.64 × 0.64 = 26% of the time. The other 74% of rollouts wander off it.

The spread σ works the same way for continuous codes. It is the model’s own statement of how much to trust its mean. A model that misses by 0.6 pays 0.946 with σ = 0.5, as in the example. The same miss with an overconfident σ = 0.2 costs 0.36 / 0.08 + ½ ln(2π × 0.04) = 4.5 − 0.69 = 3.81. So the loss rewards a model that knows how wrong it tends to be. That honest spread is what lets a rollout show a fan of futures instead of one thin line.

You train a deterministic network with mean squared error on a context that has two equally likely futures, at −1 and +1. What does it learn to output?

Chapter 4

Mixtures for Many Futures

Let the network output a weighted blend of Gaussians, train it by scoring the true next code, and sample from it to roll forward

A robot hand moves toward a cup, and a post stands in the way. In the recordings the hand goes around the post. Sometimes it passes on the left, and sometimes on the right. Both are good. The path through the middle hits the post.

Chapter 3 ended on the problem. A network trained with squared error returns the average of the two paths, which runs through the post. A single Gaussian does no better, because one bell has one hump, and the hump would sit in the middle too. We need an output that can say: either here, or there.

A blend of bells (slide 21)

The lecture’s answer is a mixture: a weighted sum of several Gaussians. Each component is one bell that describes one region of possible outcomes. Each weight says how likely that region is. With M components the density is a sum over them.

p(y | c)
The density of the next code y given the context c. A mixture can have several humps.
∑k=1M
Add up the M components.
πk(c)
The weight of component k. All M weights are positive and add to 1. They also depend on the context.
N(y; μk, Σk)
A Gaussian bell with its own centre μk and spread Σk, both read from the context.

For the robot hand, two components do the job. One bell sits left of the post, one sits right, and each carries weight 0.5. The middle gets almost no density, so the model never proposes a path through the post.

What the network outputs (slide 22)

A mixture density network (MDN) is an ordinary neural network whose last layer outputs these parameters. The lecture’s summary is exact: an MDN predicts parameters, and it does not output one chosen future. It reads the context c, which is the code history plus the current action, and returns three groups of numbers.

The weights π1:M come from a softmax over M logits, so they are positive and sum to 1, as in Chapter 3. The means μ1:M are unconstrained, because a centre can be anywhere. The scales σ1:M must be positive, so the network outputs a raw number and applies a softplus, the smooth function ln(1 + ex), which is always above zero. The slide adds a small ε. Why? A bell that shrinks to a needle has an enormous density at its centre, and the loss would chase that. The ε keeps every bell at least a little wide.

Count the numbers. For a code with 1 coordinate and M = 3, the network outputs 3 logits, 3 means and 3 raw scales: 9 numbers. For a code with 32 coordinates, each component needs 32 means and 32 scales (if the bells are axis-aligned), plus one logit, so the count grows quickly. Choose M as a design setting: too small and a real region goes unmodeled, too large and some bells spike on single examples.

The loss scores the truth (slide 23)

To train, we use the NLL from Chapter 3, now on the mixture. Take the recorded next code y* = zt+1. Evaluate every component at y*, multiply each by its weight, add, and take −log.

ℒ(c, y*) = −log [ ∑k πk(c) N(y*; μk(c), σk2(c)) ]

The slide stresses that training requires no component sampling. You never choose a component. Gradients flow through the sum, so every bell is nudged in proportion to how much it explained y*. Work one case. Let there be two components, weights 0.5 and 0.5, means −1 and +1, and scales 0.3 and 0.3. The true next code is y* = 1.1. The right-hand bell has density (1 / (0.3 × 2.507)) × exp(−0.5 × (0.1 / 0.3)2) = 1.330 × 0.946 = 1.258. The left-hand bell is 2.1 / 0.3 = 7 scales away, so its density is near zero. The mixture gives 0.5 × 1.258 = 0.629, and the loss is −ln 0.629 = 0.46. A single Gaussian with mean 0 and scale 1.0 gives density 0.218 at the same point, and a loss of 1.52. The mixture pays a third as much.

Fit a mixture

The blue ticks are 40 recorded next codes for one context: two clusters. Choose how many bells the network may use. The fit runs live. Then press Sample 20 to pick a bell, then a value.

Toy data, ours and seeded: 40 values from two bells at −1 and +1 with spread 0.25. The fit uses 60 steps of expectation maximization, a standard way to fit a mixture; a real MDN reaches a similar fit by gradient descent on the loss above. The NLL is the average over the 40 values, in nats.

Sampling and rolling forward (slide 24)

To roll forward, the model needs one code. It samples in two stages. First draw a component, k ~ Cat(π(ct)), which picks “left of the post” or “right of the post”. Then draw the code from that bell, ẑt+1 ~ N(μk, Σk). Then update the context, take the next supplied action, and repeat. Because the first draw commits to a branch, each rollout is a single plausible future.

The lecture closes the section with a pointer. Richer alternatives, diffusion and flow models, come in later lectures. The MDN is the simple member of that family, and it is the one the 2018 World Models system used.

In training an MDN, how is the loss computed from the recorded next code?

Chapter 5

Predicting in Feature Space

Study three models that predict the next feature directly, and find what stops their codes from collapsing to a constant

Think of a cook who tastes a stew and writes one line in a notebook: “salty, thick, hot”. Next to it she writes what she expects after adding water: “less salty, thinner, hot”. She never writes the full recipe or draws the pot. She predicts in her own short notes. The notes are the features, and the next three models do exactly this: they predict the next feature directly.

What we have built so far (slides 25 and 26)

The lecture pauses for a recap. A latent world model has three parts in a row. First, encode each observation into a code, ot → zt, either continuous or discrete, deterministic or stochastic. Second, retain predictive state, a context ct built from the code and action history, (z≤t, a<t) → ct. A model state, a belief or an internal memory carries it. Third, predict under an action, (ct, at) → ẑt+1, as a direct output or as a distribution plus a selection rule.

Training compares predictions with observed codes or observations. Rolling forward reuses the predicted state or code for the next action, and decodes to observations when the model has a decoder. The lecture names the four design choices: the representation, the retained state, the prediction rule and the training objective. Slide 26 then asks how seven real models answer them, and walks through each. This chapter takes the first three: TD-MPC, V-JEPA 2-AC and LeWorldModel.

TD-MPC: predict the next code, keep a slow copy (slides 27 to 31)

TD-MPC (Hansen, Wang and Su, 2022) is the simplest. Its encoder is deterministic: zt = Eθ(ot), and the input may include recent frames. Its dynamics network is deterministic too: ẑt+1 = Fθ(zt, at). A rollout reuses the predicted latent at the next step, so every step stays in latent space.

The training loss is a latent consistency loss: the squared L2 distance, the sum of squared differences, between the prediction and the code of the real next frame.

ℒcons
The consistency loss. It is small when the predicted next code matches the encoded real next frame.
Fθ(Eθ(ot), at)
The prediction: encode the current frame with the online encoder, then step the dynamics under the action.
sg[Eθ̄(ot+1)]
The target: encode the next frame with a slow copy of the encoder. The sg means stop gradient: treat the target as fixed, and send no learning signal through it.

The target comes from a target encoder, a slow-moving copy of the online encoder. It follows the online weights by an exponential moving average (EMA): θ̄ ← (1 − τ)θ̄ + τθ. With an illustrative τ = 0.01, a target weight of 1.00 facing an online weight of 2.00 moves to 0.99 × 1.00 + 0.01 × 2.00 = 1.01. The copy drifts slowly, so the target does not jump around under the predictor’s feet.

Now the warning on slide 30. The consistency loss alone permits collapse. If the encoder outputs one constant for every frame, E(o) = c, and the dynamics returns it, F(c, a) = c, then the loss is exactly zero. So the full TD-MPC objective also includes reward prediction and value learning. The rewards force the code to keep what matters for the task. The slide’s videos show the latent transition and the agent running (the Dog Run task), and the lecture’s note is that predictions stay in latent space while the task video shows the behavior in the environment.

V-JEPA 2-AC: freeze the encoder (slides 32 to 35)

V-JEPA 2 (Assran et al., 2025) is a video model, and the lecture focuses on its action-conditioned stage, V-JEPA 2-AC. Its encoder is a pretrained network with deterministic features, frozen during this stage. The predictor keeps a history of features, actions and robot poses inside itself, and predicts the next feature deterministically.

The loss uses L1 error, the sum of absolute differences, in two parts. The teacher-forced term predicts each next feature from the recorded history. The two-step rollout term feeds the first prediction back and checks the second one. So L = Lteacher forcing + Ltwo-step rollout, and both compare with the frozen encoder’s features of the real frame. This is Chapter 2’s remedy for the training-rollout gap in a small dose. The robot then uses feature-space predictions to choose actions. A frozen encoder cannot collapse, because nothing in this stage trains it.

LeWorldModel: learn it all, and spread the codes (slides 36 to 39)

LeWorldModel (Maes et al., 2026) goes the other way. It learns the encoder and the predictor jointly, end to end from pixels, with deterministic continuous features. The history lives inside the predictor, ẑt+1 = Fψ(zhistory, ahistory). The loss is MSE prediction plus a regularizer called SIGReg.

SIGReg answers Chapter 2’s open question, what stops a constant code. It takes many random one-dimensional projections of the codes in a batch, and pushes each projection toward a standard bell, N(0, 1). A constant code fails that test at once. Across samples, the embeddings are encouraged toward N(0, I). The lecture adds a careful caveat: this does not make the prediction p(zt+1 | history, action) Gaussian. It constrains how the codes spread across the data, not what the next code looks like. A separate visualization decoder, not part of training, lets you look at predicted frames. Control uses the latent predictions.

Why a constant code wins

Eight inputs get eight codes on the line. The scale s multiplies every code. Drag s toward zero, or press Train to let gradient descent decide. Then turn the spread penalty on and train again.

1.00

Illustrative toy, ours: the predictor misses by 0.3 on the scale of the codes, so the prediction loss is 0.09 s2. The penalty is 0.5 (s2 − 1)2, which asks the codes to have spread 1; real SIGReg tests random projections of the batch against N(0, 1) and is more general. The collapse itself is the lecture’s (slide 30).

Watch what the loss alone does. Its curve is a bowl with the bottom at s = 0, where all eight codes sit on one point and the loss is zero. The model has “solved” prediction by predicting nothing. With the penalty on, the bottom moves to s = 0.95, and the codes keep their spread. The minimum is a little under 1, because the prediction loss still tugs the codes inward a bit.

Which design lets a model train its encoder and predictor together on prediction loss alone without collapsing?

Chapter 6

The 2018 World Models System

Walk through vision, memory and dreaming in the model that made latent world models famous, in the order it is trained

Imagine you have driven the same track a thousand times. Now close your eyes. You can still “see” the next bend coming, and you can steer through it in your head. You do not replay pixels. You hold a feel for the road, and your feel moves forward as you imagine turning the wheel.

In 2018 David Ha and Jürgen Schmidhuber built a machine that does this. Their paper, World Models, let an agent learn a compact model of a game, then practise inside the model’s dream. Lecture 7 introduced the idea. Our world-models Gleam tells the whole story. Here we open the machine and follow how it is trained, step by step, with the lecture’s own diagrams in mind.

The parts: vision and memory (slides 41 and 42)

The system has a vision part (called V) and a memory part (called M). A small controller (C) reads their outputs and picks the action. The lecture’s slides cover V and M, because they are the world model. They hold every design choice the lecture asks about.

V is a variational autoencoder, as in our VAE lesson. It looks at one frame ot and gives a distribution over codes, qφ(zt | ot), then samples a stochastic code zt ~ qφ(· | ot). The slide shows driving frames from CarRacing next to their VAE reconstructions. The lecture’s note is worth keeping: the VAE encodes and reconstructs observed frames, and temporal prediction is the MDN-RNN’s job. V knows nothing about time.

M is a recurrent network, an RNN. It reads a code and an action and updates a hidden vector, the memory ht. The update is deterministic: ht = fθ(ht−1, zt−1, at−1). The roles are split cleanly. The code zt encodes the current frame. The memory ht summarizes earlier codes and actions. (If you want to see how an RNN carries a memory, our recurrent-network lesson builds one.)

From ht, M predicts the next code with a mixture density network, the head from Chapter 4. That MDN is the prior pθ(zt | ht): its belief about the code before it sees the frame. So the memory update is deterministic and the code update is probabilistic. The slide’s rule is the one to remember: while observing, sample the frame VAE posterior; during rollout, sample the MDN prior.

Sizes and data flow

Concretely, a camera frame is an array of pixels, say 64 × 64 × 3. V shrinks it to a code of tens of numbers, say z: 32. M keeps a memory a few hundred numbers wide, say h: 256, and its MDN outputs weights, means and scales for a few components (Chapter 4). These sizes are illustrative, to show the shape of the data; they are not the lecture’s. The point is the squeeze. The thing M predicts is a few dozen numbers, not twelve thousand.

Training in two stages (slide 43)

The system does not train V and M together. The slide lists two stages, and a third use.

Stage 1: train V. Collect frames from play. Train the VAE to reconstruct each frame, with the usual KL term from Lecture 7 (a score of how far the code’s distribution sits from a plain prior) that keeps the code close to that prior. No time, no actions.

Stage 2: freeze V, fit M. The lecture’s wording is: freeze V; fit M to recorded latent sequences. Run the frozen encoder over a recorded episode to turn every frame into a code. For each step, M reads the history z≤t, a≤t and must give high probability to the recorded next code. The loss is the MDN negative log likelihood from Chapter 4.

ℒM = −log pψ(zt+1 | z≤t, a≤t)

Notice that the frozen encoder appears twice on the slide: once for the frame at time t, and once for the frame at t + 1, whose code becomes the observed target. M scores the recorded next code. Because V is frozen, the target does not move while M learns.

Use: imagined rollouts (slide 44). Now cut the camera. Start from a real code and memory. Sample ẑt from the MDN prior, feed it and the next action into the RNN, get the next memory, sample again. The latent and the memory continue by themselves. If you want to look, send each ẑt through the VAE decoder to get a picture ôt. The slide labels it clearly: the decoded observation is a model output. It is not a real frame.

Three phases of one system

Pick a phase. Teal outlines learn. Blue outlines are frozen. Grey parts are not used in that phase. Press Dream one step in phase 3 to count imagined steps.

The three phases and their losses are the lecture’s (slides 41, 43 and 44). The box layout is ours, and the step counter only counts; it runs no network.

Why stage the training? Each stage has one clear job and a fixed target. V needs only pictures, and M never chases a code that is still changing. The price is that V learns what looks important, not what predicts well. If the car’s speed is a tiny detail in the picture, V may blur it away, and M can never get it back. The joint models of Chapter 5 and the RSSM of Chapter 7 exist to fix that.

In the 2018 World Models system, what does the second training stage do?

Chapter 7

RSSM and the Dreamer Family

Let the state posterior see the history, match it to a history-based prior, and imagine by sampling that prior

You wake up and, before you open the curtains, you guess the weather. Yesterday was grey, so you guess rain. That guess is your prior: what you expect before you look. Then you open the curtains and see sun. Your new belief is a posterior: your expectation, corrected by what you saw. A good guesser has a prior that lands close to the posterior, so the correction is small.

Chapter 6’s system gave each frame a code from the frame alone. It never used yesterday’s grey sky to read today’s picture. The model in this chapter, the recurrent state-space model (RSSM), does. It comes from PlaNet (Hafner et al., 2019) and the Dreamer family reuses it. The lecture compares it with World Models on one slide, and that comparison is our map.

One state with two parts (slides 45 and 46)

The RSSM keeps the same two ingredients as before: a deterministic RNN memory and a probabilistic latent. It names them ht and ut. The memory updates by ht = fθ(ht−1, ut−1, at−1). The pair s̃t = (ht, ut) is the model state. The memory carries history in a way that can be copied exactly. The latent ut carries what is random.

There are two ways to get ut, and they are the center of the model. A frame encoder first turns the picture into features, et = Eφ(ot). Then:

qφ(u | h, e)
The state posterior. It sees the memory h and the features e of the current frame. Use it when a real frame is available.
pθ(u | h)
The transition prior. It sees the memory only. Use it when there is no frame, which is imagination.
pθ(o | h, u)
The decoder. Compare Chapter 6: here the history h enters the decoder too, as the red arrows on the slide show.

Two modes of one machine (slides 47 and 48)

Slide 47 puts the two systems side by side. World Models 2018 draws zt ~ qφ(· | ot): the current frame only. The RSSM draws ut ~ qφ(· | ht, Eφ(ot)): the history plus the current observation. History helps the reading. If you know a car was braking, a blurry frame is easier to read.

Slide 48 shows imagining without observations. The encoder branch is cut. You sample ut ~ pθ(ut | ht) from the prior and continue with ht+1 = fθ(ht, ut, at). The lecture’s sentence: during imagination, sample the prior and continue; the observation branch is unavailable. This is the same two-mode rule as in Chapter 6, now with history on both branches.

The loss that glues them together (slide 49)

Now the question: how is the prior taught to be good? Slide 49 gives one loss per time step.

ℒt = Eq[ −log pθ(ot | ht, ut) ] + β DKL( qφ(ut | ht, et) ‖ pθ(ut | ht) )

The first term is reconstruction. Draw u from the posterior, decode, and score the real frame. The second term is a KL divergence, a measure of how far one distribution sits from another, scaled by a weight β. It pulls the posterior toward the prior and, because both are trained, it pulls the prior toward the posterior. So the prior learns to predict, from history alone, the code that the real frame would have produced. That is exactly what imagination needs. The lecture’s phrase: joint temporal training, where the posterior is matched to a history-dependent prior.

Compare the 2018 recipe. There the VAE KL pulled each frame code toward a fixed standard prior, and the memory model was fitted afterward in a second stage. Here observation modeling and temporal prediction train together, with the temporal prior in the KL.

For a feel of the numbers, take one-number Gaussians. Let the prior be N(0, 1) and the posterior N(0.8, 0.5). The KL of the posterior from the prior is ln(1 / 0.5) + (0.52 + 0.82) / (2 × 12) − 0.5 = 0.693 + 0.445 − 0.5 = 0.64 nats. If the real frame surprises the model more, so the posterior sits at 2.0, the KL grows to 0.693 + 2.125 − 0.5 = 2.32 nats.

Prior meets posterior

The purple curve is the prior, from history alone. The blue curve is the posterior, after seeing the frame. Slide the surprise, then press Train the prior a few times.

0.80
0.50
1.0

Illustrative toy, ours: one-number Gaussians and the closed-form KL, ln(σp / σq) + (σq2 + (μq − μp)2) / (2 σp2) − 1/2. Each press of Train moves the prior half way to the posterior, standing in for many gradient steps. The loss form is the lecture’s (slide 49).

Train the prior and watch the KL fall to zero. When the prior equals the posterior, imagination samples exactly what observation would have shown. Press Imagine 12 before and after training. Before, the dreamed values scatter around the wrong centre. After, they land where the posterior would.

The Dreamer family (slides 50 and 51)

Dreamer builds agents on this state. The lecture lists the world-model choices across versions. Dreamer V1 to V3 are recurrent state-space models. V1 uses a Gaussian stochastic state. V2 uses a categorical state, a choice among labelled codes as in Chapter 3, and a trick called KL balancing that weighs the two directions of the KL differently. V3 uses a categorical RSSM with more robust training. Dreamer 4 changes the architecture to a Transformer world model: video frames pass through a causal tokenizer into continuous video tokens, and block-causal dynamics map tokens plus supplied actions to the next tokens. Shortcut forcing trains the dynamics to denoise future representations in few generation steps. The lecture’s summary: V4 changes the world-model architecture, and latent imagination remains the shared idea.

What does the KL term in the RSSM loss train the transition prior to do?

Chapter 8

Images as Tokens: IRIS and Photon-1

Turn each frame into discrete codes and predict them with a Transformer, then count what that costs

Read this line: “the cat sat on the”. You expect the next word to be “mat”. A language model does the same thing, one word at a time, and it does it very well. Now suppose a video game screen were also a sentence. Each frame is a short string of words from a small dictionary, and a game is a long story told in those words. If so, a language model could predict the next frame.

That is the move of this chapter. Chapter 0 gave us the dictionary, a codebook. Chapter 3 gave us the loss for picking the next word. Two models put them together and replace the RNN with a Transformer, a network that reads a whole list of tokens at once and learns which earlier tokens matter for the next one.

IRIS: a game as a sequence of codes (slides 52 to 55)

IRIS (Micheli, Alonso and Fleuret, a 2022 preprint, ICLR 2023) has the title Transformers are Sample-Efficient World Models. It has four parts in a row, and the lecture draws each of them.

First, a discrete tokenizer: an encoder plus a codebook, as in Chapter 0, turns each frame ot into K codes, kt,1, …, kt,K. The encoder is deterministic: each patch is assigned to its nearest codebook entry. Second, an autoregressive Transformer reads the code history and the actions. Its state is the code and action history itself, held inside the sequence model, with no separate memory vector. Third, it outputs a categorical distribution over the next codes, and we sample or take the argmax, exactly as in Chapter 3. Fourth, an image decoder looks up each code’s vector in the codebook and decodes them to the next frame ôt+1.

The losses are separate. The autoencoder learns from reconstruction and codebook terms. The Transformer learns from token cross-entropy, which is the discrete negative log likelihood of Chapter 3 applied to each next code. The slide notes the training loop repeats updates of both the autoencoder and the Transformer, rather than training them once in sequence.

Generation runs in two orders, and the lecture points them out. The first order is across time: frame t, then the action at, then frame t + 1. The second is within each frame: code 1, then code 2, up to code K, each conditioned on the earlier ones. The slide’s figure shows a recorded trajectory above and a model rollout below that uses the recorded actions, with the first frames given as conditioning. An agent trained on these imagined trajectories then plays Atari games.

Photon-1: the same recipe at scale (slides 56 to 59)

Photon-1 (David Li and Jonathan Li, an Induction Labs technical report, July 2026) is titled Scaling Video Pretraining with Imagination Models. The lecture gives the size: 106 billion parameters with 5 billion active, 575 million frames, and 960 discrete tokens per frame. Its structure matches IRIS, and the slide lists the parts.

The encoder is a differential FSQ. FSQ, finite scalar quantization, is a cousin of Chapter 0’s lookup that rounds each number of the feature to a small set of levels, so no learned codebook is needed. The word “differential” means the report codes frame changes, not whole frames. The state is the token history inside the Transformer, and the prediction is probabilistic autoregressive next-token. The learning is video pretraining first, then later action supervised fine-tuning (SFT) and reinforcement learning (RL). One slide shows predicted ball positions in billiards after a fine-tune that rendered them.

Slide 58 lines the two up. They share a recipe: discrete visual representations plus autoregressive next-token prediction. They differ in training.

PartIRISPhoton-1
Visual representationVQ image codesFSQ codes of frame differences
Dynamics dataAction-labeled Atari interactionsVideo pretraining without action labels
ActionsCondition imagined next framesAfter SFT: predict next state, then action
Image decoderAutoencoder reconstruction; rollout renderingAuxiliary visualization only

The lecture adds a fair caution. Photon-1 is a scaled extension of this modeling family, and the representation and the training stages also change. So it is not simply IRIS made bigger.

What tokens cost (slide 66, shown here because it belongs to this design)

Tokens bring a price. Predicting a frame code by code is serial: code 2 waits for code 1. If a frame has K codes and you imagine H frames, you need about H × K prediction steps. Take 10 frames with Photon-1’s 960 codes each: 9,600 serial steps. The lecture’s remedy for this cost is frame or block prediction, where the model outputs all the features of a frame in one go. Compression, fewer tokens, helps too. Play with the counts below.

Count the serial steps

The bars use a log scale: each tick is ten times more steps. Choose codes per frame, then slide the number of imagined frames.

10

The 960 codes per frame is the lecture’s number for Photon-1; 16 and 256 are illustrative picks. The count H × K is the lecture’s (slide 66). Real steps differ in cost, so the lecture says to compare actual latency too.

The slide pairs each cost with a repair and an honest trade. Many serial token steps call for fewer tokens or frame prediction, and compression may lose information. Repeated computation over a long context calls for caching or context compression, which costs memory or forgets history. Many iterative sampling steps call for distillation or shortcut methods, which need extra training and come in later lectures.

A token-by-token model predicts 12 frames of 100 codes each. How many serial prediction steps does it need, and what does frame-at-a-time prediction cut that to?

Chapter 9

Comparing Designs and Rollout Failures

Line the seven models up, then name the ways a rollout goes wrong and the repair for each

A mechanic opens the hood of seven different cars. The engines look alike at first. Each has pistons, a fuel line and a cooling loop. But the details differ: where the fuel mixes, how the heat leaves, what happens when it gets cold. To pick a car for a trip, she needs a chart that lines the parts up. This chapter is that chart for world models, and then a list of the ways each engine fails on the road.

The seven side by side (slides 60 and 61)

Slides 60 and 61 hold two tables. We join them here, because they answer the four design questions of Chapter 5: what the representation and history are, how the next step is predicted, how the model is learned, and what keeps the codes useful.

ModelRepresentation and historyNext-step predictionHow it is learnedWhat constrains the codes
TD-MPCStacked frames, then a compact latent stateDeterministic latent transitionJoint, task-orientedConsistency plus reward and value signals
V-JEPA 2-ACFrozen features plus action and pose historyPoint prediction (L1 training)Pretrain, then freeze the encoderFixed pretrained target features
LeWorldModelJointly learned features plus causal historyPoint prediction (MSE training)Joint encoder and predictorPredictive objective plus SIGReg
World Models (2018)Frame VAE code plus RNN memoryMDN over the next codeVAE, then MDN-RNNReconstruction plus framewise VAE KL
RSSM / Dreamer V1 to V3Memory plus stochastic model statePrior; posterior when observations arriveJoint temporal inference and dynamicsObservation modeling plus prior and posterior agreement
IRISDiscrete image codes plus causal historyCategorical next-token probabilitiesSeparate losses; repeated updates of both modulesReconstruction through discrete codes
Photon-1Discrete frame-change codes plus causal historyCategorical next-token probabilitiesVideo pretraining, then action learningEncoder training details not specified in the report

Read down the columns. Three models predict one point. Two give a distribution, through a mixture or through a prior and posterior. Two give categorical odds over codes. Each model also answers the collapse question in its own way: fixed targets, task signals, a spread penalty, a KL term, or reconstruction. The lecture ends the pair of tables on one sentence: a learned representation becomes useful for world modeling through its temporal and action-conditioned role. A code is good when it helps predict the next step under an action, not when it merely rebuilds a picture. The lecture also warns that these are model-component comparisons, not complete agent descriptions.

Failure 1: training and rollout disagree (slide 62)

You met this in Chapter 2. Training reads recorded inputs, z1, z2, and aims at z3. A free-running rollout reads a seed and then its own outputs, ẑ1, ẑ2, and produces ẑ3. The remedy on the slide: train on multi-step predictions that feed generated outputs back as inputs. V-JEPA 2-AC does a little of it, with teacher forcing plus a two-step rollout loss. The slide’s footer points to two further ideas, scheduled sampling and diffusion forcing, which also mix generated inputs into training.

Failure 2: errors compound (slide 63)

The hero at the top is this slide. Let us derive why the rollout drifts. The truth is xt+1 = xt + at. The model is x̂t+1 = g x̂t + at. Define the error et = x̂t − xt. Subtract: et+1 = g x̂t − xt. Add and remove g xt: the right side is g(x̂t − xt) + (g − 1) xt. So:

et+1 = g et + (g − 1) xt

There are two terms. The second, (g − 1) xt, is the fresh mistake of this step. It is all there is if the model reads the true state, because then et = 0. The first, g et, is the old error passed forward and multiplied by g. With g = 1.08, the old error grows 8% each step on top of the new one. Run it: e1 = 0.08 × 1 = 0.08. Then e2 = 1.08 × 0.08 + 0.08 × 2 = 0.246. Keep going to e12 = 5.82, while the one-step error at step 12 is only 0.16. The lecture’s remedies: multi-step consistency in training, a validated rollout horizon, and state updates whenever a real observation arrives.

Failure 3: missing information and unfamiliar inputs (slide 64)

Two different problems share a slide. In the first, information is missing: the same observation arises from different histories, as in Chapter 1. The cure is relevant history and memory. In the second, the query lies outside the data coverage, a scene the model never saw. The cure is to expand coverage and to validate new conditions. The lecture’s sharp sentence: a richer output distribution cannot recover information that the representation discarded. A mixture on top of a code that lost the speed still has no speed to draw from. It adds that longer context and ensemble disagreement (several models trained apart, whose answers differ when the input is unfamiliar) can help, but neither guarantees reliable out-of-distribution prediction.

Failure 4: many futures, one sample (slide 65)

When the future is truly ambiguous, one rollout is one draw. The slide shows four imagined trajectories from the same first frame (IRIS, Figure 2). Use probabilistic dynamics and several samples. But the lecture adds a warning that this device tests: more randomness does not repair model bias. Evaluate the predicted distribution, not only one sample.

A fan of futures

Blue shows where the real system goes, a band of one spread either side. The warm lines are sampled rollouts of the model. Add samples and watch the average. Then raise the bias.

4
0.05
0.30

Illustrative toy, ours. The truth moves +1 per step with a random wobble of 0.3 per step. The model moves 1 plus its bias per step, with its own noise. The idea, that sampling does not fix bias, is the lecture’s (slide 65).

With bias zero, the average of the samples closes in on the true mean as you add samples, and the fan has about the width of the blue band. With bias 0.05, every step is 5% too fast, and after 12 steps the average sits 0.6 beyond the truth, whatever the sample count. More draws shrink the noise in the average. They never remove a bias that every draw shares.

Failure 5: the cost of tokens (slide 66)

Chapter 8 counted H × K serial steps for token-by-token rollout. The repairs trade speed for something else: compression can lose information, caching uses memory or forgets history, and distillation needs more training. Always compare real latency, because steps differ in cost.

In the error recursion et+1 = g et + (g − 1) xt, which term is the only one that remains when the model reads the true state at every step?

Chapter 10

The Field Guide

Review the lecture on one page: the numbers, an evaluation checklist, the core code, and where the course goes next

You can now say, from memory, what Lecture 8 built. A latent world model encodes each observation into a code, keeps a predictive state from the history, and predicts the next code under a supplied action. A rollout feeds those predictions back in, so training on recorded inputs (teacher forcing) differs from running on your own outputs. The prediction can be one point, a categorical distribution, a Gaussian or a mixture, and the loss is the negative log likelihood. Seven models answer the design questions differently, and each has its own way to stop the codes from collapsing. Every rollout still has to face compounding error, missing information, unfamiliar inputs, many futures and the serial cost of tokens.

The numbers and terms in one table

ItemValue or meaningWhere
Toy codebook lookupv = (0.6, −0.1) is nearest to e3 at squared distance 0.020, slide 7
Rollout error at gain 1.08g = 1.08: one-step error 0.160, rollout error 5.823 at step 12Hero, 9, slide 63
Error recursionet+1 = g et + (g − 1) xt9, slide 63
Discrete lossℒ = −log πk*3, slide 18
Fixed-spread GaussianNLL = ‖y − μ‖2 / (2σ2) + C, so the best mean is E[y | c]3, slide 20
Mixture loss−log ∑k πk N(y*; μk, σk2)4, slide 23
TD-MPC targetEMA copy, θ̄ ← (1 − τ)θ̄ + τθ, with stop gradient5, slide 29
V-JEPA 2-AC lossL1 teacher forcing plus L1 two-step rollout5, slide 34
RSSM loss−log p(o | h, u) + β KL(q ‖ p)7, slide 49
Photon-1 size106B parameters, 5B active, 575M frames, 960 tokens per frame8, slide 57
Serial costH frames × K codes is about HK steps8, 9, slide 66

How would you evaluate a world model? (slide 67)

The lecture ends with a discussion slide: choose one model and say which checks apply. Here are its five checks, each with the Chapter that explains why.

CheckConcrete testWhy
Data splitHold out entire episodes, not adjacent frames.Neighbouring frames look alike, so a random split leaks the answer.
Rollout horizonSame initial evidence and actions; compare one-step and H-step rollout.Chapter 9: one-step error hides compounding.
Probabilistic outputUse held-out NLL when available; check calibration or coverage (do events given 70% odds happen about 70% of the time, and does the stated range contain the truth as often as it claims).Chapters 3 and 4: score the whole distribution.
Representation scaleRaw latent MSE is not comparable across different encoders.Chapter 5: shrinking the codes shrinks the loss.
Use conditionsTest new scenes and actions separately; measure generation latency.Chapter 9: coverage and serial cost.

Try it on one model. For IRIS you would hold out whole Atari episodes, compare the 1-step and 20-step rollouts from the same first frames, report token cross-entropy on held-out episodes, and time the generation. For TD-MPC you could not compare its latent MSE with LeWorldModel’s, since the two encoders have different scales. You would compare the control results and the rollouts decoded to the same space.

The core as code

Three small pieces carry the whole lecture: the mixture loss, a teacher-forced training step, and a dream that feeds the model its own samples. This is a sketch for one code coordinate at a time, in PyTorch style, not a full implementation.

python
import math, torch, torch.nn.functional as F

def mdn_nll(logits, mu, raw_sigma, y):
    # logits, mu, raw_sigma: (B, M)   y: (B, 1), the recorded next code
    log_pi = F.log_softmax(logits, dim=-1)                 # weights: positive, sum to 1
    sigma  = F.softplus(raw_sigma) + 1e-3                  # positive scales, with a floor
    log_n  = -0.5*((y - mu)/sigma)**2 - torch.log(sigma) - 0.5*math.log(2*math.pi)
    return -torch.logsumexp(log_pi + log_n, dim=-1).mean()   # score every bell; no sampling

def train_step(model, z, a):                            # z: (B, T+1, D) recorded codes
    h, loss = model.init_state(z.size(0)), 0.0
    for t in range(z.size(1) - 1):
        h, p = model.step(h, z[:, t], a[:, t])             # input is the RECORDED z_t
        loss = loss + mdn_nll(*p, z[:, t+1])               # target is the recorded z_{t+1}
    return loss / (z.size(1) - 1)

@torch.no_grad()
def dream(model, z0, actions):                          # no camera: the model reads itself
    h, z, out = model.init_state(z0.size(0)), z0, [z0]
    for a in actions:
        h, (logits, mu, raw_sigma) = model.step(h, z, a)
        k = torch.distributions.Categorical(logits=logits).sample()        # pick a bell
        sigma = F.softplus(raw_sigma.gather(-1, k[:, None])) + 1e-3
        z = mu.gather(-1, k[:, None]) + sigma * torch.randn_like(sigma)    # draw from it
        out.append(z)                                      # z feeds the next step
    return torch.stack(out, dim=1)

Find the three ideas in the code. The first function never samples (Chapter 4). The second feeds recorded codes (Chapter 2, teacher forcing). The third feeds its own samples, which is exactly why Chapter 9’s failures appear.

Where it sits, and what comes next

This lecture is the middle of a bridge. Lecture 7 gave us latent variables, the ELBO and a first look at World Models. Lecture 4 gave us the state-space and belief ideas that Chapter 1 reused. The lecture’s last line names the next step: use predicted futures for planning, then learn behavior with world models. Lecture 9 takes up planning and control, and Lecture 10 turns to policies and value with world models.

Keep going

Credit: this lesson re-teaches Lecture 8 of Penn CIS 6280: World Models (Fall 2026) by Jiatao Gu. The structure follows the lecture’s own, and every device and example is ours except where a note says the number is the lecture’s. The slides are on the course site, and the papers named in each chapter are the primary sources.

The takeaway. A world model is judged by its rollout, not by its one-step loss. Choose what the code keeps, keep enough history, predict a distribution when the future is ambiguous, and test the model on its own outputs.
Why does the lecture say to hold out entire episodes when you test a world model, instead of random frames?

Now press Teach and explain, from memory, how a latent world model is built and run: the code, the memory, the prediction rule, the loss, and the rollout. Then go back to the instrument at the top, set the gain to 1.15, and say why the warm line leaves the green one.

CIS 6280 Lecture 8: Latent World Models
The course site · Lecture slides · Back to Gleams