Policy and Value Learning with World Models

A world model only predicts. To act, an agent can search plans with the cross-entropy method, or learn a policy and a value in imagination. Start with the search: refit a Gaussian and watch the cost fall.

Revisit representations, latent prediction, LeWM, RSSM and CEM, then see how Dreamer turns imagined futures into a reusable policy and a value.

Press Run one round a few times with CEM, then switch to random guessing and run the same rounds. Watch the green plans crowd toward the blue diamond, the best answer. Then we build, piece by piece, everything underneath.

You need a feel for averages (add up, divide by how many) and a little Python for the code. We build the rest from zero.

Based on Lecture 10 of Penn CIS 6280: World Models (Fall 2026), taught by Jiatao Gu. Lecture slides. Every idea is re-taught here in our own words and devices; it follows Lecture 9, Planning and Control, and revisits Lectures 6 to 9.

Search for a plan

The toy system and its cost come from the lecture’s classroom example (slides 56 to 60): a point at position 0 and speed 0 must reach position 1 at speed 0 with two actions a0 and a1, each between −1 and 1, and the cost is (a0 − 1)² + (a0 + a1)² + 0.1(a0² + a1²). The lecture works four plans by hand; here 24 plans are drawn per round and the best 6 are kept, with seeded random numbers. The best cost is found on a fine grid.

Chapter 0

Predicting Is Not Choosing

See why a model that predicts the future still needs a second tool to pick an action, and meet the four parts of every agent

Picture a game of Flappy Bird. A bird falls a little every frame. Tap, and it hops up. Green pipes slide in from the right, each with a gap, and the bird must pass through every gap. You have a fraction of a second to decide: tap, or wait?

Now suppose you have a very good world model: a learned program that answers one question. “If the world looks like this and I do that, what will it look like next?” Hand it a picture of the bird and the word “tap”, and it draws the next frame. That is a real gift. It is also not enough.

The model tells you what each choice would do. It never tells you which choice to make. A map shows every road, but you still have to pick a route. This lecture is about picking the route.

One decision, repeated

The lecture opens with the Flappy Bird demo, and it shows three steps that repeat. First, the learned model predicts several possible futures, one for each sequence of tap or no tap. Second, a search called CEM (the cross-entropy method, Chapter 6) picks the sequence whose predicted future looks best. Third, the bird executes only the first action of that sequence, looks at the new frame, and starts again.

Why throw away the rest of the plan? Because new pipes keep arriving, and because the model is never perfect. A plan made now is a guess about the next few frames. After one real step you know more, so you can guess better. This habit has a name: replanning, or model predictive control (MPC). The plan looks a fixed number of steps ahead, called the horizon, and the horizon slides forward as the bird moves. The slide sums it up in four words: finite planning horizon, repeated control.

Try it. The device below is a toy of the same loop. A point must follow a wandering target. The planner picks five actions ahead, but the real world adds a small push the model did not predict. You choose how many of the five actions to execute before you plan again.

Replan sooner or later

The blue line is the target gap. The teal line is the bird. Short marks on the floor show where it replanned. The bars below show the average miss for 1 to 5 executed actions. Drag both sliders.

5
0.12

Illustrative toy, not the lecture’s Flappy Bird model. The point moves by its action plus a random push of the size you choose; the planner assumes no push and moves toward the next target value, at most 0.4 per step. The same 80 random pushes are used for every bar, so the bars differ only by how often the plan is redone.

With no surprises, every bar is the same: a perfect model needs no replanning. Add surprises and the bars climb from left to right. Each action you execute without looking carries any unseen push forward, and the pushes pile up. Replanning after every action resets the pile to zero each time.

Read the failure in three parts. The symptom: the bird drifts off the gap centre and clips a pipe. The measurement: the average miss grows with the number of unchecked actions. The fix: observe more often, even if each plan is thrown away. Planning is cheap compared with a crash.

Two ways to act with a world model

Flappy Bird replans after one action. The lecture’s second demo, PushT, executes the whole plan. In PushT a flat robot pusher must slide a T-shaped block onto a goal outline. The clip on slide 4 shows the full plan run without any feedback, which is fine because the model is good enough over those few steps.

Both demos plan with CEM. The slide then names a second way to act. Instead of searching for a plan at every decision, learn two things once and reuse them. A policy is a rule that looks at the current state and says which action to take, like a habit. A value is a number that says how much total reward to expect from a state if you follow the policy, like a gut feeling that a position is good. Dreamer, the paper from 2020 that ends this lecture, learns both inside its world model. Model-based reinforcement learning (the lesson on model-based RL builds it from zero) can use either way, or mix them.

The four interfaces

Slide 5 breaks every agent in this course into four parts, each defined by what goes in and what comes out. It is worth memorising, because the rest of the lecture fills in one row at a time.

PartGoes inComes outChapters
RepresentationAn image or a history of observationsJEPA features, or a VAE posterior1, 2
DynamicsLatent context and actionsPredicted future states3, 4, 5
ObjectiveA predicted outcome and the taskA cost or a return5, 6, 8
Action selectionThe model and the objectiveAn action plan, or a policy action6, 7, 8

In shapes: a camera frame of 3 × 224 × 224 numbers goes into the representation and comes out as a short vector D, say 192 numbers. The dynamics part takes that vector and an action block and returns another vector of size D. The objective turns the final vector into one number. Action selection tries many actions and keeps the ones with the best number.

One short paragraph of logistics

Slide 2 lists four notices. Assignment 2 releases on September 30. Final project details come after fall break. Project registration is due before October 14, and groups of two to four are encouraged. Part of the material first planned for this lecture moves to after fall break, which is why the value derivation on slide 69 says “the full derivation follows after fall break”. Today’s revisit of Lectures 6 to 9 supports Assignment 2.

The road

  1. Learned targets: JEPA, the teacher, collapse and the SIGReg fix (slides 6 to 14).
  2. The VAE: reconstruct an image, and the KL price of the code (slides 15 to 20).
  3. One LeWM training step: tensors, alignment, loss, and the no-detach rule (slides 21 to 32).
  4. Rollouts: teacher forcing against open-loop prediction (slides 33 and 34).
  5. RSSM and PlaNet: memory, reward labels, and scoring imagined plans (slides 35 to 43).
  6. Goal cost and CEM: every step of one planning decision (slides 44 to 55).
  7. CEM by hand: four plans, two elites, one refit, and how it fails (slides 56 to 65).
  8. Value, policy and Dreamer (slides 66 to 72).
  9. The field guide: numbers, checklist, code, and where to go next.
Flappy Bird plans five actions with CEM but executes only the first one. Why does it replan so often?

Chapter 1

Learned Targets and the Collapse Problem

Train an encoder to predict features instead of pixels, and see why it needs a rule that keeps every image from getting the same code

You want to teach a student to sort photos, but you have no labels. So you set a game: look at part of a photo, then guess what the rest looks like. The student scores a point when the guess matches. Soon the student finds a cheat. Describe every photo as “grey blob”, and every guess matches perfectly. The score is perfect and the student has learned nothing.

This cheat has a name, collapse, and it is the main danger in this chapter. The lecture’s recap of Lectures 5 and 6 asks one question of every method: what supplies the target? Slide 6 sorts the methods into three families by the answer.

FamilyWhat supplies the target?Examples
ReconstructionThe observed data, or the missing pixelsAutoencoders, MAE; the VAE adds a probabilistic latent
ContrastiveThe matching view among many candidatesSimCLR, CLIP
Learned targetsAnother network’s encoding of a view or regionDINO, I-JEPA, V-JEPA, LeJEPA

Today uses the third family for LeWM, the world model of Chapters 3 and 4, and the first family for the RSSM route of Chapter 5. This chapter handles the third. (The Lecture 6 lesson and the JEPA lesson build these from scratch.)

JEPA: predict the features, not the pixels

A JEPA (joint embedding predictive architecture) hides part of an image. A context encoder reads the visible patches and turns them into feature vectors. A predictor then guesses the feature vector of each hidden block, given where that block sits. A second encoder, the teacher, reads the full image and produces the true feature vectors. Slide 8 gives the I-JEPA loss: the average squared distance between the predicted and the teacher features at the masked locations. The teacher features are normalised, and no gradient flows into them.

Two terms need care. A gradient says which way to nudge each weight to make a loss smaller. A stop-gradient treats the teacher’s output as a fixed number: the loss may compare to it, but never changes it. Without it, the easiest way to cut the loss is to change the answer key.

The teacher still has to improve, or the targets would stay random. Slide 9 shows how. After each training step, the teacher copies a little of the student:

θteachernew
The teacher’s weight after this step.
m
How much of the old teacher to keep. The slide uses 0.9.
θteacherold
The teacher’s weight before this step.
θstudentnew
The student’s weight after backpropagation has just updated it.

0.9 × 2 + 0.1 × 4 = 2.2 (old teacher weight 2, new student weight 4)

This is an exponential moving average (EMA): the teacher is a slow-moving shadow of the student. The two tools have different jobs. Backpropagation updates the student and the predictor. The stop-gradient keeps the loss from touching the teacher. The EMA is the only way the teacher changes.

Why agreeing is not enough

Slide 10 states the trap in two lines. Suppose the encoder maps every input x to the same constant c, so E(x) = c. Suppose the predictor just returns its input, so F(c) = c. Then the predicted feature equals the target feature for every image, and the prediction loss is exactly zero. The features match perfectly and carry no information about the input.

So a useful recipe must also keep the codes of different images different. Contrastive methods do this by pushing wrong pairs apart. I-JEPA leans on the teacher, the EMA and the architecture. LeJEPA (slide 11) adds an explicit rule, and the lecture writes it as a mix of two losses (its slide calls the weight ε; we write λ):

ℒ
The training loss.
λ
A weight between 0 and 1 that trades the two terms.
ℒagreement
Two views of the same example should get close codes.
λ
The same weight, now on the second term.
ℒSIGReg
The whole batch of codes should follow a prescribed shape, a round Gaussian cloud.

SIGReg: shape the whole cloud

Agreement looks at one example at a time. SIGReg looks at the batch. Collect the codes of B encoded views into a matrix Z. Then push the cloud of B points toward N(0, I), a Gaussian (a bell-shaped spread) centred at zero with unit spread in every direction. A collapsed batch is a single point, so it is very far from that shape.

Comparing a cloud in D dimensions with a Gaussian is hard. Slide 13 makes it easy. Pick a random unit direction u, and measure each point’s position along it: h = Zu. That gives B plain numbers. Compare those B numbers with a one-dimensional N(0, 1). Repeat for many random directions. If every projection looks Gaussian, the whole cloud does. Try it on a batch of 48 two-dimensional codes.

Agree, but collapse

Left: 48 codes in two dimensions. Right: their positions along one random direction, against the bell curve of N(0, 1). Drag the spread to zero, or squash the cloud into a line, and compare the two losses.

1.00
1.00

Illustrative toy. The second view of each code is the first plus a small random change that scales with the spread, so shrinking the cloud also shrinks the agreement loss. The shape score is our stand-in for SIGReg: for 16 fixed random directions, the average squared gap between the sorted projections and the matching quantiles of N(0, 1). The lecture only says SIGReg compares projections to N(0, 1); the LeJEPA paper’s own statistic is different.

At spread 0, the agreement loss is 0.00, a perfect score, and the shape score is at its worst. At spread 1 and roundness 1 the shape score is near 0.1 and the agreement loss is small but not zero. A line is cleverer: it has plenty of spread in one direction, so projections on some directions look fine. Other directions expose it. That is why SIGReg averages over many directions.

JEPA against the VAE

Slide 14 closes the first half with a comparison. A JEPA code must keep what is needed to predict target features. A VAE code must keep what is needed to reconstruct the image. A JEPA gives one feature vector z = E(o). A VAE gives a whole distribution q(z | o). SIGReg puts its Gaussian across different examples. The VAE puts a Gaussian on each image’s posterior, compared with a shared prior. Each prepares a later part of the lecture: JEPA for LeWM, and the VAE for the RSSM.

An encoder gives every image the code c, and the predictor returns its input. What is the prediction loss, and what does it tell you?

Chapter 2

The VAE: Reconstruct, and Pay for the Code

Learn a code that rebuilds the image, then see why a second term charges the code for every bit it uses

You describe a photo of a brick corridor to a friend over the phone. You may say only eight numbers. Your friend must redraw the corridor from them. If you pick the numbers well, the redraw looks close. If you may say as many numbers as you like, you could read out every pixel, and you would have learned nothing about corridors.

A variational autoencoder (VAE) plays this game. It has two networks. The encoder turns an observation o (an image) into a short code z, called the latent. The decoder turns z back into an image. Training makes the rebuilt image match the original (slide 15). The lecture revisits it now because the RSSM of Chapter 5 uses the same machinery.

The code is a distribution, not a point

A plain autoencoder gives one code per image. The VAE encoder gives a belief. For one fixed image it outputs a mean μ(o) and a spread σ(o) for every coordinate of the code. Together they define q(z | o), the posterior: “given this image, the code is probably near here, give or take this much”. To rebuild, you sample one z from q and hand it to the decoder.

Sampling raises a puzzle. Training needs gradients, and a random draw has none: you cannot differentiate “pick a random number”. Slide 16 shows the standard way out, the reparameterization trick. Draw the randomness first, from a fixed distribution that does not depend on the encoder, and build z from it.

z
The sampled code that goes into the decoder.
μ(o)
The encoder’s best guess for the code of this image.
σ(o)
How unsure the encoder is, per coordinate. The ⊙ sign means multiply coordinate by coordinate.
ε
A fresh draw of standard noise. It never touches the encoder’s weights.

μ = 0.5, σ = 0.2, ε = −1.5  ⇒  z = 0.5 + 0.2 × (−1.5) = 0.2

Now z is a plain formula in μ and σ. The slope of z with respect to μ is 1. The slope with respect to σ is ε, here −1.5. The decoder’s loss can send gradients back through both. The noise is an input, like the image itself.

The loss has two jobs

Slide 17 writes the VAE loss as two terms. The first is the reconstruction term, the average of −log p(o | z): how surprised the decoder is by the true image, given the code. It rewards a code that carries information about the image. The second is a KL divergence, a measure of how far one distribution sits from another. It compares the image’s posterior q(z | o) with the prior p(z) = N(0, I), the round Gaussian that stands for “a code before seeing any image”.

Why reconstruct the image that went in? Slide 18 answers with a fact about the decoder. If the decoder is a Gaussian with a fixed spread, then −log p(o | z) is proportional to the squared error between the decoded and the true image. So “maximise the likelihood” and “make the squared pixel error small” are the same instruction. The target is the same image the encoder saw. Nobody supplies a label.

Why a KL term at all? Without it, the encoder can make σ tiny and spread the means of different images far apart. Each image gets its own private, exact code, and the decoder rebuilds perfectly. That is the “read out every pixel” cheat. The KL charges for every way a posterior departs from the prior. Slide 19 gives the price for a Gaussian posterior:

DKL
The price of this image’s posterior, in nats (a unit of information, like bits but with natural logs).
½
A constant from the Gaussian formula.
∑j
Add up the price of every code coordinate j.
μj2
Charges for a mean away from zero.
σj2
Charges for a spread larger than 1.
log σj2
Charges for a spread smaller than 1. A very sharp code costs a lot.

μ = 1, σ = 0.5: ½ [1 + 0.25 − (−1.386) − 1] = ½ × 1.636 = 0.818 nats

The price is exactly zero when μ = 0 and σ² = 1, that is, when the posterior equals the prior and the code says nothing about the image. Every bit of information in the code has to earn its price by helping the reconstruction. Slide 19 calls this “a balance between reconstruction and an image-dependent code”. Find that balance below for a one-number code.

Find the code

The image is the number 1.5 (blue line). The decoder just returns z. Drag the mean and the spread of the code. Purple is your code q, pink is the prior N(0, 1). Then press Draw z to sample with reparameterization.

1.50
0.20

Illustrative toy with exact formulas. Reconstruction cost is the expected value of ½(z − 1.5)², which is ½[(μ − 1.5)² + σ²]; the KL is the lecture’s formula for one coordinate. The best code is where both slopes are zero: μ = 0.75 and σ = 0.707.

Start with the mean on the image and a tiny spread, as the device does. Reconstruction is almost free, at 0.02, but the KL costs about 2.25 nats because a sharp, off-centre code is very different from the prior. Press the best-code button. The mean moves halfway to the image, to 0.75, and the spread widens to 0.707. The decoder is now a little off, and the code is much cheaper. The total falls from 2.27 to 0.91.

Read the failure in three parts. The symptom: reconstructions look perfect but samples from the prior look like noise. The measurement: a large KL per image, with σ far below 1. The fix: keep the KL term, and watch the two terms separately rather than only their sum.

Two losses, still no actions

Slide 20 lays JEPA and VAE side by side. JEPA compares a learned feature of another view or region. The VAE compares the observed input under a decoder likelihood. JEPA usually outputs one embedding. The VAE outputs a distribution q(z | o). JEPA’s distribution constraint is recipe dependent, for example SIGReg across the batch. The VAE pays a KL to p(z) per observation. JEPA learns features that are easy to predict. The VAE learns inference and a generative latent model. Both so far describe a single frame. Neither knows what an action does. The next chapter adds actions. (For the full VAE story, see the Lecture 7 lesson and VAE and VQ-VAE.)

The reparameterization trick writes z = μ + σ × ε. What does this buy us?

Chapter 3

One LeWM Training Step

Follow a recorded log of images and actions through the encoder and predictor, and see exactly which numbers the loss compares

A robot arm pushes a T-shaped block across a table. A camera records every frame, and the controller records every push. That log is all the data you have. You do not touch the robot again. From this log alone, you must build a model that can later answer: “if the block looks like this and I push like that, what will it look like next?”

The model in the lecture is LeWM (LeWorldModel, Maes et al. 2026). It has two learned parts. The encoder E turns each frame into a vector z, the embedding. The predictor F takes the vectors seen so far plus the recorded actions and guesses the next vector. Slide 22 draws it in one line: “predict the next latent from this state and action”, written ẑt+1 = F(zt, at).

Training and planning change different things

Before the details, slide 21 draws a line that the rest of the lecture respects. Training and planning are two different jobs.

Model trainingPlanning at time t
InputsRecorded images and actionsThe current context and a goal image
ComparesPredicted and observed next embeddingsThe predicted outcome and the desired goal
UpdatesEncoder and predictor weightsThe mean and variance of an action proposal
Real worldData already collectedMoves only when actions run

Keep it in mind: this chapter changes weights, and the model never leaves the log. Chapter 6 freezes the weights and changes the actions.

The tensors in one batch

A tensor is a grid of numbers with a fixed shape. Slide 23 lists the four that one training batch holds. Here B is the batch size, T the number of frames in a clip, D the embedding size, and A_block the numbers in one action block.

TensorShapeMeaning
obsB × T × 3 × 224 × 224Recorded images
actionsB × T × A_blockRecorded action blocks
embB × T × DThe encoded observed images
pred, targetB × (T − 1) × DAligned next-step pairs

Why T − 1 at the end? A clip of T frames has only T − 1 “next frames” to predict. The last frame has no successor in the clip.

The predictor reads history, never the future

Slide 24 quotes the paper. The predictor is a transformer with 6 layers, 16 attention heads and 10% dropout, about 10 million parameters. It reads a history of frame embeddings with a causal mask: each position may look only at itself and earlier positions. It brings in the action through AdaLN (adaptive layer normalisation), where the action sets the scale and shift inside each layer. The AdaLN weights start at zero, which the paper says steadies training. A projector follows the predictor, the same design as the one after the encoder. All parts learn jointly under one loss.

The mask matters because a model that can peek at frame t + 1 while predicting it would score perfectly and learn nothing. Slide 25 shows the alignment for four frames. Press a row below to see what each output may use.

What may each output see?

Four observed frames z0 to z3. Pick an output row. Teal cells are the frames and actions it may read; the green cell is its target.

Slide 25’s table, drawn. The fourth output has no target in a four-frame slice, so it is left out of the loss: that is why the loss uses pred[:, :-1] against emb[:, 1:].

The device shows the rule. Output k reads frames z0 to zk and actions u0 to uk, and its target is zk+1. The last output has no target, so it drops out. That is why the loss pairs pred[:, :-1] with emb[:, 1:].

The loss, by hand

Slide 26 gives the prediction loss: the squared distance between the predicted next embedding and the encoder’s embedding of the real next frame. The prediction is the model under the recorded action. The target is the encoder output for the real next image. Slide 27 works one case with D = 2. The prediction is (0.2, −0.1) and the target is (0.5, 0.3).

(0.2 − 0.5)² + (−0.1 − 0.3)² = 0.09 + 0.16 = 0.25  ⇒  MSE = 0.25 / 2 = 0.125

One number comes out. It sends gradients to two places. Slide 29 states the part that surprises people: the target is not detached. The loss is ‖F(E(o), u) − E(onext)‖², and the encoder E appears on both sides. So the gradient with respect to the encoder weights has two paths: the prediction path and the target path. The recorded action and the raw images stay fixed. LeWM updates the encoder and the predictor together, with no stop-gradient.

That makes the shortcut of slide 28 dangerous. If the encoder gives every observation the same code, E(o) = c, and the predictor returns c, then the loss is exactly zero. The next term closes this door. Slide 30 adds SIGReg from Chapter 1, applied across the batch at each time slice.

Both branches get the gradient

The warm dot is the prediction. The blue dot is the target code. Press Train one step and watch both move. Turn on the stop-gradient and the target stays still. Try the constant encoder last.

The starting values are slide 27’s: prediction (0.2, −0.1), target (0.5, 0.3), D = 2. Each step moves a point by half of the gap: the gradient of ‖a − b‖²/D with respect to a is 2(a − b)/D, which equals a − b when D = 2, and the learning rate is 0.5. In the real model the codes come from a shared encoder, so this two-dot picture is a simplification. The SIGReg stand-in in the read-out is the penalty (s − 1)², where s is the spread of a batch of codes; with every code equal, s = 0.

Without the stop-gradient, one step brings both dots to the midpoint, (0.35, 0.10), and the loss falls to zero. With it, only the prediction moves, and it closes half the gap each step while the target never moves. So without it, the target code itself drifts. That drift is the lever of collapse: the easiest way to shrink the loss is to move the answer key. Press the constant encoder. The loss reads 0.000, a perfect score, and the SIGReg stand-in reads its maximum 1.0, weighted by λ = 0.1.

Read the failure in three parts. The symptom: training loss near zero and planning that never reaches any goal. The measurement: the spread of the batch’s embeddings, which falls toward zero. The fix: the SIGReg term, with weight 0.1 in the lecture’s code.

The whole step in code

Slide 31 makes two side remarks. LeWM predicts the next embedding and regularises embeddings with SIGReg. To see its predictions as pictures, fit a separate decoder afterwards. In the paper’s PushT ablation (Table 7), LeWM without a decoder loss scored a success rate of 96.0 ± 2.83, and with a pixel loss added it scored 86.0 ± 7.54. The slide is careful: this is specific to the experiment, and the RSSM route of Chapter 5 does use image reconstruction. Slide 32 then shows the step as six lines.

# one LeWM training step. Shapes: obs [B,T,3,224,224], actions [B,T,A_block]
emb  = encoder(obs)                                       # [B, T, D]
pred = predictor(emb, actions)                            # causal history, [B, T, D]
L_pred = mse(pred[:, :-1], emb[:, 1:])                      # aligned next-step pairs, no detach
L_reg  = mean([SIGReg(emb[:, t]) for t in range(T)])      # across the batch, each time slice
loss = L_pred + 0.1 * L_reg
optimizer.zero_grad(); loss.backward(); optimizer.step()
A LeWM run reports a prediction loss of 0.000 early in training, yet planning fails. What should you check first?

Chapter 4

Rollouts: Training with Help, Planning Alone

See why a model that scores well one step at a time can drift when it must feed its own guesses back in

You follow a GPS through a strange city. At every turn the screen shows your true position. Even if you misjudge one turn, the next turn starts from the truth, so the mistakes never pile up. Now the GPS fails. You must drive five turns by dead reckoning: you only count how far you went and which way you turned. A small error at turn one is still there at turn two. By turn five you are on the wrong street.

Slide 33 draws exactly this split for a world model. The two situations have names. Teacher forcing means every training step feeds the model the real context, taken from the recording, however wrong its last guess was. Open-loop prediction means the model feeds its own predicted latents back in as the context for the next prediction. The slide’s figure shows a PushT clip twice: the real frames on one row, and the imagined frames on a second row that slowly drift away from them.

Why training uses help

The loss of Chapter 3 pairs pred[:, :-1] with emb[:, 1:]. Every prediction in that line starts from encoded real frames. That is teacher forcing, and it is the only thing a log can offer: the log holds no frames that the model imagined. It also makes training fast, since all positions run at once in one forward pass.

Planning cannot use it. At decision time there is one real observation, the current frame. Every later frame is a prediction. To score a plan of five actions, the model must predict step one, add that prediction to its context, predict step two from the longer context, and so on. The training loss never asked whether those later contexts, made of its own output, stay close to reality.

The error adds up

A toy shows the effect with numbers. A point goes round a circle of radius 1. The true motion turns it by a fixed angle each step. The model knows that angle almost right, but it is off by a small amount δ each step. After one step from a true position, the model’s point is off by the chord of an angle δ. After h open-loop steps the angles have added, so the point is off by the chord of an angle hδ.

error after h open-loop steps = 2 sin(hδ / 2)  ⇒  δ = 5°: h = 1 gives 0.087, h = 5 gives 0.433

That is five times larger after five steps, for the same model. Teacher forcing would report 0.087 at every step. Play with it below.

Dead reckoning

Blue dots are the real path. Warm dots are the model rolled out on its own from the first frame. Teal ticks are one-step guesses made from each real frame. Drag the model’s error and the horizon.

5.0°
5

Illustrative toy, not LeWM: a point turning 30° per step on a circle of radius 1, with a model whose turn is wrong by the angle you set. Errors are exact chord lengths, 2 sin(hδ/2), capped by the circle’s diameter, 2.

Set the error to 5° and look at the bars. The teal bars, the one-step errors, stay flat. The warm bars climb. Raise the steps ahead to 12 and the open-loop point is off by 1.0, half the way across the circle. Real dynamics can be worse: where small differences grow, as in a balancing task, errors multiply rather than add.

Read the failure in three parts. The symptom: a plan that looks perfect in imagination misses in the real world. The measurement: the open-loop error on held-out clips at horizon H, compared with the one-step error. The fix: keep the horizon short, and replan often, as Chapter 0 showed. LeWM plans only five model steps ahead (slide 64). Training with multi-step rollouts is another general remedy, but this lecture does not claim LeWM uses it.

Every candidate keeps its own history

Slide 34 spells out a point that is easy to get wrong in code. CEM, from Chapter 6, scores hundreds of candidate plans at once. Each candidate is a separate story about the future. Candidate A starts from the same observed latent history as candidate B. Then they split. A’s first predicted latent depends on A’s first action block, and is appended to A’s history only. B does the same with B’s own. From then on, A’s predictions use A’s history and A’s actions, and nothing of B’s.

StageCandidate ACandidate B
StartThe same observed latent historyThe same observed latent history
First transitionUse action block uA,0Use action block uB,0
Next inputAppend the predicted zA,1Append the predicted zB,1
Later transitionUse A’s history and actionsUse B’s history and actions

In shapes: the observed context is 1 × T_hist × D. Copy it N times to get N × T_hist × D, one private copy per candidate. At each model step the predictor reads N × (T_hist + h) × D and the matching actions, and returns one new latent per candidate, N × D. Each branch appends its own. With N = 300 and H = 5 (slide 64), the model runs five times, each time on 300 branches at once. If you shared one history across candidates, every plan would be scored on a blend of all the others.

A model is off by 5° per step on a turning point. Why is its error after 5 open-loop steps about five times its one-step error?

Chapter 5

RSSM and PlaNet: Memory, Reward and Scored Plans

Add a notebook and a reward predictor to the world model, so it can score imagined futures that no real environment ever produced

You are playing a video game, and the screen only shows what is in front of you. A door closed behind you, but you remember it is there. You keep a small notebook in your head: where you have been, what you heard. You also keep a guess about things you cannot see. Both help you decide where to go, and both are partly wrong.

PlaNet (Hafner et al., 2019) builds this into a world model. It is the second route in Chapter 1: reconstruct the image, like the VAE of Chapter 2, and add a reward. The model is the RSSM, the recurrent state-space model.

Where reward labels come from

A reward is a number the task hands out after each step, such as +1 for reaching the goal. Slide 35 shows the data-collection loop from the PlaNet paper. The agent calls env.step(a) inside an action-repeat loop. Each call returns a reward and the next observation. With action repeat R, the same action runs R times, and the training label sums the R small rewards. The result goes into the replay set as a triple (o, a, r).

That is a real reward, from the task’s rules. But an imagined state never passed through env.step. Nobody can ask the task what an imagined state was worth. So PlaNet learns a reward predictor: a small network that guesses the reward from the model’s own state.

Rewards can be tricky to learn from. Slide 36 shows the Reacher task, where a two-joint arm must touch a red ball. The reward is 1 if the fingertip is within a threshold distance of the goal, and 0 otherwise. Take a threshold of 0.06. At distance 0.04 the reward is 1. At 0.08 it is 0, and at 0.20 it is also 0. A near miss looks exactly like a far miss.

A sparse reward

The red disk is the goal. The hand is the teal dot. Drag the distance, or press a preset from the slide.

0.080

The rule is the lecture’s r = 1[d ≤ R_goal + R_tip], with R_goal + R_tip = 0.06 as in the slide’s example. The drawing is schematic and not to scale.

The reward is almost always 0, and it tells the learner nothing about how close it was. This is what sparse means. A reward predictor trained on such labels has little to learn from until the arm gets lucky.

The RSSM: a notebook plus a guess

The RSSM splits its state in two (slide 37). The deterministic state ht is the notebook: a vector that a recurrent network updates from the last state, the last guess and the last action. It carries memory and has no randomness. The stochastic state st is the guess about what is happening now. It is sampled, so it can stand for several possibilities that fit the same memory. Slide 38 gives four rules:

There are two ways to get st. With a real image in hand, sample from the posterior q(st | ht, ot), the VAE encoder of Chapter 2. With no future image, as in imagination, sample from the prior p(st | ht). Training makes the prior match what the posterior found, so that imagination, which must use the prior, stays on track.

The reward predictor, by hand

The reward head looks at the state and writes a number: r̂t = ρ(ht, st). Slide 39 treats it as a Gaussian with unit spread. Then −log N(r; r̂, 1) equals half the squared error, plus a constant. With the observed r = 1 and a prediction of 0.4, the squared-error term is ½ × 0.6² = 0.18. The slide stresses a point: this loss fits recorded rewards. It does not push the predicted reward up.

Slide 40 takes a one-weight head to show who gets the gradient. Let r̂ = w · s, with w = 2 and s = 0.2, so r̂ = 0.4. The label is 1.

∂L/∂w = (r̂ − r) · s = (−0.6)(0.2) = −0.12    ∂L/∂s = (r̂ − r) · w = (−0.6)(2) = −1.2

The state s gets a gradient ten times larger than the weight. That is the point of the slide: the reward error trains the head and the shared inference and recurrent path that produced s. The state is shaped to keep what predicts reward. The observed reward is a label, not an extra input to q(st | ht, ot). Try it.

Who learns from a reward error

The head predicts r̂ = w × s. The label is 1. Press Train and watch the prediction climb. The bars show how large each gradient is.

Slide 40’s numbers: w = 2, s = 0.2, label r = 1, loss ½(r̂ − r)². Each step moves both w and s by 0.1 times their gradient. A real state s is a vector made by the encoder and the recurrent network, so its gradient flows on into their weights; here it is one number.

At the start the loss is 0.18 and the gradient on s is −1.2. After one step s has moved from 0.20 to 0.32, while w moved by only 0.012. After 20 steps the prediction is 1.00, almost exactly the label. Most of the change came from s, which rose from 0.20 to 0.49, while w moved only from 2.00 to 2.04. That is the whole idea: a state that is useless for the reward gets reshaped.

The whole RSSM loss

Slide 41 puts it together. Summed over time, three things are charged:

Observation loss
Keeps the scene information, as in the VAE.
Reward loss
Keeps the task-relevant information, like the device above.
KL
Pulls the future prior toward the states the posterior inferred from real data.

Scoring an imagined plan

To plan, PlaNet starts from the current state and imagines each candidate action sequence U = (at, …, at+H−1) forward using the prior only, since no images exist in the future. At each imagined step the reward head writes a predicted reward. Slide 42 adds them up into a predicted return:

Ĝ(U)
The predicted return of plan U.
∑j=1H
Add the imagined steps, from step 1 to the horizon H.
γj−1
A discount that shrinks later rewards. PlaNet uses γ = 1, so it shrinks nothing.
r̂t+j
The reward head’s guess at imagined step j.

With γ = 1 and predicted rewards (0.2, 0.8, 1.0), the return is 2.0. CEM minimises a cost, so use J(U) = −Ĝ(U). Change the numbers below to feel the discount.

Score a plan

Three imagined steps, each with a predicted reward. Drag the rewards and the discount. The bars show each step’s contribution.

0.20
0.80
1.00
1.00

The slide’s numbers: rewards (0.2, 0.8, 1.0) and γ = 1 give Ĝ = 2.0. The rewards are predictions made on states from the prior; a wrong reward head or a drifting state makes the score wrong in the same way.

Two ways to score a plan

Slide 43 sets the two systems side by side. PlaNet needs reward labels, learns a reward head, sums predicted rewards into a return, and prefers high return. LeWM needs no reward labels at all. It receives a goal image at planning time and prefers plans whose final predicted embedding is close to the goal embedding. Chapter 6 builds that cost.

QuantityHow it is computedHow the agent uses it
Observed reward rtThe task rule on a real stepA label in PlaNet’s replay set
Predicted rewardThe reward head on (ht, st)Scores one imagined step
Predicted return Ĝ(U)The sum of predicted step rewardsPlaNet prefers high return
LeWM terminal cost J(U)‖ẑt+H − zgoal‖²Prefer low cost, given a goal image
In PlaNet, the observed reward is used as…

Chapter 6

A Goal Image and Every Step of CEM

Turn a picture of the goal into a cost, then search for the actions that reach it by sampling, scoring, keeping the best and refitting

You are solving a sliding puzzle, and a photo of the finished puzzle lies on the table. You cannot touch the pieces yet. In your head you try a few sequences of moves. For each one you picture the final board, and you ask: how much does it look like the photo? You pick the sequence whose final board looks most like it. Then you make the first move, look at the real board, and think again.

That is planning with a goal image, and it is exactly what LeWM does at decision time. Everything in this chapter is the machinery behind that sentence.

The cost: distance to the goal

Slide 44 defines the cost of a predicted future. Encode the goal image once with the same encoder, to get a vector zg = enc(og). Predict where a plan ends, giving ẑH. Then the cost is the squared distance between the two.

C(ẑH)
The cost of ending in predicted state ẑH. Lower is better.
ẑH
The embedding the model predicts after H steps of the plan.
zg
The embedding of the goal image, computed once.

The distance is measured in the full embedding space, with all D coordinates. Slide 45 warns about a trap. The slides often draw a two-dimensional picture of embeddings made by PCA (a way to squash many dimensions into two for drawing). That picture is only for the eye. Two plans that look close in the picture can be far apart in full space, so the scoring must never use it.

Slide 46 then states the planning problem. The optimisation variable is the plan U = (u0, …, uH−1), a list of H action blocks. The encoder and predictor stay fixed. We are looking for the U with the smallest cost: a* = argmin C(ẑH). The search is over actions, not over weights.

CEM in four verbs

The cross-entropy method (CEM) searches without any gradients. It needs only to try plans and compare costs. Slide 47 gives the algorithm, and it has four verbs that repeat for K rounds:

  1. Sample. Draw N candidate plans from a Gaussian over whole plans.
  2. Score. Roll each plan out in the frozen model and compute its cost.
  3. Keep the elites. Pick the Ne plans with the lowest cost.
  4. Refit. Make the Gaussian match the elites: their mean and their spread.

Slide 48 names the four indices, so no one mixes them up. The real time is t, and it moves only when an action really runs. Inside a plan, h = 0, …, H−1 counts the model’s imagined steps. The search round k = 0, …, K−1 changes the Gaussian’s mean and variance. The candidate i = 1, …, N is one plan in one round.

A distribution over whole plans

A plan has H action blocks, each with A_block numbers, so U lives in a space of H × A_block numbers. Slide 49 puts a Gaussian on that whole space. Each coordinate has its own mean and its own variance: qk(U) = N(vec(U); μk, diag(vk)). The word diag means the coordinates vary independently. Each round draws N plans, a tensor of shape N × H × A_block.

Slide 50 starts it plainly: μ = 0, v = 1. To sample, draw standard noise ε and compute U = μ + √v ⊙ ε, exactly the trick of Chapter 2. Then clip the result to the allowed action range. Slide 51 rolls every plan out: copy the same observed context into N separate branches (Chapter 4), apply each branch’s action block, and append each predicted embedding. Slide 52 scores each: Ji is the sum over the D coordinates of the squared gap between its final embedding and the goal. That gives one number per candidate. All plans share the same goal and the same start.

Watch all four verbs in a toy world where each action block is a push of two numbers, the model adds the pushes up, and the goal sits at (1.4, 0.9). The numbers are the lecture’s PushT budget: 300 plans, 30 elites, 5 steps.

One CEM round, phase by phase

Each dot is where one of 300 plans ends. The blue diamond is the goal. Press Next phase to sample, score, keep the 30 elites and refit. The dashed ring is the Gaussian the search now holds.

Illustrative toy, not LeWM: the latent state is 2D, the model adds each of the 5 action blocks to it (so the model is exact), the start is (0, 0), each action number is clipped to −0.5 to 0.5, and the search starts at μ = 0, v = 1. N = 300 and Ne = 30 are the budget slide 64 reports; the random draws are seeded.

Press the phase button through one round. The first sample is a wide cloud around the start, because the first Gaussian is centred on “do nothing”. Scoring shades each dot by its cost. The elites are the 30 dots nearest the goal. The refit shrinks the ring and slides it toward them. After five rounds the cloud is a much tighter knot around the goal: in our seeded run the 30th-best plan of the fifth round ends about 0.13 away, and the spread per action number falls from 1.00 to about 0.14.

Why the refit is a mean and a variance

Why exactly these two formulas? Slide 55 shows that they are the answer to a question: “which Gaussian makes the elite plans most probable?” Pick the Gaussian parameters η that maximise the average log-probability of the elites. For a Gaussian with independent coordinates, calculus gives a clean result:

μnew = mean(Uelite)     vnew = mean( (Uelite − μnew)² )

The new mean is the average elite plan, coordinate by coordinate. The new variance is how far the elites spread around it. Slide 54 adds the next sampling step: draw a small perturbation around the fitted mean, with spread √v. The fitted spread sets the size of the next round’s steps. Wide elites mean wide exploring. Tight elites mean fine tuning.

The same thing as code

def cem(model, z_ctx, z_goal, lo, hi, H=5, N=300, Ne=30, K=30):
    mu  = zeros(H, A)                                       # [H, A_block]
    var = ones(H, A)
    for k in range(K):
        U = (mu + var.sqrt() * randn(N, H, A)).clamp(lo, hi)   # sample   [N, H, A]
        z = rollout(model, z_ctx, U)                         # frozen model, N branches -> [N, D]
        J = ((z - z_goal) ** 2).sum(-1)                   # score    [N]
        elite = U[J.topk(Ne, largest=False).indices]       # keep     [Ne, H, A]
        mu, var = elite.mean(0), elite.var(0, unbiased=False)   # refit
    return mu                                            # or the best plan seen

Read the failure in three parts. The symptom: the planner returns plans that look good in the model but miss the goal. The measurement: compare the cost of the returned plan in the model with the real distance to the goal after executing it. The fix: the next chapter shows the usual causes, and Chapter 4 showed the usual cure.

In one CEM round, which two quantities change from round k to round k + 1?

Chapter 7

CEM by Hand, and How It Goes Wrong

Work one whole round with four plans on paper, then see three ways the search can settle on the wrong answer

You are driving a toy car along a straight line, two moves only. Each move is a push, forward or back. You want the car to stop exactly one metre from where it started. A stop matters: arriving at speed would carry it past. With so few moves you can write every number on a napkin. That is slides 56 to 61, and it is the best way to trust CEM before the big model hides the numbers.

A system small enough to calculate

Slide 56 gives the world: position p and speed v. Each step, the position moves by the speed, and the speed moves by the push a.

pnext = p + v     vnext = v + a     start: p0 = 0, v0 = 0     goal: p2 = 1, v2 = 0     a0, a1 in [−1, 1]

A plan is two numbers, (a0, a1). Slide 57 rolls four plans through the world. Take (1, 1). Step one: p1 = 0 + 0 = 0, and v1 = 0 + 1 = 1. Step two: p2 = 0 + 1 = 1, and v2 = 1 + 1 = 2. The car is at the right place, doing speed 2. The other three work the same way:

Plan (a0, a1)After a0: (p1, v1)End: (p2, v2)
(0, 0)(0, 0)(0, 0)
(1, 1)(0, 1)(1, 2)
(1, −1)(0, 1)(1, 0)
(0.5, −0.5)(0, 0.5)(0.5, 0)

The cost, with a little effort

Slide 58 turns the goal into one number. Three terms add up. The first, (a0 − 1)², is the squared error of the final position, since p2 equals a0. The second, (a0 + a1)², is the squared final speed, since v2 = a0 + a1. The third, 0.1(a0² + a1²), charges for effort. The slide says plainly that this third term is “an added classroom choice”, not part of the goal. Slide 59 fills the table. Check (1, −1) yourself: 0 + 0 + 0.1 × 2 = 0.20.

Two plans have the lowest costs, 0.20 and 0.30. Take them as the elites. Slide 60 refits the mean as their average, coordinate by coordinate: ½[(1, −1) + (0.5, −0.5)] = (0.75, −0.75). Plug it into the cost: (0.75 − 1)² = 0.0625, then (0.75 − 0.75)² = 0, then 0.1 × (0.5625 + 0.5625) = 0.1125, for a total of 0.175. The slide’s remark is worth a pause: the mean differs from every sampled candidate, and here it is better than both elites.

Slide 61 does the same for the spread. The elites lie +0.25 and −0.25 from the mean in the first coordinate, and the reverse in the second. The variance is the average squared deviation, vnew = (0.0625, 0.0625), and the standard deviation is its square root, (0.25, 0.25). The next Gaussian is centred on (0.75, −0.75) with spread 0.25. Change the effort weight below and see which plans become elites.

Slides 56 to 61, live

Four plans in the plan square. Green rings mark the elites. The teal cross is the new mean and the dashed box is one standard deviation. Drag the effort weight, or keep three elites instead of two.

0.10

The four plans, the three-term cost and the refit formulas are slides 57 to 61. At the default weight 0.1 you get the slide’s numbers: costs 1.00, 4.20, 0.20, 0.30, mean (0.75, −0.75) with cost 0.175, and spread (0.25, 0.25). Other weights and a third elite are our additions.

At weight 0.1 the table matches the slide to the digit. Raise the weight to 1 and effort dominates: the quiet plan (0, 0) becomes an elite, because doing nothing is cheap. At weight 0 only the goal matters. With three elites, the refit mean drifts toward the worse plan, and the spread grows. That is the knob Ne: fewer elites make the search sharper and riskier.

Three ways CEM goes wrong

Slide 62 lists them in one line each.

The next device shows the last two at once. The cost has two valleys, at −1 and +1, with a hill between them. Run CEM from the middle and read the cost at the refit mean. Then switch to a model that is wrong about which valley is deeper.

Between two good answers

Blue line: the real cost of a one-number plan. Warm dashed line: the cost the model believes. Dots are sampled plans, green when elite. The teal bell is the Gaussian CEM holds.

40

Illustrative toy. Equal valleys: the cost is min((a − 1)², (a + 1)²) and the model is right. Wrong model: the real cost is min((a − 1)² + 0.6, (a + 1)²), so the left valley is better by 0.6, but the model believes min((a − 1)², (a + 1)² + 0.6). Each round keeps the best 25% of the plans (at least 2). The random draws are seeded; press Reset for a new run from the same start.

With 40 plans and two equal valleys, the first refit puts the mean at −0.03, on the hill, where the real cost is 0.94. Six rounds later the mean is at 0.56, still short of a valley, with real cost 0.19. Switch to the wrong model, and the search is happy: by round 3 it sits at +1 with a spread of 0.02, sure of itself, and the real cost is 0.60 instead of 0.00. Drop the plans to 12 and the equal-valley run happens to fall into one valley; luck, not skill.

Read the failure in three parts. The symptom: the returned plan costs more than its elites did, or the agent misses in the real world though the model was happy. The measurement: the real cost at the refit mean, next to the real cost of the best elite, and the real cost after execution against the model’s. The fix: more samples, a better model, or replanning to rescue a bad first guess.

LeWM’s planning budget

Slide 64 reports what LeWM actually uses, from the paper’s appendix. Each planning step draws N = 300 plans and keeps Ne = 30 elites. The horizon is H = 5 model steps. On PushT the search runs up to 30 CEM rounds. Then the model executes all five planned steps before replanning, which is why the PushT clip on slide 65 shows no feedback replanning. Compare this with Flappy Bird in Chapter 0, which executes one action and replans. Both are legal uses of the same planner; the choice depends on how good the model is and how fast the world surprises you.

The elites of one CEM round sit in two separate valleys of the cost, one on each side of a hill. What can go wrong at the refit?

Chapter 8

Value, Policy and Dreamer

Stop searching from scratch at every decision: learn a habit that picks actions and a gut feeling that scores states, both inside the model

A taxi driver does not run a search before every turn. After years of practice she has a habit: at this kind of junction, with this kind of traffic, she takes this lane. She also has a gut feeling about positions: “this side street will be slow, even though it looks short”. A chess beginner searches every move. A master looks two moves ahead and trusts a feeling about the position they reach.

Those are the two things that Chapters 5 to 7 lack, and the last seven slides of the lecture add them. CEM is a planner: it searches again at every decision, and it forgets everything after. Slide 66 introduces the other route: model-based reinforcement learning uses predicted consequences to improve the expected return, the total reward an agent can expect over time. The lecture’s example is Dreamer (Hafner et al., ICLR 2020), and its three-part figure tells the story: (a) learn the dynamics from experience, (b) learn behaviour in imagination, (c) act in the environment.

In everyday words: Dreamer plays with its model, not with the world. It lives through many imagined futures, learns what works, and then walks into the real world with a habit. (The model-based RL lesson, the world models lesson and the actor-critic lesson build the parts from zero.)

Search against habit

Slide 67 puts the two routes in one table, with the caution that they are not rivals. CEM names a search method. Model-based RL names how prediction improves behaviour. A single RSSM can serve both.

QuestionPlanning with CEM (PlaNet)Learning a policy (Dreamer)
How is the model used?To score candidate action sequencesTo generate imagined trajectories for actor-critic updates
What improves?A proposal over plans, at this one decisionShared policy and value weights, kept across decisions
How is an action chosen?Run a search on predicted returnsEvaluate the learned actor, one forward pass
What do they share?An RSSM predicts states and rewardsThe same RSSM predicts states and rewards

Value: the part beyond the horizon

A plan looks only H steps ahead. What if the best thing happens at step H + 1? A planner that stops at H cannot see it. Slide 68 fixes this with a value function. First, the state: xt = (ht, st), the RSSM’s notebook and guess together. Then:

Vπ(xt)
The value of state xt: the total reward to expect from here on, if the policy π keeps acting.
∑k=0∞
Add up every future step, with no end.
γk
A discount below 1. A reward k steps away counts γk times as much, so far rewards matter less.
rt+k+1
The reward received k + 1 steps from now.

Reward predicts one step. Value estimates the rest. Now a plan’s score gets a tail: add up the H imagined rewards, then add the value of the state where the plan stops, discounted by γH. The slide works one case with H = 2 and γ = 0.9. Plan A earns rewards (5, 0) and ends in a state worth 0. Plan B earns (0, 0) and ends in a state worth 10.

A: 5 + 0.9 × 0 + 0.9² × 0 = 5.0     B: 0 + 0.9 × 0 + 0.9² × 10 = 8.1

Without the tail, a planner would pick A. With it, B wins, because B ends where the good things are. Move the sliders below.

Count what lies beyond the horizon

Plan A earns (5, 0) then stops in a state worth the A slider. Plan B earns (0, 0) then stops in a state worth the B slider. Turn the value tail off to see what a plain H-step planner sees.

0.0
10.0
0.90

Slide 68’s example: H = 2, γ = 0.9, rewards (5, 0) and (0, 0), end-state values 0 and 10, giving 5 and 8.1. The score is ĜH = ∑k=0H−1 γk r̂t+k+1 + γH V(x̂t+H). All numbers are the slide’s or your own sliders.

With the default sliders B scores 8.1 and A scores 5.0. Switch the tail off and B falls to 0.0: a planner with no value is blind to everything past step 2. Now push γ down to 0.5. B’s tail is worth 0.25 × 10 = 2.5, and A wins again. A heavy discount makes the agent short-sighted, which is sometimes the right call.

Learning the value from imagined returns

Where does the value network Vψ come from? It is trained inside the model (slide 69). From a state xt, let the current policy pick an action, at ~ πφ(· | xt), and let the model imagine the next state, x̂t+1 ~ pθ(· | xt, at). Build a target from the predicted reward plus the value of the next state:

yt = r̂t+1 + γ (1 − dt+1) · sg[ Vψ(x̂t+1) ]     LV(ψ) = ½ E ( Vψ(xt) − sg[yt] )²

Two pieces need words. The flag d is 1 if the episode ended, which cuts the future off. The sg is the stop-gradient of Chapter 1: the target is treated as a fixed number, so the critic is pulled toward it and the target is not pulled back. Using your own estimate of the next state inside your target is called bootstrapping. A worked case: r̂ = 1, γ = 0.9, d = 0 and V(next) = 10 give y = 1 + 0.9 × 10 = 10. If V(xt) is 8, the loss is ½ (8 − 10)² = 2, and the critic is pulled up toward 10. The slide adds that Dreamer uses multi-step λ-returns, a blend of returns of different lengths, and that the full derivation comes after fall break.

Policy gradient: a reusable habit

Now the habit. Slide 70 recalls the policy gradient: nudge the policy’s weights so that actions that did better than expected become more likely.

∇φJ
Which way to change the policy weights φ to raise the expected return J.
∇φ log πφ(a | x)
The direction that makes this action more likely in this state.
A(x, a)
The advantage: how much better this action did than the value predicted, A = Ĝ − Vψ(x). Positive pushes the action up; negative pushes it down.

The slide makes the contrast sharp. CEM updates a plan proposal, and throws it away after the decision. The policy gradient updates a policy that serves every future decision. Try the update on a single imagined state with three actions whose true returns are 2, 5 and 3.

Better than expected goes up

Warm bars: the policy’s probability of each action. The blue marker is the value V, the policy’s average return. Below, each action’s advantage. Press Update the policy.

Illustrative toy: one imagined state, three actions with returns 2, 5 and 3, and a softmax policy. The update is the exact policy gradient for a softmax, which is πa (returna − V) for each logit, with step size 0.3. A real actor is a network over many states, and the returns come from imagined rollouts.

At the start the policy is uniform. V = (2 + 5 + 3) / 3 = 3.33. The advantages are −1.33, +1.67 and −0.33. After one update, the middle action rises from 33.3% to 39.1%. After 20 updates it holds 93.4%. Notice that the best action is not chosen by search. It simply gets pushed up each time it beats the average.

Dreamer: differentiate through the imagination

Slide 71 shows the last trick. In Dreamer the imagined trajectory is built from differentiable pieces: at = gφ(xt, εt) and xt+1 = fθ(xt, at, ξt), where ε and ξ are the noise draws of the reparameterization trick from Chapter 2. So the score ĜH is a smooth function of φ, and you can backpropagate straight through actions, states, rewards and values to the actor’s weights: φ ← φ + α ∇φ E[ĜH]. During the actor update the world model (θ) and the critic (ψ) are held fixed, but derivatives still flow through them. The RSSM supplies the dynamics. The actor and the critic learn how to use them.

  1. Learn the dynamics from real experience, with the RSSM loss of Chapter 5.
  2. Imagine short trajectories from real starting states, using the actor for actions.
  3. Update the critic toward imagined returns, and the actor by backpropagating through the imagination.
  4. Act in the real world with the actor, one cheap forward pass, and store the new experience.

Slide 72 closes the lecture in three lines. Representation and dynamics losses make future states predictable. CEM searches actions at a decision, while a learned policy reuses behaviour across decisions. Value connects finite imagination to long-term return.

Read the failure in three parts. The symptom: the agent scores high in imagination and low in the real world. The measurement: imagined return against real return for the same actor. The fix: keep imagination short, retrain the model on the states the actor actually visits, and treat a big gap as a model fault, never as an actor success.

A planner has horizon H = 2. Plan B earns nothing in two steps but ends in a state with value 10. Why add γH V to its score?

Chapter 9

The Field Guide

Review the lecture on one page: the failures, the numbers, a checklist, the core code, and where it leads next

You can now say, from memory, what Lecture 10 argued. Representation and dynamics losses make futures predictable. A cost or a reward makes futures comparable. Then there are two ways to turn that into action. CEM searches a Gaussian over whole plans at each decision. A learned policy and value keep the lesson of many imagined lives and reuse it at every decision. Each part has its own way to fail, and a good engineer names the failure before touching the code.

Six failures, six tests

FailureSymptomWhat to measure
Embedding collapse (Ch 1, 3)Prediction loss near zero, planning uselessSpread of embeddings across a batch
Open-loop drift (Ch 4)Plans look perfect in imagination, miss in realityH-step error against one-step error
Sparse reward (Ch 5)The reward head learns almost nothingShare of steps with a nonzero reward
Split valleys (Ch 7)The refit mean is worse than its elitesCost at the mean against the best elite
Model exploited (Ch 7)A confident plan that fails when executedReal cost after executing, against the model’s
Blind beyond the horizon (Ch 8)The planner ignores a distant payoffPlan score with and without a value tail

Try the habit on six made-up cases. Each lists what a run’s diagnostics show. Name the failure. One case may look alarming and be fine.

Name the failure

Read the run. Press the failure that explains it.

All six cases are made up to practise the habit; every number in them is illustrative, not a measurement of a real system.

The numbers that matter

WhatNumber or formWhy it matters
EMA teacher (slide 9)0.9 × 2 + 0.1 × 4 = 2.2The teacher is a slow shadow of the student.
Collapse (slide 10)E(x) = c gives prediction loss 0Agreement alone is not learning.
LeJEPA loss (slide 11)(1 − λ) agreement + λ SIGRegA shape rule on the whole batch.
VAE loss (slides 17, 19)−log p(o | z) + KL(q ‖ N(0, I))Information has a price; zero at μ = 0, σ² = 1.
LeWM tensors (slide 23)obs B×T×3×224×224; emb B×T×D; pred, target B×(T−1)×DPairs are aligned by one step.
MSE by hand (slide 27)(0.2, −0.1) against (0.5, 0.3): 0.25 / 2 = 0.125One scalar feeds both branches.
LeWM loss (slides 30, 32)L_pred + 0.1 × L_SIGRegNo detach, the regulariser stops the collapse.
Decoder ablation (slide 31)PushT success 96.0 ± 2.83 without, 86.0 ± 7.54 with a pixel lossSpecific to that experiment.
Sparse reward (slide 36)threshold 0.06: d = 0.04 gives 1; 0.08 and 0.20 give 0A near miss looks like a far miss.
Reward head (slides 39, 40)loss 0.18; gradients −0.12 on w, −1.2 on sThe reward error also reshapes the state.
PlaNet return (slide 42)γ = 1, rewards (0.2, 0.8, 1.0) give 2.0CEM minimises −Ĝ.
CEM budget (slide 64)N = 300, Ne = 30, H = 5, up to 30 rounds on PushTThe real planner is this small.
CEM by hand (slides 59 to 61)costs 1.00, 4.20, 0.20, 0.30; mean (0.75, −0.75) costs 0.175; spread (0.25, 0.25)The mean can beat every sample.
Value tail (slide 68)H = 2, γ = 0.9: A scores 5, B scores 8.1Value sees past the horizon.

A checklist for any world-model agent

The core, as code

Three short sketches gather the lecture. encoder, predictor, SIGReg, rollout, actor, critic, reward_head and rssm.imagine stand for whatever your library provides. The CEM sketch is in Chapter 6.

# 1. LeWM training step, Ch 3: obs [B,T,3,224,224], actions [B,T,A_block]
emb  = encoder(obs)                                  # [B, T, D]
pred = predictor(emb, actions)                       # causal history
loss = mse(pred[:, :-1], emb[:, 1:]) + 0.1 * mean([SIGReg(emb[:, t]) for t in range(T)])

# 2. Reward head, Ch 5: x = (h, s) from the RSSM; r [B]
r_hat  = reward_head(x)                              # [B]
loss_r = 0.5 * ((r - r_hat) ** 2).mean()                # a label, not an input to q

# 3. Dreamer-style update inside the model, Ch 8: x0 [B, dim], horizon H
x, rs, G = x0.detach(), [], 0
for k in range(H):
    a = actor(x).rsample()                           # a = g(x, eps), differentiable
    x = rssm.imagine(x, a)                          # prior only, no image
    G = G + gamma**k * reward_head(x)                # predicted r_{t+k+1}
G = G + gamma**H * critic(x)                       # the value tail
actor_loss  = -G.mean()                              # model and critic fixed, gradients flow through
critic_loss = 0.5 * ((critic(x0) - G.detach()) ** 2).mean()  # sg on the target

Where this sits, and what comes next

Slide 72 gives the thread: from learned predictions to better decisions. Lectures 5 to 7 asked what an encoder should keep (Lecture 5, Lecture 6) and how to keep many futures alive (Lecture 7). This lecture fills in the action-selection row of the four interfaces. The value derivation with λ-returns, which slide 69 defers, comes after fall break. The course site’s file for Lecture 11 is named for Dreamer, diffusion and flow matching.

Keep going

Course credit. This lesson re-teaches Lecture 10 of Penn CIS 6280 World Models (Fall 2026), taught by Jiatao Gu. The slides are linked at the top of this page and in the footer: Lecture 10 slides. The course site is jiataogu.me/cis6280-world-models. The explanations, devices and toy numbers on this page are our own; the numbers labelled from slides come from the slides.

The takeaway. A world model gives you futures. A cost or a reward ranks them. Search spends compute at every decision. A policy and a value spend it once, then reuse it. Every link in the chain can fail quietly, so name what you measure at each link.
A run reports a prediction loss near zero, and the planner returns confident plans that never reach the goal. The embeddings of different frames are almost identical. Which fix does the lecture’s recipe suggest?

Now press Teach and explain, from memory, how one LeWM step is built, how one CEM round refits, and how a value and a policy are learned inside a model. Then go back to the hero and run the search again.

CIS 6280 Lecture 10: Policy and Value Learning with World Models
The course site · Lecture slides · PlaNet · Dreamer · Back to Gleams