A world model only predicts. To act, an agent can search plans with the cross-entropy method, or learn a policy and a value in imagination. Start with the search: refit a Gaussian and watch the cost fall.
Revisit representations, latent prediction, LeWM, RSSM and CEM, then see how Dreamer turns imagined futures into a reusable policy and a value.
Press Run one round a few times with CEM, then switch to random guessing and run the same rounds. Watch the green plans crowd toward the blue diamond, the best answer. Then we build, piece by piece, everything underneath.
You need a feel for averages (add up, divide by how many) and a little Python for the code. We build the rest from zero.
Based on Lecture 10 of Penn CIS 6280: World Models (Fall 2026), taught by Jiatao Gu. Lecture slides. Every idea is re-taught here in our own words and devices; it follows Lecture 9, Planning and Control, and revisits Lectures 6 to 9.
The toy system and its cost come from the lecture’s classroom example (slides 56 to 60): a point at position 0 and speed 0 must reach position 1 at speed 0 with two actions a0 and a1, each between −1 and 1, and the cost is (a0 − 1)² + (a0 + a1)² + 0.1(a0² + a1²). The lecture works four plans by hand; here 24 plans are drawn per round and the best 6 are kept, with seeded random numbers. The best cost is found on a fine grid.
Chapter 0
See why a model that predicts the future still needs a second tool to pick an action, and meet the four parts of every agent
Picture a game of Flappy Bird. A bird falls a little every frame. Tap, and it hops up. Green pipes slide in from the right, each with a gap, and the bird must pass through every gap. You have a fraction of a second to decide: tap, or wait?
Now suppose you have a very good world model: a learned program that answers one question. “If the world looks like this and I do that, what will it look like next?” Hand it a picture of the bird and the word “tap”, and it draws the next frame. That is a real gift. It is also not enough.
The model tells you what each choice would do. It never tells you which choice to make. A map shows every road, but you still have to pick a route. This lecture is about picking the route.
The lecture opens with the Flappy Bird demo, and it shows three steps that repeat. First, the learned model predicts several possible futures, one for each sequence of tap or no tap. Second, a search called CEM (the cross-entropy method, Chapter 6) picks the sequence whose predicted future looks best. Third, the bird executes only the first action of that sequence, looks at the new frame, and starts again.
Why throw away the rest of the plan? Because new pipes keep arriving, and because the model is never perfect. A plan made now is a guess about the next few frames. After one real step you know more, so you can guess better. This habit has a name: replanning, or model predictive control (MPC). The plan looks a fixed number of steps ahead, called the horizon, and the horizon slides forward as the bird moves. The slide sums it up in four words: finite planning horizon, repeated control.
Try it. The device below is a toy of the same loop. A point must follow a wandering target. The planner picks five actions ahead, but the real world adds a small push the model did not predict. You choose how many of the five actions to execute before you plan again.
The blue line is the target gap. The teal line is the bird. Short marks on the floor show where it replanned. The bars below show the average miss for 1 to 5 executed actions. Drag both sliders.
Illustrative toy, not the lecture’s Flappy Bird model. The point moves by its action plus a random push of the size you choose; the planner assumes no push and moves toward the next target value, at most 0.4 per step. The same 80 random pushes are used for every bar, so the bars differ only by how often the plan is redone.
With no surprises, every bar is the same: a perfect model needs no replanning. Add surprises and the bars climb from left to right. Each action you execute without looking carries any unseen push forward, and the pushes pile up. Replanning after every action resets the pile to zero each time.
Read the failure in three parts. The symptom: the bird drifts off the gap centre and clips a pipe. The measurement: the average miss grows with the number of unchecked actions. The fix: observe more often, even if each plan is thrown away. Planning is cheap compared with a crash.
Flappy Bird replans after one action. The lecture’s second demo, PushT, executes the whole plan. In PushT a flat robot pusher must slide a T-shaped block onto a goal outline. The clip on slide 4 shows the full plan run without any feedback, which is fine because the model is good enough over those few steps.
Both demos plan with CEM. The slide then names a second way to act. Instead of searching for a plan at every decision, learn two things once and reuse them. A policy is a rule that looks at the current state and says which action to take, like a habit. A value is a number that says how much total reward to expect from a state if you follow the policy, like a gut feeling that a position is good. Dreamer, the paper from 2020 that ends this lecture, learns both inside its world model. Model-based reinforcement learning (the lesson on model-based RL builds it from zero) can use either way, or mix them.
Slide 5 breaks every agent in this course into four parts, each defined by what goes in and what comes out. It is worth memorising, because the rest of the lecture fills in one row at a time.
| Part | Goes in | Comes out | Chapters |
|---|---|---|---|
| Representation | An image or a history of observations | JEPA features, or a VAE posterior | 1, 2 |
| Dynamics | Latent context and actions | Predicted future states | 3, 4, 5 |
| Objective | A predicted outcome and the task | A cost or a return | 5, 6, 8 |
| Action selection | The model and the objective | An action plan, or a policy action | 6, 7, 8 |
In shapes: a camera frame of 3 × 224 × 224 numbers goes into the representation and comes out as a short vector D, say 192 numbers. The dynamics part takes that vector and an action block and returns another vector of size D. The objective turns the final vector into one number. Action selection tries many actions and keeps the ones with the best number.
Slide 2 lists four notices. Assignment 2 releases on September 30. Final project details come after fall break. Project registration is due before October 14, and groups of two to four are encouraged. Part of the material first planned for this lecture moves to after fall break, which is why the value derivation on slide 69 says “the full derivation follows after fall break”. Today’s revisit of Lectures 6 to 9 supports Assignment 2.
Chapter 1
Train an encoder to predict features instead of pixels, and see why it needs a rule that keeps every image from getting the same code
You want to teach a student to sort photos, but you have no labels. So you set a game: look at part of a photo, then guess what the rest looks like. The student scores a point when the guess matches. Soon the student finds a cheat. Describe every photo as “grey blob”, and every guess matches perfectly. The score is perfect and the student has learned nothing.
This cheat has a name, collapse, and it is the main danger in this chapter. The lecture’s recap of Lectures 5 and 6 asks one question of every method: what supplies the target? Slide 6 sorts the methods into three families by the answer.
| Family | What supplies the target? | Examples |
|---|---|---|
| Reconstruction | The observed data, or the missing pixels | Autoencoders, MAE; the VAE adds a probabilistic latent |
| Contrastive | The matching view among many candidates | SimCLR, CLIP |
| Learned targets | Another network’s encoding of a view or region | DINO, I-JEPA, V-JEPA, LeJEPA |
Today uses the third family for LeWM, the world model of Chapters 3 and 4, and the first family for the RSSM route of Chapter 5. This chapter handles the third. (The Lecture 6 lesson and the JEPA lesson build these from scratch.)
A JEPA (joint embedding predictive architecture) hides part of an image. A context encoder reads the visible patches and turns them into feature vectors. A predictor then guesses the feature vector of each hidden block, given where that block sits. A second encoder, the teacher, reads the full image and produces the true feature vectors. Slide 8 gives the I-JEPA loss: the average squared distance between the predicted and the teacher features at the masked locations. The teacher features are normalised, and no gradient flows into them.
Two terms need care. A gradient says which way to nudge each weight to make a loss smaller. A stop-gradient treats the teacher’s output as a fixed number: the loss may compare to it, but never changes it. Without it, the easiest way to cut the loss is to change the answer key.
The teacher still has to improve, or the targets would stay random. Slide 9 shows how. After each training step, the teacher copies a little of the student:
0.9 × 2 + 0.1 × 4 = 2.2 (old teacher weight 2, new student weight 4)
This is an exponential moving average (EMA): the teacher is a slow-moving shadow of the student. The two tools have different jobs. Backpropagation updates the student and the predictor. The stop-gradient keeps the loss from touching the teacher. The EMA is the only way the teacher changes.
Slide 10 states the trap in two lines. Suppose the encoder maps every input x to the same constant c, so E(x) = c. Suppose the predictor just returns its input, so F(c) = c. Then the predicted feature equals the target feature for every image, and the prediction loss is exactly zero. The features match perfectly and carry no information about the input.
So a useful recipe must also keep the codes of different images different. Contrastive methods do this by pushing wrong pairs apart. I-JEPA leans on the teacher, the EMA and the architecture. LeJEPA (slide 11) adds an explicit rule, and the lecture writes it as a mix of two losses (its slide calls the weight ε; we write λ):
Agreement looks at one example at a time. SIGReg looks at the batch. Collect the codes of B encoded views into a matrix Z. Then push the cloud of B points toward N(0, I), a Gaussian (a bell-shaped spread) centred at zero with unit spread in every direction. A collapsed batch is a single point, so it is very far from that shape.
Comparing a cloud in D dimensions with a Gaussian is hard. Slide 13 makes it easy. Pick a random unit direction u, and measure each point’s position along it: h = Zu. That gives B plain numbers. Compare those B numbers with a one-dimensional N(0, 1). Repeat for many random directions. If every projection looks Gaussian, the whole cloud does. Try it on a batch of 48 two-dimensional codes.
Left: 48 codes in two dimensions. Right: their positions along one random direction, against the bell curve of N(0, 1). Drag the spread to zero, or squash the cloud into a line, and compare the two losses.
Illustrative toy. The second view of each code is the first plus a small random change that scales with the spread, so shrinking the cloud also shrinks the agreement loss. The shape score is our stand-in for SIGReg: for 16 fixed random directions, the average squared gap between the sorted projections and the matching quantiles of N(0, 1). The lecture only says SIGReg compares projections to N(0, 1); the LeJEPA paper’s own statistic is different.
At spread 0, the agreement loss is 0.00, a perfect score, and the shape score is at its worst. At spread 1 and roundness 1 the shape score is near 0.1 and the agreement loss is small but not zero. A line is cleverer: it has plenty of spread in one direction, so projections on some directions look fine. Other directions expose it. That is why SIGReg averages over many directions.
Slide 14 closes the first half with a comparison. A JEPA code must keep what is needed to predict target features. A VAE code must keep what is needed to reconstruct the image. A JEPA gives one feature vector z = E(o). A VAE gives a whole distribution q(z | o). SIGReg puts its Gaussian across different examples. The VAE puts a Gaussian on each image’s posterior, compared with a shared prior. Each prepares a later part of the lecture: JEPA for LeWM, and the VAE for the RSSM.
Chapter 2
Learn a code that rebuilds the image, then see why a second term charges the code for every bit it uses
You describe a photo of a brick corridor to a friend over the phone. You may say only eight numbers. Your friend must redraw the corridor from them. If you pick the numbers well, the redraw looks close. If you may say as many numbers as you like, you could read out every pixel, and you would have learned nothing about corridors.
A variational autoencoder (VAE) plays this game. It has two networks. The encoder turns an observation o (an image) into a short code z, called the latent. The decoder turns z back into an image. Training makes the rebuilt image match the original (slide 15). The lecture revisits it now because the RSSM of Chapter 5 uses the same machinery.
A plain autoencoder gives one code per image. The VAE encoder gives a belief. For one fixed image it outputs a mean μ(o) and a spread σ(o) for every coordinate of the code. Together they define q(z | o), the posterior: “given this image, the code is probably near here, give or take this much”. To rebuild, you sample one z from q and hand it to the decoder.
Sampling raises a puzzle. Training needs gradients, and a random draw has none: you cannot differentiate “pick a random number”. Slide 16 shows the standard way out, the reparameterization trick. Draw the randomness first, from a fixed distribution that does not depend on the encoder, and build z from it.
μ = 0.5, σ = 0.2, ε = −1.5 ⇒ z = 0.5 + 0.2 × (−1.5) = 0.2
Now z is a plain formula in μ and σ. The slope of z with respect to μ is 1. The slope with respect to σ is ε, here −1.5. The decoder’s loss can send gradients back through both. The noise is an input, like the image itself.
Slide 17 writes the VAE loss as two terms. The first is the reconstruction term, the average of −log p(o | z): how surprised the decoder is by the true image, given the code. It rewards a code that carries information about the image. The second is a KL divergence, a measure of how far one distribution sits from another. It compares the image’s posterior q(z | o) with the prior p(z) = N(0, I), the round Gaussian that stands for “a code before seeing any image”.
Why reconstruct the image that went in? Slide 18 answers with a fact about the decoder. If the decoder is a Gaussian with a fixed spread, then −log p(o | z) is proportional to the squared error between the decoded and the true image. So “maximise the likelihood” and “make the squared pixel error small” are the same instruction. The target is the same image the encoder saw. Nobody supplies a label.
Why a KL term at all? Without it, the encoder can make σ tiny and spread the means of different images far apart. Each image gets its own private, exact code, and the decoder rebuilds perfectly. That is the “read out every pixel” cheat. The KL charges for every way a posterior departs from the prior. Slide 19 gives the price for a Gaussian posterior:
μ = 1, σ = 0.5: ½ [1 + 0.25 − (−1.386) − 1] = ½ × 1.636 = 0.818 nats
The price is exactly zero when μ = 0 and σ² = 1, that is, when the posterior equals the prior and the code says nothing about the image. Every bit of information in the code has to earn its price by helping the reconstruction. Slide 19 calls this “a balance between reconstruction and an image-dependent code”. Find that balance below for a one-number code.
The image is the number 1.5 (blue line). The decoder just returns z. Drag the mean and the spread of the code. Purple is your code q, pink is the prior N(0, 1). Then press Draw z to sample with reparameterization.
Illustrative toy with exact formulas. Reconstruction cost is the expected value of ½(z − 1.5)², which is ½[(μ − 1.5)² + σ²]; the KL is the lecture’s formula for one coordinate. The best code is where both slopes are zero: μ = 0.75 and σ = 0.707.
Start with the mean on the image and a tiny spread, as the device does. Reconstruction is almost free, at 0.02, but the KL costs about 2.25 nats because a sharp, off-centre code is very different from the prior. Press the best-code button. The mean moves halfway to the image, to 0.75, and the spread widens to 0.707. The decoder is now a little off, and the code is much cheaper. The total falls from 2.27 to 0.91.
Read the failure in three parts. The symptom: reconstructions look perfect but samples from the prior look like noise. The measurement: a large KL per image, with σ far below 1. The fix: keep the KL term, and watch the two terms separately rather than only their sum.
Slide 20 lays JEPA and VAE side by side. JEPA compares a learned feature of another view or region. The VAE compares the observed input under a decoder likelihood. JEPA usually outputs one embedding. The VAE outputs a distribution q(z | o). JEPA’s distribution constraint is recipe dependent, for example SIGReg across the batch. The VAE pays a KL to p(z) per observation. JEPA learns features that are easy to predict. The VAE learns inference and a generative latent model. Both so far describe a single frame. Neither knows what an action does. The next chapter adds actions. (For the full VAE story, see the Lecture 7 lesson and VAE and VQ-VAE.)
Chapter 3
Follow a recorded log of images and actions through the encoder and predictor, and see exactly which numbers the loss compares
A robot arm pushes a T-shaped block across a table. A camera records every frame, and the controller records every push. That log is all the data you have. You do not touch the robot again. From this log alone, you must build a model that can later answer: “if the block looks like this and I push like that, what will it look like next?”
The model in the lecture is LeWM (LeWorldModel, Maes et al. 2026). It has two learned parts. The encoder E turns each frame into a vector z, the embedding. The predictor F takes the vectors seen so far plus the recorded actions and guesses the next vector. Slide 22 draws it in one line: “predict the next latent from this state and action”, written ẑt+1 = F(zt, at).
Before the details, slide 21 draws a line that the rest of the lecture respects. Training and planning are two different jobs.
| Model training | Planning at time t | |
|---|---|---|
| Inputs | Recorded images and actions | The current context and a goal image |
| Compares | Predicted and observed next embeddings | The predicted outcome and the desired goal |
| Updates | Encoder and predictor weights | The mean and variance of an action proposal |
| Real world | Data already collected | Moves only when actions run |
Keep it in mind: this chapter changes weights, and the model never leaves the log. Chapter 6 freezes the weights and changes the actions.
A tensor is a grid of numbers with a fixed shape. Slide 23 lists the four that one training batch holds. Here B is the batch size, T the number of frames in a clip, D the embedding size, and A_block the numbers in one action block.
| Tensor | Shape | Meaning |
|---|---|---|
| obs | B × T × 3 × 224 × 224 | Recorded images |
| actions | B × T × A_block | Recorded action blocks |
| emb | B × T × D | The encoded observed images |
| pred, target | B × (T − 1) × D | Aligned next-step pairs |
Why T − 1 at the end? A clip of T frames has only T − 1 “next frames” to predict. The last frame has no successor in the clip.
Slide 24 quotes the paper. The predictor is a transformer with 6 layers, 16 attention heads and 10% dropout, about 10 million parameters. It reads a history of frame embeddings with a causal mask: each position may look only at itself and earlier positions. It brings in the action through AdaLN (adaptive layer normalisation), where the action sets the scale and shift inside each layer. The AdaLN weights start at zero, which the paper says steadies training. A projector follows the predictor, the same design as the one after the encoder. All parts learn jointly under one loss.
The mask matters because a model that can peek at frame t + 1 while predicting it would score perfectly and learn nothing. Slide 25 shows the alignment for four frames. Press a row below to see what each output may use.
Four observed frames z0 to z3. Pick an output row. Teal cells are the frames and actions it may read; the green cell is its target.
Slide 25’s table, drawn. The fourth output has no target in a four-frame slice, so it is left out of the loss: that is why the loss uses pred[:, :-1] against emb[:, 1:].
The device shows the rule. Output k reads frames z0 to zk and actions u0 to uk, and its target is zk+1. The last output has no target, so it drops out. That is why the loss pairs pred[:, :-1] with emb[:, 1:].
Slide 26 gives the prediction loss: the squared distance between the predicted next embedding and the encoder’s embedding of the real next frame. The prediction is the model under the recorded action. The target is the encoder output for the real next image. Slide 27 works one case with D = 2. The prediction is (0.2, −0.1) and the target is (0.5, 0.3).
(0.2 − 0.5)² + (−0.1 − 0.3)² = 0.09 + 0.16 = 0.25 ⇒ MSE = 0.25 / 2 = 0.125
One number comes out. It sends gradients to two places. Slide 29 states the part that surprises people: the target is not detached. The loss is ‖F(E(o), u) − E(onext)‖², and the encoder E appears on both sides. So the gradient with respect to the encoder weights has two paths: the prediction path and the target path. The recorded action and the raw images stay fixed. LeWM updates the encoder and the predictor together, with no stop-gradient.
That makes the shortcut of slide 28 dangerous. If the encoder gives every observation the same code, E(o) = c, and the predictor returns c, then the loss is exactly zero. The next term closes this door. Slide 30 adds SIGReg from Chapter 1, applied across the batch at each time slice.
The warm dot is the prediction. The blue dot is the target code. Press Train one step and watch both move. Turn on the stop-gradient and the target stays still. Try the constant encoder last.
The starting values are slide 27’s: prediction (0.2, −0.1), target (0.5, 0.3), D = 2. Each step moves a point by half of the gap: the gradient of ‖a − b‖²/D with respect to a is 2(a − b)/D, which equals a − b when D = 2, and the learning rate is 0.5. In the real model the codes come from a shared encoder, so this two-dot picture is a simplification. The SIGReg stand-in in the read-out is the penalty (s − 1)², where s is the spread of a batch of codes; with every code equal, s = 0.
Without the stop-gradient, one step brings both dots to the midpoint, (0.35, 0.10), and the loss falls to zero. With it, only the prediction moves, and it closes half the gap each step while the target never moves. So without it, the target code itself drifts. That drift is the lever of collapse: the easiest way to shrink the loss is to move the answer key. Press the constant encoder. The loss reads 0.000, a perfect score, and the SIGReg stand-in reads its maximum 1.0, weighted by λ = 0.1.
Read the failure in three parts. The symptom: training loss near zero and planning that never reaches any goal. The measurement: the spread of the batch’s embeddings, which falls toward zero. The fix: the SIGReg term, with weight 0.1 in the lecture’s code.
Slide 31 makes two side remarks. LeWM predicts the next embedding and regularises embeddings with SIGReg. To see its predictions as pictures, fit a separate decoder afterwards. In the paper’s PushT ablation (Table 7), LeWM without a decoder loss scored a success rate of 96.0 ± 2.83, and with a pixel loss added it scored 86.0 ± 7.54. The slide is careful: this is specific to the experiment, and the RSSM route of Chapter 5 does use image reconstruction. Slide 32 then shows the step as six lines.
# one LeWM training step. Shapes: obs [B,T,3,224,224], actions [B,T,A_block] emb = encoder(obs) # [B, T, D] pred = predictor(emb, actions) # causal history, [B, T, D] L_pred = mse(pred[:, :-1], emb[:, 1:]) # aligned next-step pairs, no detach L_reg = mean([SIGReg(emb[:, t]) for t in range(T)]) # across the batch, each time slice loss = L_pred + 0.1 * L_reg optimizer.zero_grad(); loss.backward(); optimizer.step()
Chapter 4
See why a model that scores well one step at a time can drift when it must feed its own guesses back in
You follow a GPS through a strange city. At every turn the screen shows your true position. Even if you misjudge one turn, the next turn starts from the truth, so the mistakes never pile up. Now the GPS fails. You must drive five turns by dead reckoning: you only count how far you went and which way you turned. A small error at turn one is still there at turn two. By turn five you are on the wrong street.
Slide 33 draws exactly this split for a world model. The two situations have names. Teacher forcing means every training step feeds the model the real context, taken from the recording, however wrong its last guess was. Open-loop prediction means the model feeds its own predicted latents back in as the context for the next prediction. The slide’s figure shows a PushT clip twice: the real frames on one row, and the imagined frames on a second row that slowly drift away from them.
The loss of Chapter 3 pairs pred[:, :-1] with emb[:, 1:]. Every prediction in that line starts from encoded real frames. That is teacher forcing, and it is the only thing a log can offer: the log holds no frames that the model imagined. It also makes training fast, since all positions run at once in one forward pass.
Planning cannot use it. At decision time there is one real observation, the current frame. Every later frame is a prediction. To score a plan of five actions, the model must predict step one, add that prediction to its context, predict step two from the longer context, and so on. The training loss never asked whether those later contexts, made of its own output, stay close to reality.
A toy shows the effect with numbers. A point goes round a circle of radius 1. The true motion turns it by a fixed angle each step. The model knows that angle almost right, but it is off by a small amount δ each step. After one step from a true position, the model’s point is off by the chord of an angle δ. After h open-loop steps the angles have added, so the point is off by the chord of an angle hδ.
error after h open-loop steps = 2 sin(hδ / 2) ⇒ δ = 5°: h = 1 gives 0.087, h = 5 gives 0.433
That is five times larger after five steps, for the same model. Teacher forcing would report 0.087 at every step. Play with it below.
Blue dots are the real path. Warm dots are the model rolled out on its own from the first frame. Teal ticks are one-step guesses made from each real frame. Drag the model’s error and the horizon.
Illustrative toy, not LeWM: a point turning 30° per step on a circle of radius 1, with a model whose turn is wrong by the angle you set. Errors are exact chord lengths, 2 sin(hδ/2), capped by the circle’s diameter, 2.
Set the error to 5° and look at the bars. The teal bars, the one-step errors, stay flat. The warm bars climb. Raise the steps ahead to 12 and the open-loop point is off by 1.0, half the way across the circle. Real dynamics can be worse: where small differences grow, as in a balancing task, errors multiply rather than add.
Read the failure in three parts. The symptom: a plan that looks perfect in imagination misses in the real world. The measurement: the open-loop error on held-out clips at horizon H, compared with the one-step error. The fix: keep the horizon short, and replan often, as Chapter 0 showed. LeWM plans only five model steps ahead (slide 64). Training with multi-step rollouts is another general remedy, but this lecture does not claim LeWM uses it.
Slide 34 spells out a point that is easy to get wrong in code. CEM, from Chapter 6, scores hundreds of candidate plans at once. Each candidate is a separate story about the future. Candidate A starts from the same observed latent history as candidate B. Then they split. A’s first predicted latent depends on A’s first action block, and is appended to A’s history only. B does the same with B’s own. From then on, A’s predictions use A’s history and A’s actions, and nothing of B’s.
| Stage | Candidate A | Candidate B |
|---|---|---|
| Start | The same observed latent history | The same observed latent history |
| First transition | Use action block uA,0 | Use action block uB,0 |
| Next input | Append the predicted zA,1 | Append the predicted zB,1 |
| Later transition | Use A’s history and actions | Use B’s history and actions |
In shapes: the observed context is 1 × T_hist × D. Copy it N times to get N × T_hist × D, one private copy per candidate. At each model step the predictor reads N × (T_hist + h) × D and the matching actions, and returns one new latent per candidate, N × D. Each branch appends its own. With N = 300 and H = 5 (slide 64), the model runs five times, each time on 300 branches at once. If you shared one history across candidates, every plan would be scored on a blend of all the others.
Chapter 5
Add a notebook and a reward predictor to the world model, so it can score imagined futures that no real environment ever produced
You are playing a video game, and the screen only shows what is in front of you. A door closed behind you, but you remember it is there. You keep a small notebook in your head: where you have been, what you heard. You also keep a guess about things you cannot see. Both help you decide where to go, and both are partly wrong.
PlaNet (Hafner et al., 2019) builds this into a world model. It is the second route in Chapter 1: reconstruct the image, like the VAE of Chapter 2, and add a reward. The model is the RSSM, the recurrent state-space model.
A reward is a number the task hands out after each step, such as +1 for reaching the goal. Slide 35 shows the data-collection loop from the PlaNet paper. The agent calls env.step(a) inside an action-repeat loop. Each call returns a reward and the next observation. With action repeat R, the same action runs R times, and the training label sums the R small rewards. The result goes into the replay set as a triple (o, a, r).
That is a real reward, from the task’s rules. But an imagined state never passed through env.step. Nobody can ask the task what an imagined state was worth. So PlaNet learns a reward predictor: a small network that guesses the reward from the model’s own state.
Rewards can be tricky to learn from. Slide 36 shows the Reacher task, where a two-joint arm must touch a red ball. The reward is 1 if the fingertip is within a threshold distance of the goal, and 0 otherwise. Take a threshold of 0.06. At distance 0.04 the reward is 1. At 0.08 it is 0, and at 0.20 it is also 0. A near miss looks exactly like a far miss.
The red disk is the goal. The hand is the teal dot. Drag the distance, or press a preset from the slide.
The rule is the lecture’s r = 1[d ≤ R_goal + R_tip], with R_goal + R_tip = 0.06 as in the slide’s example. The drawing is schematic and not to scale.
The reward is almost always 0, and it tells the learner nothing about how close it was. This is what sparse means. A reward predictor trained on such labels has little to learn from until the arm gets lucky.
The RSSM splits its state in two (slide 37). The deterministic state ht is the notebook: a vector that a recurrent network updates from the last state, the last guess and the last action. It carries memory and has no randomness. The stochastic state st is the guess about what is happening now. It is sampled, so it can stand for several possibilities that fit the same memory. Slide 38 gives four rules:
There are two ways to get st. With a real image in hand, sample from the posterior q(st | ht, ot), the VAE encoder of Chapter 2. With no future image, as in imagination, sample from the prior p(st | ht). Training makes the prior match what the posterior found, so that imagination, which must use the prior, stays on track.
The reward head looks at the state and writes a number: r̂t = ρ(ht, st). Slide 39 treats it as a Gaussian with unit spread. Then −log N(r; r̂, 1) equals half the squared error, plus a constant. With the observed r = 1 and a prediction of 0.4, the squared-error term is ½ × 0.6² = 0.18. The slide stresses a point: this loss fits recorded rewards. It does not push the predicted reward up.
Slide 40 takes a one-weight head to show who gets the gradient. Let r̂ = w · s, with w = 2 and s = 0.2, so r̂ = 0.4. The label is 1.
∂L/∂w = (r̂ − r) · s = (−0.6)(0.2) = −0.12 ∂L/∂s = (r̂ − r) · w = (−0.6)(2) = −1.2
The state s gets a gradient ten times larger than the weight. That is the point of the slide: the reward error trains the head and the shared inference and recurrent path that produced s. The state is shaped to keep what predicts reward. The observed reward is a label, not an extra input to q(st | ht, ot). Try it.
The head predicts r̂ = w × s. The label is 1. Press Train and watch the prediction climb. The bars show how large each gradient is.
Slide 40’s numbers: w = 2, s = 0.2, label r = 1, loss ½(r̂ − r)². Each step moves both w and s by 0.1 times their gradient. A real state s is a vector made by the encoder and the recurrent network, so its gradient flows on into their weights; here it is one number.
At the start the loss is 0.18 and the gradient on s is −1.2. After one step s has moved from 0.20 to 0.32, while w moved by only 0.012. After 20 steps the prediction is 1.00, almost exactly the label. Most of the change came from s, which rose from 0.20 to 0.49, while w moved only from 2.00 to 2.04. That is the whole idea: a state that is useless for the reward gets reshaped.
Slide 41 puts it together. Summed over time, three things are charged:
To plan, PlaNet starts from the current state and imagines each candidate action sequence U = (at, …, at+H−1) forward using the prior only, since no images exist in the future. At each imagined step the reward head writes a predicted reward. Slide 42 adds them up into a predicted return:
With γ = 1 and predicted rewards (0.2, 0.8, 1.0), the return is 2.0. CEM minimises a cost, so use J(U) = −Ĝ(U). Change the numbers below to feel the discount.
Three imagined steps, each with a predicted reward. Drag the rewards and the discount. The bars show each step’s contribution.
The slide’s numbers: rewards (0.2, 0.8, 1.0) and γ = 1 give Ĝ = 2.0. The rewards are predictions made on states from the prior; a wrong reward head or a drifting state makes the score wrong in the same way.
Slide 43 sets the two systems side by side. PlaNet needs reward labels, learns a reward head, sums predicted rewards into a return, and prefers high return. LeWM needs no reward labels at all. It receives a goal image at planning time and prefers plans whose final predicted embedding is close to the goal embedding. Chapter 6 builds that cost.
| Quantity | How it is computed | How the agent uses it |
|---|---|---|
| Observed reward rt | The task rule on a real step | A label in PlaNet’s replay set |
| Predicted reward | The reward head on (ht, st) | Scores one imagined step |
| Predicted return Ĝ(U) | The sum of predicted step rewards | PlaNet prefers high return |
| LeWM terminal cost J(U) | ‖ẑt+H − zgoal‖² | Prefer low cost, given a goal image |
Chapter 6
Turn a picture of the goal into a cost, then search for the actions that reach it by sampling, scoring, keeping the best and refitting
You are solving a sliding puzzle, and a photo of the finished puzzle lies on the table. You cannot touch the pieces yet. In your head you try a few sequences of moves. For each one you picture the final board, and you ask: how much does it look like the photo? You pick the sequence whose final board looks most like it. Then you make the first move, look at the real board, and think again.
That is planning with a goal image, and it is exactly what LeWM does at decision time. Everything in this chapter is the machinery behind that sentence.
Slide 44 defines the cost of a predicted future. Encode the goal image once with the same encoder, to get a vector zg = enc(og). Predict where a plan ends, giving ẑH. Then the cost is the squared distance between the two.
The distance is measured in the full embedding space, with all D coordinates. Slide 45 warns about a trap. The slides often draw a two-dimensional picture of embeddings made by PCA (a way to squash many dimensions into two for drawing). That picture is only for the eye. Two plans that look close in the picture can be far apart in full space, so the scoring must never use it.
Slide 46 then states the planning problem. The optimisation variable is the plan U = (u0, …, uH−1), a list of H action blocks. The encoder and predictor stay fixed. We are looking for the U with the smallest cost: a* = argmin C(ẑH). The search is over actions, not over weights.
The cross-entropy method (CEM) searches without any gradients. It needs only to try plans and compare costs. Slide 47 gives the algorithm, and it has four verbs that repeat for K rounds:
Slide 48 names the four indices, so no one mixes them up. The real time is t, and it moves only when an action really runs. Inside a plan, h = 0, …, H−1 counts the model’s imagined steps. The search round k = 0, …, K−1 changes the Gaussian’s mean and variance. The candidate i = 1, …, N is one plan in one round.
A plan has H action blocks, each with A_block numbers, so U lives in a space of H × A_block numbers. Slide 49 puts a Gaussian on that whole space. Each coordinate has its own mean and its own variance: qk(U) = N(vec(U); μk, diag(vk)). The word diag means the coordinates vary independently. Each round draws N plans, a tensor of shape N × H × A_block.
Slide 50 starts it plainly: μ = 0, v = 1. To sample, draw standard noise ε and compute U = μ + √v ⊙ ε, exactly the trick of Chapter 2. Then clip the result to the allowed action range. Slide 51 rolls every plan out: copy the same observed context into N separate branches (Chapter 4), apply each branch’s action block, and append each predicted embedding. Slide 52 scores each: Ji is the sum over the D coordinates of the squared gap between its final embedding and the goal. That gives one number per candidate. All plans share the same goal and the same start.
Watch all four verbs in a toy world where each action block is a push of two numbers, the model adds the pushes up, and the goal sits at (1.4, 0.9). The numbers are the lecture’s PushT budget: 300 plans, 30 elites, 5 steps.
Each dot is where one of 300 plans ends. The blue diamond is the goal. Press Next phase to sample, score, keep the 30 elites and refit. The dashed ring is the Gaussian the search now holds.
Illustrative toy, not LeWM: the latent state is 2D, the model adds each of the 5 action blocks to it (so the model is exact), the start is (0, 0), each action number is clipped to −0.5 to 0.5, and the search starts at μ = 0, v = 1. N = 300 and Ne = 30 are the budget slide 64 reports; the random draws are seeded.
Press the phase button through one round. The first sample is a wide cloud around the start, because the first Gaussian is centred on “do nothing”. Scoring shades each dot by its cost. The elites are the 30 dots nearest the goal. The refit shrinks the ring and slides it toward them. After five rounds the cloud is a much tighter knot around the goal: in our seeded run the 30th-best plan of the fifth round ends about 0.13 away, and the spread per action number falls from 1.00 to about 0.14.
Why exactly these two formulas? Slide 55 shows that they are the answer to a question: “which Gaussian makes the elite plans most probable?” Pick the Gaussian parameters η that maximise the average log-probability of the elites. For a Gaussian with independent coordinates, calculus gives a clean result:
μnew = mean(Uelite) vnew = mean( (Uelite − μnew)² )
The new mean is the average elite plan, coordinate by coordinate. The new variance is how far the elites spread around it. Slide 54 adds the next sampling step: draw a small perturbation around the fitted mean, with spread √v. The fitted spread sets the size of the next round’s steps. Wide elites mean wide exploring. Tight elites mean fine tuning.
def cem(model, z_ctx, z_goal, lo, hi, H=5, N=300, Ne=30, K=30): mu = zeros(H, A) # [H, A_block] var = ones(H, A) for k in range(K): U = (mu + var.sqrt() * randn(N, H, A)).clamp(lo, hi) # sample [N, H, A] z = rollout(model, z_ctx, U) # frozen model, N branches -> [N, D] J = ((z - z_goal) ** 2).sum(-1) # score [N] elite = U[J.topk(Ne, largest=False).indices] # keep [Ne, H, A] mu, var = elite.mean(0), elite.var(0, unbiased=False) # refit return mu # or the best plan seen
Read the failure in three parts. The symptom: the planner returns plans that look good in the model but miss the goal. The measurement: compare the cost of the returned plan in the model with the real distance to the goal after executing it. The fix: the next chapter shows the usual causes, and Chapter 4 showed the usual cure.
Chapter 7
Work one whole round with four plans on paper, then see three ways the search can settle on the wrong answer
You are driving a toy car along a straight line, two moves only. Each move is a push, forward or back. You want the car to stop exactly one metre from where it started. A stop matters: arriving at speed would carry it past. With so few moves you can write every number on a napkin. That is slides 56 to 61, and it is the best way to trust CEM before the big model hides the numbers.
Slide 56 gives the world: position p and speed v. Each step, the position moves by the speed, and the speed moves by the push a.
pnext = p + v vnext = v + a start: p0 = 0, v0 = 0 goal: p2 = 1, v2 = 0 a0, a1 in [−1, 1]
A plan is two numbers, (a0, a1). Slide 57 rolls four plans through the world. Take (1, 1). Step one: p1 = 0 + 0 = 0, and v1 = 0 + 1 = 1. Step two: p2 = 0 + 1 = 1, and v2 = 1 + 1 = 2. The car is at the right place, doing speed 2. The other three work the same way:
| Plan (a0, a1) | After a0: (p1, v1) | End: (p2, v2) |
|---|---|---|
| (0, 0) | (0, 0) | (0, 0) |
| (1, 1) | (0, 1) | (1, 2) |
| (1, −1) | (0, 1) | (1, 0) |
| (0.5, −0.5) | (0, 0.5) | (0.5, 0) |
Slide 58 turns the goal into one number. Three terms add up. The first, (a0 − 1)², is the squared error of the final position, since p2 equals a0. The second, (a0 + a1)², is the squared final speed, since v2 = a0 + a1. The third, 0.1(a0² + a1²), charges for effort. The slide says plainly that this third term is “an added classroom choice”, not part of the goal. Slide 59 fills the table. Check (1, −1) yourself: 0 + 0 + 0.1 × 2 = 0.20.
Two plans have the lowest costs, 0.20 and 0.30. Take them as the elites. Slide 60 refits the mean as their average, coordinate by coordinate: ½[(1, −1) + (0.5, −0.5)] = (0.75, −0.75). Plug it into the cost: (0.75 − 1)² = 0.0625, then (0.75 − 0.75)² = 0, then 0.1 × (0.5625 + 0.5625) = 0.1125, for a total of 0.175. The slide’s remark is worth a pause: the mean differs from every sampled candidate, and here it is better than both elites.
Slide 61 does the same for the spread. The elites lie +0.25 and −0.25 from the mean in the first coordinate, and the reverse in the second. The variance is the average squared deviation, vnew = (0.0625, 0.0625), and the standard deviation is its square root, (0.25, 0.25). The next Gaussian is centred on (0.75, −0.75) with spread 0.25. Change the effort weight below and see which plans become elites.
Four plans in the plan square. Green rings mark the elites. The teal cross is the new mean and the dashed box is one standard deviation. Drag the effort weight, or keep three elites instead of two.
The four plans, the three-term cost and the refit formulas are slides 57 to 61. At the default weight 0.1 you get the slide’s numbers: costs 1.00, 4.20, 0.20, 0.30, mean (0.75, −0.75) with cost 0.175, and spread (0.25, 0.25). Other weights and a third elite are our additions.
At weight 0.1 the table matches the slide to the digit. Raise the weight to 1 and effort dominates: the quiet plan (0, 0) becomes an elite, because doing nothing is cheap. At weight 0 only the goal matters. With three elites, the refit mean drifts toward the worse plan, and the spread grows. That is the knob Ne: fewer elites make the search sharper and riskier.
Slide 62 lists them in one line each.
The next device shows the last two at once. The cost has two valleys, at −1 and +1, with a hill between them. Run CEM from the middle and read the cost at the refit mean. Then switch to a model that is wrong about which valley is deeper.
Blue line: the real cost of a one-number plan. Warm dashed line: the cost the model believes. Dots are sampled plans, green when elite. The teal bell is the Gaussian CEM holds.
Illustrative toy. Equal valleys: the cost is min((a − 1)², (a + 1)²) and the model is right. Wrong model: the real cost is min((a − 1)² + 0.6, (a + 1)²), so the left valley is better by 0.6, but the model believes min((a − 1)², (a + 1)² + 0.6). Each round keeps the best 25% of the plans (at least 2). The random draws are seeded; press Reset for a new run from the same start.
With 40 plans and two equal valleys, the first refit puts the mean at −0.03, on the hill, where the real cost is 0.94. Six rounds later the mean is at 0.56, still short of a valley, with real cost 0.19. Switch to the wrong model, and the search is happy: by round 3 it sits at +1 with a spread of 0.02, sure of itself, and the real cost is 0.60 instead of 0.00. Drop the plans to 12 and the equal-valley run happens to fall into one valley; luck, not skill.
Read the failure in three parts. The symptom: the returned plan costs more than its elites did, or the agent misses in the real world though the model was happy. The measurement: the real cost at the refit mean, next to the real cost of the best elite, and the real cost after execution against the model’s. The fix: more samples, a better model, or replanning to rescue a bad first guess.
Slide 64 reports what LeWM actually uses, from the paper’s appendix. Each planning step draws N = 300 plans and keeps Ne = 30 elites. The horizon is H = 5 model steps. On PushT the search runs up to 30 CEM rounds. Then the model executes all five planned steps before replanning, which is why the PushT clip on slide 65 shows no feedback replanning. Compare this with Flappy Bird in Chapter 0, which executes one action and replans. Both are legal uses of the same planner; the choice depends on how good the model is and how fast the world surprises you.
Chapter 8
Stop searching from scratch at every decision: learn a habit that picks actions and a gut feeling that scores states, both inside the model
A taxi driver does not run a search before every turn. After years of practice she has a habit: at this kind of junction, with this kind of traffic, she takes this lane. She also has a gut feeling about positions: “this side street will be slow, even though it looks short”. A chess beginner searches every move. A master looks two moves ahead and trusts a feeling about the position they reach.
Those are the two things that Chapters 5 to 7 lack, and the last seven slides of the lecture add them. CEM is a planner: it searches again at every decision, and it forgets everything after. Slide 66 introduces the other route: model-based reinforcement learning uses predicted consequences to improve the expected return, the total reward an agent can expect over time. The lecture’s example is Dreamer (Hafner et al., ICLR 2020), and its three-part figure tells the story: (a) learn the dynamics from experience, (b) learn behaviour in imagination, (c) act in the environment.
In everyday words: Dreamer plays with its model, not with the world. It lives through many imagined futures, learns what works, and then walks into the real world with a habit. (The model-based RL lesson, the world models lesson and the actor-critic lesson build the parts from zero.)
Slide 67 puts the two routes in one table, with the caution that they are not rivals. CEM names a search method. Model-based RL names how prediction improves behaviour. A single RSSM can serve both.
| Question | Planning with CEM (PlaNet) | Learning a policy (Dreamer) |
|---|---|---|
| How is the model used? | To score candidate action sequences | To generate imagined trajectories for actor-critic updates |
| What improves? | A proposal over plans, at this one decision | Shared policy and value weights, kept across decisions |
| How is an action chosen? | Run a search on predicted returns | Evaluate the learned actor, one forward pass |
| What do they share? | An RSSM predicts states and rewards | The same RSSM predicts states and rewards |
A plan looks only H steps ahead. What if the best thing happens at step H + 1? A planner that stops at H cannot see it. Slide 68 fixes this with a value function. First, the state: xt = (ht, st), the RSSM’s notebook and guess together. Then:
Reward predicts one step. Value estimates the rest. Now a plan’s score gets a tail: add up the H imagined rewards, then add the value of the state where the plan stops, discounted by γH. The slide works one case with H = 2 and γ = 0.9. Plan A earns rewards (5, 0) and ends in a state worth 0. Plan B earns (0, 0) and ends in a state worth 10.
A: 5 + 0.9 × 0 + 0.9² × 0 = 5.0 B: 0 + 0.9 × 0 + 0.9² × 10 = 8.1
Without the tail, a planner would pick A. With it, B wins, because B ends where the good things are. Move the sliders below.
Plan A earns (5, 0) then stops in a state worth the A slider. Plan B earns (0, 0) then stops in a state worth the B slider. Turn the value tail off to see what a plain H-step planner sees.
Slide 68’s example: H = 2, γ = 0.9, rewards (5, 0) and (0, 0), end-state values 0 and 10, giving 5 and 8.1. The score is ĜH = ∑k=0H−1 γk r̂t+k+1 + γH V(x̂t+H). All numbers are the slide’s or your own sliders.
With the default sliders B scores 8.1 and A scores 5.0. Switch the tail off and B falls to 0.0: a planner with no value is blind to everything past step 2. Now push γ down to 0.5. B’s tail is worth 0.25 × 10 = 2.5, and A wins again. A heavy discount makes the agent short-sighted, which is sometimes the right call.
Where does the value network Vψ come from? It is trained inside the model (slide 69). From a state xt, let the current policy pick an action, at ~ πφ(· | xt), and let the model imagine the next state, x̂t+1 ~ pθ(· | xt, at). Build a target from the predicted reward plus the value of the next state:
yt = r̂t+1 + γ (1 − dt+1) · sg[ Vψ(x̂t+1) ] LV(ψ) = ½ E ( Vψ(xt) − sg[yt] )²
Two pieces need words. The flag d is 1 if the episode ended, which cuts the future off. The sg is the stop-gradient of Chapter 1: the target is treated as a fixed number, so the critic is pulled toward it and the target is not pulled back. Using your own estimate of the next state inside your target is called bootstrapping. A worked case: r̂ = 1, γ = 0.9, d = 0 and V(next) = 10 give y = 1 + 0.9 × 10 = 10. If V(xt) is 8, the loss is ½ (8 − 10)² = 2, and the critic is pulled up toward 10. The slide adds that Dreamer uses multi-step λ-returns, a blend of returns of different lengths, and that the full derivation comes after fall break.
Now the habit. Slide 70 recalls the policy gradient: nudge the policy’s weights so that actions that did better than expected become more likely.
The slide makes the contrast sharp. CEM updates a plan proposal, and throws it away after the decision. The policy gradient updates a policy that serves every future decision. Try the update on a single imagined state with three actions whose true returns are 2, 5 and 3.
Warm bars: the policy’s probability of each action. The blue marker is the value V, the policy’s average return. Below, each action’s advantage. Press Update the policy.
Illustrative toy: one imagined state, three actions with returns 2, 5 and 3, and a softmax policy. The update is the exact policy gradient for a softmax, which is πa (returna − V) for each logit, with step size 0.3. A real actor is a network over many states, and the returns come from imagined rollouts.
At the start the policy is uniform. V = (2 + 5 + 3) / 3 = 3.33. The advantages are −1.33, +1.67 and −0.33. After one update, the middle action rises from 33.3% to 39.1%. After 20 updates it holds 93.4%. Notice that the best action is not chosen by search. It simply gets pushed up each time it beats the average.
Slide 71 shows the last trick. In Dreamer the imagined trajectory is built from differentiable pieces: at = gφ(xt, εt) and xt+1 = fθ(xt, at, ξt), where ε and ξ are the noise draws of the reparameterization trick from Chapter 2. So the score ĜH is a smooth function of φ, and you can backpropagate straight through actions, states, rewards and values to the actor’s weights: φ ← φ + α ∇φ E[ĜH]. During the actor update the world model (θ) and the critic (ψ) are held fixed, but derivatives still flow through them. The RSSM supplies the dynamics. The actor and the critic learn how to use them.
Slide 72 closes the lecture in three lines. Representation and dynamics losses make future states predictable. CEM searches actions at a decision, while a learned policy reuses behaviour across decisions. Value connects finite imagination to long-term return.
Read the failure in three parts. The symptom: the agent scores high in imagination and low in the real world. The measurement: imagined return against real return for the same actor. The fix: keep imagination short, retrain the model on the states the actor actually visits, and treat a big gap as a model fault, never as an actor success.
Chapter 9
Review the lecture on one page: the failures, the numbers, a checklist, the core code, and where it leads next
You can now say, from memory, what Lecture 10 argued. Representation and dynamics losses make futures predictable. A cost or a reward makes futures comparable. Then there are two ways to turn that into action. CEM searches a Gaussian over whole plans at each decision. A learned policy and value keep the lesson of many imagined lives and reuse it at every decision. Each part has its own way to fail, and a good engineer names the failure before touching the code.
| Failure | Symptom | What to measure |
|---|---|---|
| Embedding collapse (Ch 1, 3) | Prediction loss near zero, planning useless | Spread of embeddings across a batch |
| Open-loop drift (Ch 4) | Plans look perfect in imagination, miss in reality | H-step error against one-step error |
| Sparse reward (Ch 5) | The reward head learns almost nothing | Share of steps with a nonzero reward |
| Split valleys (Ch 7) | The refit mean is worse than its elites | Cost at the mean against the best elite |
| Model exploited (Ch 7) | A confident plan that fails when executed | Real cost after executing, against the model’s |
| Blind beyond the horizon (Ch 8) | The planner ignores a distant payoff | Plan score with and without a value tail |
Try the habit on six made-up cases. Each lists what a run’s diagnostics show. Name the failure. One case may look alarming and be fine.
Read the run. Press the failure that explains it.
All six cases are made up to practise the habit; every number in them is illustrative, not a measurement of a real system.
| What | Number or form | Why it matters |
|---|---|---|
| EMA teacher (slide 9) | 0.9 × 2 + 0.1 × 4 = 2.2 | The teacher is a slow shadow of the student. |
| Collapse (slide 10) | E(x) = c gives prediction loss 0 | Agreement alone is not learning. |
| LeJEPA loss (slide 11) | (1 − λ) agreement + λ SIGReg | A shape rule on the whole batch. |
| VAE loss (slides 17, 19) | −log p(o | z) + KL(q ‖ N(0, I)) | Information has a price; zero at μ = 0, σ² = 1. |
| LeWM tensors (slide 23) | obs B×T×3×224×224; emb B×T×D; pred, target B×(T−1)×D | Pairs are aligned by one step. |
| MSE by hand (slide 27) | (0.2, −0.1) against (0.5, 0.3): 0.25 / 2 = 0.125 | One scalar feeds both branches. |
| LeWM loss (slides 30, 32) | L_pred + 0.1 × L_SIGReg | No detach, the regulariser stops the collapse. |
| Decoder ablation (slide 31) | PushT success 96.0 ± 2.83 without, 86.0 ± 7.54 with a pixel loss | Specific to that experiment. |
| Sparse reward (slide 36) | threshold 0.06: d = 0.04 gives 1; 0.08 and 0.20 give 0 | A near miss looks like a far miss. |
| Reward head (slides 39, 40) | loss 0.18; gradients −0.12 on w, −1.2 on s | The reward error also reshapes the state. |
| PlaNet return (slide 42) | γ = 1, rewards (0.2, 0.8, 1.0) give 2.0 | CEM minimises −Ĝ. |
| CEM budget (slide 64) | N = 300, Ne = 30, H = 5, up to 30 rounds on PushT | The real planner is this small. |
| CEM by hand (slides 59 to 61) | costs 1.00, 4.20, 0.20, 0.30; mean (0.75, −0.75) costs 0.175; spread (0.25, 0.25) | The mean can beat every sample. |
| Value tail (slide 68) | H = 2, γ = 0.9: A scores 5, B scores 8.1 | Value sees past the horizon. |
Three short sketches gather the lecture. encoder, predictor, SIGReg, rollout, actor, critic, reward_head and rssm.imagine stand for whatever your library provides. The CEM sketch is in Chapter 6.
# 1. LeWM training step, Ch 3: obs [B,T,3,224,224], actions [B,T,A_block] emb = encoder(obs) # [B, T, D] pred = predictor(emb, actions) # causal history loss = mse(pred[:, :-1], emb[:, 1:]) + 0.1 * mean([SIGReg(emb[:, t]) for t in range(T)]) # 2. Reward head, Ch 5: x = (h, s) from the RSSM; r [B] r_hat = reward_head(x) # [B] loss_r = 0.5 * ((r - r_hat) ** 2).mean() # a label, not an input to q # 3. Dreamer-style update inside the model, Ch 8: x0 [B, dim], horizon H x, rs, G = x0.detach(), [], 0 for k in range(H): a = actor(x).rsample() # a = g(x, eps), differentiable x = rssm.imagine(x, a) # prior only, no image G = G + gamma**k * reward_head(x) # predicted r_{t+k+1} G = G + gamma**H * critic(x) # the value tail actor_loss = -G.mean() # model and critic fixed, gradients flow through critic_loss = 0.5 * ((critic(x0) - G.detach()) ** 2).mean() # sg on the target
Slide 72 gives the thread: from learned predictions to better decisions. Lectures 5 to 7 asked what an encoder should keep (Lecture 5, Lecture 6) and how to keep many futures alive (Lecture 7). This lecture fills in the action-selection row of the four interfaces. The value derivation with λ-returns, which slide 69 defers, comes after fall break. The course site’s file for Lecture 11 is named for Dreamer, diffusion and flow matching.
Course credit. This lesson re-teaches Lecture 10 of Penn CIS 6280 World Models (Fall 2026), taught by Jiatao Gu. The slides are linked at the top of this page and in the footer: Lecture 10 slides. The course site is jiataogu.me/cis6280-world-models. The explanations, devices and toy numbers on this page are our own; the numbers labelled from slides come from the slides.
Now press Teach and explain, from memory, how one LeWM step is built, how one CEM round refits, and how a value and a policy are learned inside a model. Then go back to the hero and run the search again.