A world model only predicts. To act, an agent tries many action plans in imagination, scores each against a goal, and runs the first step. Drag the model error and watch re-planning hold the goal.
Learn how a robot turns a model that predicts into a choice of what to do: the cost, the search, the feedback loop, and the ways it fails.
Pick Plan once, then slide the real thrust down to 70%. The cart stops short. Now pick Plan every step and watch it recover. Then we build, piece by piece, the cost, the loop, and three ways to search for actions: CEM, MPPI and gradients.
You need a feel for position and speed (speed is how fast position changes) and what a probability is. We build the rest from zero.
Based on Lecture 9 of Penn CIS 6280: World Models (Fall 2026), taught by Jiatao Gu. Lecture slides. Every idea is re-taught here in our own words and devices; it follows Lecture 8, Latent World Models.
Toy cart, illustrative numbers. Position p and speed v update as p+v and v plus thrust, for 12 steps; the target is position 1 with speed 0. The model assumes thrust works at 100%; the slider is how well it really works. The cost is the lecture's: (p12 − 1)² + v12² + 0.1 × the sum of squared thrusts. The plan is the exact least-squares answer of Chapter 3.
Chapter 0
See why a model that predicts well still cannot act until you add a goal and a search
A first note on logistics. Lecture 9 met on Thursday 24 September 2026, on Zoom, because the instructor was travelling. Assignment 1 was due that day, and office hours ran from 4:30 to 5:30 PM on campus. That is all the logistics there are. The rest of this page is the idea.
Picture a small cart on a rail. You want it to roll to a mark one metre away and stop exactly there. You cannot try it first, because the cart is expensive and the rail ends in a wall. But you do have something valuable: a world model, a program that takes the cart’s situation and a push, and says what the situation will be a moment later.
The last lecture was about building that program. We gave it a state (what the cart is doing now) and an action (a push), and it predicted the next state. Lecture 9 asks the next question. The model can tell you what happens if you push. It cannot tell you which push to make. How do you get from a model that predicts to a cart that arrives?
Start with the plainest answer. Write down a few candidate plans, ask the model to imagine each one, and keep the best. The lecture draws exactly this. From the same current information, three plans fan out: do nothing, keep accelerating, or accelerate and then brake. The model predicts two or more steps of each. Only one ends at the goal and stays there.
Notice what changed. Before, we compared a prediction with what really happened, to fix the model. Now the model stays fixed. We compare predictions with what we want, to fix the plan. The lecture says it in one line: Lecture 8 gave actions and predicted; Lecture 9 uses predictions to choose actions.
The device below is that idea in miniature. The cart starts at rest at position 0. The target is position 1, and the cart must be stopped there, so its speed must be 0. Pick one of the three plans, and the model draws the cart’s predicted path over six steps.
Each button is one plan of six pushes. The purple path is what the model predicts. The pink line is the target. Try all three and compare where each one ends.
Toy cart, illustrative numbers: position changes by the speed, speed changes by the push, six steps from rest. The cost is the lecture’s arrive-and-stop cost with effort weight 0.1 (Chapter 2 explains it). The pushes are chosen so the third plan lands exactly on the target.
The first plan leaves the cart at 0. The second reaches position 1, but the cart is still moving at speed 0.4, so it sails past the mark one step later. The third arrives and stops. A person sees that at a glance. A computer needs a number that says so, and that number is the task cost in the read-out. Lower is better. Chapter 2 builds it.
Planning has two jobs, and it helps to keep them apart. The first is predict: given a start and a plan, what happens? That is the world model. The second is choose: which plan has the lowest cost? That is a search. The model never decides anything. It answers “what if” again and again, and a separate piece of code asks the questions.
This split gives the lecture its shape. First we say what the cost is. Then we plan in the one case where the math is exact, a model made of straight lines and bell curves. Then we meet three ways to search when the model is a neural network and the exact answer is gone: CEM, MPPI and gradients. We wrap all three in a feedback loop called model predictive control. Next we plug in the latent models of Lecture 8. Last we ask where it breaks.
The hero at the top already shows the loop. A cart plans from a model that thinks every push works at full strength. When the real cart is weaker, a plan made once lands short. A plan remade at every step does not, because each step starts from where the cart really is.
Chapter 1
Meet the five latent models from Lecture 8 in one place, and see why their small errors grow when you imagine far ahead
Before we plan, we need something to plan with. The lecture spends its first ten slides on a recap, and it is worth a slow look, because every planner later in the lesson plugs into one of these models. You can skip the details of how each one is trained. What matters is what each one offers to a planner: a way to start, a way to step forward, and a way to say how good the end is.
All five are latent models. A latent is a short list of numbers that stands for a picture. A camera frame has millions of pixels, but what matters (where the arm is, where the cup is) fits in a few hundred numbers. An encoder squeezes the frame into that list. The dynamics part then predicts the next list from the current list and an action. Planning in the small list is far cheaper than planning in pixels.
Pick a family below. Each card gives the four things the lecture lists for it: the encoder, the state it keeps, how it predicts, and how it is trained. The first two families are deterministic: the same input always gives the same output. The next three are stochastic: they hold a probability, and a rollout draws a sample from it.
Tap a family. Read how it starts a rollout, how it steps, and what you must carry along for every candidate plan.
Summarised from the lecture’s slides 3 to 10 (Hansen et al., TD-MPC; Maes et al., LeWorldModel; Ha and Schmidhuber, World Models, 2018; Hafner et al., PlaNet and Dreamer; Micheli et al., IRIS). “Carried state” is what a planner must copy for each candidate plan.
Two details matter later. First, TD-MPC keeps only the latent z, and its next latent is a function of the current one and the action: ẑt+1 = F(zt, at). The prediction is then fed back in, so the model can run for many steps without seeing a new picture. The lecture adds a warning: squared-L2 consistency alone permits collapse, where the encoder maps every frame to the same point and the loss is trivially zero. The full method also trains on reward and value, which keeps the latent useful.
Second, the stochastic families keep memory. A recurrent network (RNN) carries a hidden summary ht of everything so far. The RSSM, the model behind PlaNet and Dreamer, has two paths. To observe, it samples the state ut from a posterior that sees the history and the new frame. To imagine, it samples ut from a prior that sees only the history, then updates ht+1 = f(ht, ut, at). The lecture is firm on this point: during imagination the observation branch is not available. A planner lives entirely on the imagination path.
Slide 11 shows the danger of imagining far. Think of a person who walks and counts steps, but whose stride estimate is 8% too long. After one step, the guess is nearly right. After twelve steps, the guess has drifted by a whole stride or more. Each step feeds a slightly wrong position into the next step, so the errors do not cancel. They pile up. This is compounding error.
The lecture makes it exact with a one-line world. The true system is xt+1 = xt + at. The model is slightly off: x̂t+1 = g x̂t + at, with gain g = 1.08. The actions are +1 for six steps, then −1 for six steps. Define the error et = x̂t − xt. Then et+1 = g et + (g − 1) xt: the old error is multiplied by g and a new error is added.
Now the two ways to use the model. One-step prediction hands the model the true state at every step, so it can only be wrong by one fresh slip. Free-running rollout hands it its own last guess, so slips accumulate. At step 12 the lecture reports a one-step error of 0.160 and a rollout error of 5.823. That is 36 times larger. Move the knobs and see it.
Blue is the true system. Purple is the model running free on its own guesses. Raise the gain error, then lower the number of steps between real observations and watch the purple line get pulled back.
The lecture’s constructed example: x0 = x̂0 = 1, g = 1.08, actions +1 for six steps then −1 for six. At g = 1.08 and no real looks the model ends at 6.823 against a true 1, a rollout error of 5.823; fed the true state every step the error is 0.160. Looking every N steps resets the model’s state to the true one at steps 0, N, 2N and so on.
The remedy list on the slide has three parts: multi-step consistency (train the model on its own multi-step rollouts, not only on single steps), a validated rollout horizon (do not trust the model past the step where its error is known to be small), and state updates when real observations arrive (the slider above). The third one is the seed of model predictive control in Chapter 4.
Slide 12 adds a second price of imagination: time. A token model like IRIS predicts a frame as a run of K codes, one after another. For H frames that is about H × K serial steps. Each step must wait for the one before. The slide lists fixes and their tradeoffs. Fewer tokens or frame-level prediction give fewer steps, but compression may lose detail. Caching saves repeated work, but memory grows or history is forgotten. Distillation cuts iterative sampling steps, but needs more training. Hold this in mind. Chapter 10 returns to it as the planning budget.
Chapter 2
Turn “arrive and stop” into one number a planner can minimise, and see the loop that uses it
You tell a friend: “Park the car at the mark.” They understand. A computer does not. It needs a score for every way the parking could end, so that it can say one ending is better than another. That score is the task cost, written J. Low means good. The planner’s whole job is to find the actions with the lowest J.
The lecture builds the score for the cart from Chapter 0: arrive at position 1 and stop. Think about what can go wrong. The cart can stop short. It can reach the mark at speed, and so shoot past. Or it can get there smoothly but waste a great deal of energy. Each failure is one term of the score.
Play with the three terms. The device starts at the lecture’s three snapshots: no action, at the target but still moving, and at the target and stopped. Then drag the sliders to build your own ending.
The bar is J, cut into its three terms. Pick a snapshot, or drag the sliders for the final position, the final speed and the total effort.
The formula is the lecture’s (slide 14, target position 1, effort weight 0.1). The effort values in the snapshots are illustrative: 0 for no action, 0.2 for the fast arrival and 0.07 for the smooth one.
Look at the middle snapshot. The cart is exactly on the mark, so the position term is zero. But it is moving at speed 2, and that single term costs 4. Without the speed term, a planner would call “touch the mark at full speed” a perfect plan. A cost does not say what you meant. It says what you wrote. Chapter 10 comes back to this.
It is easy to mix up training the model with planning. Slide 15 puts them side by side, and the contrast is the key idea of the lecture.
| Learn to predict | Plan for a task | |
|---|---|---|
| What is minimised | The loss L between the model’s prediction fθ(st, at) and the observed next state | The cost J of the model’s imagined rollout of a candidate plan, measured against the goal g |
| Compared with | What actually happened | What we want to happen |
| What changes | The model parameters θ | The candidate actions A, with θ held fixed |
The slide ends with a line worth remembering: an accurate prediction is useful only after we decide which outcomes to prefer. The model tells you the weather. The cost tells you whether you want an umbrella.
Slide 16 draws the planner as a box, and the data flow is what makes it a real system. Inside the box, three parts talk. Action search proposes candidate actions A, a list of H pushes. The world model turns each into imagined consequences. The task evaluator turns the consequences and the goal into one cost, J(A). The result goes back to the search, which proposes better actions. All of this is internal and happens before the cart moves. The model parameters stay fixed throughout.
Only then does a real action leave the box. The environment responds, a new observation comes back, and the current information is updated. The dashed arrows of the slide are imagination. The solid arrows are the real world. Here is the evaluator as code, for the cart:
pythondef rollout(s0, A, f): # s0: (2,) = [position, speed] A: (H,) pushes S = [s0] for a in A: S.append(f(S[-1], a)) # the fixed world model, one step return np.array(S) # (H + 1, 2) def cost(S, A, goal=1.0, lam=0.1): pH, vH = S[-1] return (pH - goal)**2 + vH**2 + lam * (A**2).sum() # one float: lower is better
Chapter 3
Plan inside a world made of straight lines and bell curves, where the best action sequence has a closed-form answer
Before we plan with a neural network, the lecture plans in the one world where nothing is hard: the linear-Gaussian state-space model (LGSSM) of Lecture 4. If you built a Kalman filter, you know it. The point of doing this first is that every step has an exact answer. Later, with learned models, each step becomes an approximation. You will then know exactly what was lost.
Imagine tracking the cart again, but now the cart is jostled by small random bumps and you only see it through a noisy camera. The model has two lines. The state moves as sk+1 = A sk + B ak + wk: the next state is a straight-line function of the current state and the push, plus a bump w drawn from a bell curve with spread Q. The camera sees ok = C sk + νk, with its own noise R. In the cart, s is [position, speed], and A adds the speed to the position.
The planner does not hold the true state. It holds a belief, written p(st | Ht) = N(μt, Pt): a bell curve with centre μt (the best guess) and spread Pt (the doubt), given the history Ht of everything seen so far. The Kalman filter computes it. The planning variables are the next H pushes, A = (at, …, at+H−1). The model and the belief are given. The pushes are what we choose.
Fix a candidate plan. What will the cart do? Because no new camera frames arrive during imagination, the belief only flows forward, and the lecture gives two update lines, shown with their symbols below.
Do one step by hand. Let the doubt start at P0 = diag(0.01, 0.01) and let the bumps be Q = diag(0, 0.01). With A = [[1, 1], [0, 1]], the stretched doubt AP0A⊤ is [[0.02, 0.01], [0.01, 0.01]]. Add Q and P1 = [[0.02, 0.01], [0.01, 0.02]]. The trace (the sum of the diagonal) is 0.04. Now push with a = 0.2 from μ0 = (0, 0): μ1 = (0, 0.2). The push moved the guess. It did not touch the doubt.
That last sentence is the surprise of slide 18. With additive noise, the covariance is the same for every candidate plan. Actions change where the cloud is centred. They do not change how wide it is.
Now score a plan. A random future means a random cost, so the lecture takes the expected cost: the average over all the futures the belief allows. For a squared distance there is an exact identity: if s ~ N(μ, P), then E[‖s − g‖²] = ‖μ − g‖² + tr(P). The average squared miss is the squared miss of the centre, plus the width of the cloud.
Put the identity into the plan’s cost, J(A) = ∑j=1H [‖μj − g‖² + tr(Pj)] + λ ∑j=0H−1 ‖at+j‖². The trace terms do not depend on the actions, as we just saw. So when we compare plans, they cancel out of the comparison. The best plan is A* = argmin [ ∑ ‖μj(A) − g‖² + λ ∑ ‖a‖² ], where “argmin” means “the plan that gives the smallest value”. Planning under noise needs only the mean path.
One more fact finishes it. Because the dynamics are linear, each μj is a straight-line function of the pushes. A square of a straight-line function is a bowl, so J is a quadratic bowl in A. A bowl has one bottom, and a short piece of linear algebra finds it. Try it: scale the exact plan up and down, and watch the cost trace the bowl.
The purple line is the mean path for a plan of 6 pushes. The band is one standard deviation of position. Scale the plan, then raise the bumps. Watch what moves and what does not.
Toy cart, illustrative numbers: A = [[1, 1], [0, 1]], B = [0, 1], H = 6, target (1, 0), P0 = diag(0.01, 0.01), bumps Q = diag(0, q), effort weight 0.1. The exact plan solves the normal equations of the sum-over-steps cost. The bowl on the right is J as you scale that plan.
Read the device. As you drag the bump size, the band widens and J rises, but the best scale stays at 100%. The covariance moved the whole bowl up without shifting its bottom. As you drag the scale, the band does not change at all, only the centre does.
The exact answer, for the curious, is the least-squares solution. Stack the predicted means into one vector m = mfree + G a, where G collects how each push changes each later state. Then minimise ‖m − g‖² + λ‖a‖². The answer is a* = (G⊤G + λI)−1 G⊤(g − mfree). That is one matrix solve. The hero used this same formula.
Now see what the lecture is building toward. The recipe has four steps: initialise from the belief, predict the mean and covariance, evaluate the expected quadratic cost, and optimise a quadratic in the actions. Slide 22 will put this next to a learned model, and each of the four steps will change.
Chapter 4
Plan a long way ahead, act on only the first step, look at what really happened, and plan again
You drive to a friend’s house you have never visited. You look at the map and plan the whole route: left, right, right, left. Then you drive. At the first turn a road is closed. You do not follow the old plan into the barrier. You look up, find where you are now, and plan again from there. You are not a worse planner for it. You are a better driver.
That habit has a name. Model predictive control (MPC) is a loop of three moves. Plan: find the best action sequence for the next H steps. Act: carry out only the first action of it. Look: take the real new observation and update what you know. Then the loop starts over from the new situation. The lecture writes it in two lines.
Why throw away a good plan? Because it was made with a model, and a model is a guess. Chapter 1 showed that predictions drift. The first action of a plan is the one whose prediction is most reliable, because it is the nearest. The last action is the one that depends on the most guesses. MPC trusts the near, checks it against the world, and re-aims. The hero already shows the gain: a plan made once stopped short, and a plan remade at every step did not.
Slide 23 names two numbers that are easy to confuse. The prediction horizon H is how far ahead each plan looks: how many model steps are imagined. The execution interval K is how many of the planned actions are carried out before the next look. The slide plans H = 5 steps in both pictures. In one, K = 1: act once, observe, replan. In the other, K = 3: act three times blind, then observe and replan.
Think of H as headlights and K as how long you drive between glances at the map. Long headlights let you see a distant turn. Frequent glances let you catch a mistake early. They are separate dials, and they cost different things. A large H makes every plan more expensive to compute. A small K makes you plan more often. A large K is cheap, and it leaves you blind for longer.
Try it. The cart below has a real thrust at 80% of what the model believes, and a gust adds speed at step 8. The planner replans from the real state every K steps with a horizon of H. Set K to 1, then to 5, and watch how long the gust bends the path.
Teal ticks mark each replan. The blue path is the real cart. Change H and K, and switch the gust on and off.
Toy cart, illustrative numbers. The model believes thrust works at 100%; the real cart gets 80%. The gust adds 0.08 to the speed at step 8. Each plan is the exact least-squares plan for the sum-over-steps cost of Chapter 3, started from the real state, with effort weight 0.1. 20 steps in all. “Worst miss” is the largest distance from the target between step 10 and step 20.
Results on this toy: with K = 1 the cart is back within 0.06 of the target after the gust. With K = 5 the worst miss is about 0.21. The cart planned in open loop for five steps and had no way to know it had been pushed. In both cases the plan itself was the same kind of calculation. Only the feedback differed.
The slide titled “Planning with a learned latent world model” gives a four-row table. We keep the loop and the objective. We replace the calculations inside it. Here it is with a plain gloss on each cell.
| Step | Linear-Gaussian + quadratic cost | Learned latent world model |
|---|---|---|
| Initialise | The current Gaussian belief | An inferred latent state, or the retained history |
| Predict | Exact mean and covariance recursion | A learned rollout; sample futures if needed |
| Evaluate | Exact expected quadratic cost | A task cost on the predicted outcomes |
| Optimise | A quadratic in the actions: one solve | Generally a nonlinear function of the actions: a search |
The new target is A*t ≈ argmin Epθ(S | Ht, A)[Cost(S, A; g)]. Read it as: among action sequences, pick the one whose imagined outcomes, averaged over the model’s own randomness, cost least. The symbol “≈” is honest. A search will not find the exact minimum. The next three chapters are three different searches.
Chapter 5
Throw a cloud of guessed plans at the model, keep the best few, and aim the next cloud where they landed
You are blindfolded and want to find the highest point of a hilly field. You throw twenty darts at random. You walk to where they landed and pick up the five that landed highest. Then you throw twenty more, this time near the middle of those five. After three rounds, your darts bunch up on the hilltop. You never needed a map of the slope. You only needed to compare heights.
That is the cross-entropy method (CEM), the search PlaNet uses. The field is the cost J(A) of an action sequence. A dart is a candidate plan. The height of a dart is the cost your world model predicts for it. CEM never asks which way is downhill. It only needs the model to roll out a plan and return a number.
The lecture keeps the plan short so we can check by hand: H = 2 pushes, A = (a0, a1), on the cart from the hero, starting at rest. The cart ends at position a0 with speed a0 + a1. So the cost is J = (a0 − 1)² + (a0 + a1)² + 0.1(a0² + a1²). CEM keeps a bell curve over plans: qk(A) = N(μ(k), diag v(k)), a centre and a spread for each push. The index k counts search rounds. Real time has not moved.
Step one is sample: draw N candidates from the bell curve. The slide supplies four. Step two is roll out and score: send each through the model and take its cost. Check the table.
| Candidate (a0, a1) | End position | End speed | Cost J | Elite? |
|---|---|---|---|---|
| (0, 0) | 0 | 0 | 1 + 0 + 0 = 1.00 | no |
| (1, 1) | 1 | 2 | 0 + 4 + 0.2 = 4.20 | no |
| (1, −1) | 1 | 0 | 0 + 0 + 0.2 = 0.20 | yes |
| (0.5, −0.5) | 0.5 | 0 | 0.25 + 0 + 0.1 = 0.30 | yes |
Step three is select the elites: the Ne candidates with the lowest cost. Here Ne = 2, so (1, −1) and (0.5, −0.5) win. Step four is refit: fit a new bell curve to the elites. Their average is the new centre, and their spread is the new spread.
Those are the slide’s own numbers: μ0 = (0.75, −0.75) and v0 = (0.0625, 0.0625). The new centre has cost 0.0625 + 0 + 0.1125 = 0.175, far below the first batch’s best. Then repeat the whole thing, before acting: sample, score, keep elites, refit, sample again. The device runs it for you.
The shading is the cost of each pair of pushes (darker is cheaper). Dots are candidates, filled ones are elites. The ellipse is the bell curve for the next round. The pink star is the true best. Press Next round several times.
The cost and the four-candidate batch are the lecture’s (slides 24 to 27; candidates (0,0), (1,1), (1,−1), (0.5,−0.5)). Later batches are seeded random draws, clipped to the action bounds [−1, 1], started from centre (0, 0) with spread 0.5, with a spread floor of 0.0025 and full refit rate. The true best is solved exactly from the quadratic.
Try small elites. With Ne = 1 the spread collapses to the floor after one round, and the search can freeze on a lucky but poor point. With Ne near N the elites are not selective, and the centre barely moves. The lecture marks this honestly: small batches need not improve the mean every round. Its three rounds show mean costs 0.175, 0.19773 and 0.17575, and the second is worse than the first.
Slide 28 gives it in code. Read the shapes: A is (N, H, da), one plan per row of the batch; J is (N,); the elite set is Ne indices. Two safeguards are optional but wise. A refit rate α blends the new fit with the old one, so a noisy batch cannot throw the centre far. A variance floor vmin keeps the bell curve from shrinking to a point.
pythonmu, v = init() # (H, d_a) each for k in range(I): # I search rounds A = clip(sample_gaussian(mu, v, N), action_bounds) # (N, H, d_a) J = rollout_and_score(A, current_information, goal) # (N,) one cost per plan E = np.argsort(J)[:N_e] # the N_e cheapest mu_hat = A[E].mean(axis=0) v_hat = ((A[E] - mu_hat)**2).mean(axis=0) mu = (1 - alpha) * mu + alpha * mu_hat v = np.maximum(v_min, (1 - alpha) * v + alpha * v_hat) return clip(mu, action_bounds) # the plan; MPC runs mu[0] then replans
Count the work. Each search round rolls out N plans, each H steps long, so a decision costs about I × N rollouts of H model steps. With the lecture’s cost model that is where the time goes. And remember what CEM does not do. It does not change the world model. It does not even look at how the model works inside. That is its strength: any model that can roll out and score is a model CEM can use, including a non-differentiable one.
Chapter 6
Wiggle the plan you already have, let cheap futures count for more, and nudge the plan toward the average of the good wiggles
A sailor holds a course. Every few minutes she tries a small change: a degree left, two degrees right, a touch more sail. She watches which tries went faster. Then she does not jump to the single best try. She blends all of them, giving the faster ones a bigger say. Her course improves a little with each round, and she never drew a map of the wind.
That is MPPI, model predictive path integral control. It differs from CEM in one respect that matters. CEM starts each round from a bell curve and keeps only the elites. MPPI starts from a current plan, Ā, and uses every sample, each with a weight. The plan is a list of H pushes, such as the orange line on the slide. The world model is fixed. The only moving part is the plan.
Make M copies of the plan and add noise to each: Ai = Ā + εi, with εi ~ N(0, Σa). The noise εi has one number for every push. The matrix Σa sets how wide the wiggles are. Roll each copy through the model and compute its cost Si. Small wiggles give a local search that is careful. Big wiggles explore, but most of them land on bad plans.
Now each copy gets a share of influence. Costs are not shares, because a cost can be any size and a share must be between 0 and 1. The lecture uses a softmax of the negative cost, which turns any list of scores into shares that sum to 1.
Check the slide’s example by hand. Three samples have corrected costs 0, 1 and 3, and τ = 1. The raw scores are e0 = 1, e−1 = 0.368 and e−3 = 0.050. Their total is 1.418. So the weights are 1/1.418 = 0.705, 0.368/1.418 = 0.259 and 0.050/1.418 = 0.035. A sample that costs 3 more than the best gets about 5% of the best sample’s say.
The temperature τ is the dial that matters. A small τ makes the vote winner-take-all: nearly all weight on the best sample, which is almost the same as picking it. A large τ flattens the vote: every sample counts about the same, and the plan barely improves.
The update is a weighted average of the wiggles: ΔA = ∑i wi εi, and the new plan is Ānew = Ā + ΔA. Wiggles that went well pull the plan their way. Wiggles that went badly barely count. Repeat around the new plan if time allows. Then execute the first action, observe, and shift the remaining plan forward one step to start the next decision. That shift is a warm start: the old plan is a good guess for the new one.
The thick orange line is the plan: six pushes for the cart. Press the main button to sample wiggles (their darkness shows their weight), then press it again to update the plan. Change τ and see the weights move.
Toy cart, illustrative numbers: H = 6 pushes from rest, the hero’s cost (target position 1, speed 0, effort weight 0.1). The device uses the plain rollout cost in units of 0.02 above the best sample as S; the full MPPI formulation adds control and sampling corrections that this toy leaves out. The initial plan is a steady push of 0.05. Samples are seeded, so the first batch is the same on every visit.
Run it for four or five rounds at τ = 1. The plan swings from a steady push to a push-then-brake shape, and the cost drops. Now set τ very high and run again: the plan crawls. Set τ very low: it jumps toward whichever single wiggle was luckiest, and a noisy batch can send it the wrong way. The read-out gives the effective number of samples, 1 / ∑ wi². It is 1 when one sample takes all the weight and M when all share equally. It tells you how many wiggles actually voted.
Both are sampling searches that never need a gradient, and both keep the model fixed. CEM fits a new bell curve to a hard-selected elite set, so the plan is a mean of the elites and the spread shrinks. MPPI holds one plan and averages weighted wiggles, so every sample counts a little. MPPI’s noise width stays whatever you set. That makes it simple to warm-start and a good fit to continuous control, which is why TD-MPC’s published planner is in the MPPI family (Chapter 9).
Chapter 7
When the model is differentiable, ask it which way each push should change, and roll the whole plan downhill
Back in the dark field, the dart thrower has a second option. She stands still and feels the ground with her foot. The ground tilts to the left, so she steps left. She never throws a dart. A tilt is more informative than a thousand darts, if you can feel it. CEM and MPPI sample costs. Gradient planning reads the slope.
The gradient of the cost J with respect to the plan A is a list with one number per push. Entry j says: if I raise push j a little, how much does the cost rise? Step each push against its entry and the cost falls. The recipe is the lecture’s one line: A ← A − η ∇A J(A; θ), with the model parameters θ fixed. The number η is the step size.
This only works when the rollout and the cost are differentiable: smooth enough that a small change in a push causes a small, computable change in the cost. Neural-network world models usually are. Then a deep-learning library does the hard part.
Why does an early push matter to the final cost? Because it changes every state after it. Take a rollout sj+1 = fθ(sj, aj) and a cost with a final-state part and an effort part, J(A) = ℓ(sH, g) + ∑ r(aj). The slide gives the gradient for push j.
On the cart it is easy to read. Push j adds 1 to the final speed, and adds H − 1 − j to the final position (it speeds the cart for the remaining steps). So ∂J/∂aj = 2(pH − 1)(H − 1 − j) + 2vH + 0.2aj. Early pushes move the position the most. A library computes this for any model by running the rollout forward, then backpropagating from the cost to the actions, with the model’s weights marked as fixed.
Real actuators have limits: a motor cannot push beyond its range. After each gradient step, project the plan back into the valid set: A(k+1) = Π(A(k) − ηk ∇J). For simple box bounds, projection just clips each push to [amin, amax]. The slide adds a warning: a fixed step size does not guarantee that the cost goes down. Try it.
Top: the eight pushes of the plan, with the bound shown as dashed lines. Bottom: the cost after each step, on a log scale. Press Step or Run 20. Then push the step size past 100% of the stability limit.
Toy cart, illustrative numbers: eight pushes from rest, the hero’s cost (target 1, speed 0, effort weight 0.1), exact gradients. The step size is shown as a share of 2 / L, where L = 292 is the largest curvature of this cost; a plain gradient step on a quadratic is stable only below 100% of that. The plan starts at all zeros.
At 50% the cost drops from 1 to 0.03 in 10 steps. At 110% with no bound, the plan overshoots the bottom of the bowl by more each step and the cost grows to the hundreds. Turn the bound on, and projection catches the blow-up: the plan stays inside the box and the cost settles, although it jitters. The step size is a real design choice, and real planners often use an adaptive optimiser such as Adam.
In code, the whole inner loop is short. The model is frozen: its parameters do not require gradients, only the plan does.
pythonA = warm_start.clone().requires_grad_() # (H, d_a): shifted previous plan for k in range(n_iters): S = rollout(s0, A, f) # f has requires_grad=False on its weights J = cost(S, A, goal) # scalar J.backward() # A.grad: (H, d_a), through every model step with torch.no_grad(): A -= eta * A.grad A.clamp_(a_min, a_max) # the projection onto the box A.grad = None
If the model is stochastic, the rollout contains random draws, and a draw seems to block the gradient. The fix comes from Lecture 7: the reparameterization trick. Write the step as sj+1 = fθ(sj, aj, εj), where the noise εj comes from a fixed distribution that does not depend on the actions. Hold the noise draws fixed, differentiate through each sampled future, and average: ∇AE[J] ≈ (1/M) ∑m ∇AJ(Rollout(Ht, A, ε(m))). The slide warns that this must be checked along the whole path. Categorical sampling and mixture-component choices (as in IRIS and the MDN of World Models 2018) are not smooth, so they need extra treatment. A continuous latent alone does not guarantee a usable gradient.
The slide ends with three limits. Local search: flat regions and local minima can stall an update. Long rollouts: a gradient is a product of H Jacobians. If each shrinks the signal by 10%, then after 50 steps only 0.950 = 0.5% is left, and the early pushes see almost no signal. If each grows it by 10%, 1.150 is about 117 times too big. The first case is a vanishing gradient, and the second is an exploding one. Model error: the model’s own slope can point the wrong way, and the optimiser will follow it with great confidence. The responses on the slide are restarts, shorter horizons with feedback, and better model coverage.
Chapter 8
Score a plan by the average cost of its possible futures, never by the cost of the average future
You choose between two routes to the airport. Route A takes 30 minutes on average. Route B takes 35. A looks better. But A crosses a bridge that is sometimes stuck for an hour, and your flight leaves in 45 minutes. A’s average is better. A’s risk of missing the flight is worse. The average alone hid the problem.
The same trap waits in planning with a stochastic world model. A plan no longer has one outcome. It has a cloud of possible futures, and each future has its own cost. Slide 38 shows the smallest possible case. Plan A leads to two equally likely futures with costs 0 and 10. Plan B leads to two equally likely futures that both cost 4. Which plan is better?
Take the expected cost: each future’s cost times its probability, added up. Plan A: ½ × 0 + ½ × 10 = 5. Plan B: ½ × 4 + ½ × 4 = 4. Plan B wins, even though plan A has the single best outcome. If a bad outcome is expensive and you will only get one try, that is the right call. (If rare failures matter a great deal, the lecture adds, specify an acceptable failure probability as well as the average cost.)
Here is a shortcut that looks sensible. Predict the average future state, E[S], and compute the cost of that one state, J(E[S]). It needs a single rollout instead of many. The slide states plainly that this is wrong: E[J(S)] ≠ J(E[S]) in general.
Why? Think of a cost that squares the miss, like ours. A square is a bowl, and for a bowl the average of the heights is higher than the height at the average. Two futures at positions 0 and 2, each with probability ½, average to position 1. The cost of position 1 is (1 − 1)² = 0. But each future really costs (0 − 1)² = 1 and (2 − 1)² = 1, so the expected cost is 1. The average future sits on the target. No actual future does.
Each plan ends at a random position. Plan A aims straight at the target but is shaky. Plan B aims a little short but is steady. The yellow bowl is the cost of a miss. Drag A’s shakiness, then change how many futures you sample.
Toy numbers, illustrative: cost is the squared miss (position − 1)². Plan A ends at 1 with the spread you set; plan B ends at 0.8 with spread 0.1. The futures are seeded random draws; “Draw new futures” changes the seed. The true expected costs are spread² for A and 0.04 + 0.01 = 0.05 for B.
Set the spread to 0.8 and look at the read-out. The cost of A’s average future is 0, so the shortcut calls A perfect and picks it. The true expected cost of A is 0.64 (the spread squared), and the average over a few sampled futures lands near it, far worse than B’s 0.05. The shortcut chose the wrong plan. Now cut the sample count to 2 or 3 and redraw a few times. The sampled average becomes noisy and can pick wrongly in the other direction. That is the price of sampling: a small M is cheap and unreliable.
The slide ends with a sentence that is easy to skim and important to keep: sampling candidate actions and sampling the futures of one plan are different operations. They sit at different levels of the loop. Count what a decision costs in the stochastic case.
Nest them and the data is a block of shape (N, M, H, ds): N plans, M futures each, H steps, ds numbers per state. The cost of a plan is the mean over the M axis, giving a vector of N costs. That vector is what CEM sorts and MPPI weighs. The compute grows as the product, and Chapter 10 puts it in one formula.
Slide 39 sums up Chapters 5 to 7. All three optimise actions inside MPC, with the model parameters fixed. They differ in how they turn costs into a better plan, and in what they require from the model.
| Method | How it updates a plan | Needs action gradients? | Good when |
|---|---|---|---|
| CEM | Refit the proposal to the elites | No | The model is a black box; the cost has jumps |
| MPPI | Average the cost-weighted perturbations | No | Continuous control with a warm start |
| Gradient-based | Step along the negative cost gradient | Yes | The rollout is smooth and the horizon short |
The last column is our reading, not the slide’s. It follows from what each method needs: gradients need smoothness, and samples only need a score. A smooth, short rollout gives gradients cheaply. A rollout with discrete tokens or contact has no useful slope, so sampling takes over.
Chapter 9
Show the robot a photo of the goal, and plan in feature space with a model that never saw that goal
You want a robot arm to put a block where it sits in a photograph. You cannot write “the block is at x = 0.31, y = 0.12”, because you do not have the coordinates. You have a picture. So how does a cost like (p − 1)² become a cost made from a picture?
The lecture’s answer is short. Use the model’s own encoder. Encode the goal photo og into the same feature space as the robot’s camera frames, giving zg = E(og). Let the world model imagine where a candidate plan leaves the robot, as features ẑt+H(A). The cost is the distance between those two points in feature space.
The slide ends with the key sentence: the goal enters the evaluator. It need not enter the dynamics model. The model was trained to predict what happens under actions, with no goal in mind. Give it a new goal photo and the same frozen model plans toward it. Nothing is retrained.
Try it. A toy robot lives in a two-number feature space. A fixed, frozen model predicts its next point from its action. Click anywhere to place the goal, or use a preset, then press Plan. The plan is found by CEM from Chapter 5, over H = 5 steps. Press Act one step to run MPC, with K = 1.
Blue is the robot’s real feature point. Purple lines are imagined plans (grey: candidates, purple: the chosen plan). Click the plane to move the pink goal. Then press Plan, or Act one step to run the loop.
Toy, illustrative numbers: a two-number feature space, a frozen step z′ = z + 0.3a plus a small fixed bend, actions in [−1, 1], H = 5, CEM with 40 candidates, 6 elites and 3 rounds (seeded). The real world equals the model here, so only the goal changes. The camera-angle switch adds a fixed offset of (0.5, −0.4) to the encoded goal: the lecture notes that viewpoint changes can affect the goal-feature distance.
Place a goal far away. The first plan cannot reach it in five steps, so its predicted end is only closer. After one action and a replan, it is closer again. This is MPC doing the work a single long plan cannot. Now switch on the camera-angle toggle. The robot goes to a point that matches the encoded photo perfectly, and that point is not where you meant. A cost made from features is only as good as the features. The distance is a task choice.
Slide 41 gives the checklist that lets one planner serve many models. Four boxes in a row: the current context (what the model needs to start), a latent rollout under candidate actions A, the predicted future features, and a task evaluator. Before planning you ask three questions. What must we initialise (which observations, memory or inferred state)? What does action search need (are candidate actions valid, and are futures sampled or deterministic)? How do evaluation and feedback work (what score ranks plans, and when does the next real observation arrive)? The advice: initialise with enough history to predict consequences, and clone that context for every candidate.
The remaining slides walk the family from Chapter 1 through this checklist.
Notice the pattern. Each model changes what you initialise and what a candidate carries. None of them changes the loop. That is why the lecture draws one diagram at the end and reuses it.
Chapter 10
Tell five failures apart: a goal that is not the one you meant, a blind model, a hard search, a short horizon and a missed deadline
A planner can fail without any bug in the code. Every line may run as written, and the robot still does the wrong thing. The lecture closes with six slides on why, and they give one habit: when a closed-loop system fails, ask which part failed. There are five parts, and each has a different cure. We take them in the lecture’s order.
You want a robot to carry a full cup to a table. Writing the cost is harder than it sounds. Do you penalise a spill? How much? Is any spot on the table fine? Slide 50 offers a shortcut: show a successful run and say “this is what success looks like.” A goal photo of the cup at the target does that, as Chapter 9 showed.
But one example does not say everything. Slide 51 lists three behaviours against two readings of the demonstration. “Carry the full cup carefully” satisfies both “match the final goal image” and “arrive without spilling.” “Spill, then arrive upright” may match the goal image and fails the no-spill reading. “Arrive safely at another acceptable placement” may fail the image match and pass the no-spill reading. A single photo cannot tell the readings apart. More demonstrations, comparisons, constraints or feedback can add the missing information, and if rare failures matter, give an acceptable failure probability as well as an average cost.
The planner chooses whatever its model says is cheapest. If the model is wrong in a way that makes a bad action look good, the planner will find that action, and love it. Slide 52 shows a box pushed to a goal behind a wall. The model has never seen the wall stop anything, so it predicts the box slides through and lands on the goal with a distance near zero. In reality the box hits the wall and stops far short. The cost rule is the same in both pictures. The model got the motion wrong.
The planner picks the push strength that the model says ends nearest the goal. The purple ghost is where the model thinks the box ends. The blue box is where it really ends. Slide the model’s belief from “passes through” to “stops at the wall”.
Toy, illustrative numbers: the box starts at 0, the wall stands at 0.48, the goal is at 1, and a push of strength u would carry the box 1.2u in free space. The real box stops at the wall. The model lets a share of the carry-on pass through; the planner picks u from 0 to 1 that minimises its predicted miss, with a tiny effort penalty of 0.01 u.
At 100% the model predicts a miss of 0.00 and the real miss is 0.52. This is the failure of slide 52. The symptom: the plan looks perfect in imagination and fails in the world. The measurement: compare the predicted end with the real end for the same action. The gap is the model error. The fix on the slide: collect interaction data around the obstacle, and restrict actions that the data does not support. A planner must not be asked to use a model outside the region where the model was trained. The same slide cites Chua et al. (PETS) for this point.
Now suppose the model is right. Predicting the outcome of one plan is cheap. Finding a good plan among all plans is a search over a huge, uneven space, and slide 53 shows why. A box stands between the start and the goal. A useful first action may move you away from the goal, around the obstacle. A planner that rewards getting closer will take the nearest dead end. Restarts, structured proposals and hierarchical search help. None removes the difficulty in general.
Slide 54 is the same obstacle with a different cause. Even a perfect search cannot choose a detour whose benefit arrives after the end of its horizon. Try the numbers. A route straight at the goal reaches a dead end 0.4 away. A detour first moves away from the goal, then curves round and arrives. The planner scores each route by its distance to the goal at the end of the horizon H.
The orange line is the straight route into a dead end. The purple line is the detour. The vertical teal line is the horizon H. The planner picks the route with the lower line at H.
Illustrative numbers drawn for this lesson, not from the lecture: the straight route closes 0.15 per step from distance 1.0 and stalls at 0.4 after step 4; the detour opens up to 1.48 by step 4, then closes 0.2 per step and arrives at step 12.
With H below 10 the straight route looks closer, so the planner takes the dead end. At H = 10 the detour’s distance (0.28) finally beats the dead end (0.40). Remedies from the slide: a longer H, subgoals (intermediate targets), or temporal abstraction (planning in bigger steps). Learned terminal values, which Lecture 10 covers, let a short horizon see far by summarising the future in one number.
Planning costs compute, and the controller has a deadline. Slide 55 gives the count: the rollout work is about I × N × M × H, the product of search rounds, candidate plans, futures per plan and model steps. The slide adds a careful note: this counts model transitions, not wall-clock time, and autoregressive token generation adds work within each step. Drag the four factors and see what a decision costs.
Each coloured block is one factor, drawn as a length on a log scale. Multiplying the factors adds the lengths. Raise one factor and see which one moves the total most.
The product formula is the lecture’s (slide 55). The default values are illustrative, not from any paper. One “model step” is one call of the transition function for one candidate and one future; real latency also depends on batching and hardware.
At the defaults the work is 3 × 100 × 5 × 20 = 30,000 model steps per decision. If you must act every 50 ms, that is a hard limit. The slide’s ways out: warm starts, useful proposals, batching, shorter rollouts, or larger action chunks. A larger K saves planning calls but delays feedback (Chapter 4).
Slide 56 puts it all in one table. When the closed loop fails, ask the question for the part you suspect, and measure real task outcomes under stated compute and interaction budgets.
| If the failure is… | Ask | In this lesson |
|---|---|---|
| Objective | Does the score actually represent success? | The cup, the full-speed arrival |
| Information | Can the available observations tell the relevant states apart? | A goal photo from another angle |
| Model | Do predicted consequences match actual outcomes? | The wall |
| Search | Can the optimiser find a good plan within this model? | The dead end, the short horizon |
| Execution | Can we plan, act and observe quickly enough? | I × N × M × H, and the interval K |
Chapter 11
Review the loop, the numbers and the code, then look ahead to learning the behaviour instead of searching for it
You can now rebuild Lecture 9 from memory. A world model predicts. A cost says what you want. A search tries action sequences against the model and keeps the cheapest. A feedback loop executes the first action, looks, and plans again. The lecture’s closing diagram says it in five words: predict, evaluate, optimise actions, execute, observe. Then it asks one question: what changes during planning, and what stays fixed? The model parameters stay fixed. The actions change, and the plan is rebuilt after every real observation.
| What | Number | Why it matters |
|---|---|---|
| Arrive-and-stop cost | J = (pH − 1)² + vH² + 0.1 ∑ a² | Three failures, three terms |
| Compounding error, gain 1.08 | rollout error 5.823 vs one-step 0.160 at step 12 | Look at reality often |
| LGSSM prediction | Pj+1 = APjA⊤ + Q; covariance independent of the plan | Planning needs only the mean path |
| Horizon and interval | H = 5, K = 1 or 3 | Look-ahead versus how often you check |
| CEM, four candidates | costs 1.00, 4.20, 0.20, 0.30; new centre (0.75, −0.75), spread 0.0625 | Elites refit the proposal |
| CEM round means | 0.175, 0.19773, 0.17575 | A small batch need not improve each round |
| MPPI weights, costs 0, 1, 3, τ = 1 | 0.705, 0.259, 0.035 | Cheap futures vote more |
| Two plans, random futures | expected cost 5 vs 4 | E[J(S)] is not J(E[S]) |
| LeWorldModel | H = K = 5 model steps × frame skip 5 = 25 environment steps | Larger K delays correction |
| V-JEPA 2-AC | L1 goal distance, CEM, K = 1 | Replan after every action |
| Planning work | ∝ I × N × M × H | A decision has a deadline |
Here is MPC with a CEM search and a goal photo, as one function. Read it with the chapters in mind: the encoder is Chapter 9, the rollout is Chapter 1, the cost is Chapter 2, the search is Chapter 5, and the replanning loop is Chapter 4.
pythondef act(model, enc, history, goal_img, H=5, N=40, Ne=6, I=3): z0 = enc(history) # current information: (d_z,) zg = enc(goal_img) # the goal enters only here: (d_z,) mu, v = np.zeros((H, d_a)), np.full((H, d_a), 0.5) for k in range(I): # search rounds; real time has not advanced A = clip(mu + np.sqrt(v) * rng.normal(size=(N, H, d_a)), lo, hi) # (N, H, d_a) Z = model.rollout(z0, A) # frozen model: (N, H + 1, d_z); clone z0 per candidate J = ((Z[:, -1] - zg) ** 2).sum(-1) # squared L2 goal distance: (N,) E = np.argsort(J)[:Ne] # elites mu = A[E].mean(0) v = np.maximum(1e-3, A[E].var(0)) # variance floor return mu[0] # MPC: send the first action, then observe and call again
The last lecture built the models and asked them to predict given actions. This one used the predictions to choose actions. The next lecture turns to learned behaviour. Planning searches from scratch at every step, and every step costs compute. Slide 58 asks the question that opens Lecture 10: can experience reduce the need to search from scratch? The answer is to learn a policy π(a | s) that proposes actions directly, a value V(s) that scores a state, and an action value Q(s, a) that scores a state and an action together. Dreamer and TD-MPC connect imagined futures to learning how to act. You have already met the seeds: TD-MPC’s score ends with a learned value, and its search takes proposals from a learned policy.
The course site lists the readings and assignments for every lecture. The papers behind this one are the ones named on the slides: PlaNet (Hafner et al.), Dreamer, TD-MPC (Hansen et al.), World Models (Ha and Schmidhuber), IRIS (Micheli et al.), V-JEPA 2 (Assran et al.), LeWorldModel (Maes et al.), the MPPI paper (Williams et al.), PETS (Chua et al.), and the MIT trajectory-optimisation notes.
Primary sources named on the slides: PlaNet, Dreamer, TD-MPC, World Models, IRIS, V-JEPA 2 and PETS.
Now press Teach and explain one planning decision out loud, from memory: what the cost says, how the search finds a plan, why only the first action runs, and what you would check if the real robot disagreed with the imagined one. Then go back to the cart at the top, drag the real thrust to 50% and to 130%, and explain why re-planning every step holds the target when planning once does not.