Planning and Control with World Models

A world model only predicts. To act, an agent tries many action plans in imagination, scores each against a goal, and runs the first step. Drag the model error and watch re-planning hold the goal.

Learn how a robot turns a model that predicts into a choice of what to do: the cost, the search, the feedback loop, and the ways it fails.

Pick Plan once, then slide the real thrust down to 70%. The cart stops short. Now pick Plan every step and watch it recover. Then we build, piece by piece, the cost, the loop, and three ways to search for actions: CEM, MPPI and gradients.

You need a feel for position and speed (speed is how fast position changes) and what a probability is. We build the rest from zero.

Based on Lecture 9 of Penn CIS 6280: World Models (Fall 2026), taught by Jiatao Gu. Lecture slides. Every idea is re-taught here in our own words and devices; it follows Lecture 8, Latent World Models.

Plan a cart to the target

70%

Toy cart, illustrative numbers. Position p and speed v update as p+v and v plus thrust, for 12 steps; the target is position 1 with speed 0. The model assumes thrust works at 100%; the slider is how well it really works. The cost is the lecture's: (p12 − 1)² + v12² + 0.1 × the sum of squared thrusts. The plan is the exact least-squares answer of Chapter 3.

Chapter 0

From Prediction to Choice

See why a model that predicts well still cannot act until you add a goal and a search

A first note on logistics. Lecture 9 met on Thursday 24 September 2026, on Zoom, because the instructor was travelling. Assignment 1 was due that day, and office hours ran from 4:30 to 5:30 PM on campus. That is all the logistics there are. The rest of this page is the idea.

Picture a small cart on a rail. You want it to roll to a mark one metre away and stop exactly there. You cannot try it first, because the cart is expensive and the rail ends in a wall. But you do have something valuable: a world model, a program that takes the cart’s situation and a push, and says what the situation will be a moment later.

The last lecture was about building that program. We gave it a state (what the cart is doing now) and an action (a push), and it predicted the next state. Lecture 9 asks the next question. The model can tell you what happens if you push. It cannot tell you which push to make. How do you get from a model that predicts to a cart that arrives?

Three pushes, one prediction each

Start with the plainest answer. Write down a few candidate plans, ask the model to imagine each one, and keep the best. The lecture draws exactly this. From the same current information, three plans fan out: do nothing, keep accelerating, or accelerate and then brake. The model predicts two or more steps of each. Only one ends at the goal and stays there.

Notice what changed. Before, we compared a prediction with what really happened, to fix the model. Now the model stays fixed. We compare predictions with what we want, to fix the plan. The lecture says it in one line: Lecture 8 gave actions and predicted; Lecture 9 uses predictions to choose actions.

The device below is that idea in miniature. The cart starts at rest at position 0. The target is position 1, and the cart must be stopped there, so its speed must be 0. Pick one of the three plans, and the model draws the cart’s predicted path over six steps.

Imagine three plans

Each button is one plan of six pushes. The purple path is what the model predicts. The pink line is the target. Try all three and compare where each one ends.

Toy cart, illustrative numbers: position changes by the speed, speed changes by the push, six steps from rest. The cost is the lecture’s arrive-and-stop cost with effort weight 0.1 (Chapter 2 explains it). The pushes are chosen so the third plan lands exactly on the target.

The first plan leaves the cart at 0. The second reaches position 1, but the cart is still moving at speed 0.4, so it sails past the mark one step later. The third arrives and stops. A person sees that at a glance. A computer needs a number that says so, and that number is the task cost in the read-out. Lower is better. Chapter 2 builds it.

Two jobs, kept apart

Planning has two jobs, and it helps to keep them apart. The first is predict: given a start and a plan, what happens? That is the world model. The second is choose: which plan has the lowest cost? That is a search. The model never decides anything. It answers “what if” again and again, and a separate piece of code asks the questions.

This split gives the lecture its shape. First we say what the cost is. Then we plan in the one case where the math is exact, a model made of straight lines and bell curves. Then we meet three ways to search when the model is a neural network and the exact answer is gone: CEM, MPPI and gradients. We wrap all three in a feedback loop called model predictive control. Next we plug in the latent models of Lecture 8. Last we ask where it breaks.

The hero at the top already shows the loop. A cart plans from a model that thinks every push works at full strength. When the real cart is weaker, a plan made once lands short. A plan remade at every step does not, because each step starts from where the cart really is.

  1. The models we plan with: five latent models from Lecture 8, and why their errors pile up.
  2. A goal becomes a cost: one number that says how good an outcome is.
  3. Planning when the math is exact: straight-line dynamics and bell-curve noise.
  4. Model predictive control: plan, act once, look, plan again.
  5. Search by sampling: CEM: keep the best guesses and refit.
  6. Search by weighting: MPPI: let cheap futures vote on the plan.
  7. Search by slope: gradients: roll the plan downhill through the model.
  8. Futures are random: compare expected costs, not the cost of the average.
  9. Planning in a latent model: a goal image, a frozen model, five model families.
  10. What can go wrong: a vague goal, a wrong model, a hard search, a small budget.
  11. Field guide: the numbers, the code, and where the course goes next.
When a planner searches for good actions, what stays fixed and what changes?

Chapter 1

The Models We Plan With

Meet the five latent models from Lecture 8 in one place, and see why their small errors grow when you imagine far ahead

Before we plan, we need something to plan with. The lecture spends its first ten slides on a recap, and it is worth a slow look, because every planner later in the lesson plugs into one of these models. You can skip the details of how each one is trained. What matters is what each one offers to a planner: a way to start, a way to step forward, and a way to say how good the end is.

All five are latent models. A latent is a short list of numbers that stands for a picture. A camera frame has millions of pixels, but what matters (where the arm is, where the cup is) fits in a few hundred numbers. An encoder squeezes the frame into that list. The dynamics part then predicts the next list from the current list and an action. Planning in the small list is far cheaper than planning in pixels.

Five families, one table

Pick a family below. Each card gives the four things the lecture lists for it: the encoder, the state it keeps, how it predicts, and how it is trained. The first two families are deterministic: the same input always gives the same output. The next three are stochastic: they hold a probability, and a rollout draws a sample from it.

Compare the model families

Tap a family. Read how it starts a rollout, how it steps, and what you must carry along for every candidate plan.

Summarised from the lecture’s slides 3 to 10 (Hansen et al., TD-MPC; Maes et al., LeWorldModel; Ha and Schmidhuber, World Models, 2018; Hafner et al., PlaNet and Dreamer; Micheli et al., IRIS). “Carried state” is what a planner must copy for each candidate plan.

Two details matter later. First, TD-MPC keeps only the latent z, and its next latent is a function of the current one and the action: ẑt+1 = F(zt, at). The prediction is then fed back in, so the model can run for many steps without seeing a new picture. The lecture adds a warning: squared-L2 consistency alone permits collapse, where the encoder maps every frame to the same point and the loss is trivially zero. The full method also trains on reward and value, which keeps the latent useful.

Second, the stochastic families keep memory. A recurrent network (RNN) carries a hidden summary ht of everything so far. The RSSM, the model behind PlaNet and Dreamer, has two paths. To observe, it samples the state ut from a posterior that sees the history and the new frame. To imagine, it samples ut from a prior that sees only the history, then updates ht+1 = f(ht, ut, at). The lecture is firm on this point: during imagination the observation branch is not available. A planner lives entirely on the imagination path.

Why far-away predictions go wrong

Slide 11 shows the danger of imagining far. Think of a person who walks and counts steps, but whose stride estimate is 8% too long. After one step, the guess is nearly right. After twelve steps, the guess has drifted by a whole stride or more. Each step feeds a slightly wrong position into the next step, so the errors do not cancel. They pile up. This is compounding error.

The lecture makes it exact with a one-line world. The true system is xt+1 = xt + at. The model is slightly off: x̂t+1 = g x̂t + at, with gain g = 1.08. The actions are +1 for six steps, then −1 for six steps. Define the error et = x̂t − xt. Then et+1 = g et + (g − 1) xt: the old error is multiplied by g and a new error is added.

Now the two ways to use the model. One-step prediction hands the model the true state at every step, so it can only be wrong by one fresh slip. Free-running rollout hands it its own last guess, so slips accumulate. At step 12 the lecture reports a one-step error of 0.160 and a rollout error of 5.823. That is 36 times larger. Move the knobs and see it.

Watch the error compound

Blue is the true system. Purple is the model running free on its own guesses. Raise the gain error, then lower the number of steps between real observations and watch the purple line get pulled back.

1.08
never

The lecture’s constructed example: x0 = x̂0 = 1, g = 1.08, actions +1 for six steps then −1 for six. At g = 1.08 and no real looks the model ends at 6.823 against a true 1, a rollout error of 5.823; fed the true state every step the error is 0.160. Looking every N steps resets the model’s state to the true one at steps 0, N, 2N and so on.

The remedy list on the slide has three parts: multi-step consistency (train the model on its own multi-step rollouts, not only on single steps), a validated rollout horizon (do not trust the model past the step where its error is known to be small), and state updates when real observations arrive (the slider above). The third one is the seed of model predictive control in Chapter 4.

Generation costs time

Slide 12 adds a second price of imagination: time. A token model like IRIS predicts a frame as a run of K codes, one after another. For H frames that is about H × K serial steps. Each step must wait for the one before. The slide lists fixes and their tradeoffs. Fewer tokens or frame-level prediction give fewer steps, but compression may lose detail. Caching saves repeated work, but memory grows or history is forgotten. Distillation cuts iterative sampling steps, but needs more training. Hold this in mind. Chapter 10 returns to it as the planning budget.

A model has gain 1.08 where the real system has 1. Which setup keeps its error smallest at step 12?

Chapter 2

A Goal Becomes a Cost

Turn “arrive and stop” into one number a planner can minimise, and see the loop that uses it

You tell a friend: “Park the car at the mark.” They understand. A computer does not. It needs a score for every way the parking could end, so that it can say one ending is better than another. That score is the task cost, written J. Low means good. The planner’s whole job is to find the actions with the lowest J.

The lecture builds the score for the cart from Chapter 0: arrive at position 1 and stop. Think about what can go wrong. The cart can stop short. It can reach the mark at speed, and so shoot past. Or it can get there smoothly but waste a great deal of energy. Each failure is one term of the score.

J
The task cost of one whole plan. One number. Lower is better.
(pH − 1)²
Position error. pH is the position after H steps; 1 is the target. Squaring makes a miss of 0.5 cost four times a miss of 0.25, and removes the sign.
vH²
Remaining motion. The speed at the end should be 0, so any speed costs, left or right.
0.1 ∑ aj²
Action effort. Add up the squared pushes. The 0.1 is a weight: it says how much one unit of effort matters against one unit of miss.

Play with the three terms. The device starts at the lecture’s three snapshots: no action, at the target but still moving, and at the target and stopped. Then drag the sliders to build your own ending.

Build a cost from its terms

The bar is J, cut into its three terms. Pick a snapshot, or drag the sliders for the final position, the final speed and the total effort.

1.00
0.00
0.07

The formula is the lecture’s (slide 14, target position 1, effort weight 0.1). The effort values in the snapshots are illustrative: 0 for no action, 0.2 for the fast arrival and 0.07 for the smooth one.

Look at the middle snapshot. The cart is exactly on the mark, so the position term is zero. But it is moving at speed 2, and that single term costs 4. Without the speed term, a planner would call “touch the mark at full speed” a perfect plan. A cost does not say what you meant. It says what you wrote. Chapter 10 comes back to this.

Two different questions

It is easy to mix up training the model with planning. Slide 15 puts them side by side, and the contrast is the key idea of the lecture.

Learn to predictPlan for a task
What is minimisedThe loss L between the model’s prediction fθ(st, at) and the observed next stateThe cost J of the model’s imagined rollout of a candidate plan, measured against the goal g
Compared withWhat actually happenedWhat we want to happen
What changesThe model parameters θThe candidate actions A, with θ held fixed

The slide ends with a line worth remembering: an accurate prediction is useful only after we decide which outcomes to prefer. The model tells you the weather. The cost tells you whether you want an umbrella.

The loop around it

Slide 16 draws the planner as a box, and the data flow is what makes it a real system. Inside the box, three parts talk. Action search proposes candidate actions A, a list of H pushes. The world model turns each into imagined consequences. The task evaluator turns the consequences and the goal into one cost, J(A). The result goes back to the search, which proposes better actions. All of this is internal and happens before the cart moves. The model parameters stay fixed throughout.

Action search
proposes candidate actions A, shape (H, d_a): H steps, d_a numbers per action
↓
World model, fixed
rolls A forward from the current information: states (H + 1, d_s)
↓
Task evaluator
goal and states in, one number out: J(A). Back to the search.
↓ the best plan, first action only
Environment
the real action happens; a new observation comes back and updates the current information

Only then does a real action leave the box. The environment responds, a new observation comes back, and the current information is updated. The dashed arrows of the slide are imagination. The solid arrows are the real world. Here is the evaluator as code, for the cart:

pythondef rollout(s0, A, f):          # s0: (2,) = [position, speed]   A: (H,) pushes
    S = [s0]
    for a in A:
        S.append(f(S[-1], a))   # the fixed world model, one step
    return np.array(S)           # (H + 1, 2)

def cost(S, A, goal=1.0, lam=0.1):
    pH, vH = S[-1]
    return (pH - goal)**2 + vH**2 + lam * (A**2).sum()   # one float: lower is better
A plan puts the cart exactly on the target but moving at speed 2. With the lecture’s cost, why is its cost still large?

Chapter 3

Planning When the Math Is Exact

Plan inside a world made of straight lines and bell curves, where the best action sequence has a closed-form answer

Before we plan with a neural network, the lecture plans in the one world where nothing is hard: the linear-Gaussian state-space model (LGSSM) of Lecture 4. If you built a Kalman filter, you know it. The point of doing this first is that every step has an exact answer. Later, with learned models, each step becomes an approximation. You will then know exactly what was lost.

Imagine tracking the cart again, but now the cart is jostled by small random bumps and you only see it through a noisy camera. The model has two lines. The state moves as sk+1 = A sk + B ak + wk: the next state is a straight-line function of the current state and the push, plus a bump w drawn from a bell curve with spread Q. The camera sees ok = C sk + νk, with its own noise R. In the cart, s is [position, speed], and A adds the speed to the position.

The planner does not hold the true state. It holds a belief, written p(st | Ht) = N(μt, Pt): a bell curve with centre μt (the best guess) and spread Pt (the doubt), given the history Ht of everything seen so far. The Kalman filter computes it. The planning variables are the next H pushes, A = (at, …, at+H−1). The model and the belief are given. The pushes are what we choose.

Predicting under a fixed plan

Fix a candidate plan. What will the cart do? Because no new camera frames arrive during imagination, the belief only flows forward, and the lecture gives two update lines, shown with their symbols below.

μj+1 = Aμj + Ba
The best guess after one more step: push the old guess through the dynamics and add the effect of the action. Starts at μ0 = μt.
Pj+1 = APjA⊤ + Q
The doubt after one more step: the old doubt stretched by the dynamics, plus fresh bumps Q. There is no action in this line. Starts at P0 = Pt.

Do one step by hand. Let the doubt start at P0 = diag(0.01, 0.01) and let the bumps be Q = diag(0, 0.01). With A = [[1, 1], [0, 1]], the stretched doubt AP0A⊤ is [[0.02, 0.01], [0.01, 0.01]]. Add Q and P1 = [[0.02, 0.01], [0.01, 0.02]]. The trace (the sum of the diagonal) is 0.04. Now push with a = 0.2 from μ0 = (0, 0): μ1 = (0, 0.2). The push moved the guess. It did not touch the doubt.

That last sentence is the surprise of slide 18. With additive noise, the covariance is the same for every candidate plan. Actions change where the cloud is centred. They do not change how wide it is.

The expected cost has a clean form

Now score a plan. A random future means a random cost, so the lecture takes the expected cost: the average over all the futures the belief allows. For a squared distance there is an exact identity: if s ~ N(μ, P), then E[‖s − g‖²] = ‖μ − g‖² + tr(P). The average squared miss is the squared miss of the centre, plus the width of the cloud.

Put the identity into the plan’s cost, J(A) = ∑j=1H [‖μj − g‖² + tr(Pj)] + λ ∑j=0H−1 ‖at+j‖². The trace terms do not depend on the actions, as we just saw. So when we compare plans, they cancel out of the comparison. The best plan is A* = argmin [ ∑ ‖μj(A) − g‖² + λ ∑ ‖a‖² ], where “argmin” means “the plan that gives the smallest value”. Planning under noise needs only the mean path.

One more fact finishes it. Because the dynamics are linear, each μj is a straight-line function of the pushes. A square of a straight-line function is a bowl, so J is a quadratic bowl in A. A bowl has one bottom, and a short piece of linear algebra finds it. Try it: scale the exact plan up and down, and watch the cost trace the bowl.

Predict a cloud, find the bowl

The purple line is the mean path for a plan of 6 pushes. The band is one standard deviation of position. Scale the plan, then raise the bumps. Watch what moves and what does not.

100%
0.010

Toy cart, illustrative numbers: A = [[1, 1], [0, 1]], B = [0, 1], H = 6, target (1, 0), P0 = diag(0.01, 0.01), bumps Q = diag(0, q), effort weight 0.1. The exact plan solves the normal equations of the sum-over-steps cost. The bowl on the right is J as you scale that plan.

Read the device. As you drag the bump size, the band widens and J rises, but the best scale stays at 100%. The covariance moved the whole bowl up without shifting its bottom. As you drag the scale, the band does not change at all, only the centre does.

The exact answer, for the curious, is the least-squares solution. Stack the predicted means into one vector m = mfree + G a, where G collects how each push changes each later state. Then minimise ‖m − g‖² + λ‖a‖². The answer is a* = (G⊤G + λI)−1 G⊤(g − mfree). That is one matrix solve. The hero used this same formula.

Now see what the lecture is building toward. The recipe has four steps: initialise from the belief, predict the mean and covariance, evaluate the expected quadratic cost, and optimise a quadratic in the actions. Slide 22 will put this next to a learned model, and each of the four steps will change.

In the linear-Gaussian model with additive noise, what does a different candidate action sequence change about the predicted cloud of futures?

Chapter 4

Model Predictive Control

Plan a long way ahead, act on only the first step, look at what really happened, and plan again

You drive to a friend’s house you have never visited. You look at the map and plan the whole route: left, right, right, left. Then you drive. At the first turn a road is closed. You do not follow the old plan into the barrier. You look up, find where you are now, and plan again from there. You are not a worse planner for it. You are a better driver.

That habit has a name. Model predictive control (MPC) is a loop of three moves. Plan: find the best action sequence for the next H steps. Act: carry out only the first action of it. Look: take the real new observation and update what you know. Then the loop starts over from the new situation. The lecture writes it in two lines.

A*t = argmin J
At time t, the whole H-step plan with the lowest cost. J depends on the history Ht (what we know now) and the goal g.
at = [A*t]0
Keep only entry number 0, the first push. The other H − 1 pushes are thrown away, for now.
p(st+1 | Ht+1)
After the real observation arrives, the belief is updated by the filter. The next plan starts from this new belief. This is the feedback.

Why throw away a good plan? Because it was made with a model, and a model is a guess. Chapter 1 showed that predictions drift. The first action of a plan is the one whose prediction is most reliable, because it is the nearest. The last action is the one that depends on the most guesses. MPC trusts the near, checks it against the world, and re-aims. The hero already shows the gain: a plan made once stopped short, and a plan remade at every step did not.

Two dials: how far to look, how long to wait

Slide 23 names two numbers that are easy to confuse. The prediction horizon H is how far ahead each plan looks: how many model steps are imagined. The execution interval K is how many of the planned actions are carried out before the next look. The slide plans H = 5 steps in both pictures. In one, K = 1: act once, observe, replan. In the other, K = 3: act three times blind, then observe and replan.

Think of H as headlights and K as how long you drive between glances at the map. Long headlights let you see a distant turn. Frequent glances let you catch a mistake early. They are separate dials, and they cost different things. A large H makes every plan more expensive to compute. A small K makes you plan more often. A large K is cheap, and it leaves you blind for longer.

Try it. The cart below has a real thrust at 80% of what the model believes, and a gust adds speed at step 8. The planner replans from the real state every K steps with a horizon of H. Set K to 1, then to 5, and watch how long the gust bends the path.

Set the horizon and the interval

Teal ticks mark each replan. The blue path is the real cart. Change H and K, and switch the gust on and off.

6

Toy cart, illustrative numbers. The model believes thrust works at 100%; the real cart gets 80%. The gust adds 0.08 to the speed at step 8. Each plan is the exact least-squares plan for the sum-over-steps cost of Chapter 3, started from the real state, with effort weight 0.1. 20 steps in all. “Worst miss” is the largest distance from the target between step 10 and step 20.

Results on this toy: with K = 1 the cart is back within 0.06 of the target after the gust. With K = 5 the worst miss is about 0.21. The cart planned in open loop for five steps and had no way to know it had been pushed. In both cases the plan itself was the same kind of calculation. Only the feedback differed.

What stays the same, what changes

The slide titled “Planning with a learned latent world model” gives a four-row table. We keep the loop and the objective. We replace the calculations inside it. Here it is with a plain gloss on each cell.

StepLinear-Gaussian + quadratic costLearned latent world model
InitialiseThe current Gaussian beliefAn inferred latent state, or the retained history
PredictExact mean and covariance recursionA learned rollout; sample futures if needed
EvaluateExact expected quadratic costA task cost on the predicted outcomes
OptimiseA quadratic in the actions: one solveGenerally a nonlinear function of the actions: a search

The new target is A*t ≈ argmin Epθ(S | Ht, A)[Cost(S, A; g)]. Read it as: among action sequences, pick the one whose imagined outcomes, averaged over the model’s own randomness, cost least. The symbol “≈” is honest. A search will not find the exact minimum. The next three chapters are three different searches.

An MPC controller plans 5 steps ahead and executes 3 of them before looking again. What are H and K, and what is the cost of raising K?

Chapter 5

Search by Sampling: CEM

Throw a cloud of guessed plans at the model, keep the best few, and aim the next cloud where they landed

You are blindfolded and want to find the highest point of a hilly field. You throw twenty darts at random. You walk to where they landed and pick up the five that landed highest. Then you throw twenty more, this time near the middle of those five. After three rounds, your darts bunch up on the hilltop. You never needed a map of the slope. You only needed to compare heights.

That is the cross-entropy method (CEM), the search PlaNet uses. The field is the cost J(A) of an action sequence. A dart is a candidate plan. The height of a dart is the cost your world model predicts for it. CEM never asks which way is downhill. It only needs the model to roll out a plan and return a number.

One round, with every number

The lecture keeps the plan short so we can check by hand: H = 2 pushes, A = (a0, a1), on the cart from the hero, starting at rest. The cart ends at position a0 with speed a0 + a1. So the cost is J = (a0 − 1)² + (a0 + a1)² + 0.1(a0² + a1²). CEM keeps a bell curve over plans: qk(A) = N(μ(k), diag v(k)), a centre and a spread for each push. The index k counts search rounds. Real time has not moved.

Step one is sample: draw N candidates from the bell curve. The slide supplies four. Step two is roll out and score: send each through the model and take its cost. Check the table.

Candidate (a0, a1)End positionEnd speedCost JElite?
(0, 0)001 + 0 + 0 = 1.00no
(1, 1)120 + 4 + 0.2 = 4.20no
(1, −1)100 + 0 + 0.2 = 0.20yes
(0.5, −0.5)0.500.25 + 0 + 0.1 = 0.30yes

Step three is select the elites: the Ne candidates with the lowest cost. Here Ne = 2, so (1, −1) and (0.5, −0.5) win. Step four is refit: fit a new bell curve to the elites. Their average is the new centre, and their spread is the new spread.

μk
The new centre: the plain average of the elite plans, push by push. For the table: ((1 + 0.5)/2, (−1 − 0.5)/2) = (0.75, −0.75).
vk
The new spread: the average squared distance of each elite from the new centre, push by push. The symbol ⊗2 means square each entry. For the table: (0.25² + 0.25²)/2 = 0.0625 in each push.

Those are the slide’s own numbers: μ0 = (0.75, −0.75) and v0 = (0.0625, 0.0625). The new centre has cost 0.0625 + 0 + 0.1125 = 0.175, far below the first batch’s best. Then repeat the whole thing, before acting: sample, score, keep elites, refit, sample again. The device runs it for you.

Run CEM on two pushes

The shading is the cost of each pair of pushes (darker is cheaper). Dots are candidates, filled ones are elites. The ellipse is the bell curve for the next round. The pink star is the true best. Press Next round several times.

16
4

The cost and the four-candidate batch are the lecture’s (slides 24 to 27; candidates (0,0), (1,1), (1,−1), (0.5,−0.5)). Later batches are seeded random draws, clipped to the action bounds [−1, 1], started from centre (0, 0) with spread 0.5, with a spread floor of 0.0025 and full refit rate. The true best is solved exactly from the quadratic.

Try small elites. With Ne = 1 the spread collapses to the floor after one round, and the search can freeze on a lucky but poor point. With Ne near N the elites are not selective, and the centre barely moves. The lecture marks this honestly: small batches need not improve the mean every round. Its three rounds show mean costs 0.175, 0.19773 and 0.17575, and the second is worse than the first.

The whole algorithm

Slide 28 gives it in code. Read the shapes: A is (N, H, da), one plan per row of the batch; J is (N,); the elite set is Ne indices. Two safeguards are optional but wise. A refit rate α blends the new fit with the old one, so a noisy batch cannot throw the centre far. A variance floor vmin keeps the bell curve from shrinking to a point.

pythonmu, v = init()                                            # (H, d_a) each
for k in range(I):                                         # I search rounds
    A = clip(sample_gaussian(mu, v, N), action_bounds)    # (N, H, d_a)
    J = rollout_and_score(A, current_information, goal)  # (N,) one cost per plan
    E = np.argsort(J)[:N_e]                              # the N_e cheapest
    mu_hat = A[E].mean(axis=0)
    v_hat  = ((A[E] - mu_hat)**2).mean(axis=0)
    mu = (1 - alpha) * mu + alpha * mu_hat
    v  = np.maximum(v_min, (1 - alpha) * v + alpha * v_hat)
return clip(mu, action_bounds)                              # the plan; MPC runs mu[0] then replans

Count the work. Each search round rolls out N plans, each H steps long, so a decision costs about I × N rollouts of H model steps. With the lecture’s cost model that is where the time goes. And remember what CEM does not do. It does not change the world model. It does not even look at how the model works inside. That is its strength: any model that can roll out and score is a model CEM can use, including a non-differentiable one.

In one CEM round, 100 candidate plans are scored and the 10 cheapest are kept. What is refit, and from what?

Chapter 6

Search by Weighting: MPPI

Wiggle the plan you already have, let cheap futures count for more, and nudge the plan toward the average of the good wiggles

A sailor holds a course. Every few minutes she tries a small change: a degree left, two degrees right, a touch more sail. She watches which tries went faster. Then she does not jump to the single best try. She blends all of them, giving the faster ones a bigger say. Her course improves a little with each round, and she never drew a map of the wind.

That is MPPI, model predictive path integral control. It differs from CEM in one respect that matters. CEM starts each round from a bell curve and keeps only the elites. MPPI starts from a current plan, Ā, and uses every sample, each with a weight. The plan is a list of H pushes, such as the orange line on the slide. The world model is fixed. The only moving part is the plan.

Step 1: perturb

Make M copies of the plan and add noise to each: Ai = Ā + εi, with εi ~ N(0, Σa). The noise εi has one number for every push. The matrix Σa sets how wide the wiggles are. Roll each copy through the model and compute its cost Si. Small wiggles give a local search that is careful. Big wiggles explore, but most of them land on bad plans.

Step 2: turn costs into weights

Now each copy gets a share of influence. Costs are not shares, because a cost can be any size and a share must be between 0 and 1. The lecture uses a softmax of the negative cost, which turns any list of scores into shares that sum to 1.

wi
The normalised influence of sample i on the update. All weights add up to 1.
exp[−(Si − β)/τ]
A raw score that is 1 for the best sample and falls toward 0 as the cost rises. The temperature τ sets how fast it falls.
∑ exp[…]
The total of all raw scores. Dividing by it makes the weights sum to 1.
β
The lowest cost in the batch. Subtracting it changes no weight, but it keeps the exponentials from underflowing to zero when costs are large.

Check the slide’s example by hand. Three samples have corrected costs 0, 1 and 3, and τ = 1. The raw scores are e0 = 1, e−1 = 0.368 and e−3 = 0.050. Their total is 1.418. So the weights are 1/1.418 = 0.705, 0.368/1.418 = 0.259 and 0.050/1.418 = 0.035. A sample that costs 3 more than the best gets about 5% of the best sample’s say.

The temperature τ is the dial that matters. A small τ makes the vote winner-take-all: nearly all weight on the best sample, which is almost the same as picking it. A large τ flattens the vote: every sample counts about the same, and the plan barely improves.

Step 3: update and act

The update is a weighted average of the wiggles: ΔA = ∑i wi εi, and the new plan is Ānew = Ā + ΔA. Wiggles that went well pull the plan their way. Wiggles that went badly barely count. Repeat around the new plan if time allows. Then execute the first action, observe, and shift the remaining plan forward one step to start the next decision. That shift is a warm start: the old plan is a good guess for the new one.

Perturb, weigh, update

The thick orange line is the plan: six pushes for the cart. Press the main button to sample wiggles (their darkness shows their weight), then press it again to update the plan. Change τ and see the weights move.

1.00
0.030

Toy cart, illustrative numbers: H = 6 pushes from rest, the hero’s cost (target position 1, speed 0, effort weight 0.1). The device uses the plain rollout cost in units of 0.02 above the best sample as S; the full MPPI formulation adds control and sampling corrections that this toy leaves out. The initial plan is a steady push of 0.05. Samples are seeded, so the first batch is the same on every visit.

Run it for four or five rounds at τ = 1. The plan swings from a steady push to a push-then-brake shape, and the cost drops. Now set τ very high and run again: the plan crawls. Set τ very low: it jumps toward whichever single wiggle was luckiest, and a noisy batch can send it the wrong way. The read-out gives the effective number of samples, 1 / ∑ wi². It is 1 when one sample takes all the weight and M when all share equally. It tells you how many wiggles actually voted.

CEM and MPPI side by side

Both are sampling searches that never need a gradient, and both keep the model fixed. CEM fits a new bell curve to a hard-selected elite set, so the plan is a mean of the elites and the spread shrinks. MPPI holds one plan and averages weighted wiggles, so every sample counts a little. MPPI’s noise width stays whatever you set. That makes it simple to warm-start and a good fit to continuous control, which is why TD-MPC’s published planner is in the MPPI family (Chapter 9).

In MPPI, what does lowering the temperature τ do to the update?

Chapter 7

Search by Slope: Gradient Planning

When the model is differentiable, ask it which way each push should change, and roll the whole plan downhill

Back in the dark field, the dart thrower has a second option. She stands still and feels the ground with her foot. The ground tilts to the left, so she steps left. She never throws a dart. A tilt is more informative than a thousand darts, if you can feel it. CEM and MPPI sample costs. Gradient planning reads the slope.

The gradient of the cost J with respect to the plan A is a list with one number per push. Entry j says: if I raise push j a little, how much does the cost rise? Step each push against its entry and the cost falls. The recipe is the lecture’s one line: A ← A − η ∇A J(A; θ), with the model parameters θ fixed. The number η is the step size.

This only works when the rollout and the cost are differentiable: smooth enough that a small change in a push causes a small, computable change in the cost. Neural-network world models usually are. Then a deep-learning library does the hard part.

The chain of influence

Why does an early push matter to the final cost? Because it changes every state after it. Take a rollout sj+1 = fθ(sj, aj) and a cost with a final-state part and an effort part, J(A) = ℓ(sH, g) + ∑ r(aj). The slide gives the gradient for push j.

∇ajJ
How the total cost changes when push j changes. One entry of the gradient.
∇r(aj)
The direct effort part: a bigger push costs more right away.
∂sH/∂aj
How the final state responds to push j. It is a chain of Jacobians (matrices of local slopes), one for each step between j and the end.
∇sHℓ
How the final cost changes when the final state moves. This is the “error signal” at the end of the rollout.
∂f / ∂s ···
The product of the model’s step slopes. This is the same chain rule that backpropagation uses to train a network, applied here to the actions instead of the weights.

On the cart it is easy to read. Push j adds 1 to the final speed, and adds H − 1 − j to the final position (it speeds the cart for the remaining steps). So ∂J/∂aj = 2(pH − 1)(H − 1 − j) + 2vH + 0.2aj. Early pushes move the position the most. A library computes this for any model by running the rollout forward, then backpropagating from the cost to the actions, with the model’s weights marked as fixed.

Walking downhill, within the bounds

Real actuators have limits: a motor cannot push beyond its range. After each gradient step, project the plan back into the valid set: A(k+1) = Π(A(k) − ηk ∇J). For simple box bounds, projection just clips each push to [amin, amax]. The slide adds a warning: a fixed step size does not guarantee that the cost goes down. Try it.

Roll the plan downhill

Top: the eight pushes of the plan, with the bound shown as dashed lines. Bottom: the cost after each step, on a log scale. Press Step or Run 20. Then push the step size past 100% of the stability limit.

50%

Toy cart, illustrative numbers: eight pushes from rest, the hero’s cost (target 1, speed 0, effort weight 0.1), exact gradients. The step size is shown as a share of 2 / L, where L = 292 is the largest curvature of this cost; a plain gradient step on a quadratic is stable only below 100% of that. The plan starts at all zeros.

At 50% the cost drops from 1 to 0.03 in 10 steps. At 110% with no bound, the plan overshoots the bottom of the bowl by more each step and the cost grows to the hundreds. Turn the bound on, and projection catches the blow-up: the plan stays inside the box and the cost settles, although it jitters. The step size is a real design choice, and real planners often use an adaptive optimiser such as Adam.

In code, the whole inner loop is short. The model is frozen: its parameters do not require gradients, only the plan does.

pythonA = warm_start.clone().requires_grad_()          # (H, d_a): shifted previous plan
for k in range(n_iters):
    S = rollout(s0, A, f)                              # f has requires_grad=False on its weights
    J = cost(S, A, goal)                              # scalar
    J.backward()                                       # A.grad: (H, d_a), through every model step
    with torch.no_grad():
        A -= eta * A.grad
        A.clamp_(a_min, a_max)                         # the projection onto the box
    A.grad = None

Random futures, and four limits

If the model is stochastic, the rollout contains random draws, and a draw seems to block the gradient. The fix comes from Lecture 7: the reparameterization trick. Write the step as sj+1 = fθ(sj, aj, εj), where the noise εj comes from a fixed distribution that does not depend on the actions. Hold the noise draws fixed, differentiate through each sampled future, and average: ∇AE[J] ≈ (1/M) ∑m ∇AJ(Rollout(Ht, A, ε(m))). The slide warns that this must be checked along the whole path. Categorical sampling and mixture-component choices (as in IRIS and the MDN of World Models 2018) are not smooth, so they need extra treatment. A continuous latent alone does not guarantee a usable gradient.

The slide ends with three limits. Local search: flat regions and local minima can stall an update. Long rollouts: a gradient is a product of H Jacobians. If each shrinks the signal by 10%, then after 50 steps only 0.950 = 0.5% is left, and the early pushes see almost no signal. If each grows it by 10%, 1.150 is about 117 times too big. The first case is a vanishing gradient, and the second is an exploding one. Model error: the model’s own slope can point the wrong way, and the optimiser will follow it with great confidence. The responses on the slide are restarts, shorter horizons with feedback, and better model coverage.

A planner uses a gradient step size above the stable limit, with no action bounds. What does the cost do?

Chapter 8

Futures Are Random: Compare Expected Costs

Score a plan by the average cost of its possible futures, never by the cost of the average future

You choose between two routes to the airport. Route A takes 30 minutes on average. Route B takes 35. A looks better. But A crosses a bridge that is sometimes stuck for an hour, and your flight leaves in 45 minutes. A’s average is better. A’s risk of missing the flight is worse. The average alone hid the problem.

The same trap waits in planning with a stochastic world model. A plan no longer has one outcome. It has a cloud of possible futures, and each future has its own cost. Slide 38 shows the smallest possible case. Plan A leads to two equally likely futures with costs 0 and 10. Plan B leads to two equally likely futures that both cost 4. Which plan is better?

Take the expected cost: each future’s cost times its probability, added up. Plan A: ½ × 0 + ½ × 10 = 5. Plan B: ½ × 4 + ½ × 4 = 4. Plan B wins, even though plan A has the single best outcome. If a bad outcome is expensive and you will only get one try, that is the right call. (If rare failures matter a great deal, the lecture adds, specify an acceptable failure probability as well as the average cost.)

The mistake: plan with the average future

Here is a shortcut that looks sensible. Predict the average future state, E[S], and compute the cost of that one state, J(E[S]). It needs a single rollout instead of many. The slide states plainly that this is wrong: E[J(S)] ≠ J(E[S]) in general.

Why? Think of a cost that squares the miss, like ours. A square is a bowl, and for a bowl the average of the heights is higher than the height at the average. Two futures at positions 0 and 2, each with probability ½, average to position 1. The cost of position 1 is (1 − 1)² = 0. But each future really costs (0 − 1)² = 1 and (2 − 1)² = 1, so the expected cost is 1. The average future sits on the target. No actual future does.

Score by the cloud, not its centre

Each plan ends at a random position. Plan A aims straight at the target but is shaky. Plan B aims a little short but is steady. The yellow bowl is the cost of a miss. Drag A’s shakiness, then change how many futures you sample.

0.80
10

Toy numbers, illustrative: cost is the squared miss (position − 1)². Plan A ends at 1 with the spread you set; plan B ends at 0.8 with spread 0.1. The futures are seeded random draws; “Draw new futures” changes the seed. The true expected costs are spread² for A and 0.04 + 0.01 = 0.05 for B.

Set the spread to 0.8 and look at the read-out. The cost of A’s average future is 0, so the shortcut calls A perfect and picks it. The true expected cost of A is 0.64 (the spread squared), and the average over a few sampled futures lands near it, far worse than B’s 0.05. The shortcut chose the wrong plan. Now cut the sample count to 2 or 3 and redraw a few times. The sampled average becomes noisy and can pick wrongly in the other direction. That is the price of sampling: a small M is cheap and unreliable.

Two different things to sample

The slide ends with a sentence that is easy to skim and important to keep: sampling candidate actions and sampling the futures of one plan are different operations. They sit at different levels of the loop. Count what a decision costs in the stochastic case.

Nest them and the data is a block of shape (N, M, H, ds): N plans, M futures each, H steps, ds numbers per state. The cost of a plan is the mean over the M axis, giving a vector of N costs. That vector is what CEM sorts and MPPI weighs. The compute grows as the product, and Chapter 10 puts it in one formula.

Three searches, one table

Slide 39 sums up Chapters 5 to 7. All three optimise actions inside MPC, with the model parameters fixed. They differ in how they turn costs into a better plan, and in what they require from the model.

MethodHow it updates a planNeeds action gradients?Good when
CEMRefit the proposal to the elitesNoThe model is a black box; the cost has jumps
MPPIAverage the cost-weighted perturbationsNoContinuous control with a warm start
Gradient-basedStep along the negative cost gradientYesThe rollout is smooth and the horizon short

The last column is our reading, not the slide’s. It follows from what each method needs: gradients need smoothness, and samples only need a score. A smooth, short rollout gives gradients cheaply. A rollout with discrete tokens or contact has no useful slope, so sampling takes over.

Two equally likely futures end at positions 0 and 2, and the target is 1 with a squared-miss cost. What is the cost of the average future, and the expected cost?

Chapter 9

Planning in a Latent World Model

Show the robot a photo of the goal, and plan in feature space with a model that never saw that goal

You want a robot arm to put a block where it sits in a photograph. You cannot write “the block is at x = 0.31, y = 0.12”, because you do not have the coordinates. You have a picture. So how does a cost like (p − 1)² become a cost made from a picture?

The lecture’s answer is short. Use the model’s own encoder. Encode the goal photo og into the same feature space as the robot’s camera frames, giving zg = E(og). Let the world model imagine where a candidate plan leaves the robot, as features ẑt+H(A). The cost is the distance between those two points in feature space.

Jgoal(A)
The cost of plan A: how far the imagined end of the plan is from the goal, in feature space.
d(ẑt+H(A), E(og))
The distance between two vectors: where the model says the plan ends, and where the goal photo lives. The goal image goes through the frozen encoder E, the same one used on camera frames.
d(u, v) = ‖u − v‖²2
Squared L2 distance: the sum of squared differences of the entries. The slide calls it one choice. The next slides use it (LeWorldModel) and the L1 version, a sum of absolute differences (V-JEPA 2-AC).

The slide ends with the key sentence: the goal enters the evaluator. It need not enter the dynamics model. The model was trained to predict what happens under actions, with no goal in mind. Give it a new goal photo and the same frozen model plans toward it. Nothing is retrained.

Try it. A toy robot lives in a two-number feature space. A fixed, frozen model predicts its next point from its action. Click anywhere to place the goal, or use a preset, then press Plan. The plan is found by CEM from Chapter 5, over H = 5 steps. Press Act one step to run MPC, with K = 1.

Plan toward a goal point

Blue is the robot’s real feature point. Purple lines are imagined plans (grey: candidates, purple: the chosen plan). Click the plane to move the pink goal. Then press Plan, or Act one step to run the loop.

Toy, illustrative numbers: a two-number feature space, a frozen step z′ = z + 0.3a plus a small fixed bend, actions in [−1, 1], H = 5, CEM with 40 candidates, 6 elites and 3 rounds (seeded). The real world equals the model here, so only the goal changes. The camera-angle switch adds a fixed offset of (0.5, −0.4) to the encoded goal: the lecture notes that viewpoint changes can affect the goal-feature distance.

Place a goal far away. The first plan cannot reach it in five steps, so its predicted end is only closer. After one action and a replan, it is closer again. This is MPC doing the work a single long plan cannot. Now switch on the camera-angle toggle. The robot goes to a point that matches the encoded photo perfectly, and that point is not where you meant. A cost made from features is only as good as the features. The distance is a task choice.

The same interface, many models

Slide 41 gives the checklist that lets one planner serve many models. Four boxes in a row: the current context (what the model needs to start), a latent rollout under candidate actions A, the predicted future features, and a task evaluator. Before planning you ask three questions. What must we initialise (which observations, memory or inferred state)? What does action search need (are candidate actions valid, and are futures sampled or deterministic)? How do evaluation and feedback work (what score ranks plans, and when does the next real observation arrive)? The advice: initialise with enough history to predict consequences, and clone that context for every candidate.

The remaining slides walk the family from Chapter 1 through this checklist.

Notice the pattern. Each model changes what you initialise and what a candidate carries. None of them changes the loop. That is why the lecture draws one diagram at the end and reuses it.

A robot has a frozen world model and a frozen encoder. You give it a new goal photo it never saw. What changes in the planning loop?

Chapter 10

What Can Go Wrong

Tell five failures apart: a goal that is not the one you meant, a blind model, a hard search, a short horizon and a missed deadline

A planner can fail without any bug in the code. Every line may run as written, and the robot still does the wrong thing. The lecture closes with six slides on why, and they give one habit: when a closed-loop system fails, ask which part failed. There are five parts, and each has a different cure. We take them in the lecture’s order.

1. The objective: success is hard to write down

You want a robot to carry a full cup to a table. Writing the cost is harder than it sounds. Do you penalise a spill? How much? Is any spot on the table fine? Slide 50 offers a shortcut: show a successful run and say “this is what success looks like.” A goal photo of the cup at the target does that, as Chapter 9 showed.

But one example does not say everything. Slide 51 lists three behaviours against two readings of the demonstration. “Carry the full cup carefully” satisfies both “match the final goal image” and “arrive without spilling.” “Spill, then arrive upright” may match the goal image and fails the no-spill reading. “Arrive safely at another acceptable placement” may fail the image match and pass the no-spill reading. A single photo cannot tell the readings apart. More demonstrations, comparisons, constraints or feedback can add the missing information, and if rare failures matter, give an acceptable failure probability as well as an average cost.

2. The model: it plans through a wall

The planner chooses whatever its model says is cheapest. If the model is wrong in a way that makes a bad action look good, the planner will find that action, and love it. Slide 52 shows a box pushed to a goal behind a wall. The model has never seen the wall stop anything, so it predicts the box slides through and lands on the goal with a distance near zero. In reality the box hits the wall and stops far short. The cost rule is the same in both pictures. The model got the motion wrong.

Plan through a wall

The planner picks the push strength that the model says ends nearest the goal. The purple ghost is where the model thinks the box ends. The blue box is where it really ends. Slide the model’s belief from “passes through” to “stops at the wall”.

100%

Toy, illustrative numbers: the box starts at 0, the wall stands at 0.48, the goal is at 1, and a push of strength u would carry the box 1.2u in free space. The real box stops at the wall. The model lets a share of the carry-on pass through; the planner picks u from 0 to 1 that minimises its predicted miss, with a tiny effort penalty of 0.01 u.

At 100% the model predicts a miss of 0.00 and the real miss is 0.52. This is the failure of slide 52. The symptom: the plan looks perfect in imagination and fails in the world. The measurement: compare the predicted end with the real end for the same action. The gap is the model error. The fix on the slide: collect interaction data around the obstacle, and restrict actions that the data does not support. A planner must not be asked to use a model outside the region where the model was trained. The same slide cites Chua et al. (PETS) for this point.

3. The search: a good model is still hard to plan with

Now suppose the model is right. Predicting the outcome of one plan is cheap. Finding a good plan among all plans is a search over a huge, uneven space, and slide 53 shows why. A box stands between the start and the goal. A useful first action may move you away from the goal, around the obstacle. A planner that rewards getting closer will take the nearest dead end. Restarts, structured proposals and hierarchical search help. None removes the difficulty in general.

4. The horizon: delayed rewards need a long look

Slide 54 is the same obstacle with a different cause. Even a perfect search cannot choose a detour whose benefit arrives after the end of its horizon. Try the numbers. A route straight at the goal reaches a dead end 0.4 away. A detour first moves away from the goal, then curves round and arrives. The planner scores each route by its distance to the goal at the end of the horizon H.

Look far enough ahead

The orange line is the straight route into a dead end. The purple line is the detour. The vertical teal line is the horizon H. The planner picks the route with the lower line at H.

5

Illustrative numbers drawn for this lesson, not from the lecture: the straight route closes 0.15 per step from distance 1.0 and stalls at 0.4 after step 4; the detour opens up to 1.48 by step 4, then closes 0.2 per step and arrives at step 12.

With H below 10 the straight route looks closer, so the planner takes the dead end. At H = 10 the detour’s distance (0.28) finally beats the dead end (0.40). Remedies from the slide: a longer H, subgoals (intermediate targets), or temporal abstraction (planning in bigger steps). Learned terminal values, which Lecture 10 covers, let a short horizon see far by summarising the future in one number.

5. The budget: the best plan that arrives late is no plan

Planning costs compute, and the controller has a deadline. Slide 55 gives the count: the rollout work is about I × N × M × H, the product of search rounds, candidate plans, futures per plan and model steps. The slide adds a careful note: this counts model transitions, not wall-clock time, and autoregressive token generation adds work within each step. Drag the four factors and see what a decision costs.

Count the rollout work

Each coloured block is one factor, drawn as a length on a log scale. Multiplying the factors adds the lengths. Raise one factor and see which one moves the total most.

3
100
5
20

The product formula is the lecture’s (slide 55). The default values are illustrative, not from any paper. One “model step” is one call of the transition function for one candidate and one future; real latency also depends on batching and hardware.

At the defaults the work is 3 × 100 × 5 × 20 = 30,000 model steps per decision. If you must act every 50 ms, that is a hard limit. The slide’s ways out: warm starts, useful proposals, batching, shorter rollouts, or larger action chunks. A larger K saves planning calls but delays feedback (Chapter 4).

The diagnosis table

Slide 56 puts it all in one table. When the closed loop fails, ask the question for the part you suspect, and measure real task outcomes under stated compute and interaction budgets.

If the failure is…AskIn this lesson
ObjectiveDoes the score actually represent success?The cup, the full-speed arrival
InformationCan the available observations tell the relevant states apart?A goal photo from another angle
ModelDo predicted consequences match actual outcomes?The wall
SearchCan the optimiser find a good plan within this model?The dead end, the short horizon
ExecutionCan we plan, act and observe quickly enough?I × N × M × H, and the interval K
A planner’s imagined rollout ends exactly on the goal, but the real robot stops far short, with the same action. Which part of the closed loop should you check first?

Chapter 11

Field Guide: Planning on One Page

Review the loop, the numbers and the code, then look ahead to learning the behaviour instead of searching for it

You can now rebuild Lecture 9 from memory. A world model predicts. A cost says what you want. A search tries action sequences against the model and keeps the cheapest. A feedback loop executes the first action, looks, and plans again. The lecture’s closing diagram says it in five words: predict, evaluate, optimise actions, execute, observe. Then it asks one question: what changes during planning, and what stays fixed? The model parameters stay fixed. The actions change, and the plan is rebuilt after every real observation.

The numbers that matter

WhatNumberWhy it matters
Arrive-and-stop costJ = (pH − 1)² + vH² + 0.1 ∑ a²Three failures, three terms
Compounding error, gain 1.08rollout error 5.823 vs one-step 0.160 at step 12Look at reality often
LGSSM predictionPj+1 = APjA⊤ + Q; covariance independent of the planPlanning needs only the mean path
Horizon and intervalH = 5, K = 1 or 3Look-ahead versus how often you check
CEM, four candidatescosts 1.00, 4.20, 0.20, 0.30; new centre (0.75, −0.75), spread 0.0625Elites refit the proposal
CEM round means0.175, 0.19773, 0.17575A small batch need not improve each round
MPPI weights, costs 0, 1, 3, τ = 10.705, 0.259, 0.035Cheap futures vote more
Two plans, random futuresexpected cost 5 vs 4E[J(S)] is not J(E[S])
LeWorldModelH = K = 5 model steps × frame skip 5 = 25 environment stepsLarger K delays correction
V-JEPA 2-ACL1 goal distance, CEM, K = 1Replan after every action
Planning work∝ I × N × M × HA decision has a deadline

The whole planner as code

Here is MPC with a CEM search and a goal photo, as one function. Read it with the chapters in mind: the encoder is Chapter 9, the rollout is Chapter 1, the cost is Chapter 2, the search is Chapter 5, and the replanning loop is Chapter 4.

pythondef act(model, enc, history, goal_img, H=5, N=40, Ne=6, I=3):
    z0 = enc(history)                        # current information: (d_z,)
    zg = enc(goal_img)                       # the goal enters only here: (d_z,)
    mu, v = np.zeros((H, d_a)), np.full((H, d_a), 0.5)
    for k in range(I):                         # search rounds; real time has not advanced
        A = clip(mu + np.sqrt(v) * rng.normal(size=(N, H, d_a)), lo, hi)   # (N, H, d_a)
        Z = model.rollout(z0, A)               # frozen model: (N, H + 1, d_z); clone z0 per candidate
        J = ((Z[:, -1] - zg) ** 2).sum(-1)       # squared L2 goal distance: (N,)
        E = np.argsort(J)[:Ne]                  # elites
        mu = A[E].mean(0)
        v  = np.maximum(1e-3, A[E].var(0))      # variance floor
    return mu[0]                              # MPC: send the first action, then observe and call again

A checklist for a planner that is not working

  1. Objective. Print the cost for a plan you know is good and one you know is bad. Is the good one lower?
  2. Information. Do two states you need to tell apart give two different encodings?
  3. Model. For the plan the planner chose, compare the predicted end with the real end. How big is the gap?
  4. Search. Run the search longer, or with more candidates. Does the cost fall? If it does, the search was the limit.
  5. Horizon. Does a longer H change the chosen action? If it does, the horizon was too short.
  6. Execution. Is the planning time under the control period? If not, cut I, N, M or H, or raise K.

Where this sits

The last lecture built the models and asked them to predict given actions. This one used the predictions to choose actions. The next lecture turns to learned behaviour. Planning searches from scratch at every step, and every step costs compute. Slide 58 asks the question that opens Lecture 10: can experience reduce the need to search from scratch? The answer is to learn a policy π(a | s) that proposes actions directly, a value V(s) that scores a state, and an action value Q(s, a) that scores a state and an action together. Dreamer and TD-MPC connect imagined futures to learning how to act. You have already met the seeds: TD-MPC’s score ends with a learned value, and its search takes proposals from a learned policy.

The course site lists the readings and assignments for every lecture. The papers behind this one are the ones named on the slides: PlaNet (Hafner et al.), Dreamer, TD-MPC (Hansen et al.), World Models (Ha and Schmidhuber), IRIS (Micheli et al.), V-JEPA 2 (Assran et al.), LeWorldModel (Maes et al.), the MPPI paper (Williams et al.), PETS (Chua et al.), and the MIT trajectory-optimisation notes.

Keep going

Primary sources named on the slides: PlaNet, Dreamer, TD-MPC, World Models, IRIS, V-JEPA 2 and PETS.

The takeaway. A world model never chooses. A cost says what you want, a search tries plans against the frozen model, and feedback re-aims every step. When the loop fails, ask which of the five parts failed (the objective, the information, the model, the search or the execution), and change only that one.
A robot with a goal photo, a frozen latent model and CEM reaches wrong spots on a new camera angle but works on the old one. Which part of the diagnosis table points to the cause, and what is the first fix to try?

Now press Teach and explain one planning decision out loud, from memory: what the cost says, how the search finds a plan, why only the first action runs, and what you would check if the real robot disagreed with the imagined one. Then go back to the cart at the top, drag the real thrust to 50% and to 130%, and explain why re-planning every step holds the target when planning once does not.

CIS 6280 Lecture 9: Planning and Control with World Models
The course site · Lecture slides · Back to Gleams