Nobody has the equations for a robot's camera feed. So learn them — in a tiny latent space — and let the robot rehearse a thousand futures by dreaming instead of moving.
You learned in Lecture 1 that robot learning is hard because data is expensive — every datum has to be acted out on a real robot, in real time, at risk to the hardware. The cleanest answer to "where does the data come from?" is breathtakingly simple to say: learn a model of the world, then generate as much practice data as you like inside it, for free. That is model-based reinforcement learning (MBRL), and it is the most sample-efficient family of methods we have.
Lectures 12–14 assumed you already knew the dynamics — you had clean equations for how a state and an action produce the next state, and you used trajectory optimization or MPC on top of them. But step back and ask the uncomfortable question for a real robot: who actually wrote down those equations?
For a clean cart-pole, a textbook did. But your robot's only window onto the world is a camera — a stream of pixel grids, maybe 64×64×3 numbers per frame, twenty times a second. There is no equation that maps "this image of a coffee mug" plus "rotate gripper 5 degrees" to "the next image." The physics of contact, occlusion, lighting, and clutter is hopeless to hand-derive.
Why does a learned model buy sample efficiency? Because one real interaction can be reused a hundred times. A model-free method throws each transition away after one gradient step. A model-based method fits a model to that transition, then rolls the model forward thousands of times "in imagination" — squeezing far more learning out of the same scarce real experience.
Both agents reach the same skill. The bar shows how many real environment steps each spends to get there. Slide how many imagined rollouts the model-based agent runs per real step, and watch its real-step budget shrink — that's data being manufactured for free.
The catch — the one that haunts the whole lecture — is that the dreamed futures are only as good as the learned model. A small per-step prediction error compounds over a long rollout, exactly the villain from Lecture 1, now living inside your own head. Half of MBRL research is fighting that compounding. We'll meet it head-on in Chapter 7.
Start with the simplest version, before any pixels. Suppose the world is time-invariant: the same state and action always produce the same next state. We want a learned function — call it f-hat, written f̂θ — that predicts the next state from the current state and action. We fit it by minimizing prediction error on logged transitions:
The instructor calls this "a supervised regression problem" — and then immediately warns that it is not ordinary supervised learning, for two deep reasons.
In ImageNet the images arrive independently and you do not get to pick them. Here, the transitions in your dataset D were generated by some policy acting in the real dynamics. Change the policy and you change the data distribution. So a question with no analog in supervised learning appears: what policy should we use to collect data? Collect with a bad policy and your model is sharp only in regions you never need.
The number we minimize — prediction error — is not what we ultimately care about. We care about good control. The instructor's blunt warning: "A good model doesn't necessarily mean good or easy control/planning." A model can have tiny average error yet be locally so steep or jagged that any controller built on it is unstable. That decoupling — accurate-but-useless — is why naive MBRL fails, and why the lecture spends slides on regularizing the model (spectral normalization, Lipschitz bounds, physics priors) so that it is not just accurate but controllable.
For a low-dimensional robot (say a 4-DOF arm), a state x might be a length-8 vector (4 joint angles + 4 velocities) and an action u a length-4 vector of torques. The model is a small MLP: input a length-12 vector concat(x, u), output a length-8 vector x′. We train it on logged tuples. So far this is exactly the regression above. The interesting break comes when x stops being eight clean numbers and becomes a 64×64×3 image — the subject of Chapter 2.
The teal curve is the true (unknown) dynamics. The warm curve is a learned fit. Crank the wiggle: an overfit model can hug the data points yet become so jagged that a controller stepping along it will jerk wildly. Low error, bad for control.
Now the camera. An observation is an image o of shape 64×64×3 — about 12,288 numbers. Could we learn dynamics directly in pixel space: given this image and an action, predict the next image? In principle yes; in practice it is a disaster. Predicting 12,288 numbers per step is enormously expensive, the prediction is dominated by irrelevant texture and noise, and you cannot roll thousands of such predictions forward in parallel.
The key move of deep MBRL is to introduce a small latent state — a compact vector, call it s, of maybe 32 to 256 numbers — and do everything there.
Read the data flow carefully, because this is the realization the lecture demands:
enc: 64×64×3 image → s (length 32). A convolutional net that maps a high-dimensional image to a small vector.dyn: (s_t length 32, a_t length 6) → s_{t+1} length 32. A small net predicting the next latent from the current latent and action.rew: s_t length 32 → scalar r̂. Predicts the reward from the latent state alone.dec: s → reconstructed 64×64×3 image. Used only at training time to force the latent to retain real information; thrown away at planning time.Hafner, the Dreamer author, puts it precisely: this latent model "mimics a non-linear Kalman filter or latent state-space model — but it is conditioned on actions and predicts rewards, letting the agent imagine the outcomes of potential action sequences without executing them." In other words: it is exactly the Kalman-filter idea from the estimation lectures, generalized to images and made action-aware.
A toy "image" of 12,288 numbers is squeezed by the encoder into a latent of just a handful. Slide the latent size and watch the memory-per-step (the cost of one rollout step) collapse — and with it, how many parallel dreams you can afford.
Now we make the latent model concrete. Visual control is a POMDP: the agent never sees the true state, only images o. Dreamer answers with a recurrent state-space model built from three learned components (the lecture's bullet, expanded):
| Component | What it maps | Plain meaning |
|---|---|---|
| Representation model p(st | st−1, at−1, ot) | past latent + action + this image → latent st | The encoder, with memory. It sees the new image. Used while interacting with the real world. |
| Transition model q(st | st−1, at−1) | past latent + action → latent st | Predicts the next latent without the image. This is what lets us imagine forward blind. |
| Reward model q(rt | st) | latent st → scalar reward | Scores how good a latent state is, so imagined rollouts can be evaluated. |
The split between the representation model (sees the image) and the transition model (does not) is the heart of it. The representation model is used during real interaction, where images keep arriving. The transition model is used during imagination, where there are no images — it must hallucinate the next latent from the previous latent and the action alone. Training forces these two to agree, so the transition model learns to predict what the encoder would have produced.
Because a single image is ambiguous (you cannot read velocity off one frame), the latent carries a recurrent memory: st depends on st−1. The model accumulates history into the latent, exactly the way a Kalman filter accumulates a belief. That is why the lecture calls it a "non-linear Kalman filter": a recurrent state-space model that integrates a stream of partial observations into a compact, Markovian belief — and, unlike a Kalman filter, is conditioned on actions and predicts reward.
Press Step to push one frame through. Watch a real image get encoded into a latent (the small bar), the action injected, the transition model predict the next latent and reward, and a decoder rebuild the image. Toggle Imagine to cut the camera off — now only the transition model runs, exactly as in a dream.
Here is the payoff that gives Dreamer its name. Once the transition model and reward model are trained, the agent stops needing the real world to practice. It picks a latent state it has actually visited, then rolls the transition model forward in latent space — choosing actions from its policy, collecting predicted rewards — never touching a camera, never moving a motor. Each such rolled-out latent trajectory is an imagined trajectory, and the agent trains its behavior on these.
Make it tangible with a toy linear latent. Say the latent is 2-dimensional and the (learned) transition is
and the reward is r̂(s) = −(s0)² (we want the first latent coordinate near zero). Start at a real, encoded latent s0 = (1.0, 0.0). The policy says "push with a0 = −1." One imagined step:
That single line of arithmetic — one matrix-vector multiply — just produced a labeled training example: "in state (1.0, 0.0), taking action −1 leads to state (1.0, −0.5) earning reward −1." We paid zero real environment steps for it. Repeat from a thousand different starting latents, for a horizon of fifteen each, and you have fifteen thousand training transitions from a handful of real interactions. That multiplication — one real step into a thousand dreamed ones — is the engine of sample efficiency.
Let's measure the multiplication in code: fit a one-step latent model from a little real data, then use it to manufacture many transitions and verify the imagined trajectory matches the truth at short horizon.
We can now dream short rollouts. But Chapter 4's caveat bites: if we only sum rewards inside a 15-step dream, the agent is short-sighted — it ignores everything that happens after step 15. Dreamer's central contribution fixes this with an actor-critic living entirely in latent space.
| Model | Maps | Job |
|---|---|---|
| Actor (policy) a = π(s) | latent s → action a | Choose the action in each imagined latent state. This is what we deploy. |
| Critic (value) v(s) | latent s → scalar | Estimate the total discounted future reward from latent s — summarizing everything beyond the dream horizon. |
The trick: roll out a short imagined trajectory s0, s1, …, sH. The return we train the actor to maximize is the sum of dreamed rewards plus the critic's value at the final dreamed state:
That tail term, γH v(sH), is the critic standing in for "and good things keep happening after the dream ends." The critic is trained for Bellman consistency on imagined rewards (its value should equal one dreamed reward plus the discounted value of the next dreamed state), and the actor is trained to maximize the target.
Let's build the heart of phase 2: bootstrapping a short imagined rollout with a learned value, and showing it gets you the true long-horizon return that a truncated sum misses.
Dreamer learns a policy in imagination and then runs it. TD-MPC takes the other classic route from Lectures 12–14: instead of committing to a learned policy, it plans afresh at every step — model predictive control — but in the learned latent space, and with a learned value to see past the planning horizon. It is the marriage of MPC and TD-learning, hence the name.
TD-MPC learns the same latent ingredients — an encoder to a latent z, a latent dynamics model, and a reward model — plus a learned value (cost-to-go), trained by TD-learning (the same bootstrapping idea as Dreamer's critic). The difference is entirely in how it acts.
The sampling-and-refitting loop is MPPI (Model Predictive Path Integral) — the same sampling-based MPC you met for offroad driving in Lecture 14, except the rollouts happen in a learned latent space instead of a known simulator, and the score of each plan is "sum of predicted latent rewards + learned value at the end." CEM (the cross-entropy method) is a close cousin used by PETS.
| Dreamer | TD-MPC | |
|---|---|---|
| How it acts | Amortized policy: train an actor in imagination, just run it | Online planning: re-solve a short MPC at every step in latent space |
| Sees past horizon via | Critic value on imagined rollouts | TD-learned value on planned rollouts |
| Optimizer | Analytic value-gradient backprop through the dream | Sampling (MPPI / CEM), gradient-free |
| Runtime cost | Cheap at deploy (one network pass) | Heavier at deploy (many rollouts per step) |
Each faint line is one candidate action sequence rolled out in the latent model. Press Plan to sample a batch; MPPI reweights toward high-scoring plans and the warm path is the chosen one. Toggle Use value to add the learned tail — without it, short plans are myopic and stop short of the goal.
In Lecture 1 you met compounding error in the real world: a behavior-cloned policy makes one mistake, lands off-distribution, and spirals. Model-based RL has the same villain, now inside the learned model. The lecture writes the planning objective with the model nested inside itself:
Look at those nested f̂'s. At step 3 you are feeding the model's own output back into itself, twice. Each pass adds a little error, and errors stacked through a function multiply rather than cancel. A model that is 99% accurate per step can be wildly wrong after twenty self-applications. This is challenge 5, and it is why a model you fit perfectly to one-step data can still produce useless long rollouts.
You already measured this in Chapter 4's lab: the imagined rollout matched the truth at step 3 but its error never shrank as the horizon grew — that is the compounding starting. The same arithmetic that manufactures cheap data also manufactures cheap error, one self-application at a time. The remedies above are all about giving that error fewer steps to compound through, or a smoother space to compound in.
Time to put every idea in one moving picture. A robot must reach a goal using a learned latent model. You hold the dials that define the whole lecture: how much real data trained the model, how short the dream horizon is, and whether a value covers the tail. Watch the imagined plan (dashed) versus where the robot really ends up (solid).
Press Run. The robot encodes its state, dreams a plan (the faint imagined path), executes it in the real world (the warm path), and re-plans. The teal dot is the goal. Break it: train on little data (inaccurate model), dream too long (compounding), turn the value off (myopic) — and watch the warm path miss the goal even though the dream looked great.
The most instructive combination: a sharp model with a too-long dream and the value off. The imagined plan confidently swoops to the goal — and the real robot, executing it, drifts wide, because the late part of the dream was fiction. Then turn the value on and shorten the horizon: the dream is humbler, but the real robot arrives. That gap between a beautiful dream and a real arrival is the entire engineering discipline of deep MBRL.
You now own deep model-based RL's two flagships. Both rest on a single idea — learn a compact latent world model from rich observations, then act inside it — and split only on how they act: Dreamer trains a policy by dreaming; TD-MPC re-plans online in the latent space. Everything else (encoder, latent dynamics, reward head, value) they share.
Lecture 14 gave you planning when the dynamics are known; this lecture learned them when they are not. Lecture 16 zooms out to structured world models — models that bake in objects, contacts, and physics so they generalize better and compound error less. And the instructor's closing philosophical question lingers: is the world model the next frontier toward AGI? Sora is a "preliminary large world model" (it predicts video but is not conditioned on actions); RFM-1 is a robotics foundation model whose world model is conditioned on actions — exactly the action-conditioned latent prediction you just learned, scaled up.
"Learn the world; then rehearse a thousand futures before you move once."