A bird's-eye tour of the whole field: why a brain exists for movement, what a policy really is, and the ladder of methods from copying an expert to learning a world from scratch.
There is a small marine animal called the sea squirt. In its youth it swims, a tadpole-shaped larva with a primitive eye, a tail, and a tiny nervous system, drifting through the ocean hunting for a place to live. When it finds a good rock, it cements its head to the surface, stops moving forever — and then does something startling. It digests its own brain and nervous system and absorbs them for nutrients.
The neuroscientist Daniel Wolpert tells this story to make one blunt claim, which is the thesis of this entire lecture: "We have a brain for one reason and one reason only, and that's to produce adaptable and complex movements." Once the sea squirt no longer needs to act in the world, it has no further use for a brain. So it eats it.
This reframes what "intelligence" even means. We are dazzled by systems that play chess or write essays, and we wave away the fact that a one-year-old child can pick up a strange spoon, on a strange table, and bring it to her mouth without a second thought. But which of those is actually hard for a machine?
Hans Moravec noticed the answer decades ago. Moravec's paradox states that the hard problems are easy and the easy problems are hard. In Moravec's own 1988 words: it is comparatively easy to make computers play checkers or pass intelligence tests, and difficult or impossible to give them the perception and mobility skills of a one-year-old. Abstract reasoning is recent and rare in evolution; perception and movement are billions of years old and exquisitely tuned. Evolution spent three and a half billion years refining motor control and only the last sliver of time on language.
Hover or click each rung. Things humans find effortless (perception, grabbing a mug) took evolution the longest and are hardest for robots. Things we find impressive (chess, language) are youngest and, for machines, the easiest.
So why is grabbing a mug so hard? A tempting answer is "our hardware is just better — human muscles beat robot motors." The instructor pushes back: that's likely not the bottleneck. A cheap teleoperated robot (the Mobile ALOHA system costs under thirty-two thousand dollars and has only sixteen degrees of freedom) can do delicate kitchen tasks beautifully when a human is driving it. The motors are fine. What's missing is the algorithm that closes the loop the way a brain does: perception, planning, control, and action, all coupled together, all reacting to each other in real time.
If a brain exists to produce action, then we need a precise picture of how action and perception fit together. Reinforcement learning gives us the cleanest one, and it is just two boxes and two arrows. It comes straight out of the opening chapter of Sutton and Barto.
On one side is the agent — the decision-maker, the thing we are building. On the other side is the environment — everything else: the floor, gravity, the mug, the wind. They talk to each other in a never-ending cycle.
That loop is the heartbeat of every method in this course. The agent never gets to see the whole future; it only ever sees where it is now, picks an action, and gets pushed somewhere new. The single arrow that makes this hard — and makes it a control problem and not just a prediction problem — is the one from the action back into the environment: your action changes the next state you have to deal with.
The environment hands back a single number at every step: the reward, written r(s, a) — a score for taking action a in state s. Sutton and Barto elevate this to a foundational claim called the reward hypothesis:
This is a remarkably strong and slightly uncomfortable claim. "Walk without falling," "stack the blocks," "win the soccer match" — the bet is that each can be boiled down to one number per step, and that maximizing the long-run total of that number produces the behavior you wanted. The agent's true objective is not the reward at this instant; it is the return — the sum of all future rewards, with rewards further in the future counted a little less (a far-off reward is worth less today, the same way a dollar next year beats nothing but is worth less than a dollar now).
Let's make that concrete. Suppose a walking robot earns plus one for every step it stays upright and minus ten the moment it falls. Over an episode it racks up a sequence of rewards; the return is what we add up. The lab below builds exactly this accumulation — the quantity every algorithm in the course is secretly trying to grow.
We keep saying the agent "chooses an action." Let's pin down the object that does the choosing, because it is the thing we actually learn and the thing every method in the course produces. It is called the policy, written π (the Greek letter pi).
The instructor draws this with the most important example in modern robotics: a policy that maps an image to a control. The slide for imitation learning shows it literally — a network learns the mapping "from state (image) to control (steering direction)." So let's trace the actual data, because a policy is not an abstract arrow; it is a concrete function with concrete shapes flowing through it.
Say the robot is a self-driving car and its sensor is a forward camera.
Read that top to bottom and the magnitude of the problem jumps out: the policy compresses a high-dimensional input (an image is a huge, raw, mostly-irrelevant blob of pixels) down to a low-dimensional output (a handful of motor numbers). The instructor names this the central difficulty of unifying perception and control: "How to get to the low-dimensional manifold of meaningful things from raw input." Most of those 150,000 pixels are sky and asphalt; the policy must learn which few features (the lane lines, the car ahead) actually decide the steering.
The flip side is just as hard for some robots: a humanoid has a high-dimensional output too — dozens of joints — and the search space of action sequences grows exponentially with the number of steps. Perception is hard because the input is huge; control is hard because the output space is huge; doing both at once is robot learning's signature challenge.
You don't need a deep network to feel what a policy is. The simplest one maps state to action with a single straight line: action = slope × state + offset. We can learn that line from demonstrations by plain supervised regression — show it states paired with the expert's actions, and fit the line that best reproduces them. That is the seed of imitation learning, which we'll meet on the ladder next chapter.
A toy 8×8 camera frame (left) is fed to a policy that must squeeze it to a single steering number (right gauge). Drag the lane offset slider: the bright lane pixels shift, and a good policy turns that shift into the correct steer. The point is the funnel — thousands of pixels in, one action out.
So we want a good policy. How do we get one? The lecture surveys the family of methods this course will teach, and they are not a random list — they form a ladder of difficulty, ordered by how much help the robot is given and how much it must figure out for itself. Climb it and the assumptions get weaker, the data gets harder to come by, and the autonomy goes up.
| Rung | Method | What it's given | How it learns the policy |
|---|---|---|---|
| 1 (easiest) | Supervised / Imitation Learning | Expert demonstrations (state–action pairs) | Copy the expert: regress action on state |
| 2 | Model-free RL | Only a reward signal; trial and error | Make "good stuff" likelier, "bad stuff" rarer |
| 3 | Model-based RL | Reward + ability to learn a world model | Learn the dynamics, then plan inside it |
| 4 (hardest data) | Offline RL | A fixed logged dataset — no interaction | Squeeze a policy from old data, no exploring |
Let's walk the rungs, because the distinction between them is the spine of the syllabus.
Imitation learning is the easiest because someone already solved the task: a human expert. You collect demonstrations and train the policy to reproduce the expert's action at each state — exactly the regression you just coded. The catch (the whole subject of Lecture 6) is that the moment your copy makes a small mistake, it drifts into states the expert never showed it, and the errors snowball. But when good demonstrations exist, nothing is faster.
Model-free RL throws away the expert. The robot learns purely from interacting with the environment — trial and error, guided only by reward. The instructor's one-line summary of the basic policy-gradient algorithm REINFORCE: "make good stuff more likely and bad stuff less likely." No model of the world is built; the robot just tries things and reinforces what worked. It's powerful and general, but desperately data-hungry, because trial and error in the physical world is slow and risky.
Model-based RL adds a clever twist: instead of only reacting, the robot learns a model of the world's dynamics — a "world model" that predicts the next state from the current state and action — and then plans inside that model. The instructor's framing: no simulator handed to you, so you collect data from the real robot, learn a model, design a policy against it, and deploy. Because you can "imagine" rollouts in your learned model instead of executing every one for real, it is far more sample-efficient — it squeezes more behavior out of fewer real interactions.
Offline RL is the strictest: you get a fixed dataset of logged experience and you may not interact with the world at all — no exploring, no trying things, because online interaction is expensive and unsafe. The instructor flags the key contrast: unlike imitation learning, offline RL does not rely on expert data — the logged trajectories can be mediocre or random, and the algorithm must still distill a good policy from them. That makes it powerful and treacherous in equal measure.
Click a rung to see what that method is handed and what it must figure out for itself. The bars show the trade: more help (left) means easier learning but a ceiling set by your demos; less help (right) means harder, hungrier learning but the freedom to surpass any expert.
There's a fault line running underneath that whole ladder, and it's worth surfacing on its own because it explains why the rungs differ. It is the question: does the robot have a model of the world's dynamics, or must it learn one?
A model of the dynamics (also called a transition model or world model) is a function that answers: if I'm in this state and I take this action, what state do I end up in? If you have it, you can plan — simulate the consequences of actions in your head before committing to any.
| Situation | You have the dynamics… | Then you can… |
|---|---|---|
| Explicit / known model | by hand, from physics (mass, friction, geometry) | use classical optimal control: LQR, MPC, trajectory optimization |
| Learned model | by fitting it from data the robot collects | do model-based RL: plan inside the learned model |
| No model | you don't have it and don't even try to build it | do model-free RL: pure trial and error |
The instructor draws the contrast crisply: model-free RL assumes the dynamics are unknown and doesn't even attempt to learn them — it just reacts. Model-based methods either get the model handed to them (explicit physics) or learn it from real data and then design a controller against it. When the physics is clean and known, classical control with a known model is breathtaking — this is how Apollo flew to the Moon and how legged robots do backflips. When the physics is unknowable (contact, deformable objects, a changing floor), you must learn the model, or skip it entirely.
Let's feel that leverage directly. The lab pits two learners against each other on the same task: a model-free learner that must visit every state to learn its value, versus a model-based learner that builds a one-step map of the world and then propagates value through it. With the same handful of real transitions, the model-based learner figures out the right move from far more states.
Recall Moravec's ladder from Chapter 0: low-level motor skills sit at the bottom (ancient, hard for machines), high-level planning and language near the top (recent, easy for machines). That same split structures how a real robot is built. Intelligent behavior is almost never one flat policy from pixels to torques — it is a hierarchy.
At the top sits the high-level plan: "to make a salad, first grab the bowl, then chop the lettuce, then toss." This is symbolic, slow, and abstract — the domain where language models now shine (the instructor cites systems that use a large language model to lay out the steps). At the bottom sit the low-level skills: the actual joint torques that reach for the bowl and close the gripper without crushing it. Fast, continuous, reactive — the part that's genuinely hard.
The lecture's deeper point is that these levels can't be designed in isolation. Classical robotics built perception, planning, and control as separate modules that barely talked to each other — and errors at one interface got amplified by the next. The instructor's striking example: a human teleoperator succeeds where an autonomous robot fails on identical hardware, precisely because the human runs perception, planning, control, and actuation as one tightly coupled closed loop. The same lesson appears in drone racing: it took a team over seven years to beat the human champion, and even then only on one known track. Humans win by coupling the levels; robot learning's promise is to learn that coupling instead of hand-designing it.
A robot (warm) must reach the goal (teal) but a wall blocks the straight path. Flat mode steers directly at the goal and jams against the wall. Hierarchy mode picks a subgoal (the gap, a purple waypoint) first, then a simple reach skill drives to it — one hard problem split into two easy ones.
The lecture spends its middle showing real systems, so the survey isn't abstract. It's worth knowing the gallery, because it tells you which ideas already work in the wild and which are still aspirational.
| System | Recipe | Rung |
|---|---|---|
| DM Control Soccer; Gran Turismo Sophy (beats human drivers, no collisions) | Deep RL in simulation + distributed training + careful reward design | Model-free RL |
| Eureka (dexterous pen-spinning) | A language model writes the reward function | Reward design via LLM |
| DeepMind 1-v-1 soccer robots; DATT trajectory tracking | Train in sim, deploy on real hardware (Sim2Real) + online adaptation | Sim2Real |
| RoboCook (best-systems paper) | Learn a structured world model of dough/tools, plan in it | Model-based RL |
| Rope-whipping to a target; DNN dynamics + sampling-based control | Collect real data, learn a "delta" dynamics model, plan | Model-based RL |
| Diffusion policy; Mobile ALOHA | Collect human teleoperation demos, imitation-learn the policy | Imitation |
Notice the spread: every rung of the Chapter 3 ladder has a flagship success. NVIDIA's Isaac Gym (massively parallel simulation) and data-driven safety filters round out the toolkit — the safety filter sits on top of any policy and vetoes actions that would break hardware, because, as Lecture 1 put it, physics doesn't forgive.
Then the instructor shows the bloopers: robots fumbling tasks "that seem very simple for humans." The 2015 DARPA Robotics Challenge is full of humanoids toppling over while opening a door. Even Mobile ALOHA, which can do beautiful kitchen work under teleoperation, fumbles when running autonomously.
The instructor frames the open problem honestly with the four pillars from Lecture 1 — algorithm, data, computation, hardware — and asks: how far are we from embodied intelligence in the real physical world? Robot learning has made far more progress at improving each module (better perception, better control) than at unifying them into one seamless brain. Closing that gap — getting perception, planning, and control to co-adapt the way they do in a crow tracking a target or a person placing their feet on rough ground — is the real frontier, and the reason the field is wide open for you.
Time to put the tour in one place. A robot must reach a sequence of goals in a world with an unknown push (a "wind"). You hold the dials that map onto the whole survey, and you can watch one method outclimb another in real time.
Press Run. The robot (warm) chases goals (teal dots) while an invisible wind pushes it. Switch the method and crank the wind: the imitation heuristic and the reactive model-free policy both sag as the wind rises, while the model-based agent learns the wind and leans into it. Watch the return climb.
The instructive sweep is to set the wind high and step through the three methods. Imitation drifts because its straight-line "expert" never anticipated a current; model-free fights the wind reactively but with a steady-state offset; model-based, which estimates the push and cancels it, sails to the goals with the highest return. That progression — copy, react, model — is the ladder you climbed in Chapter 3, made visible.
You now have the aerial view. Lecture 1 told you what robot learning is and why it's hard; this lecture showed you the landscape of methods that attack it. Here is how the topic groups hang together — the same ladder you climbed, now labelled as the syllabus.
You came in from Lecture 1 — What Is Robot Learning?. To turn each thread of this overview into full understanding, these companion Gleams expand the rungs:
"We have a brain for one reason and one reason only — to produce adaptable and complex movements." — Daniel Wolpert