Learning to make sequential decisions in the physical world. Three words that carve out a field — and a single quadratic that explains why it is so much harder than ImageNet, AlphaGo, or GPT.
On the first day of CMU's 16-831, the instructor asks a question that sounds like a joke: "Is a microwave a robot?" It is not a joke. It is the fastest way to find the edge of an entire research field.
A microwave senses (you press "90 seconds"), it acts (it heats), and it stops. A factory welding arm is more impressive: it traces the same weld seam thousands of times a day with millimeter precision. Both are machines that do things in the physical world. Neither is what this course means by a robot that learns.
Here is the difference. Bolt that welding arm to a slightly different car, or let the part arrive rotated five degrees, and it welds thin air. It never looked. It runs a fixed script — what engineers call open-loop control: act first, never check the result. The microwave is the same: it never measures whether your food is actually hot.
Picture instead a robot arm asked to pick up a mug it has never seen, on a table it has never seen, in lighting it has never seen. It cannot run a script. It must sense the scene, think about where the mug is and how to grasp it, act by moving — and then sense again, because the mug may have slipped, the lighting may have changed, its own gripper may be in the way. That ring — sense, think, act, sense again — is the closed loop, and it is the object this entire course studies.
Press Run. Watch a robot try to reach a target. Toggle Closed loop off to make it a microwave: it commits to one plan and never looks again. Drag the disturbance slider to push the world around mid-motion.
With the loop closed, the robot shrugs off the disturbance — it sees it drifted and corrects. With the loop open, the same disturbance ends with the gripper somewhere in space, confidently wrong. That gap is the whole reason the field exists.
"Learning to make sequential decisions in the physical world." The instructor decomposes this sentence into three independent properties. Each one, on its own, names a huge area of AI. Robot learning is the rare corner where all three hold at once.
Learning means the behavior is data-driven: the robot improves from data rather than from a human hand-deriving every rule. The alternative is the entire classical toolbox — search & planning, classic control, optimal control — where an engineer writes down a model and solves for the controller. Those methods are not "dumb"; as we will see in Chapter 6 they fly rockets. They just do not learn from experience.
Sequential means the current decision influences the next state, and therefore the next decision. Your choices have consequences that come back to you. The alternative is the one-shot world: a bandit (each pull is independent) or standard supervised learning (each image is classified independently, and getting one wrong does not change the next image). Sequential structure is what makes the problem a control problem and not just a prediction problem.
Physical world means the robot is embedded in a real, closed loop with messy dynamics, gravity, friction, noisy sensors, and zero tolerance for catastrophic mistakes. This is what people mean by embodied intelligence. The alternative is a clean digital sandbox: RL for games (Atari, Go) or LLMs living entirely in token-space.
Click a system to light up which of the three properties it has. Only when all three are on do you land in the Robot Learning corner.
| System | Learning? | Sequential? | Physical? | What it is |
|---|---|---|---|---|
| ImageNet classifier | ✓ | × | × | Supervised learning |
| Multi-armed bandit | ✓ | × | × | One-shot decision learning |
| AlphaGo | ✓ | ✓ | × | Game RL (clean, simulated) |
| GPT / LLM | ✓ | ✓* | × | Token-space generation |
| Hand-tuned MPC drone | × | ✓ | ✓ | Classical control |
| Learned robot policy | ✓ | ✓ | ✓ | Robot learning |
*An LLM's autoregressive generation is sequential within a sequence of tokens, but those tokens do not move a body through a world with consequences — there is no physics in the loop.
Chapter 0 gave us the picture; now let us give it symbols, because the rest of the course speaks in these symbols. Don't worry — there is no calculus here, just four named quantities and one arrow.
The single most important arrow is the last one: the action changes the next state. In symbols, the next state is produced by the world's transition dynamics:
That feedback is exactly what supervised learning lacks. In supervised learning your prediction on image #5 has no effect on image #6 — the data is fixed. Here, a clumsy action at step 5 drops you into a state you may never have trained on by step 6. Hold that thought; in Chapter 3 it becomes the central difficulty of the entire field.
An open-loop controller decides the whole action sequence up front from the initial observation and then executes blindly: a0, a1, … computed once. A closed-loop controller (also called feedback control) re-reads the observation before every action: at = π(ot). When the world is perfectly known and never disturbed, the two are identical. Add one gust of wind and they diverge completely.
Both controllers want the cart (dot) to sit on the dashed target. Wind kicks it randomly. The open-loop plan was computed once at t=0; the closed-loop policy re-measures and pushes back. Crank the wind and watch one of them fall apart.
Here is the single most important idea in this entire introduction, and it is the reason you cannot just throw supervised learning at a robot and call it done.
Suppose we teach a robot to drive by imitation: we record an expert driver and train a policy π to copy the expert's action at each state. On the states the expert visited, our policy is good — say it picks the wrong action only with a small probability ε (epsilon), the per-step error rate. Over a trajectory of H steps, how bad can it get?
The naive answer — the answer supervised learning gives you — is ε·H. If you err 5% of the time over 100 steps, expect about 5 mistakes. That answer is wrong, and dangerously optimistic.
Ross and Bagnell (2010) made this precise. For plain imitation (behavior cloning), the worst-case extra cost grows like ε·H² — quadratic in the horizon, not linear. Double the episode length and you quadruple the trouble.
Take ε = 0.05 and H = 100. The naive linear guess says ε·H = 5 bad steps. But once the policy first slips off-distribution — which happens on average after about 1/ε = 20 steps — it tends to stay lost for the remaining ~80 steps. The expected number of "lost" steps is therefore far closer to 80 than to 5. The error did not add up; it compounded.
The teal line is the expert's demonstrated path. The warm dot is our cloned policy. Raise the per-step error and press Run: once it leaves the demonstrated band (shaded), it has never seen those states and spirals away.
Now let's measure the quadratic ourselves rather than take it on faith. The lab below runs thousands of cloned-policy rollouts and plots cost against horizon. Your job is to encode the one fact that makes the curve bend upward: once you make a mistake, you stay lost.
GPT works for a reason worth stating plainly. The instructor lists the recipe: architecture = transformer; data = web text, books, wikis; loss = next-token prediction; optimization = SGD; generation = autoregressive. Four of those five transfer to robots almost unchanged. The one that does not is data — and it breaks everything.
Language models drink from a firehose: trillions of tokens, already written, free to scrape, perfectly labeled (the next word is the label). There is no equivalent firehose of robot experience. Every datum of robot data has to be acted out — on a real robot in real time, or in a simulator that you then have to trust. It is slow, expensive, dangerous to collect, and it goes stale the moment the robot or task changes.
Even compared to RL on games like Atari or Go, the physical world is a different beast. The instructor's contrast list:
| Game RL (AlphaGo, DQN) | Robot learning |
|---|---|
| Known & static rules | Unknown & dynamic world |
| One specific task | Many tasks, many environments |
| Clear reward (win / score) | Unclear, hard-to-specify goals |
| Offline / batch learning is enough | Requires online adaptation |
| Slow, untimed moves | Fast real-time action (e.g. 50 Hz) |
| Failures are free (just restart) | Failures break hardware — physics doesn't forgive |
| Discrete, digital world | Continuous physical world |
Look at the "50 Hz" and "failures break hardware" rows together and you see the bind: the robot must decide twenty times a second, forever, and a single bad decision can snap a leg. You cannot brute-force a billion episodes the way AlphaGo self-played millions of games — each episode costs real seconds and risks real damage. Sample efficiency is not a nicety in robotics; it is the whole game.
A learner needs some number of environment steps. At a fixed control rate, how much wall-clock robot time is that — and how does it compare to an LLM's token feast?
Throughout the lecture one diagram keeps reappearing: four blocks labelled Algorithm, Data, Computation, Hardware. It is the instructor's mental model of what builds embodied intelligence, and the punchline is a single sentence: data increases the "upper bound" of algorithm, computation, and hardware.
What does "upper bound" mean? Any fixed, hand-designed system has a ceiling on how well it can ever do, set by the assumptions baked into it. The instructor lists where those assumptions crack:
Learning from data attacks all five: instead of assuming the model, you fit it; instead of one tuning, you adapt; instead of a fixed structure, a deep network expands the policy class. Data doesn't replace good algorithms, computation, or hardware — it raises how far each of them can take you.
There's a second, related point hiding in that diagram. Classical robotics separately designs each module: a perception block feeds a planning block feeds a control block, each built and tuned in isolation. That's clean, but errors at one interface get amplified by the next, and no module knows the others' weaknesses. Learning lets you train modules together — even end-to-end from pixels to torques — so they co-adapt to cover each other's blind spots.
The dashed line is the ceiling a system can reach. More data lifts the ceiling. Toggle between a modular pipeline (errors compound across interfaces, lower ceiling) and an end-to-end learned system (co-adapted, higher ceiling).
Before celebrating learning, the instructor shows a humbling gallery of robots doing spectacular things with no learning at all. This matters: it tells you exactly when learning earns its keep and when it is solving a problem classical methods already nailed.
| System | Method (no learning) |
|---|---|
| Apollo 11 guidance | Optimal control + robust control |
| Boston Dynamics Atlas humanoid | Trajectory optimization + Model Predictive Control (MPC) |
| Offroad autonomy | Sampling-based MPC |
| High-speed robot hand (Ishikawa–Komuro, 2009) | Fast vision + classical feedback |
| Tailsitter aerobatics (MIT LIDS) | Trajectory generation + nonlinear control |
The thread connecting all five: when you have a good model of the dynamics, classical control is breathtaking. Apollo flew people to the Moon on optimal control. Atlas does backflips on trajectory optimization and MPC — methods you will meet in Lectures 12–14. The robot hand catches a falling ball because its vision and feedback loop run faster than the ball can fall.
To feel why model quality dominates classical control, play with a controller that plans using an internal model of the system — and dial up how wrong that model is. With a perfect model it's flawless. With a 30% mismatch it staggers. That mismatch is precisely the gap learning and adaptation are built to close.
A model-based controller drives the cart to the target using its belief about the system gain. Slide the model error up and watch tracking degrade — the controller is confidently solving the wrong problem.
From here on, the course needs a precise vocabulary for "sequential decisions." That vocabulary is the Markov Decision Process (MDP). You'll spend whole lectures on it later (Lectures 5 and 7); right now we just install the words, because you'll hear them constantly.
| Symbol | Name | Plain meaning |
|---|---|---|
| s | State | The situation of the world right now. |
| a | Action | What the robot does. |
| r(s,a) | Reward | A number scoring how good that action was here. Bigger is better. |
| P(s′|s,a) | Transition | The world's dynamics: where you land next. |
| π(a|s) | Policy | The robot's decision rule. What we learn. |
| γ | Discount | A number in [0,1) saying how much the future matters vs. now. |
The robot's goal is to maximize its return — the total reward over time, with future rewards discounted by γ at each step:
Why discount? Two reasons. Mathematically, γ < 1 keeps an infinite sum finite. Intuitively, it encodes "a reward now beats the same reward later" — the same reason a dollar today beats a dollar next year.
Picture a 4-step corridor: each move costs −0.1 (effort), and arriving at the goal pays +1.0. Under a "always go right" policy the reward sequence is r = (−0.1, −0.1, −0.1, +1.0). With γ = 0.9:
Notice the goal reward of +1.0, four steps away, is only "worth" 0.729 today — that's 0.9³. The effort costs near the start bite at nearly full value; the payoff at the end is discounted. That tension — pay now, rewarded later — is the heart of every sequential decision. Let's compute it in code, then you fix the discounting.
A 5-cell corridor. Slide the discount and watch the return of the "go right to the goal" policy change — a far-sighted agent (γ→1) values the goal; a myopic one (γ→0) sees only the next step's cost.
Time to put every idea in one place. This robot lives in an unknown, windy world and must reach a sequence of goals. You hold the four dials that define the whole field:
Press Run. The robot chases goals (teal dots). Break it: open the loop, crank noise and wind, and watch the success rate collapse — then switch adaptation on and watch it recover.
The most instructive combination: max wind with adaptation off (the robot fights a current it can't see), then the same wind with adaptation on (it learns the current and leans into it). That switch — from blind to adaptive — is the difference between a robot that works in the lab and one that works in the world. Let's build the adaptation step itself.
You now own the conceptual spine. Everything in 16-831 is an answer to one of the two hard problems you met: compounding error (Chapter 3) and where the data comes from (Chapter 4). Here is how the eight topic groups hang on that spine.
This lesson is the front door to the 16-831 series. Continue with Lecture 2 — Robot Learning: An Overview. To go deeper on individual ideas, these companion Gleams expand each thread:
"The robot must act in a world that acts back."