A quadruped trained only in simulation must walk on ice, mud, and stairs it never saw — and recover its gait within a single step. How? Train a teacher that can peek at the true physics, then distill it into a student that infers that physics from the last fraction of a second of motion.
You train a quadruped to walk. Every episode happens inside a physics simulator, on a desktop, overnight. The robot learns a beautiful trot. Then you carry it outside.
The pavement is fine. Then it steps onto a patch of ice — the friction drops to a third of anything it saw in training. A step later it sinks into mud — the ground gives way under each foot. Then it hits a flight of stairs. None of these were in the simulator. A policy that froze its behavior at training time would slip, sink, and topple within one or two steps.
Here is the brutal constraint, and it is the whole reason this lecture exists. The robot cannot stop, run twenty experiments on the ice to "figure out the friction," and then resume. Physics doesn't forgive (Lecture 1): a single wrong step on ice is a fall, and a fall can break the hardware. Adaptation has to happen online, in fractions of a second, from the motion the robot has already produced — with no falls allowed.
Press Run. Both quadrupeds trotted perfectly on the training ground (firm, friction = 1.0). Now the terrain changes underfoot — ice, mud, firm ground — in bands. The teal robot runs an adaptive policy that senses the change and re-tunes its push; the red robot runs the frozen training policy. Slide how hard the terrain deviates from training.
The frozen policy isn't broken — it is solving the wrong problem. It pushes off the ice as if the ice were the firm training ground, so it slips. The adaptive policy notices "my last few steps moved me less than I expected; the ground must be slippery" and leans harder. That noticing-and-adjusting, done online from recent experience, is the entire subject of the lecture.
"Adaptation" is a slippery word, so the lecture splits it cleanly in two. Both are about a single policy coping with a situation that wasn't fixed at training time — but they differ in what changes.
The robot's body-plus-environment behaves differently than it did in training. Recall the MDP transition from Lecture 1: the next state is drawn from the world's dynamics given the current state and action. Dynamics-wise adaptation says the dynamics also depend on a hidden environment parameter — wind on a drone, friction under a foot, a payload bolted to a car. In symbols, the transition becomes:
Here et (the environment parameter) is the thing that was constant in training and is now different — and it might even drift over time (wind gusts). Crucially, et is unknown at test time: the robot has no sensor that reads "friction = 0.3." It must be inferred.
Here the physics is the same but you ask the robot to do something new. Task-wise adaptation makes the reward depend on a task parameter g — a target location, a trajectory to track, even a language instruction like "get the groceries." The reward becomes r(st, at, g). Unlike the hidden environment, g is usually known (you tell the robot the goal); the challenge is that the specific goal may be unseen in training. This is the world of goal-conditioned RL (the subject of the Multi-Task & Goal-Conditioned RL Gleam).
| Dynamics-wise | Task-wise | |
|---|---|---|
| What changes | The transition / physics | The reward / goal |
| Hidden variable | Environment param e (wind, friction, payload) | Task param g (goal, trajectory, language) |
| Known at test time? | No — must infer it | Usually yes — you tell it |
| Policy form | π(s, z), z = estimate of e | π(s, g) |
| Example | Drone in a wind tunnel; quadruped on ice | One humanoid policy tracking many motions |
The two can co-exist — a humanoid asked to track a new dance (task-wise) while carrying an unknown backpack (dynamics-wise). But because the hard, unforgiving case is the unknown physics, this lecture concentrates on dynamics-wise adaptation: inferring an environment parameter you cannot directly see.
Let us make the goal precise, because the rest of the lecture is just different ways to achieve it. We want two objects working together.
Read the two objects carefully, because the split is the whole idea:
Why split it this way instead of one giant network? Because the two pieces face completely different constraints. The policy is a slow, careful, RL-trained object — expensive to learn, never updated in the field. The adaptation law is a tiny, fast inference run every few milliseconds. Separating them lets each be exactly what it needs to be. This split — a frozen policy + a small online estimate — is the architecture you will see in three different disguises (adaptive control, RMA, Neural-Fly) before the lecture ends.
One more contrast worth installing now: zero-shot vs. few-shot adaptation. Zero-shot means the adaptation law needs no labeled data and no reward signal at deployment — it just reads the passive history of what already happened (RMA and Neural-Fly are essentially zero-shot in this sense). Few-shot means the robot is allowed a handful of labeled trials in the new setting before it must perform (classic meta-RL). Zero-shot is the holy grail for hardware, because trials cost falls.
Before any neural networks, control theory already solved a version of this problem, and it is the cleanest place to feel what "estimate the hidden parameter online" means. This is the same adaptive-control idea you met in Lecture 12 — we are now reading it as a template for everything that follows.
Take the simplest uncertain system the lecture writes down. A quantity s evolves under our control a plus an unknown constant disturbance e (think: a steady wind, a payload, a calibration error):
Our goal is to drive s to zero. The obvious thing is a proportional controller — push back in proportion to how far off you are: a = −2s. Substitute it in and the closed loop becomes ṡ = −2s + e. Where does it settle? At equilibrium ṡ = 0, so 0 = −2s + e, giving:
The proportional controller leaves a permanent steady-state offset — it fights the wind but never fully cancels it, because to hold any nonzero push it needs a nonzero error to push against. The disturbance wins a little, forever.
Put a number on it. Say the unknown wind is e = 0.6. The proportional controller settles at s = e/2 = 0.3 — almost a third of a unit off, permanently. Now watch a discretized integral controller wake up to it. Let the estimate (the running integral) start at 0 and nudge it each step by a small fraction of the current error. Step 1: the cart sits at the offset 0.3, so the estimate climbs to, say, 0.18. With more push, the offset shrinks to 0.21; the estimate climbs to 0.31. Next the offset is 0.14, estimate 0.40. The pattern is unmistakable: the offset marches toward 0 while the estimate marches toward the true wind 0.6. After enough steps the estimate is 0.6 and the cart sits exactly on the mark, the push perfectly cancelling the wind. No one told the controller the wind was 0.6 — it backed that number out of the history of how far off it kept being.
That is the entire skeleton of adaptive control, and of this whole lecture: a controller that holds an internal estimate of the hidden parameter, updated from the history of what it has observed, and uses that estimate to cancel the disturbance. The lecture names two flavors:
| Flavor | Idea | Does the estimate have to be "true"? |
|---|---|---|
| MIAC — model-identification adaptive control | Run system identification while the system runs; explicitly try to make the estimate converge to the real parameter. | Yes — aims for estimate → true e. More intuitive, but stability is delicate. |
| MRAC — model-reference adaptive control | Specify an ideal "reference" closed-loop behavior; adapt until the real system behaves like the reference. | No — getting e exactly right is neither necessary nor the point; matching the desired behavior is. |
Tuck the MRAC remark away — "the estimate need not be the true parameter, only good enough to produce the right action" — because it is the precise insight that lets RMA replace a physical parameter with an abstract learned code in Chapter 6.
Adaptive control hand-derives the adaptation law from a model. But what if you want the adaptation itself to be learned — learned in such a way that it generalizes to environments you've never seen? That is the promise of meta-learning, and it is the conceptual bridge between classical adaptive control and the learned teacher-student methods that follow.
Ordinary learning fits one set of parameters to one dataset. Meta-learning is "learning to learn": you train across many tasks so that, faced with a brand-new task, you can adapt to it from a tiny amount of data. The guiding principle, from Vinyals et al., is blunt and useful: "test and train conditions must match." If the robot will have to adapt from 5 seconds of experience at deployment, then train it by repeatedly forcing it to adapt from 5 seconds of experience. Practice the exact thing you'll be tested on.
Meta-learning needs an inner loop that turns "a little data from task i" into "the right parameters for task i." The lecture contrasts two styles:
| Style | How it adapts | Trade-off |
|---|---|---|
| Black-box adaptation | A neural network directly predicts the task-specific parameters from the task's small dataset — a learned function from data to weights. | Flexible, but "guessing" weights can be unreliable. |
| MAML (Model-Agnostic Meta-Learning) | Leverages gradient information: the task-specific parameters are one gradient-descent step from a meta-learned initialization. Meta-training finds an initialization that is one step away from good on every task. | More principled — uses the same optimizer at test time as it was trained for. |
Notice the RMA connection forming already. RMA's adaptation module is exactly a black-box adaptation: a network that maps recent history (the "small dataset" of this episode) directly to the extrinsics (the "task-specific parameters"). Meta-learning is the lens that explains why training such a module in simulation should generalize to real terrain.
When the "tasks" become full MDPs — each with its own dynamics — meta-learning becomes meta-RL: the inner adaptation is now an RL update, and you meta-train a policy that can adapt quickly to a new MDP. The lecture flags a particularly clean variant, model-based meta-RL: apply meta-learning to the learned dynamics model rather than the policy. You learn a representation of the dynamics that adapts fast, then plan or control on top of it — which is precisely the recipe behind the drone work that became Neural-Fly (learn a dynamics representation, adapt it online, control with it).
Adaptive control is beautiful when you can write the equations. But a quadruped on natural terrain has contact, deformable mud, slipping feet — physics you cannot model cleanly (Lecture 1's "modeling the world is hard"). So we replace the hand-derived adaptation law with a learned one. The trick that makes learning tractable is teacher-student training, also called privileged learning.
The idea sounds almost like cheating, and that's the point. Inside the simulator, you know everything — the exact friction of every patch, the payload mass, the motor strength, the terrain height. In the real world you know none of it. Teacher-student exploits this asymmetry in two stages.
The canonical example the lecture cites for driving is "Learning by Cheating". Stage 1 trains a privileged agent that sees the ground-truth state — the exact positions and velocities of every other car, the true traffic-light state. With that god's-eye view it learns to drive well. Stage 2 then trains a sensorimotor student that only sees the camera, supervised to copy the privileged agent's decisions.
Click the stages. Watch what each policy is allowed to see (its inputs) and notice that the student is trained to match the teacher's action using only onboard signals. The privileged channel (the dashed peek at true physics) exists in sim and is cut at deployment.
For locomotion the lecture is specific about what goes in each channel. The teacher's privileged information is "almost everything available in sim" — contact states, terrain geometry, friction, applied disturbances. The student is restricted to proprioceptive history only: the IMU and joint-angle readings the real A1 robot actually has. And the teacher here is trained with RL (PPO), not imitation — there is no human expert for a quadruped gait.
Let's actually distill. The lab below has a privileged teacher whose action is a clean function of the true environment parameter the student can't see. Your job is the student: fit a regression from the observable history to the teacher's action, then prove the student matches the teacher without ever reading the privileged parameter.
RMA (Kumar, Fu, Pathak & Malik, 2021) is the cleanest instance of the whole pattern, and it is what lets that quadruped from Chapter 0 walk on sand, mud, and stairs it never saw, adapting in fractions of a second. It is teacher-student with one crucial twist: the student does its matching in latent space, not action space.
RMA has exactly two pieces, and they are trained in two phases.
In simulation, take the privileged environment parameters et (mass, center of mass, friction, terrain height, motor strength) and pass them through an environment-factor encoder μ that squashes them into a small latent vector:
This zt, the extrinsics, is fed alongside the current state xt and previous action at−1 into the base policy π, which outputs the joint targets at. The encoder μ and the policy π are trained together with model-free RL (PPO). After Phase 1 you have a policy that walks beautifully — if you hand it the true extrinsics. You can't, in the real world. That's Phase 2's job.
Now freeze the base policy. Train an adaptation module φ that estimates the extrinsics without ever seeing e — using only the recent history of states and actions:
φ is trained by plain supervised regression in simulation: minimize the gap between its estimate ẑt and the true latent zt the encoder produced. Both quantities are available in sim, so this is just a fitting problem — no falls, no RL.
This is where MRAC's remark from Chapter 3 pays off. RMA does not try to recover the true friction or mass. It targets the encoder's latent z — a low-dimensional code whose only job is to make the policy act correctly. Several different physical situations that act the same can map to the same z (identifiability is no longer a worry), and the whole thing is optimized end-to-end for good actions, not for "true" numbers. We don't need a correct system ID; we need the right behavior.
To make "data flow" more than a slogan, here are the actual shapes that move through RMA, so you could implement it. The numbers are illustrative of the A1 setup.
Phase 2 changes only the source of z. Instead of e → μ → z, the adaptation module φ takes a sliding window of the last ~50 (state, action) pairs — a tensor of shape (50, ~42) — runs it through a 1-D convolution over time, and outputs ẑ of shape (8,). The training target is the Phase-1 latent z; the loss is just the squared distance between ẑ and z. Same 8-number interface, different input. That is why you can swap φ in for μ at deployment without touching the frozen policy: they produce the identical-shaped z, and the policy never knows the difference.
A variant the lecture flags: the student doesn't have to predict a latent. In DATT (Deep Adaptive Trajectory Tracking, 2023) the student predicts the unknown dynamics parameter e directly. RMA's choice (predict the latent z) and DATT's choice (predict e) are two points on the same spectrum — what matters is that a fast module estimates something from history that lets the frozen policy act right.
The phrase "infer the physics from recent motion" can sound mystical. It isn't. Let's do it with numbers on the simplest possible example, then code it, so you trust the mechanism completely.
A robot pushes its foot with a known command and watches how far it actually slides. The true relationship is: distance slid = command × (1 − friction). On firm ground (friction near 0) a push of 1.0 slides nearly 1.0; on ice (friction near 0) — wait, that's backwards on purpose. Let's instead model the cleaner, RMA-flavored relation the lecture uses: the actual motion is the commanded motion scaled by a hidden environment gain e, plus a little sensor noise. The robot does not know e; it must infer it.
Say the true hidden gain is e = 0.4 (a slippery world: each unit of command produces only 0.4 units of motion). The robot logs three recent steps:
| Step | Command a | Observed motion y (= 0.4·a + noise) |
|---|---|---|
| 1 | 1.0 | 0.42 |
| 2 | 2.0 | 0.78 |
| 3 | 3.0 | 1.23 |
How do we get the single best estimate of e from these three (command, motion) pairs? This is least squares: find the ê that minimizes the sum of squared prediction errors ∑(y − e·a)². For a single scale factor, calculus gives a clean closed form — the estimate is the dot product of commands-and-motions divided by the dot product of commands-with-themselves:
Plug in our three steps. The numerator is (1.0)(0.42) + (2.0)(0.78) + (3.0)(1.23) = 0.42 + 1.56 + 3.69 = 5.67. The denominator is 1² + 2² + 3² = 1 + 4 + 9 = 14. So:
From three noisy observations of its own motion, the robot recovered the hidden gain (true value 0.4) to within 1%. That is the adaptation module, stripped to its mathematical core: a least-squares read-out of the hidden physics from the recent history of what the robot did and what happened. A policy that now uses ê = 0.405 instead of assuming the nominal e = 1.0 will scale its commands correctly and stop slipping.
The dashed line is the unknown true gain e. Press Step to feed the estimator one more (command, motion) observation; the warm dot is its running least-squares estimate. Watch it lock onto the truth within a handful of steps — that's "fractions of a second" of motion. Slide the sensor noise up and see the estimate get jittery but stay centered.
Now make it real. The lab below builds the adaptation module as exactly this least-squares read-out, then proves that a policy using the estimate tracks a target far better than one that stubbornly assumes the nominal physics.
RMA estimates a latent code for a frozen policy. Neural-Fly (O'Connell, Shi et al., Science Robotics 2022) shows the same skeleton in a different and gorgeous form: it adapts the force model of a drone flying agile maneuvers through strong, gusting wind — achieving centimeter-level tracking where conventional controllers drift.
The problem: the aerodynamic force on a quadrotor in wind is a nasty unmodeled residual — turbulent, nonlinear, wind-dependent. You cannot write its equation. But Neural-Fly makes two observations that crack it open, and they are exactly the pretraining-plus-finetuning insight from the lecture's deep-learning detour:
Neural-Fly trains a neural network — using only ~12 minutes of flight data across 6 wind conditions — to be a wind-invariant basis. Call its output φ(x): a small set of basis functions of the drone's state x. The training algorithm (DAIML) uses a discriminator to force φ to be wind-independent, so that all the wind-specific information is pushed out into a separate set of coefficients. The residual aerodynamic force is then modeled as the basis mixed by linear coefficients:
Here φ(x) is the frozen, wind-invariant basis (the shared representation) and a is a short vector of wind-specific linear coefficients — the low-dimensional "knob." This is the same frozen-thing-plus-small-online-estimate split as RMA, just rearranged: the basis is frozen, the coefficients adapt.
In flight, the drone observes the actual residual force it experiences (the gap between commanded and measured acceleration). It then updates the coefficients a online — a composite adaptation law that is, at heart, a recursive least-squares fit of a to make φ(x)·a match the observed residual force. The basis never changes; only these few coefficients chase the wind. Because the basis is concise and fixed, the fit is fast and stable — fast enough to track a wind that is itself changing.
Why "composite"? The lecture's adaptation law blends two error signals into the coefficient update, and it pays to see why each matters:
Blending them is what makes the proof of exponential stability go through — the closed loop is provably well-behaved, which is why Neural-Fly can be trusted on hardware flying through narrow gates in a wind tunnel. Compare the magnitudes the lecture reports: against a strong nonlinear baseline controller, Neural-Fly cut tracking error by about two-thirds. The expressive part (the deep basis, ~hundreds of thousands of weights) was learned once from 12 minutes of flight; online, only a handful of coefficients move. That ratio — enormous frozen model, tiny fast estimate — is the entire economy of the method.
Build it. The lab below gives you a frozen learned basis and a true (unknown) wind that creates a residual force. Your job is the online coefficient update that makes the predicted residual cancel the real one — and you'll watch the tracking error fall as the coefficients lock on.
Time to put RMA's whole loop in one place. A quadruped trots across terrain whose hidden gain changes in bands. You control how severe the terrain gets, how fast the adaptation module estimates, and whether adaptation is on at all — and you watch the estimated gain chase the truth while the robot stays upright.
Press Run. The top strip shows the robot crossing colored terrain bands (each a different hidden gain). The bottom plot tracks the true gain (dashed) vs the adaptation module's estimate. Turn adaptation off to watch the estimate freeze at the nominal value and the robot's slip-error climb; turn it on and watch the estimate chase every band.
The most instructive combination: crank the terrain change rate to max with adaptation off — the robot's slip-error spikes on every new band because it keeps pushing as if the ground were firm. Then flip adaptation on: the estimate jumps to each new gain within a few steps and the slip-error collapses. That switch — from a frozen policy fighting an invisible world to an adaptive one that reads the world from its own motion — is the difference between a robot that works in the lab and one that works on a hiking trail.
You now own the spine of adaptive robot learning. Every method in this lecture is a different costume on one idea: train an expressive thing once that is a function of the hidden situation, then estimate the hidden situation online from recent experience. Here's how the pieces hang together.
This closes a loop the series opened in Lecture 1: that robot learning requires online adaptation because the physical world changes and physics doesn't forgive. It also extends the adaptive control from Lecture 12 — the integral term you met there is the simplest adaptation law in the family RMA and Neural-Fly generalize with learning.
"The robot cannot run experiments on the ice. It must read the ice from the way it is already moving."