A policy that is flawless in simulation can fall flat on its face the instant it touches a real robot. The gap between the two — and the surprisingly simple trick that closes it — is the whole story of this lecture.
You spend a week training a quadruped to walk. In simulation it is magnificent — it trots, it climbs stairs, it recovers from shoves, it never falls. You upload the exact same policy to the real robot, set it on the floor, and press go. It takes two steps, its legs splay, and it collapses.
Nothing about the policy changed. The neural network weights are byte-for-byte identical. What changed is everything around the policy: the floor has friction the simulator never modeled, the motors respond a few milliseconds late, the legs weigh slightly more than the CAD file said, and the camera image is grainier than any rendered frame. The policy is answering a question about a world that no longer exists.
This is the reality gap: the difference between the simulated world a policy was trained in and the real world it must run in. And sim2real is the name for the entire research effort to cross that gap — to take a policy born in simulation and make it work on hardware.
A policy was tuned for one exact friction value (the dashed line). Press Deploy and drag the real friction away from where it trained. The overfit policy degrades fast as reality drifts; later we will build the flat robust curve that does not.
Notice the shape of the failure. Right where the real friction matches the simulator (the dashed line), the overfit policy is perfect. It is only perfect there. Step a little off, and performance falls off a cliff. The policy did not learn to walk; it learned to walk on this exact floor.
If simulation introduces a whole new problem — the reality gap — why not just train on the real robot and skip it? Because of the villain from Lecture 1: the data problem. Reinforcement learning is staggeringly sample-inefficient. A walking policy can need ten million environment steps to converge. At 50 Hz on one real robot, that is over fifty days of nonstop, supervised, breakage-prone operation — and that is before the first thing you want to change.
Simulation manufactures the data firehose that robots otherwise lack. The instructor lists four advantages, and each one is a direct rebuttal to a reason real-robot training is painful:
| Advantage of simulated data | What it buys you |
|---|---|
| Cheap, fast, scalable | Thousands of environments run in parallel on one GPU. A quadrotor policy has been trained in 18 seconds on a laptop; IsaacGym runs thousands of robots at once. |
| Safe | RL explores by trying random, often catastrophic actions. In sim a crash costs nothing; on hardware it snaps a leg. |
| Labeled (free oracle) | The simulator knows the ground-truth state — exact contact forces, terrain height, every object's pose. In the real world those labels are expensive or impossible to get. |
| No wear and tear | A real robot degrades with every rollout. A simulated one is brand new every episode, forever. |
That third advantage — the free oracle — is quietly the most powerful. In simulation you can read off quantities that no real sensor can ever measure: the precise friction coefficient under each foot, the wind vector hitting a drone, the mass of a payload. Hold onto that fact. In Chapter 6 it becomes the key that unlocks one of the most successful sim2real recipes there is — the privileged teacher.
Slide how many environment steps a policy needs. On a real robot that is wall-clock days of risky operation; in a massively-parallel simulator it is minutes.
"Sim2real is never easy," the lecture warns, "because it is hard to exactly replicate the real world." But how the replication fails matters, because the two failure modes call for different fixes. The instructor splits the gap cleanly in two, using the language of the model-based RL lectures, where a simulator is just the world's transition function: the next state is produced by feeding the current state, the action, and some environment parameters through the dynamics.
A parametric mismatch is the friendlier of the two. The simulator's structure is correct — it is the standard rigid-body dynamics — but a coefficient is wrong. The robot's mass in the CAD file was 2.0 kg; the real one with its cables and battery is 2.3 kg. Friction varies with the floor. These are numbers, and numbers can be randomized over (Chapter 3) or fitted (Chapter 5).
A non-parametric mismatch is deeper. The simulator simply does not model an effect. A drone's simple thrust model knows nothing about the swirling aerodynamics in a tight turn (the problem Neural-Fly tackles); a rigid-body sim has no notion of a deformable tire's contact patch. You cannot fix a missing effect by changing a parameter — the term for it is not even in the equations.
Click each real-world effect to classify it. A wrong number in the right equations is parametric; a missing effect is non-parametric.
Here is the idea that reshaped sim2real, and it is almost insultingly simple. Instead of training the policy in one carefully-tuned simulator and hoping it matches reality, you randomize the simulator. Every episode, you roll a fresh world: a new mass, a new friction, a new motor delay, new textures, new lighting. The policy is forced to succeed across thousands of different worlds.
Formally, recall the dynamics carry an environment parameter alongside the state and action. Domain randomization samples that parameter freshly each episode and trains a single policy that works for the whole range. The original 2017 paper put it perfectly:
The original paper applied this to perception: they trained an object detector on simulated images with absurd, non-realistic random textures, lighting, and camera angles — and it transferred to real photos accurately to 1.5 cm, with no real images in training at all. But the idea generalizes far beyond pixels. Today people randomize every part of the pipeline: dynamics (mass, inertia, friction), sensors (noise, bias, dropout), actuation (latency, gain), and visuals (textures, lighting). What you randomize is whatever you think might differ between sim and real.
The instructor frames domain randomization as an approximation of robust control. Robust control's goal is a controller that performs well not for one system, but for a whole set of possible systems — the worst case is bounded. Domain randomization achieves the same thing by brute force: rather than prove a worst-case bound mathematically, it simply trains on the whole set and lets the policy discover behaviors that survive all of them. The price is a real one, and Chapter 4 is entirely about it: a policy that must work everywhere is rarely the best policy anywhere.
Let us make the central claim concrete. We will train two policies — one on a single fixed friction, one over a randomized range — and measure each on a set of held-out "real" frictions it never saw. The randomized one should win on the worst case. You will write the one line that turns single-value training into range training.
Domain randomization sounds like a free lunch: randomize everything, get robustness. It is not. There is a fundamental tension hiding in the width of the randomization range, and getting it wrong fails in two opposite ways.
Too narrow, and you are back to overfitting. If you randomize friction only between 0.95 and 1.05 but the real floor is 1.4, the policy never trained near reality and transfers badly — the same cliff from Chapter 0, just slightly wider.
Too wide, and a different problem appears: the policy may fail to train at all. If the mass can be anything from a feather to a boulder, no single set of weights can be good across that whole range. The optimal action in a feather-light world is the wrong action in a boulder-heavy one. The policy is asked to be excellent everywhere, gives up, and settles for being mediocre everywhere — or never converges.
Slide the randomization width. The real-world success curve is low when the range is too narrow (reality falls outside training) and falls again when it is too wide (the policy can't learn a single good behavior). Somewhere in the middle is the peak.
The lab below measures this curve directly. You will sweep the randomization width, train a policy at each width over that range, and score it on the true real gain. The curve you produce is hump-shaped — that hump is the entire reason "just randomize more" is bad advice.
Domain randomization makes the policy robust to the gap. System identification attacks the gap from the other side: it makes the simulator match the real robot. Instead of "be robust to any friction," it asks "what is the actual friction, measured from real data?"
The recipe is the inverse of simulation. In simulation you know the parameters and predict the motion. In system ID you observe the real motion and solve backward for the parameters that best explain it. You run the real robot, log how it moved, and fit the simulator's coefficients — mass, friction, motor gain — so that the same commands produce the same motion in sim as they did in reality.
Let us do the fit by hand. The real robot's motion came from a true (unknown) gain plus sensor noise. We have the commands and the noisy responses. Recover the gain by least squares — the same math that fits a line through scattered points.
Domain randomization builds a fixed policy that ignores which world it is in. But there is a smarter target. What if, instead of being robust to every world, the policy could figure out which world it is in and adapt to it on the fly? A policy that reads the environment and adjusts is an approximation of adaptive control — and it does not conflict with randomization; you can do both.
The trouble is obvious. To adapt to the environment parameter — the friction, the payload, the wind — the policy would need to know it. And in the real world that parameter is exactly what you cannot measure. The robot has no friction sensor. So how can a policy that depends on the unknown parameter ever run on hardware?
The answer is one of the most elegant recipes in robot learning, and it leans entirely on the "free oracle" from Chapter 1. It runs in three phases:
A nice degree of freedom: the student need not copy the teacher's actions. It can match the teacher in some other space. RMA has the student reproduce the teacher's compressed environment encoding in a latent space. Agile But Safe has the student predict safety-relevant rays. DATT (a drone controller) has the student predict the unknown dynamics directly. In every case the principle is identical: distill privileged knowledge into something the real robot can compute from its own senses.
Press Run. The world's hidden parameter (a wind, in blue) drifts over time. The teacher sees it directly and tracks perfectly. The student can't see it — it must infer it from its recent motion history, lagging slightly but catching up. A fixed (non-adaptive) policy just suffers the offset.
There is one more way to dodge the reality gap, and it is almost philosophical: don't learn the part of the system that differs most between sim and real.
The lowest-level dynamics — exactly how a foot's contact force translates into motion in the next millisecond — are precisely where simulators are least accurate. So instead of learning a low-level controller that commands raw motor torques (and inherits all that mismatch), you learn a higher-level policy that outputs abstract commands — a desired body velocity, a gait frequency — and hand those to a classical low-level controller that the physics actually supports.
This is the structure of CAJun, a continuous-jumping quadruped: a learned high-level RL policy decides the desired base velocity, gait timing, and a swing-leg residual, while a low-level Model Predictive Controller solves for the ground reaction forces. Because the learned part lives in the well-behaved abstract space and the messy contact physics is handled by classical control, CAJun needs little domain randomization, trains in about 20 minutes, jumps farther, and is more robust than an end-to-end policy.
None of these methods is required to fully close the gap on its own. A common final step is to fine-tune in the real world: take the sim-trained policy as a strong starting point and let it keep learning from a small amount of real data. Because the policy already nearly works, this needs orders of magnitude less real experience than training from scratch — the dangerous, sample-hungry early exploration happened safely in simulation. Sim gets you 95% of the way; a little real data finishes the job.
In 2023, a system called Swift (Kaufmann et al., Nature) did something no autonomous system had done: it beat human world champions in head-to-head first-person-view drone racing — a sport where pilots scream a quadrotor through gates at over 100 km/h on the ragged edge of control. The policy was trained almost entirely in simulation. It is the clearest case study of every idea in this lecture working together.
Why is racing the perfect crucible for sim2real? Because at the limits of flight, every source of the reality gap is screaming at once: the aerodynamics are violently non-parametric (turbulence, propeller wash), latency is fatal at those speeds, and the camera-based perception is noisy. A policy that overfit its simulator would be confidently wrong about the one moment that decides the race.
Swift's recipe is this lecture's checklist:
Play with the forces below. Crank the unmodeled aerodynamics (the non-parametric gap), then turn on randomization + residual correction + online adaptation and watch a drone that was missing gates start threading them.
Press Run. The drone (warm) chases gates (teal). Raise the unmodeled aerodynamics to widen the reality gap, then flip on the three tools and watch the gate-clear rate climb back up.
The instructive sequence: max aero with everything off (the drone fights an unseen force and clips gates), then add randomization (it stops overfitting and clears more), then add adaptation (it learns the force and threads them). That progression — from brittle to robust to adaptive — is sim2real in miniature, and it is how Swift flew.
You now own the conceptual map of sim2real. Every method in this lecture is an answer to a single question — how does a policy born in simulation survive the real world? — and they attack the reality gap from complementary directions.
| Method | What it does to the gap | Best at | Limit |
|---|---|---|---|
| Domain randomization | Makes the policy robust to it | Unknown / unmodelable parameters | Wide ranges blunt performance & hurt training |
| System ID | Shrinks it (tunes the sim) | Parametric mismatch (wrong numbers) | Can't add missing (non-parametric) physics |
| Learning to adapt | Senses & cancels it online | Slowly-varying hidden parameters | Needs a privileged-teacher / student pipeline |
| Higher-level policies | Avoids exposing the learned part to it | Systems with a good classical low-level controller | Requires that trustworthy low-level controller |
| Real-world fine-tuning | Erases the residual gap with real data | Final polish on a near-working policy | Still costs (a little) real, risky data |
This is Lecture 21 of the CMU 16-831 series. The natural neighbors are Lecture 20 — Exploration (just before) and Lecture 22 — Safe RL (just after, where "physics doesn't forgive" gets its own lecture). To go deeper on the ideas here:
"With enough variability in the simulator, the real world may appear to the model as just another variation."