Behavior cloning drifts off the road and the cost compounds. Three ideas fix it: put the expert back in the loop, match whole distributions instead of single steps, and learn to copy demonstrations that branch.
Last lecture we taught a robot to drive by behavior cloning (BC): record an expert, then train a policy to copy the expert's action at every state, exactly like a supervised classifier. On the states the expert visited, the clone is good. And yet on the open road it veers off and crashes. Why?
Here is the villain again, in one sentence. The first time the clone picks a slightly wrong action, it lands in a state the expert never visited — a little too close to the curb. Nothing in the training data says how to recover from there, because the expert was never there. So the clone makes another mistake, drifts further into unseen territory, and snowballs. This is distribution shift (also called covariate shift): the states the policy actually visits at test time stop matching the states it was trained on.
So the question for this whole lecture is blunt: how do we fix imitation learning so the error stops compounding? Today gives three answers, each a different philosophy.
The teal band is where the expert demonstrated. Behavior cloning is fine until its first slip, then it spirals through states it never saw. Toggle DAgger on: the expert has labeled those recovery states, so the policy steers back. Raise the per-step error and run again.
Behavior cloning has one fatal asymmetry. It trains on the expert's states but is tested on the learner's states — and those two sets diverge the moment the learner slips. DAgger (Dataset Aggregation, Ross, Gordon & Bagnell, 2011) fixes the asymmetry head-on: collect training data from the states the learner actually visits, and have the expert say what it would have done there.
The trick is one short loop. Read it slowly — every method today is a variation on putting the right data in front of the policy.
Notice what each actor does. The learner generates the states (it decides where the trajectory goes); the expert generates the labels (it decides the correct action there). That division of labor is the whole idea. BC let the expert pick both the states and the labels — which is exactly why the learner's states went unseen.
Picture a 100-step drive. Iteration 1's policy slips at step 20 and ends up hugging the curb for steps 20–100. DAgger feeds all 80 of those curb-states to the expert, who labels each with "steer left, hard." Those 80 labeled pairs join D. Iteration 2's policy, trained on D, now knows how to steer back from the curb — so it slips less, and the states it visits look more and more like the expert's. The training distribution and the test distribution converge. That convergence is the cure for compounding error, and we make it precise next chapter.
BC's cost grows like ε·H². DAgger's grows like ε·H — linear. That single change of exponent is the entire payoff, so let us see why it happens, with no heavy math.
The quadratic in BC came from a mismatch: the policy was trained on the expert's state distribution but evaluated on its own. The bound has a worst-case factor that scales with the horizon because once you're off-distribution you can stay lost for the rest of the episode. DAgger removes the mismatch by training on its own induced distribution of states — the very thing it is evaluated on. When the training distribution equals the test distribution, there is no covariate shift left to amplify, and the cost collapses back to linear.
The simulation below races the two bounds as you stretch the horizon. BC's curve bends upward (quadratic); DAgger's stays a straight line (linear). At short horizons they're close. Stretch H and the gap explodes — which is exactly the regime real robots live in, where an episode is hundreds or thousands of control steps.
Both policies err with the same per-step rate ε. The warm curve is behavior cloning (stays lost after a slip → cost ~ε·H²); the teal line is DAgger (recovers → cost ~ε·H). Slide ε and the horizon to watch the gap.
Now let's measure the DAgger improvement ourselves instead of trusting the curve. The lab simulates DAgger iterations: each iteration rolls out the current policy, the expert relabels the visited states, and we retrain. As recovery data accumulates, the policy's drift on a held-out test rollout should fall — far below the BC baseline. Your one line is the aggregation step that makes the data grow.
DAgger fixes where the data comes from. But there is a second, sneakier failure of behavior cloning that more data alone cannot fix — and the lecture spends real time on it, because solving it is what motivates GAIL and Diffusion Policy.
Human demonstrators are multimodal: faced with the same scene, they don't always do the same thing. Two drivers reach a rock in the road; one swerves left, one swerves right. Both are perfect demonstrations. Now train a regression policy — a network that outputs a single action and is fit by mean-squared error to copy the demos. What does it learn at the rock?
This is why the lecture says behavior cloning needs powerful models, not just more data. The instructor lists two upgrades that go beyond a plain feed-forward regressor:
The expert demonstrated swerving left and swerving right around the obstacle — both valid. The regression (mean) policy averages them and drives straight into the rock. Toggle to a multimodal policy: it samples one mode and goes around. Press Run to send a car down each.
Let's nail the failure in code so it isn't hand-waving. The lab fits a single mean to a bimodal set of demonstrated actions and measures how badly the mean misses — then compares it to a policy that picks the nearest mode.
Before the generative models, the lecture offers a clever detour. Sometimes the hard part isn't where the data comes from — it's that learning directly from raw sensors is brutally hard because the observation is high-dimensional (camera pixels, lidar). The idea: don't imitate a human; imitate a machine that knows more than the final robot ever will.
This is the privileged teacher (or teacher–student) recipe, and it comes in two stages:
| System | Teacher sees (privileged) | Student sees |
|---|---|---|
| Autonomous driving (CARLA) | Ground-truth lane, traffic-light state, other vehicles' positions/velocities | Front-camera pixels |
| Legged locomotion (RMA, in sim) | Terrain, friction, contact forces, applied disturbances | Proprioceptive history (IMU, joint angles) |
| Agile drone flight | An MPC controller that knows the full state | Onboard sensing |
One subtlety the lecture flags: the student doesn't have to learn in the teacher's action space. In RMA (Rapid Motor Adaptation) the student predicts the teacher's compressed latent — an encoding of the privileged info — rather than raw actions, which is easier to regress from a history of proprioception. The principle stands: an easy-to-train teacher manufactures clean supervision for a hard-to-train student.
A robot must hold the center line under an unknown wind. The privileged teacher is handed the true wind and cancels it instantly. The student sees only its recent positions and must infer the wind from how it has been pushed — lagging at first, then catching up. Slide the wind and run.
DAgger and privileged teachers both lean on a queryable expert. GAIL (Generative Adversarial Imitation Learning, Ho & Ermon, 2016) throws that crutch away. The setting: you have a fixed batch of expert trajectories, you cannot query the expert for more, and you are not given any reward function. Just demonstrations. How do you learn a policy?
The shift in viewpoint is the whole lesson. BC matched the expert's action at each state, one timestep at a time — which is what made it fragile to compounding error. GAIL instead matches the distribution of state–action pairs the policy produces to the distribution the expert produces. If the two distributions are indistinguishable, the policy behaves like the expert — trajectory-wide, not just per-step.
The discriminator D is a classifier. It takes a single state–action pair (s, a) — the data point GAIL works with is the pair, written z = (s, a) on the slides — and outputs one number in [0, 1]: its estimate of the probability that this pair came from the expert rather than from the policy. Near 1 means "I think the expert did this"; near 0 means "I think the policy did this."
You train D by ordinary binary classification: feed it expert pairs labeled 1 and policy pairs labeled 0, and minimize cross-entropy. A good D learns a decision boundary that separates expert behavior from policy behavior in (state, action) space.
Each dot is a state–action pair. Teal = expert, warm = current policy. The shaded line is the discriminator's learned boundary (it scores everything to one side as "expert"). Press Train D to refit the boundary, then Policy step to nudge the policy's pairs toward the expert cluster — fooling D. Repeat: the two clusters merge.
Now the data flow. GAIL repeats three steps until the discriminator can no longer tell expert from policy. The instructor's exact phrasing: sample from the student, update the discriminator to classify teacher vs. student, then train the student to minimize the discriminator's accuracy.
Step 3 is the part that surprises people, so let's be precise about where the policy's gradient comes from. There is no expert reward and no per-step imitation label. Instead, GAIL hands the policy a synthetic reward built from the discriminator's score: a pair that D thinks is expert pays well; a pair D recognizes as policy pays poorly. The policy is then optimized by reinforcement learning on that synthetic reward — not by supervised regression to actions. That is the deep difference from BC and even DAgger.
The lecture asks: can GAIL model multimodal behavior? Yes — like a GAN, the policy is a sampler from a distribution, so it can put mass on both "swerve left" and "swerve right" as long as both look expert to D. That is a genuine advantage over mean-regression BC. The price is adversarial training's notorious instability: the discriminator and policy are locked in a minimax game, and like GANs, GAIL can oscillate, collapse modes, or need careful tuning to converge. There is no free lunch — you trade BC's brittleness for an optimization that is harder to stabilize.
Let's build the reward signal itself: a tiny logistic discriminator that scores expert vs. policy samples, and the reward the policy would climb.
GAIL captures multimodal behavior but is hard to train. Is there a way to get the multimodality without the adversarial game? Yes — and it is the hottest idea in modern imitation learning: Diffusion Policy. It takes the diffusion model from Lecture 4 (the engine behind image generators) and points it at actions instead of pixels.
The recipe matches the multimodal upgrade from Chapter 3: replace the single-action output with a generative model of the action distribution. Diffusion is just an unusually good generative model, because it can represent sharp, multi-peaked distributions that a Gaussian or even a Gaussian mixture struggle with.
The two-process picture, stated for actions:
Two engineering choices the lecture and the Diffusion Policy paper stress. First, diffusion policies usually predict a short sequence (chunk) of future actions, not a single action — the same action chunking idea behind ACT (Action Chunking with Transformers, built on a conditional VAE). Predicting a chunk keeps the motion coherent and reduces the dithering that comes from re-deciding every single step. Second, the denoiser is conditioned on the observation, so the generated actions are appropriate to what the robot currently sees.
The curve is the demonstrated action distribution: two peaks (left swerve, right swerve). Press Sample to drop noisy candidate actions (warm) at random and watch them denoise — slide downhill into one peak or the other. The blue line is where mean-regression would put its single action: stuck in the dead valley between the peaks.
| Generative model + IL | How it represents the action distribution | Note |
|---|---|---|
| Gaussian mixture (GMM) | Predefined fixed number of Gaussian modes | Simple; must guess the mode count |
| Conditional VAE (ACT) | Latent variable decoded to an action chunk | Action chunking + temporal ensemble |
| GAN (GAIL) | Adversarial sampler | Multimodal but unstable to train |
| Diffusion (Diffusion Policy) | Iterative denoising of an action chunk | Sharp multimodality, stable training |
Time to put all three fixes side by side on the same road. The car must drive a winding lane that splits around an obstacle (a multimodal demo) under per-step noise. You choose the method and watch what breaks and what survives.
Pick a method, set the per-step noise, and press Run. The teal band is the expert lane; it forks around the red obstacle. Watch BC drift and crash, DAgger recover but average into the rock, and the multimodal policy thread one fork cleanly.
The instructive sequence: run BC at noise 0.08 (it drifts and hits the rock), switch to DAgger (it stops drifting but still aims at the rock because it averages the fork), then Diffusion (it both stays on the lane and picks a fork). The three failures and fixes of this whole lecture, in one screen: drift → DAgger; averaging → generative; both at once → a modern diffusion policy.
You arrived with one broken algorithm — behavior cloning — and leave with three repairs and the judgment to choose between them. Here is the whole lecture on one card.
| Situation | Best tool |
|---|---|
| You can query the expert in any state | DAgger (directly cures drift) |
| You have a simulator with ground-truth state | Privileged teacher → student distillation |
| Fixed demos, no expert access, no reward | GAIL (distribution matching) |
| Demos branch (multiple valid actions) | Diffusion Policy / conditional VAE (ACT) |
| Demonstrators depend on recent history | Sequence model over an observation window |
This lesson is Lecture 6 of the CMU 16-831 series. Continue along the spine, and branch out to the companion Gleams that expand each thread:
"Don't train on where the expert went — train on where you went, labeled by the expert."