CS224R HW1 — Imitation Learning: BC, Flow Matching & DAgger

Chapter 0: The Wall

You have trained a bird to fly through gaps in pipes. It learned by watching an expert — thousands of frames of "here is what I saw, here is the height I aimed for." On the easy course, one gap per pipe, it is flawless: it threads every opening and never crashes.

Then you switch to the hard course. Now each pipe has two gaps, an upper one and a lower one, and the bird can take either. And your flawless policy — the same network, the same training, the same loss — flies straight into the solid wall between the two gaps. Every single time.

It is not confused. It is not undertrained. It is doing exactly what you asked it to do. That is the unsettling part, and unpacking it is the whole point of this homework.

The crash you cannot train away

Two valid demonstrations from the same state: one expert aims at the upper gap (y = 0.70), another aims at the lower gap (y = 0.30). Press play. Watch where a mean-seeking policy — one trained to minimize squared error — decides to fly.

upper gap y 0.70

lower gap y 0.30

The big reveal. The averaging policy does not pick a gap. It averages the two heights the expert used and aims for the point exactly between them — which is precisely where the wall is. Two perfectly good answers, blended into one catastrophic one.

This is not a bug in your code. It is a mathematical property of the loss function you chose. Fixing it is what launches the rest of imitation learning — and this lesson walks the whole road: how behavior cloning works, why the wall appears, and the two very different escapes (a richer model called flow matching, and a data trick called DAgger).

On the hard course, the expert sometimes aims at the upper gap and sometimes at the lower gap — from the very same observation. Before you know any of the math, what is the most likely reason a "predict the average action" policy crashes?

It did not see enough training data; more demonstrations would fix it. The average of "aim high" and "aim low" is "aim at the middle" — and the middle is solid wall, not a gap. The physics simulator is broken and the bird cannot actually reach either gap.

Where we are headed. Ten chapters. We build the setup (the bird, the observation, the paradigm), implement behavior cloning by hand, watch it fail on hard mode, then fix it two ways. There is a Forge Studio (the ⚒ button in the mode row) where you build every function the real HW1 asks for — the policy MLP, MSE, the flow-matching loss and sampler, the deterministic expert, and the DAgger relabel loop — on a live Flappy Bird instrument, and watch the bird stop crashing the moment your code goes green.

Chapter 1: The Bird & The Paradigm

Before we can fix anything we need to know the machine. The environment is a physics-based Flappy Bird. A bird falls under gravity. Pipes scroll left. Each pipe has one gap (easy) or two (hard). The bird must thread the gaps without touching a pipe, the ceiling, or the floor.

The action: a target height, not a flap

The agent does not press "flap." At every timestep it outputs a single number: a target y-position, normalized to [0, 1], where 0 is the top of the screen and 1 is the bottom. A built-in PD controller — a proportional-derivative feedback loop — converts that target into thrust, so the bird has momentum and cannot teleport. Aim for a height; the controller flies you there with realistic lag.

Why this matters for learning. Because the bird has momentum, a good policy must anticipate: to be centered on a gap when the pipe arrives, you have to start climbing several frames early. That lag is exactly why predicting a whole plan of future actions — Chapter 3 — beats predicting one action at a time.

The observation: four numbers

The policy sees a 4-dimensional vector, every entry normalized to roughly [0, 1]:

Index	Meaning	Note
`obs[0]`	distance to the next pipe	counts down as the pipe approaches
`obs[1]`	y-position of gap 1 (upper)	the two gaps are ~0.40 normalized (about 179 px) apart
`obs[2]`	y-position of gap 2 (lower)	on easy mode, equal to gap 1
`obs[3]`	the bird's current y-position	the bird's velocity is not observed

The hidden hard part is already here. On hard mode obs[1] ≠ obs[2] — there are two gaps — but nothing in the observation tells the policy which gap the expert will pick. Same four numbers, two valid answers. That is the seed of the whole multimodality problem.

Two difficulty modes

Mode	Pipes	Expert	Plain BC
Easy	one gap (`gap1 == gap2`)	always aims at the one gap → unimodal	works fine
Hard	alternating single- and double-gap pipes	randomly picks a gap → multimodal	fails spectacularly

Imitation learning vs reinforcement learning

Internalize this distinction before anything else. Imitation learning (IL) is not reinforcement learning (RL). They solve the same kind of problem — choose good actions — from completely different signals.

	Reinforcement Learning	Imitation Learning
Data source	trial and error in the environment	expert demonstrations — no env interaction while training
Signal	a reward function	the expert's actions, treated as labels
Algorithm family	Q-learning, policy gradients, …	supervised learning
Hard part	exploration, sparse reward, credit assignment	distribution shift, multimodal experts
Analogy	learn chess by self-play	learn chess by watching humans

The imitation-learning loop, in three boxes

1 · Collect

Run the expert. Log every (state, action) pair into a dataset D.

↓

2 · Fit

Train a network π_θ(s) to output the expert's action at each state — plain supervised learning.

↓

3 · Deploy

Run π_θ in the environment. No reward, no exploration, no Bellman equation.

Why start a robot-learning course here? Behavior cloning is the simplest baseline, and every advanced method (DAgger, GAIL, offline RL, RLHF) is built on top of it. It is also how most real robot policies are bootstrapped — Tesla Autopilot, OpenAI's dactyl, Stanford ALOHA all start from BC pretraining. And it exposes the deep questions — distribution shift, multimodality, inductive bias — that the rest of the course spends its time solving.

In this environment, what does the policy's action actually control?

A binary flap: 1 to thrust up, 0 to fall. The bird's velocity directly. A target y-height in [0,1]; a PD controller turns that target into thrust, so the bird has momentum.

Chapter 2: Behavior Cloning

Strip behavior cloning (BC) down to its essence and it is literally just supervised learning. If you have ever trained an image classifier, you have already done every mechanical step of BC. The only thing that changes is the data: states in, expert actions out.

The dataset

Run the expert in the environment N times. Record every (state, action) pair. You get

D = { (s₁, a₁^*), (s₂, a₂^*), …, (s_N, a_N^*) }

where each a_i^* is the height the expert aimed for at state s_i. In the real homework the default is roughly 500 demonstration episodes, which window out to several thousand training pairs.

The objective

Find the network parameters θ that make the policy's action match the expert's action, averaged over the whole dataset:

θ^* = arg min_θ (1/N) ∑_i L( π_θ(s_i), a_i^* )

Here L is a loss that measures how far off the prediction is. For Problem 1 of the homework, L is mean squared error. Problem 2 swaps in a generative loss (flow matching). Same overall structure — different L.

Concept → realization. The data flow is exactly a regressor's. Input tensor s of shape [B, 4]. Output tensor π_θ(s) of shape [B, 20] (twenty future target heights — Chapter 3 explains the 20). Target tensor a^* of shape [B, 20]. The loss reduces the two to a single scalar. Backprop, Adam step, repeat. Nothing exotic.

Why BC is not trivially perfect

The optimization is easy. The hardness lives in three failure modes — and each of the follow-up problems attacks one:

Failure mode	What goes wrong	Fixed by
Multimodal experts	two valid actions per state get averaged into an invalid one (the wall)	Problem 2 · flow matching or Problem 3 · deterministic expert
Distribution shift	small errors compound until the bird is in states the expert never visited	Problem 3 · DAgger
Causal confusion	the network learns a shortcut that works on training data but not in deployment	better data / representations (beyond HW1)

Regression fits the demonstrations — that is all it does

A single state, several expert demonstrations (dots), and the line a mean-squared-error fit draws through them. Drag the slider to add spread. On unimodal data the fit sits right on the cluster. Keep this picture — Chapter 5 shows what happens when the dots split into two clusters.

demo spread 0.04

What is behavior cloning, in one sentence?

Learning a reward function from the expert, then optimizing it with RL. Treating (state, expert-action) pairs as a supervised dataset and training a network to predict the action from the state. Running the policy and rewarding it whenever it matches the expert.

Chapter 3: Action Chunking

Most textbook BC predicts one action at a time: see a state, output an action, repeat. Modern robot-learning policies — Diffusion Policy, ALOHA, RT-1, π₀ — do something that looks strange at first: they predict a whole chunk of future actions in a single forward pass.

π_θ(s_t) = ( a_t, a_t+1, a_t+2, …, a_t+T−1 )

In this homework ACTION_CHUNK = 20: the policy predicts twenty future target heights at once. But it does not execute all twenty. At rollout time it executes only the first EXECUTE_STEPS = 10 before firing again. That "predict 20, execute 10, re-plan" pattern is called receding-horizon control.

Receding horizon: predict a plan, walk half of it, re-plan

The blue bar is the 20-step plan the policy just predicted. The solid segment is the 10 steps actually executed; the faded segment is discarded. Press play: watch the plan slide forward, always re-planning before the executed part runs out.

Why predict more than you execute?

Three concrete reasons, each a real engineering decision:

1 · Temporal consistency. A single-step policy can flip-flop — "up" this frame, "down" the next — because each decision is made fresh, independently, in the presence of noise. A chunk forces the network to commit to a coherent 20-step plan. On the hard course this is decisive: it lets the policy commit to one gap and route smoothly through it, instead of dithering between gaps every frame.

2 · Reduced compounding error. Every time you query the policy you roll the dice on a small error. Querying once every 10 steps instead of every step means roughly 10× fewer "rolls" per episode. If the per-query errors are roughly independent, fewer queries means lower accumulated error variance over the 1000-step episode.

3 · Multi-step reasoning. Predicting 20 steps ahead lets the policy plan around the geometry — begin the climb early (remember the momentum lag from Chapter 1), commit to a gap, thread it. A single-step policy has to rediscover that plan every frame, and gets tangled in noise.

The chunk-size tradeoff

Chunk vs execute	Effect
large chunk, execute few (T ≫ T_exec)	strong temporal consistency, but the policy commits to a stale plan for longer
chunk == execute	always responsive to fresh observations, but weaker consistency
predict 20, execute 10 (this HW)	the standard compromise — predict more than you execute, re-query before the buffer runs dry

How chunking shapes the network

This is the whole reason the policy's output has 20 dimensions. Input: a state of shape [B, 4]. Output: a chunk of shape [B, 20] — each output number is one future target height. The action_dim argument of BCPolicy defaults to 20 because that is the chunk size. Keep that number in mind; it is the shape everything downstream must match.

Predict 20, execute 10. The last 10 predicted actions are computed and then thrown away every re-plan. They are not wasted — asking the network to think 20 steps out is what makes the executed 10 coherent. It is cheaper to over-predict than to under-plan.

The policy predicts ACTION_CHUNK=20 actions but only executes EXECUTE_STEPS=10 before re-querying. What happens to the other 10?

They are executed after the first 10, so all 20 run before re-planning. They are computed but discarded — the policy re-plans after 10 steps, giving temporal consistency without committing to a stale plan for too long. They are averaged with the first 10 to smooth the trajectory.

Chapter 4: The MLP & The MSE Loss

Now the two design decisions that are Problem 1: the network architecture and the loss. Both are textbook — the interesting part (Chapter 5) is what they do together on bimodal data.

A three-layer MLP

The policy, BCPolicy, is a plain multilayer perceptron — the simplest neural network, a stack of matrix multiplies with nonlinearities between them:

state (4-D)

the observation [dist, gap1, gap2, bird_y]

↓

Linear(4 → 256) · ReLU

first hidden layer

↓

Linear(256 → 256) · ReLU

second hidden layer

↓

Linear(256 → 20) · Sigmoid

output: 20 target heights, each in (0, 1)

Why an MLP and nothing fancier? The problem is small — a 4-D input, a 20-D output, simple physics. A convolutional net would be overkill (no spatial structure). A transformer would be overkill (no sequential input structure). Two hidden layers of 256 units is the natural choice. (Compare: Problem 2's flow-matching policy is a 1D U-Net, because learning a vector field is a much richer function class.)

Why ReLU in the hidden layers? ReLU(x) = max(0, x). It is cheap, and — critically — it does not saturate for positive inputs, so gradients stay healthy (they are 1 or 0) no matter the magnitude. Sigmoid or tanh would flatten and vanish the gradient across layers.

Why Sigmoid on the output? The action is a target height in [0, 1]. The sigmoid σ(x) = 1 / (1 + e^−x) maps any real number into (0, 1), so the network can never emit an out-of-range height. Baking the constraint into the architecture gives a clean gradient signal everywhere, instead of a kink from clamping after the fact.

In PyTorch it is six lines. nn.Sequential(nn.Linear(4,256), nn.ReLU(), nn.Linear(256,256), nn.ReLU(), nn.Linear(256,20), nn.Sigmoid()). The one silent trap: forgetting super().__init__() — without it, PyTorch never registers the layer parameters, the optimizer updates nothing, and training "runs" while the loss never moves. The starter code includes it for you.

The MSE loss

Mean squared error is the average, over the dataset and over all 20 chunk dimensions, of the squared gap between prediction and expert action:

L_MSE(θ) = (1/N) ∑_i || π_θ(s_i) − a_i^* ||² = (1/N) ∑_i ∑_k=1..20 ( π_θ(s_i)[k] − a_i^*[k] )²

The one fact you must carry forward

What does MSE regression actually converge to? Take the expected squared error at a state and set the derivative with respect to the prediction p to zero:

d/dp E[ (p − a)² ] = 2 · E[ p − a ] = 2 · ( p − E[a] ) = 0 ⇒ p = E[a]

The MSE-minimizing prediction is the conditional mean of the expert's actions at that state:

π_θ^*(s) = E_{a^* ~ expert}[ a^* | s ]

Remember this. MSE gives you the average expert action at each state. For a unimodal expert (one gap, one answer) that average is exactly right. For a multimodal expert (two gaps, two answers) the average is the point between them — and that is the whole disaster of Chapter 5.

Why squared error and not absolute error?

Loss	Minimizer	Tradeoff
L2 · MSE · (p−a)²	conditional mean	smooth gradient; but averages multimodal targets
L1 · MAE · \|p−a\|	conditional median	robust to outliers; non-smooth at 0, harder for SGD

For smooth control tasks, MSE is the standard. Its smoothness is a gift for optimization — and its mean-seeking is a curse for multimodal data.

What does an MSE-trained regressor output, in expectation, at a given state?

The single most frequent (mode) expert action at that state. The median expert action at that state. The conditional mean — the average of all expert actions at that state.

Chapter 5: The Multimodality Trap

This is the conceptual climax of Problem 1. Everything before was setup; everything after is a fix. Spend time here.

The setup

Hard mode alternates single-gap and double-gap pipes. When the expert sees a double-gap pipe it hovers at the midpoint while far away, then — when it gets within a commit distance of the pipe — randomly picks one of the two gaps and aims there. Same observation, two different choices, decided by a coin flip inside the expert. So the dataset literally contains:

State s (a double-gap configuration), action = aim at the upper gap, y = 0.70.
State s (the same state), action = aim at the lower gap, y = 0.30.

Two valid expert actions, from an identical state, differing only by the expert's private coin flip. The observation carries no signal that would let any function distinguish them.

What MSE does with that

From Chapter 4: the MSE minimizer is the conditional mean. The conditional mean of 0.70 and 0.30 is

(0.70 + 0.30) / 2 = 0.50

So the trained policy outputs 0.50 at this state — which, because the two gaps are about 0.40 apart (~179 px) and each opening is only about 0.167 tall (75 px, i.e. half-opening ±0.084), is exactly the solid wall between them. The policy is not doing anything wrong. It is faithfully reproducing the conditional mean of the expert distribution. But the conditional mean is not a valid action.

Watch the mean walk into the wall as the demos split

Start with all demos aiming at one gap (unimodal). Drag the slider to send more and more of them to the other gap. The dashed line is the MSE optimum — the mean of the demos. Watch it leave the safe gap and drift into the wall as the data becomes bimodal.

fraction aiming lower gap 0.50

The fundamental issue, stated precisely

MSE assumes p(a | s) is unimodal. It assumes there is a single best action at each state. When the expert's behavior is genuinely multimodal — several valid actions per state — the mean of those modes can be arbitrarily far from any of them, and can be catastrophically bad. Here it is the worst point in the entire action space: the wall.

Why this is the lesson of HW1

Problem 1 is designed to fail on hard mode. You will run it, see a mean episode length far below the 1000-step cap, and the homework asks you to explain why in a couple of sentences. The answer is exactly: multimodality of the expert + mean-seeking of MSE.

And there are two clean escapes, which take orthogonal routes:

Fix	Idea	Changes
Problem 2 · Flow matching	make the model richer — learn the full distribution p(a \| s), then sample one mode	the model (data untouched)
Problem 3 · DAgger	make the data unimodal — a deterministic expert that always picks the same gap	the data (model untouched)

Both work. Real robot-learning systems often use both at once: a rich generative policy trained on the cleanest data available.

The writeup, in three sentences. "On hard mode the expert chooses between two valid actions (upper or lower gap) randomly at the same state, making the action distribution multimodal. The MSE loss converges to the conditional mean of expert actions, so the policy outputs the average of the two gaps — a target between them, where the wall is. This makes the bird crash, so the episode lengths are short."

If demo A aims at the upper gap (y=0.7) and demo B aims at the lower gap (y=0.3), both from the same state, what does the MSE-trained policy output there, and why is it a problem?

0.7, because the network memorizes the last demo it saw. 0.5, the mean of the two — which lands on the wall between the gaps, so the bird crashes. Either 0.7 or 0.3 at random, which is fine because both are valid.

Chapter 6: Flow Matching — fix the model

The first escape keeps the data exactly as-is and makes the model richer. Instead of a regressor that outputs one number per state, we train a generative model that learns the whole distribution of expert actions at each state — and then samples from it. Same state, different samples: sometimes the upper gap, sometimes the lower, never the average. This is flow matching, the tool of Problem 2. (For a gentler standalone tour see the site's Flow Matching Gleam.)

Regressor vs generative model

	Regressor (Problem 1)	Generative model (Problem 2)
Given a state, returns	one deterministic action	a sample from p(a \| s)
Same state twice	same action	possibly different actions
On bimodal data	the mean (the wall)	either mode — never the mean

The deep idea. A regressor outputs one answer per input. A generative model outputs a distribution per input, from which you sample. For unimodal targets the two are equivalent. For multimodal targets, only the generative model preserves the modes. Each Flappy Bird rollout samples once, commits to a gap, and routes smoothly through it.

The flow-matching idea: transport noise into data

Flow matching generates a sample from a complex distribution by continuously carrying a sample from a simple one — Gaussian noise — along a learned vector field. Picture the 20-D action space (think of it as a plane). At every point and time, an arrow says "flow this way." Follow the arrows from time τ=0 to τ=1 and a noise sample is carried to a data sample.

dx/dτ = v_θ(x, s, τ), x(0) ~ N(0, I), x(1) ~ p(a | s)

The magic is that different noise samples get carried to different modes. Noise near the "upper-gap attractor" flows to the upper gap; noise near the "lower-gap attractor" flows to the lower gap. That is how multimodality survives.

Training: regress the velocity along straight lines

The training objective is startlingly simple. Take a clean expert action a₁. Draw independent noise a₀ ~ N(0, I). Pick a random time τ ~ U(0,1) and interpolate along the straight line between them:

a_τ = τ · a₁ + (1 − τ) · a₀

Differentiate that straight line and the velocity is constant — it does not even depend on τ:

d a_τ / dτ = a₁ − a₀

So the loss is: regress the network's output toward that constant difference.

L_FM(θ) = E [ || v_θ(a_τ, s, τ) − (a₁ − a₀) ||² ]

It is still MSE — on the right thing. Plain BC does MSE on actions, which collapses to the mean. Flow matching does MSE on velocities at random interpolation points. Averaged over many noise/data pairs, the network learns E[a₁ − a₀ | a_τ=x] — a field that transports noise into the full data distribution, modes intact. Same loss class, opposite inductive bias.

Sampling: Euler integration

To generate an action for state s, start from noise and take small steps in the direction of the learned velocity. With num_steps = 20, the step size is h = 1/20 = 0.05:

x_τ+h = x_τ + h · v_θ(x_τ, s, τ)

Repeat 20 times from τ=0 to τ=1, then clamp the result to [0,1]. Because the training paths are straight lines, few steps are needed — that is flow matching's practical edge over diffusion (curved paths, many steps).

Noise splits into two gaps — the vector field at work

Twenty noise samples on the left (τ=0) integrate rightward under a learned field toward a bimodal target — upper gap (teal) and lower gap (blue). Press play. Notice: no sample ends in the wall between them. Each one commits.

The knife-edge case. Set num_steps = 1 and flow matching collapses back into plain MSE regression: a single Euler step from noise toward the conditional mean of the velocity gives you the conditional mean of the action — the wall again. The multimodality is only preserved because you integrate through many timesteps. Straightness is what lets 20 steps suffice.

Why does flow matching solve the multimodality problem that broke plain MSE?

It uses a bigger network, so it memorizes both gaps and averages them more accurately. It learns a vector field that carries different noise samples to different modes, so each rollout samples one gap instead of averaging — no point ever lands on the wall. It changes the loss to L1, whose minimizer is the median rather than the mean.

Chapter 7: DAgger — fix the data

The second escape keeps the model and the loss exactly as Problem 1 — plain MLP, plain MSE — and changes the data. It is the tool of Problem 3, and it fixes two things at once: the multimodality you just met, and a second, more fundamental failure of BC called distribution shift.

The distribution-shift problem

BC trains the policy to mimic the expert at the states the expert visits. But at deployment the policy visits the states its own behavior produces. It makes a small error, lands in a slightly weird state the expert never quite visited, makes a slightly larger error there, drifts further, and by fifty timesteps in it is in state space the expert never saw and the policy has no idea what to do.

Errors compound: the drift off the expert's states

The green band is the states the expert visited (the only states BC was trained on). Press play: the agent starts inside it, but each small error nudges it out, and the further out it goes the worse it acts — a runaway. Raise the per-step error to see the drift accelerate.

per-step error 0.020

It is fundamental, not a tuning issue. Ross & Bagnell (2010) proved BC's total error scales as O(T²·ε) in the episode horizon T. For T=1000 (this homework) even a 1% per-step error compounds toward 100%. You cannot train your way out with a smaller ε — the states themselves drift to regions absent from the training data.

DAgger: iteratively relabel the policy's own states

DAgger (Dataset Aggregation; Ross, Gordon, Bagnell 2011) closes the gap directly. Each round: roll out the current policy to collect the states it actually visits, ask the expert what to do at those states, add those (state, expert-action) pairs to the dataset, and retrain. Over rounds, the training distribution converges to the deployment distribution.

1 · Roll out

Run the current policy π_k; collect the states it visits.

↓

2 · Relabel

Query the expert at each visited state — expert actions, not policy actions.

↓

3 · Aggregate

Add the new pairs to the growing dataset D.

↓

4 · Retrain

Plain MSE BC on all of D. Repeat for 5 rounds.

The one-sentence insight. BC trains on the expert's state distribution; DAgger trains on the policy's own state distribution — with expert labels there. That is why its error is only O(T·ε) (linear) instead of O(T²·ε). Critically: the policy provides the states, the expert provides the labels. Swap those and you learn nothing — you would just be training the policy to keep doing what it already does.

The deterministic-expert trick

Here is the cleverest part of HW1. The original expert is multimodal — it randomly picks a gap. If we relabeled with it, we would keep adding multimodal labels, and MSE would keep averaging into the wall. DAgger would fix distribution shift but not multimodality.

So for relabeling we swap in a deterministic expert that always picks the same gap (gap 1, the upper one) when it commits:

Original expert (multimodal):

if dist < commit_dist:
    target_gap = np.random.choice([0, 1])  # coin flip!
    committed = True

Deterministic expert (unimodal):

if dist < commit_dist:
    committed = True
    raw_target = float(gap1_y)  # ALWAYS gap 1

Two birds, one line. That single change — raw_target = float(gap1_y) — turns multimodal data into unimodal data. Now every label says "gap 1, gap 1, gap 1," so the conditional mean is gap 1, a valid action. DAgger's aggregation fixes distribution shift; the deterministic expert fixes multimodality. Same MLP, same MSE as Problem 1 — just cleaner data. Which gap you pick is arbitrary; consistency is the point.

DAgger with action chunking

One wrinkle: the policy predicts 20-step chunks but executes 10 (receding horizon). During a rollout you query the expert at every step, storing a per-step list of (state, expert-action). Afterward you window that list into chunks: state s_t gets paired with the next 20 expert actions [a^*_t, …, a^*_t+19]. Any 20 consecutive expert actions form a valid expert chunk, so windowing makes new training pairs for free — the same pattern the original demo collector uses.

In DAgger, who provides the states and who provides the action labels — and why does it matter?

The expert provides both, so the data is always clean. The policy provides the states (rollout), the expert provides the labels — so the policy gets supervision exactly at the states where it makes errors. The policy provides both, so it learns to reproduce its own behavior.

Chapter 8: Showcase — Three Fixes, One Bird

Here is the payoff. Three methods — plain BC regression, flow matching, DAgger — attacking the same problem: a multimodal expert on a long-horizon control task. Pick a method, press play, and watch the bird fly the hard course. The one thing that changes across methods is where the policy decides to aim when it sees a double-gap pipe.

Fly the hard course — pick your fix

Buttons choose the policy. BC-MSE aims for the mean of the two gaps → the wall → crash. Flow samples one gap and commits. DAgger was retrained on deterministic-expert labels and always takes gap 1. Watch the "aiming at" readout and the crash counter.

What the three methods teach

Method	Hard-mode result	How it solves the problem
BC regression (P1)	~200–400 steps	doesn't — the baseline failure. MSE averages the modes into the wall.
Flow matching (P2)	~700–1000 steps	keep the data, enrich the model — a generative policy samples one mode per rollout.
DAgger (P3, final round)	~800–1000 steps	keep the model, fix the data — deterministic relabeling makes the training distribution unimodal and covers the deployment states.

Two orthogonal cures. Flow matching makes the model richer; DAgger makes the data cleaner. Real robot-learning systems often combine them — a diffusion or flow policy trained with DAgger-style relabeling. The three problems of HW1 are the three ideas you will keep meeting for the rest of robot learning.

The costs are not equal. Flow matching pays at inference (20 forward passes of Euler integration per action, versus 1 for the MLP). DAgger needs an interactive expert you can query at any state — cheap in simulation, expensive when the expert is a human teleoperator. BC is cheapest of all and simply wrong here. Pick your fix by your constraint.

Now build the whole homework yourself. The Showcase animates the outcome; the Forge Studio (the ⚒ button in the mode row) walks you through every function HW1 asks you to write — BCPolicy.forward, mse_loss, flow_matching_loss, the schedule’s interpolate + sample, DeterministicExpert.act, and rollout_and_relabel — on a live Flappy Bird stage. The bird crashes until your code goes green, then clears the gap. Finish the forge and you have implemented the assignment, at browser scale.

Chapter 9: Field Guide — Cheat Sheet & Self-Quiz

Everything worth carrying out of HW1, on one page. If you can reconstruct this from memory, you can teach it.

The equations

BC objective: θ^* = arg min_θ E_(s,a^*)~D [ || π_θ(s) − a^* ||² ]

MSE minimizer: π_θ^*(s) = E[ a^* | s ] (the conditional mean)

Flow training: a_τ = τ a₁ + (1−τ) a₀, target v = a₁ − a₀

Flow sampling (Euler): x_τ+h = x_τ + h · v_θ(x_τ, s, τ)

DAgger error: O(T·ε) vs BC error: O(T²·ε)

The three problems, one table

	P1 · BC-MSE	P2 · Flow Matching	P3 · DAgger
What you change	— (baseline)	the model	the data
Model	3-layer MLP	1-D U-Net (given)	3-layer MLP (same as P1)
Loss	MSE on actions	MSE on velocities	MSE on actions (same as P1)
Fixes multimodality?	no	yes (samples a mode)	yes (deterministic expert)
Fixes distribution shift?	no	no	yes (relabels visited states)
Inference cost	1 forward pass	20 (Euler)	1 forward pass
Needs interactive expert?	no	no	yes

The real starter-code contracts

You implement	File	In one line
`BCPolicy.__init__ / forward`	networks.py	the `nn.Sequential` MLP; `return self.net(state)`
`mse_loss`	losses.py	`F.mse_loss(policy(s), a)`
`FlowMatchingSchedule.interpolate`	networks.py	`x_t = tx1+(1-t)eps`; `v = x1-eps`
`FlowMatchingSchedule.sample`	networks.py	Euler loop over `num_steps`, then `clamp(0,1)`
`flow_matching_loss`	losses.py	interpolate → model → `mse_loss(v_pred, v_target)`
`DeterministicExpert.act`	dagger.py	`raw_target = float(gap1_y)` when committed
`rollout_episode` / `rollout_and_relabel`	dagger.py	roll out policy, relabel with expert, window into chunks

Named failure modes & their symptoms

Failure mode	Symptom you would actually see
Multimodal collapse	hard-mode episode length stuck ~300 while easy mode is ~1000; the bird hits the wall between two open gaps
Forgot `super().__init__()`	training loss never moves off its random-init value; optimizer has no parameters to update
Forgot the output Sigmoid	predictions leave [0,1]; unstable, random-looking performance even on easy mode
Wrong `action_dim`	outputs one number instead of a 20-chunk; the training loop crashes on a shape mismatch
Flow: querying model at τ+h	subtly off-distribution samples; performance drops with no crash
Flow: `num_steps = 1`	collapses back to MSE — the wall returns, multimodality lost
DAgger: labels from policy not expert	no improvement over rounds; the policy just reinforces its own mistakes
DAgger: forgot `obs.copy()`	every stored state is identical (the last one) — numpy aliasing bug

Self-quiz

If you set num_steps = 1 in the flow-matching sampler, what does the method reduce to?

A perfect one-shot sampler — faster and just as good. Plain MSE regression — one Euler step lands on the conditional mean, so multimodality is lost (the wall returns). A diffusion model with a curved path.

Suppose you ran DAgger but relabeled with the original (multimodal) expert instead of the deterministic one. Would it fix distribution shift? Would it fix multimodality?

It fixes both — DAgger always fixes everything. It fixes neither — you need flow matching for both. It fixes distribution shift (states still get aggregated) but NOT multimodality (the labels stay bimodal, so MSE still averages into the wall).

Why does BC's error scale as O(T²) while DAgger's is only O(T)?

Because DAgger uses a larger network that generalizes better. BC's deployment distribution drifts away from training, so per-step error is amplified by a deviation that itself grows with time. DAgger keeps the training distribution covering deployment, so per-step error stays bounded. Because DAgger executes shorter episodes.

Take it back to class

If a friend asks "Why does behavior cloning fail on hard mode?" — you say:

"It's not BC's fault — it's the loss. MSE regression converges to the conditional mean of the expert's actions. When the expert is bimodal — sometimes go up, sometimes go down — the mean is in the middle, which is a wall. The fix is either a richer model (flow matching captures both modes) or cleaner data (a deterministic expert that always picks the same gap, which DAgger relabels with while also curing distribution shift)."

Where this connects

This forge is the entry point to the CS224R arc. Related on the site:

Imitation Learning — the standalone Gleam on BC, DAgger, and inverse RL.
Flow Matching — the generative model in full, from scratch.
CS224R · RLHF & DPO and RL for VLAs — where imitation meets reinforcement.
Offline RL — HW3's territory: an expert dataset plus value learning.

"What I cannot create, I do not understand." — Feynman. You have now created all three: the mean-seeking regressor, the noise-transporting flow, and the self-correcting relabeler. Build them in the Studio and the understanding is yours.

Companion & practice forge for Stanford CS 224R Homework 1 (Imitation Learning). Credits the public course materials. Implement the core math yourself; this is not a solution answer bank.