You can't write down a reward for "drive politely" — but you can show examples. So run reinforcement learning backward: from an expert's demonstrations, recover the reward function they were secretly optimizing. The catch is the answer is never unique — and the fix is the single most elegant idea in the lecture.
You are building a self-driving car and your boss says: make it drive politely. Reasonable request. So you sit down to write the reward function — the number the car maximizes — and you freeze.
What is "polite," numerically? Plus ten for leaving a gap? Minus five for tailgating? How much is a smooth lane change worth versus arriving thirty seconds sooner? Should yielding to a merging truck score higher than yielding to a merging sports car? Every weight you pick is a guess, the guesses interact, and the moment the car does something rude you discover the reward you wrote rewarded the wrong thing. Hand-designing rewards for rich human behavior is, in practice, brutally hard. This is the problem the whole lecture exists to solve.
But notice something. You may not be able to write the reward — yet you have something almost as good. You have demonstrations: hours of footage of good human drivers being polite. They never told you their reward either. They just drove. The question that defines this lecture is:
Why bother recovering a reward at all, when you already have demonstrations you could just copy? Because the reward is the intent, and intent transfers. The lecture's golf analogy is perfect: copying an expert's exact swing frame-by-frame (behavior cloning) breaks the instant the ball lies somewhere new. But if you recover what the swing was for — get the ball in the hole with few strokes — you can re-plan a fresh swing for any lie. The reward is the compact, portable thing. The trajectory is the disposable thing.
IRL belongs to the family of learning from demonstrations. The lecture lays out the family so you can place it:
| Demonstrations are… | Use this | Reward needed? |
|---|---|---|
| good (expert) | imitation learning — including IRL | No (IRL recovers one) |
| mixed / suboptimal | offline RL (Lecture 17) | Yes (given) |
| good, and you want a world model | model-based RL | Yes (given) |
IRL sits in the imitation column: it assumes the demonstrations are good and that no reward is available — the same setting as behavior cloning. The difference is what it produces. Behavior cloning produces a policy. IRL produces a reward, and only then a policy.
Two teal expert paths cross a gridworld toward a goal, steering around a hazard. Press Recover reward and watch a reward heatmap form: cells the expert sought become warm, cells it avoided stay cold. The expert never told us the reward — we inferred it from where it chose to go.
Both behavior cloning and IRL start from the exact same data — a dataset of expert trajectories D = {(s0, a0), (s1, a1), …}. The difference is what they learn from it, and that difference is everything.
Behavior cloning (BC) treats imitation as plain supervised learning: state in, action out. It fits a policy π(a|s) to predict, for each state the expert visited, the action the expert took. It never asks why the expert did that — it just memorizes the mapping. Lecture 6 showed the cost of this: the moment the cloned policy drifts to a state the expert never visited, it has no idea what to do, errors compound, and the trajectory cost grows like ε·H² in the horizon H. BC has no model of the goal, so it cannot recover.
IRL assumes the expert is (approximately) optimal for some reward, and tries to find that reward. Formally, it looks for a reward r* such that the expert policy π* is the one that maximizes expected reward:
Read that aloud: "find the reward r* under which the expert's policy earns more expected reward than any other policy." Once you have r*, you run ordinary RL on it to get a policy — and that policy generalizes, because it is solving the goal, not parroting a state-to-action lookup table.
Imagine the expert demonstrated driving on a straight road. BC saw "go straight" in those states and learned exactly that. Drop it on a curved road — new states — and it has nothing. But IRL recovers the reward "stay centered in the lane, avoid the shoulder." Run RL with that reward on the curved road and it plans a correct curved trajectory it was never shown. Same reward, new environment, correct new behavior. That re-planning step is exactly what BC lacks.
The lecture is honest about the cost: IRL is more work. You typically need an extra RL loop to optimize a policy for the recovered reward, and that loop is expensive. The tradeoff:
| Behavior cloning | Inverse RL | |
|---|---|---|
| Learns | a policy π(a|s) directly | a reward r(s,a), then a policy via RL |
| Generalizes to new states/envs? | poorly (no goal model) | well (re-plan with the reward) |
| Output interpretable? | no (a black-box mapping) | yes (an explicit reward) |
| Compute cost | cheap (one supervised fit) | expensive (inner RL loop) |
| Compounding error? | yes (ε·H²) | mitigated (policy solves the goal) |
Here is the trouble at the heart of IRL, and the lecture states it bluntly: IRL is an ill-posed problem. Given a set of expert demonstrations, there is not one reward that explains them — there are infinitely many. The problem is fundamentally under-determined. Before we can solve it we have to understand exactly how it is broken.
Consider the simplest disaster. Set the reward to zero everywhere: r(s,a) = 0 for all states and actions. Under that reward, every policy earns the same total reward (zero), so the expert's policy is — trivially — an optimal one. It is a maximizer. There is nothing to distinguish it from any other policy. The all-zeros reward "explains" the expert's behavior perfectly, and tells you absolutely nothing.
The problem is deeper than constants. Suppose the expert always walks to a goal in the top-right corner of a grid. Many different reward maps make exactly that path optimal: one that puts a big reward only at the goal; one that adds a gentle slope rising toward the goal; one that puts small bonuses along the diagonal the expert happens to walk. All three are different functions, all three make the same trajectory optimal, and the demonstrations cannot tell them apart — the expert only showed you the winning path, not what every alternative would have scored.
Toggle between Reward A (a single spike at the goal) and Reward B (a smooth slope toward the goal). The heatmaps are different functions — but the greedy optimal path they induce (warm arrows) is identical. The expert's demonstration is consistent with both. That is the ambiguity: the data does not pick one.
There is a third layer, which the MaxEnt paper (Ziebart et al., 2008) calls out and which matters once demonstrations are imperfect. We will soon summarize the expert by a statistic — how often it visits each kind of state. But many different stochastic mixtures of policies produce the very same statistics. So even after we agree on a target statistic to match, the space of behaviors that match it is still huge. Three layers of ambiguity, stacked.
To make IRL tractable, nearly every classical algorithm makes one modeling choice: the reward is linear in features. This single assumption is what lets the math close, so let us build it carefully.
A feature φ(s) is a vector of numbers describing a state — things you can measure. For a driver: distance to the car ahead, speed relative to the limit, lane offset, time to a lane boundary. For our gridworld: "is this cell grass?", "is this cell road?", "distance to goal." Features are given; you (or a neural network) design them. They are the vocabulary the reward is allowed to talk about.
The reward of a state is then a weighted sum of its features, with an unknown weight vector θ:
Here θ is the unknown we want to recover (the driver's priorities) and φ is known (the measurable features). IRL has gone from "recover an arbitrary function of state" to "recover d numbers." That is a massive simplification, and it is why the linear form is so popular. (The lecture notes φ itself can be a deep network — the weights on top stay linear, which is all the math below needs.)
Now the beautiful part. If the reward is linear in features, then the value of a policy — its expected discounted return — is also linear in θ. Watch the algebra; it is short and it is the whole reason this works. The value of a policy from a start state is the expected discounted sum of rewards:
Substitute the linear reward rθ(s) = θTφ(s):
Now θ is a constant vector — it does not depend on time or the trajectory — so pull it outside the expectation and the sum:
The thing left inside — the expected discounted sum of feature vectors collected as the policy runs — gets its own name. It is the expected feature count (or feature expectation) of the policy, written μπ:
You do not need to know the policy in closed form to get μπ. Just run the policy (or read the demonstrations) and average. Given m trajectories, the Monte-Carlo estimate of the feature counts is the average over trajectories of the discounted feature sum along each one:
This is exactly how we summarize the expert: collect its demonstrations, compute their average discounted feature counts μE. That vector μE is the entire fingerprint of the expert that IRL will try to match.
A 6×6 world has two features per cell: φ1 = "closeness to the goal corner" and φ2 = "is this a hazard cell." Slide the weights θ1, θ2 and watch rθ = θ1φ1 + θ2φ2 repaint the map. A positive θ1 and negative θ2 recovers "head to the goal, avoid hazards."
Chapter 3 gave us the lever. Here is how Abbeel & Ng (2004) pull it. Their insight, which the lecture calls "both necessary and sufficient," is this:
The proof is one line, and it follows straight from the key identity. The gap in value between the expert and the learner, under any weights θ, is:
Look at the right-hand side. If the feature counts match — if (μE − μπL) is the zero vector — then the gap is θT·0 = 0, no matter what θ is. The learner matches the expert's value even though we never figured out θ. By the Cauchy–Schwarz inequality the gap is also bounded by the size of the mismatch: |VE − VπL| ≤ ‖θ‖ · ‖μE − μπL‖. Shrink the mismatch, and you shrink the worst-case value gap. That is why "make the feature counts close" is the right objective.
This gives IRL a concrete number to drive to zero. Define the feature-matching residual as the gap between the expert's feature counts and the learner's:
A residual of zero means the learner is, value-wise, indistinguishable from the expert under the linear-reward family. A large residual means we are not behaving like the expert yet. This single scalar is the compass for the whole IRL loop in Chapter 8.
The teal dot is the expert's feature counts μE in 2D feature space. Press Run IRL loop: the warm dot is the current learner's μπL, and each iteration nudges it toward the expert. Watch the residual (the gap) shrink toward zero — the goal of feature matching.
Let us compute these feature counts ourselves and confirm the headline fact: a policy's residual against the expert is exactly zero when it is the expert's own policy. This is the lab that makes "feature matching" concrete.
We have the objective (match feature counts) and we have the linear value identity (V = θTμ). Now: how do you actually find θ? The lecture presents two pre-MaxEnt recipes. Both are tie-breakers that pick a reward making the expert look as good as possible relative to alternatives.
The original IRL paper had a clean idea: pick the θ that maximizes the gap between the expert's value and the value of other policies. If the expert is optimal under the true reward, then the true reward makes the expert beat every alternative. So search for the θ that pushes that advantage as large as possible:
Because value is linear in θ (Chapter 3), this objective and its constraints are all linear in the unknown θ — so it is a linear program, which solves efficiently. Two practical notes from the lecture. First, you need a bound on θ (say |θi| ≤ 1) or the "maximize the gap" objective runs off to infinity. Second, where do the "other policies" come from? You generate them — by running RL on the current reward guess, or hand-designing them — and the whole thing iterates: solve for θ, train a new competing policy π with reward θTφ, add it to the pool, re-solve θ. This iteration is the skeleton of the IRL loop.
Max-margin IRL borrows the philosophy of the support vector machine: don't just make the expert better than the alternatives — make it better by the largest possible margin. Find the θ (with ‖θ‖ ≤ 1) that maximizes the smallest advantage of the expert over any competing policy:
The largest margin gives the most robust reward — the one least likely to be a near-tie that flips with a little noise. Abbeel & Ng famously used this to teach an autonomous helicopter aerobatics from a human pilot's demonstrations: recover the pilot's reward, then run RL to fly maneuvers the pilot never explicitly demonstrated.
Let us see the core ambiguity that motivates this directly: build a tiny gridworld and show that two genuinely different weight vectors θ produce the identical greedy policy. No tie-breaker, no unique answer — this is the disease the rest of the lecture cures.
We are stuck with infinitely many rewards (and infinitely many behavior-distributions) that all match the expert's feature counts. We need a principled way to pick exactly one. Maximum-entropy IRL (Ziebart, Maas, Bagnell, Dey, AAAI 2008 — from CMU) gives the most elegant answer in the lecture, and it comes from a single idea borrowed from physics and information theory.
The principle of maximum entropy (Jaynes, 1957) says: among all probability distributions consistent with what you know, choose the one with the highest entropy — the most spread out, the most uncertain, the one that assumes the least beyond your constraints. Entropy measures uncertainty; the highest-entropy distribution is the one that commits to nothing it was not forced to commit to. The lecture frames it as Occam's razor: the simpler explanation is to be preferred.
Ziebart et al. put a probability distribution over entire trajectories ζ (a sequence of states/actions), not over single actions. The constraint is feature matching: the distribution's expected feature counts must equal the expert's. Solving "maximize entropy subject to that constraint" with Lagrange multipliers gives a clean closed form — a distribution where the probability of a trajectory is exponential in its reward:
Let us read every symbol. φζ = ∑s ∈ ζ φ(s) is the trajectory's total feature count, so θTφζ is its total reward (the reward of a path is the sum of its state rewards). The partition function Z(θ) = ∑ζ exp(θTφζ) is the normalizer — it sums the exponentials over all trajectories so the probabilities add to one. This is exactly a Boltzmann / softmax distribution over paths.
Now finding the reward is clean: choose θ to maximize the log-likelihood of the demonstrated trajectories under this model. And the gradient of that log-likelihood is the most satisfying result in the lecture — it is exactly the feature-matching residual:
That is breathtakingly simple. To increase the likelihood of the expert's data, nudge θ in the direction of (expert feature counts − the current model's feature counts). When the two match, the gradient is zero and we stop — at exactly the feature-matched solution from Chapter 4. Maximum entropy hasn't replaced feature matching; it has uniquely identified which feature-matching solution to take: the one with the highest entropy. Ziebart et al.'s contribution was an efficient dynamic-programming way to compute the model's expected feature counts E[φζ] (the hard term), making the gradient practical.
Five candidate trajectories have feature-based rewards. Slide the reward weight θ. Each bar is P(ζ) ∝ exp(θ · reward). At θ = 0 the distribution is uniform (maximum entropy, no preference). As θ grows, probability piles exponentially onto the high-reward path — but never collapses entirely. This is the MaxEnt path distribution.
Formulas hide until you push numbers through them. Let us compute an actual maximum-entropy trajectory distribution and an actual gradient step, with a pencil, so the symbols become arithmetic.
Take a tiny world with three candidate trajectories. Use a single feature so θ is a single number. Each trajectory's total feature count φζ (its "goodness") is:
So trajectory 3 is the "best" (highest feature count), trajectory 1 the worst. Suppose the reward weight is θ = 1. The reward of each trajectory is θ·φζ, i.e. 1, 2, and 3.
The MaxEnt distribution is P(ζ) ∝ exp(θ·φζ). Compute the exponentials:
Z is just their sum — the normalizer:
Divide each weight by Z:
They sum to 1.000, as a probability distribution must. Read the result: the best trajectory gets 66.5% of the probability mass — clearly preferred — but the others keep a real, non-zero share (24.5% and 9.0%). The distribution is tilted toward high reward, not collapsed onto it. That residual spread is the entropy MaxEnt insists on keeping.
Now suppose the expert demonstrated trajectory 3 (the good one). The expert's empirical feature count is μE = φ3 = 3. The model's expected feature count under the current distribution is the probability-weighted average:
The log-likelihood gradient from Chapter 6 is expert minus model:
The gradient is positive, so a gradient-ascent step raises θ. Raising θ makes the high-feature trajectory 3 (which the expert actually showed us) even more probable — pushing the model's expected feature count up toward the expert's 3. Exactly the right direction. Iterate, and E[φ] climbs toward μE until the gradient vanishes at the feature-matched, maximum-entropy reward.
Let us implement that single gradient step and confirm, independently, that it moves the reward weight in the direction that makes the expert's trajectory more likely.
Everything now assembles into one loop — the "general recipe" the lecture states up front. IRL is iterative because you cannot solve for the reward in one shot: you guess a reward, see what optimal behavior it implies, compare that to the expert, and correct. It is reinforcement learning running inside a reward-search loop.
Step 2 is why IRL is costly: each outer iteration runs a full RL solve. Step 4 is the MaxEnt gradient from Chapter 6 — or, in max-margin IRL, an LP/SVM step — but the structure is identical: the update direction is always the feature-count difference. When the residual hits zero, the learner matches the expert's value under every linear reward (Chapter 4), and θ is the recovered reward.
Two expert paths (teal) head to the goal avoiding a hazard. Press Run IRL: each iteration solves for the best policy under the current reward, compares its feature counts to the expert's, and updates the reward weights. Watch the reward heatmap sharpen and the residual bar shrink — the loop converging on the expert's intent.
You arrived unable to write a reward for "drive politely." You leave able to recover a reward from demonstrations, understand why the answer is never unique, and pick the single least-biased one. Here is the whole lecture on one page.
| Method | Tie-breaker / principle | Handles noisy demos? |
|---|---|---|
| LP-IRL (Ng & Russell 2000) | maximize expert's value advantage (linear program) | poorly |
| Max-margin / apprenticeship (Abbeel & Ng 2004) | maximize the SVM-style margin | poorly |
| MaxEnt IRL (Ziebart et al. 2008) | maximum entropy — least-committal P(ζ) ∝ exp(reward) | well |
| GAIL (2016) | adversarial feature/distribution matching | well (model-free, no inner RL) |
"The simpler explanation is to be preferred." — Occam's razor, the principle MaxEnt IRL turns into math. You just recovered a reward three times.