You have a giant log of robot experience and you are not allowed to collect more. Can you still learn a policy better than the one that gathered the data? Yes — but only if you defeat the one failure that wrecks naive Q-learning on logged data: the agent overvalues actions it never tried.
A self-driving car company has driven millions of miles. A warehouse robot has been teleoperated through ten thousand pick-and-place demonstrations. A hospital has years of treatment records. In every case there is a mountain of logged experience sitting on disk — and a hard rule: you may not go collect more. New on-road data points cost one to two months each. A bad exploratory action smashes a robot arm into a shelf. In drug discovery a wrong "action" is a dead patient. So the only data you will ever have is the data you already have.
This is offline reinforcement learning (also called batch RL): learn the best policy you can from a fixed, static dataset of transitions, with zero new interaction with the environment. No exploring. No trying. Just the log.
Why not just run our beautiful off-policy Q-learning from Lecture 8 on the log? Q-learning is off-policy, which means (we proved it) its target r + γmaxa′Q(s′,a′) is independent of which policy collected the data. So a static log of transitions is exactly the kind of data Q-learning was built to consume. It should just work… right?
It does not. It fails catastrophically, and in a way that gets worse, not better, the longer you train. The reason is hidden inside that innocent-looking max. Watch it happen.
A one-step problem (a "bandit"): one state, several actions, but the log only ever tried the teal actions in the middle — the grey actions on the edges were never logged. The dashed curve is the true reward. Press Train naive: the fitted value model (warm curve) matches the truth where it has data, but extrapolates wildly upward on the unseen edge actions — and the greedy policy (the marker) marches straight to the highest fantasy. Then press Train conservative to see the fix this lecture builds.
The naive model is not wrong because it had too little data. It is wrong because it confidently guessed about actions it can never test, and the max in the policy then deliberately hunts for the largest guess. The data tells you nothing about the edges, and the policy walks straight off the edge. Fixing this is the entire lecture.
The cheapest thing you can do with a log of demonstrations is copy them. Behavioral cloning (BC) — supervised learning of "in this state, do what the log did" — is the simplest offline method. So a natural question: if all we have is a fixed dataset and no interaction, isn't offline RL just imitation with extra steps?
No. And the difference is the whole reason offline RL exists. Imitation can only reproduce the average behavior in the log. Offline RL aims to produce a policy that is better than anything in the log, by recombining good fragments of different trajectories. The lecture calls this stitching.
Suppose your log contains two mediocre trajectories through four waypoints A, B, C, D:
No single logged trajectory goes from A to the goal C. An imitator starting at A will follow Trajectory 1 to D — it never reaches C, because no demonstration did. But the two trajectories both pass through B. A value-based offline method can learn that B has high value (because from B you can reach C, as Trajectory 2 shows), and therefore stitch: go A → B (from Trajectory 1), then B → C (from Trajectory 2). The result — A → B → C — is a path better than any trajectory in the data.
This is the payoff and the promise. The machinery that makes stitching possible is multi-step dynamic programming: the Bellman backup propagates the value of C backward through B and on to A. It is the same value-bootstrapping you learned for Q-learning — and it is also exactly what drags in unseen actions and overestimation. Power and peril, same mechanism.
The two faint paths are the logged trajectories (A→B→D and D→B→C). Press Imitate: a cloning policy from A follows the data to D — the goal C is never reached. Press Stitch: value propagation marks B as the high-value junction, and the offline RL policy composes A→B (path 1) with B→C (path 2) to reach the goal. Green = the stitched, better-than-data path.
Let us find the exact line where everything breaks. Recall the offline TD loss — standard fitted Q-iteration on the dataset D, exactly as in Lecture 8:
Read the loss carefully. The states s, the actions a, and the next states s′ all come from the dataset D — those are safe, they actually happened. But look at the inner expression maxa′ Q(s′,a′). The maximization ranges over all actions a′, including actions that were never taken in s′ in the entire log. That is the leak.
Function approximators do not return a flat zero on inputs they have not seen; they extrapolate, and the extrapolation has random error in both directions. Now apply a max over actions to a vector of noisy estimates. The max systematically picks out whichever action got the luckiest positive error. So the bootstrap target is biased too high — this is the same "max of noisy estimates overestimates" effect behind Double DQN, but offline it is unbounded because there is no real reward signal to ever pull the fantasy back down.
Here is the feedback loop. Trace it once and you will never forget it:
Take a single state s′ with three actions. The log only ever took action a₁ (in-distribution), whose true value is 5. Actions a₀ and a₂ were never logged; the function approximator extrapolates them with optimistic error. Say it currently estimates:
With γ = 0.95 and reward r = 0 on this transition, the naive target uses the max:
The honest in-data value of the next state is only 5, so the honest target would be 0.95 × 5 = 4.75. The max inflated the backup by 8.55 − 4.75 = 3.8, and it did so by trusting a₀, an action the log never tried. Now Q(s,a) gets pulled toward 8.55. Next iteration, the model extrapolates a₀ even higher (say 11), the target becomes 0.95 × 11 = 10.45, and up it spirals. The numbers run away because the max keeps rewarding the most over-optimistic unseen action.
Three actions at state s′. The teal bar (a₁) is in the data — its value stays pinned at the truth (5). The grey OOD bars (a₀, a₂) extrapolate. Press Naive backup repeatedly: the max keeps picking the highest OOD bar and feeding it back, so the fantasies climb without bound (the warm "target" line runs off the top). Press Conservative backup: a penalty pushes the OOD bars back down toward the data, the max settles on the true in-data value, and the target stops diverging.
The lecture is blunt about a tempting but wrong escape hatch: "just collect more data." We meet that misconception head-on in the next chapter.
To talk about "actions in the data" precisely we need one piece of vocabulary. The fixed log was collected by some policy — a human teleoperator, an old controller, a mixture of many. Whatever it was, call it the behavior policy πβ(a|s): the distribution of actions that actually appear in the dataset at each state. The dataset D is samples from it.
An action a at state s is in-distribution (in-support) if πβ(a|s) is meaningfully positive — the log contains examples of it. It is out-of-distribution (OOD) if πβ(a|s) ≈ 0 — the log essentially never tried it. Distributional shift is the gap between the actions the learned policy π wants to take and the actions πβ actually provided. The bigger that gap, the more the value function is being asked about regions it has no evidence for.
If overestimation came from too little data, more data would cure it. It does not. The lecture shows the log-scale plot: as you add data, the observed reward and the (cumulative) reward model diverge further, because more data of the same wrong distribution still leaves the OOD actions unsupported. The optimizer always finds the gap.
The shaded hump is the behavior data πβ over the action axis. The dashed curve is the true reward; the warm curve is the learned model (accurate under the hump, wild outside it). The marker is where the policy's optimizer lands. Slide the policy's aggressiveness: at 0 it stays inside the data (safe, modest reward); push it right and its peak leaves the hump — the model promises a huge value but the true reward there (the cross) collapses. That gap is distributional shift made visible.
The most direct response to "the policy keeps leaving the data" is: don't let it. A policy-constraint method modifies the policy-improvement step so the learned policy π cannot stray too far from the behavior policy πβ. If π only ever proposes actions the data supports, then max never queries an OOD action, and the death spiral never starts.
Concretely, instead of the unconstrained greedy improvement π = argmaxaQ(s,a), we constrain it:
Here D(·,·) is a divergence between distributions (for example KL divergence), and ε is a leash length. The policy is allowed to improve over the data — but only within a ball of radius ε around the behavior policy. This is the family the lecture cites: BCQ (Batch-Constrained Q-learning, Fujimoto et al. 2018), BRAC (Behavior-Regularized Actor-Critic, Wu et al. 2019), and relatives.
The trouble with constraints is the radius ε. Too tight and you have re-invented imitation — the policy clings to πβ and gives up the stitching that motivated offline RL in the first place. Too loose and OOD actions leak back in and the values blow up again. Worse, a single global ε is crude: in well-covered states you could safely roam, while in thinly-covered states even a small step is OOD. Matching the leash to the local data density is exactly what the next two families do more gracefully.
| ε (leash) | Behavior | Failure mode |
|---|---|---|
| Too tight | Clings to the data | No improvement — just imitation, no stitching |
| Just right | Improves within support | (the sweet spot, hard to find) |
| Too loose | Leaves support | OOD actions leak in → overestimation returns |
Policy constraints work, and they were the first thing that worked. But notice what they do not do: they constrain the policy, leaving the value model itself free to be wildly over-optimistic off-support. The next idea attacks the value model directly.
Policy constraints attack step 1 of the spiral (don't let the policy pick OOD actions). Conservatism attacks the value model itself: don't bother constraining the policy — just make sure the value model never over-estimates an OOD action in the first place. If the unseen actions are valued low, the max will simply avoid them on its own.
This is the idea behind Conservative Q-Learning (CQL) (Kumar, Zhou, Tucker, Levine, NeurIPS 2020). The lecture frames it as a question: "which value models do not hurt?" Answer: ones that are pessimistic about actions they have not seen. CQL learns such a model by adding one extra term to the standard Bellman regression:
Read the two pieces of the new term. The first expectation, Ea~μ[Q(s,a)], pushes down the Q-values of actions drawn from some broad distribution μ — in practice the actions the model currently thinks are best, which are exactly the dangerous high-value unseen ones. The second expectation, E(s,a)~D[Q(s,a)], pulls up the Q-values of the actions actually in the data. The coefficient α sets the strength.
Back to the three actions at s′ from Chapter 2: in-data action a₁ with true value 5, and OOD fantasies Q(a₀) = 9, Q(a₂) = 7. The naive max gave 9, inflating the target. Now apply a conservative penalty: push down the over-valued OOD actions by, say, the gap above the in-data value. After the penalty the model has been driven to:
Now the max is max(4.2, 5.0, 4.6) = 5.0 — it lands on the in-data action, exactly the honest value. The target becomes 0.95 × 5.0 = 4.75, the correct value, instead of the runaway 8.55. Conservatism converted the max from a fantasy-hunter into a truth-seeker, by making sure no unseen action out-scores the data. That is the entire trick.
The lecture notes a sharper danger in the full RL (multi-step) case than in the one-step bandit: because the target itself depends on the policy, the exploitation is more severe — an over-valued OOD action does not just mislead one decision, it poisons every backup that bootstraps through it. So conservatism matters even more once you bootstrap.
Let's implement the conservative penalty and verify it clamps OOD values below the in-data maximum.
The first two fixes both still touch OOD actions — constraints limit how far the policy reaches into them, conservatism evaluates them in order to push them down. Implicit Q-Learning (IQL) (Kostrikov, Nair, Levine, 2021) takes the most radical stance: never query the Q-function on an out-of-sample action at all. If you only ever evaluate Q at state-action pairs that appear in the data, there is no fantasy to inflate. The leak isn't sealed — it's removed.
Recall from Lecture 8 that SARSA's target uses the action actually taken next in the data, Q(s′,a′) with a′ from the log — never a max, never an OOD action. So a SARSA-style backup is automatically in-sample and safe:
But this only learns the value of the behavior policy — it cannot improve over the data, because it averages over whatever actions the log happened to take, good and bad. We want the value of the best in-support action without ever naming it. That is where the clever bit comes in.
IQL treats the state value V(s) as a random variable: fix the state, and the value Q(s,a) varies as the action a varies over the data distribution. The maximum of that random variable (over in-support actions) is exactly the best in-data value we want. We just can't compute a hard max without querying actions. So IQL estimates an upper expectile of the distribution instead — a statistic that, as it approaches 1, approaches the in-support maximum, but is computable purely from samples in the data.
An expectile is the asymmetric-loss cousin of a quantile. The τ-expectile is the value m that minimizes an asymmetric squared error: errors on one side are weighted more than the other.
Decode it. When τ = 0.5 both signs of the residual u get weight 0.5 — that is ordinary mean-squared error, so the ½-expectile is the mean. When τ = 0.9, residuals where the target was above the estimate (u > 0) get weight 0.9, while residuals below get only 0.1. The fit is pulled upward, toward the high values. As τ → 1, the estimate chases the maximum of the in-support values — but it got there by re-weighting in-sample residuals, never by evaluating an unseen action.
If you took the expectile of the full TD target r + γQ(s′,a′), you would catch not only "a good action" but also "a lucky transition" — a sample that happened to land in a great next state by chance of the dynamics. IQL avoids this with two networks. A value network Vψ(s) fits the upper expectile of Q over actions only, and then Q is fit with plain MSE over transitions, averaging out the dynamics luck:
Notice every expectation is over data tuples only — (s,a) and (s,a,s′) straight from D. No a′ is ever maximized; no Vψ or Q is ever evaluated at an action the log didn't take. The policy is extracted afterward by advantage-weighted regression (weight the behavior-cloning loss by exp of the advantage A = Q − V), which also touches only dataset actions. IQL is, in the lecture's words, just SARSA-style TD with an asymmetric L2 loss — almost shockingly simple, and state-of-the-art on D4RL.
The dots are the values Q(s,a) for the actions the data took at one state (a noisy cloud). The line is the fitted expectile Vψ(s). At τ = 0.5 it sits at the mean. Slide τ up: the asymmetric loss up-weights the high residuals and the estimate climbs toward the in-support max (dashed) — without ever evaluating an action outside the cloud. That is IQL's whole move.
Let's compute an expectile by hand in code and confirm it lands above the mean and below the max — the precise property that makes it a safe, in-sample stand-in for the max.
Time to put the whole argument on one small problem and measure it. We will build a tiny contextual bandit where the data covers only some actions, learn a value model, and compare three policies: the unconstrained greedy policy (chases the max, including OOD fantasies), and a support-constrained policy (only allowed to pick actions the data covers). We will score each by its true reward — the thing we actually care about and can compute only because this is a toy.
This is the same lesson as the cliff in Lecture 8 — the optimistic max is dangerous when you cannot correct it — but offline it is sharper still, because there is no future interaction at all to wash out the mistake.
That assertion — constrained beats unconstrained on true reward — is the empirical heart of offline RL. Policy constraints, CQL, and IQL are three different mechanisms that all enforce some version of it. The lecture's benchmark plot (D4RL) shows the same ranking at scale: imitation < policy-constraint methods < conservative/in-sample methods, especially on the "stitching" maze tasks.
Everything so far has been value-based: learn a Q or V, extract a policy. There is a completely different way to use a fixed log, and it sidesteps the OOD problem from another angle. Diffuser (Janner, Du, Tenenbaum, Levine, 2022) treats offline decision-making as conditional trajectory generation with a diffusion model.
The textbook recipe is: fit a one-step dynamics model from the log, then hand it to a trajectory optimizer that searches for a high-reward plan. The trouble — and it is the same trouble as before, wearing a different hat — is that a powerful optimizer exploits the learned model's errors. Its "optimal" plans look like adversarial examples: sequences that the model wrongly predicts are great, the same way the max wrongly valued OOD actions. Compounding one-step errors over a long horizon make it worse.
Three properties make this work for offline control:
The connection to the rest of the lecture: Diffuser stays on the data manifold by construction — a sampled trajectory is, by definition, drawn from the distribution of trajectories the model learned from the log. It never has to evaluate an OOD action, for the same structural reason IQL never does: it only ever produces things that look like the data. Different machinery, same governing principle — do not trust regions the data never covered.
A trajectory through a 2-D maze. Press Denoise to step the diffusion process: it starts as pure noise (a scribble) and is iteratively refined into a smooth, goal-reaching plan — with start and goal inpainted (held fixed) and a return guide pulling it toward the reward. Notice it denoises the whole path at once, not one step at a time.
You arrived able to run Q-learning on any stream of transitions. You leave knowing why that fails on a frozen log and three structurally different ways to make it work. Here is the whole lecture on one page.
| Idea | What it does | How it blocks the OOD leak | Cost / catch |
|---|---|---|---|
| Naive offline QL | FQI on the log | It doesn't — max queries OOD actions | Death spiral; diverges |
| Behavioral cloning | Copy the data | Never leaves the data | Can't beat the data (no stitching) |
| Policy constraint (BCQ, BRAC) | Keep π near πβ | Policy can't pick OOD actions, so max can't either | Leash ε hard to set; too tight = imitation |
| Conservatism (CQL) | Push down OOD Q-values | OOD actions can't out-score the data, so max avoids them | Can be over-conservative; un-learns when fine-tuning (Cal-QL fixes) |
| In-sample (IQL) | Upper expectile of in-data values | Never evaluates an OOD action at all | Expectile τ is a knob; needs separate V, Q nets |
| Diffuser | Denoise whole trajectories | Samples stay on the data manifold by construction | Generative + slow sampling; not value-based |
"The values of actions that are too different from those in the data are unlikely to be estimated accurately." — the one sentence from the IQL paper that, taken seriously, generates every algorithm in this lecture.