A frozen dataset of someone else's wandering — an incomplete prefix and a successful suffix from different starts. Plain imitation copies the observed action distribution. But a value-guided policy can stitch compatible good halves into a candidate full path. Two algorithms explore that idea: AWAC leans the policy toward high-advantage data actions; IQL learns an upper expectile of learned action values at data states, avoiding sampled next-action queries in its value backup.
A companion & practice forge for Stanford's CS 224R Homework 3 (Offline RL). It credits the public course materials and teaches you to implement representative core math yourself — it is not a copy-paste answer bank or a complete homework trainer. The starter-code contracts here (the double-Q TD target, the AWAC exp-weighted actor, IQL expectile loss and Q-target, advantage estimates, and Polyak update) are shrunk to browser-scale NumPy exercises — no torch, no D4RL, no cluster.
You are handed a box of driving logs and told to build a good driver. You cannot drive yourself — no test track, no simulator, no do-overs. Just the logs. That is offline reinforcement learning: learn a policy from a fixed dataset of someone else's experience, with no way to try things out.
Here is the twist that makes it interesting. Imagine a tiny puck — a PointMass — that must navigate to a goal. Your dataset contains two incomplete kinds of trip, and neither one contains the complete desired start-to-goal route:
Copy trip A and you get halfway, then stall. Copy trip B and you start elsewhere. But look closer: A's good first half meets a state near the beginning of B's successful suffix. If the state and dynamics really match, splice them — A's first half, then B's suffix — and you have a candidate route that appears in no single trip in the dataset. That proposed splice is called stitching; evaluation must still verify that the transition is physically feasible.
The start is bottom-left, the goal is top-right. Trip A (teal) does the first leg well then stalls; Trip B (purple) starts elsewhere and finishes well. This schematic assumes their marked state is compatible; toggle Stitch to inspect the candidate route.
The catch: learning a value function from a fixed dataset is dangerous. The moment you ask "what's the value of an action the data never tried?", a neural network extrapolates without a direct label. The error can have either sign, but a policy that searches many noisy estimates preferentially selects positive mistakes, which can then contaminate later targets. AWAC and IQL use different controls for this risk; neither makes learned values infallible.
Before AWAC or IQL, we pin down the world and the data. HW3 uses a PointMass navigation task and, in the full assignment, an 8-DOF AntMaze with continuous robot actions. PointMass is the right thing to hold in your head because it is 2-D: you can draw the trajectories and inspect a candidate stitch.
A puck lives in a 2-D maze with walls. Its observation is its normalized x and y position; there is no velocity coordinate in this PointMass observation. Its action is one of five discrete choices: stay, down, up, left, or right. The motion is noisy, so the same choice need not produce an identical displacement every time. The reward is sparse: you get 0 at the goal and −1 every other step. A successful episode with T transitions therefore sums to −(T−1) when its final transition receives zero — for 47 transitions, 46·(−1)+0=−46. Time-limit truncation and goal termination must follow the evaluator's mask convention.
| Piece | PointMass | Why it matters |
|---|---|---|
| Observation s | normalized x and y position, continuous | Continuous positions need a function approximator; the action choices remain discrete. |
| Action a | Discrete(5): stay, down, up, left, right | Integer action indices select among five noisy motion commands. |
| Reward r | 0 at goal, −1 otherwise (sparse) | This reduces the particular loitering incentive, but does not guarantee an ungameable reward. |
| Discount γ | 0.99 | Future reward matters almost as much as now — long-horizon credit. |
The defining constraint: during training you never touch the environment. No rollouts, no exploration. You read from a fixed set of transitions:
collected by some other policy — a mixture of decent and mediocre drivers. It never grows. If a transition is not in D, you will never see it.
HW3 ships a curated pointmass_stitching_dataset.npz designed to test exactly one thing. Its trajectories have:
| Statistic | Value | Meaning |
|---|---|---|
| Transitions / complete episodes | 8,184 / 79 | Done-delimited archive; no leftover tail after the final episode. |
| Observation / next-observation shapes | [8184, 2] / [8184, 2] | Two normalized position coordinates in each transition. |
| Action range | 0–4 | Integer PointMass action indices. |
| Rewards | −1 or 0 | Sparse reward values in the checked NPZ archive. |
| Return max / min / mean | −46 / −150 / about −102.99 | The handout reports about −104; this differs from the checked archive mean, for an unresolved reason. |
A useful test is to train an offline RL algorithm and compare it with −46 under the same start states, reward, horizon, and terminal convention. A return better than the archive's best is evidence consistent with composition, but it does not by itself prove that stitching caused the improvement; inspect routes and repeat the evaluation. Top-trajectory filtering changes the cloning data distribution, but a state-conditioned policy can still generalize beyond any one recorded path, so −46 is not a universal behavior-cloning ceiling.
A schematic fixture inspired by the dataset, not a plot of every archived transition. Most dim paths wander and stall. Two are highlighted: the good first half and the good second half whose marked states nearly meet. Drag the slider to vary the illustrative background paths.
Here is the natural idea, and why it fails so badly it broke offline RL for years. You have a fixed dataset. Just run your favorite off-policy algorithm on it — train a Q-network with Bellman TD, extract the greedy policy. What could go wrong?
A Q-network is a neural net. Feed it a state and an action, it returns a number. For state-action pairs in the dataset, the loss supplies observations, but the learned number can still be inaccurate. At actions the data barely covers or never tried, the network extrapolates and its error can have either sign. Nothing directly labels every out-of-distribution (OOD) action — not the data, not the loss. When a policy searches many noisy estimates, it is more likely to select a value that happens to be too high.
Now watch the risk loop. The policy searches for a high Q. If an OOD action a* has an accidental positive error, the policy may move toward a*. A critic target can then depend on that selected estimate, allowing the error to propagate. The policy may chase it harder and evaluation can deteriorate. This is a possible actor-critic feedback loop, not a guarantee that every run diverges or that Q-values become infinite.
The horizontal axis is a one-dimensional action in an illustrative toy model. The data cloud covers a middle region; the dashed curve is a chosen reference value. Drag training forward to see one possible upward extrapolation and the greedy policy's selection. Real errors can have either sign and need measurement.
Modern offline RL has two families of fixes, and AWAC and IQL are one from each:
| Family | The discipline | HW3's example |
|---|---|---|
| Constrain the policy | Use data actions in the actor loss to encourage a policy near the data; the parametric policy can still generalize outside observed actions. | AWAC |
| Constrain the value | Use a state-only value in the IQL Q backup, removing the sampled next-action query from that value-learning path. | IQL |
AWAC extracts the policy by weighted behavior cloning on logged actions, encouraging a data anchor without guaranteeing strict support. IQL introduces a state-only value function trained by expectile regression, so its Q target uses V(s′) and does not need a sampled next action. Same risk, two controls. We build both.
Both algorithms rest on the same two objects: the Q-function and the value function. We recap them precisely, because the entire art of offline RL is computing them without asking a question the data can't answer.
The policy-specific Q-function Qπ(s, a) is the expected return if you take action a in state s, then follow policy π. An optimal Q function assumes an optimal continuation, which is a different object. The ordinary policy value Vπ(s) is the expected return from state s under that policy — it does not fix an action:
Read Vπ(s) as "how good is being here if I act as my policy would." IQL later fits a different statistic, an upper expectile of learned action values, so its V is not automatically a policy average. The advantage ties a chosen action to its state baseline:
Positive advantage: this specific action was better than the state's average. Negative: worse. Advantage is the clean signal both AWAC and IQL use to decide which dataset actions to imitate more strongly — it strips out the "how good is this state" baseline and leaves only "how good was this action, here."
V(s) — a state-dependent baseline — centers the signal on zero, isolating the per-action difference. AWAC's baseline is a sampled policy-action value; IQL's is a learned upper expectile, so call an action positive when it is above the chosen baseline.The Q-function refers to itself one step ahead. The Bellman target for a transition (s, a, r, s′, d) is:
The (1 − d) mask is the terminal case: if the supplied mask says the episode ended (d = 1), there is no future under that convention, so the target is just the reward. Time-limit truncation may require a separate treatment in a different evaluator. For a nonterminal example, reward minus one, discount zero point ninety-nine, and next-state value two give a target of zero point ninety-eight. Then Q regresses toward that target by squared error. The offline question is: how do you compute the next-state value while controlling unsupported action queries?
| Approach | Value of s′ | OOD risk |
|---|---|---|
| Naive Q-learning | maxa′ Q(s′, a′) | High risk — unrestricted search can select unsupported positive errors. |
| AWAC | min(Q̄1, Q̄2)(s′, a′), a′ ~ π | Reduced, not removed — the actor is data-anchored, but sampled actions can still drift. |
| IQL | V(s′) — a state-only function | No sampled next-action query — learned Q and V can still be inaccurate. |
This is a schematic grid, not the exact PointMass maze map. The goal cell has value 0; other displayed cells receive a toy negative step value. Press step to run illustrative backups: value ripples outward from the goal. A state-conditioned value can rank a better continuation at a shared state, subject to the learned estimates and environment dynamics.
s appears in the wandering tail of trip A and in the good middle of trip B. The Q-function does not need a trajectory identifier; it can learn that B's logged continuation at s has higher estimated value. A policy may then prefer that observed continuation even though no full trajectory did A's first half followed by B's second. The value function is state-aware, but its ranking can still be wrong and the splice must be evaluated in the environment.AWAC — Advantage-Weighted Actor-Critic (Nair et al., 2020) — is the "constrain the policy" answer. It has two halves: a critic that learns Q by Bellman TD, and an actor extracted by advantage-weighted regression. This chapter builds the critic; the next builds the actor.
AWAC's critic is exactly the off-policy critic from HW2 — two independent Q-networks with slow-moving target copies. For a batch of transitions (s, a, r, s′, d), the target is:
and the loss trains both online critics toward it:
Three pieces, each with a job:
| Piece | Name | Why it's there |
|---|---|---|
two Q̄1, Q̄2 | target networks | Slow, Polyak-averaged copies of the online critics. A regression target must be stable; if it moved every step you'd chase your own tail and diverge. |
min(·, ·) | clipped double-Q | The max over noisy estimates is biased upward. Taking the min reduces that upward selection pressure, but it is not a lower-bound guarantee and can still be above the true value. |
(1 − d) | terminal mask | If the supplied d marks s′ terminal, the target is just r. Time-limit masks need the task's specified convention. |
a′ comes from — the one place the critic touches the actorTo evaluate min(Q̄1, Q̄2)(s′, a′) you need an action a′ at the next state. AWAC samples it from the current actor: a′ ~ π(·|s′) (gradients off — it's used only in the target). This is what makes AWAC an actor-critic rather than pure offline Q-learning.
a′ is only approximately in-distribution. AWAC's actor loss is anchored on data actions, but a categorical policy can assign mass to unseen choices and a continuous policy has tails outside observed samples. The target Q can therefore evaluate slightly OOD actions, and errors can leak in. A stronger anchor or larger dataset may reduce the drift; neither is a support guarantee. IQL removes the sampled next-action query from its value backup — hold that scoped claim for Chapter 6.no_grad disciplineTwo rules that are bugs if broken. First, the target is computed inside a stop-gradient block: gradients must flow only through the prediction Q(s,a), never through the target. The optimizer owns online Q parameters, not the target copies, so a large loss needs broader diagnosis: inspect optimizer groups, target gradients, finite values, reward scale, masks, and shapes. Second, the (1 − d) mask — miss it and terminal states bootstrap under the wrong convention. You will write this target as Kernel 1 in the Studio.
Two critics estimate the same reference value, shown with independent toy noise. The max selects whichever estimate got lucky and increases upward pressure. The min of the two often reduces that pressure, but it can still sit above the truth on an individual draw. Reshuffle to see why this is a bias-control heuristic, not a bound.
min(Q̄1, Q̄2) in the Bellman target instead of a single Q or the max?The critic gives us Q. Now the actor. This is where AWAC earns its name and does its stitching — and where the "constrain the policy" discipline lives.
The AWAC actor is trained by advantage-weighted regression: imitate the dataset, but weight each state-action pair by the exponential of its advantage.
Read it piece by piece:
log πψ(a|s) — the log-probability the policy assigns to the dataset's action. This is ordinary behavior cloning.(s, a) ~ D — the action comes from the data, not a sampled policy action. The actor is asked "given someone took a at s, how much should its log probability change?" This data anchor discourages drift, but a parametric policy can still assign mass to unseen actions.A(s, a) = Q(s, a) − V(s) — the advantage. In AWAC, V(s) is a single-sample estimate: sample aπ ~ π(·|s) and use Q(s, aπ).exp(A / λ) — the weight. Always positive; larger when the action beat the state's average. λ is a temperature.exp(A/λ) factor is the only difference: it up-weights the dataset's better-than-average actions and down-weights the worse ones. So AWAC keeps a data anchor while leaning toward estimated good parts; the parametric actor can still generalize beyond observed actions. Because it up-weights a good action at each state regardless of which trajectory it came from, this is one route by which stitching might emerge.The exponential is not arbitrary — it is the closed-form solution to "maximize expected advantage while staying within a KL-ball of the data policy." The temperature λ is the Lagrange multiplier of that KL constraint:
| λ | exp(A/λ) behavior | Resulting policy |
|---|---|---|
| → 0 (small) | sharper relative weights; the cap can flatten several large positives | aggressive tilt over estimated advantages; risks overfit |
| starter default 0.1 | temperature is separate from the weight cap | inspect the weight distribution rather than assume one best action |
| → ∞ (large) | all weights ≈ 1 | plain BC — the same likelihood objective without an advantage tilt |
With small λ and a large advantage, exp(A/λ) can overflow — exp(20) is already 4.8×108. The starter clamps the post-exponential weights at 50: exp_weights.clamp(max=50). That limits a single value but can still let several capped examples dominate, so log the unclipped and clipped ranges. A robust variant can clamp the log weight before exponentiation; keep that recommendation distinct from the starter's order. You will write both the advantage estimate and the exp-weight as Kernels 2 and 3.
Each bar is a dataset action at one state; its height is the starter's capped exponential weight exp(A/λ). Green = better than estimated average, red = worse. Drag λ: small sharpens relative weights and can saturate multiple positive bars at 50; large flattens toward plain BC.
(s, a) pairs, so the objective is pulled toward observed actions. That makes critic samples a′ ~ π(·|s′) more likely to be near the data, but not guaranteed: categorical policies can put mass on unseen choices and continuous policies have tails. The cycle is an empirical risk to monitor, not a resolved support proof.(s, a) pairs rather than sampled policy actions?AWAC works, but its critic target still samples a′ ~ π, and a learned actor can wander outside observed support. IQL — Implicit Q-Learning (Kostrikov et al., 2021) — removes that sampled next-action query from its value backup. Its Bellman target uses a state-only value function V(s′), so no action is chosen in that particular value loop. The question becomes: how do you train that V? The answer is IQL's signature: expectile regression.
If you regress V(s) toward the Q-values of the dataset's actions with ordinary squared error, the minimizer is the mean:
But the mean is too pessimistic. The dataset at a state mixes good actions and mediocre ones; the mean dilutes the good with the bad. We don't want the average dataset action's value — we want the value of the good dataset actions. Not the mean, and not the max (the max would extrapolate OOD, the very thing we're avoiding). Something between the mean and the max.
The expectile is that "something between." It is an asymmetric statistic computed with squared — not absolute — error, which keeps it smooth for gradient descent. It is not a percentile: its position depends on weighted residual magnitudes, not on a count of samples. The expectile loss with parameter τ ∈ (0,1) is:
The 𝟙[u<0] is the indicator — 1 when u is negative, 0 otherwise. So the weight in front of u² is:
u > 0 (Q above V — V is too low): weight is τ;u < 0 (Q below V — V is too high): weight is 1 − τ.With τ = 0.9, being too low is penalized 9× more than being too high. So to minimize loss, V gets pushed up, but the balance is not a percentage-count rule. For the five values negative two, zero, one, three, and eight, the mean is two and the zero point nine expectile is about five point six nine; only one of the five values lies above it. That is the upper expectile: a soft preference for high learned values among the dataset actions.
τ = 0.9 learns an upper statistic of the learned Q estimates for dataset actions — optimistic toward good actions without asking the actor for a next action. At τ = 0.5 the two weights are equal and you're back to MSE (the mean). Sweep τ toward 1 and the scalar fit often rises toward the largest input Q, but a neural V has no universal range guarantee when it generalizes.The dots are Q estimates for dataset actions at one state. The fitted V (vertical line) is the τ-expectile. At τ = 0.5 it lands on the mean (dashed). Drag τ up and this scalar fixture rises toward the largest input; it is not a percentile or a neural-network range guarantee. The shading shows the asymmetric penalty: Q above V receives τ, while Q below V receives 1−τ.
Once V is trained by expectile regression on dataset (s, a) pairs, IQL's Q-target becomes:
Notice what's gone from this backup: no a′, no policy sample, and no min(Q, Q)(s′, a′). V(s′) is a function of the next state alone. The Q and V training path uses dataset actions for the current-state critic terms, so IQL removes this sampled next-action query. It does not make learned Q values exact, prevent the actor from generalizing, or rule out every OOD evaluation elsewhere.
V with expectile regression (τ near 0.9) instead of plain MSE?With the expectile V in hand, the rest of IQL falls out cleanly. IQL runs three learned updates plus a target refresh per step, in a deliberate order, over the critic, value, actor, and target roles.
| Network | Role | Trained by |
|---|---|---|
| Q1, Q2 (+ targets Q̄1, Q̄2) | estimate Q(s,a) at dataset actions | TD toward y = r + γ(1−d)V(s′) |
| V (state-only) | upper expectile of Q over dataset actions | expectile regression toward Q̄(s,a) |
AWAC had two roles (actor plus critic). IQL adds the V-network as its own component — that extra network lets the Q backup avoid sampling a next action. It reduces one OOD query; it does not buy universal OOD freedom for every actor or evaluation query.
V regresses (by the expectile loss) toward the target Q at the dataset action:
Why target Q, not online Q? Same reason as any target network: if V regressed onto the online Q it would chase a target that moves every step. The slow target Q is stable on the timescale of one V update. The seven-kernel sequence writes this as Kernel 4 alongside the expectile loss; the next kernel is the Q update.
Now the payoff. The Q-target uses V(s′) directly:
This is the same terminal-masked, stop-gradient Bellman target from Chapter 4 — but with the sampled min(Q, Q)(s′, a′) machinery replaced by a single call to V(s′). No policy sample enters this backup. You'll write this as Kernel 5.
Here is the elegant part: IQL extracts its policy exactly like AWAC does. Advantage-weighted regression, same equation:
The main difference from AWAC is how the advantage is computed. AWAC uses a single-sample V(s) ≈ Q(s, aπ) baseline, which adds sample noise. IQL uses a learned V-network, so it removes that one-sample baseline noise. This may make the actor signal cleaner, but it does not guarantee lower total estimation error or faster convergence.
V using the current target Q; (2) extract the actor using the just-updated V; (3) update Q bootstrapping on V(s′); (4) Polyak-update the target Qs. Each reads the freshest available value and moves one step. And crucially: nowhere does IQL sample a′ ~ π(·|s′). The policy never enters the value loops at all — it only consumes the trained Q and V to decide which dataset actions to imitate.An abstract role graph, not an animation of update order. V regresses toward the expectile of target Q; Q regresses toward r + γV(s′); the policy reads Q and V for its advantage. The arrows show these dependencies; the policy consumes values but no sampled next action feeds the Q target. That removes one OOD query from the value backup.
a′ ~ π(·|s′) sampled inside the value-learning loop?Both algorithms extract their policy with the same weighted-likelihood form. Their main difference is how they compute the baseline and Q target. That design difference motivates an experiment; it does not decide a universal winner in advance.
| Component | AWAC | IQL |
|---|---|---|
| Actor loss | Identical: −mean( log π(a|s) · exp(A/λ) ) | |
| Q-target | r + γ min(Q̄1,Q̄2)(s′,a′), a′ ~ π | r + γ V(s′) — no actor |
| V estimate | Q(s, aπ) — single MC sample | separate V-net, expectile regression |
| Networks | actor + 2Q + 2 target Q = 5 | actor + 2Q + 2 target Q + V = 6 |
| OOD risk | possible if the actor drifts | no sampled next action in the Q backup; other generalization risk remains |
| Advantage variance | one-sample baseline noise | that baseline noise is removed; total variance is task- and training-dependent |
| Knobs | λ only | τ (expectile) and λ |
| Compare with | matched seeds, checkpoints, returns, goal rate, lengths, and trajectory inspection | |
On a short task, AWAC's sampled-action exposure may matter less. On a long-horizon task like a medium maze, TD errors can compound over many bootstrap steps. AWAC's actor may drift and its critic may evaluate slightly OOD a′; IQL removes that sampled next-action query from its value backup. Whether this produces more reliable values or higher return is an empirical question.
a′ ~ π(·|s′), which can drift outside the data as training progresses, letting OOD evaluation errors enter and propagate. (2) IQL replaces that sample with a learned V(s′) trained from dataset action values, removing the sampled next-action query in the Q backup. (3) On longer-horizon tasks, test whether that narrower value-learning path improves final performance rather than treating the mechanism as a theorem.Tuning IQL's expectile is a real deliverable. The pattern:
| τ | V learns | Effect |
|---|---|---|
| 0.2 | lower expectile | baseline shifts down; relative actor weights still depend on Q differences |
| 0.5 | the mean (MSE) | still weighted by Q minus the mean, so not plain BC |
| 0.7–0.9 | upper expectile | often a useful comparison range; measure the resulting policy |
| 0.99 | near upper tail | may be sensitive to noisy Q estimates; verify finite values and returns |
The assignment's useful comparison is to run at least two expectiles, such as τ = 0.2 and τ = 0.9, under the same seeds and checkpoints. The direction and size of any return difference belong in the measured results, not in this schematic lesson.
These curves are hard-coded teaching fixtures, normalized to a generic return axis. They do not report AWAC or IQL runs. The dashed −46 reference is a PointMass archive statistic and should not be overlaid as an AntMaze scale. Use matched seeds, final checkpoints, and trajectory figures for a real comparison.
A = Q(s,a) − V(s) compared with AWAC?Everything comes together here. This is a schematic payoff: a value-guided policy is shown stitching two compatible half-paths into a candidate route to the goal — a route that appears in no single trajectory of the dataset. A real learned rollout must be measured in the environment.
The arena draws the same two illustrative trips from Chapter 0: trip A does the first leg well then stalls; trip B starts elsewhere and finishes well. Behavior cloning is state-conditioned, so it can generalize, but ordinary cloning does not use return to prefer a continuation at a shared state. The schematic IQL path uses a radial field and a marked handoff to show the intended mechanism: a value estimate can rank B's logged continuation above A's stall when the state and dynamics match. This is a concept illustration, not an executed trained policy.
Press Play to inspect the BC and IQL schematic paths. The displayed curves, handoff, and radial field are synthetic fixtures; they are not trained rollouts or a learned value network. Toggle the field to see the visual explanation being illustrated, then use real environment evaluation to test a policy.
And the mechanism is exactly the value function from Chapter 3, learned safely by the expectile trick from Chapter 6, extracted by the advantage-weighting from Chapters 5 and 7. Every piece you built is on screen at once.
The debugging reality of offline RL. When training misbehaves, the symptom usually points straight at one of the ideas in this lesson. Read the symptom, find the cause.
| Symptom | Likely cause | Fix |
|---|---|---|
| Q-values climb without bound; eval return crashes | Possible OOD selection, max instead of min, bad reward scale, terminal mask, finite-value, or shape issue | Inspect queries and masks; use the intended min or IQL state-value target; check optimizer groups and finite diagnostics |
| Critic loss explodes to 1e6+ | A missing no_grad is one possibility, alongside reward scale, nonfinite outputs, masks, optimizer parameters, and broadcasting | Detach the target, then check target gradients, optimizer ownership, values, masks, and shapes |
| Q near goal creeps up; erratic near terminals | Possible missing (1−d) mask or a mismatch between goal and time-limit terminal semantics | Use the task's supplied terminal/truncation convention and verify the Bellman mask |
| Policy just reproduces the data; no improvement | Low τ changes the V baseline but does not automatically yield BC; also inspect Q differences, λ, weight clipping, and data coverage | Compare expectiles and temperatures under matched seeds; log advantages and weights before tuning |
| V unstable, tracks noise | Possible high expectile with noisy Q estimates, finite-value failure, or an optimization issue | Check Q/V ranges and gradients; compare a lower expectile such as 0.9 with the task's protocol |
| Actor loss is positive | For categorical log probabilities, a positive negative-log-likelihood is ordinary; continuous log densities can be positive or negative | Check the objective sign, finite values, action distribution, and gradients with a controlled example |
inf/nan in the actor loss | exp(A/λ) may overflow, or inputs may already be nonfinite | Check finite advantages; distinguish the starter's post-exp cap from a robust pre-exp log clamp |
| Shape-mismatch silent garbage (loss "fine" but never learns) | [B] reward broadcast against a [B,1] Q → [B,B] | .squeeze(-1) the V/Q outputs to [B]; print shapes |
exp(A/λ), while categorical mass and continuous tails still need support diagnostics.V from learned Q estimates at dataset actions; this narrows one extrapolation path, but does not guarantee correct values or a hard-task win.This lesson completes the CS224R homework companion sequence. It builds on HW1 (Imitation Learning: BC, Flow Matching, DAgger) and HW2 (Online RL: Q-learning, PPO, off-policy actor-critic) — the Bellman recursion you saw scaled to neural nets there is exactly the engine that, here, learns a value function from a fixed dataset. Related Gleams: Offline RL, Off-Policy Actor-Critic, RL Algorithms.