Stop learning a value function and then arguing about it. Optimize the policy directly — one elegant identity turns "good trajectories should be more likely" into a gradient you can actually estimate from rollouts.
For two lectures you have learned to be a value-function person. You estimate Q(s, a) — "how good is taking action a in state s?" — and then you act by picking the action with the biggest Q. Q-learning, DQN, all of it, is built on that one move: learn the value, then read the policy off it.
That move quietly assumes one thing: that you can take a max over actions. With four buttons on a joystick, fine — check all four Q-values, pick the biggest. But a robot arm's action is a vector of seven continuous joint torques. There is no list of actions to scan. "Take the max over a continuous, seven-dimensional space" is its own hard optimization problem, solved inside every single decision, at fifty hertz, forever.
Here is the cleanest example, straight from the slides. The Linear Quadratic Regulator (LQR) is a classic control problem — quadratic costs, linear dynamics. When you solve it, two objects fall out. The optimal value function is a quadratic form in the state. The optimal policy is just a linear map: action = −K times state. One is a fat matrix sandwiched in a quadratic; the other is a single gain matrix K.
Which would you rather learn? A short linear rule, or the bulky quadratic surface whose only purpose is to imply that linear rule? Sometimes the policy is simply the smaller, friendlier object. Learning it directly skips a hard intermediate problem.
A one-state world with three actions; only the middle one pays off. The bars are the policy's action probabilities — what π would output. Press Step to nudge the policy by the policy-gradient rule (rewarded actions get pushed up). Watch a vague, near-uniform policy sharpen onto the good action. No Q-table, no max — the policy itself moves.
Notice what the demo is doing and what it is not. It never asks "what is the value of being in this state?" It only ever asks "did that action lead to reward? then make it more likely." That single instinct — turn up the probability of actions that paid off — is everything. The rest of this lesson is the math that makes it precise and the engineering that makes it usable.
Before we can take a gradient, we need a thing to take the gradient of. Let's nail down the objective with no hand-waving, because every later step refers back to it.
Our policy is πθ(a | s) — a probability distribution over actions, with knobs θ (the network weights). A run of the robot produces a trajectory τ = (s0, a0, s1, a1, …). Each trajectory earns a return R(τ) — the total reward summed along it. Different runs give different trajectories because the policy is random and the world is random.
So the only sensible thing to maximize is the expected return — the average return if you rolled out this policy over and over:
Read it aloud: J of theta is the average total reward of trajectories drawn from the policy with parameters theta. The subscript τ ∼ pθ(τ) is doing real work — it says the trajectories are sampled using this very policy. Change θ and you change which trajectories you tend to see. That coupling is the whole difficulty, and we will return to it.
Stare at that product. Only the middle factor — the policy — depends on θ. The start distribution p(s0) is fixed by the environment; the transition dynamics p(st+1|st, at) are the laws of physics, which the robot does not control. We will use that fact in Chapter 3 to make most of this expression vanish. Tuck it away.
Yes — with the most honest tool there is, Monte Carlo sampling. Roll out the current policy N times, get N trajectories, average their returns. That sample average is an unbiased estimate of the true expected return.
This is the same trick you would use to estimate "the average roll of a die": you cannot reason it out from a formula you do not have, so you roll many times and average. Evaluating J is easy. The hard part — the part this whole lecture is about — is getting its gradient ∇θJ(θ), so we know which way to nudge θ.
Everything in this lecture hinges on one tiny calculus identity. The slides call it the "useful identity"; in the wider literature it is the log-derivative trick or the score-function trick. It is two lines, and once you have it the policy gradient theorem is almost free.
Start from the ordinary derivative of a logarithm. For any positive function f(θ), the chain rule gives:
Now just multiply both sides by f(θ) to clear the denominator:
That is the whole trick. Apply it with f = pθ(τ), the trajectory probability:
The quantity ∇θ log πθ(a | s) that this produces has a name worth knowing: the score function. Intuitively it points in the direction of θ-space that would make the action a more probable in state s. Push θ along the score and you turn up the dial on "do a here." That phrase — "the direction that makes this action more likely" — is the mental image to carry for the rest of the lecture.
Make it real. Suppose in some state the policy is a softmax over two actions with a single logit knob θ: P(left) = σ(θ), P(right) = 1 − σ(θ), where σ is the logistic function. Then a short calculation gives the scores:
Read the meaning off the signs. If you took left, the score is positive — raising θ makes left more likely, exactly the action you took. If you took right, the score is negative — you'd lower θ to make right more likely. The score always points "toward making the action I just took more probable." Multiply it by how good the outcome was, and you have a learning signal.
A two-action softmax policy with one logit θ (slider). The two bars are P(left) and P(right). The arrows are the score ∇θlogπ for each action — right-pointing means "raise θ to make me likelier". Slide θ and watch the scores: the action that is currently rare has the bigger-magnitude score (more room to grow).
This is the "black-board derivation" the slides promise. It is three steps, and at the end something beautiful happens: the unknown physics cancels out completely. Follow each line; none of them is harder than the chain rule.
By the definition of expectation, the expected return is the return of each trajectory weighted by how probable that trajectory is, summed (integrated) over all trajectories:
Push the gradient inside the integral. Only pθ(τ) depends on θ (the return R(τ) is just a number for a given trajectory), so:
Now use the log-derivative trick from Chapter 2 to replace ∇θ pθ(τ) with pθ(τ)·∇θ log pθ(τ):
The leading pθ(τ) re-creates an expectation. So:
Recall the trajectory probability from Chapter 1: a product of the start, the policy at each step, and the transition at each step. The log of a product is a sum of logs:
Now take the gradient with respect to θ. Look at each piece:
Substitute back, and you have the Policy Gradient Theorem in its clean form:
Read it as a sentence: the gradient of expected return is the average, over trajectories, of (how good the trajectory was) times (the sum of scores of every action you took in it). Good trajectories (big R) push their actions' scores up hard; bad ones push gently or down. That is the entire algorithm in one line.
The theorem gives an expectation. We already know how to turn an expectation into something computable: Monte Carlo sampling. Roll out N trajectories with the current policy, and replace the expectation with the sample average. That is the entire REINFORCE algorithm (Williams, 1992; the slides credit the Sutton et al. lineage):
Spelled out as a loop:
Time to make the estimator concrete in code. The lab below puts a one-step policy on a 2-armed bandit — the simplest world where the policy-gradient idea is visible end to end. You will assemble the REINFORCE estimate from its two ingredients (return times score) and then check that following it actually raises expected reward.
The formula is correct, but you should never trust a formula you cannot feel. Let's build the intuition the slides emphasize, and then compute one full REINFORCE step by hand so the abstraction touches the ground.
Look again at the REINFORCE update: each action's score — the "make me more likely" direction — is weighted by the return R(τ) of the trajectory it lived in. So:
That is it. Good stuff is made more likely; bad stuff is made less likely. Guanya Shi's phrase: REINFORCE "simply formalizes the notion of trial and error." There is no clever credit-assignment machinery here — just "did the whole episode go well? then do more of what you did."
Two actions, one state, softmax policy with logit θ = 0, so π(left) = π(right) = 0.5. The scores at θ = 0 (from Chapter 2, with σ(0) = 0.5):
We sample 4 one-step episodes. Suppose the outcomes are:
| Episode | Action | Score | Return R(τ) | R · score |
|---|---|---|---|---|
| 1 | left | +0.5 | +1 | +0.5 |
| 2 | left | +0.5 | +1 | +0.5 |
| 3 | right | −0.5 | 0 | 0 |
| 4 | right | −0.5 | 0 | 0 |
The REINFORCE gradient estimate is the average of the last column:
It is positive, so gradient ascent raises θ: θ ← 0 + η·0.25. Raising θ raises π(left). And indeed left was the rewarding action — the estimator correctly decided to make left more likely. With learning rate η = 1, θ moves to +0.25, so π(left) climbs from 0.50 to σ(0.25) ≈ 0.562. One step, one nudge in the right direction. Repeat and it sharpens onto left.
The same four episodes. Toggle each episode's return between the rewarding action being left or right, and watch the running gradient estimate — and which way θ (hence π) would move. Confirm: the gradient always points toward making the rewarded action likelier.
REINFORCE is unbiased: average enough rollouts and the estimate converges to the true gradient. But "enough" is the catch. The slides put it bluntly: "What is wrong with REINFORCE? It has high variance!" Two trajectories from the same policy can produce wildly different gradient estimates, and you are averaging only a handful of them.
Three compounding reasons:
Each warm dot is a REINFORCE gradient estimate from a small batch of N noisy episodes; the teal line is the true gradient. Shrink the batch (slider) and watch the estimates spray all over — some even point the wrong way. The cloud's spread is the variance.
The first variance reduction is so obvious in hindsight that it feels like cheating — and it costs us nothing in bias. It comes from a single physical fact: the future cannot influence the past.
In vanilla REINFORCE, every action at is weighted by the entire trajectory return R(τ) — including rewards that arrived before that action was even taken. But a decision at time t' cannot possibly have caused a reward at an earlier time t < t'. Weighting at by rewards from the past is pure noise: it adds variance without adding any real credit.
Formally, replace the single trajectory return R(τ) multiplying action at with the sum of rewards from t onward:
That inner sum — the reward-to-go from step t — is exactly an estimate of the action-value Q(st, at): "expected return if I take this action here and continue." So a second, equivalent way to write the policy gradient theorem is:
A 6-step episode with a reward at each step (bars). Click an action step. Red = the rewards vanilla REINFORCE wrongly credits to it (including past ones); teal = the reward-to-go (only t' ≥ t) that causality says it can actually influence. The red-only past rewards are the noise we throw away.
Now let's verify the policy-gradient estimator is actually the gradient — by checking it against a brute-force finite difference. This is the gold-standard correctness test for any gradient estimator, and it's how you'd debug your own implementation.
Here is the most important — and most misunderstood — variance trick. We are going to subtract a number from the return before weighting the score, and somehow that lowers variance without changing the gradient we are estimating. It sounds like a free lunch. The slides prove it is real.
Pick any function b(s) that depends only on the state, not the action — call it a baseline. Subtract it from Q before weighting:
The astonishing part: this equals the gradient without the baseline. The reason is that the extra term we added — E[ b(s)·∇θ log πθ(a|s) ] — is exactly zero. Here is the slides' proof, in plain steps:
Walk it: the log-derivative trick turns ∇logπ back into ∇π/π, the π cancels, b(s) pulls out front (it doesn't depend on a), the gradient swaps with the integral, and the integral of a probability distribution over all actions is 1 — a constant, whose gradient is 0. The baseline term vanishes identically. Any b(s) is allowed.
Without a baseline, suppose every trajectory in some region has return between +90 and +110. Vanilla REINFORCE pushes every action's probability up (all returns positive), then sorts them out only by the small differences. Huge weights, tiny relative signal — that is high variance. Subtract the average return, ~100, and now returns become −10 to +10: above-average actions go up, below-average go down. Same expected direction, far smaller weights, far less noise.
The slides give the most common choice: b(s) = V(s), the state-value function. Then Q(s, a) − V(s) is the advantage A(s, a) — "how much better was this action than the state's average action?" That advantage-weighted gradient is the backbone of every modern policy-gradient method (A2C, PPO).
Two clouds of gradient estimates from the same noisy episodes: warm = no baseline, teal = baseline subtracted (the average return). The dashed line is the true gradient. Slide the return offset up — without a baseline the warm cloud's spread explodes; the teal cloud stays tight because it cancels the offset.
Let's prove the free lunch with our own numbers: measure the variance of the gradient estimate with and without a baseline, and confirm the baseline cuts variance while leaving the mean (the true gradient) untouched.
You now own a complete, self-contained method: a way to improve a policy directly, derived from one calculus identity, made practical by Monte Carlo, and de-noised by two unbiased tricks. Let's place it in the landscape and hand you the one-page summary.
| Q-Learning (value-based) | Policy Gradient (policy-based) | |
|---|---|---|
| What it learns | A value Q(s,a); policy is implicit (argmax) | The policy πθ directly |
| Continuous actions | Awkward (max over actions) | Natural (output the action) |
| On/off-policy | Off-policy — can replay an old buffer | On-policy — needs fresh rollouts |
| Model | Model-free | Model-free (physics cancels in the gradient) |
| Main weakness | Instability with function approximation | High variance, sample-hungry |
| Stochastic policy | Not naturally | Yes — it is a distribution |
Neither family dominates. The future of the course is the marriage of the two: keep the policy-gradient structure (the score weighted by how good the action was) but replace the noisy reward-to-go with a learned value estimate — a critic. That hybrid is actor-critic, the subject of the very next lecture, and the foundation of PPO and SAC that run real robots today.
This lesson sits between the value-based and actor-critic chapters of the 16-831 series. Continue the thread:
"Make the good trajectories more likely. The rest is variance reduction."