Actor-Critic Methods — Introduction to Robot Learning (CMU 16-831)

Chapter 0: A Gradient You Can Barely See

In Lecture 9 we built REINFORCE: roll out a whole episode, measure its total return, and nudge the policy so that every action it took becomes more likely in proportion to how good the whole episode turned out. It works. It is also, in the instructor's blunt words, high variance.

Here is the problem in one picture. Suppose a robot took a perfectly sensible action early in an episode, but then — thirty steps later, for unrelated reasons — the episode ended in disaster. REINFORCE doesn't know which step was to blame. It credits the entire trajectory's return to every action in it. A good action inside a bad episode gets punished; a lucky-but-dumb action inside a good episode gets rewarded.

The noise is not a detail — it is the bottleneck. Because each gradient estimate is built from one or a few full-episode returns, two runs of the same policy can produce gradients pointing in opposite directions, purely from luck in the dice the environment rolled. You can average many rollouts to calm it, but each rollout costs real robot time. On a physical robot you cannot afford ten thousand episodes to average away the noise.

So the lecture asks a sharp question. Could we keep a running estimate of how good a state is — a learned "expected score from here" — and use it to subtract off the boring, predictable part of the return? Then the gradient would only respond to the part that is genuinely surprising: did this action do better or worse than what we already expected from this state?

That single idea — a learned judge that says "from here, you should expect about this much" — is the critic. The policy that chooses actions is the actor. Train them together and you get the actor–critic family. Before any math, feel the variance problem yourself.

Same policy, same state — watch the gradient signal jump around

Each Roll samples one episode's return and plots the gradient weight REINFORCE would use on the first action. Toggle Use a critic baseline to subtract the state's expected value first. With the baseline, the weights cluster near zero and only the genuinely good/bad actions stand out.

Use a critic baseline (subtract expected value)

Ready.

Where we are headed. Every method in this lecture is one knob on the same tradeoff: how much do we trust a learned estimate (low noise, possibly biased) versus the raw observed return (unbiased, very noisy)? The critic, the advantage, n-step returns, and GAE are four points on that dial. Keep the dial picture in your head.

Why does REINFORCE produce such noisy gradient estimates?

Because neural networks are inherently random. Because the learning rate is always too high. Because it weights each action by the whole episode's return, so an action's update depends on everything that happened before and after it — mostly luck unrelated to that action. Because it never uses a discount factor.

Chapter 1: The Trick That Costs Nothing — Baselines

Let us write down where Lecture 9 left us. The policy gradient, after applying causality (an action can only be credited with rewards that come after it), is an expectation over states and actions:

∇_θ J(θ) ∝ E_{s∼p_π, a∼π} [ Q^π(s,a) · ∇_θ log π_θ(a|s) ]

Read it slowly. The term ∇_θ log π_θ(a|s) is the direction in parameter space that makes action a more likely in state s. The scalar Q^π(s,a) — the action-value, the expected discounted return if you take action a in state s and then follow the policy — is the weight: how hard to push in that direction. In REINFORCE we estimate Q with the observed reward-to-go from each sampled action.

Now the trick. The instructor proves a small, beautiful fact: you can subtract any function b(s) that depends only on the state — called a baseline — and the gradient's expectation does not change at all:

∇_θ J(θ) ∝ E_s,a [ ( Q^π(s,a) − b(s) ) · ∇_θ log π_θ(a|s) ]

Why subtracting b(s) is free

The reason is one line of algebra. The extra term we introduced is the expectation of b(s) · ∇_θ log π_θ(a|s). Because b(s) does not depend on the action, we can pull it outside the sum over actions, and what is left is the policy's own gradient summed over all its actions:

∑_a π_θ(a|s) · ∇_θ log π_θ(a|s) = ∑_a ∇_θ π_θ(a|s) = ∇_θ ∑_a π_θ(a|s) = ∇_θ 1 = 0

The middle step uses the identity π · ∇ log π = ∇ π (the log-derivative trick). The action probabilities always sum to 1, and the gradient of the constant 1 is zero. So the baseline contributes nothing in expectation — it is unbiased.

Free lunch, almost. Subtracting b(s) changes nothing in expectation, but it absolutely changes the variance of the single-sample estimate. Pick b(s) well and the variance plummets; the gradient stops swinging wildly run-to-run. That is the entire payoff — same target, far steadier aim.

So which b(s) is best? Intuitively: subtract off "how good this state typically is", so that the weight on an action measures only whether it beat the state's average. The natural choice is the state-value function V^π(s) — the expected return from state s when you simply follow the current policy. That is the seed of the critic, and the subject of the next two chapters.

Slide the baseline — watch the gradient variance collapse

Returns from one state scatter around their mean (the dashed line). The gradient weight is return − baseline. Slide the baseline toward the mean and the bars shrink toward zero spread — the estimate gets quieter without moving its average.

Baseline b0.00

…

Subtracting a state-dependent baseline b(s) from the policy gradient…

changes which policy we converge to. leaves the gradient's expectation unchanged (it stays unbiased) but can sharply reduce its variance. only works if b depends on the action too. always increases the variance.

Chapter 2: The Advantage — How Much Better Than Average?

Set the baseline to the state-value: b(s) = V^π(s). The weight on each action becomes a quantity with its own name and its own deep meaning — the advantage function:

A^π(s,a) = Q^π(s,a) − V^π(s)

Translate it into English. Q^π(s,a) is "expected return if I take action a here, then follow my policy." V^π(s) is "expected return if I just follow my policy from here" — the average over the actions my policy would have picked. Their difference is exactly:

The advantage is "how much better than average was this action?" A^π(s,a) > 0 means action a beat the policy's own typical choice in this state — make it more likely. A^π(s,a) < 0 means it underperformed the average — make it less likely. A^π = 0 means "exactly as good as usual, no signal." This is the cleanest possible learning signal: it has already factored out everything about the state that the action had no control over.

With the advantage, the policy gradient takes the form that the rest of deep RL is built on:

∇_θ J(θ) ∝ E_s,a [ A^π(s,a) · ∇_θ log π_θ(a|s) ]

Why this kills variance — concretely

Imagine a state where every action leads to a return around +100, give or take a little. Raw REINFORCE weights every action by ~+100 and shoves them all up hard — even though the policy learns nothing about which action to prefer (they're all the same). The advantage, by contrast, subtracts the ~+100 average, leaving small numbers that say "this one was +2 better, that one was −3 worse." The huge, shared, uninformative offset is gone. What remains is pure signal.

Raw return vs. advantage — the offset that teaches nothing

Three actions in one state, each with a noisy return around a high common mean. Warm bars = raw return (the weight REINFORCE uses). Teal bars = advantage (return minus V). Crank the common return level: the warm bars grow huge and nearly equal; the teal bars stay small and show the real ranking.

Common return level V60

…

In a state, taking action a yields expected return Q = 12, while the policy's average from that state is V = 10. The advantage A(s,a) is…

120 — multiply Q by V. −2 — the action is worse than average. +2 — the action does 2 better than the policy's average, so make it more likely. 0 — advantage is always zero on average over actions.

Chapter 3: The Critic — Learning V^π(s)

We want the advantage, which needs V^π(s). But V^π is exactly the thing we don't know — if we knew the value of every state we'd nearly have the problem solved. So we learn it. A second network, the critic V_φ with its own parameters φ, takes a state and outputs a single number: its estimate of the expected return from that state.

How do we train it? This is just policy evaluation — the problem from Lecture 7. The most direct option:

Option 1 — Monte Carlo regression

Roll out the policy, record the actual reward-to-go from each state, and fit the critic to those numbers by regression. Minimize the squared error between the critic's prediction and the observed return:

L(φ) = ½ ∑ ( G_t − V_φ(s_t) )²

where G_t is the true observed discounted reward-to-go from state s_t. This is unbiased — G_t is a real sample of the return — but it inherits the same fat variance as REINFORCE, because each G_t is a full noisy rollout. We have moved the noise, not removed it. The instructor asks: can we do better?

The critic does a different job from the actor. The actor learns what to do (a distribution over actions). The critic learns how good a situation is (one number per state). The actor is trained by the policy gradient; the critic is trained by regression toward returns. They are two networks solving two different supervised-ish problems, coupled only through the data the actor generates and the advantage the critic supplies. We map the full data flow in Chapter 5.

The danger: a wrong critic

There is a catch the lecture flags explicitly. We use V_φ inside the advantage that steers the actor. If V_φ is a bad estimate of the true V^π, the advantage is wrong, and the actor is pushed in a wrong direction. Subtracting an inaccurate baseline can even add bias — the very thing baselines were supposed to avoid. So the critic must be good enough. This tension — a learned critic reduces variance but can introduce bias — is the heartbeat of the whole lecture, and we make it precise next with the TD trick.

What is the critic's job in actor–critic, and how is the simplest version trained?

It picks actions; it is trained by the policy gradient. It picks actions; it is trained by regression to returns. It estimates the expected return from a state, V(s); the Monte Carlo version is trained by regressing it onto the observed reward-to-go. It estimates the transition dynamics of the world.

Chapter 4: Bootstrapping — the TD Error

"Can we do better?" than waiting for a full noisy rollout? Yes — and the idea is the single most important move in value-based RL. Recall the Bellman equation: the value of a state equals the immediate reward plus the discounted value of wherever you land next:

V^π(s) = E_a∼π [ r(s,a) + γ V^π(s′) ]

So instead of waiting for the whole future, take one real step, observe the reward r and the next state s′, and use the critic's own estimate of the rest. That gives a target:

y = r + γ V_φ(s′)

and we train the critic to match it: L(φ) = ( y − V_φ(s) )². Using your own current estimate to build the target for your own update is called bootstrapping — pulling yourself up by your own bootstraps. The target y is a bootstrapped estimate: one real reward plus a guess about everything after.

The bias–variance trade, made concrete. The Monte Carlo target uses the entire observed future — unbiased, but it carries the noise of every random reward and transition to the end of the episode (high variance). The one-step bootstrap target uses one real reward and then trusts V_φ — far less noise (low variance), but if V_φ is wrong, the target is wrong too (bias). The critic trades a little bias for a lot less variance. On a robot, where each sample is expensive, that is usually the right trade.

The TD error is a one-step advantage

Here is the payoff that ties everything together. Plug the one-step target into the advantage, A = Q − V, using the same Bellman idea for Q(s,a) ≈ r + γV(s′):

A(s,a) ≈ ( r + γ V_φ(s′) ) − V_φ(s) = δ

This quantity δ (delta) is the temporal-difference (TD) error: the gap between what we actually got plus where we landed, and what we expected. And it does double duty:

It is the critic's training signal: nudge V_φ(s) toward y = V_φ(s) + δ, i.e. shrink δ.
It is the actor's learning signal: a one-step estimate of the advantage A(s,a). Positive δ → the action did better than expected → make it more likely.

One number, two jobs. The TD error δ trains the critic and steers the actor. That single shared scalar is why actor–critic is so elegant: a unified currency of surprise that both networks consume. There is no max in sight (unlike Q-learning), which is why the critic update is more stable — it evaluates the policy you actually have, rather than chasing a moving greedy target.

Bootstrap vs. Monte Carlo — bias and variance, side by side

A true value (dashed). The Monte Carlo estimate is centered on it but scatters widely. The one-step bootstrap is tight but offset by the critic's error. Slide the critic error up: the teal cloud stays tight but drifts off-target (bias); the warm cloud stays centered but never gets quiet (variance).

Critic error (bias)0.20

…

The TD error δ = r + γV(s′) − V(s) is positive. What does that mean and what do we do?

The episode is over; stop training. Things went better than the critic expected — raise V(s) toward the target, and make this action more likely (positive one-step advantage). The action was bad; make it less likely. The discount γ is too large.

Now build the advantage from raw returns yourself and watch its variance fall against the baseline.

Chapter 5: Two Networks, One Loop — the Data Flow

Let us make the architecture concrete — what I cannot create, I do not understand. Actor–critic is two function approximators sharing one experience stream. Here is exactly what flows where, with shapes.

Observation o (shape: obs_dim)

The state the robot sees this step.

↓ fed into BOTH networks

Actor π_θ(a|o) → action distribution (shape: action_dim)

Outputs e.g. a Gaussian's mean+std, or softmax logits. Sample an action a.

Critic V_φ(o) → one scalar

Outputs a single number: expected return from o.

↓ take action a in the world; observe r and next obs o′

TD error δ = r + γV_φ(o′) − V_φ(o) (scalar)

The shared currency of surprise.

↓ δ splits two ways

Critic update: φ ← φ + α_v · δ · ∇_φV_φ(o)

Regress V toward the bootstrapped target r + γV(o′).

Actor update: θ ← θ + α_π · δ · ∇_θlog π_θ(a|o)

Push action a more likely if δ > 0, less if δ < 0.

↻ move to o′, repeat

Notice the design decisions, each made for a reason:

Why two separate heads? The actor outputs a distribution over actions; the critic outputs a single value. Different output shapes, different losses. Many implementations share the early layers (one trunk, two heads) so perception features are reused, then split.
Why a separate actor learning rate α_π vs. critic α_v? The critic usually learns faster — a wrong critic poisons the actor's advantage, so you want V to be roughly right before trusting its δ.
Why "no max"? The critic evaluates the current policy (V^π), not the best possible action (as Q-learning's max does). Evaluating a fixed target is more stable than chasing a greedy one.

Is actor–critic on-policy or off-policy? The standard version is on-policy. The gradient is an expectation over states and actions drawn from the current policy π_θ. The moment you reuse stale data from an old policy, that expectation is over the wrong distribution — the gradient is biased. (Lecture's later "off-policy actor–critic" and SAC fix this by switching from V to a Q-critic and re-sampling actions from the current policy — the topic of the next lecture.)

Now implement one full step of both updates on a tiny problem and confirm the critic moves toward its target while the actor shifts toward the higher-advantage action.

Why is standard actor–critic on-policy?

Because the critic uses a max operator. Because it never uses a replay buffer by definition of the word. Because the policy gradient is an expectation over states and actions from the current policy — data from an old policy makes the gradient estimate biased. Because the actor and critic share parameters.

Chapter 6: A Worked Example, By Hand

Numbers make it real. Take a tiny scenario and compute every quantity with a calculator — an advantage, one critic update, one actor update, and an n-step return — so you could reproduce the algorithm from memory.

The setup

A robot is in state s. Our critic currently believes:

V_φ(s) = 5.0 — "from here I expect a return of about 5."
The robot takes action a, receives reward r = 2.0, and lands in s′ where V_φ(s′) = 6.0.
Discount γ = 0.9.

Step 1 — the one-step advantage (TD error)

The bootstrapped target is the reward plus the discounted next value:

y = r + γ V_φ(s′) = 2.0 + 0.9 × 6.0 = 2.0 + 5.4 = 7.4

The TD error — our one-step advantage estimate — is the target minus the current value:

δ = A(s,a) ≈ y − V_φ(s) = 7.4 − 5.0 = +2.4

Positive: the action did 2.4 better than the critic expected. Good news — we will make this action more likely.

Step 2 — the critic update

Regress V toward the target. With critic learning rate α_v = 0.5:

V_φ(s) ← V_φ(s) + α_v · δ = 5.0 + 0.5 × 2.4 = 5.0 + 1.2 = 6.2

The critic moved from 5.0 toward the target 7.4 — it now expects more from this state. Notice it did not jump all the way to 7.4; the learning rate damps the move, because the target is itself only a one-sample estimate.

Step 3 — the actor update

Say the actor is a 2-action softmax with equal logits, so before the update P(a) = 0.5. The update raises the chosen action's logit by α_π · δ. With α_π = 0.5, the logit of a rises by 0.5 × 2.4 = 1.2, so logits go from (0, 0) to (1.2, 0). The new probability of the chosen action:

P(a) = e^1.2 / (e^1.2 + e⁰) = 3.320 / (3.320 + 1.0) = 3.320 / 4.320 ≈ 0.769

The chosen action's probability climbed from 0.50 to about 0.77. The positive advantage pushed the policy toward it — exactly as intended.

You just ran one full actor–critic step by hand. One TD error δ = +2.4 did everything: it told the critic to raise V from 5.0 to 6.2, and it told the actor to raise the action's probability from 0.50 to 0.77. Had the reward been small — say r = 0 so y = 5.4 and δ = +0.4 — both moves would have been gentler; had δ gone negative, the critic would lower V and the actor would demote the action.

Step 4 — a 3-step return by hand

Now a multi-step target, the bridge to the next chapter. Suppose over three steps the robot collects rewards r₀ = 2, r₁ = 1, r₂ = 3, and the critic's value of the state after step 3 is V(s₃) = 4, with γ = 0.9. The 3-step return sums the first three real rewards (discounted) and then bootstraps with the critic:

G⁽³⁾ = r₀ + γ r₁ + γ² r₂ + γ³ V(s₃)

G⁽³⁾ = 2 + 0.9(1) + 0.81(3) + 0.729(4) = 2 + 0.9 + 2.43 + 2.916 = 8.246

Compare to the 1-step target from the same start, which would be r₀ + γV(s₁): it trusts the critic after just one real reward. The 3-step version uses three real rewards before trusting the critic — more real signal, less reliance on the (possibly wrong) critic, but more accumulated reward noise. That is the n-step knob, and we turn it next.

With V(s) = 5, r = 2, V(s′) = 6, γ = 0.9, the TD error is +2.4. If the critic learning rate is 0.5, the updated V(s) is…

7.4 — jump straight to the target. 6.2 — move halfway toward the target: 5.0 + 0.5×2.4. 5.0 — it doesn't move. 2.4 — it becomes the TD error.

Chapter 7: n-step Returns and GAE — the Dial

We now have two extremes for the advantage. The 1-step TD error uses one real reward and trusts the critic for everything after — low variance, but biased by the critic's error. The Monte Carlo advantage uses the entire observed return — unbiased, but high variance. Everything good lives in between.

n-step returns

The obvious interpolation: take n real rewards, then bootstrap. The n-step advantage is

A⁽ⁿ⁾ = r_t + γ r_t+1 + … + γⁿ⁻¹ r_t+n−1 + γⁿ V(s_t+n) − V(s_t)

Small n → near the 1-step TD error (low variance, more bias). Large n → near Monte Carlo (high variance, less bias). The lecture notes this is exactly what eligibility traces implement efficiently.

GAE — a weighted blend of all n at once

Rather than pick a single n, Generalized Advantage Estimation (GAE, Schulman et al. 2016 — the slide that "unifies PG and actor–critic") takes an exponentially-weighted average over all n-step advantages, controlled by a single knob λ in [0, 1]. The clean way to write it: let δ_t = r_t + γV(s_t+1) − V(s_t) be each step's TD error. Then

A^GAE_t = δ_t + (γλ) δ_t+1 + (γλ)² δ_t+2 + … = ∑_k≥0 (γλ)^k δ_t+k

The single parameter λ sweeps the whole bias–variance line:

λ = 0: only the first term survives. A^GAE = δ_t — the pure 1-step TD advantage. Lowest variance, most bias from the critic.
λ = 1: the discounted sum of all future TD errors telescopes into the Monte Carlo advantage (full return minus V). Highest variance, least bias.
0 < λ < 1: a smooth interpolation. In practice λ ≈ 0.95 is a sweet spot — most of the bias killed, variance still manageable.

λ is the bias–variance dial, in one number. Turn it down toward 0 to lean on the critic (quiet but possibly biased); turn it up toward 1 to lean on real rewards (honest but noisy). GAE is why modern actor–critic methods — A2C, PPO — get steady and low-bias advantages: they set λ once and let the exponential weighting blend every horizon automatically.

Sweep λ — bias and variance of the advantage estimate

The advantage estimator's bias (from trusting an imperfect critic) and its variance (from real reward noise) as λ goes 0→1. At λ=0 it is the 1-step TD advantage (low variance, high bias); at λ=1 it is Monte Carlo (high variance, ~zero bias). The total error dips in the middle — that minimum is the sweet spot.

λ0.95

…

Implement GAE yourself and confirm the two endpoints: λ = 0 reproduces the 1-step TD advantage, λ = 1 reproduces the full Monte Carlo advantage.

In GAE, what does turning λ from 1 down toward 0 do to the advantage estimate?

Moves it from Monte Carlo (high variance, low bias) toward the 1-step TD advantage (low variance, more bias). Moves it from low variance toward high variance. Has no effect — λ only changes the discount. Removes the critic entirely.

Chapter 8: Showcase — Actor and Critic, Learning Together

Everything in one place. A robot lives on a small grid. The critic learns a value for every cell (shown as a heatmap — warm = high value, near the goal). The actor learns which way to move from each cell (shown as arrows). Both train from the same TD errors, online, as the robot wanders.

Grid actor–critic — the critic's value map and the actor's arrows improve together

Press Train. The goal (teal ring) pays +1. Watch the heatmap fill in from the goal outward (the critic learning V) while the arrows swing to point along the value gradient (the actor learning the policy). Raise λ to use longer-horizon advantages; raise reward noise to see why a learned critic earns its keep.

GAE λ0.90

Reward noise0.20

Ready. The value map starts flat and the arrows random.

Two things to notice as it trains. First, value propagates backward from the goal — the cell next to the goal learns its worth first, then its neighbors, then theirs. That is bootstrapping in action: each cell's target borrows its neighbor's current estimate. Second, the arrows follow the value — the moment a cell's neighbor looks more valuable, the actor's TD error there goes positive and the arrow swings toward it. The two networks bootstrap each other into a good policy without a single full Monte Carlo rollout.

The instructive experiment. Crank reward noise to the max with λ near 1 (Monte Carlo): the value map flickers and the arrows thrash, because each update trusts a very noisy full return. Now drop λ toward 0 (lean on the critic): the map settles — quieter, even if slightly biased early. You are watching the bias–variance dial from Chapter 7 decide whether learning is stable. That is the entire lecture, live.

Chapter 9: Connections & Cheat Sheet

You started with a gradient you could barely see and ended with two networks that quiet it down by sharing one signal. Here is the whole lecture in one frame, then where it leads.

REINFORCE (Lecture 9)

Weight each action by the whole episode's return. Unbiased, very high variance.

↓ subtract a state-only baseline b(s) — free in expectation

Baseline = V^π(s)

Weight becomes the advantage A = Q − V: "how much better than average?"

↓ learn V with a second network (the critic)

Actor–Critic

Critic bootstraps via TD error δ = r + γV(s′) − V(s); δ trains the critic AND steers the actor.

↓ blend 1-step TD ↔ Monte Carlo with one knob

n-step / GAE(λ)

λ=0 is 1-step TD (low variance, biased); λ=1 is Monte Carlo (unbiased, noisy). λ≈0.95 in practice.

Cheat sheet — the vocabulary you now own.

Baseline b(s): subtract any state-only function from the gradient — unbiased, lowers variance.
Advantage A(s,a) = Q(s,a) − V(s): how much better than the policy's average this action was. Best baseline = V.
Critic V_φ(s): a learned net estimating expected return; trained by regression (Monte Carlo) or bootstrapping (TD).
TD error δ = r + γV(s′) − V(s): the surprise. Trains the critic; doubles as a 1-step advantage for the actor. No max → more stable than Q-learning.
Bias–variance: the critic trades a little bias for a lot less variance vs. Monte Carlo.
n-step / GAE(λ): one dial from low-variance/biased (small n, λ→0) to unbiased/noisy (large n, λ→1).
On-policy: standard actor–critic needs data from the current policy; reusing stale data biases the gradient.

Where to go next on this site

This lesson sits in the model-free RL arc of the 16-831 series. Continue and go deeper:

Lecture 9 — Policy Gradients — the REINFORCE estimator and variance reduction this lecture builds on.
Lecture 11 — Advanced RL (PPO & SAC) — PPO clips the actor update on GAE advantages; SAC is the off-policy, max-entropy actor–critic teased at the end of this lecture.
Actor–Critic — a from-zero companion Gleam on the same family.
Off-Policy Actor–Critic — replay buffers, a Q-critic, and re-sampling actions from the current policy (DDPG, SAC).
Policy Gradients, Q-Learning, MDPs — the surrounding model-free toolbox.

The test of this lesson. Open the original Lecture 10 slides. You should be able to explain, from memory: why a baseline is free, why V is the best one, what the advantage means, why the TD error trains the critic and steers the actor at once, why standard actor–critic is on-policy, and how λ in GAE dials bias against variance. If you can, you are ready for PPO and SAC.

"The actor chooses; the critic keeps score. Together they hear the signal through the noise."

Actor–Critic Methods