Advanced RL: TRPO, PPO, DDPG & SAC — Introduction to Robot Learning (CMU 16-831)

Chapter 0: One Bad Step and the Policy Collapses

You have a working policy gradient agent. It rolls out the current policy, estimates the gradient of expected return, and steps the parameters uphill: θ′ = θ + α ∇_θ J(θ). Everything depends on one number you picked by hand — the step size α.

Pick it too small and training crawls; you burn a million expensive robot rollouts to inch forward. Pick it too big and something much worse happens. The new policy is so different from the old one that the data you just collected — rolled out under the old policy — no longer describes the world the new policy will see. You step confidently in a direction that was only good locally, land far away, and the return falls off a cliff.

Why a bad step is unrecoverable. In supervised learning a too-large step just means a noisy loss; next batch you correct it. In RL the policy generates its own data. A bad step moves you to a region where the new policy visits bad states, collects bad rollouts, and the next gradient is computed from that garbage. The collapse feeds itself — you can't simply "try again" from clean data because the policy is the data source.

So the entire chapter we are about to walk through is one idea expressed five different ways. We want to take the biggest step we safely can — big enough to learn fast, small enough that the post-step policy still resembles the one we collected data under. That "still resembles" condition is called a trust region, and it is the spine of TRPO, PPO, and (in a different guise) DDPG and SAC.

The cliff — one step on the return landscape

The curve is true expected return as a function of how far you move the policy. The straight tangent is what your gradient thinks happens. Slide the step size: a small step lands where the gradient promised; a big step walks off the local peak and over the cliff, where return crashes.

Step size α 0.20

Small step: the gradient's promise holds.

Notice the asymmetry. Near zero, the tangent line and the true curve agree — the linear approximation behind the gradient is trustworthy. Far out, they diverge wildly. The gradient is a local object; it tells you nothing about what happens once you've moved far enough that the world looks different. A trust region is just a fence that keeps you in the zone where the local picture is still roughly true.

The thesis of the lecture. Every algorithm here answers "how big a step is safe?" differently. NPG measures distance in policy space, not parameter space. TRPO caps the KL divergence between old and new policy with a hard constraint. PPO approximates that cap with a cheap clip you can run with plain SGD. DDPG/TD3/SAC sidestep the on-policy step entirely by reusing old data from a replay buffer — which needs its own kind of care. Keep the cliff picture in your head; we hang every method on it.

Why is a too-large policy-gradient step in RL more dangerous than a too-large step in supervised learning?

Because RL uses a different optimizer than SGD. Because the learning rate is harder to type. The policy generates its own data: a bad step moves it to a region where it collects bad rollouts, so the next gradient is computed from garbage and the collapse feeds itself. Because RL gradients are always exactly zero.

Chapter 1: The Taxonomy — Where These Algorithms Live

Before the new machinery, let us place it on the map you already know. The lecture splits model-free RL into three families by what they directly estimate.

Family	What it learns	Examples
Value-based	The optimal Q-function; the policy is implicit (act greedily).	FQI, DQN
Policy-gradient	The policy directly, by ascending the return gradient.	REINFORCE, NPG, TRPO, PPO
Actor-critic	A critic for the current policy, used to improve the actor.	DDPG, TD3, SAC

Everything in this lecture is either policy-gradient (the TRPO/PPO branch) or actor-critic (the DDPG/SAC branch). Why are there so many? Because there is no perfect RL algorithm — each one trades off three things the lecture names explicitly.

Sample efficiency — how many environment steps to reach good performance. For robots, where each step is real seconds and real wear, this can dominate.
Stability & ease of use — how robust the method is to hyperparameters and random seeds. An algorithm that only works after weeks of tuning is a research toy, not a tool.
Assumptions — stochastic vs deterministic policy, continuous vs discrete actions, finite vs infinite horizon.

A subtlety about "efficient". The lecture warns: more sample-efficient does not always mean shorter wall-clock time. In a fast simulator you can generate millions of cheap samples per minute, so an inefficient-but-parallelizable on-policy method (PPO) can finish a wall-clock race against a sample-efficient-but-serial off-policy method (SAC). On a real robot, where samples are precious, the ranking flips. Match the algorithm to where your data is cheap.

The two axes that organize everything

Two binary questions sort all six algorithms cleanly, and we will return to this grid as our final cheat sheet:

On-policy vs off-policy. On-policy methods (PPO, A2C) can only learn from data the current policy generated; after one update the data is stale and thrown away. Off-policy methods (DDPG, TD3, SAC) reuse a replay buffer of old transitions, so they squeeze far more learning out of each sample — at the cost of extra care to stay stable.
Stochastic vs deterministic policy. A stochastic policy outputs a distribution over actions (it samples); a deterministic policy outputs one exact action. Stochasticity is built-in exploration; determinism is precise but needs noise bolted on to explore.

The branch we are about to climb. The on-policy / stochastic corner is where the step-size problem bites hardest, because you keep throwing away data — so you'd better make each step count and never collapse. That is exactly what NPG → TRPO → PPO fix. The off-policy corner (DDPG/TD3/SAC) attacks the other lever: reuse so much old data that sample efficiency stops being the bottleneck.

PPO is on-policy; SAC is off-policy. On a real robot where every environment step costs real seconds and hardware wear, which property of SAC matters most?

SAC uses a deterministic policy, so it is faster to evaluate. SAC reuses a replay buffer of old transitions, so it extracts far more learning per real-world sample. SAC never needs a critic. SAC requires no hyperparameter tuning at all.

Chapter 2: Natural Gradient — Distance in the Right Space

Vanilla policy gradient already has a hidden way to talk about step size. We can rewrite "step uphill with learning rate α" as a constrained optimization: maximize the linearized improvement, subject to not moving the parameters too far.

max_θ′ (θ′ − θ)^⊤ ∇_θ J(θ) s.t. ‖θ′ − θ‖² ≤ ε

Read it left to right: among all new parameter vectors θ′ close to the old θ, pick the one that most increases the (linearized) objective. The fence is a ball of radius ε in parameter space. Solving this gives back ordinary gradient ascent — ε just plays the role of α.

The flaw the lecture flags. Why measure the fence with the 2-norm of the parameter change? Some directions in θ barely nudge the policy; others, the same numerical distance, swing the action distribution wildly. A fixed ball in parameter space is a lopsided, distorted ball in the space we actually care about — policy space. The 2-norm is measuring the wrong thing.

The fix is to measure distance where it matters: directly between the two policies. Constrain a divergence between the old and new action distributions, not between their parameter vectors:

max_θ′ (θ′ − θ)^⊤ ∇_θ J(θ) s.t. D(π_θ, π_θ′) ≤ ε

Natural Policy Gradient (NPG) makes this concrete. For small steps, the KL divergence between the policies is well-approximated by a quadratic form in the parameter change, with the Fisher information matrix F as its metric:

D(π_θ, π_θ′) ≈ (θ′ − θ)^⊤ F (θ′ − θ)

F is the local "ruler" of policy space: it stretches the directions that change the policy a lot and shrinks the ones that barely matter. Solving the constrained problem with this metric gives the natural gradient update:

θ′ = θ + α F⁻¹ ∇_θ J(θ)

That F⁻¹ "pre-bends" the raw gradient so a unit step moves the same policy distance in every direction. The same numerical ε now means the same amount of behavioral change — which is exactly the trust region we wanted.

Parameter-space ball vs. policy-space ball

The teal dashed circle is a fixed ball in parameter space. The warm ellipse is what that same ball looks like measured in policy space, warped by the Fisher metric. Drag the anisotropy slider: when one direction changes the policy much more than the other, the naive ball lets you step too far along the sensitive axis. The natural gradient re-shapes the step so it's a true circle in policy space.

Anisotropy 3.0

A round ball in θ is a stretched ellipse in policy space.

Take-away. Step size should be measured in units of behavioral change, not raw parameter distance. NPG installs that idea via the Fisher metric. TRPO (next) turns the approximate KL constraint into an exact, enforced one; PPO turns it into something you can run without ever forming or inverting F.

Why does NPG constrain a divergence between policies (via the Fisher matrix) instead of the plain 2-norm of the parameter change?

Because the 2-norm is slower to compute. Because parameters must always be unit vectors. Because equal parameter distances can cause very unequal changes in the action distribution; the Fisher metric measures the step in units of actual policy change. Because the Fisher matrix removes the need for a reward signal.

Chapter 3: TRPO — A Hard KL Trust Region

TRPO starts from a beautiful identity. The difference in return between a new policy and the old one can be written exactly as an expectation of the old policy's advantage A, taken over states the new policy visits:

J(θ′) − J(θ) = E_{τ ~ π_θ′} [ ∑_t γ^t A^π_θ(s_t, a_t) ]

If we could maximize the right-hand side directly, we'd be doing exact policy improvement. There's a chicken-and-egg problem, though: the expectation is over trajectories from the new policy π_θ′ — the very thing we are still solving for. We can't sample from a policy we don't have yet.

The key move. If the new policy is close enough to the old one, the states it visits are nearly the states the old one visited. So we cheat: sample states from the old policy (which we have, and already rolled out), and correct the action mismatch with an importance-sampling ratio — the ratio of new-to-old action probability. The cheat is only valid if the policies stay close. That "stay close" is the trust region, and it is what makes the whole approximation honest.

Concretely, we maximize a surrogate objective — the old-policy-sampled advantage, reweighted by the probability ratio — subject to a hard cap on the KL divergence between old and new:

max_θ′ E_{s,a ~ π_θ} [ ^{π_θ′(a|s)}⁄_{π_θ(a|s)} · A^π_θ(s, a) ]

s.t. E_{s ~ π_θ} [ D_KL( π_θ(·|s) ‖ π_θ′(·|s) ) ] ≤ δ

Read the objective: if an action had positive advantage, increase its probability ratio (make it more likely under the new policy); if negative, decrease it. The KL cap δ is the leash that keeps the importance-sampling approximation valid — the same trust region NPG hinted at, now explicitly enforced rather than approximated by a single F⁻¹ step.

Why importance sampling needs the leash — a number

The importance-sampling trick is "evaluate the new policy using the old policy's data, reweighted by the ratio." It is exact in expectation, but its variance explodes when the policies drift apart. Feel it concretely. Suppose under the old policy an action had probability 0.5, and the new policy makes it probability 0.9. The reweighting ratio is 0.9 / 0.5 = 1.8 — mild. Now suppose the new policy makes it 0.99 while the old policy had it at 0.05. The ratio is 0.99 / 0.05 = 19.8 — a single sample now carries twenty times its honest weight, and a handful of such samples swamp the whole estimate.

ratio = ^π_new(a|s)⁄_{π_old(a|s)} : ^0.9⁄_0.5 = 1.8 (safe) vs ^0.99⁄_0.05 = 19.8 (blows up)

This is the deep reason the trust region is not optional bookkeeping — it is what keeps these ratios near 1 so the estimate stays sane. The KL cap on the policy distributions directly bounds how large any reweighting ratio can get. Keep this picture: the leash isn't to be cautious for its own sake; it's to keep the math we used to justify the step from falling apart. PPO's clip, next chapter, attacks the exact same ratios from the other end — it caps them sample by sample instead of capping the distribution-level KL.

Step inside the KL ball on a reward surface

Background shading is true return (brighter = better). The teal circle is the trust region — the set of new policies within KL ≤ δ of where you stand. TRPO moves to the best point inside that circle, not the global optimum. Slide δ to grow or shrink the leash, then press Step to take several constrained updates and watch the path climb safely.

Trust radius δ 0.14

Ready — standing at the start.

Bigger δ lets each step jump further and climb faster — until it's large enough that the linear/importance-sampling approximation breaks and a step overshoots into worse territory (the cliff from Chapter 0). Smaller δ is safe but slow. TRPO's guarantee is that, within the leash, every step is a genuine improvement.

Why TRPO works but hurts. The KL constraint gives a real monotonic-improvement guarantee — theoretically gorgeous. But enforcing it means solving a constrained, non-convex optimization every update: estimate the Fisher matrix, do a conjugate-gradient solve, then a line search to satisfy the KL cap. It is complicated, slow, and incompatible with tricks like dropout or sharing layers between the policy and value networks. This pain is precisely what PPO was invented to remove.

In TRPO's surrogate objective we sample states from the OLD policy but want the return of the NEW policy. What single condition makes that substitution valid?

The reward must be deterministic. The advantage must always be positive. The new policy must stay close to the old one (small KL), so old and new visit nearly the same states; the importance-sampling correction then holds. The action space must be discrete.

Chapter 4: PPO — The Clipped Surrogate, Step by Step

PPO keeps TRPO's idea — reweight the advantage by the probability ratio, stay near the old policy — but throws away the expensive constrained solve. Instead of a hard KL leash, it bakes the "stay close" into the objective itself with a clip, so you can optimize it with ordinary minibatch SGD for several epochs.

Define the probability ratio — new action probability over old, at the actions you actually took:

r_t(θ′) = ^{π_θ′(a_t | s_t)}⁄_{π_θ(a_t | s_t)}

When the new policy hasn't moved, r = 1. If the new policy makes an action more likely, r > 1; less likely, r < 1. The plain (TRPO-style) surrogate term is just r times the advantage: r_t · A_t. Maximizing it with no leash is exactly the cliff — nothing stops r from running off to 5 or 0.01.

The clipped objective, built one piece at a time

PPO compares two quantities and takes the minimum:

L^CLIP(θ′) = E_t [ min( r_t A_t, clip(r_t, 1−ε, 1+ε) · A_t ) ]

Two new pieces. clip(r, 1−ε, 1+ε) simply pins the ratio inside the band [1−ε, 1+ε] — with the standard ε = 0.2, that's [0.8, 1.2]. Anything below 0.8 becomes 0.8; anything above 1.2 becomes 1.2. The outer min takes the smaller of the unclipped term and the clipped term. Let us see exactly when each one bites, splitting on the sign of the advantage.

Case 1 — the action was good (A > 0). We want to push its probability up, so we'd like r to grow past 1. The clipped term caps the reward for that at r = 1+ε = 1.2. Once r exceeds 1.2, the clipped term (1.2·A) is smaller than the unclipped term (r·A), so the min picks the clipped one — and it's flat. Gradient zero. Pushing this good action even more likely earns nothing. The update stops voluntarily before it walks off the cliff.

Case 2 — the action was bad (A < 0). We want to push its probability down, so r should shrink below 1. Here the clip floor at 1−ε = 0.8 matters. Once r drops below 0.8, the clipped term is 0.8·A; because A is negative, the unclipped r·A (with small r) is the larger (less negative) number, so the min picks the unclipped one — which keeps a gradient. A very bad action keeps getting pushed down; it is not let off the hook.

That asymmetry is the entire genius. The clip removes the incentive to overshoot in the helpful direction, but it never stops you from fixing a genuinely bad action. The lecture states it as a warning about the alternative: if you used only the clipped term, a "very bad" ratio (r ≪ 1 when A > 0) would not be punished — the min is what restores that punishment.

The min makes it a pessimistic lower bound. Because we always take the smaller of the two terms, L^CLIP ≤ the unclipped surrogate everywhere. PPO optimizes a conservative under-estimate of improvement — it never lets the objective reward a step more than the honest, un-clipped quantity would. That pessimism is what keeps it inside an implicit trust region without any KL solve.

The clipped objective vs. the unclipped one

The dashed gray line is the unclipped surrogate r·A. The warm curve is L^CLIP — what PPO actually optimizes. Toggle the sign of the advantage and watch which side goes flat (zero gradient) and which keeps a slope. The flat region is where one update can no longer move the policy too far.

Clip ε 0.20

Advantage A > 0 (good action) — uncheck for A < 0

Good action: flat once the ratio passes 1+ε.

Because the clip enforces the trust region per sample inside a plain loss, PPO can do what TRPO cannot: multiple epochs of minibatch SGD on the same batch of data. That reuse is why PPO is dramatically simpler to implement, plays nicely with dropout and shared networks, and — empirically — matches or beats TRPO's sample efficiency. It is the workhorse behind robot locomotion policies and the RLHF step that aligns large language models.

For a GOOD action (A > 0), the new policy already makes it far more likely so the ratio r = 1.8, well above 1+ε = 1.2. What does L^CLIP do?

Keeps rewarding the increase, so r is pushed even higher. The min selects the clipped term (1.2·A), which is flat — zero gradient, so the update stops pushing this action further. Pushes the action's probability to zero. Throws an error because r > 1.

Chapter 5: Compute the Clip by Hand

Formulas are slippery until you run a number through them. Let us evaluate L^CLIP for several probability-ratio values, with ε = 0.2 so the band is [0.8, 1.2], for both signs of the advantage. The per-sample objective is

L = min( r · A, clip(r, 0.8, 1.2) · A )

Good action: A = +2

We want r above 1. Watch where the clip starts to bite.

ratio r	r·A (unclipped)	clip(r)	clip(r)·A	min → L	gradient?
0.90	1.80	0.90	1.80	1.80	yes
1.00	2.00	1.00	2.00	2.00	yes
1.10	2.20	1.10	2.20	2.20	yes
1.20	2.40	1.20	2.40	2.40	edge
1.50	3.00	1.20	2.40	2.40 (clipped)	no — flat
2.00	4.00	1.20	2.40	2.40 (clipped)	no — flat

At r = 1.5: the unclipped term is 1.5 × 2 = 3.0; the clipped term is clip(1.5)=1.2, so 1.2 × 2 = 2.4. The min is 2.4. Push r from 1.5 to 2.0 and L stays glued at 2.4 — the clip bit at r = 1.2 and froze the reward. No further gradient, so SGD stops enlarging this already-favored action. That is the safe-step mechanism, in arithmetic.

Bad action: A = −2

We want r below 1 (make a bad action less likely). Now the asymmetry shows.

ratio r	r·A (unclipped)	clip(r)	clip(r)·A	min → L	gradient?
1.00	−2.00	1.00	−2.00	−2.00	yes
0.80	−1.60	0.80	−1.60	−1.60	edge
0.50	−1.00	0.80	−1.60	−1.60 (clipped)	no — flat
1.50	−3.00	1.20	−2.40	−3.00 (unclipped)	yes — punished!

Two things to feel here. First row r = 0.5: we have already pushed the bad action well below its old probability (good!), and the clip stops us from being rewarded for pushing it to zero — the min picks the clipped −1.60, which is flat. Done; move on.

The last row is the punch line. r = 1.5 with A < 0 means the new policy made a bad action more likely — a real mistake. The unclipped term −3.00 is smaller (more negative) than the clipped −2.40, so the min keeps −3.00, which still has gradient. The optimizer is pulled hard to fix this. This is the lecture's warning made arithmetic: if PPO used only the clipped term, this "very bad" ratio would cap at −2.40 and be under-punished; the min restores the full penalty.

Now make the machine do it — and verify your hand answers against it.

With ε = 0.2 and a GOOD action (A = +2), you compute L at r = 1.5. What value, and is there still a gradient pushing r higher?

L = 3.0, gradient still present. L = 1.5, gradient still present. L = 2.4 (= 1.2·A), and it is flat — no gradient, so the step stops growing this action. L = 4.0, gradient still present.

Chapter 6: DDPG & TD3 — Deterministic Off-Policy Control

PPO is on-policy and stochastic: it throws data away after each update. For continuous control on a real robot, that waste hurts. The other branch — DDPG (Deep Deterministic Policy Gradient) — reuses every transition from a replay buffer and learns a precise, deterministic action map.

DDPG has two networks. A critic Q_φ(s, a) estimates the value of taking action a in state s, trained by the same Bellman/replay machinery as DQN. And an actor — a deterministic policy μ_θ(s) that outputs one action, no sampling.

How the actor learns: gradient flows through the critic

Here is the elegant part. The actor wants to output the action that the critic rates highest. So we maximize the critic's value at the actor's own action, and the chain rule routes the gradient through the critic into the actor:

∇_θ J ≈ E_{s ~ D} [ ∇_a Q_φ(s, a) |_{a=μ_θ(s)} · ∇_θ μ_θ(s) ]

Read the data flow. The critic tells you, at the current action, which direction in action space increases value — that's ∇_a Q. The actor's Jacobian ∇_θ μ then says how to nudge the weights to move the output action in that good direction. The actor is dragged toward whatever the critic currently values. Because the critic learns from a replay buffer of any past data, DDPG is fully off-policy — that's its sample-efficiency win.

Make the shapes concrete, because the chain rule here is easy to wave at and hard to picture. Say the state is a 17-dimensional robot observation and the action is a 6-dimensional torque vector. The actor maps 17 numbers in to 6 numbers out: μ_θ: ℝ¹⁷ → ℝ⁶. The critic maps the 17 + 6 = 23 numbers (state concatenated with action) to a single scalar value. To improve the actor we ask the critic for ∇_a Q — a 6-vector, one number per torque, saying "raise this torque, lower that one." We feed that 6-vector backward through the actor network and it becomes a gradient on the actor's weights. No reward, no probability ratio, no log-prob appears anywhere — the learning signal is purely "the critic says move the action this way." That is what makes it the deterministic policy gradient.

Off-policy reuse needs care. Reusing old data is free learning — until it isn't. The critic is trained on its own predictions (bootstrapping), and a deterministic actor with no exploration noise will keep proposing the actions the critic happens to over-estimate. The error feeds back on itself, the critic's Q-values explode, and the policy chases a mirage. This overestimation bias is the failure mode TD3 was built to fix.

TD3: three patches that stabilize DDPG

TD3 (Twin Delayed DDPG) keeps DDPG's skeleton and adds three fixes, each targeting a specific instability:

Twin critics. Train two Q-networks and use the smaller of the two as the learning target. Taking the min systematically counteracts overestimation — if one critic is too optimistic, the other reins it in.
Target policy smoothing. Add a little clipped noise to the action when computing the target, so the critic can't exploit a sharp, spurious peak in its own value surface. It forces Q to be smooth in a neighborhood.
Delayed actor updates. Update the actor (and target networks) less often than the critic, so the actor chases a critic that has had time to settle rather than a noisy moving target.

The actor must still explore, and a deterministic policy explores nothing on its own — so DDPG/TD3 bolt exploration noise onto the action during data collection. That bolt-on is exactly what SAC, next, replaces with something principled.

Let's make the critic-gradient mechanism concrete. The lab below recovers the action-gradient direction — the very signal that drives the DDPG actor — numerically.

In DDPG, how does the deterministic actor μ_θ(s) learn which action to output?

By sampling many actions and keeping the luckiest one. By copying a human expert's demonstrations. By following the critic's action-gradient ∇_aQ through the chain rule, nudging its output toward actions the critic values higher. By maximizing the KL divergence to the old policy.

Chapter 7: SAC — Maximum-Entropy RL

SAC (Soft Actor-Critic) keeps DDPG's off-policy, replay-buffer efficiency but makes two changes that together make it the most robust continuous-control algorithm on the list: the policy is stochastic again, and the objective rewards not just return but entropy.

The entropy of a policy is a measure of how spread-out (unpredictable) its action distribution is. A policy that always picks the same action has zero entropy; one that spreads probability evenly across actions has maximum entropy. SAC adds an entropy bonus to every step's reward, scaled by a temperature α:

J(π) = E [ ∑_t r(s_t, a_t) + α · H( π(· | s_t) ) ]

In words: succeed at the task and act as randomly as possible while doing so. The temperature α is the trade dial — large α values exploration heavily (the policy stays spread out); small α collapses toward pure reward-maximization (and toward DDPG-like determinism).

Why an entropy bonus helps — two concrete payoffs

Exploration that doesn't die. A pure-reward agent that stumbles onto one decent strategy will lock onto it and stop trying alternatives — premature convergence to a local optimum. The entropy bonus pays the agent to keep its options open, so it keeps probing for something better. Exploration is built into the objective, not bolted on as external noise like in DDPG.
Robustness. A high-entropy policy doesn't commit to one razor-thin solution; it keeps mass on several near-optimal actions. If the world shifts a little — friction changes, a payload is added — it still has a backup action in its distribution rather than a single brittle plan. The SAC paper reports unusually consistent performance across random seeds for exactly this reason.

The entropy term is a trust region in disguise. By paying for spread, SAC resists collapsing the policy onto a sharp peak — the same disease (a brittle, over-committed update) that TRPO/PPO fight with KL caps and clips. PPO keeps the new policy near the old one; SAC keeps the policy near uniform. Different fence, same instinct: don't over-commit.

The entropy bonus as a number

For a discrete policy with action probabilities p, the (Shannon) entropy is H = − ∑_i p_i log p_i. Take a 3-action policy that is nearly deterministic, p = (0.90, 0.05, 0.05):

H = −[ 0.90 ln 0.90 + 0.05 ln 0.05 + 0.05 ln 0.05 ] = −[ −0.0948 − 0.1498 − 0.1498 ] ≈ 0.394

Now a spread-out policy p = (0.34, 0.33, 0.33):

H = −[ 0.34 ln 0.34 + 0.33 ln 0.33 + 0.33 ln 0.33 ] ≈ 1.098

The uniform-ish policy has nearly three times the entropy (1.098 vs 0.394). With temperature α = 0.2, the entropy bonus alone is 0.22 vs 0.08 per step — a standing reward the spread-out policy collects just for staying exploratory. Maximizing return plus αH tilts SAC toward keeping that breadth until the task reward is strong enough to justify narrowing.

Temperature dial — explore or collapse

Three actions; one has slightly higher task value. Slide the temperature α: at high α the SAC policy stays spread (high entropy, exploring); at low α it collapses onto the single best action (low entropy, exploiting). The bar heights are the policy's action probabilities; the readout shows the entropy and the entropy bonus αH.

Temperature α 0.50

Moderate temperature: balanced exploration.

Let's compute and verify the entropy itself, and confirm the headline fact: a hotter (more uniform) softmax policy has higher entropy.

SAC maximizes reward PLUS an entropy bonus αH. What is the main benefit of the entropy term?

It makes the critic unnecessary. It pays the policy to stay spread out, so it keeps exploring and stays robust instead of collapsing onto one brittle action. It forces the policy to be deterministic. It removes the need for a replay buffer.

Chapter 8: The 2×2 Map — Click to Place Every Algorithm

Chapter 1 promised that two binary questions sort every algorithm. Here is the grid, and it is the single most useful picture to carry out of this lecture. The two axes:

On-policy ↔ off-policy — throw data away after each update, or reuse a replay buffer.
Stochastic ↔ deterministic policy — output a distribution to sample from, or one exact action.

The algorithm map — click any cell

Click a quadrant (or an algorithm chip) to see which methods live there and why. The horizontal axis is on-policy → off-policy; the vertical axis is stochastic → deterministic.

Click an algorithm to place it.

	On-policy (data thrown away)	Off-policy (replay buffer reused)
Stochastic policy	REINFORCE, A2C, PPO, TRPO	SAC
Deterministic policy	—	DDPG, TD3

Read the columns. The whole on-policy / stochastic cell is the policy-gradient family — REINFORCE up through PPO — and it is exactly where the step-size problem lives, because each update discards its data. NPG, TRPO and PPO are the progressively cheaper trust regions that tame it.

The off-policy column is the actor-critic family that buys sample efficiency through replay. DDPG and TD3 sit in the deterministic row; SAC sits in the stochastic row because its entropy objective wants a distribution to spread. Notice the empty cell: on-policy and deterministic is essentially unused — a deterministic on-policy method would throw away data and have no built-in exploration, the worst of both.

The one decision rule. Cheap, fast simulator with parallel rollouts → lean on-policy / PPO: simple, robust, parallelizes beautifully. Expensive real-robot samples → lean off-policy / SAC: squeezes every transition, explores via entropy, and is famously stable across seeds. Need raw deterministic precision and you can manage its instability → TD3. That is the practitioner's flowchart in one sentence.

In the 2×2 map, which cell is essentially empty, and why?

Off-policy / stochastic — because replay buffers can't hold distributions. On-policy / stochastic — because PPO made it obsolete. On-policy / deterministic — it discards data AND has no built-in exploration, the worst of both. Off-policy / deterministic — because DDPG and TD3 don't exist.

Chapter 9: Connections & Cheat Sheet

You now own the advanced model-free toolbox — the algorithms people actually reach for in robot learning. Every one of them answered the Chapter 0 question (how big a step is safe?) with a different fence.

Vanilla PG

Step uphill with hand-tuned α. One bad step collapses the policy.

↓ measure distance in policy space, not parameters

NPG

Pre-bend the gradient by the Fisher metric F⁻¹. Equal ε = equal behavioral change.

↓ enforce the policy-distance cap exactly

TRPO

Maximize the importance-sampled advantage s.t. KL ≤ δ. Monotonic but slow.

↓ approximate the cap with a cheap clip

PPO

min(r·A, clip(r)·A). Plain SGD, multi-epoch, the on-policy workhorse.

↻ or skip on-policy entirely: reuse a replay buffer

DDPG → TD3 → SAC

Off-policy actor-critic. Deterministic + twin critics (TD3) or stochastic + entropy bonus (SAC).

Cheat sheet — the vocabulary you now own.

Step-size problem = one too-large on-policy step lands off the local peak and the policy collapses (it generates its own data).
Trust region = a fence keeping the new policy close enough to the old that the gradient/importance-sampling approximation stays valid.
NPG = natural gradient F⁻¹∇J; measures the step in policy space via the Fisher metric.
TRPO = max importance-sampled advantage s.t. KL(π_old‖π_new) ≤ δ. Hard constraint, monotonic improvement, expensive.
PPO = L^CLIP = min(r·A, clip(r,1−ε,1+ε)·A). The clip flattens over-eager good actions; the min still punishes bad ones. A pessimistic lower bound, not a hard constraint.
Probability ratio r = π_new(a|s) / π_old(a|s); = 1 when unchanged.
DDPG = deterministic actor + critic; actor follows ∇_aQ through the chain rule; off-policy via replay.
TD3 = DDPG + twin critics (min target) + target smoothing + delayed actor, all to kill overestimation.
SAC = off-policy, stochastic, maximizes reward + αH(π); entropy bonus = built-in exploration & robustness.

The 2×2 cheat-sheet table

	On-policy	Off-policy
Stochastic	REINFORCE, A2C, PPO	SAC
Deterministic	—	DDPG, TD3

Two misconceptions to leave behind. (1) The PPO clip is not a hard constraint. It does not forbid a large ratio — it just stops rewarding one, by zeroing the gradient. A single big-magnitude advantage can still drag r outside the band; that is why real PPO also does early-stopping on KL. (2) Off-policy reuse is not free. Replaying old data buys sample efficiency but invites overestimation bias and divergence — the entire TD3 patch-list and SAC's twin critics exist to manage exactly that.

Where to go next on this site

This lesson is Lecture 11 of the 16-831 series. The natural neighbors:

Lecture 10 — Actor-Critic — the advantage, baselines, and the actor-critic skeleton these methods build on.
Lecture 12 — Model-Based Control Basics — the next branch: learn a world model and plan, for sample efficiency model-free RL can't match.
RL Algorithms — the model-free landscape, value vs policy vs actor-critic, from absolute zero.
Off-Policy Actor-Critic — replay buffers, target networks, and the deterministic/soft variants in depth.
Policy Gradients & Actor-Critic — the foundations of the TRPO/PPO branch.

The test of this lesson. Open the original Lecture 11 slides. You should be able to explain, from memory: why a constrained 2-norm step is the wrong fence, what NPG fixes with the Fisher matrix, why TRPO's KL constraint is correct but painful, how the PPO clip-and-min flattens over-eager good actions yet still punishes bad ones, how the DDPG actor learns through the critic's gradient, what the three TD3 patches fix, and why SAC's entropy bonus buys exploration and robustness. If you can place all six on the 2×2 map, you're ready for Lecture 12.

"Take the biggest step you safely can — then take another."