In simulation a crash is free; on a real robot it breaks the leg. So how do you let a policy explore and learn while guaranteeing it never enters a state it must not? The answer is a thin layer that lets the policy propose, then projects every unsafe action onto the nearest safe one.
A reinforcement-learning agent playing Atari dies ten thousand times before it gets good. Each death costs nothing — the emulator resets, the score zeroes, and it tries again. That is the whole reason game RL works: it can afford to fail at industrial scale.
Now bolt that same trial-and-error learner onto a quadruped walking along the edge of a table. The first time it stumbles off the edge, you do not get a reset. You get a shattered actuator, a thousand-dollar repair, and a robot that cannot collect any more data because it is in pieces. Put a person in the room instead of a table edge and the stakes stop being about money.
This is the line Lecture 1 drew when it said physics doesn't forgive. In games, failures are free. In the physical world, some states are catastrophic — once you enter them, the episode is not just lost, it is irreversible. A safe-RL system must learn a good policy while never visiting those states even once, not even during exploration.
The lecture's instructor frames this with a deceptively simple picture: a robot near a wall. The robot's learned policy proposes a velocity that would drive it straight into the wall. The interesting question is not "how do we train a policy that never proposes a bad action?" — that is hopeless during early learning. The question is: can we wrap the policy in a layer that catches every bad proposal and nudges it to the closest safe alternative, before the motor ever moves?
A robot (warm dot) is learning to move right toward a goal, but its early policy is noisy and sometimes jerks toward the cliff edge (red zone). Toggle the safety filter and press Run. With the filter off it eventually falls; with it on, every action that would cross the edge is clipped, so it explores and reaches the goal without ever leaving the table.
With the filter off, crank the noise and watch the dot eventually tip over the edge — game over, no recovery. With the filter on, the same noisy policy never falls, because the layer between policy and motor refuses to execute any action that would carry the robot past the boundary. The learning still happens; the catastrophe does not.
"Safety" sounds obvious until you try to write it down. The lecture's first move is to make it precise, and the precise version is a sentence worth memorizing: safety is a constraint in an MDP. The robot still wants to maximize reward (or minimize cost) over time — that part is unchanged. What safety adds is a forbidden region the trajectory may not enter and a set of allowed actions it may not leave.
Read it in plain words. Here xt is the state, ut is the control (the action), ct is the cost we pay at each step, and ft is the dynamics — how the world moves you to the next state. The first two lines are just an ordinary control problem. The last line is safety: the state must stay inside the safe set Xt (no collisions, no falling off the table) and the action must stay inside the admissible set Ut (no torque beyond what the motor can survive). Everything in this lecture is a different way of enforcing that last line.
The instructor splits safety into two types, and the split organizes the entire field:
| Type | What it constrains | Example |
|---|---|---|
| Policy-wise | The closed-loop behavior — e.g. the policy must keep the system stable (a small push doesn't make it diverge). | If a gust nudges a drone, it returns to its path rather than spiraling away. |
| State-wise | The states the trajectory is allowed to occupy — a forbidden region in state space. | Collision avoidance; staying a meter from a human; not leaving the table. |
Policy-wise safety asks: if we perturb the robot a little, will it fall or recover? That is the language of stability — the nominal trajectory and any slightly-perturbed trajectory should stay close (we make this rigorous in Chapter 8). State-wise safety asks: does the robot ever physically enter a bad state? That is collision avoidance, and it exists in almost every real robotic system. Most of this lesson lives in the state-wise world, because that is where "never hit the person" lives.
Not all guarantees are equal. The lecture grades them on a ladder, and knowing which rung a method sits on tells you exactly how much to trust it:
The most common first instinct, and the most dangerous, is this: "I'll just give the robot a big negative reward whenever it crashes. Then RL will learn to avoid crashing." It feels airtight. It is not. Understanding exactly why is the conceptual hinge of the whole lecture.
An RL agent maximizes expected return — the average, discounted total reward. A crash penalty is just one term inside that average. The optimizer is therefore free to trade it off: if crashing pays −100 but happens only rarely, while a slightly-too-aggressive style earns +5 on every other run, the math may decide that occasionally crashing is worth it. The agent is not malfunctioning; it is doing exactly what you asked — maximizing the average — and the average does not care about the worst case.
Below, an agent chooses how aggressively to drive toward a goal that sits just past a cliff. A bigger penalty shifts its preferred aggression down — but as long as the reward for speed is positive, there is always a penalty-large-enough question. And notice: even a penalty that makes crashing "rare" still means a nonzero crash rate, which in robotics means eventually, a broken robot.
An agent picks its aggression to maximize expected return: faster driving earns more reward but raises the chance of going over the cliff (which pays the penalty). Slide the crash penalty up. The optimizer backs off — but the crash probability only shrinks, it never reaches zero. A hard constraint would pin it to exactly zero.
There is a second, subtler problem the lecture stresses repeatedly: the penalty teaches nothing until the robot has actually crashed. RL learns the value of a state by visiting it. To learn "this state is worth −100", the agent must enter that state — which, for a catastrophic state, is precisely what we forbade. Reward shaping asks the robot to learn the danger by experiencing it, but the whole point of safety is that it must not experience it. The villain of Lecture 1, compounding error, returns here as: you cannot Monte-Carlo your way to "never".
Chapter 2 showed that a hand-picked penalty is arbitrary: too small and it gets ignored, too big and it makes the robot a useless coward. The constrained MDP (CMDP) replaces that guesswork with a clean specification. Instead of folding safety into the reward, keep them separate: maximize reward as usual, subject to a budget on a second quantity — the expected cost of unsafe behavior.
Here rt is the ordinary task reward, ct is a separate cost signal that fires when the robot does something unsafe (gets within a meter of a human, draws too much current), and d is the cost budget you are willing to tolerate. This is far better than a penalty because d is interpretable — "no more than 25 near-misses per episode, in expectation" — rather than a magic penalty weight you tune by trial and error.
The CMDP is, as the lecture says bluntly, highly non-convex and very hard to solve directly. The standard trick is the Lagrangian: fold the constraint back into the objective, but with a learned multiplier λ instead of a guessed one.
Read λ as a thermostat. We optimize the policy to maximize L (it wants reward) and adjust λ to enforce the budget. If the policy is spending more cost than the budget d allows, λ climbs — making unsafe behavior more expensive — until the policy backs off. If the policy is comfortably under budget, λ drifts down toward zero, freeing the policy to chase reward. The multiplier finds its own level, automatically, instead of you guessing the penalty weight. This is exactly what penalty-tuning wished it was.
That last line is the whole update: nudge λ up when cost exceeds the budget, down when it is under, never letting it go negative. Let's implement that thermostat and watch it pull a budget-busting policy back under the line.
Every RL method has a beating heart called exploration: to find a better policy, the agent must occasionally try actions it is unsure about. In a game, an unsure action that turns out badly costs a life and teaches a lesson. On a robot, an unsure action that turns out badly might be the action that ends the robot. This is the deepest tension in the whole field, and it has a name: safe exploration.
Spell out the chicken-and-egg. To know whether an action is safe, you generally have to try things and observe consequences. But the consequences of unsafe actions are exactly the catastrophes you are forbidden from experiencing. So you must explore enough to learn a good policy, while never stepping outside the region you already know is safe.
This buys safety at a price, and the price is the central tradeoff of the entire lecture: conservatism versus performance. A very cautious agent stays far from every boundary — it will never crash, but it also never discovers the high-reward behaviors that live near the edge (the fastest racing line hugs the wall). A bold agent explores aggressively and may find the optimum — or may fall off the cliff first. Safe robot learning is the art of tuning that dial.
A robot explores a corridor with a high-reward zone right next to the forbidden wall (red). Slide the safety margin: a large margin keeps the robot far from the wall (zero crashes, but it never reaches the reward); a tiny margin lets it grab the reward but risks crossing the wall. Watch reward and crash-risk trade off as you slide.
Notice the shape: as you reduce the margin, expected reward climbs (you reach more of the corridor) but crash risk climbs too, and at some point it spikes. The sweet spot is a margin small enough to capture most of the reward yet large enough to keep risk at zero. The methods in the rest of this lecture are all attempts to find — and then certify — that sweet spot, so the robot can sit as close to the boundary as is provably safe and not one inch closer.
Here is the mathematical object that turns "please stay safe" into "you cannot leave the safe set." Define a single scalar function of the state, h(x), the safety function, with one rule: h(x) ≥ 0 means safe, h(x) < 0 means unsafe. The boundary — the wall — is exactly where h(x) = 0. Think of h as "signed distance to danger": large and positive deep in safe territory, shrinking to zero at the edge, negative past it.
Defining "safe" as a region is easy. The hard part is staying in it over time, because the robot keeps moving. A naive rule like "only act when h(x) ≥ 0" fails: you might be safe right now but barreling toward the wall so fast that no action next step can save you. By the time h hits zero it is too late to stop.
Read the dot on top as "rate of change of h". The right-hand side, −γ·h(x), is the speed limit on how fast you may approach danger. Walk through what it enforces, because this single line is the heart of the lecture:
It is an automatic, state-dependent speed limit toward the cliff: fast when far, slow when near, frozen at the edge. The constant γ > 0 is the aggressiveness knob — large γ lets you ride right up to the wall before braking (high performance, small margin); small γ brakes early (conservative, big margin). That is precisely the conservatism dial from Chapter 4, now made into a tunable number with a guarantee attached.
And here is the payoff, the property the whole construction buys you: forward invariance. If you start inside the safe set (h ≥ 0) and you obey the CBF condition at every instant, you provably stay inside the safe set forever. The safe set becomes a region the state mathematically cannot leave — a trap door that only opens inward. We will see that invariance in action in Chapter 7.
The robot sits at distance h from the wall (h = 0). The teal curve is the CBF speed limit: the most negative rate of change of h that is still allowed at each distance. Far from the wall, big approach speeds are fine; near it, the allowed approach speed shrinks to zero. Slide γ to see the aggressiveness knob reshape the limit.
Now we cash in the barrier function. The architecture the lecture champions is beautiful in its modularity: let the learned policy do whatever it wants, then pass its action through a filter that catches anything unsafe and minimally edits it. The RL policy and the safety guarantee are completely decoupled — you can swap in any policy, however reckless, and the filter still protects the robot.
Here is the data flow, the Concept-plus-Realization core of this whole lesson:
Step 3 is the famous CBF-QP (control-barrier-function quadratic program), from Ames et al. 2017. "QP" because the optimization is a quadratic program: minimize the squared distance from the nominal action (a quadratic objective) subject to the barrier inequality (a linear constraint). It is tiny — often a handful of variables — and solves in microseconds, fast enough to run at the robot's control rate. In one sentence: the safety filter projects an unsafe action onto the closest safe one.
The word minimally-invasive is doing real work here. The filter does not impose its own agenda; it changes the policy's action by the smallest possible amount needed to restore safety, and not a bit more. When the policy is already safe, the filter is invisible — u* = unom. It only wakes up at the boundary. So you get the full performance of the learned policy everywhere it's safe, and a hard guarantee exactly where you need one.
Make it concrete with the simplest possible case. A robot is at position p = 2.0 and there is a wall at p = 0; the safe set is p ≥ 0, so a natural safety function is just h(p) = p (distance to the wall). The action is a velocity v, and the dynamics are ṗ = v, so ḣ = v. With γ = 1, the barrier condition ḣ ≥ −γh becomes:
So at p = 2.0, any velocity down to −2.0 is safe — you may move toward the wall at up to 2 units/sec. Now suppose the policy proposes vnom = −3.5 (charging at the wall too fast). It violates the limit. The filter finds the nearest velocity that satisfies it: the boundary value v* = −2.0. It clips the proposed −3.5 to −2.0 — the closest safe value — and that is what the motor executes. As the robot approaches the wall, h shrinks, the allowed velocity −γh rises toward zero, and the robot is forced to slow to a stop exactly at the boundary. Let's build that projection.
Time to see every idea move at once. A robot lives in a 2D arena with a circular keep-out zone (a person, an obstacle) in the middle. Its learned policy is greedy: it always heads straight for whatever goal you place, even if that line plows through the obstacle. The CBF safety filter sits between the policy and the wheels, projecting each greedy velocity onto the nearest safe one.
The safety function here is h(x) = (distance from obstacle center) − (radius): positive outside the keep-out circle, zero on its edge, negative inside. The filter enforces ḣ ≥ −γh. Watch what emerges: the robot drives straight when the obstacle isn't in the way, then smoothly arcs around the keep-out zone the moment its greedy heading would violate the barrier — not because the policy got smart, but because the filter bent each action by the minimum needed. The robot grazes the boundary and rides it, never entering. That is forward invariance you can watch.
Click anywhere to set a goal (teal). The robot (warm) always wants to go straight there (faint red arrow = nominal action), but the safety filter projects that onto the nearest safe velocity (green arrow = executed action) so the trajectory can never enter the red keep-out circle. Toggle the filter off to watch the greedy policy drive straight through. Slide γ to change how tightly it hugs the obstacle.
The instructive experiment: set a goal on the far side of the obstacle so the straight line goes through it, then toggle the filter. Off, the warm dot drives a beeline into the red zone — collision. On, the green executed-arrow peels away from the red nominal-arrow exactly at the boundary, carving an arc around the circle, then snaps back to straight-line once the obstacle is behind it. The policy never changed; only the filter intervened, and only where it had to.
Now let's certify that invariance in code — not by watching, but by proving the minimum value of h over a full rollout never goes negative under the filter, while it does without it.
Chapters 5–7 handled state-wise safety — staying out of forbidden regions. The lecture's other half is policy-wise safety: stability. The question is the one from Chapter 1: if we perturb the robot a little, does it recover, or does the error grow? A stable system, nudged off its nominal trajectory, stays in a tube around it; an unstable one diverges.
The lecture grades stability in three strengths. Stable: perturbations stay bounded (the gap between the nominal and perturbed trajectories never blows up). Asymptotically stable: the gap not only stays bounded but shrinks to zero — the robot returns to its path. Exponentially stable: it returns fast, the gap decaying like a falling exponential. For a linear closed-loop system, there is a crisp test: stability is governed by the spectral radius — the largest absolute eigenvalue of the closed-loop matrix. If every eigenvalue sits inside the unit circle, errors shrink each step and the system is (exponentially) stable; outside, they grow.
The deep parallel: a Lyapunov function is to stability what a barrier function is to state safety. Both are scalar certificates of a state; both turn a hard global property into a simple local condition on a derivative. Barrier: "h can't decrease past the wall." Lyapunov: "V can only decrease toward the goal." Modern Lyapunov learning simply learns such a certificate (and a policy that maximizes the region it covers) with a neural network, instead of hand-deriving it.
The flagship case study ties everything together. A drone landing near the ground feels messy aerodynamic ground effect that is brutally hard to model analytically. Neural-Lander (Shi et al., ICRA 2019) approximates that unknown force with a deep network and feeds it into the controller. But there is a catch with teeth: a controller built on a learned dynamics term is only provably stable if that network is well-behaved — specifically, if its Lipschitz constant (how fast its output can change as the input changes) is bounded below 1.
The lecture rounds out the toolbox with two more families worth a name. Hamilton-Jacobi reachability computes, exactly, the set of states from which danger is unavoidable no matter what you do — and keeps you out of it; the Agile But Safe legged-locomotion work uses a reachability-based filter to run a quadruped at 3 m/s with collision-avoidance guarantees. And reach-avoid value networks learn a value function whose sign tells you whether a state can both reach the goal and stay safe — a model-free, scalable cousin of the barrier idea.
You now own the whole landscape. Safety is a constraint in an MDP; you enforce it at one of three strengths, on one of two kinds of constraint, with a small family of tools. Here is how every method in the lecture hangs on that spine.
This lesson is Lecture 22 of the 16-831 series. Continue with Lecture 23 — Multi-Task, Adaptive & Transfer Learning, and revisit where the data came from in Lecture 21 — Sim2Real (safety filters are how a sim-trained policy survives first contact with reality). Companion Gleams that expand individual threads:
"Let the policy be bold. Make the filter the part that never gets it wrong."