Model-Based Control Basics

Chapter 1: The State-Space Model — How the World Evolves

Before we can control a system we need a precise way to say how it moves. Control theory writes every system in one of two near-identical forms, and the whole lecture lives in them.

The state x is a vector that captures everything you need to know about the system right now — for a cart, its position and velocity; for a drone, position, velocity, angle, and angular rate. The control input u is what you get to choose — a thrust, a torque, a wheel speed. The dynamics f says how the state changes given the current state and your input.

The dot in ẋ means rate of change — how fast the state is moving this instant. The continuous form says "the velocity of the state is some function of state and input." The discrete form says "next step's state is some function of this step's state and input." They describe the same idea; the discrete one is what a computer actually iterates, and the continuous one (the instructor's choice for today) makes stability easiest to define because the physical world is continuous.

The simplest possible dynamics — the integrators

Two toy systems carry most of the intuition, so meet them now.

The single integrator is ẋ = u: whatever you command is the velocity of the state. Push right, it slides right; release, it stops instantly. Think of a frictionless puck where your input is its speed.

The double integrator is ẍ = u (two dots = acceleration): your input is the acceleration, not the velocity. This is Newton's law for a unit mass — force equals acceleration. It is the honest model of a cart, a drone axis, a robot joint. The subtlety, which Chapter 4 will make vivid, is that commanding acceleration is much harder to stabilize than commanding velocity: you can't stop on a dime, because momentum carries you past the target.

Linear systems — the form we can actually solve

When f is linear, the model collapses to two matrices:

Here A is the n×n matrix describing how the state evolves on its own (the open-loop, uncontrolled physics), and B is the n×m matrix describing how your input pushes the state. The double integrator written this way uses x = [position; velocity]: position's rate is velocity, velocity's rate is the input. So A moves position from velocity, and B injects u into velocity. Almost no real robot is exactly linear — but Chapter 9 shows you can linearize a nonlinear system around an operating point and get an A and B that are accurate nearby. Linear systems are where the clean theory lives, so we earn the right to use it.

Chapter 2: Feedback — Closing the Loop

We have the dynamics. Now the goal. The instructor lists what a feedback policy might be asked to do: stabilize the system to a reference point, track a moving trajectory, minimize a cost (this is where RL was born), be adaptive to uncertainty, and be robust to disturbance. Today's spine is the first one — hold a target — because everything else is built on it.

A feedback policy is a rule u = π(x): you look at the current state and decide the input from it. Plug it into the dynamics and you get the closed-loop system, the system with the controller wired in:

The notation f_cl emphasizes the point: once the policy is fixed, there is no input left to choose — the state evolves all on its own according to a new, controller-shaped law. Designing a controller is choosing π so that this closed-loop law behaves the way you want.

The proportional controller — push harder the farther you are

The simplest useful policy is the proportional (P) controller: the input is proportional to the error, pushing back toward the reference. For the single integrator ẋ = u, with target 0, take u = −kx:

Read this. If x is positive (we're to the right of the target), the input is negative (push left); the bigger the error, the harder the push. The constant k > 0 is the gain — a single dial for how aggressively the controller reacts. The solution of this equation is a decaying exponential, x(t) = e^−kt x(0): the error melts away to zero, faster for larger k. We just turned a system that did nothing (u = 0 leaves it frozen wherever it started) into one that homes in on the target.

Open-loop trajectory optimization vs. closed-loop policy

The instructor draws this distinction explicitly. Open-loop trajectory optimization (planning) solves for a whole sequence of states and inputs ahead of time — great for "what path should I take," useless on its own against surprises. Closed-loop policy optimization finds the rule u = π(x) that reacts. Real systems use both: plan a trajectory, then wrap a feedback controller around it to hug that trajectory despite disturbances. We'll see exactly that in Chapter 6.

Chapter 3: PID — Proportional, Derivative, Integral

The P controller is enough for the single integrator, but it falls apart on the double integrator — the honest model of a real cart or drone axis. Here is why, and here is the fix, term by term. By the end you'll understand the most widely deployed controller on Earth: PID.

P alone isn't enough — the double integrator

Take ẍ = u, target 0, and apply pure proportional, u = −x. The closed-loop law is ẍ = −x, whose solution is x(t) = x(0) cos(t) + ẋ(0) sin(t) — a sine wave that never decays. The controller keeps pushing back toward the target, the mass keeps overshooting through it (momentum!), and you get an undying oscillation. Stable, technically, but useless: it never settles.

Add D — the derivative term (damping)

The derivative controller adds a term proportional to how fast the error is changing. For the double integrator: u = −x − 2ẋ. The closed-loop law becomes ẍ = −x − 2ẋ, which now decays — the solution is x(t) = (x(0) + (x(0)+ẋ(0))t)e^−t, sliding smoothly to zero. The −2ẋ term is a brake: when the cart is racing toward the target, D pushes against the motion, dissipating the energy that would have caused overshoot. P + D together is the PD controller, the workhorse of robot joint control.

Read the two gains as physical dials. K_p is stiffness — how hard the controller pulls toward the target (a stronger spring). K_d is damping — how hard it resists motion (a stronger shock absorber). Too little K_d and you overshoot and ring; too much and you crawl home sluggishly. There is a sweet spot, and you'll find it in the widget below.

Add I — the integral term (kill the steady offset)

One problem remains. Suppose a constant disturbance d shoves the system — a steady wind, a payload, gravity on a tilted arm. A P or PD controller leaves a steady-state offset: it settles, but slightly off-target, because at the offset the spring force exactly balances the disturbance and the error stops shrinking. The fix is the integral controller: accumulate the error over time and feed it back.

The integral term has a beautiful logic: as long as any error persists, the integral keeps growing, so the input keeps ramping — until the error is driven to exactly zero. The instructor shows the disturbed single integrator with u = −2x − ∫x: differentiate both sides and you get ẍ = −2ẋ − x, which is exponentially stable and drives the error to zero despite the constant disturbance. Integral control, he notes, is the simplest form of adaptive control — it silently learns the constant push and cancels it.

A PID step by hand

Let's compute one control step with real numbers so the formula stops being abstract. Discrete double integrator, target 0, gains K_p = 1.0, K_d = 2.0, K_i = 0.1, time step Δt = 0.1. Suppose right now the position is x = 0.5, the velocity is ẋ = −0.3 (already moving back toward target), and the accumulated error integral so far is I = 0.8.

The control output is the weighted sum:

Notice what each term said. P wanted to push left (−0.5) because we're to the right of target. D pushed right (+0.6) because we're already moving left fast — D is saying "ease off the brake, you'll overshoot." I added a small left nudge (−0.08) from the leftover accumulated error. The three negotiate, and the net command is a gentle +0.02. Then the integral updates: I ← I + x·Δt = 0.8 + 0.5·0.1 = 0.85, ready for the next step. That single line, run 50 times a second, is a PID controller.

Three experiments worth running: (1) set K_d = 0 and watch it oscillate — no damping. (2) Raise K_d and the ringing dies. (3) With K_i = 0 and wind on, notice it settles below the target — the steady offset. Now raise K_i and watch the trace creep back up to exactly the target. You just saw all three terms earn their letters.

Let's build it in code — the full PID loop — and prove that adding the integral term is what finally erases the offset.

Chapter 4: Stability — Does It Actually Come Home?

We've been tuning controllers by eye. But "looks like it settles" is not engineering. Control theory's job, the instructor says, is to rigorously define and analyze what we mean by a good controller. The most basic property of all is stability: if you perturb the system, does it come back?

There are three flavors, and they form a ladder of increasing strength.

The formal definitions are about ε and δ: a solution is stable if for every error tolerance ε you pick, there's a starting-distance δ small enough that staying within δ at the start keeps you within ε forever. Asymptotically stable adds that the gap eventually goes to zero. Exponentially stable adds that it goes to zero at least as fast as a decaying exponential. The instructor notes a clean fact: for linear systems, asymptotic and exponential stability are the same thing. So for the linear systems we care about, "converges" already means "converges fast."

Lyapunov's idea — stability without solving the ODE

For a high-dimensional system you can't just solve the differential equation and stare at it. Lyapunov gave a way to prove stability without solving anything. The trick: find an energy-like function V(x) that is positive everywhere except zero at the equilibrium, and show that it always decreases along the system's motion. If the energy can only go down and is bounded below by zero, the system must drain to the bottom — the equilibrium.

The instructor's example makes it concrete. Take the single integrator ẋ = −Kx with the energy V(x) = ½ x^Tx (squared distance from the target). Its rate of change along the motion is V̇ = x^Tẋ = −x^TKx. If K is positive definite, this is strictly negative whenever x ≠ 0 — in fact V̇ ≤ −2λ_min(K)·V, which is the exponential-stability condition. So the system is exponentially stable whenever the gain K is positive definite, proven without ever solving the ODE. That is the power of the energy argument: it scales to any dimension.

Chapter 5: Eigenvalues — Reading Stability Off the Matrix

For linear systems, stability is not something you have to simulate or guess. It is written, exactly, in the eigenvalues of the closed-loop matrix. This is the single most useful test in linear control, and it's a one-line check.

Wire a linear feedback law u = −Kx into the linear system ẋ = Ax + Bu. The closed loop becomes:

where K is the gain matrix (an m×n matrix) and A_cl = A − BK is the closed-loop matrix. The solution of ẋ = A_clx is governed by the eigenvalues of A_cl — each mode of the system decays or grows like e^λt where λ is an eigenvalue. So:

Why the real part? Write an eigenvalue as λ = σ + iω. Then e^λt = e^σt(cosωt + i sinωt). The real part σ sets the envelope: σ < 0 means a shrinking exponential (decay), σ > 0 means a growing one (blow-up), σ = 0 means constant amplitude (eternal oscillation). The imaginary part ω just sets how fast it oscillates inside that envelope. So the sign of the real part is stability. The more negative, the faster it decays — that's the exponential rate λ from Chapter 4.

A stability check by hand

Take the double integrator with a PD controller and verify it's stable through eigenvalues alone. State x = [position p; velocity v]. The open-loop matrices are A = [[0, 1], [0, 0]] (position's rate is velocity; velocity's rate is zero on its own) and B = [[0], [1]] (input is acceleration). The PD law u = −K_pp − K_dv means K = [K_p, K_d]. The closed-loop matrix:

Its eigenvalues solve the characteristic equation det(A_cl − λI) = 0, which works out to λ² + K_dλ + K_p = 0. Plug in K_p = 1, K_d = 2 (the instructor's PD example): λ² + 2λ + 1 = 0, so (λ + 1)² = 0, giving a repeated eigenvalue at λ = −1. Both eigenvalues have real part −1 < 0 — Hurwitz — so the closed loop is exponentially stable, decaying like e^−t. That matches exactly the ODE solution from Chapter 3. Now set K_d = 0: λ² + 1 = 0 gives λ = ±i, real part 0 — the eternal oscillation. The eigenvalues told us everything without solving a single trajectory.

Note the discrete-time twist, since real controllers run in discrete time: for x_t+1 = A_clx_t, the stable region is not the left half-plane but the unit circle — every eigenvalue must have magnitude (spectral radius) strictly less than 1, so each mode λ^t shrinks. Same idea, different boundary. We'll use exactly this test in the eigenvalue code lab.

Let's verify the closed-loop stability test directly from the matrices.

Chapter 6: Disturbance, Robustness, and Tracking

Real systems don't match your model exactly. There's a gust, a payload, an unmodeled friction. The instructor frames the uncertainty hierarchy crisply: robust control handles a disturbance that is "wrong and not learnable" (you just need to survive it), and adaptive control handles one that is "wrong but learnable" (you can estimate and cancel it). Let's see why a good controller is naturally robust, and how to track a moving target.

Why exponential stability buys you robustness

Add a disturbance d to the model: ẋ = f(x, u) + d. The goal of robust control is input-to-state stability (ISS) — not necessarily converging to exactly zero, but staying within an error ball whose size is proportional to the disturbance. The instructor's example nails it: the disturbed single integrator ẋ = −kx + d settles to a steady error of d/k. Bigger gain k → smaller residual error. The system doesn't blow up; it just sits a little off, by an amount you can shrink.

And to eliminate a constant disturbance entirely, you already know the trick from Chapter 3: integral control. The instructor calls it the simplest adaptive controller — it silently estimates the constant push and cancels it. More sophisticated adaptive control (MRAC, L1 adaptive, the Neural-Fly work) estimates richer, time-varying disturbances, but the integral term is the seed of the whole idea.

Tracking a trajectory — add a feedforward term

So far the target was a fixed point. Often we want to track a moving reference x_d(t) — follow a path, not just hold still. The instructor's tracking controller for the double integrator ẍ = u is:

The first two terms are PD on the tracking error e = x − x_d. The last term, ẍ_d, is the feedforward: the acceleration the desired trajectory itself demands. Why include it? Define the error e = x − x_d and substitute — the feedforward exactly cancels the ẍ_d that appears, leaving clean error dynamics:

This is just the stable PD equation from Chapter 5, now governing the error rather than the position. The feedforward does the "predictable" work of following the trajectory; the feedback only has to mop up the deviations. Without it, the feedback would always lag, constantly chasing a target it can't anticipate. For a fixed setpoint, ẋ_d and ẍ_d are zero and it collapses back to plain PD. Feedforward + feedback is the standard pattern: plan the path (open-loop), wrap feedback around it (closed-loop), exactly the split from Chapter 2.

Chapter 7: LQR — The Optimal Linear Controller

PID gains we tuned by hand and by eye. But "which gains are best?" is itself a question with a clean answer. The instructor's bullet from Lecture 1 returns: minimize a cost — this is where RL started. Set up the right cost over a linear system and the optimal controller drops out in closed form. This is the Linear-Quadratic Regulator (LQR), and it is the centerpiece of the whole lecture.

The cost — state error versus control effort

LQR asks: drive the state to zero while not spending too much control. It writes that trade-off as a quadratic cost integrated over time:

Two terms, two penalties. x^TQ x penalizes the state for being away from zero — the bigger the error, the bigger the cost; Q (positive semidefinite) weights which states you care about. u^TR u penalizes the control effort — how much "gas" you burn; R (positive definite) weights how expensive your actuators are. The entire art of LQR is the ratio Q/R.

The answer is a single gain matrix

Here is the surprise that the whole lecture has been building toward. For the linear system ẋ = Ax + Bu with this quadratic cost, the optimal policy is linear state feedback — u = −Kx — the very form we've been using all along. The best possible controller, out of all possible controllers, is just a matrix times the state:

where P is a positive-definite matrix that solves the Algebraic Riccati Equation (ARE):

You do not solve this by hand — you call an LQR routine in Python or MATLAB, the instructor stresses. But understand what it gives you. The matrix P is, in a precise sense, the optimal cost-to-go: x^TPx is the minimum remaining cost from state x — the value function from your RL lectures, now for a continuous linear system. The gain K* reads that value off and turns it into the cheapest action. And here is the kicker:

Where does this closed form come from? The instructor names two classical routes: dynamic programming (the Hamilton-Jacobi-Bellman equation — the continuous-time cousin of the value iteration you saw in Lecture 7) and calculus of variations (Pontryagin's maximum principle). Both land on the same Riccati equation. Next lecture derives the discrete-time version step by step; today we just need to see the structure and feel the trade-off. The lab below makes that trade-off something you can play with.

Chapter 8: Discrete LQR by Hand — The Riccati Recursion

The continuous Riccati equation is solved by a library call. But the finite-horizon discrete version — the one the next lecture derives and the one you'd actually code for model-based RL — is solved by a beautiful backward recursion, and it is small enough to run by hand. Doing so reveals exactly where the gain comes from: it's dynamic programming, run backward in time, exactly like value iteration.

The setup: discrete linear system x_t+1 = A x_t + B u_t, and a cost summed over a finite horizon of N steps, with a terminal penalty:

The cost-to-go from any state turns out to be quadratic, V_t(x) = x^TP_tx, and the matrices P_t are computed backward from the end. Start at the terminal cost, P_N = Q_f, then sweep backward. At each step the optimal gain and the next P are:

This is the backward Riccati recursion. Read it as one step of dynamic programming: given the cost-to-go from the next state (P_t+1), find the action that minimizes "cost now + cost-to-go from where that action lands you." The minimization is over a quadratic, so it has a closed form — that's where the matrix inverse comes from. The result is both the optimal gain K_t for this step and the updated cost-to-go P_t. Sweep all the way back to t = 0 and you have a gain for every step.

One Riccati step with real numbers

Let's do the cleanest possible scalar example so every operation is a number, not a matrix. Take a scalar system A = 1, B = 1 (so x_t+1 = x_t + u_t), with cost weights Q = 1, R = 1, and a terminal penalty Q_f = 1. Start the backward sweep at P_N = Q_f = 1.

One step back, compute K_N−1 first:

So at this step the optimal action is u = −0.5·x — move halfway toward the target. Why not all the way (u = −x, landing exactly on 0)? Because control is costly (R = 1): the cost of slamming there outweighs the benefit. Halfway is the optimal compromise. Now the cost-to-go update:

So P grew from 1 to 1.5: states farther in the past cost more, because more steps of error and effort lie ahead. Take another step back with P = 1.5: K = 1.5/(1+1.5) = 0.6, and P = 1 + 1.5 − 1.5·0.6 = 1.6. The gains and P-values march toward a steady value as the horizon grows — and that fixed point is exactly the solution of the algebraic Riccati equation from Chapter 7. The infinite-horizon LQR gain is the limit of this backward recursion. Beautiful: finite-horizon DP, run long enough, is the steady-state optimal controller.

Now build the full backward recursion and prove the gain it produces actually stabilizes the system.

Chapter 9: Connections — Linearization, and Where This Goes

You now have the control-theory foundation that all of model-based RL stands on. Before the cheat sheet, one loose end: real robots are nonlinear. How do we use this beautiful linear theory on a nonlinear drone?

Linearization — the bridge from nonlinear to linear

The instructor's answer is linearization. Around an equilibrium point (x_e, u_e) where the system is at rest, approximate the nonlinear f by its first derivatives — the Jacobians:

The 2-D drone makes it vivid: at hover (level, motionless, thrust balancing gravity), the nasty trig term with sinθ and cosθ flattens to a simple linear matrix — the dynamics become "horizontal acceleration is proportional to tilt angle, vertical acceleration is proportional to extra thrust." Near hover, the drone is a linear system, so you design an LQR controller for the linearized A and B and it works beautifully — as long as you don't tilt too far. There's also feedback linearization, which algebraically cancels the nonlinearity (the instructor mentions it but skips the theory). And when linearization isn't enough — aggressive maneuvers, big tilts — you re-linearize at every step along a trajectory, which is exactly the iLQR/DDP and SQP machinery of the next lectures.

The honest caveats

• We assumed the model is known. The whole point of model-based RL is to learn f from data when it isn't — system identification, model learning, or online adaptation — then run exactly today's machinery on the learned model.
• Linear-quadratic is a narrow corner. LQR's closed form needs linear dynamics and quadratic cost. Real problems have constraints (don't hit the wall, don't exceed motor limits) and nonlinearity — which is why MPC, SQP, iLQR, and sampling-based methods exist (next lectures).
• Feedback needs the state. u = −Kx assumes you can measure x. When you can't, you need a state estimator — a Kalman filter — and the combination (estimator + LQR) is the classic LQG controller.

Where to go next on this site

• Lecture 11 — Advanced RL (PPO, SAC) — the previous lecture; the model-free world this one departs from. Useful contrast: trial-and-error policy search vs. designed feedback.
• Lecture 13 — Optimal Control & Planning I — the next lecture; MPC, discrete LQR derivation, SQP/iLQR — how to make decisions over a horizon when the system is nonlinear and constrained.
• RL Algorithms — the model-free toolbox; see how the cost-minimization view of LQR connects to the reward-maximization view of RL (cost = −reward).
• Lecture 7 — Value & Policy Iteration — the dynamic-programming spine; the Riccati recursion is its continuous-state, quadratic-cost cousin.
• Kalman Filter — the state estimator you pair with LQR when you can't measure the full state directly.

"The optimal controller for a linear system with a quadratic cost is a single matrix on the state. Everything harder — constraints, nonlinearity, unknown dynamics — is the rest of the course earning its keep."