A noisy sensor reports only where a cart is. A Kalman filter still recovers its hidden speed, and keeps that speed error small while raw sensor differences go wild.
Learn how a world model keeps a hidden state, how it predicts the next state, how one new observation corrects that guess, and how the model learns its own numbers.
Drag the sensor noise up, then switch between the three ways to estimate. Watch the speed panel at the bottom. Then we build, piece by piece, the model, the Bayes loop, the Kalman filter, and the learning that sits underneath.
You need a rough idea of a probability (a number from 0 to 1 for how likely something is) and what a matrix is (a grid of numbers that turns one list of numbers into another). We teach the rest from zero.
Based on Lecture 4 of CIS 6280: World Models (University of Pennsylvania, Fall 2026), taught by Jiatao Gu, with the course’s lecture slides, its handwritten LGSSM notes and the dynamax Kalman filter tracking notebook. Every idea is re-taught here in our own words and our own devices.
Illustrative: a 1D cart simulated here with the lecture’s model (position p, speed v, a known push a, one noisy position sensor, step rule p+v+½a and v+a) and process noise of standard deviation 0.1. The Kalman filter is the lecture’s predict then correct. Errors are root-mean-square over steps 5 to 59.
Chapter 0
See why one sensor reading cannot tell two different futures apart, and why a model must carry more than it can see
A delivery cart rolls down a long corridor. At the far end, a sensor reports where the cart is once every second. The reading wobbles a little each time, because no sensor is perfect.
You need to say where the cart will be in four seconds. You have the latest reading: 3 metres from the wall. Is that enough?
It is not. A cart that creeps at 1 metre per second and a cart that races at 2 metres per second can both be at 3 metres right now. The reading looks the same. The futures are very different. What is missing is the speed, and the sensor never reports it.
This lecture is about that gap. It gives a name to the missing piece, a model for how the missing piece changes, and an exact method to guess it from the readings. The method is the Kalman filter, from 1960, and it is still the backbone of many tracking and robot systems.
Lecture 4 of CIS 6280 opens with three course updates. An Ed Discussion board is open for your questions. The final exam falls in the last week of teaching. The release of Assignment 1 moves to Thursday, September 10. That is all the logistics there are, so we move on to the idea.
Lecture 3 built simulators: programs that hold the full truth about a toy world and step it forward. The slide shows two T-shaped blocks pushed around in a simulator. The camera picture shows the outcome: where the blocks sit and touch. The simulator knows more: speeds and physical numbers too.
Four words from that lecture matter now. Learn them well, because every later chapter uses them.
In Lecture 3 you could read st straight out of the simulator. A real learner cannot. It may receive only ot. When the state is there but you cannot see it, we call it a hidden state. It is like the engine of a car: you hear it and feel it, but you never look at it directly.
A world model is a model that predicts what happens next in the world. If the world has a hidden state, the model must keep a guess of that state, because the next observation depends on it. Everything in this lecture serves that one need.
Both carts sit at 3 metres, so the sensor frame is identical. Set each cart’s hidden speed with the sliders and watch where each reading will be over the next four steps.
Illustrative: no push is applied, so each cart moves p + v every step. The speeds are yours to set; the lecture’s own 1D model appears in Chapter 3.
Set both speeds the same and the two futures merge. Set them apart and the readings drift away from each other, although the first frame never changed. The frame shows the position. The future needs the speed too.
So the plan for this lesson has three parts. First, a model of how a hidden state moves and how it shows itself through the sensor. Second, a way to keep a running guess of the state while readings arrive. Third, a way to learn the model’s numbers from recorded data. The hero at the top of the page already runs the second part: it guesses both position and speed from one noisy position sensor.
Chapter 1
Separate three ways to remember the past, and learn what makes a state the right thing to remember
You watch a video of a ball. You pause on one frame. Where will the ball be in a moment? You cannot say, because a still frame has no motion in it. Now you step back one frame. The ball moved one square to the right. Suddenly you can guess: one more square, then another.
You did not learn a new fact about the world. You used the past to recover something the present frame hid. That small act is the heart of this chapter. An agent has three different ways to carry the past with it, and the lecture names all three.
Lecture 2 set the goal of a world model. Given what the agent knows so far and a plan of actions, predict what comes next. The lecture writes it as pθ(Yfuture | ht, afuture). Read it aloud: the probability of the future outcomes Y, given the history and the actions we choose. The symbol θ stands for the model’s own numbers, which Chapter 2 names.
The history, written ht, is everything the agent has seen and done up to now: ht = (o0:t, a0:t−1). The small notation o0:t means the list o0, o1, up to ot. Notice the actions stop one step earlier, because the action at has not been taken yet.
Keeping the whole history is safe, since nothing is thrown away. It is also a problem. The history grows by one more observation and action every step. After an hour of video it is enormous, and the model must reread all of it to make each prediction.
A representation is a short code that stands in for something larger. The lecture’s version is zt = Eφ(ot; o<t, a<t). An encoder E, with its own learned numbers φ, reads the current observation and a view of the past, and returns one code zt. The lecture calls zt a contextualized representation: it describes the current observation using past context.
There is a catch, and the lecture states it plainly. A representation makes no promise to be a full summary. Prediction may still need the whole sequence of codes, z≤t. The code helps, but it does not close the question. Lecture 5 returns to how such codes are learned.
The third way is the neat one. Suppose you could name a quantity st that holds everything the future needs. Then the past would add nothing once you knew st. That is the Markov property, and a quantity with it is a Markov state.
Think of a save file in a video game. The file holds the level, your health and your items. You can load it tomorrow and play on, and nothing about how you got there matters. A replay of every button press holds the same information, but you would not want to carry it around. The save file is a Markov state. The replay is a history.
Now the key move, and it matters for the whole lecture. The Markov property talks about prediction when the state is known. In a real system you often do not know st. You hold only codes z≤t and past actions. To find the state you must do state inference: compute p(st | z≤t, a<t), a distribution over states that says how sure you are. Inferring an unknown state can still need the history. The result describes your uncertainty about st, which is different from st itself.
So the three ways line up like this. The history is complete but too big. A representation is small but may not be complete. A Markov state is small and complete, but you usually have to infer it. The rest of the lecture builds the machine for that inference.
Take the cart from Chapter 0. A frame shows position pt = 5.0. The cart’s speed is 2.0, so the next position is about 7.0. Three guesses are possible.
Guess one uses position only: “the cart stays at 5.0.” It misses by 2.0. Guess two uses the history: the last two positions were 3.0 and 5.0, so the cart moved 2.0 and should move 2.0 more. That gives 7.0, an almost perfect guess, from history alone. Guess three uses the state (position 5.0, speed 2.0) and adds them: 7.0. The state and the two-frame history give the same answer, because the state is a compact record of the history that matters.
Each dot is one simulated cart. Across is the position you predict. Up is where the cart really went. Dots on the diagonal are perfect. Choose what the predictor may look at.
Illustrative: 90 simulated carts with positions from 0 to 10, speeds from −2 to 2, and process noise of standard deviation 0.1 at each step. “Misses by” is the root-mean-square gap between the guess and the true next position.
The picture shows why we want a state. With position only, the dots form a wide cloud, because the missing speed spreads the outcomes. With two positions or with the full state, the dots hug the diagonal. History and state carry the same information. The state is simply the smaller, tidier package.
Chapter 2
Write a world with a hidden state as three small probability rules, then run them to generate a whole run of the world
Imagine you write the rules of a tiny world on three index cards. Card one says how the world starts. Card two says how it moves when you push it. Card three says what a sensor would show you at any moment. Shuffle nothing, add nothing else. With those three cards you can play the world forward as often as you like, and every run will be a little different.
That is a state-space model (SSM). The lecture calls it a world model with a Markov state: a generative process over trajectories. A trajectory is one full run: the states, the observations and the actions in order.
The lecture draws the process as four moves. Each move uses one probability law, and each law has a plain meaning.
The symbol θ stands for all the numbers that define the model. Estimating θ from recorded trajectories is the job of Chapter 9. Notice what the readings depend on: only the current state, never the older readings. Notice what the next state depends on: only the current state and action. That is the Markov property from Chapter 1, now built into the model.
Because each step uses just one law, the chance of a whole trajectory is a product of small pieces. This is the same trick you saw for sentences in a language model: multiply one chance per step.
A model needs real formulas for its three laws. The lecture picks the oldest and cleanest choice, the linear-Gaussian state-space model (LGSSM), from R. E. Kalman’s paper of 1960. It makes three promises.
First, the state is a list of real numbers, a vector st in Rn. For our cart n = 2: position and speed. Second, the maps are linear: the next state is a fixed matrix times the current state, plus a fixed matrix times the action, and the reading is a fixed matrix times the state. Third, all randomness is Gaussian: the familiar bell curve, fully described by its centre (the mean) and its spread (the covariance, a grid of numbers that records how wide the spread is in each direction and how directions move together).
The lecture writes the three laws like this.
s0 ~ N(μ0, P0) st+1 = A st + B at + εst, εst ~ N(0, Q) ot = C st + εot, εot ~ N(0, R)
Take the pieces one at a time. The process noise εs is a small random kick that the world adds to the state at each step: a bump in the corridor, a gust of wind. Its size is set by Q. The observation noise εo is the sensor’s wobble, set by R. The matrix A says how the state moves by itself, B says how the action pushes it, and C says which part of the state the sensor can read.
The data flow has fixed shapes. For the cart, s is a 2-vector, a is one number, o is one number, A is 2 by 2, B is 2 by 1, C is 1 by 2, Q is 2 by 2 and R is a single number. Chapter 3 fills them in.
Why choose such strict rules? Because two things stay true, and they make everything else easy. Add a linear map to a Gaussian and you still have a Gaussian. Observe a Gaussian through a linear sensor and the updated belief is still Gaussian. So the whole uncertainty about the state fits in two objects, a mean and a covariance, forever. The lecture says it this way: under these assumptions, prediction and correction have closed-form Gaussian updates.
The lecture adds one more assumption: independence. The start state and all noise terms are independent of each other, and the noise at one time is independent of the noise at every other time. Each kick and each wobble is a fresh draw. Run the model below and see this for yourself.
Pick an action, then press Step. Blue dashed lines are the hidden state. Yellow dots are the sensor readings. Change a noise slider to start a fresh run.
Illustrative: the cart model of Chapter 3 with start law N(μ0 = [0, 0], P0 = diag(1, 0.25)) and the noise levels shown, as standard deviations. The numbers in the read-out are this run’s own draws.
Run it with both noises at zero. The trajectory is a clean curve that the three laws fully decide. Add process noise and the true path wanders. Add sensor noise and the yellow dots scatter around the path. The two noises do different jobs, and a filter must tell them apart: process noise changes the truth, sensor noise only changes what you see of the truth.
A failure worth knowing: the noise terms are assumed independent. Suppose a real sensor drifts, so each reading is wrong in the same direction as the last. The symptom is a filter that is too sure of itself. You measure it by checking whether the residuals, the gaps between predicted and actual readings, correlate from one step to the next. The fix is to put the drift into the state, so the model can estimate it too.
Chapter 3
Turn a story of motion and sensing into the matrices A, B and C, and see what each one does to a state
Back to the corridor. The cart has a position p and a speed v. You can push it, and the push changes its speed. One sensor reads the position, with a wobble. That is the whole world. The lecture uses it to make the abstract model concrete, and so do we.
The state is two numbers stacked in a vector: st = [pt, vt]. The observation ot is one number. The action at is also one number, and the lecture is careful to say what it is: an acceleration-like push applied over one model step. It is not the relative target command from Lecture 3.
Let one model step last one unit of time. In that time the cart coasts at speed v, so its position grows by v. If you also push with acceleration a, the speed grows by a. A push that builds up smoothly over the step moves the cart an extra half of a, because the cart only has the full extra speed at the end. This is the rule from school physics, distance = v t + ½ a t2 with t = 1.
pt+1 = pt + vt + ½ at + εpt vt+1 = vt + at + εvt ot = pt + εot
The first two rules say how the state evolves. The last says what the sensor measures: one noisy number, the position. The lecture adds a remark that carries the whole chapter. Speed shapes future positions, but the sensor does not read it directly. Repeated position readings can still reveal the speed, because the speed leaves a trace in how the position changes. Chapter 4 proves exactly when that works.
A matrix is a grid of numbers that turns one list of numbers into another. Multiply a matrix by a vector and each output number becomes a weighted sum of the input numbers. The three rules above are three such sums, so we can pack them into three matrices.
Now the three rules are the lecture’s two lines: st+1 = A st + B at + εst and ot = C st + εot. The shapes match what Chapter 2 promised: A is 2 by 2, B is 2 by 1, C is 1 by 2.
Start from s = [2, 1]: the cart is at 2 and moves at 1. Push with a = 0.5, and ignore noise for now.
First, A s = [1·2 + 1·1, 0·2 + 1·1] = [3, 1]. The cart coasted from 2 to 3 and kept its speed. Second, B a = [½·0.5, 1·0.5] = [0.25, 0.5]. The push adds 0.25 to the position and 0.5 to the speed. Add the two: st+1 = [3.25, 1.5]. Finally the sensor computes C st+1 = 3.25. It reports a position and says nothing about the speed of 1.5. Try other numbers below.
Set the cart’s state and the push. The purple ghost is where A alone takes the cart. The blue cart is the next state after the push. The yellow tick is what the sensor would read.
Noise-free on purpose, so each term of the arithmetic shows. The matrices are the lecture’s: A = [[1, 1], [0, 1]], B = [½, 1]T, C = [1, 0], one model step is one unit of time.
Slide the speed to zero and A does nothing: the cart stays where it is, and only the push moves it. Slide the push to zero and only A acts. Look at the second number in the pbox: it never reaches the sensor line.
A failure to watch for. Suppose you write B wrongly, say B = [0, 1], and forget the ½ a term. The symptom: after every push, your estimate of the position lags a little behind the truth. The measurement: the residuals, the gaps between the predicted and the real readings, spike exactly when the action is non-zero. The fix: correct B, or if the physics is truly uncertain, give the model a larger Q so it does not trust its own rule.
The lecture closes this block with a promise. Once the three laws are fixed, the same model supports three different operations. They are the plan for the rest of the lesson.
Chapter 4
Test, before you build any filter, whether two different states could ever produce the same readings
A detective has a list of clues and two suspects. If the clues fit both suspects equally well, no amount of clever reasoning will pick one. The detective needs a new clue, not a better method. A state estimator is in the same position. Before you build one, ask a blunt question: could the clues ever tell the possible states apart?
The lecture calls this observability. It asks: from the history ht = (o0:t, a0:t−1), can the agent work out the state st that the model uses to predict? The answer can fail in two quite different ways, and it helps to keep them apart.
The second problem is the one a Kalman filter handles. The first is a flaw in the setup, and a filter cannot repair it. So the lecture isolates the first one with a clean test.
Throw the noise away: set both noise terms to zero. Keep what you know for sure: the matrices A, B and C, and the action sequence. Now ask the structural question. Given a sequence of readings, can you uniquely determine the state at the start of the window?
Try it on the cart. One ideal reading says ot = pt. That fixes the position exactly. The speed could be anything: 0, 1, 5. A single frame is not enough to recover a two-number state, which is just what Chapter 0 showed.
Add one more reading. The next position obeys pt+1 = pt + vt + ½ at, and the sensor reads ot+1 = pt+1. Rearrange, with the push at known: ot+1 − ½ at = pt + vt. Since pt = ot, the speed is vt = ot+1 − ot − ½ at. Take readings of 2.0 and 3.25 with a push of 0.5. Then v = 3.25 − 2.0 − 0.25 = 1.0. The second reading exposed the speed, although the sensor never reads speed.
The same two equations in matrix form make the test mechanical.
[ot, ot+1 − ½ at]T = [[1, 0], [1, 1]] [pt, vt]T, det = 1, rank = 2
The determinant (det) of a 2 by 2 matrix [[a, b], [c, d]] is ad − bc. A nonzero determinant means the matrix can be undone: from the two readings you can solve back for the two unknowns. The rank counts how many rows carry independent information. Rank 2 for 2 unknowns is exactly what we want.
Now generalize. Collect H readings, ot to ot+H−1. Each reading is the state pushed through A some number of times, then read by C: ot = C st, ot+1 = C A st + C B at, and so on. Subtract the known action terms, because you know them, and stack what remains.
Here is the test, in the lecture’s own words. If the rank of OH is below n, then some nonzero change δ has OH δ = 0. Then OH(s + δ) = OH s: the state s and the state s + δ produce identical readings. That is permanent ambiguity. If OH has full column rank, no such δ exists, and the state is uniquely determined.
Note the lecture’s careful caveat. One reading may not fix the state uniquely. With a prior and an observation model you can still say which states are more likely. Ambiguity limits what the data alone can settle, not what a Bayesian estimate can say.
The dot is the true state. Everything in the green or red mark gives exactly the same noise-free readings as the dot. Pick a sensor, then add readings with the slider.
Noise-free, with the lecture’s A = [[1, 1], [0, 1]]. The true state is [3, 1]. A state s′ is drawn when OH(s′ − s) = 0, found from the null direction of the first nonzero row of OH.
Start with the position sensor and one reading. The red line is vertical: every speed at that position fits. Drag the slider to two readings and the line collapses to a point. Now switch to the speed sensor. No number of readings helps: the rows of OH are all [0, 1], so the position can slide along the red line forever. The symptom of such a design is a position estimate whose uncertainty never shrinks. The measurement is rank(OH) = 1 at every H. The fix is a new sensor that reads position, not more speed readings.
Chapter 5
Keep a belief about the hidden state, widen it when the world moves, and sharpen it when a reading arrives
You are walking through a dark house and trying to keep track of where you are. You know you took two steps forward. That shifts your idea of your position, and you feel a little less sure, because steps are never exact. Then your hand touches a doorframe. Now you are more sure again, and your idea jumps toward the door.
You just did what this chapter describes. You held a belief about where you are. You moved the belief when you acted. You reshaped it when you sensed something. Two moves, repeated forever. The lecture gives them names and proves they are exact.
At any time the world is in one actual state st. The agent never sees it. All the agent holds is the history ht = (o0:t, a0:t−1). From that, the agent builds the posterior p(st | ht): a distribution that gives every possible state a chance. Other books call it the belief state. The lecture stresses the difference: st is not the same thing as p(st | ht). One is a fact about the world. The other is the agent’s uncertainty about that fact.
This matters because the transition law needs st and the agent does not have it. So the agent carries the whole belief into the next prediction instead.
The agent takes an action at. Where is the world now? The agent cannot say for sure, so it tries every possible current state. For each one, it asks where the transition law would send it. Then it weights each answer by how likely that starting state was. The lecture writes this as an integral. An integral here is just a sum over a continuous range of states.
p(st+1 | ht, at) = ∫ p(st+1 | st, at) p(st | ht) dst
The first factor is the transition law, from the model. The second is the current belief, from the agent. The lecture derives this in three short moves: add the unknown state and then sum it out (marginalize it), split the joint chance with the product rule, and cut away what the assumptions make irrelevant. Two assumptions do the cutting. The transition is Markov. And the agent picks its action using only the history, so the action tells you nothing more about st.
The same recipe predicts a whole future. For a planned list of actions, the chance of the future outcomes is the world model’s prediction from each possible state, weighted by the agent’s belief: p(Yfuture | ht, afuture) = ∫ p(Yfuture | st, afuture) p(st | ht) dst. The world model supplies the first factor. The belief supplies the second. This is the exact link back to Lecture 2’s goal.
A new reading ot+1 arrives. Which of the predicted states explain it well? Bayes’ rule answers. Multiply each predicted state’s chance by how likely it would be to produce this reading, then rescale so the chances add up to 1.
The reading reweights the predicted states. States that explain it gain weight. States that do not lose weight.
A robot lives in a corridor of five cells, numbered 0 to 4. Its belief is [0, 0.2, 0.6, 0.2, 0]. It tries to move one cell right, but the move works only 80% of the time and otherwise the robot stays put.
Predict. Each new cell gets 0.8 of the belief from the cell on its left plus 0.2 of its own. Cell 1: 0.8·0 + 0.2·0.2 = 0.04. Cell 2: 0.8·0.2 + 0.2·0.6 = 0.28. Cell 3: 0.8·0.6 + 0.2·0.2 = 0.52. Cell 4: 0.8·0.2 = 0.16. Cell 0 gets 0. The new belief is [0, 0.04, 0.28, 0.52, 0.16]. It adds to 1, and it is wider than before.
Correct. The robot sees a door. Cell 3 has a door, and the sensor says “door” with chance 0.9 there and 0.1 elsewhere. Multiply: [0, 0.004, 0.028, 0.468, 0.016]. These add to 0.516, the normalizer. Divide each by 0.516: [0, 0.008, 0.054, 0.907, 0.031]. The belief jumped to 90.7% on cell 3.
The last slide of this block ties it together. After observing ot you hold p(st | ht). After the action you hold the prediction. After observing ot+1 you hold p(st+1 | ht+1). That last object has the same role at time t + 1 that the first had at time t. So it becomes the input to the next prediction, and the loop repeats for every time step. The belief is a running summary of the whole history.
The bars are the belief over 24 cells. Press Predict to move the belief by the action. Then place the sensor reading and press Correct. The yellow curve is the likelihood of the reading.
Illustrative: a discrete version of the loop on 24 cells with Gaussian move noise and Gaussian sensor noise, starting from a belief centred on cell 6 with spread 1.5. “Spread” is the standard deviation of the belief, in cells.
Press Predict a few times without correcting. The bars drift right and the spread grows, because every move adds doubt. Skip the predict step in a real system and the symptom is a belief that is too sharp: the robot is sure, and wrong. The measurement is simple: the error grows larger than the spread the belief reports. The fix is always to widen the belief when the world moves.
Chapter 6
Push the mean through the dynamics, grow the covariance, and see why a belief stretches and tilts when time passes
A ship’s navigator, long before satellites, worked out the ship’s position by dead reckoning: start from the last known position, add the speed and heading, and mark the new spot on the chart. The mark is a best guess. Every hour at sea makes the guess worse, so the navigator draws a bigger circle around the mark. That growing circle is the whole story of this chapter, written in matrices.
The man who made it exact was Rudolf Kalman. His 1960 paper, A New Approach to Linear Filtering and Prediction Problems, appeared in the Journal of Basic Engineering. The lecture gives the split of roles clearly. The LGSSM supplies the world-model probability laws. The Kalman filter is exact recursive inference under those laws.
The filter needs the model of Chapter 2 and a few promises. The start state s0 and every noise variable are independent of each other, across sources and across time. The covariances must behave: P0 ⪰ 0 and Q ⪰ 0 mean “positive semidefinite”, a polite way to say a covariance can never describe a negative spread. R ≻ 0 means the sensor noise is strictly positive, so it is never perfectly exact. This last point will keep a certain matrix invertible in Chapter 7.
Then the filter runs in three stages. Initialize: after the first reading o0, Gaussian conditioning gives the first filtered belief N(μ0|0, P0|0). Predict: a linear transition plus independent Gaussian noise gives another Gaussian. Correct: a linear-Gaussian reading gives yet another Gaussian. Because each stage returns a Gaussian, the loop repeats for every t, and the belief always lives in the same two-object family.
The lecture uses a two-index habit that is worth learning once. In μt+1|t, the first index is the time of the state we describe, and the second is the time of the latest reading we have used. So μt+1|t = E[st+1 | ht, at] is the mean of the next state before its reading arrives, and μt+1|t+1 is the mean after. The lecture shortens them to (μ−, P−) before and (μ+, P+) after. The pair (μ, P) is the entire belief. The mean alone is a point estimate, and it hides how unsure you are.
What is the average next state? Take the average of the rule st+1 = A st + B at + εst. Three facts finish it. The action at is chosen from the history alone, so it tells us nothing new about st, and the average current state is μt|t. The noise has mean zero. And the action is known, so it shifts every candidate state by the same vector and adds no spread.
μt+1|t = A μt|t + B at
Said plainly: run the old mean through the same physics you would run a single state through. The lecture adds a warning. If the actuation itself were unsure, you would need an extra random term or a bigger Q.
Now the spread. Write the current error as et = st − μt|t. Then the next error is et+1|t = A et + εst: the old error, pushed through A, plus a fresh kick. The covariance is the average of the error times its transpose (the transpose flips a grid over its diagonal, so rows become columns). Expand the square. Two cross terms appear, and both are zero, because the new kick is independent of the old error.
The result is the full prediction: p(st+1 | ht, at) = N(A μt|t + B at, A Pt|t AT + Q). The previous steps found its mean and covariance. Gaussian closure supplies the family name.
Take P0|0 = [[0.5, 0], [0, 1]] and Q = [[1.5, 0], [0, 1]]. First A P: row one is [1·0.5 + 1·0, 1·0 + 1·1] = [0.5, 1], and row two is [0, 1]. Then (A P) AT: row one is [0.5·1 + 1·1, 0.5·0 + 1·1] = [1.5, 1], and row two is [1, 1]. Add Q to get P− = [[3, 1], [1, 2]].
Read what happened. The position variance jumped from 0.5 to 3. One part came from Q (1.5) and one from the speed doubt leaking into position (1). The speed variance rose from 1 to 2. And the off-diagonal entry is now 1. It says: if the cart is further along than expected, it is probably also faster than expected. Position and speed are now correlated. That tilt is the seed of the next chapter’s trick.
The ellipse marks where the cart probably is (two standard deviations). Move the first slider to apply more predict steps. Change the second to scale the process noise Q.
The start belief and noise are the lecture’s numbers: μ = [0, 1], P = diag(0.5, 1), Q = scale × diag(1.5, 1), no action. One predict step with scale 1 gives the lecture’s P− = [[3, 1], [1, 2]].
Slide the steps to 5 and watch. The ellipse grows without limit and leans harder to the right. Nothing in the loop shrinks it. The symptom of a filter that only predicts is a position doubt that keeps growing: after 5 steps the position spread is several times the start. The measurement is the covariance itself, which grows at every step. The fix is the next chapter: bring in the readings.
Chapter 7
Compare the reading to the prediction, split the gap by how much each side deserves trust, and see how a position sensor moves the speed
Your model says the cart should be at 1.5. The sensor says 2.5. One of them is wrong, or both are a little wrong. How far should you move your belief toward the sensor? A friend who trusts the sensor fully says all the way. A friend who trusts the model fully says not at all. The right answer is in between, and it depends on how much each of them has earned your trust.
The Kalman correction computes that in-between answer exactly. It uses a number called the Kalman gain, written K. Think of K as a trust slider. At 0 you keep the prediction. At 1 you jump to the reading. The filter sets the slider for you, from the two uncertainties.
Before a reading comes, the filter can already say what it expects. The predicted reading is ôt+1|t = C μ−. The matrix C turns a state vector (n numbers) into a reading vector (d numbers). For the cart n = 2, d = 1, and C picks the position. The hat marks a prediction made before seeing ot+1.
The filter can also say how far off that guess may be. The reading is o = C s + εo, so the error in the predicted reading is C times the state error plus the sensor wobble. The two are independent, so their covariances add.
St+1 = C P− CT + R, p(ot+1 | ht, at) = N(C μ−, St+1)
S is the innovation covariance: how wide the predicted reading is. This Gaussian is exactly the normalizer from Chapter 5’s Bayes rule. When the real reading arrives, the gap between it and the prediction is the innovation (also called the residual): ν = ot+1 − C μ−. It lives in reading space. It is the surprise.
To fix the state from a surprise, the filter needs to know how the state error and the reading error relate. The lecture puts them in one joint Gaussian. The state block is P−, the reading block is S, and the cross block is Cov(s, o) = P− CT. The cross block is the key: it records how much a state error shows up in the reading.
The lecture’s goal: split the state error into a part that the reading determines and a leftover that is independent of the reading. Call the state error es = s − μ− and the reading error eo = o − C μ−. For any matrix L, write es = L eo + r, with the leftover r = es − L eo.
For Gaussians, two quantities with zero cross-covariance are independent. So choose L to make Cov(r, eo) = 0. That says P− CT − L S = 0, which gives L = P− CT S−1. That L is the Kalman gain.
The course’s handwritten notes reach the same gain by a second route. They write the posterior as the product of the sensor’s bell curve and the prediction’s bell curve, then find where its slope is zero and how sharply it curves. That gives P+ with an inverse that is the sum CT R−1 C + (P−)−1: precisions add, where a precision is one divided by a variance: a sharp belief has a big precision. Two routes, one answer, which is a good sign.
Drop to one dimension to see the slider. Let the state be a single position, C = 1, the prediction be μ− = 10 with P− = 4, and the sensor noise be R = 1. Then S = 4 + 1 = 5 and K = 4 / 5 = 0.8. A reading of 12 gives ν = 2, so μ+ = 10 + 0.8 · 2 = 11.6. The new variance is P+ = 4 − 0.8 · 5 · 0.8 = 0.8. The belief moved most of the way to the sensor, because the prediction was the shakier of the two. Make R = 16 instead and K = 4 / 20 = 0.2, so the mean barely moves to 10.4.
Now the cart. The prediction has mean [1.5, 2] and P− = [[3, 1], [1, 2]]. With C = [1, 0], the cross block P− CT is the first column of P−: [3, 1]. The second entry is 1, not 0. That is the position-speed correlation from Chapter 6. So the gain has a nonzero speed entry, and a surprise in position changes the speed estimate. This is the trick behind the hero: the sensor reads only position, but the correlation carries the news to speed.
The purple ellipse is the prediction. The yellow band is the reading and its noise. The warm ellipse is the corrected belief. Move the reading and the sensor noise.
The lecture’s numbers: μ− = [3/2, 2], P− = [[3, 1], [1, 2]], C = [1, 0]. At reading 2.5 and R = 1 the device reproduces the lecture’s correction: K = [3/4, 1/4], μ+ = [9/4, 9/4].
Slide R up to 6. The band widens, K falls, and the warm ellipse stays near the purple one: the filter ignores a sloppy sensor. Slide R down to 0.1 and the warm ellipse snaps to the band. Now the failure. Suppose you set R much too small for a sensor that is really noisy. The symptom: the estimate jitters, chasing every reading. The measurement: the average of ν2 / S over many steps lands far above 1, because the real surprises are larger than the filter expected. The fix: raise R until that average sits near 1.
The lecture ends the block with one box. Start from N(μt|t, Pt|t). Predict after action at: μ− = A μ + B a, P− = A P AT + Q. Correct after reading ot+1: ν = o − C μ−, S = C P− CT + R, K = P− CT S−1, μ+ = μ− + K ν, P+ = P− − K S KT. The new pair is the starting pair for the next step. It summarizes all the history this model needs.
Chapter 8
Work the lecture’s position and speed numbers from start to finish, and find out which entry lets a position sensor change the speed
Formulas are easy to nod at and hard to trust. The cure is to push real numbers through them with a pencil. This chapter does that for the lecture’s cart, one full predict and correct step, so that nothing in Chapters 5 to 7 stays a symbol.
First, the numbers. The matrices are the cart’s: A = [[1, 1], [0, 1]], B = [½, 1]T, C = [1, 0]. The noise levels are Q = [[1.5, 0], [0, 1]] and R = 1. The first push is a0 = 1.
The lecture starts from the prior N([0, 1], I): a mean position of 0, a mean speed of 1, and a covariance that is the identity (spread 1 in each, no link between them). Then the first reading arrives, o0 = 0, and conditioning gives the first filtered belief.
The innovation is ν = 0 − 0 = 0, so the mean does not move. The reading width is S = 1 + 1 = 2. The gain is K = [1, 0] / 2 = [½, 0]: position gets half the trust, and speed gets none, because the prior says position and speed are unrelated. The covariance becomes P0|0 = [[1 − ½·2·½, 0], [0, 1]] = [[½, 0], [0, 1]]. We start from μ0|0 = [0, 1] and P0|0 = diag(½, 1).
Before we compute, learn how to read a covariance matrix for this cart. It is a 2 by 2 grid. The top-left entry, Var(p), is how unsure we are about position. The bottom-right entry, Var(v), is how unsure we are about speed. The two off-diagonal entries are equal, and each is Cov(p, v): how the two errors rise and fall together. A positive value means “if the cart is further along than we think, it is probably faster than we think too”. A zero means the two doubts are unrelated.
The mean goes through the physics. A μ = [0 + 1, 1] = [1, 1]. The push adds B a0 = [½, 1]. So μ− = [3/2, 2].
The covariance, from Chapter 6, is A P AT + Q = [[3/2, 1], [1, 1]] + [[3/2, 0], [0, 1]] = [[3, 1], [1, 2]]. Look at the two entries on the diagonal. They are the doubts about position (3) and about speed (2). Now look at the off-diagonal entry, 1. It is the cross-covariance: how the two errors move together. It is new. Our start belief had a zero there. The prediction made it nonzero, because position depends on speed.
Suppose the sensor reports o1 = 5/2. The predicted reading is C μ− = 3/2, so the surprise is ν = 5/2 − 3/2 = 1. The reading width is S = C P− CT + R = 3 + 1 = 4.
The gain is K = P− CT / S. The product P− CT is the first column of P−, which is [3, 1]. Divide by 4: K = [3/4, 1/4]. The mean update is μ+ = [3/2, 2] + [3/4, 1/4] · 1 = [9/4, 9/4]. The covariance update is P+ = P− − K S KT = [[3, 1], [1, 2]] − 4 · [[9/16, 3/16], [3/16, 1/16]] = [[3/4, 1/4], [1/4, 7/4]].
Check three things. The position doubt fell from 3 to 3/4, a good drop from one reading. The speed doubt fell from 2 to 7/4, a small drop. And the speed estimate moved from 2 to 9/4, although the sensor reads only position. The lecture’s sentence says it all: cross-covariance carries information, so a position-only sensor can update the speed.
Move through the three stages. Choose the reading o1. Then switch the cross-covariance off to see what the speed estimate loses.
The matrices, noise levels and the push are the lecture’s. The lecture works only o1 = 5/2; the other two readings are our own, to show the same arithmetic moving. “Drop the cross-covariance” zeroes the off-diagonal of P− before the correction, as a thought experiment.
At o1 = 2.5, stage Correct, the box shows exactly the numbers above. Now press the red toggle. The off-diagonal of P− is forced to 0, the gain on speed becomes 0, and the speed estimate stays at 2 whatever the sensor says. That is a real failure, not just a thought experiment: some code approximates the covariance by its diagonal to save work. The symptom is a speed estimate that never reacts to position readings. The measurement: the second entry of the gain K is exactly 0 at every step. The fix: keep the full covariance matrix.
Chapter 9
Choose the model numbers that make a recorded run most probable, and see the Kalman filter hand you that probability for free
So far we knew the model. Someone gave us A, B, C, Q and R. In real life nobody does. You have a recording of sensor readings and the actions that were taken, and you must work out how noisy the sensor is, how hard the world kicks the state, and what the dynamics are. This is learning, and it is a different job from the filtering you just did.
The lecture draws the line cleanly. In inference the parameters θ = (μ0, P0, A, B, C, Q, R) are fixed and we estimate hidden states. In learning θ is unknown, the states stay hidden, and the recorded readings decide the fit.
Maximum likelihood says: pick the θ under which the data you saw was most probable. The data is only the readings o0:T, given the known actions. The states are never recorded. So the score must add up over every hidden path that could have produced the readings. This is the marginal likelihood.
ℓ(θ) = log pθ(o0:T | a0:T−1) = log ∫ pθ(s0:T, o0:T | a0:T−1) ds0:T, θ★ = argmaxθ ℓ(θ)
That integral runs over a whole trajectory of hidden states, which sounds hopeless. The Kalman filter makes it easy. Split the chance of the whole recording into one chance per reading, each given everything before it: p(o0:T) = p(o0) ∏ p(ot | ht−1, at−1). And each of those chances is the Gaussian from Chapter 7: N(ot; C μt|t−1, St). The filter already computes the centre and the width at every step.
Take the negative log of one factor, so that “more probable” becomes “lower cost”. The lecture writes it like this.
The two middle terms pull in opposite directions, and that is the beauty of the score. Say the surprise was ν = 2. With a predicted width S = 5, the cost is ½[1.84 + ln 5 + 4/5] = ½[1.84 + 1.61 + 0.80] = 2.12. Inflate the width to S = 20 and the error term shrinks to 0.20, but ln 20 = 3.00 grows more, and the cost rises to 2.52. Shrink it to S = 1 and the error term balloons to 4.0: cost 2.92. The best width is S = ν2 = 4, with cost 2.11. A model cannot cheat by declaring itself uncertain, and it cannot cheat by being overconfident.
Add up the costs over the whole recording. The first reading uses the prior, S0 = C P0 CT + R. The later ones use the filter’s own νt and St. That sum is L(θ) = −ℓ(θ).
Direct gradients. L(θ) is a smooth function of the model numbers. Run the filter forward, add up the costs, and let automatic differentiation (software that works out the slope of the cost by itself) carry derivatives back through every predict and correct step. Then take a step downhill: θ ← θ − η ∇θ L(θ), where η is the learning rate, the size of each downhill step. A change to θ changes every later prediction, so the gradient flows through the whole loop. The covariances must stay legal: P0 and Q positive semidefinite, and R positive definite so that S is always invertible.
Expectation-maximization (EM). This is the optional second route. It improves the same objective by alternating two steps. The E-step fixes θ and uses the complete recording to compute a posterior over the hidden trajectory. That offline pass, which uses readings from the future as well, is called smoothing. The M-step then refits θ using the smoothed states and their products, and for linear-Gaussian models this refit often has a closed-form answer. EM improves a lower bound on the likelihood that touches the true likelihood at the current θ, so an exact E-step and M-step can never make the likelihood worse. The lecture is honest about the trade: convenient updates for classical models, but no promise of faster convergence or of the same local answer as gradients.
A recorded run of 60 readings is fixed. The curve is the total cost L(θ) from the Kalman filter, for every assumed sensor noise. Drag the assumed noise and look for the lowest point.
Illustrative: one simulated cart run with the hero’s model, true sensor noise 1.0 and true process noise 0.1, and a vague prior (position spread 5, speed spread 2). The cost is the sum of Lt over all 60 readings, computed by the Kalman recursion in your browser. Noise levels are standard deviations.
Put the assumed noise at 0.4 and the cost is high: the model thinks the sensor is exact, so honest wobble looks like a shock. Drag it up and the cost falls, bottoms out near the true value of 1, then rises again as the model grows needlessly vague. The data picks the answer. A real fit does the same dance in many dimensions at once.
A failure to watch. A flexible model can fit one recording almost perfectly by shrinking R toward zero and explaining every wobble as real motion. The symptom is a model that looks great on its training run and poorly on a new one. The measurement is the cost on a held-out run, which jumps. The fix: constrain R to stay positive, and always check on data the fit never saw.
The block ends with four roles. The world model is the initial, transition and observation laws, set by θ. Filtering gives the online posterior p(st | o0:t, a0:t−1; θ). Prediction gives future readings p(ot+1 | ht, at; θ). The objective is the marginal log likelihood, and the optimizer is direct gradients or EM.
Chapter 10
Swap the continuous state for a handful of modes, then meet the five places where the linear-Gaussian recipe stops being right
A fitness band cannot see whether you are sitting, walking or running. It sees a stream of small shakes. You are always in one of three modes, and the shakes hint at which. There is no smooth position or speed here. There are just three boxes, and you are in exactly one.
The lecture calls this a hidden Markov model (HMM): a state-space model whose state is discrete. The state st takes one of L values. The three laws are the same three cards as before, now written as tables: an initial table p(s0), a transition table p(st+1 | st), and an observation table p(ot | st). For simplicity the lecture leaves out actions here.
The Bayes loop of Chapter 5 is unchanged. The integrals just become sums over L boxes, and the belief is a list of L probabilities instead of a mean and a covariance.
predict: p(st+1 = j | o0:t) = ∑i=1L p(st+1 = j | st = i) p(st = i | o0:t) correct: p(st+1 = j | o0:t+1) ∝ p(ot+1 | st+1 = j) p(st+1 = j | o0:t)
A worked step with illustrative numbers. Let the modes be Parked, Walking and Running, with belief [0.5, 0.3, 0.2]. A sticky transition table says Parked stays Parked 80% of the time, Walking stays 70%, Running stays 70%. Predict. Parked: 0.8·0.5 + 0.1·0.3 + 0.05·0.2 = 0.44. Walking: 0.15·0.5 + 0.7·0.3 + 0.25·0.2 = 0.335. Running: 0.05·0.5 + 0.2·0.3 + 0.7·0.2 = 0.225. The three add to 1.
The band now reports a high shake. The observation table gives that reading 0.02 chance if Parked, 0.15 if Walking and 0.8 if Running. Multiply: 0.0088, 0.0503 and 0.18. They sum to 0.239. Divide: the new belief is [0.037, 0.210, 0.753]. One shake moved the band from 20% Running to 75%. Try it yourself.
Each press feeds the band one reading. The filter predicts with the sticky transition table, then corrects with the observation table. Dashed outlines show the prediction.
Illustrative tables. Transitions (rows are the current mode): Parked [0.8, 0.15, 0.05], Walking [0.1, 0.7, 0.2], Running [0.05, 0.25, 0.7]. Observations (low, medium, high): Parked [0.85, 0.13, 0.02], Walking [0.15, 0.7, 0.15], Running [0.02, 0.18, 0.8]. Start belief [0.5, 0.3, 0.2].
Learning works the same way as before: maximize the marginal likelihood, by direct gradients or by EM. For an HMM the E-step is the forward-backward pass, which smooths over the discrete states.
The lecture closes with a numbered list. Each item is a promise of the linear-Gaussian model, and what to do when the promise fails.
Item four deserves a picture. Slide the gap below and watch where the single Gaussian puts its weight.
The cart will pass the obstacle on the left or on the right. Blue is the true spread of futures. Warm is the one Gaussian with the same mean and variance. Drag the gap.
Illustrative: two equal routes, each a bell curve of spread 0.5, centred half the gap either side of the obstacle. The obstacle fills positions from −0.5 to 0.5. The warm curve matches the mean (0) and variance of the two-route mixture, as a Kalman prediction would.
At a gap of 3, the true chance of ending inside the obstacle is about 2%. The Gaussian says about 25%. If a planner trusts that Gaussian, it will fear a collision that cannot happen on average, and it may also aim straight at the obstacle’s centre, the one place nobody goes. The symptom: a plan that steers through a wall. The measurement: compare the filter’s mass in a region with the fraction of real runs that end there. The fix: a model with several modes.
The lecture zooms out. A sequence model can carry the useful past in two ways. One is an explicit generative state with named transition and observation laws: a simulator, an LGSSM, an HMM. The other is a recurrent predictive memory: history compressed inside a predictor, as in an RNN’s hidden memory or a video model’s latent. That memory may live on the agent’s side, not as a world-side variable. The lecture’s summary rule is a good one to keep: choose a valid state and probability laws first, then choose inference and learning methods that fit those assumptions.
Next week, Lecture 5, asks how to learn useful codes zt = Eφ(ot; o<t, a<t) from raw observations with no state labels. Three questions drive it: what should stay unchanged (invariance), what should change in a predictable way (equivariance), and what keeps the code from collapsing to nothing. A contextualized code need not be Markov, so a predictor may still need the whole sequence of codes.
Chapter 11
Keep the numbers, the terms, a checklist and the core code, and see where each one leads next
A state-space model writes a world as three laws: how a run starts, how the hidden state moves under actions, and how a sensor shows the state. The linear-Gaussian version makes all three exact and tiny: a mean and a covariance carry the whole belief. The Kalman filter keeps that belief by a loop of predict and correct, and a position-only sensor can fix the speed through the cross-covariance. Observability tells you if the readings could ever reveal the state. The same filter hands you the likelihood, and the likelihood trains the model by gradients or EM.
| Term | What it is | The cart’s value |
|---|---|---|
| State st | Everything the next step needs; hidden | [p, v], n = 2 |
| Observation ot | The noisy sensor reading | p + noise, d = 1 |
| A, B, C | Transition, input and observation matrices | [[1, 1], [0, 1]], [½, 1]T, [1, 0] |
| Q, R | Process noise and sensor noise covariances | diag(3/2, 1), 1 |
| Observability OH | Stack of C Ak; full column rank means no ambiguity | rank 2 from H = 2 |
| Prediction (μ−, P−) | A μ + B a, and A P AT + Q | [3/2, 2], [[3, 1], [1, 2]] |
| Innovation ν and S | Surprise, and its predicted spread | 1 and 4 |
| Gain K | P− CT S−1, the trust slider | [3/4, 1/4] |
| Correction (μ+, P+) | μ− + K ν, and P− − K S KT | [9/4, 9/4], [[3/4, 1/4], [1/4, 7/4]] |
| Cost L(θ) | Sum of ½[d log 2π + log |S| + νT S−1 ν] | minimized by gradients or EM |
All the cart values come from the lecture’s worked example (slides 38 and 39).
This is the whole filter step plus the cost of one reading, in NumPy. It reproduces the lecture’s example: starting from μ = [0, 1], P = diag(0.5, 1), a = 1 and o = 2.5 it returns μ+ = [2.25, 2.25] and P+ = [[0.75, 0.25], [0.25, 1.75]].
pythonimport numpy as np A = np.array([[1., 1.], [0., 1.]]) # (n, n) = (2, 2) B = np.array([[.5], [1.]]) # (n, m) = (2, 1) C = np.array([[1., 0.]]) # (d, n) = (1, 2) Q = np.diag([1.5, 1.0]); R = np.array([[1.0]]) def kalman_step(mu, P, a, o): """mu (n,1), P (n,n), action a (m,1), reading o (d,1).""" # predict mu_m = A @ mu + B @ a # (n,1) P_m = A @ P @ A.T + Q # (n,n) # correct nu = o - C @ mu_m # (d,1) innovation S = C @ P_m @ C.T + R # (d,d) K = P_m @ C.T @ np.linalg.inv(S) # (n,d) gain mu_p = mu_m + K @ nu P_p = P_m - K @ S @ K.T # one-reading cost, summed over t to get L(theta) nll = 0.5 * (len(o) * np.log(2 * np.pi) + np.log(np.linalg.det(S)) + (nu.T @ np.linalg.inv(S) @ nu).item()) return mu_p, P_p, nll
To learn A, Q or R, wrap this step in a loop over the recording, add up nll, and let an automatic-differentiation library take gradients. The course’s reading, the dynamax Kalman filter tracking notebook, tracks an object in two dimensions with a four-number state (two positions, two speeds) and a position-only sensor, and runs it with a real library, including offline smoothing.
State-space models are the oldest answer to “what does the agent believe about a world it cannot see?”, and they sit under many modern world models. A learned latent state in a video model plays the role of st. A belief state in a decision problem is the posterior of Chapter 5. Keep going with these lessons.
Now explain, from memory, how a position-only sensor can change a speed estimate, and why the belief must widen between readings. Then go back to the hero, raise the sensor noise, and watch the filter handle it.