State-Space Models

A noisy sensor reports only where a cart is. A Kalman filter still recovers its hidden speed, and keeps that speed error small while raw sensor differences go wild.

Learn how a world model keeps a hidden state, how it predicts the next state, how one new observation corrects that guess, and how the model learns its own numbers.

Drag the sensor noise up, then switch between the three ways to estimate. Watch the speed panel at the bottom. Then we build, piece by piece, the model, the Bayes loop, the Kalman filter, and the learning that sits underneath.

You need a rough idea of a probability (a number from 0 to 1 for how likely something is) and what a matrix is (a grid of numbers that turns one list of numbers into another). We teach the rest from zero.

Based on Lecture 4 of CIS 6280: World Models (University of Pennsylvania, Fall 2026), taught by Jiatao Gu, with the course’s lecture slides, its handwritten LGSSM notes and the dynamax Kalman filter tracking notebook. Every idea is re-taught here in our own words and our own devices.

Track a cart you cannot see

1.0

Illustrative: a 1D cart simulated here with the lecture’s model (position p, speed v, a known push a, one noisy position sensor, step rule p+v+½a and v+a) and process noise of standard deviation 0.1. The Kalman filter is the lecture’s predict then correct. Errors are root-mean-square over steps 5 to 59.

Chapter 0

Why a World Model Keeps a Hidden State

See why one sensor reading cannot tell two different futures apart, and why a model must carry more than it can see

A delivery cart rolls down a long corridor. At the far end, a sensor reports where the cart is once every second. The reading wobbles a little each time, because no sensor is perfect.

You need to say where the cart will be in four seconds. You have the latest reading: 3 metres from the wall. Is that enough?

It is not. A cart that creeps at 1 metre per second and a cart that races at 2 metres per second can both be at 3 metres right now. The reading looks the same. The futures are very different. What is missing is the speed, and the sensor never reports it.

This lecture is about that gap. It gives a name to the missing piece, a model for how the missing piece changes, and an exact method to guess it from the readings. The method is the Kalman filter, from 1960, and it is still the backbone of many tracking and robot systems.

A short note on the course

Lecture 4 of CIS 6280 opens with three course updates. An Ed Discussion board is open for your questions. The final exam falls in the last week of teaching. The release of Assignment 1 moves to Thursday, September 10. That is all the logistics there are, so we move on to the idea.

What Lecture 3 gave us, and what it hid

Lecture 3 built simulators: programs that hold the full truth about a toy world and step it forward. The slide shows two T-shaped blocks pushed around in a simulator. The camera picture shows the outcome: where the blocks sit and touch. The simulator knows more: speeds and physical numbers too.

Four words from that lecture matter now. Learn them well, because every later chapter uses them.

In Lecture 3 you could read st straight out of the simulator. A real learner cannot. It may receive only ot. When the state is there but you cannot see it, we call it a hidden state. It is like the engine of a car: you hear it and feel it, but you never look at it directly.

A world model is a model that predicts what happens next in the world. If the world has a hidden state, the model must keep a guess of that state, because the next observation depends on it. Everything in this lecture serves that one need.

Compare two carts that look the same

Both carts sit at 3 metres, so the sensor frame is identical. Set each cart’s hidden speed with the sliders and watch where each reading will be over the next four steps.

1.0
−1.0

Illustrative: no push is applied, so each cart moves p + v every step. The speeds are yours to set; the lecture’s own 1D model appears in Chapter 3.

Set both speeds the same and the two futures merge. Set them apart and the readings drift away from each other, although the first frame never changed. The frame shows the position. The future needs the speed too.

So the plan for this lesson has three parts. First, a model of how a hidden state moves and how it shows itself through the sensor. Second, a way to keep a running guess of the state while readings arrive. Third, a way to learn the model’s numbers from recorded data. The hero at the top of the page already runs the second part: it guesses both position and speed from one noisy position sensor.

The road

  1. State, history and representation: what “Markov” means, and why a state beats a pile of old frames.
  2. The state-space model: three probability laws that generate a whole trajectory.
  3. A cart, one sensor and three matrices: the lecture’s 1D example, read as motion and sensing.
  4. Observability: when can readings reveal the hidden state at all?
  5. The Bayes loop: predict the next state, then correct it with the new reading.
  6. Kalman predict: how the mean and the uncertainty move forward.
  7. Kalman correct: the gain, and why a position sensor can fix the speed.
  8. One step by hand: the lecture’s numbers, worked end to end.
  9. Learning the model: marginal likelihood, gradients and EM.
  10. Other state spaces and the limits: hidden Markov models, and five ways the linear-Gaussian model breaks.
  11. Field guide: numbers, checklist, core code, and where to go next.
Two carts give the same position reading now. Four steps later their readings are far apart. What did the first reading leave out?

Chapter 1

State, History and Representation

Separate three ways to remember the past, and learn what makes a state the right thing to remember

You watch a video of a ball. You pause on one frame. Where will the ball be in a moment? You cannot say, because a still frame has no motion in it. Now you step back one frame. The ball moved one square to the right. Suddenly you can guess: one more square, then another.

You did not learn a new fact about the world. You used the past to recover something the present frame hid. That small act is the heart of this chapter. An agent has three different ways to carry the past with it, and the lecture names all three.

Way one: keep the whole history

Lecture 2 set the goal of a world model. Given what the agent knows so far and a plan of actions, predict what comes next. The lecture writes it as pθ(Yfuture | ht, afuture). Read it aloud: the probability of the future outcomes Y, given the history and the actions we choose. The symbol θ stands for the model’s own numbers, which Chapter 2 names.

The history, written ht, is everything the agent has seen and done up to now: ht = (o0:t, a0:t−1). The small notation o0:t means the list o0, o1, up to ot. Notice the actions stop one step earlier, because the action at has not been taken yet.

Keeping the whole history is safe, since nothing is thrown away. It is also a problem. The history grows by one more observation and action every step. After an hour of video it is enormous, and the model must reread all of it to make each prediction.

Way two: compress it into a representation

A representation is a short code that stands in for something larger. The lecture’s version is zt = Eφ(ot; o<t, a<t). An encoder E, with its own learned numbers φ, reads the current observation and a view of the past, and returns one code zt. The lecture calls zt a contextualized representation: it describes the current observation using past context.

There is a catch, and the lecture states it plainly. A representation makes no promise to be a full summary. Prediction may still need the whole sequence of codes, z≤t. The code helps, but it does not close the question. Lecture 5 returns to how such codes are learned.

Way three: keep a state that is Markov

The third way is the neat one. Suppose you could name a quantity st that holds everything the future needs. Then the past would add nothing once you knew st. That is the Markov property, and a quantity with it is a Markov state.

p(st+1 | s0:t, a0:t)
The chance of each next state if you know every earlier state and every action. This is the long, honest version.
p(st+1 | st, at)
The chance of each next state if you know only the current state and the action you take now. The lecture says: given this state and the next action, the model can predict without earlier states.

Think of a save file in a video game. The file holds the level, your health and your items. You can load it tomorrow and play on, and nothing about how you got there matters. A replay of every button press holds the same information, but you would not want to carry it around. The save file is a Markov state. The replay is a history.

Now the key move, and it matters for the whole lecture. The Markov property talks about prediction when the state is known. In a real system you often do not know st. You hold only codes z≤t and past actions. To find the state you must do state inference: compute p(st | z≤t, a<t), a distribution over states that says how sure you are. Inferring an unknown state can still need the history. The result describes your uncertainty about st, which is different from st itself.

So the three ways line up like this. The history is complete but too big. A representation is small but may not be complete. A Markov state is small and complete, but you usually have to infer it. The rest of the lecture builds the machine for that inference.

A worked example with the cart

Take the cart from Chapter 0. A frame shows position pt = 5.0. The cart’s speed is 2.0, so the next position is about 7.0. Three guesses are possible.

Guess one uses position only: “the cart stays at 5.0.” It misses by 2.0. Guess two uses the history: the last two positions were 3.0 and 5.0, so the cart moved 2.0 and should move 2.0 more. That gives 7.0, an almost perfect guess, from history alone. Guess three uses the state (position 5.0, speed 2.0) and adds them: 7.0. The state and the two-frame history give the same answer, because the state is a compact record of the history that matters.

Predict the next position three ways

Each dot is one simulated cart. Across is the position you predict. Up is where the cart really went. Dots on the diagonal are perfect. Choose what the predictor may look at.

Illustrative: 90 simulated carts with positions from 0 to 10, speeds from −2 to 2, and process noise of standard deviation 0.1 at each step. “Misses by” is the root-mean-square gap between the guess and the true next position.

The picture shows why we want a state. With position only, the dots form a wide cloud, because the missing speed spreads the outcomes. With two positions or with the full state, the dots hug the diagonal. History and state carry the same information. The state is simply the smaller, tidier package.

A learner only sees codes z and its own past actions, never the true state. What does the lecture say about the Markov property here?

Chapter 2

The State-Space Model: Three Laws That Make a Trajectory

Write a world with a hidden state as three small probability rules, then run them to generate a whole run of the world

Imagine you write the rules of a tiny world on three index cards. Card one says how the world starts. Card two says how it moves when you push it. Card three says what a sensor would show you at any moment. Shuffle nothing, add nothing else. With those three cards you can play the world forward as often as you like, and every run will be a little different.

That is a state-space model (SSM). The lecture calls it a world model with a Markov state: a generative process over trajectories. A trajectory is one full run: the states, the observations and the actions in order.

The four moves of one run

The lecture draws the process as four moves. Each move uses one probability law, and each law has a plain meaning.

  1. Initialize and observe. Draw a start state from the initial law pθ(s0). Then draw the first reading from the observation law pθ(o0 | s0).
  2. Intervene. Choose an action at. The model does not draw this. You, or a policy, pick it.
  3. Evolve. Draw the next state from the transition law pθ(st+1 | st, at).
  4. Observe. Draw the new reading from the observation law pθ(ot+1 | st+1). Then go back to move two.

The symbol θ stands for all the numbers that define the model. Estimating θ from recorded trajectories is the job of Chapter 9. Notice what the readings depend on: only the current state, never the older readings. Notice what the next state depends on: only the current state and action. That is the Markov property from Chapter 1, now built into the model.

Because each step uses just one law, the chance of a whole trajectory is a product of small pieces. This is the same trick you saw for sentences in a language model: multiply one chance per step.

pθ(s0:T, o0:T | a0:T−1)
The chance of one whole run: every hidden state and every reading from time 0 to T, given the actions you chose.
pθ(s0)
The initial law: how the run starts.
∏ pθ(ot | st)
One observation law for each time step, multiplied together. Each reading depends only on the state at its own time.
∏ pθ(st+1 | st, at)
One transition law for each step, multiplied together. Each state depends only on the previous state and the action.

The classical choice: linear and Gaussian

A model needs real formulas for its three laws. The lecture picks the oldest and cleanest choice, the linear-Gaussian state-space model (LGSSM), from R. E. Kalman’s paper of 1960. It makes three promises.

First, the state is a list of real numbers, a vector st in Rn. For our cart n = 2: position and speed. Second, the maps are linear: the next state is a fixed matrix times the current state, plus a fixed matrix times the action, and the reading is a fixed matrix times the state. Third, all randomness is Gaussian: the familiar bell curve, fully described by its centre (the mean) and its spread (the covariance, a grid of numbers that records how wide the spread is in each direction and how directions move together).

The lecture writes the three laws like this.

s0 ~ N(μ0, P0)    st+1 = A st + B at + εst,  εst ~ N(0, Q)    ot = C st + εot,  εot ~ N(0, R)

Take the pieces one at a time. The process noise εs is a small random kick that the world adds to the state at each step: a bump in the corridor, a gust of wind. Its size is set by Q. The observation noise εo is the sensor’s wobble, set by R. The matrix A says how the state moves by itself, B says how the action pushes it, and C says which part of the state the sensor can read.

The data flow has fixed shapes. For the cart, s is a 2-vector, a is one number, o is one number, A is 2 by 2, B is 2 by 1, C is 1 by 2, Q is 2 by 2 and R is a single number. Chapter 3 fills them in.

Why choose such strict rules? Because two things stay true, and they make everything else easy. Add a linear map to a Gaussian and you still have a Gaussian. Observe a Gaussian through a linear sensor and the updated belief is still Gaussian. So the whole uncertainty about the state fits in two objects, a mean and a covariance, forever. The lecture says it this way: under these assumptions, prediction and correction have closed-form Gaussian updates.

The lecture adds one more assumption: independence. The start state and all noise terms are independent of each other, and the noise at one time is independent of the noise at every other time. Each kick and each wobble is a fresh draw. Run the model below and see this for yourself.

Run the three laws step by step

Pick an action, then press Step. Blue dashed lines are the hidden state. Yellow dots are the sensor readings. Change a noise slider to start a fresh run.

0.10
0.6

Illustrative: the cart model of Chapter 3 with start law N(μ0 = [0, 0], P0 = diag(1, 0.25)) and the noise levels shown, as standard deviations. The numbers in the read-out are this run’s own draws.

Run it with both noises at zero. The trajectory is a clean curve that the three laws fully decide. Add process noise and the true path wanders. Add sensor noise and the yellow dots scatter around the path. The two noises do different jobs, and a filter must tell them apart: process noise changes the truth, sensor noise only changes what you see of the truth.

A failure worth knowing: the noise terms are assumed independent. Suppose a real sensor drifts, so each reading is wrong in the same direction as the last. The symptom is a filter that is too sure of itself. You measure it by checking whether the residuals, the gaps between predicted and actual readings, correlate from one step to the next. The fix is to put the drift into the state, so the model can estimate it too.

In the linear-Gaussian model, which term describes how wrong the sensor is, and which describes how the world itself gets pushed off course?

Chapter 3

A Cart, One Sensor and Three Matrices

Turn a story of motion and sensing into the matrices A, B and C, and see what each one does to a state

Back to the corridor. The cart has a position p and a speed v. You can push it, and the push changes its speed. One sensor reads the position, with a wobble. That is the whole world. The lecture uses it to make the abstract model concrete, and so do we.

The state is two numbers stacked in a vector: st = [pt, vt]. The observation ot is one number. The action at is also one number, and the lecture is careful to say what it is: an acceleration-like push applied over one model step. It is not the relative target command from Lecture 3.

The physics, one step at a time

Let one model step last one unit of time. In that time the cart coasts at speed v, so its position grows by v. If you also push with acceleration a, the speed grows by a. A push that builds up smoothly over the step moves the cart an extra half of a, because the cart only has the full extra speed at the end. This is the rule from school physics, distance = v t + ½ a t2 with t = 1.

pt+1 = pt + vt + ½ at + εpt     vt+1 = vt + at + εvt     ot = pt + εot

The first two rules say how the state evolves. The last says what the sensor measures: one noisy number, the position. The lecture adds a remark that carries the whole chapter. Speed shapes future positions, but the sensor does not read it directly. Repeated position readings can still reveal the speed, because the speed leaves a trace in how the position changes. Chapter 4 proves exactly when that works.

From story to matrices

A matrix is a grid of numbers that turns one list of numbers into another. Multiply a matrix by a vector and each output number becomes a weighted sum of the input numbers. The three rules above are three such sums, so we can pack them into three matrices.

Now the three rules are the lecture’s two lines: st+1 = A st + B at + εst and ot = C st + εot. The shapes match what Chapter 2 promised: A is 2 by 2, B is 2 by 1, C is 1 by 2.

A step with every number shown

Start from s = [2, 1]: the cart is at 2 and moves at 1. Push with a = 0.5, and ignore noise for now.

First, A s = [1·2 + 1·1, 0·2 + 1·1] = [3, 1]. The cart coasted from 2 to 3 and kept its speed. Second, B a = [½·0.5, 1·0.5] = [0.25, 0.5]. The push adds 0.25 to the position and 0.5 to the speed. Add the two: st+1 = [3.25, 1.5]. Finally the sensor computes C st+1 = 3.25. It reports a position and says nothing about the speed of 1.5. Try other numbers below.

Push a state through A, B and C

Set the cart’s state and the push. The purple ghost is where A alone takes the cart. The blue cart is the next state after the push. The yellow tick is what the sensor would read.

2.0
1.0
0.50

    

Noise-free on purpose, so each term of the arithmetic shows. The matrices are the lecture’s: A = [[1, 1], [0, 1]], B = [½, 1]T, C = [1, 0], one model step is one unit of time.

Slide the speed to zero and A does nothing: the cart stays where it is, and only the push moves it. Slide the push to zero and only A acts. Look at the second number in the pbox: it never reaches the sensor line.

A failure to watch for. Suppose you write B wrongly, say B = [0, 1], and forget the ½ a term. The symptom: after every push, your estimate of the position lags a little behind the truth. The measurement: the residuals, the gaps between the predicted and the real readings, spike exactly when the action is non-zero. The fix: correct B, or if the physics is truly uncertain, give the model a larger Q so it does not trust its own rule.

One model, three jobs

The lecture closes this block with a promise. Once the three laws are fixed, the same model supports three different operations. They are the plan for the rest of the lesson.

  1. Modeling (generate). The initial, transition and observation laws define a whole distribution over trajectories. You can sample runs from it, as Chapter 2 did.
  2. Inference. With the model fixed, compute the posterior pθ(st | o0:t, a0:t−1), the chance of each hidden state given what you have seen. In this lecture that computation is the Kalman filter (Chapters 5 to 8).
  3. Learning. Estimate the model’s own numbers, μ0, P0, A, B, C, Q and R, from recorded trajectories by maximum likelihood, meaning: choose the numbers that make the data most probable. One algorithm for this is called EM (Chapter 9).
For the cart, what does the top row of A = [[1, 1], [0, 1]] do?

Chapter 4

Observability: Can the Readings Reveal the State at All?

Test, before you build any filter, whether two different states could ever produce the same readings

A detective has a list of clues and two suspects. If the clues fit both suspects equally well, no amount of clever reasoning will pick one. The detective needs a new clue, not a better method. A state estimator is in the same position. Before you build one, ask a blunt question: could the clues ever tell the possible states apart?

The lecture calls this observability. It asks: from the history ht = (o0:t, a0:t−1), can the agent work out the state st that the model uses to predict? The answer can fail in two quite different ways, and it helps to keep them apart.

The second problem is the one a Kalman filter handles. The first is a flaw in the setup, and a filter cannot repair it. So the lecture isolates the first one with a clean test.

The noise-free test

Throw the noise away: set both noise terms to zero. Keep what you know for sure: the matrices A, B and C, and the action sequence. Now ask the structural question. Given a sequence of readings, can you uniquely determine the state at the start of the window?

Try it on the cart. One ideal reading says ot = pt. That fixes the position exactly. The speed could be anything: 0, 1, 5. A single frame is not enough to recover a two-number state, which is just what Chapter 0 showed.

Add one more reading. The next position obeys pt+1 = pt + vt + ½ at, and the sensor reads ot+1 = pt+1. Rearrange, with the push at known: ot+1 − ½ at = pt + vt. Since pt = ot, the speed is vt = ot+1 − ot − ½ at. Take readings of 2.0 and 3.25 with a push of 0.5. Then v = 3.25 − 2.0 − 0.25 = 1.0. The second reading exposed the speed, although the sensor never reads speed.

The same two equations in matrix form make the test mechanical.

[ot, ot+1 − ½ at]T = [[1, 0], [1, 1]] [pt, vt]T,   det = 1,   rank = 2

The determinant (det) of a 2 by 2 matrix [[a, b], [c, d]] is ad − bc. A nonzero determinant means the matrix can be undone: from the two readings you can solve back for the two unknowns. The rank counts how many rows carry independent information. Rank 2 for 2 unknowns is exactly what we want.

The observability matrix

Now generalize. Collect H readings, ot to ot+H−1. Each reading is the state pushed through A some number of times, then read by C: ot = C st, ot+1 = C A st + C B at, and so on. Subtract the known action terms, because you know them, and stack what remains.

õt:t+H−1
The stacked readings with the known action effects subtracted. The tilde marks that clean-up.
OH
The observability matrix. Its rows are C, then C A, then C A squared, and so on. It has d times H rows (d is the size of one reading) and n columns (n is the size of the state).
st
The unknown state at the start of the window.

Here is the test, in the lecture’s own words. If the rank of OH is below n, then some nonzero change δ has OH δ = 0. Then OH(s + δ) = OH s: the state s and the state s + δ produce identical readings. That is permanent ambiguity. If OH has full column rank, no such δ exists, and the state is uniquely determined.

Note the lecture’s careful caveat. One reading may not fix the state uniquely. With a prior and an observation model you can still say which states are more likely. Ambiguity limits what the data alone can settle, not what a Bayesian estimate can say.

Find the states that look identical

The dot is the true state. Everything in the green or red mark gives exactly the same noise-free readings as the dot. Pick a sensor, then add readings with the slider.

1

    

Noise-free, with the lecture’s A = [[1, 1], [0, 1]]. The true state is [3, 1]. A state s′ is drawn when OH(s′ − s) = 0, found from the null direction of the first nonzero row of OH.

Start with the position sensor and one reading. The red line is vertical: every speed at that position fits. Drag the slider to two readings and the line collapses to a point. Now switch to the speed sensor. No number of readings helps: the rows of OH are all [0, 1], so the position can slide along the red line forever. The symptom of such a design is a position estimate whose uncertainty never shrinks. The measurement is rank(OH) = 1 at every H. The fix is a new sensor that reads position, not more speed readings.

A robot has a sensor that reads only its speed, and its position is part of the state. After a thousand readings, what can you say about the position?

Chapter 5

The Bayes Loop: Predict, Then Correct

Keep a belief about the hidden state, widen it when the world moves, and sharpen it when a reading arrives

You are walking through a dark house and trying to keep track of where you are. You know you took two steps forward. That shifts your idea of your position, and you feel a little less sure, because steps are never exact. Then your hand touches a doorframe. Now you are more sure again, and your idea jumps toward the door.

You just did what this chapter describes. You held a belief about where you are. You moved the belief when you acted. You reshaped it when you sensed something. Two moves, repeated forever. The lecture gives them names and proves they are exact.

The world has one state, the agent has a belief

At any time the world is in one actual state st. The agent never sees it. All the agent holds is the history ht = (o0:t, a0:t−1). From that, the agent builds the posterior p(st | ht): a distribution that gives every possible state a chance. Other books call it the belief state. The lecture stresses the difference: st is not the same thing as p(st | ht). One is a fact about the world. The other is the agent’s uncertainty about that fact.

This matters because the transition law needs st and the agent does not have it. So the agent carries the whole belief into the next prediction instead.

Move one: predict

The agent takes an action at. Where is the world now? The agent cannot say for sure, so it tries every possible current state. For each one, it asks where the transition law would send it. Then it weights each answer by how likely that starting state was. The lecture writes this as an integral. An integral here is just a sum over a continuous range of states.

p(st+1 | ht, at) = ∫ p(st+1 | st, at)  p(st | ht)  dst

The first factor is the transition law, from the model. The second is the current belief, from the agent. The lecture derives this in three short moves: add the unknown state and then sum it out (marginalize it), split the joint chance with the product rule, and cut away what the assumptions make irrelevant. Two assumptions do the cutting. The transition is Markov. And the agent picks its action using only the history, so the action tells you nothing more about st.

The same recipe predicts a whole future. For a planned list of actions, the chance of the future outcomes is the world model’s prediction from each possible state, weighted by the agent’s belief: p(Yfuture | ht, afuture) = ∫ p(Yfuture | st, afuture) p(st | ht) dst. The world model supplies the first factor. The belief supplies the second. This is the exact link back to Lecture 2’s goal.

Move two: correct

A new reading ot+1 arrives. Which of the predicted states explain it well? Bayes’ rule answers. Multiply each predicted state’s chance by how likely it would be to produce this reading, then rescale so the chances add up to 1.

p(st+1 | ht+1)
The new posterior, after the reading. Here ht+1 = (ht, at, ot+1).
p(ot+1 | st+1)
The observation likelihood: how likely this reading is if the state is st+1. The lecture’s observation assumption says the current state alone sets the sensor.
p(st+1 | ht, at)
The prediction from move one.
p(ot+1 | ht, at)
The normalizer: the chance of the reading before you saw it. It is one number that makes the result add to 1.

The reading reweights the predicted states. States that explain it gain weight. States that do not lose weight.

A worked example on five cells

A robot lives in a corridor of five cells, numbered 0 to 4. Its belief is [0, 0.2, 0.6, 0.2, 0]. It tries to move one cell right, but the move works only 80% of the time and otherwise the robot stays put.

Predict. Each new cell gets 0.8 of the belief from the cell on its left plus 0.2 of its own. Cell 1: 0.8·0 + 0.2·0.2 = 0.04. Cell 2: 0.8·0.2 + 0.2·0.6 = 0.28. Cell 3: 0.8·0.6 + 0.2·0.2 = 0.52. Cell 4: 0.8·0.2 = 0.16. Cell 0 gets 0. The new belief is [0, 0.04, 0.28, 0.52, 0.16]. It adds to 1, and it is wider than before.

Correct. The robot sees a door. Cell 3 has a door, and the sensor says “door” with chance 0.9 there and 0.1 elsewhere. Multiply: [0, 0.004, 0.028, 0.468, 0.016]. These add to 0.516, the normalizer. Divide each by 0.516: [0, 0.008, 0.054, 0.907, 0.031]. The belief jumped to 90.7% on cell 3.

The loop

The last slide of this block ties it together. After observing ot you hold p(st | ht). After the action you hold the prediction. After observing ot+1 you hold p(st+1 | ht+1). That last object has the same role at time t + 1 that the first had at time t. So it becomes the input to the next prediction, and the loop repeats for every time step. The belief is a running summary of the whole history.

Run the predict and correct loop

The bars are the belief over 24 cells. Press Predict to move the belief by the action. Then place the sensor reading and press Correct. The yellow curve is the likelihood of the reading.

3 cells
1.0
cell 12
1.5

Illustrative: a discrete version of the loop on 24 cells with Gaussian move noise and Gaussian sensor noise, starting from a belief centred on cell 6 with spread 1.5. “Spread” is the standard deviation of the belief, in cells.

Press Predict a few times without correcting. The bars drift right and the spread grows, because every move adds doubt. Skip the predict step in a real system and the symptom is a belief that is too sharp: the robot is sure, and wrong. The measurement is simple: the error grows larger than the spread the belief reports. The fix is always to widen the belief when the world moves.

After a predict step, the belief gets wider. After a correct step with an informative reading, what usually happens, and why?

Chapter 6

Kalman Predict: Where the Belief Goes Next

Push the mean through the dynamics, grow the covariance, and see why a belief stretches and tilts when time passes

A ship’s navigator, long before satellites, worked out the ship’s position by dead reckoning: start from the last known position, add the speed and heading, and mark the new spot on the chart. The mark is a best guess. Every hour at sea makes the guess worse, so the navigator draws a bigger circle around the mark. That growing circle is the whole story of this chapter, written in matrices.

The man who made it exact was Rudolf Kalman. His 1960 paper, A New Approach to Linear Filtering and Prediction Problems, appeared in the Journal of Basic Engineering. The lecture gives the split of roles clearly. The LGSSM supplies the world-model probability laws. The Kalman filter is exact recursive inference under those laws.

What must be true for the filter to be exact

The filter needs the model of Chapter 2 and a few promises. The start state s0 and every noise variable are independent of each other, across sources and across time. The covariances must behave: P0 ⪰ 0 and Q ⪰ 0 mean “positive semidefinite”, a polite way to say a covariance can never describe a negative spread. R ≻ 0 means the sensor noise is strictly positive, so it is never perfectly exact. This last point will keep a certain matrix invertible in Chapter 7.

Then the filter runs in three stages. Initialize: after the first reading o0, Gaussian conditioning gives the first filtered belief N(μ0|0, P0|0). Predict: a linear transition plus independent Gaussian noise gives another Gaussian. Correct: a linear-Gaussian reading gives yet another Gaussian. Because each stage returns a Gaussian, the loop repeats for every t, and the belief always lives in the same two-object family.

Reading the notation

The lecture uses a two-index habit that is worth learning once. In μt+1|t, the first index is the time of the state we describe, and the second is the time of the latest reading we have used. So μt+1|t = E[st+1 | ht, at] is the mean of the next state before its reading arrives, and μt+1|t+1 is the mean after. The lecture shortens them to (μ−, P−) before and (μ+, P+) after. The pair (μ, P) is the entire belief. The mean alone is a point estimate, and it hides how unsure you are.

Predict the mean

What is the average next state? Take the average of the rule st+1 = A st + B at + εst. Three facts finish it. The action at is chosen from the history alone, so it tells us nothing new about st, and the average current state is μt|t. The noise has mean zero. And the action is known, so it shifts every candidate state by the same vector and adds no spread.

μt+1|t = A μt|t + B at

Said plainly: run the old mean through the same physics you would run a single state through. The lecture adds a warning. If the actuation itself were unsure, you would need an extra random term or a bigger Q.

Predict the covariance

Now the spread. Write the current error as et = st − μt|t. Then the next error is et+1|t = A et + εst: the old error, pushed through A, plus a fresh kick. The covariance is the average of the error times its transpose (the transpose flips a grid over its diagonal, so rows become columns). Expand the square. Two cross terms appear, and both are zero, because the new kick is independent of the old error.

Pt+1|t
The covariance of the next state before its reading arrives. This is P−.
A P AT
The existing uncertainty after the dynamics rotate and scale it. For the cart, A mixes speed into position, so doubt about speed becomes doubt about position.
Q
The new uncertainty the process noise adds in this step.

The result is the full prediction: p(st+1 | ht, at) = N(A μt|t + B at, A Pt|t AT + Q). The previous steps found its mean and covariance. Gaussian closure supplies the family name.

The cart, with every number

Take P0|0 = [[0.5, 0], [0, 1]] and Q = [[1.5, 0], [0, 1]]. First A P: row one is [1·0.5 + 1·0, 1·0 + 1·1] = [0.5, 1], and row two is [0, 1]. Then (A P) AT: row one is [0.5·1 + 1·1, 0.5·0 + 1·1] = [1.5, 1], and row two is [1, 1]. Add Q to get P− = [[3, 1], [1, 2]].

Read what happened. The position variance jumped from 0.5 to 3. One part came from Q (1.5) and one from the speed doubt leaking into position (1). The speed variance rose from 1 to 2. And the off-diagonal entry is now 1. It says: if the cart is further along than expected, it is probably also faster than expected. Position and speed are now correlated. That tilt is the seed of the next chapter’s trick.

Predict with no readings

The ellipse marks where the cart probably is (two standard deviations). Move the first slider to apply more predict steps. Change the second to scale the process noise Q.

1
1.0

    

The start belief and noise are the lecture’s numbers: μ = [0, 1], P = diag(0.5, 1), Q = scale × diag(1.5, 1), no action. One predict step with scale 1 gives the lecture’s P− = [[3, 1], [1, 2]].

Slide the steps to 5 and watch. The ellipse grows without limit and leans harder to the right. Nothing in the loop shrinks it. The symptom of a filter that only predicts is a position doubt that keeps growing: after 5 steps the position spread is several times the start. The measurement is the covariance itself, which grows at every step. The fix is the next chapter: bring in the readings.

The cart’s predicted covariance is P− = A P AT + Q. What does the term A P AT mean in words?

Chapter 7

Kalman Correct: The Gain Is a Trust Slider

Compare the reading to the prediction, split the gap by how much each side deserves trust, and see how a position sensor moves the speed

Your model says the cart should be at 1.5. The sensor says 2.5. One of them is wrong, or both are a little wrong. How far should you move your belief toward the sensor? A friend who trusts the sensor fully says all the way. A friend who trusts the model fully says not at all. The right answer is in between, and it depends on how much each of them has earned your trust.

The Kalman correction computes that in-between answer exactly. It uses a number called the Kalman gain, written K. Think of K as a trust slider. At 0 you keep the prediction. At 1 you jump to the reading. The filter sets the slider for you, from the two uncertainties.

Predict the reading before it arrives

Before a reading comes, the filter can already say what it expects. The predicted reading is ôt+1|t = C μ−. The matrix C turns a state vector (n numbers) into a reading vector (d numbers). For the cart n = 2, d = 1, and C picks the position. The hat marks a prediction made before seeing ot+1.

The filter can also say how far off that guess may be. The reading is o = C s + εo, so the error in the predicted reading is C times the state error plus the sensor wobble. The two are independent, so their covariances add.

St+1 = C P− CT + R,    p(ot+1 | ht, at) = N(C μ−, St+1)

S is the innovation covariance: how wide the predicted reading is. This Gaussian is exactly the normalizer from Chapter 5’s Bayes rule. When the real reading arrives, the gap between it and the prediction is the innovation (also called the residual): ν = ot+1 − C μ−. It lives in reading space. It is the surprise.

How the state and the reading move together

To fix the state from a surprise, the filter needs to know how the state error and the reading error relate. The lecture puts them in one joint Gaussian. The state block is P−, the reading block is S, and the cross block is Cov(s, o) = P− CT. The cross block is the key: it records how much a state error shows up in the reading.

Deriving the gain

The lecture’s goal: split the state error into a part that the reading determines and a leftover that is independent of the reading. Call the state error es = s − μ− and the reading error eo = o − C μ−. For any matrix L, write es = L eo + r, with the leftover r = es − L eo.

For Gaussians, two quantities with zero cross-covariance are independent. So choose L to make Cov(r, eo) = 0. That says P− CT − L S = 0, which gives L = P− CT S−1. That L is the Kalman gain.

K
The gain. The top is how strongly each state part co-moves with the reading. The bottom is how wide the reading is. Big state doubt and a quiet sensor give a big K.
μ+ = μ− + K ν
The corrected mean: the prediction plus the surprise, scaled by the gain. The surprise ν lives in reading space (d numbers). The product K ν lives in state space (n numbers).
P+ = P− − K S KT
The corrected covariance. It is smaller than P−, because the leftover r is all that remains uncertain once the reading is known.

The course’s handwritten notes reach the same gain by a second route. They write the posterior as the product of the sensor’s bell curve and the prediction’s bell curve, then find where its slope is zero and how sharply it curves. That gives P+ with an inverse that is the sum CT R−1 C + (P−)−1: precisions add, where a precision is one divided by a variance: a sharp belief has a big precision. Two routes, one answer, which is a good sign.

One number to feel it

Drop to one dimension to see the slider. Let the state be a single position, C = 1, the prediction be μ− = 10 with P− = 4, and the sensor noise be R = 1. Then S = 4 + 1 = 5 and K = 4 / 5 = 0.8. A reading of 12 gives ν = 2, so μ+ = 10 + 0.8 · 2 = 11.6. The new variance is P+ = 4 − 0.8 · 5 · 0.8 = 0.8. The belief moved most of the way to the sensor, because the prediction was the shakier of the two. Make R = 16 instead and K = 4 / 20 = 0.2, so the mean barely moves to 10.4.

Why a position sensor can fix the speed

Now the cart. The prediction has mean [1.5, 2] and P− = [[3, 1], [1, 2]]. With C = [1, 0], the cross block P− CT is the first column of P−: [3, 1]. The second entry is 1, not 0. That is the position-speed correlation from Chapter 6. So the gain has a nonzero speed entry, and a surprise in position changes the speed estimate. This is the trick behind the hero: the sensor reads only position, but the correlation carries the news to speed.

Set the trust slider and correct

The purple ellipse is the prediction. The yellow band is the reading and its noise. The warm ellipse is the corrected belief. Move the reading and the sensor noise.

2.50
1.0

    

The lecture’s numbers: μ− = [3/2, 2], P− = [[3, 1], [1, 2]], C = [1, 0]. At reading 2.5 and R = 1 the device reproduces the lecture’s correction: K = [3/4, 1/4], μ+ = [9/4, 9/4].

Slide R up to 6. The band widens, K falls, and the warm ellipse stays near the purple one: the filter ignores a sloppy sensor. Slide R down to 0.1 and the warm ellipse snaps to the band. Now the failure. Suppose you set R much too small for a sensor that is really noisy. The symptom: the estimate jitters, chasing every reading. The measurement: the average of ν2 / S over many steps lands far above 1, because the real surprises are larger than the filter expected. The fix: raise R until that average sits near 1.

The whole step in one place

The lecture ends the block with one box. Start from N(μt|t, Pt|t). Predict after action at: μ− = A μ + B a, P− = A P AT + Q. Correct after reading ot+1: ν = o − C μ−, S = C P− CT + R, K = P− CT S−1, μ+ = μ− + K ν, P+ = P− − K S KT. The new pair is the starting pair for the next step. It summarizes all the history this model needs.

The sensor noise R grows much larger while everything else stays fixed. What happens to the Kalman gain and the corrected mean?

Chapter 8

One Kalman Step by Hand

Work the lecture’s position and speed numbers from start to finish, and find out which entry lets a position sensor change the speed

Formulas are easy to nod at and hard to trust. The cure is to push real numbers through them with a pencil. This chapter does that for the lecture’s cart, one full predict and correct step, so that nothing in Chapters 5 to 7 stays a symbol.

First, the numbers. The matrices are the cart’s: A = [[1, 1], [0, 1]], B = [½, 1]T, C = [1, 0]. The noise levels are Q = [[1.5, 0], [0, 1]] and R = 1. The first push is a0 = 1.

Step zero: start the filter

The lecture starts from the prior N([0, 1], I): a mean position of 0, a mean speed of 1, and a covariance that is the identity (spread 1 in each, no link between them). Then the first reading arrives, o0 = 0, and conditioning gives the first filtered belief.

The innovation is ν = 0 − 0 = 0, so the mean does not move. The reading width is S = 1 + 1 = 2. The gain is K = [1, 0] / 2 = [½, 0]: position gets half the trust, and speed gets none, because the prior says position and speed are unrelated. The covariance becomes P0|0 = [[1 − ½·2·½, 0], [0, 1]] = [[½, 0], [0, 1]]. We start from μ0|0 = [0, 1] and P0|0 = diag(½, 1).

Before we compute, learn how to read a covariance matrix for this cart. It is a 2 by 2 grid. The top-left entry, Var(p), is how unsure we are about position. The bottom-right entry, Var(v), is how unsure we are about speed. The two off-diagonal entries are equal, and each is Cov(p, v): how the two errors rise and fall together. A positive value means “if the cart is further along than we think, it is probably faster than we think too”. A zero means the two doubts are unrelated.

Predict

The mean goes through the physics. A μ = [0 + 1, 1] = [1, 1]. The push adds B a0 = [½, 1]. So μ− = [3/2, 2].

The covariance, from Chapter 6, is A P AT + Q = [[3/2, 1], [1, 1]] + [[3/2, 0], [0, 1]] = [[3, 1], [1, 2]]. Look at the two entries on the diagonal. They are the doubts about position (3) and about speed (2). Now look at the off-diagonal entry, 1. It is the cross-covariance: how the two errors move together. It is new. Our start belief had a zero there. The prediction made it nonzero, because position depends on speed.

Correct

Suppose the sensor reports o1 = 5/2. The predicted reading is C μ− = 3/2, so the surprise is ν = 5/2 − 3/2 = 1. The reading width is S = C P− CT + R = 3 + 1 = 4.

The gain is K = P− CT / S. The product P− CT is the first column of P−, which is [3, 1]. Divide by 4: K = [3/4, 1/4]. The mean update is μ+ = [3/2, 2] + [3/4, 1/4] · 1 = [9/4, 9/4]. The covariance update is P+ = P− − K S KT = [[3, 1], [1, 2]] − 4 · [[9/16, 3/16], [3/16, 1/16]] = [[3/4, 1/4], [1/4, 7/4]].

Check three things. The position doubt fell from 3 to 3/4, a good drop from one reading. The speed doubt fell from 2 to 7/4, a small drop. And the speed estimate moved from 2 to 9/4, although the sensor reads only position. The lecture’s sentence says it all: cross-covariance carries information, so a position-only sensor can update the speed.

Step through the lecture’s numbers

Move through the three stages. Choose the reading o1. Then switch the cross-covariance off to see what the speed estimate loses.


    

The matrices, noise levels and the push are the lecture’s. The lecture works only o1 = 5/2; the other two readings are our own, to show the same arithmetic moving. “Drop the cross-covariance” zeroes the off-diagonal of P− before the correction, as a thought experiment.

At o1 = 2.5, stage Correct, the box shows exactly the numbers above. Now press the red toggle. The off-diagonal of P− is forced to 0, the gain on speed becomes 0, and the speed estimate stays at 2 whatever the sensor says. That is a real failure, not just a thought experiment: some code approximates the covariance by its diagonal to save work. The symptom is a speed estimate that never reacts to position readings. The measurement: the second entry of the gain K is exactly 0 at every step. The fix: keep the full covariance matrix.

In the lecture’s example the sensor reads only position, yet the speed estimate moves from 2 to 9/4. Which number makes that possible?

Chapter 9

Learning the Model From Readings

Choose the model numbers that make a recorded run most probable, and see the Kalman filter hand you that probability for free

So far we knew the model. Someone gave us A, B, C, Q and R. In real life nobody does. You have a recording of sensor readings and the actions that were taken, and you must work out how noisy the sensor is, how hard the world kicks the state, and what the dynamics are. This is learning, and it is a different job from the filtering you just did.

The lecture draws the line cleanly. In inference the parameters θ = (μ0, P0, A, B, C, Q, R) are fixed and we estimate hidden states. In learning θ is unknown, the states stay hidden, and the recorded readings decide the fit.

The score: how probable is the recording?

Maximum likelihood says: pick the θ under which the data you saw was most probable. The data is only the readings o0:T, given the known actions. The states are never recorded. So the score must add up over every hidden path that could have produced the readings. This is the marginal likelihood.

ℓ(θ) = log pθ(o0:T | a0:T−1) = log ∫ pθ(s0:T, o0:T | a0:T−1) ds0:T,   θ★ = argmaxθ ℓ(θ)

That integral runs over a whole trajectory of hidden states, which sounds hopeless. The Kalman filter makes it easy. Split the chance of the whole recording into one chance per reading, each given everything before it: p(o0:T) = p(o0) ∏ p(ot | ht−1, at−1). And each of those chances is the Gaussian from Chapter 7: N(ot; C μt|t−1, St). The filter already computes the centre and the width at every step.

The cost of one reading

Take the negative log of one factor, so that “more probable” becomes “lower cost”. The lecture writes it like this.

Lt(θ)
The cost of reading t: minus the log chance the model gave it.
d log 2π
A constant that depends only on the size d of one reading.
log |St|
The predictive width. It is large when the model spreads its bets over a wide range of readings.
νT S−1 ν
The normalized prediction error: the surprise, measured in units of the predicted spread. It is large when the reading lands far from the prediction, relative to that spread.

The two middle terms pull in opposite directions, and that is the beauty of the score. Say the surprise was ν = 2. With a predicted width S = 5, the cost is ½[1.84 + ln 5 + 4/5] = ½[1.84 + 1.61 + 0.80] = 2.12. Inflate the width to S = 20 and the error term shrinks to 0.20, but ln 20 = 3.00 grows more, and the cost rises to 2.52. Shrink it to S = 1 and the error term balloons to 4.0: cost 2.92. The best width is S = ν2 = 4, with cost 2.11. A model cannot cheat by declaring itself uncertain, and it cannot cheat by being overconfident.

Add up the costs over the whole recording. The first reading uses the prior, S0 = C P0 CT + R. The later ones use the filter’s own νt and St. That sum is L(θ) = −ℓ(θ).

Two ways to lower the cost

Direct gradients. L(θ) is a smooth function of the model numbers. Run the filter forward, add up the costs, and let automatic differentiation (software that works out the slope of the cost by itself) carry derivatives back through every predict and correct step. Then take a step downhill: θ ← θ − η ∇θ L(θ), where η is the learning rate, the size of each downhill step. A change to θ changes every later prediction, so the gradient flows through the whole loop. The covariances must stay legal: P0 and Q positive semidefinite, and R positive definite so that S is always invertible.

Expectation-maximization (EM). This is the optional second route. It improves the same objective by alternating two steps. The E-step fixes θ and uses the complete recording to compute a posterior over the hidden trajectory. That offline pass, which uses readings from the future as well, is called smoothing. The M-step then refits θ using the smoothed states and their products, and for linear-Gaussian models this refit often has a closed-form answer. EM improves a lower bound on the likelihood that touches the true likelihood at the current θ, so an exact E-step and M-step can never make the likelihood worse. The lecture is honest about the trade: convenient updates for classical models, but no promise of faster convergence or of the same local answer as gradients.

Find the noise level the data prefers

A recorded run of 60 readings is fixed. The curve is the total cost L(θ) from the Kalman filter, for every assumed sensor noise. Drag the assumed noise and look for the lowest point.

0.40
0.10

Illustrative: one simulated cart run with the hero’s model, true sensor noise 1.0 and true process noise 0.1, and a vague prior (position spread 5, speed spread 2). The cost is the sum of Lt over all 60 readings, computed by the Kalman recursion in your browser. Noise levels are standard deviations.

Put the assumed noise at 0.4 and the cost is high: the model thinks the sensor is exact, so honest wobble looks like a shock. Drag it up and the cost falls, bottoms out near the true value of 1, then rises again as the model grows needlessly vague. The data picks the answer. A real fit does the same dance in many dimensions at once.

A failure to watch. A flexible model can fit one recording almost perfectly by shrinking R toward zero and explaining every wobble as real motion. The symptom is a model that looks great on its training run and poorly on a new one. The measurement is the cost on a held-out run, which jumps. The fix: constrain R to stay positive, and always check on data the fit never saw.

The lecture’s summary

The block ends with four roles. The world model is the initial, transition and observation laws, set by θ. Filtering gives the online posterior p(st | o0:t, a0:t−1; θ). Prediction gives future readings p(ot+1 | ht, at; θ). The objective is the marginal log likelihood, and the optimizer is direct gradients or EM.

In the cost of one reading, why can a model not lower its cost just by claiming a very large predictive covariance S?

Chapter 10

Other State Spaces, and Where This Model Breaks

Swap the continuous state for a handful of modes, then meet the five places where the linear-Gaussian recipe stops being right

A fitness band cannot see whether you are sitting, walking or running. It sees a stream of small shakes. You are always in one of three modes, and the shakes hint at which. There is no smooth position or speed here. There are just three boxes, and you are in exactly one.

The lecture calls this a hidden Markov model (HMM): a state-space model whose state is discrete. The state st takes one of L values. The three laws are the same three cards as before, now written as tables: an initial table p(s0), a transition table p(st+1 | st), and an observation table p(ot | st). For simplicity the lecture leaves out actions here.

Filtering with finite sums

The Bayes loop of Chapter 5 is unchanged. The integrals just become sums over L boxes, and the belief is a list of L probabilities instead of a mean and a covariance.

predict: p(st+1 = j | o0:t) = ∑i=1L p(st+1 = j | st = i) p(st = i | o0:t)    correct: p(st+1 = j | o0:t+1) ∝ p(ot+1 | st+1 = j) p(st+1 = j | o0:t)

A worked step with illustrative numbers. Let the modes be Parked, Walking and Running, with belief [0.5, 0.3, 0.2]. A sticky transition table says Parked stays Parked 80% of the time, Walking stays 70%, Running stays 70%. Predict. Parked: 0.8·0.5 + 0.1·0.3 + 0.05·0.2 = 0.44. Walking: 0.15·0.5 + 0.7·0.3 + 0.25·0.2 = 0.335. Running: 0.05·0.5 + 0.2·0.3 + 0.7·0.2 = 0.225. The three add to 1.

The band now reports a high shake. The observation table gives that reading 0.02 chance if Parked, 0.15 if Walking and 0.8 if Running. Multiply: 0.0088, 0.0503 and 0.18. They sum to 0.239. Divide: the new belief is [0.037, 0.210, 0.753]. One shake moved the band from 20% Running to 75%. Try it yourself.

Filter three hidden modes

Each press feeds the band one reading. The filter predicts with the sticky transition table, then corrects with the observation table. Dashed outlines show the prediction.

Illustrative tables. Transitions (rows are the current mode): Parked [0.8, 0.15, 0.05], Walking [0.1, 0.7, 0.2], Running [0.05, 0.25, 0.7]. Observations (low, medium, high): Parked [0.85, 0.13, 0.02], Walking [0.15, 0.7, 0.15], Running [0.02, 0.18, 0.8]. Start belief [0.5, 0.3, 0.2].

Learning works the same way as before: maximize the marginal likelihood, by direct gradients or by EM. For an HMM the E-step is the forward-backward pass, which smooths over the discrete states.

Five places the recipe bends

The lecture closes with a numbered list. Each item is a promise of the linear-Gaussian model, and what to do when the promise fails.

  1. The state must make the dynamics Markov. If p(st+1 | s0:t, a0:t) differs from p(st+1 | st, at), the chosen coordinates are not a sufficient state. Position alone is the example, since it omits the speed. Repair: add the missing piece to the state, or learn a representation that keeps it.
  2. Dynamics may be nonlinear. Contact and saturation change how the next state responds in different regions. Use st+1 = fθ(st, at) + εst. The prediction may no longer be Gaussian.
  3. Image observations need a nonlinear sensor. Pixels depend on geometry, perspective, lighting, visibility and occlusion in ways no single matrix C can describe. Use ot ~ pθ(ot | st). The standard Kalman correction is then generally not exact.
  4. One Gaussian merges separate futures. If the cart could go left or right around an obstacle, the Kalman prediction still reports one bell curve, and its centre can sit on the obstacle. Use a richer family that holds several modes.
  5. Parameters can change with time. A time-varying model, st+1 = At st + Bt at + ε, ot = Ct st + ε, is still linear-Gaussian, so exact Kalman filtering still works. Unknown drift or context must instead go in the state, be modeled adaptively, or be learned from more data.

Item four deserves a picture. Slide the gap below and watch where the single Gaussian puts its weight.

Split the future around an obstacle

The cart will pass the obstacle on the left or on the right. Blue is the true spread of futures. Warm is the one Gaussian with the same mean and variance. Drag the gap.

3.00

Illustrative: two equal routes, each a bell curve of spread 0.5, centred half the gap either side of the obstacle. The obstacle fills positions from −0.5 to 0.5. The warm curve matches the mean (0) and variance of the two-route mixture, as a Kalman prediction would.

At a gap of 3, the true chance of ending inside the obstacle is about 2%. The Gaussian says about 25%. If a planner trusts that Gaussian, it will fear a collision that cannot happen on average, and it may also aim straight at the obstacle’s centre, the one place nobody goes. The symptom: a plan that steers through a wall. The measurement: compare the filter’s mass in a region with the fraction of real runs that end there. The fix: a model with several modes.

Two families of memory, and what comes next

The lecture zooms out. A sequence model can carry the useful past in two ways. One is an explicit generative state with named transition and observation laws: a simulator, an LGSSM, an HMM. The other is a recurrent predictive memory: history compressed inside a predictor, as in an RNN’s hidden memory or a video model’s latent. That memory may live on the agent’s side, not as a world-side variable. The lecture’s summary rule is a good one to keep: choose a valid state and probability laws first, then choose inference and learning methods that fit those assumptions.

Next week, Lecture 5, asks how to learn useful codes zt = Eφ(ot; o<t, a<t) from raw observations with no state labels. Three questions drive it: what should stay unchanged (invariance), what should change in a predictable way (equivariance), and what keeps the code from collapsing to nothing. A contextualized code need not be Markov, so a predictor may still need the whole sequence of codes.

A robot can drive left or right around a wall, and a Kalman filter predicts one Gaussian over its future position. What goes wrong?

Chapter 11

Field Guide: The Whole Lecture on One Page

Keep the numbers, the terms, a checklist and the core code, and see where each one leads next

A state-space model writes a world as three laws: how a run starts, how the hidden state moves under actions, and how a sensor shows the state. The linear-Gaussian version makes all three exact and tiny: a mean and a covariance carry the whole belief. The Kalman filter keeps that belief by a loop of predict and correct, and a position-only sensor can fix the speed through the cross-covariance. Observability tells you if the readings could ever reveal the state. The same filter hands you the likelihood, and the likelihood trains the model by gradients or EM.

The numbers and terms

TermWhat it isThe cart’s value
State stEverything the next step needs; hidden[p, v], n = 2
Observation otThe noisy sensor readingp + noise, d = 1
A, B, CTransition, input and observation matrices[[1, 1], [0, 1]], [½, 1]T, [1, 0]
Q, RProcess noise and sensor noise covariancesdiag(3/2, 1), 1
Observability OHStack of C Ak; full column rank means no ambiguityrank 2 from H = 2
Prediction (μ−, P−)A μ + B a, and A P AT + Q[3/2, 2], [[3, 1], [1, 2]]
Innovation ν and SSurprise, and its predicted spread1 and 4
Gain KP− CT S−1, the trust slider[3/4, 1/4]
Correction (μ+, P+)μ− + K ν, and P− − K S KT[9/4, 9/4], [[3/4, 1/4], [1/4, 7/4]]
Cost L(θ)Sum of ½[d log 2π + log |S| + νT S−1 ν]minimized by gradients or EM

All the cart values come from the lecture’s worked example (slides 38 and 39).

A checklist for the lecture

The core as code

This is the whole filter step plus the cost of one reading, in NumPy. It reproduces the lecture’s example: starting from μ = [0, 1], P = diag(0.5, 1), a = 1 and o = 2.5 it returns μ+ = [2.25, 2.25] and P+ = [[0.75, 0.25], [0.25, 1.75]].

pythonimport numpy as np

A = np.array([[1., 1.], [0., 1.]])   # (n, n) = (2, 2)
B = np.array([[.5], [1.]])               # (n, m) = (2, 1)
C = np.array([[1., 0.]])                 # (d, n) = (1, 2)
Q = np.diag([1.5, 1.0]); R = np.array([[1.0]])

def kalman_step(mu, P, a, o):
    """mu (n,1), P (n,n), action a (m,1), reading o (d,1)."""
    # predict
    mu_m = A @ mu + B @ a                    # (n,1)
    P_m  = A @ P @ A.T + Q                   # (n,n)
    # correct
    nu = o - C @ mu_m                        # (d,1) innovation
    S  = C @ P_m @ C.T + R                   # (d,d)
    K  = P_m @ C.T @ np.linalg.inv(S)        # (n,d) gain
    mu_p = mu_m + K @ nu
    P_p  = P_m - K @ S @ K.T
    # one-reading cost, summed over t to get L(theta)
    nll = 0.5 * (len(o) * np.log(2 * np.pi) + np.log(np.linalg.det(S))
                 + (nu.T @ np.linalg.inv(S) @ nu).item())
    return mu_p, P_p, nll

To learn A, Q or R, wrap this step in a loop over the recording, add up nll, and let an automatic-differentiation library take gradients. The course’s reading, the dynamax Kalman filter tracking notebook, tracks an object in two dimensions with a four-number state (two positions, two speeds) and a position-only sensor, and runs it with a real library, including offline smoothing.

Where it sits, and where to go

State-space models are the oldest answer to “what does the agent believe about a world it cannot see?”, and they sit under many modern world models. A learned latent state in a video model plays the role of st. A belief state in a decision problem is the posterior of Chapter 5. Keep going with these lessons.

The takeaway. A world model with a hidden state is a belief that you keep honest. Predict widens it when the world moves. Correct narrows it when a reading arrives. The Kalman filter does both exactly, as long as the world is linear and Gaussian, and the cross-covariance lets a sensor that sees one number repair another.

Now explain, from memory, how a position-only sensor can change a speed estimate, and why the belief must widen between readings. Then go back to the hero, raise the sensor noise, and watch the filter handle it.

The lecture says to choose a valid state and probability laws first, and the algorithms after. Why in that order?
CIS 6280 Lecture 4: State-Space Models
The course site · Lecture slides · Handwritten notes · dynamax tracking notebook · Back to Gleams