Principles of Robot Autonomy I · Lesson 11 of 15 · Stanford AA274A

SLAM & Factor Graphs

To build a map you must know where you are. To know where you are you need a map. SLAM breaks the chicken-and-egg by solving both at once — first with a filter that grows uncertain, then with a graph that snaps the whole world straight.

Prerequisites: Lesson 9 (features) + a 2×2 matrix + a little Python. That's it.
12
Chapters
10
Simulations
0
Assumed Knowledge

Chapter 0: The Chicken and the Egg

You wake a robot in a building it has never seen. No GPS — you are indoors. No map — nobody gave it one. It has wheels that count their own turns and a laser that reports how far the nearest walls are. You ask it one thing: drive around, and come back knowing both where you went and what the building looks like.

Pause on how circular that is. To draw the map, the robot must place each wall at the right spot — but "the right spot" is measured relative to the robot, so it needs to know where the robot was when it saw that wall. To know where the robot is, it compares what it sees now against the map — but the map is the thing it is still building. Position needs the map; the map needs position. Neither comes first.

The one-sentence problem. Simultaneous Localization And Mapping (SLAM) is estimating the robot's trajectory and a map of the environment at the same time, from nothing but its own motion estimates and its measurements of the world. The two unknowns are coupled: an error in pose corrupts the map, and an error in the map corrupts the pose.

The naive escape is "just trust the wheels." Count wheel turns, integrate them into a position, and call that the robot's pose. This is dead reckoning — reckoning your position purely from your own motion, the way a ship's navigator did before satellites. It is seductive because it needs no map at all.

It also fails, and fails in a specific, ruinous way. Every wheel-tick estimate has a tiny error — a wheel slips a millimeter, the floor is not perfectly flat, the turn was 30.4° not 30°. Each error is small. But you integrate them: today's position is yesterday's position plus today's motion, forever. Small errors do not average out; they accumulate. The estimated path slowly bends away from the truth, and because every wall the robot draws is hung off that drifting path, the whole map smears with it.

Watch it happen. The robot below drives a loop you can see (the faint true path). Its own belief — built only from noisy odometry — is the bright path, and the walls it "maps" are pinned to that belief. Crank the slip up and run it:

Dead Reckoning Drifts — and the Map Smears With It

The robot drives a square loop. True path is faint; the dead-reckoned belief is bright purple; the mapped landmarks (orange) are placed relative to the drifting belief. Turn up odometry slip and press Run — the belief curls away from truth and the map curls with it. Notice the gap at the end: the robot thinks it is somewhere it is not.

odometry slip 2.0% ready

The bright path should end exactly where it started — it drove a closed loop. It does not. That end-of-loop gap is accumulated drift, and it is the central villain of this lesson. Dead reckoning has no way to notice it, because it never compares against anything outside itself.

So what breaks the cycle? Two ideas, and this lesson is the story of moving from the first to the second:

Where SLAM sits in the stack. Back in Lesson 1 we drew the four-module autonomy stack. SLAM is the Localization & SLAM module made whole: it consumes the features that Perception extracts (Lesson 9's corners and landmarks) plus the robot's own motion, and it produces the two things every other module depends on — the robot's pose and the map. Planning needs the map. Control needs the pose. Get SLAM wrong and everything downstream is built on sand.
Why does pure dead reckoning (integrating wheel odometry) eventually produce a badly wrong map?

Chapter 1: Stating the Problem Precisely

Chapter 0 gave us the intuition; now we say exactly what SLAM computes, because every algorithm in this lesson is a different way of estimating the same quantity. We will name every symbol as it appears.

What are the unknowns?

There are two of them, and SLAM solves for both jointly.

What are the knowns?

Two streams of data, both noisy.

The whole problem in one line. Given the controls u1:t and measurements z1:t, find the trajectory x1:t and the map m. Because everything is noisy, we don't find a single answer — we find a probability distribution over answers, the posterior, and report its peak.

Two flavors: online vs. full SLAM

There is a fork in the road, and which branch you take decides which algorithm you get.

Online (filtering) SLAM. Estimate only the posterior over the current pose and the map:

p(xt, m | z1:t, u1:t)

Past poses are summarized into the current belief and then thrown away — "marginalized out." Memory stays bounded; you can run it forever, in real time. The classic example is EKF-SLAM (Chapter 3).

Full (smoothing) SLAM. Estimate the posterior over the entire trajectory and the map:

p(x1:t, m | z1:t, u1:t)

Keep every pose. The estimate grows with time, but you can revisit and correct any past pose later — which is exactly what makes loop closure work. This is the factor-graph view (Chapters 5–10).

The single character that differs — xt versus x1:t — is the entire plot of this lesson. Filtering forgets the past to stay cheap; smoothing remembers the past to stay correct. Let's see the difference as a graphical model, where shaded circles are things we measured and open circles are what we want to find:

Online vs. Full SLAM — the graphical model

Toggle between the two. Open circles = unknowns (poses x, landmarks m). Shaded boxes = data (controls u, measurements z). In online SLAM only the current pose is kept (past poses fade). In full SLAM every pose stays in the estimate, so any of them can be corrected later.

Online: only xₜ survives; past poses are marginalized away.

Read the arrows as causality. Each pose flows to the next through a control u (the motion model). Each measurement z hangs off a (pose, landmark) pair (the measurement model). The landmarks sit to the side because they are static — every pose can see them. Those two model types are Chapter 2.

Why "keep the whole path" is not crazy. It sounds wasteful to store every pose forever. The saving grace, which Chapter 7 makes precise, is sparsity: each measurement connects only a handful of variables, so the giant problem is mostly empty and can be solved astonishingly fast. Filtering trades that sparsity away to save memory — and pays for it with correlations that never go away (Chapter 3).
What exactly distinguishes the full (smoothing) SLAM problem from the online (filtering) one?

Chapter 2: The Two Models — Motion and Measurement

Everything in SLAM is built from two probabilistic statements: how the robot moves, and what the robot sees. Get these two right and every algorithm — filter or graph — falls out of them. Let's build each from a concrete picture, then write the math, then explain every symbol.

The motion model: where am I after a control?

The robot is at pose xt−1. You command a motion ut (from odometry: "forward 0.5 m, turn 10°"). Where do you end up? In a perfect world, a deterministic function tells you:

xt = g(ut, xt−1)

Here g is the motion function — for a wheeled robot it is the unicycle/differential-drive update from Lesson 2 (rotate by the commanded turn, step forward by the commanded distance). But the world is not perfect: wheels slip, floors tilt. So we add process noise:

xt = g(ut, xt−1) + εt,     εt ~ N(0, Rt)

Reading this symbol by symbol: εt (epsilon) is a random nudge added each step. The notation ~ N(0, Rt) means "drawn from a Gaussian (bell curve) centered at zero with covariance Rt." A covariance matrix Rt is just the multi-dimensional version of "variance" — it says how big the nudge typically is in each direction, and whether errors in different directions are linked. Bigger Rt = noisier wheels = trust odometry less.

Read the noise as a confidence dial. A motion model is not "the robot goes here." It is "the robot probably goes near here, give or take this much." The give-or-take is Rt. This probabilistic reading is the whole reason SLAM can fuse motion with measurement: both are just blurry Gaussians, and combining two blurry beliefs into a sharper one is what every estimator does.

The measurement model: what do I see from here?

Now the robot, at pose xt = (x, y, θ), looks at landmark mj = (mj,x, mj,y) and its sensor reports a range r (how far) and a bearing φ (at what angle, relative to where the robot is facing). The geometry is exact — it is just the distance formula and an arctangent:

r = √( (mj,x − x)² + (mj,y − y)² )
φ = atan2(mj,y − y,  mj,x − x) − θ

Symbol by symbol: the range is the straight-line distance from robot to landmark (Pythagoras). The bearing uses atan2 — the "smart arctangent" that returns the correct angle in all four quadrants — to get the absolute direction to the landmark, then subtracts the robot's heading θ so the bearing is measured in the robot's own frame ("20° to my left"). Stacking range and bearing into a vector and adding sensor noise δt (delta):

zti = h(xt, mj) + δt,     δt ~ N(0, Qt)

where h is the measurement function (the range-bearing formula above), zti is the i-th measurement taken at time t, and Qt is the measurement-noise covariance (small Q = trust the sensor a lot).

Worked example: predict a measurement by hand

Numbers make it real. Put the robot at xt = (2, 1, 30°) and a landmark at mj = (5, 5). What should the sensor read?

  1. Differences: Δx = 5 − 2 = 3,  Δy = 5 − 1 = 4.
  2. Range: r = √(3² + 4²) = √25 = 5.0 m. (The friendly 3-4-5 triangle.)
  3. Absolute direction to landmark: atan2(4, 3) ≈ 53.1°.
  4. Bearing in robot frame: 53.1° − 30° = 23.1° — the landmark is about 23° to the robot's left.

So the predicted measurement is h = (5.0 m, 23.1°). If the actual sensor returns (5.2 m, 21°), the gap — called the innovation or residual — is (−0.2 m, +2.1°). That residual is gold: it is the disagreement between belief and reality, and every SLAM algorithm works by squeezing residuals toward zero.

Drag the robot and the landmark below and watch the predicted range and bearing update live:

Range & Bearing — drag to feel the measurement model

Drag the robot or the landmark. The wedge shows the robot's heading; the readout shows range r and bearing φ (in the robot's own frame). Rotate the robot with the slider — range stays fixed but bearing shifts, because bearing is relative to heading.

heading θ 30° r = 5.0 m, φ = 23.1°
Common misconception. "Bearing is the compass direction to the landmark." No — bearing is relative to the robot's own heading, which is why the formula subtracts θ. Two robots at the same spot facing different ways report different bearings to the same landmark. Forgetting the −θ is one of the most common SLAM bugs: your landmarks spin around the robot as it turns.
A robot at (0,0) facing 90° (straight up) sees a landmark directly to its right, at world position (3,0). What bearing φ does it report?

Chapter 3: EKF-SLAM — One Big Gaussian

The earliest SLAM algorithm took the Extended Kalman Filter you'll meet in Lesson 13 and asked: what if the "state" the filter tracks is not just the robot's pose, but the robot's pose and every landmark, stacked together? That one idea is EKF-SLAM, and it is the cleanest place to see the chicken-and-egg coupling become a number.

The augmented state

We glue the robot pose and all N landmarks into one tall vector y — the augmented state:

y = [ x, y, θ,   m1,x, m1,y,   …,   mN,x, mN,y ]

Count the entries: 3 for the pose plus 2 per landmark, so 3 + 2N numbers in all. The filter tracks a Gaussian belief over this whole vector — a mean μ (our best single guess for every coordinate) and a covariance Σ, a (3+2N)×(3+2N) matrix of all the uncertainties and — crucially — the correlations between them.

The off-diagonal blocks are the whole point. Σ has a 3×3 pose block on its diagonal, a 2×2 block per landmark, and — off the diagonal — cross-covariance blocks linking the pose to each landmark and landmarks to each other. A non-zero pose-landmark block literally encodes "if my pose is off by this much, this landmark is off by that much, in this correlated way." That is the chicken-and-egg, written as a matrix. EKF-SLAM's superpower — and its curse — is that it maintains every one of these correlations.

The two steps

Like any Kalman filter, EKF-SLAM alternates predict (the robot moved) and update (the robot saw something).

Predict. Apply the motion model to the pose part of μ (the landmarks are static, so their means don't move). Then grow the covariance — moving adds process noise Rt, so the robot's uncertainty increases:

μ̄ = g(ut, μ),     Σ̄ = Gt Σ Gt + Rt

The bar means "predicted, before the measurement." Gt is the motion Jacobian — the matrix of partial derivatives of g, which linearizes the (nonlinear) motion so the Gaussian stays a Gaussian. Because only the pose moved, Rt has nonzero entries only in its top-left 3×3 block.

Update. The robot observes landmark j and gets zt. Compute the predicted measurement h(μ̄, j), take the residual, and fold it in through the Kalman gain K:

S = Ht Σ̄ Ht + Qt
K = Σ̄ Ht S−1
μ = μ̄ + K ( zt − h(μ̄, j) )
Σ = ( I − K Ht ) Σ̄

where Ht is the measurement Jacobian (how the measurement changes as the state changes), S is the innovation covariance, and K is the trust slider between prediction and measurement. The magic is in K = Σ̄ HtS−1: because Σ̄ carries pose-landmark correlations, a single measurement of one landmark updates the pose and every correlated landmark at once. See one wall clearly, and your belief about the whole map sharpens.

Worked update: 2 poses, 1 landmark, by hand

Let's watch a correlation actually grow. We track a 1-D toy to keep the arithmetic on one line: the state is (p, ℓ) — robot position p and one landmark position , on a line.

Start. The robot is certain it is at the origin, the map is unknown:

μ = (0, 0),    Σ = [ [0.01, 0], [0, 100] ]

Read that covariance: pose variance 0.01 (tiny — we're sure where we are), landmark variance 100 (huge — no idea), and the off-diagonals are 0 — pose and landmark are, for now, uncorrelated.

Observe the landmark. The sensor says "the landmark is at distance 10," i.e. z = ℓ − p = 10, with measurement variance Q = 1. The measurement function is h = ℓ − p, so the Jacobian is H = [−1,  1] (h drops by 1 if p rises by 1; rises by 1 if ℓ rises by 1).

  1. Predicted measurement: h(μ) = 0 − 0 = 0. Residual: z − h = 10.
  2. Innovation covariance: S = HΣH + Q = 0.01 + 100 + 1 = 101.01.
  3. Gain: K = ΣHS−1. Now ΣH = [[0.01,0],[0,100]]·[−1,1] = [−0.01,  100], so K = [−0.0001,  0.990].
  4. Update the mean: μ ← μ + K·10 = (0 − 0.001,  0 + 9.90) = (−0.001,  9.90). The landmark jumped to ≈9.9 (near the measured 10); the confident pose barely moved.
  5. Update the covariance: Σ ← (I − KH)Σ. Working it out gives roughly
Σ ≈ [ [0.0099,  0.0099], [0.0099,  0.990] ]

Stare at the off-diagonal: it was 0, now it is +0.0099. The act of observing the landmark from this pose correlated them. The landmark's variance also collapsed from 100 to 0.99 — one good measurement turned "no idea" into "pretty sure." And here is the punchline of EKF-SLAM: from now on, if the robot's pose estimate ever shifts, that off-diagonal will drag the landmark along with it. The map and the trajectory are now permanently entangled in one covariance matrix.

Run the full 2-D version below. Drive the robot; covariance ellipses show the uncertainty of the pose and of each landmark. Watch the pose ellipse grow as it drives blind, then shrink the instant it re-observes a known landmark — and watch every landmark ellipse twitch in sympathy, because they are all correlated.

EKF-SLAM — covariance ellipses live

Press Drive. The robot follows a path, predicting (ellipse grows) and updating whenever a landmark is in sensor range (ellipse shrinks). Purple = pose uncertainty; orange = each landmark's uncertainty; faint orange dots are the true landmarks. Turn sensor range down to starve the filter of updates and watch uncertainty balloon.

sensor range 3.5 ready

The cost: O(N²) and why

That dense, fully-correlated covariance is exactly what makes EKF-SLAM scale badly. With N landmarks, Σ is (3+2N)×(3+2N) — it has on the order of entries, and the update touches all of them. Double the landmarks, quadruple the work and memory. A map of a building has thousands of landmarks; kills you.

The EKF-SLAM dilemma (and misconception). People assume "more landmarks = better localization, so just add more." Accurate localization does want a dense map — but EKF's O(N²) update punishes density. You're squeezed between wanting many landmarks and being able to afford few. And there's a deeper flaw: the filter linearizes once, at the current estimate, and can never revisit it. If an early pose was wrong, every measurement linearized around it is biased forever — the filter cannot go back. Both problems push us toward keeping the whole trajectory and re-optimizing — the next chapter's turn.
In EKF-SLAM, why does observing a single landmark also improve the estimate of the robot's pose (and other landmarks)?

Chapter 4: From Filtering to Optimization

EKF-SLAM works, runs online, and is beautifully simple. Yet modern SLAM systems — the ones in your phone's AR, in self-driving stacks — almost universally abandoned it. To understand why, we have to look hard at the one move that defines a filter: it throws the past away.

Marginalization: the price of forgetting

Recall from Chapter 1 that online SLAM estimates only p(xt, m | …) — the current pose. Where did poses x1, …, xt−1 go? They were marginalized out: their information was folded into the current belief and the variables themselves deleted. This is what keeps memory bounded.

But marginalization is irreversible, and it has two nasty consequences:

Filtering = compression with loss. A filter is a lossy compressor of history: it summarizes all past data into one current Gaussian. That's perfect when you only care about now and never need to revisit. It's a disaster for SLAM, where the single most valuable event — loop closure — is precisely the discovery that a past pose needs correcting.

The loop-closure argument, made concrete

Picture the drifted loop from Chapter 0. The robot has gone around and, thanks to drift, its belief about the final corner is half a meter from where it started. Then it recognizes the starting corner: "this is the same place as pose 1." That is a constraint linking pose t directly back to pose 1.

What should happen? The error shouldn't dump entirely onto the last pose — it should spread backward across the whole loop, nudging every intermediate pose a little, until the trajectory is globally consistent. But spreading error backward requires the past poses to still be there to nudge. A filter marginalized them away. A method that kept them can.

Why Smoothing Beats Filtering on a Loop

A drifted loop with a loop-closure constraint (dashed red) tying the end back to the start. Filter: the past is gone, so only the last pose can move — the closure yanks it but the loop stays kinked. Smoothing: every pose survives, so the correction distributes around the whole loop and it closes smoothly. Toggle to compare.

drifted, not yet closed

The shift this motivates is profound but simple to state: stop filtering, start optimizing. Keep the entire trajectory and map as free variables. Treat every odometry reading and every measurement as a constraint that the variables should satisfy. Then find the configuration of all variables that satisfies all constraints as well as possible — a single, global optimization problem.

The reframe in one breath. Filtering asks "given everything so far, what is my belief now?" and answers recursively, forgetting as it goes. Smoothing asks "what trajectory and map best explain all the data I've ever collected?" and answers by optimizing over everything at once. The data structure that makes the second question tractable is the factor graph — Chapter 5.

You might worry: optimizing over thousands of poses and landmarks sounds far more expensive than a filter. The opposite is true in practice, and the reason is the sparsity we flagged in Chapter 1: each constraint touches only a couple of variables, so the optimization problem is mostly empty and solvable in near-linear time. Filtering destroys that sparsity (every variable becomes correlated with every other); smoothing preserves it. Chapter 7 makes this rigorous.

What is the core reason smoothing (keeping all poses) handles loop closure better than filtering (marginalizing past poses)?

Chapter 5: Factor Graphs — Drawing the Problem

We decided to keep every variable and treat every reading as a constraint. A factor graph is the picture of exactly that — and it is not just a diagram, it is the literal data structure modern SLAM solvers store and operate on.

Two kinds of node

A factor graph is bipartite: it has two distinct kinds of node, and edges only ever connect one kind to the other.

An edge connects a factor to every variable that factor depends on — and to no others. That "and to no others" is the sparsity, made visual.

The four factor types in SLAM

Prior factor. Touches one pose (usually x1). It pins the map to a reference frame — "the first pose is the origin." Without it the whole graph could slide or spin freely (everything is relative); the prior nails it down.

Odometry factor. Touches two consecutive poses (xt−1, xt). It says "the motion between these two poses should match what odometry reported." This is the motion model as a constraint.

Observation factor. Touches one pose and one landmark (xt, mj). It says "the range/bearing from this pose to this landmark should match what the sensor reported." The measurement model as a constraint.

Loop-closure factor. Touches two non-consecutive poses (xi, xj) that turn out to be the same place. Structurally it's just another between-poses constraint — but it reaches across the graph and is the single most powerful edge there is (Chapter 8).

A factor is just a penalty. Each factor evaluates an error — the gap between what its variables predict and what was measured (the residual from Chapter 2) — and assigns a cost that grows as the error grows. A factor graph is therefore a giant sum of penalties. Solving SLAM means choosing the variables that make the total penalty as small as possible. That's it.

From graph to objective

The factor graph encodes a probability: the posterior factors into a product of per-factor likelihoods. Finding the most-probable configuration — the Maximum A Posteriori (MAP) estimate — means maximizing that product. Taking a negative logarithm (which turns products into sums and flips max into min) and using Gaussian noise (whose negative-log is a squared term) gives the form every SLAM solver actually minimizes:

x* = arg minx Σi   ei(x) Ωi ei(x)

Decoding it: the sum runs over every factor i. ei(x) is that factor's error vector (measured minus predicted). Ωi (capital omega) is the factor's information matrix — the inverse of its noise covariance, so a precise sensor (small covariance) gives a large Ω and its errors are penalized heavily. The expression eΩe is a weighted squared error (a "squared Mahalanobis distance"). So MAP SLAM is a weighted nonlinear least-squares problem: make all the residuals small, weighting each by how much you trust it.

This is the unification. Every SLAM ingredient is now one kind of term in one sum. Odometry, landmark sightings, the origin prior, loop closures — all just factors contributing eΩe. There is no special case. Add a GPS reading? It's a factor on one pose. Add an IMU? A factor between poses. The factor graph is the universal language of modern estimation, and it's why the same solver powers SLAM, visual-inertial odometry, and bundle adjustment (Chapter 9).

Build one below. Click to drop poses (purple) and landmarks (orange); the tool auto-adds odometry factors between consecutive poses and lets you draw observation factors. Watch the bipartite structure form — squares (factors) always between circles (variables).

Build a Factor Graph

Pick a tool, then click the canvas. Add pose drops a pose and auto-links it to the previous pose with an odometry factor. Add landmark drops a landmark. Observe: click a pose then a landmark to add a observation factor. Close loop: click two poses to add a loop-closure factor. Squares are factors; circles are variables.

poses: 0 · landmarks: 0 · factors: 0
In a SLAM factor graph, an observation factor connects which nodes?

Chapter 6: Solving It — Nonlinear Least Squares

Chapter 5 turned SLAM into "minimize a sum of weighted squared errors." Now we actually minimize it. The errors are nonlinear (the measurement function has square roots and arctangents), so we can't solve it in one shot. Instead we use the workhorse of robotics estimation: Gauss-Newton — linearize, solve a linear system, step, repeat.

The objective, restated

We minimize F(x) = Σi ei(x)Ωiei(x) over the full variable vector x (all poses and landmarks stacked). Each ei(x) is the residual of factor i — measured value minus what the current x predicts.

Linearize: the Jacobian

Near our current guess x, approximate each error as linear in a small step Δ using a first-order Taylor expansion:

ei(x + Δ) ≈ ei(x) + Ji Δ

where Ji is the Jacobian of factor i — the matrix of partial derivatives of its error with respect to the variables it touches. Because a factor touches only a few variables (sparsity again!), Ji is zero everywhere except in the columns for those variables.

The normal equations

Substitute the linear approximation into F, expand the squares, and demand that the derivative with respect to Δ be zero (the minimum of a quadratic). Out drops a single linear system — the normal equations:

( Σi JiΩiJi ) Δ = − Σi JiΩi ei

Name the pieces. The left matrix H = Σi JiΩiJi is the (approximate) Hessian — the curvature of the cost, also called the information matrix of the whole problem. The right side b = −Σi JiΩiei is the gradient (which way is downhill). Solve H Δ = b for the step Δ, apply it (x ← x + Δ), and repeat from the new point until Δ is tiny.

What Gauss-Newton is doing, in words. "Pretend the problem is a parabola near where I'm standing. Jump straight to the bottom of that parabola. The real problem isn't a parabola, so I didn't land at the true minimum — but I'm closer. Re-draw the parabola from here and jump again." Two or three jumps usually nails a well-behaved SLAM problem.

A tiny worked step: three poses on a line

Let's grind one full Gauss-Newton step by hand. Three poses on a line, positions x1, x2, x3. The constraints:

Give every factor information Ω = 1 for simplicity. The four errors at the current guess (start from odometry: x = (0, 1, 2)) are:

Only the loop-closure error is non-zero — the rest of the graph already agrees with itself; the conflict is entirely "odometry says 2, loop says 1.6." Build H = Σ JiJi (each term is the outer product of a row with itself), summing the four:

H = [ [3, −1, −1], [−1, 2, −1], [−1, −1, 2] ]

And the gradient side b = −Σ Jiei — only the loop term contributes (e=0.4, row [−1,0,1]):

b = −(0.4)·[−1, 0, 1] = [0.4,  0,  −0.4]

Now H is singular by itself (the graph can slide freely), but the prior fixed x1, so effectively we solve with x1 pinned. Solving HΔ = b with Δ1 = 0 gives:

Δ ≈ [0,  −0.133,  −0.267]

Update: x ← (0, 1, 2) + Δ = (0,  0.867,  1.733). Look what happened: the 0.4 m of loop-closure error did not dump entirely onto x3. It spread backwardx2 moved −0.133, x3 moved −0.267, exactly the "distribute the correction over the whole loop" behavior we wanted in Chapter 4. The optimization did automatically what the filter could not. Because these constraints are linear, one step is exact; nonlinear SLAM just repeats this a few times.

From scratch in Python

python
import numpy as np

# Each factor: (rows of Jacobian, error value, information Omega)
# Variables: x1, x2, x3  (indices 0,1,2)
def gauss_newton_step(x, factors):
    n = len(x)
    H = np.zeros((n, n))            # information / approx Hessian
    b = np.zeros(n)                 # gradient (negative)
    for (J, e, omega) in factors:
        J = np.array(J, dtype=float)
        H += J.reshape(-1,1) @ J.reshape(1,-1) * omega   # Jᵀ Ω J
        b += -J * e * omega                              # -Jᵀ Ω e
    H[0,:] = 0; H[0,0] = 1; b[0] = 0    # pin x1 via the prior
    delta = np.linalg.solve(H, b)       # solve H Δ = b
    return x + delta

x = np.array([0.0, 1.0, 2.0])           # start from odometry
factors = [
    ([1,0,0],  x[0]-0.0,            1.0),   # prior  x1=0
    ([-1,1,0], (x[1]-x[0])-1.0,     1.0),   # odo 1->2
    ([0,-1,1], (x[2]-x[1])-1.0,     1.0),   # odo 2->3
    ([-1,0,1], (x[2]-x[0])-1.6,     1.0),   # loop 1->3
]
x = gauss_newton_step(x, factors)
print(x)    # -> [0.    0.867 1.733]  the correction spread backward

And the idiomatic, batch form — stack all factor rows into one big sparse Jacobian J and weight vector, and let the linear solver do the work:

python
J = np.array([[1,0,0],[-1,1,0],[0,-1,1],[-1,0,1]], float)
e = np.array([0.0, 0.0, 0.0, 0.4])           # residuals at x=(0,1,2)
Omega = np.diag([1,1,1,1.0])                 # per-factor information
H = J.T @ Omega @ J                          # Jᵀ Ω J
b = -J.T @ Omega @ e                         # -Jᵀ Ω e
H[0,:] = 0; H[0,0] = 1; b[0] = 0             # anchor
delta = np.linalg.solve(H, b)
print(delta)    # -> [0.    -0.133 -0.267]
Misconception: "least squares means averaging the errors." No — squaring errors and weighting by information means a confident loop-closure constraint can overrule a long chain of odometry. The optimum is not an average; it's the balance point where every factor's "pull" (gradient) cancels. That's also why one bad measurement with high (wrongly-assigned) information can wreck the whole solution — the squared penalty makes outliers loud. Robust cost functions (Huber, etc.) exist precisely to mute them.
In the normal equations (JᵀΩJ)Δ = −JᵀΩe, what does the matrix JᵀΩJ represent?

Chapter 7: Sparsity — Why SLAM Is Tractable

We keep promising that solving over thousands of variables is cheap. Here is the payoff. The whole reason graph SLAM scales to entire cities is one structural fact about the Hessian H = Σi JiΩiJi: it is sparse — mostly zeros.

Where the zeros come from

Consider one factor, say an odometry factor between poses x5 and x6. Its error depends on only those two variables, so its Jacobian Ji is non-zero only in the x5 and x6 columns. When you form JiΩiJi, the result is non-zero only in the four blocks where {x5,x6} meet {x5,x6}.

Summing over all factors, Hab (the block linking variable a to variable b) is non-zero only if some factor directly connects a and b. No shared factor → a structural zero. Since each variable participates in only a handful of factors, each row of H has only a handful of non-zeros. The Hessian's sparsity pattern is exactly the adjacency structure of the factor graph.

The key contrast with the EKF. Remember Chapter 3: EKF-SLAM's covariance Σ became fully dense — every variable correlated with every other. The information matrix H = Σ−1 of the smoothing problem is the opposite: naturally sparse. Filtering's marginalization is precisely the operation that fills in the zeros (it correlates everything it touches). By refusing to marginalize, smoothing keeps H sparse — and sparse linear systems solve in near-linear time instead of cubic.

Solving the sparse system: Cholesky and ordering

To solve HΔ = b we factor H (it is symmetric positive-definite) as H = LLCholesky factorization, where L is lower-triangular — then solve two easy triangular systems. The catch: Cholesky can create new non-zeros, called fill-in, turning a sparse H into a dense L and destroying the speedup.

The fix is variable ordering: permute the rows/columns of H before factoring so fill-in stays minimal. Reorder the variables cleverly (algorithms like COLAMD do this automatically) and the factorization stays nearly as sparse as H. This single trick — sparse Cholesky with a good ordering — is what lets solvers like GTSAM and g2o optimize tens of thousands of poses in milliseconds.

The Hessian Sparsity Pattern

A pose-chain factor graph with landmarks. The grid is the Hessian H: a filled cell means "these two variables share a factor." Add loop closure drops a long-range constraint — watch one off-diagonal cell light up far from the diagonal. Marginalize a pose shows how removing a variable (the EKF move) fills in its neighbors with new correlations.

sparse band + landmark arms

Notice the structure: odometry factors put entries right next to the diagonal (a band), landmark observations add a few arms, and a loop closure plants a lone entry far off-diagonal. A band is trivially cheap to factor; the off-diagonal loop-closure entries are what good ordering manages. And marginalizing a pose — deleting its row/column — couples all its former neighbors, the dense fill-in that dooms the filter.

Misconception: "more constraints = slower." Counterintuitively, in graph SLAM adding a well-placed loop-closure factor barely costs anything (one extra off-diagonal entry) yet dramatically improves accuracy. What's expensive is density, not count. A million sparse factors solve faster than a thousand dense ones. SLAM is tractable because reality is sparse: a robot at pose 500 simply does not directly constrain a landmark it saw at pose 3.
Why is the information matrix H of a graph-SLAM problem naturally sparse, while the EKF's covariance Σ becomes dense?

Chapter 8: Loop Closure — Collapsing the Drift

We've mentioned loop closure in every chapter. Now it gets its own, because it is the single most important event in any SLAM run — the moment accumulated drift gets annihilated rather than merely slowed.

What it is

A loop closure is recognizing that the place you're at now is a place you visited before. The robot drove a long loop, drift bent its trajectory estimate, and then it comes back to a corner it already mapped and says: "I've been here. The pose I'm computing now and the pose x1 from the start are the same physical location."

Structurally, that statement is just a between-poses factor — a constraint xt − x1 ≈ 0 (or whatever relative transform the re-observation implies). But unlike an odometry factor, which only links neighbors, a loop closure reaches across the entire graph. When Gauss-Newton solves the system, that one long-range constraint forces a global correction: as we saw in Chapter 6's hand calculation, the error spreads backward over every pose in the loop, snapping the whole trajectory into consistency.

Why one edge does so much. Before closure, the only thing tying the end of the loop to its start is a long chain of noisy odometry — weak, drifted, indirect. The loop-closure factor is a direct, strong link between two far-apart poses. In the information matrix it's a single off-diagonal entry, but it short-circuits the entire chain: suddenly pose t "knows" about pose 1 without going the long way around. The optimizer redistributes all the accumulated disagreement at once.

Data association: the hard part

The geometry of loop closure is easy; recognizing the revisited place is the hard part. This is data association — deciding which current observation corresponds to which previously-seen landmark or place. Get it right and drift evaporates. Get it wrong — a false loop closure, claiming two different places are the same — and you've welded the map into a contorted, confidently-wrong shape that no amount of optimization can fix.

Watch a drifted loop snap straight. Drive the robot around; odometry drifts the loop open. Then hit Detect loop closure and Optimize — Gauss-Newton runs, and the kinked trajectory collapses onto the consistent one, dragging the map with it.

Loop Closure — the trajectory snaps straight

Press Drive loop — the robot's odometry-only estimate (purple) drifts, ending far from the start (you can see the gap). Detect loop closure adds the green constraint tying end to start. Optimize runs Gauss-Newton: watch the drifted loop collapse onto the true loop (faint), correction spread over every pose.

ready
Misconception: "loop closure is just snapping the last pose back to the start." If you only moved the final pose, you'd get a sharp kink right at the closure and a still-drifted loop everywhere else. Real loop closure is a global re-optimization: the constraint enters the factor graph and the whole trajectory re-solves, so the correction is shared smoothly across hundreds of poses. The snap you see is the entire loop relaxing at once, not one pose teleporting.
A false loop closure (asserting two different places are the same) is dangerous because:

Chapter 9: Bundle Adjustment — The Same Machine, for Cameras

Here is the beautiful part. Everything we built — factor graph, weighted least squares, Gauss-Newton, sparse Cholesky — is not specific to range-bearing SLAM. Swap in a camera and the identical machinery becomes bundle adjustment (BA), the gold standard for refining 3-D reconstructions and the engine inside structure-from-motion (Lesson 9) and visual SLAM.

The setup

In BA the variables are camera poses (where each photo was taken, in 3-D) and 3-D points (the landmarks). The data are 2-D pixel observations — where each 3-D point projected onto each image. The factor connecting camera i and point j uses the reprojection error:

eij = zij − π( posei, pointj )

where zij is the observed pixel location, π is the camera projection function (the pinhole model from Lesson 9: take a 3-D point, transform it into the camera's frame, divide by depth, apply intrinsics → a pixel), and the error is the gap, in pixels, between where the point should appear and where it was seen. "Bundle adjustment" is named for adjusting the bundle of light rays from each point so they all converge correctly.

It's the same problem wearing camera clothes. Compare to range-bearing SLAM: poses become 3-D camera poses, 2-D landmarks become 3-D points, the range-bearing model h becomes the projection model π, and the residual is now measured in pixels instead of meters and radians. The objective — minimize Σ eΩe — is letter-for-letter identical. Learn graph SLAM and you've learned bundle adjustment for free.

The same sparsity, a famous structure

BA's Hessian has a special, exploitable sparsity. Camera poses are few; 3-D points are many. Crucially, points don't observe each other and cameras don't observe each other — every factor links one camera to one point. So the point-point block of H is block-diagonal, which lets you eliminate all the points cheaply in one step (the Schur complement), shrinking the problem to just the handful of cameras. This "reduced camera system" trick is why BA over thousands of points stays fast — the same sparsity-exploitation philosophy as Chapter 7, specialized to the camera-vs-point structure.

Bundle Adjustment — minimizing reprojection error

Cameras (blue) view 3-D points (orange). The thin lines are the rays from each camera to its observation; where a ray misses the point is the reprojection error (red). Press Optimize — BA jointly nudges camera poses and point positions to shrink all the red gaps at once. Press Perturb to knock things out of alignment first.

aligned
Where you've already used it. Every time your phone builds an AR map, every 3-D scan from photogrammetry, every visual-SLAM system (ORB-SLAM, VINS) runs bundle adjustment in its back end. It's arguably the most-run optimization in all of computer vision — and it's the exact factor-graph least-squares you now understand. Lesson 9's structure-from-motion ends with a BA refinement; this is that refinement's machinery.
Bundle adjustment minimizes the reprojection error. What is that error?

Chapter 10: Showcase — Pose-Graph SLAM, End to End

Time to put every piece in one place. Below is a complete 2-D pose-graph SLAM system. You drive a robot around a loop; it accumulates drifting odometry and sights landmarks along the way; then you trigger a loop closure and watch Gauss-Newton snap the entire trajectory and map into consistency — while a side panel shows the total error dropping per iteration and the factor graph it's solving.

What to look for. (1) During Drive, the estimated path (purple) peels away from the true path (faint) — that's drift accumulating, Chapter 0. (2) Each landmark sighting adds an observation factor — the graph grows, Chapter 5. (3) Close + Optimize adds the loop-closure factor and runs Gauss-Newton (Chapter 6); the error bars fall and the loop collapses onto the truth (Chapter 8). (4) The whole correction spreads over every pose, because we kept them all (Chapter 4).
2-D Pose-Graph SLAM — drive, drift, close, optimize

Press Drive to send the robot around the loop, accumulating drifted odometry and landmark sightings. When it returns near the start, Close loop lights the green constraint. Optimize runs Gauss-Newton iteration-by-iteration — the error meter (right) drops and the trajectory + landmarks snap toward truth. Add noise makes the drift worse so the correction is more dramatic.

drift 3.5%
ready — press Drive

The error meter is the literal value of F(x) = Σi eiΩiei — the total weighted squared residual across every factor. Before closure it's small-ish (odometry agrees with itself) but the map is wrong. The instant the loop-closure factor enters, the error jumps (now there's a big residual: odometry says the loop didn't close, closure says it must). Each Gauss-Newton step then drives it back down as the variables rearrange to satisfy everyone. When it flattens, you've reached the MAP estimate — the most probable trajectory and map given all the data.

You just ran modern SLAM. This is, in miniature, exactly what GTSAM / g2o / Ceres do inside ORB-SLAM, Cartographer, and the SLAM stack of every autonomous robot: build a factor graph from odometry + observations + loop closures, then solve the sparse nonlinear least-squares problem with Gauss-Newton (or Levenberg-Marquardt) and sparse Cholesky. The toy here uses dense linear algebra for clarity, but the algorithm is the real one.

Try breaking it: crank the drift to maximum before driving, so the loop ends wildly open, then optimize — watch how much of the error one loop-closure factor can absorb. Then imagine what a false closure would do (Chapter 8): the same machinery that fixes a true loop would just as confidently warp the map for a wrong one. The optimizer trusts what you tell it; data association is where the danger lives.

Chapter 11: Connections & Cheat-Sheet

You came in with a chicken-and-egg paradox and leave with two ways to crack it: a filter that fuses motion and measurement online, and a graph that re-optimizes the whole history to kill drift. Here's the whole lesson on one page.

EKF-SLAM vs. graph-SLAM at a glance

EKF-SLAM (filtering)Graph-SLAM (smoothing)
Estimatescurrent pose + map: p(xt, m | …)full trajectory + map: p(x1:t, m | …)
Data structuredense covariance Σ (3+2N)²sparse factor graph / information matrix H
Past posesmarginalized away (gone)kept — can be re-optimized
Linearizationonce, never revisitedre-linearized each iteration
Loop closureweak — can only nudge current posestrong — correction spreads over whole loop
ScalingO(N²) per update~O(N) with sparse Cholesky + ordering
Used today?historical / small problemsyes — GTSAM, g2o, Ceres, ORB-SLAM

Cheat-sheet

The two models. Motion: xt = g(ut, xt−1) + ε, ε~N(0,R). Measurement (range-bearing): r = √(Δx²+Δy²), φ = atan2(Δy,Δx) − θ, plus noise ~N(0,Q).

EKF-SLAM. Augmented state y = [pose, all landmarks]. Predict grows Σ (+R); update shrinks it via gain K = Σ̄HS−1. Off-diagonal blocks = pose-landmark correlations = the chicken-and-egg as a matrix.

Factor graph. Variables (poses, landmarks) + factors (prior, odometry, observation, loop-closure). MAP = arg minx Σi eiΩiei — weighted nonlinear least squares.

Gauss-Newton. Linearize (Jacobians J), solve the normal equations (JΩJ)Δ = −JΩe, update x ← x + Δ, repeat. H = JΩJ is the sparse information matrix.

Sparsity. Each factor touches few variables → H is sparse → sparse Cholesky + good ordering = fast. Filtering's marginalization fills it in; smoothing keeps it sparse.

Loop closure. Recognize a revisited place → one strong long-range factor → global re-optimization collapses accumulated drift. Data association (getting it right) is the hard part; false closures are catastrophic.

Bundle adjustment. The same NLLS machinery with camera poses, 3-D points, and reprojection error e = z − π(pose, point).

Related lessons on Engineermaxxing

SLAM sits at the crossroads of perception and estimation — here's where to go deeper:

"What I cannot create, I do not understand." — Richard Feynman.
You can now build a factor graph, write the error terms, and run the Gauss-Newton step that snaps a drifted map straight. Next, Lesson 12: occupancy mapping & exploration — once you know where you are, how do you turn raw range scans into a dense, drivable map of free and occupied space?
A robot must run SLAM in real time on a long mission with many loop closures and limited drift tolerance. Which approach fits, and why?