Two Kalman filters, identical RMSE — one is lying about its confidence. This lesson builds the consistency tests, information bounds, and small-sample trial statistics that catch the liar before it gets your drone killed.
You have spent a week benchmarking two Kalman filters on the same delivery-drone dataset — the same logged flights, the same ground truth from a survey-grade RTK base station. Both filters track the drone's position. Both report a position RMSE (root-mean-square error — how far, on average, the estimate sits from truth) of exactly 0.50 m. The eval table is a tie. Which one do you ship?
It is a trick question. One of these filters will get your drone flown into a wall, and the RMSE column you have been staring at cannot tell you which.
Here is what the RMSE column hides. Every Kalman filter outputs two things at each timestep: an estimate of the state (where it thinks the drone is) and a covariance P — the filter's own report of how uncertain that estimate is. P is a claim: "I am confident to within this much." Filter A reports σ = 0.5 m of uncertainty around its estimates. Filter B reports σ = 0.05 m. Same errors on the ground, but Filter B claims ten times more certainty than Filter A.
That claim is not academic. A downstream motion planner reads P to decide how wide a safety corridor to keep around obstacles. Filter B's tiny reported σ tells the planner "the drone's position is nailed down — you can shave the corridor to centimeters." But Filter B's actual errors are half a meter. The planner is being lied to, and it will happily route the drone through a gap that isn't there.
Put a number on the harm. A common corridor rule keeps a margin of 3σ around the estimate — three standard deviations covers 99.7% of a Gaussian's mass, so a well-behaved filter's true position almost never escapes it. Filter A, reporting σ = 0.5 m, asks the planner for a 1.5 m margin — and since A's errors really are ~0.5 m, that margin genuinely contains the drone. Filter B, reporting σ = 0.05 m, asks for a 0.15 m margin. But B's true errors are 0.5 m, so the drone routinely sits 3–4× outside the corridor the planner was told was safe. Same RMSE, same drone, and one filter has quietly converted a safety margin into a coin flip.
So we need to separate two ideas that the RMSE table smears together. Accuracy is how far the estimate is from truth — RMSE and its relatives, the subject of Chapter 1. Consistency is whether the filter's own uncertainty report P matches its actual error statistics — the subject of Chapters 2 and 3. These are orthogonal axes. A filter can be accurate and inconsistent (Filter B), or mediocre-but-honest, or any combination.
Consistency is the load-bearing property, and here is why: every consumer of an estimator consumes its covariance too. Data-association gates (deciding which lidar return belongs to which track) size themselves from P. Sensor-fusion weights are computed from P. Collision-avoidance margins are set from P. Anomaly detectors threshold on P. An overconfident P (too small) means gates reject good measurements, fusion over-trusts a bad sensor, and the planner shaves margins it does not actually have.
The failure has a signature shape. An overconfident filter looks better on the metrics people usually plot — its tracks are tight and smooth — right up until a measurement outlier arrives. Then its too-small validation gate rejects the perfectly good correction, the estimate coasts on its wrong model, and the filter confidently reports it is fine while drifting meters off. Engineers call this smug divergence, and it is the classic Extended Kalman Filter death mode.
This lesson is the toolkit that catches it. Chapter 1: error metrics that don't mislead. Chapter 2: NEES, the consistency test when you have ground truth. Chapter 3: NIS, the consistency test that needs no ground truth — the one you deploy. Chapter 4: the Cramér–Rao lower bound, estimation's speed limit. Chapter 5: Monte Carlo campaign design. Chapter 6: SLAM's ATE and RPE. Chapter 7: honest robot-task statistics with small-n confidence intervals. Chapter 8 is a live lab where you break a filter in both directions and watch the tests catch it.
One ownership note so you know what we assume. You already know how a Kalman filter works — that is the kalman-filter prereq, and we never re-derive it. You already know what a confidence interval and a hypothesis test are — eval-statistics owns that machinery. Here we apply both to the special structure of state estimators: they output a distribution, not a point, so evaluation must test the distribution.
Before we build any of it, let us just see the lie. The simulation below draws a drone trajectory and two filter tracks with deliberately identical error — their RMSE matches to the decimal. Toggle the covariance ellipses and watch: Filter A's 95% ellipses comfortably contain the truth about 95% of the time; Filter B's are ten times too small and the truth sits far outside almost every one. The containment counter tells you the story RMSE couldn't.
Truth in teal. Filter A (warm) and Filter B (purple) have identical error — same RMSE to the decimal. Hit Show 95% ellipses: A's ellipses swallow the truth ~95% of the time; B's are tiny and the truth escapes almost every one. The slider heals B's lie — drag its claimed σ up and watch its containment climb back toward 95%.
Now let us make the intuition precise with a single number, in one dimension. Both filters have an estimation error of e = 0.5 m at some timestep — that is what pins their RMSE together. Filter A reports variance 0.25 (so σ = 0.5 m); Filter B reports variance 0.0025 (so σ = 0.05 m). How plausible is each filter's story about its own error?
The trick is to normalize the error by the variance the filter claims — divide the squared error by the claimed variance. This is the squared z-score: how many standard deviations out did the error land? For Filter A:
The error is exactly one σ out — completely ordinary. A Gaussian error lands at least one σ from its mean a third of the time: P(|Z| > 1) = 0.317. Filter A's story checks out. Now Filter B:
The error is ten σ out. The probability of a ten-σ error if B's covariance were honest is P(|Z| > 10) = 1.5 × 10−23. If Filter B were telling the truth about its uncertainty, you would not see this error once in the lifetime of the universe of drone flights. The same error e = 0.5 — same contribution to RMSE — produces a normalized error of 1 for the honest filter and 100 for the liar. That single normalization instantly separates them, and it is exactly what NEES will generalize to vectors in Chapter 2.
Here is that lie detector as five lines of Python. Compute the per-step error, normalize by the reported variance, and flag the fraction of steps that blow past the 95% cutoff. For one degree of freedom, the chi-square 95% cutoff is 3.84 — a normalized error above it is a 1-in-20 event.
python — the 5-line lie detector import numpy as np # err: per-step estimation errors (T,); P_diag: filter's claimed variance per step (T,) def overconfidence_rate(err, P_diag): nse = err**2 / P_diag # normalized squared error, per step return (nse > 3.84).mean() # 3.84 = chi2.ppf(0.95, 1); healthy filter ~= 0.05 # Filter A: honest. Filter B: reports 100x too-small variance. err = np.full(1000, 0.5) print(overconfidence_rate(err, np.full(1000, 0.25))) # ~0.0 (nse = 1.0 everywhere) print(overconfidence_rate(err, np.full(1000, 0.0025))) # 1.0 (nse = 100 everywhere — every step flagged)
And the two numbers you would verify against scipy, so you trust the arithmetic:
python — verify print(0.5**2/0.25, 0.5**2/0.0025) # 1.0 and 100.0 from scipy import stats print(2*stats.norm.sf(1), 2*stats.norm.sf(10)) # 0.3173 and 1.52e-23
You have seen the lie. Now we build the instruments — and the first one has nothing to do with covariance at all. Before we can test whether P is honest, we need error metrics that don't mislead us about the errors themselves. That is Chapter 1.
Your AV localization stack ships with one line in the release notes: "RMSE: 0.41 m." The safety reviewer reads it and asks three questions. In which axis — lateral or longitudinal? At what point in the drive — steady, or spiking in turns? And what about heading? You realize your one number answered none of them.
This chapter is about what a single scalar throws away. Every metric here summarizes the same underlying signal, and every summary loses information. The craft is knowing what each one hides.
Start from what error even is for a state estimator. At every timestep the error is a vector, et = xtrue(t) − x̂(t) — truth minus estimate, one entry per state component. Everything in this chapter is a different way of collapsing that vector-over-time into something you can put in a table.
Build RMSE from first principles: square each per-axis error, average over time, take the square root. The squaring is the whole personality of the metric — it punishes occasional large errors far harder than uniform small ones, which is usually what safety wants. But the same squaring means one bad second can dominate an entire run's score.
Per-axis slicing is not optional for a vehicle. Lateral and longitudinal error are not interchangeable: 0.4 m longitudinal is a comfortable stopping margin; 0.4 m lateral is a lane departure. Report RMSE per axis in the frame that physically matters — usually the vehicle body frame or the road frame, not the world frame the filter happens to run in.
Watch the mixed-state trap. Position is in meters, velocity in m/s, heading in radians. Summing squared errors across components with different units produces a physically meaningless scalar — and a filter can "improve" it by trading 0.1 rad of heading error for 0.1 m of position error as if radians and meters were the same currency. Always separate error reports by state block. And never average raw radian differences: wrap them to the range [−π, π] first, or a heading of 359° versus 1° reads as a 358° error instead of 2°.
Error-over-time curves recover what the scalar destroys: when the error happened. Plot |et| against time across the run. Does error spike during turns (model mismatch)? Grow monotonically (drift, or loss of observability)? Ring after initialization and then settle (a transient)? The shape diagnoses the cause; the scalar never can.
This is why terminal error alone misleads. A run can wander meters off mid-mission and get lucky — a loop closure or a GPS reacquisition snaps it back at the end. Terminal error reads 0.1 m; the mission actually spent 40% of its time outside its safety envelope. The worked example below shows a terminal error four times smaller than the run's true RMSE.
Finally, aggregation across runs: median versus mean of per-run RMSE. Robot logs are heavy-tailed — one diverged run out of fifty. The mean is dominated by that one disaster; the median hides it entirely. Report both, plus the max and the failure count. (eval-statistics owns the general theory of robust aggregation; here the point is that in estimation, heavy tails are the norm, not the exception.)
Let us make the terminal-error trap concrete with a five-step run you can check by hand. The per-axis errors, in meters, are:
Step 1 — RMSE in x. Square each: [0.09, 0.16, 0.25, 0.04, 0.01]. Sum = 0.55. Mean = 0.55/5 = 0.11. RMSEx = √0.11 = 0.332 m.
Step 2 — RMSE in y. Square each: [0.01, 0.04, 0.09, 0.16, 0.00]. Sum = 0.30. Mean = 0.30/5 = 0.06. RMSEy = √0.06 = 0.245 m.
Step 3 — combined position RMSE. The per-step squared norms are ex² + ey² = [0.10, 0.20, 0.34, 0.20, 0.01]. Sum = 0.85. Mean = 0.85/5 = 0.17. RMSEpos = √0.17 = 0.412 m.
Step 4 — the Pythagorean identity. Notice RMSEpos² = RMSEx² + RMSEy², because 0.11 + 0.06 = 0.17. The combined RMSE is the Pythagorean sum of the per-axis RMSEs — a useful sanity check when reading a table.
Step 5 — the trap. Terminal error is the norm of the last error: √((−0.1)² + 0.0²) = 0.1 m. A report quoting only terminal error claims 0.1 m. The run's actual RMSE is 0.412 m — 4.1× worse. And the mid-run excursion at t=2 (√0.34 = 0.583 m, the worst single step) vanishes entirely from the terminal-only story.
The aggregation trap is just as sharp across runs. Say you evaluate a filter on ten logged missions and their per-run RMSEs (meters) are [0.30, 0.32, 0.28, 0.35, 0.31, 0.29, 0.33, 0.30, 0.34, 4.10] — nine clean runs and one divergence. The mean per-run RMSE is (2.82 + 4.10)/10 = 0.692 m. The median is 0.315 m. Report only the mean and a reviewer thinks the filter is mediocre everywhere; report only the median and the divergence — the one that would crash a real drone — is completely invisible. Neither number alone is honest. The honest report is median 0.315 m, mean 0.692 m, max 4.10 m, and failures 1/10. In estimation the heavy tail is the norm: one bad run out of ten is a Tuesday, and the summary you pick decides whether you see it.
From scratch, printing every intermediate so the hand arithmetic above is reproduced line by line:
python — from scratch import numpy as np def error_report(truth, est): # truth, est: shape (T, 2) err = truth - est # (T, 2) error signal T = err.shape[0] # per-axis RMSE via explicit accumulation sx = sy = 0.0 for t in range(T): sx += err[t,0]**2; sy += err[t,1]**2 rmse_x, rmse_y = (sx/T)**0.5, (sy/T)**0.5 rmse_pos = ((sx+sy)/T)**0.5 # = sqrt(rmse_x^2 + rmse_y^2) eot = np.hypot(err[:,0], err[:,1]) # error-over-time curve return rmse_x, rmse_y, rmse_pos, eot[-1], eot.max() ex = np.array([0.3,-0.4,0.5,0.2,-0.1]); ey = np.array([0.1,0.2,-0.3,0.4,0.0]) truth = np.zeros((5,2)); est = -np.stack([ex,ey],axis=1) # truth-est = [ex,ey] print(error_report(truth, est)) # -> (0.33166, 0.24495, 0.41231, 0.1 terminal, 0.58310 max at t=2)
And the library one-liner — same answer, no loop:
python — library rmse = np.sqrt(((truth - est)**2).mean(axis=0)) # per-axis: [0.33166, 0.24495] eot = np.linalg.norm(truth - est, axis=1) # error-over-time curve print(np.hypot(*rmse), eot[-1], eot.max()) # 0.41231, 0.1, 0.58310
Play with the anatomy explorer below. The top pane drives a vehicle through straights and turns; the bottom pane draws the error-over-time curve live, with horizontal lines for RMSE, mean absolute error, and terminal error. Switch regimes and watch how the same summary lines tell radically different stories.
Top: truth (teal) and estimate (warm). Bottom: the live error norm with summary lines. Pick a failure regime and see which summaries capture it and which are blind. The terminal-error line is the one that lies when a run reconverges at the end.
The numbers — median 0.315 m, mean 0.692 m, max 4.10 m, failures 1/10 — are honest, but a table cannot show you the shape of that heavy tail. Two picture-idioms do, and they are exactly the two robotics papers reach for. The first is the box-and-whisker plot: a compact drawing of the five-number summary — minimum, first quartile (Q1), median, third quartile (Q3), maximum. For our ten runs sorted — [0.28, 0.29, 0.30, 0.30, 0.31, 0.32, 0.33, 0.34, 0.35, 4.10] — that summary is min 0.28, Q1 0.300, median 0.315, Q3 0.3375, max 4.10 (numpy's linear-interpolation convention for the quartiles).
The box spans Q1 to Q3, so its width is the IQR (interquartile range, Q3 − Q1) — here a razor-thin 0.0375 m. That tiny box is the whole story of the nine good runs: they are nearly identical, clustered within four centimetres. The whiskers reach to the most extreme points still inside the whisker fence — the standard rule draws the upper fence at Q3 + 1.5×IQR = 0.3375 + 1.5×0.0375 = 0.3938 m. Every run below that fence (0.28 through 0.35) sits under the whisker; the 4.10 m run is far past it, so the rule auto-flags it as an outlier — a lone dot, not part of the box or whiskers. The plot decided, from the data alone, that this run is a catastrophe apart.
The second idiom is the empirical CDF (ECDF) — a step function answering "what fraction of missions came in under error x?" Sort the runs, then step up by 10% at each one. You read it left to right: ninety percent of missions land under 0.35 m, the curve then goes dead flat — no run lives between 0.35 and 4.10 — until a single final step to 100% at 4.10 m. That flat plateau is the tail made visible: the catastrophe you must not average away. Drop the 4.10 run and the mean collapses from 0.692 to 0.313 m, while the median barely twitches (0.315 → 0.310 m) — the picture below lets you pull that lever and watch it happen.
Left: the five-number summary as a box (Q1–Q3), a median line, whiskers to the fence, and the 4.10 m run pinned as a labelled outlier dot (its true value is off the box's scale). Right: the error-CDF — a step up per run to 90% by 0.35 m, a flat plateau, then the last step to 100% at 4.10 m. Remove the failed run and watch the mean collapse while the median holds.
Building these summaries from absolute zero — how quartiles interpolate, why the CDF steps, what a box plot's fence actually computes — is its own small craft: see Summarizing Distributions.
Every metric in this chapter needs ground truth, and none of them touches the covariance. A filter could ace every one of them and still be the liar from Chapter 0. To catch that, we have to stop measuring the estimate and start testing the claim — which means bringing P into the math. That is NEES.
Chapter 0 caught the lie in one dimension by dividing the squared error by the claimed variance. Now we do it properly: vector errors, correlated covariances, and an exact distribution to test against — so the test stops being a hunch and becomes a hypothesis test.
Derive the statistic from the question that matters: is this error plausible under the filter's claim? The filter claims its error is drawn from e ~ N(0, P) — a zero-mean Gaussian with covariance P. To test that claim, whiten the error by the claimed covariance — divide it out. The result is the Normalized Estimation Error Squared:
Read it in 1D first, where it is just (e/σ)² — the squared z-score from Chapter 0. In multiple dimensions, P−1 is the matrix version of "divide by variance," and it does one extra job: it rotates to account for correlations between state errors. It knows that if x-error and y-error are supposed to move together, an error where they move apart is more surprising.
Geometrically, NEES measures how many covariance ellipses out the error sits. NEES = 1 means the error lands on the 1-σ ellipse; NEES = 9 means the 3-σ ellipse. An elongated P — the filter admitting large uncertainty in one direction — forgives large errors in that direction, exactly as the ellipse picture suggests.
Now the fact that turns a score into a test. If e is truly N(0, P), then after whitening, eTP−1e is a sum of n squared independent standard normals — which is, by definition, a chi-square distribution with n degrees of freedom, where n is the state dimension. Its expected value is exactly n. So for a 2D position error, a healthy filter's NEES hovers around 2.
Let us make the whitening concrete in one line so the "sum of squared standard normals" claim isn't a black box. Factor the covariance as P = LLT (a Cholesky factorization — L is the "square root" of P). Define u = L−1e. Because e ~ N(0, P), the transformed vector u has covariance L−1P L−T = L−1LLTL−T = I — it is standard, uncorrelated, unit-variance. And NEES = eTP−1e = eTL−TL−1e = uTu = u1² + u2² + … + un². That is literally n squared standard normals added up — the definition of chi-square(n). Whitening is just "rotate and rescale the error into the frame where the covariance is the identity, then measure its squared length."
Single-run bounds. For n = 2, the central 95% of the chi-square(2) distribution runs from 0.051 to 7.38. One NEES sample outside that band is a 1-in-20 event — suspicious, not damning. Consistency testing is about aggregate behavior over time and over runs, never a single sample.
Averaging is what gives the test teeth. The sum of N independent chi-square(n) variables is chi-square(N·n), so the average of N runs has bounds chi2.ppf(α, N·n)/N. For 50 runs of a 2-state filter, the 95% band tightens from [0.051, 7.38] all the way to [1.484, 2.591]. Now a persistent average of, say, 5 is unmistakable.
| Average NEES vs bounds | Diagnosis | Cause & consequence |
|---|---|---|
| Persistently above upper bound | Overconfident (P too small) | Q or R set too small, or an unmodeled bias. Gates too tight → rejects good measurements → smug divergence. The dangerous one. |
| Inside the band | Consistent | Errors match the claimed covariance. Ship it. |
| Persistently below lower bound | Underconfident (P too large) | Filter wastes information: slow convergence, gates too generous, downstream over-buffers. Safe but sluggish. |
Both extremes are tuning bugs. Overconfidence is the one that hurts people; underconfidence merely wastes performance — but it is still a defect, and Chapter 8 will make you feel why.
Practical protocol. Run NEES per timestep and plot the time series against the single-run bounds; also plot the run-averaged NEES against the tight bounds. A filter that is consistent on average but blows the bounds during turns has a maneuver-dependent model error — the time-resolved plot localizes it in a way the single number never could.
The honest limitation: NEES requires ground truth — motion capture, RTK-GPS, or simulation — because it needs xtrue. That is why NEES lives in the lab and in simulation campaigns, and why Chapter 3's NIS, which needs no truth, is what you actually deploy.
Let us compute NEES by hand for a 2-state filter, and see how correlation changes the verdict. The error is e = [0.3, −0.4] m. We test it against two different claimed covariances.
Case 1 — diagonal P. The filter claims P = [[0.09, 0], [0, 0.04]] — no correlation. Then P−1 is just [[1/0.09, 0], [0, 1/0.04]] = [[11.111, 0], [0, 25]]. So:
Interpretation: expected NEES is n = 2, and the single-run 95% band is [0.051, 7.378]. 5.0 is inside — elevated (the y-error is 2σ out) but not a violation on its own.
Case 2 — correlated P. Now the filter claims P = [[0.09, 0.02], [0.02, 0.04]] — a positive correlation between x and y errors. Invert by hand. The determinant is det(P) = 0.09×0.04 − 0.02×0.02 = 0.0036 − 0.0004 = 0.0032. For a 2×2, the inverse swaps the diagonal, negates the off-diagonal, and divides by the determinant:
Compute P−1e first. Row 1: 12.5×0.3 + (−6.25)×(−0.4) = 3.75 + 2.5 = 6.25. Row 2: (−6.25)×0.3 + 28.125×(−0.4) = −1.875 − 11.25 = −13.125. So P−1e = [6.25, −13.125]. Then:
Interpretation: 7.125 is just inside the 7.378 upper bound. The claimed positive correlation says x and y errors should move together, but this error has opposite signs (+0.3, −0.4). The correlated P finds that pattern far less plausible than the diagonal P did — 7.125 versus 5.0. NEES punishes correlation-pattern violations, not just magnitudes.
The aggregate check. If 50 independent runs averaged NEES = 7.1, that is far above the 50-run upper bound of 2.591 — a decisive overconfidence verdict, even though each single sample sat comfortably inside its single-run band. This is the whole point of averaging.
From scratch, with the 2×2 inverse done by the adjugate formula so it matches the hand steps exactly:
python — from scratch import numpy as np def inv2x2(P): a,b,c,d = P[0,0], P[0,1], P[1,0], P[1,1] det = a*d - b*c # 0.0032 for the correlated case return np.array([[ d, -b], [-c, a]]) / det # adjugate / det def nees(e, P): Pinv = inv2x2(P) Pe = Pinv @ e # two explicit dot products return float(e @ Pe) # the quadratic form e = np.array([0.3, -0.4]) print(nees(e, np.diag([0.09,0.04]))) # 5.0 print(nees(e, np.array([[0.09,0.02],[0.02,0.04]]))) # 7.125
And the library version — use solve, never an explicit inverse, in production (it is cheaper and numerically stabler):
python — library from scipy import stats nees = e @ np.linalg.solve(P, e) # same number, no explicit inverse n, N = 2, 50 lo, hi = stats.chi2.ppf([0.025,0.975], N*n) / N # [1.4844, 2.5912] for 50-run average print(stats.chi2.ppf([0.025,0.975], 2)) # single-run band [0.0506, 7.3778]
Drag the sliders in the prober below to break the filter both ways. Shrink the claimed P below the true error spread and the histogram slides right, the needle punches above the band, the dots glow red — overconfident. Inflate it and everything sags below — underconfident. The "correlate errors" slider tilts the true cloud so you can watch NEES catch a correlation mismatch, not just a magnitude one.
Left: cross-run error dots against the filter's claimed 1/2/3-σ ellipses (dots outside 3σ glow red). Right: the NEES histogram over the ideal chi-square(2) density, with the 95% band shaded and the 50-run average drawn as a needle against its tight bounds [1.48, 2.59].
NEES is powerful, but it has a fatal deployment problem: it needs xtrue, and your robot in a customer's warehouse has none. So how do you test consistency with no ground truth at all? The answer is to test a different quantity — one that reality grades for free, thirty times a second. That is NIS.
NEES needs ground truth, and your robot in a customer's warehouse has none. But there is one quantity the filter predicts before every measurement arrives — and reality grades that prediction 30 times per second. That quantity is the innovation, and testing it is the deployed filter's health check.
The key move is to shift the test from the state (which needs truth) to the measurement (which arrives for free). Before each update, the filter predicts what the measurement should be, H xpred, and how much it should scatter, the innovation covariance S = H P HT + R. The innovation is the surprise — the gap between the measurement that arrived and the one predicted:
If the filter is honest, this surprise is distributed as ν ~ N(0, S). So we whiten it exactly as we whitened the error for NEES, giving the Normalized Innovation Squared:
It is structurally identical to NEES — but built entirely from quantities the filter already computes in its gain step, at zero extra cost. Under consistency, NIS ~ chi-square(m), where m is the measurement dimension — not the state dimension. (Copying NEES's degrees of freedom into NIS code is a classic bug: a 6-state filter with 2D position measurements has NEES DOF 6 but NIS DOF 2.)
Why does this work without truth? Because the innovation sequence of a correctly-specified Kalman filter is white (uncorrelated in time), zero-mean, with covariance S. Any modeling sin — wrong Q, wrong R, a sensor bias, a missed maneuver — leaks into the innovations. The filter cannot hide from its own prediction errors, and it broadcasts them on every update.
That gives you three innovation checks in practice:
| Check | What it tests | What a failure means |
|---|---|---|
| NIS magnitude vs chi-square bounds | Variance correctness | S is wrong — Q or R mis-scaled, sensor degraded |
| Innovation whiteness (autocorrelation) | Time-independence | Time-correlated innovations = model dynamics error (e.g. Q too small, filter sluggish) |
| Innovation mean near zero, per channel | Unbiasedness | Bias — a lidar mounted 2 cm off shows up here first |
As a deployed health check, run a windowed average NIS — say the last 100 measurements — against chi-square bounds scaled by the window, and alarm when it exits. This is precisely how GNSS integrity monitors do RAIM-style fault detection, and how avionics flagged the 2024–2025 surge in GPS jamming and spoofing around the Baltic and Middle East: the innovation stream says "this measurement no longer matches my physics."
The same statistic, per measurement, is the validation gate: reject a measurement when its NIS exceeds, say, chi2.ppf(0.99, m). Now Chapter 0's failure mode is precise. An overconfident filter shrinks S, so honest measurements produce huge NIS and get rejected as outliers — starving the filter of the very corrections it needs. Smug divergence is a too-small S turning good data away at the door.
Watch the death spiral in numbers. Suppose the true innovation scatter is Strue = 4 but an overconfident filter believes Sf = 0.25 (16× too small). A perfectly ordinary measurement, one σtrue out, has raw innovation ν = 2, giving a true NIS of 2²/4 = 1 — totally normal. But the filter computes 2²/0.25 = 16, well past its 99% gate of chi2.ppf(0.99, 1) = 6.63. So it rejects the good measurement as an outlier. With no correction, the estimate coasts on its model, the next measurement is even further off, and the gate rejects that one too. The filter has locked itself out of reality — confidently, because its tiny S also makes it report near-perfect certainty the whole way down.
What NIS cannot see: it tests the filter's model of the measurement stream, not the state. A state component that no measurement observes — absolute yaw with position-only measurements, or scale in monocular visual odometry — can drift arbitrarily while the innovations stay perfectly white and well-scaled. NIS passing is necessary, not sufficient. You still need NEES campaigns in simulation (Chapter 5) to certify the whole state.
Let us work NIS by hand with the same skeleton as NEES, then compute the bounds you would actually alarm on. At one step the innovation is ν = [1.2, −0.8] and the filter's predicted innovation covariance is S = [[4, 1], [1, 2]].
Step 1 — determinant. det(S) = 4×2 − 1×1 = 8 − 1 = 7.
Step 2 — inverse. Swap the diagonal, negate the off-diagonals, divide by 7: S−1 = (1/7)×[[2, −1], [−1, 4]] = [[0.2857, −0.1429], [−0.1429, 0.5714]].
Step 3 — S−1ν. Row 1: (2×1.2 − 1×(−0.8))/7 = (2.4 + 0.8)/7 = 3.2/7 = 0.4571. Row 2: (−1×1.2 + 4×(−0.8))/7 = (−1.2 − 3.2)/7 = −4.4/7 = −0.6286.
Step 4 — NIS.
Step 5 — judge it. m = 2, so the expected NIS is 2 and the single-step 95% band is [0.051, 7.378]. 1.05 is comfortably inside — this measurement is exactly as surprising as the filter predicted. It also passes a 99% validation gate (chi2.ppf(0.99, 2) = 9.21) easily.
Step 6 — the alarm bounds. For the windowed average NIS over K = 100 steps with m = 2, the bounds are chi2.ppf(0.025, 200)/100 = 1.627 and chi2.ppf(0.975, 200)/100 = 2.411. A sustained windowed average of, say, 3.0 — even though every individual NIS sits below 7.38 — is a definitive fault. Windowing catches slow drift that single-step gating never would.
From scratch, as the online monitor a real robot would run — a ring buffer, a windowed mean, precomputed bounds:
python — from scratch: online monitor import numpy as np from scipy import stats from collections import deque class NISMonitor: def __init__(self, m, K=100): self.buf = deque(maxlen=K) self.lo, self.hi = stats.chi2.ppf([.025,.975], K*m) / K # [1.627, 2.411] def update(self, nu, S): det = S[0,0]*S[1,1] - S[0,1]*S[1,0] # 2x2 adjugate inverse Sinv = np.array([[S[1,1],-S[0,1]],[-S[1,0],S[0,0]]]) / det nis = float(nu @ Sinv @ nu) self.buf.append(nis) win = np.mean(self.buf) return {'nis': nis, 'windowed': win, 'alarm': not (self.lo < win < self.hi)} mon = NISMonitor(m=2) print(mon.update(np.array([1.2,-0.8]), np.array([[4.,1.],[1.,2.]]))) # nis=1.0514
Library version — solve for the quadratic form, chi-square for the bounds:
python — library nis = nu @ np.linalg.solve(S, nu) # 1.0514 lo, hi = stats.chi2.ppf([.025,.975], K*m) / K # windowed bounds alarm = not (lo < np.mean(buffer) < hi)
The monitor below is a live telemetry dashboard. Inject faults mid-stream and count how many steps the windowed NIS needs to catch each one. The window slider is the core tradeoff: short windows catch faults fast but false-alarm on noise; long windows are calm but slow.
Top: the tracked signal with measurements arriving. Middle: per-step NIS dots against the 95% band. Bottom: the 100-step windowed average against its tight band, with a latching ALARM lamp. Inject a fault and watch the detection latency. Shorten the window: faster detection, more false alarms.
NEES and NIS both answer "does the error match the claimed covariance?" But neither answers a deeper question: how good could the filter possibly be? Maybe 0.4 m isn't the filter's fault at all — maybe the sensors simply don't contain the information to do better. There is a hard floor, and it is computable. That is the Cramér–Rao bound.
Your filter's variance stopped improving no matter how you tune it. You have spent three days on Q and R. Is the tuning bad — or have you hit a wall that no estimator, however clever, can pass? Physics has an answer, and unlike your intuition, it is computable.
The evaluation question the bound answers is blunt: is 0.4 m the filter's fault or the sensor's fault? Without a floor, teams burn weeks tuning against a limit set by information the measurements simply do not contain — the estimation equivalent of pushing on a locked door.
Build Fisher information from intuition first. Information is how sharply the likelihood peaks as you move the parameter. If shifting the true value a little changes the data distribution a lot, the data is informative — you can pin the parameter down. If the likelihood is flat, no estimator can find it. Fisher information quantifies this as the expected squared slope of the log-likelihood — how curved the peak is.
Do the one-parameter case completely. You are estimating a constant a from N noisy samples zi = a + noise, the noise Gaussian with known σ. Each sample's log-likelihood curvature contributes 1/σ² of information, and — this is the mathematical spine of sensor fusion — information adds across independent measurements:
Now the bound itself. The Cramér–Rao lower bound (CRLB) says any unbiased estimator's variance is at least the reciprocal of the information:
An estimator that achieves the CRLB is called efficient. Efficiency is CRLB / actual variance — the fraction of the attainable information the estimator actually extracts. An efficient estimator scores 1.0; one that throws information away scores less.
The Kalman connection (no re-derivation, just the payoff): for linear-Gaussian systems, the Kalman filter is the efficient estimator — its P equals the CRLB propagated through time. That is the precise sense in which the KF is "optimal." And it is why comparing a nonlinear filter's covariance against the recursive (posterior) CRLB, computed in simulation, tells you exactly how much performance the EKF's linearization is leaving on the table — cross-link ekf, where that linearization error is the number-one real-world cause of the inconsistency Chapter 2 detects.
How the bound is used in real eval: plot the Monte Carlo RMSE of your EKF/UKF/particle filter against the √CRLB curve over time. The gap is your algorithm's inefficiency. A gap near zero means further algorithm work is wasted — only better sensors (more information) can help. This one plot has killed more doomed tuning projects than any code review.
The caveats interviewers probe: the bound covers unbiased estimators (biased ones can beat it in mean-squared error — that is what shrinkage does); it requires a regular likelihood (differentiable, with support not depending on the parameter); and it can be loose — in nonlinear problems no estimator may achieve it. It is a floor, not a promise.
Let us work the numbers. Estimate a constant voltage a from N = 25 samples corrupted by Gaussian noise with σ = 2 V.
Step 1 — information per sample. 1/σ² = 1/4 = 0.25.
Step 2 — total information. It adds over N = 25 independent samples: I = 25×0.25 = 6.25.
Step 3 — the floor. CRLB = 1/I = 1/6.25 = 0.16 V², i.e. a standard deviation of √0.16 = 0.4 V. No unbiased estimator can beat 0.4 V of spread with this data. Period.
Step 4 — the sample mean. Its variance is σ²/N = 4/25 = 0.16 V². Efficiency = CRLB/var = 0.16/0.16 = 1.0. The mean extracts every drop of information — it is efficient. (This is why "just average your samples" is not lazy; for this problem it is optimal.)
Step 5 — a lazy estimator. Average only the first 10 samples: variance = 4/10 = 0.4 V². Efficiency = 0.16/0.4 = 0.4. It wastes 60% of the available information — equivalently, it performs as if it had 10 samples, because it did only use 10.
Step 6 — the eval translation. If your deployed estimator's Monte Carlo variance is 0.4 against a floor of 0.16, the gap is algorithmic and fixable. If it is 0.17, stop tuning — buy a quieter sensor. (Halving σ quarters the floor: 0.16 → 0.04.)
Now the arithmetic behind the interview lens, because "two receivers halve the error" is the single most common Fisher-information mistake. Suppose two GPS receivers, each with per-sample information 0.25, both observing the same position.
Independent case. Information adds: I = 0.25 + 0.25 = 0.5. The variance floor is 1/I = 2.0, versus 4.0 for one receiver — the variance halved. But the reported number people care about is the standard deviation, and √2.0 = 1.414 versus √4.0 = 2.0. So the σ improved from 2.0 to 1.414 — a 29% reduction, not 50%. Cutting variance in half only cuts σ by √2.
Correlated case. Real receivers share multipath, ionospheric delay, and satellite-geometry error, so their noise is correlated with, say, ρ = 0.8. Their combined information is nowhere near 2×: as ρ → 1 the second receiver adds zero information (it is a copy of the first), and even at ρ = 0.8 the effective gain is a fraction of the independent doubling. The lesson: duplication buys almost nothing; diversity (GPS + wheel odometry + lidar, with independent error mechanisms) is what actually adds information — because independence is the hidden assumption in "information adds."
From scratch — a Monte Carlo confirmation that the mean hits the floor and the lazy estimator doesn't:
python — from scratch: Monte Carlo import numpy as np sigma, N = 2.0, 25 I = N / sigma**2; crlb = 1 / I print(I, crlb, crlb**0.5) # 6.25, 0.16, 0.4 rng = np.random.default_rng(0) z = rng.normal(3.0, sigma, (200000, N)) # 200k datasets of 25 samples print(z.mean(axis=1).var()) # ~0.16 (mean of 25 -> hits floor) print(z[:, :10].mean(axis=1).var()) # ~0.40 (first-10 -> efficiency 0.4) print(crlb / z.mean(axis=1).var(), # efficiency ~1.0 crlb / z[:, :10].mean(axis=1).var()) # efficiency ~0.4
Library form for this Gaussian-mean problem — the general dynamic case uses posterior-CRLB recursions:
python — library crlb = sigma**2 / N # 0.16 efficiency = crlb / estimates.var() # 1.0 for the sample mean # dynamic systems: filterpy / custom posterior-CRLB recursions vs your filter's P
The visualizer below makes the floor visceral. Watch three estimators race down toward the CRLB line as N grows — the sample mean rides right on the floor (efficiency 1.0), the first-10 hovers well above it (0.4), and the median sits about 64% efficient for Gaussian data, a permanent tax for using order statistics instead of the mean.
Left: the log-likelihood curve sharpening as N grows or σ shrinks — the curvature at the peak is the Fisher information. Right: variance of three estimators descending toward the CRLB floor σ²/N, with live efficiency percentages. The mean hits it; the median is stuck at ~64%.
The CRLB tells you how good a filter could be. NEES and NIS tell you whether its covariance is honest. But every one of these is measured on runs — and a single run of a stochastic system is an anecdote. To turn anecdotes into evidence, you need campaigns. That is Chapter 5.
Your new filter beat the old one on the bench trajectory: 0.45 m versus 0.50 m. You re-ran with a different noise seed and it lost. Which result do you believe? Neither. One run of a stochastic system is an anecdote, and you have been making promotion decisions on anecdotes.
Frame the problem precisely. A filter's error on any single run is a random variable — different noise realizations, initial errors, and disturbance sequences give different numbers. Evaluating on one run samples that distribution once and pretends the sample is the mean. A Monte Carlo campaign samples it M times deliberately, so you can see the distribution instead of guessing it.
Anatomy of a campaign: fix the scenario (trajectory, sensor suite); randomize what nature randomizes (measurement-noise seeds, process disturbances, initial-condition draws); run M independent trials; and log the full error and covariance traces per trial — not just summary scalars, because Chapter 1 taught us the time structure matters.
Two headline plots come out of every campaign. The first is the RMSE-over-time band: per-timestep RMSE across runs with, say, 5th–95th percentile shading. It shows transients, convergence rate, and whether the tail is tight or heavy. The second is the consistency overlay: at chosen timesteps, the cloud of actual errors across runs with the filter's predicted covariance ellipse drawn on top. If the cloud spills outside the ellipse, the filter is overconfident; if the ellipse dwarfs the cloud, underconfident. This is Chapter 2's NEES made visual.
The most actionable tuning signal there is comes from the empirical covariance. Stack the M error vectors at time t and compute their sample covariance; compare it to the filter's predicted Pt. The ratio of empirical to predicted variance per axis tells you exactly which axis is mis-tuned and by how much: a ratio of 4 means σ is underestimated 2× on that axis. The worked example runs this diagnosis.
How many runs? The question every review asks. Treat the campaign like the experiment it is: the mean-squared-error estimate has a standard error s/√M (s = the across-run standard deviation of the squared errors), so the confidence-interval width shrinks as 1/√M. Halving the CI costs 4× the runs. eval-statistics owns the CI theory; here we plug estimator quantities into it.
Comparing two filters properly — pair the runs. Give both filters the same noise seeds and compare per-seed differences. Pairing removes the shared randomness and can shrink the runs needed by an order of magnitude versus independent campaigns. If your eval harness doesn't support seed control, that is the first bug to fix.
Here is why pairing is so powerful, in one variance identity. When you compare two independent campaigns, the variance of the difference is Var(A) + Var(B) — the two noise sources stack. When you pair seeds, the difference on each seed is Di = Ai − Bi, and its variance is Var(A) + Var(B) − 2·Cov(A, B). Both filters see the same hard trajectories and easy trajectories, so their errors are strongly positively correlated — a nasty gust hurts both. That large positive covariance is subtracted. Concretely, if each filter's per-run RMSE has variance 0.01 and their correlation is 0.9, the unpaired difference-variance is 0.02 but the paired one is 0.02 − 2×0.9×0.01 = 0.002 — ten times smaller, so you need ~10× fewer runs to resolve the same true gap. Pairing doesn't change the filters; it cancels the shared luck.
Campaign-design sins that quietly ruin results: randomizing only measurement noise while keeping the trajectory fixed (you've evaluated one maneuver, not the filter); sampling initial errors from the filter's own P (circular — it presumes the consistency you are testing); and cherry-picking the seed set after seeing results. Pre-register the scenario list and seed ranges like a clinical trial.
Now the worked example. A 50-run campaign. Per-run terminal squared errors have mean MSE = 0.25 m² with across-run standard deviation s = 0.10 m². Separately, at t = 30 s the empirical error variance across runs is 0.36 m² on the x-axis while the filter's predicted P reports 0.09 m².
Step 1 — standard error of the MSE. SE = s/√M = 0.10/√50 = 0.10/7.071 = 0.01414 m².
Step 2 — 95% CI on MSE. 0.25 ± 1.96×0.01414 = 0.25 ± 0.02772 = [0.2223, 0.2777] m².
Step 3 — convert to RMSE. Square-root the endpoints (a monotone transform, so it preserves the interval). RMSE = √0.25 = 0.50 m, CI = [√0.2223, √0.2777] = [0.471, 0.527] m. So "0.50 m" really means "0.47–0.53 m at 95%." That 0.45-vs-0.50 bench comparison from single runs was noise.
Step 4 — runs to halve the CI. Width scales as 1/√M, so halving needs 4× the runs: M = 200. Quartering needs M = 800. Budget accordingly — or pair seeds to cut the variance of the comparison instead.
Step 5 — the covariance diagnosis. Empirical variance 0.36 versus predicted 0.09. Ratio = 0.36/0.09 = 4.0, so actual errors have twice the claimed σ (√4 = 2). The filter is overconfident by a factor of 2 in standard deviation on this axis — equivalently, this axis's NEES contribution runs ~4× its expected value. Fix direction: this axis's process/measurement noise model is too optimistic.
From scratch — the campaign harness computing bands, empirical covariance, and the diagnosis:
python — from scratch: campaign harness import numpy as np # errors: (M, T, n) error tensor from M seed-controlled runs def campaign_report(errors, t_check): M, T, n = errors.shape norms = np.linalg.norm(errors, axis=2) # (M, T) band = np.percentile(norms, [5,50,95], axis=0) # RMSE-over-time band # empirical covariance at t_check, by hand: subtract mean, outer products E = errors[:, t_check, :] # (M, n) mu = E.mean(axis=0) C = np.zeros((n, n)) for i in range(M): d = E[i] - mu; C += np.outer(d, d) emp_cov = C / (M - 1) # unbiased sample covariance return band, emp_cov # CI on RMSE from per-run squared errors sq (M,) sq = np.array([0.25]*50); sq_std, M = 0.10, 50 se = sq_std / np.sqrt(M) ci_mse = np.array([sq.mean() - 1.96*se, sq.mean() + 1.96*se]) print(se, ci_mse, np.sqrt(ci_mse)) # 0.01414 [0.2223 0.2777] -> RMSE [0.4715 0.5270] print(0.36/0.09, (0.36/0.09)**0.5) # variance ratio 4.0, sigma factor 2.0
Library shorthands for the three headline computations:
python — library emp_cov = np.cov(errors_at_t.T) # empirical covariance, one line band = np.percentile(err_norms, [5,50,95], axis=0) # RMSE band over time ci = mse + np.array([-1,1]) * 1.96 * sq_errs.std(ddof=1) / np.sqrt(M)
Build a campaign live below. Append runs and watch the spaghetti condense into a median line with a percentile band. Drag the timestep marker to sweep the consistency overlay along the trajectory; the mismatch slider secretly scales true noise versus filter-assumed noise so you can watch the two ellipses — predicted and empirical — diverge. Toggle paired seeds and see the comparison CI collapse.
Left: individual-run error traces accumulating, condensing into a median + band once M > 10. Right: at the marked timestep, the error cloud with the filter-predicted ellipse (teal dashed) and the empirical ellipse (warm solid), plus the live empirical/predicted variance ratio and the RMSE CI tightening as M grows.
Campaigns give us trustworthy aggregate numbers for filters that output a state. But a whole class of estimators — SLAM and odometry systems — output a trajectory in their own coordinate frame, and comparing trajectories needs its own metrics with their own blind spots. That is ATE and RPE.
Two mapping robots drive the same warehouse loop. Robot 1's trajectory looks jagged but ends exactly where it started; Robot 2's looks silky smooth but ends two meters from its start. Which SLAM system is better? Wrong question — they fail in different metrics, and a leaderboard that reports only one will crown the wrong robot.
Trajectory evaluation is special. A SLAM or odometry estimate lives in its own coordinate frame, often with an arbitrary origin (and, for monocular, an arbitrary scale). Comparing raw coordinates to ground truth would conflate frame mismatch with actual error — so every trajectory metric begins with alignment.
Umeyama alignment, in words: find the single rigid transform — rotation plus translation, optionally plus scale — that best overlays the estimated trajectory onto ground truth in the least-squares sense. Mechanically: center both point sets, compute the cross-covariance between them, take its SVD, and read the optimal rotation off the singular vectors (with a determinant fix to forbid reflections); scale is a ratio of variances, translation re-centers. We give it in code, not derivation — the point is what it removes: any error expressible as one global transform is forgiven.
ATE (Absolute Trajectory Error): after alignment, the RMSE of per-pose position differences. It measures global consistency — is the trajectory, as a shape in the world, where it should be? Loop-closure failures, drift, and map warp all show up here.
RPE (Relative Pose Error): compare motion increments — over each step (or each fixed distance Δ), the difference between the estimated relative motion and the true relative motion. It measures local accuracy: per-step odometry quality, jitter, short-horizon drift rate. A global transform cancels in the differencing, so RPE needs no alignment at all.
Here are the two blindness theorems, which the worked example proves numerically. A zigzag estimate (jittery per-step motion that averages out) has small ATE but large RPE — ATE is blind to local jitter that cancels. A smooth-drift estimate (a tiny consistent per-step bias that accumulates) has small RPE but growing ATE — RPE is blind to slow accumulation. Robot 1 and Robot 2 from the hook are exactly these two cases. You must report both.
Drift rate is the practitioner's headline: terminal (or windowed) error divided by distance traveled, quoted as a percentage — e.g. 0.4 m of drift over a 40 m loop = 0.4/40 = 1% of distance. Drift normalizes away trajectory length so systems evaluated on different courses become comparable; the KITTI odometry leaderboard's translational error is exactly this, averaged over fixed segment lengths, and it is the number VIO papers put in their abstract.
Protocol details that silently change results: aligning with scale (monocular, Sim(3)) versus without (stereo/lidar, SE(3)) — a scale fit flatters a monocular system; the number of alignment poses (first-pose-only versus full-trajectory alignment give different ATEs); and the RPE step size Δ (per-frame RPE measures jitter, per-10 m RPE measures drift). Any comparison must pin these choices. The evo package does, which is why everyone uses it.
And a caveat: trajectory metrics evaluate the pose output, not the map. A system can have great ATE and a garbage map. Full SLAM eval adds map metrics (not owned here) — cross-link classical-slam and modern-slam for the mapping side.
Now the numbers, already aligned (the identity transform is optimal here). Ground truth is 5 poses along a straight line: (0,0), (1,0), (2,0), (3,0), (4,0). Two estimates:
Step 1 — ATE for Z. Per-pose |y| errors are [0, 0.2, 0.2, 0.2, 0.2]; squared [0, 0.04, 0.04, 0.04, 0.04]; mean = 0.16/5 = 0.032; ATE = √0.032 = 0.179 m.
Step 2 — RPE for Z. True steps are all (1, 0). Estimated steps: (1, +0.2), (1, −0.4), (1, +0.4), (1, −0.4). Step errors in y: [0.2, −0.4, 0.4, −0.4]; squared [0.04, 0.16, 0.16, 0.16]; mean = 0.52/4 = 0.13; RPE = √0.13 = 0.361 m per step.
Step 3 — Z verdict. RPE (0.361) is double ATE (0.179). The jitter is invisible globally because it cancels, but every individual motion estimate is poor — terrible for velocity control despite decent map placement.
Step 4 — ATE for D. Per-pose |y| errors [0, 0.1, 0.2, 0.3, 0.4]; squared [0, 0.01, 0.04, 0.09, 0.16]; mean = 0.30/5 = 0.06; ATE = √0.06 = 0.245 m.
Step 5 — RPE for D. Estimated steps are all (1, 0.1); step error (0, 0.1) every time; RPE = 0.1 m per step — small and constant. Locally, D looks nearly perfect.
Step 6 — D verdict and drift. ATE (0.245) is 2.4× its RPE (0.1) and grows with trajectory length — the signature of accumulating drift. Drift rate: terminal error 0.4 m over 4 m traveled = 10% of distance. (Real VIO runs 0.1–1%; 10% here is pedagogical scale.)
Step 7 — the cross. Z has smaller ATE than D (0.179 < 0.245) but larger RPE (0.361 > 0.1). Rank by ATE alone and you crown the jittery system; rank by RPE alone and you crown the drifting one. Report both, always.
From scratch, matching the hand steps exactly, plus a compact Umeyama:
python — from scratch import numpy as np def ate(gt, est): # both (T, 2), already aligned return np.sqrt(np.mean(np.sum((est - gt)**2, axis=1))) def rpe(gt, est, delta=1): de = est[delta:] - est[:-delta]; dg = gt[delta:] - gt[:-delta] return np.sqrt(np.mean(np.sum((de - dg)**2, axis=1))) def umeyama(gt, est): # find R, s, t overlaying est onto gt mu_g, mu_e = gt.mean(0), est.mean(0) Sig = (gt - mu_g).T @ (est - mu_e) / len(gt) U, D, Vt = np.linalg.svd(Sig) Sfix = np.eye(2) if np.linalg.det(U) * np.linalg.det(Vt) < 0: Sfix[-1,-1] = -1 # no reflections R = U @ Sfix @ Vt return R, mu_g - R @ mu_e gt = np.array([[0,0],[1,0],[2,0],[3,0],[4,0]], float) Z = np.array([[0,0],[1,.2],[2,-.2],[3,.2],[4,-.2]]) D = np.array([[0,0],[1,.1],[2,.2],[3,.3],[4,.4]]) print(ate(gt,Z), rpe(gt,Z)) # 0.17889 0.36056 print(ate(gt,D), rpe(gt,D), np.linalg.norm(D[-1]-gt[-1])/4*100) # 0.24495 0.1 10.0% drift
The community standard is the evo toolkit — it pins the protocol so competitors and CI run identical evaluations:
bash / python — library # evo CLI: aligned ATE and RPE with pinned flags evo_ape tum gt.txt est.txt -va --align # ATE, SE(3) alignment evo_rpe tum gt.txt est.txt --delta 10 --delta_unit m # per-10m RPE (drift) # scipy for the rotation piece: Rotation.align_vectors(gt, est)
The X-ray below lets you morph the estimate and watch the two gauges disagree. Zigzag mode surges the RPE bar while ATE barely moves; drift mode grows ATE along the run while RPE stays flat. Hit "Align" to watch Umeyama snap the raw estimate onto truth and see what alignment forgives.
Ground-truth loop (teal), estimate (warm). Two live gauges — ATE (global) and RPE (local) — recompute as you morph the estimate. Zigzag surges RPE; drift grows ATE. The readout prints drift as % of distance.
ATE and RPE grade a continuous trajectory. But the ultimate question about a robot is usually binary: did it complete the task? That flips us into a completely different statistical regime — small-sample binomial evaluation, where a "success rate" is not a number at all. That is Chapter 7.
A robot-manipulation paper announces "85% success." It ran 20 trials. Another lab reports "75%" from 200 trials. Which robot would you bet on? The statistics say: you genuinely cannot tell — and that inability, hidden behind confident percentages, is the quiet scandal of robotics evaluation.
Robot eval is binomial-statistics-poor. Each real-world trial costs minutes of setup, hardware wear, and human supervision, so n = 10–50 is the norm in manipulation papers (simulation gets thousands; reality does not transfer freely). Small n means the success rate is a coarse estimate — and the field's habit of reporting it bare, without intervals, is how overclaiming happens.
Make the binomial spread visceral. A truly 75%-success robot, run for 20 trials, will produce anywhere from 11 to 18 successes in the central 95% of experiments — observed rates from 55% to 90% from the same robot. Any comparison of two 20-trial numbers within ~20 points of each other is uninformative.
Why not the textbook interval, p ± 1.96√(p(1−p)/n)? At small n and extreme p it misbehaves. At 20/20 it gives exactly [100%, 100%] — claiming certainty from 20 trials — and it can spill below 0% or above 100%. eval-statistics owns interval theory; here we import its recommendation for the binomial case: the Wilson score interval, which stays inside [0,1] and has honest coverage at small n.
Wilson works by shrinking the observed rate toward 50% by an amount that grows as n shrinks, then widening. Let us compute it by hand for 17/20 = 85%, with z = 1.96.
Step 1 — setup. p = 0.85. z² = 3.8416. z²/n = 3.8416/20 = 0.19208.
Step 2 — denominator. 1 + z²/n = 1.19208.
Step 3 — shrunken center. (p + z²/(2n)) / denom = (0.85 + 0.09604) / 1.19208 = 0.94604 / 1.19208 = 0.7936. Note it moved from 85% toward 50% — that is the small-n correction.
Step 4 — half-width.
= 1.96 × √0.008776 / 1.19208 = 1.96 × 0.09368 / 1.19208 = 0.18362 / 1.19208 = 0.15403.
Step 5 — the interval. 0.7936 ± 0.15403 = [0.6396, 0.9476] → [64.0%, 94.8%]. The honest headline for "85% success, n=20" spans thirty points. Notice the asymmetry: the interval reaches 21 points down but only 10 up, because uncertainty near the top of the scale is one-sided. To claim "above 80%" with confidence, n must grow to roughly 100+.
Success alone rewards brute force — a robot that reaches the goal by wandering the whole building "succeeds." SPL (Success weighted by Path Length) multiplies each success by shortest-path/actual-path, so inefficient successes count fractionally. It became the embodied-AI navigation standard precisely to kill wander-and-win policies. Compute it for 5 episodes with (shortest, actual) = (10,12), (8,8), (5,20), and two failures:
Step 6 — per-episode terms. success × shortest/max(shortest, actual). Ep1: 1×10/12 = 0.8333. Ep2: 1×8/8 = 1.0. Ep3 (fail): 0. Ep4: 1×5/20 = 0.25. Ep5 (fail): 0.
Step 7 — SPL. (0.8333 + 1.0 + 0 + 0.25 + 0)/5 = 2.0833/5 = 0.4167. Success rate says 60%; SPL says 41.7%. The gap quantifies inefficiency — driven by episode 4's 4×-too-long path counting as only a quarter of a success.
For missions rather than episodes, count interventions — human takeovers per mission, or mean distance/time between them. This is the AV industry's currency (disengagements per 1,000 miles); Tesla's FSD v12/v13 and community trackers argued publicly over exactly this metric in 2024–2025, and Cruise's 2024 shutdown was rooted in how an incident was reported, not just that it happened. Rare-event caution: at one intervention per 100 km, you need thousands of km before the rate estimate has one significant digit — the Poisson analog of the binomial problem above.
There is one more way small-n numbers deceive, and it is not sampling — it is selection. The winner's curse: if you run several noisy 20-trial evaluations and report the best, that maximum is biased high, because you selected for luck. Say a policy's true rate is 75%. A single 20-trial batch has a decent chance of landing at 80% or 85% by luck alone (recall the 55–90% spread). Run five batches and report the best, and you will quite often headline 90%+ — a number the policy cannot reproduce. Selecting the best checkpoint on the same trials used to report it is the same sin wearing a lab coat. The fix is a held-out evaluation set: choose on one set, report on another, never both. The "paper mode" button in the sim below enacts exactly this — it reruns batches until one gets lucky and counts how many honest ones it threw away.
From scratch — Wilson with every intermediate, plus SPL:
python — from scratch import numpy as np def wilson(k, n, z=1.959963985): p = k / n den = 1 + z*z/n # 1.19208 center = (p + z*z/(2*n)) / den # 0.79361 hw = z * np.sqrt(p*(1-p)/n + z*z/(4*n*n)) / den # 0.15403 return center - hw, center + hw # (0.63958, 0.94763) def spl(episodes): # [(success, shortest, actual), ...] terms = [s * short / max(short, act) for s, short, act in episodes] return terms, sum(terms) / len(episodes) print(wilson(17, 20)) # (0.6396, 0.9476) print(spl([(1,10,12),(1,8,8),(0,1,1),(1,5,20),(0,1,1)])) # SPL 0.41667
Library version — statsmodels and scipy both give Wilson directly:
python — library from statsmodels.stats.proportion import proportion_confint print(proportion_confint(17, 20, method='wilson')) # (0.6396, 0.9476) from scipy import stats print(stats.binomtest(17,20).proportion_ci(method='wilson')) # same print(stats.binom.ppf(.025,20,.75), stats.binom.ppf(.975,20,.75)) # 11 to 18
The honest meter below runs animated trials and draws the observed rate with its live Wilson error bar. Watch it tighten slowly as trials accumulate (the √n tax). "Paper mode" dramatizes the winner's curse: it reruns 20-trial batches until one hits 90%+ and stamps "PUBLISH!", tallying how many honest batches it threw away to get there.
Trials append as ticks/crosses. The observed rate is a dot with its Wilson interval as a live error bar; the hidden true rate is a marker you can reveal. Comparison mode shows two robots and a verdict lamp: distinguishable only when intervals separate.
You now own every diagnostic in the book — error metrics, NEES, NIS, the CRLB, campaigns, trajectory metrics, and honest trial statistics. Chapter 8 assembles them into one instrument and hands you the dials. Time to break a filter on purpose, in both directions, and watch the tests catch it.
Every diagnostic in this lesson becomes one instrument here. A 2D constant-velocity Kalman tracker follows a maneuvering target — the exact tracker from kalman-filter, nothing new inside the filter. What is new is that you control four dials: the world's true measurement noise Rtrue and true process noise Qtrue, and the filter's assumed Rf and Qf. Consistency is the study of what happens when assumed diverges from true.
The instrument panel has a live track view with the 95% covariance ellipse riding the estimate; a NEES lane (needs the sim's ground truth) and a NIS lane (computable in deployment), each with per-step dots, a windowed average, and their chi-square bands. A mismatch badge reads out the current regime: HONEST, OVERCONFIDENT, UNDERCONFIDENT, or COMPENSATING.
Experiment 1 — the overconfident filter. Set Rf well below Rtrue (the filter trusts a noisy sensor too much). Predicted: the estimate chases measurement noise, the ellipse shrinks below the actual scatter, and both NEES and NIS averages climb above the upper band. Watch which responds first, and how the windowed average trades detection speed against false alarms — Chapter 3's window lesson, now lived.
Experiment 2 — the underconfident filter. Set Rf above Rtrue. The filter over-smooths, lags maneuvers, the ellipse balloons, and NEES/NIS sink below the lower band. Internalize that consistency violations are two-sided: "conservative" filters fail the test too, and pay for it in tracking lag.
Experiment 3 — Q mismatch, the smug-divergence replay. Set Qf near zero while the target maneuvers. The filter's P collapses (it believes the model perfectly), gates tighten, and when the target turns, innovations blow through the gate. NIS spikes catastrophically while the track sails straight — the exact death mode from Chapter 0, now fully instrumented.
Experiment 4 — compensating errors, the subtle one. Raise both Rf and Qf by the same factor. The Kalman gain depends on their ratio, so the state estimates barely change and RMSE stays flat — but P scales up and NEES/NIS sink low. The moral: error metrics alone cannot even see this axis of tuning; only consistency tests constrain the absolute scale of the noise model.
Let us pin Experiment 1 with arithmetic, in one dimension — the lab's physics in miniature. Prior variance 1.0. The filter assumes Rf = 1.0, but the true measurement variance is Rtrue = 4.0.
Step 1 — gain under the assumed model. K = prior/(prior + Rf) = 1/(1+1) = 0.5.
Step 2 — reported posterior variance (what the filter prints): (1−K)×prior = 0.5×1.0 = 0.5.
Step 3 — actual posterior error variance (the truth): (1−K)²×prior + K²×Rtrue = 0.25×1.0 + 0.25×4.0 = 0.25 + 1.0 = 1.25.
Step 4 — expected NEES. actual/reported = 1.25/0.5 = 2.5, against an honest expectation of 1.0 (n = 1 DOF). The filter is overconfident by 2.5× in variance.
Step 5 — single-step band. For chi-square(1) the 95% band is [0.00098, 5.02]. One step at NEES ~2.5 would not trip it — single samples are weak evidence, as Chapter 2 warned.
Step 6 — 50-run average band. [chi2.ppf(0.025,50)/50, chi2.ppf(0.975,50)/50] = [0.647, 1.428]. The expected average 2.5 sits far above 1.428 — the campaign flags the lie decisively, even though most individual steps look innocent.
Step 7 — the twist. Had the filter been honest (Rf = 4), K = 1/(1+4) = 0.2, and reported = actual = (0.8)²×1 + (0.2)²×4 = 0.64 + 0.16 = 0.8. Note 0.8 > 0.5: the honest filter reports more uncertainty yet achieves less actual error variance (0.8 < 1.25). Overconfidence didn't just misreport — it made the estimate objectively worse, because the gain was wrong. Lying about your uncertainty corrupts the very fusion that produces the estimate.
python — the lab engine, portable to a notebook import numpy as np from scipy import stats # Mismatched 1D fusion step (Experiment 1 in miniature) prior, R_f, R_true = 1.0, 1.0, 4.0 K = prior / (prior + R_f) reported = (1 - K) * prior actual = (1 - K)**2 * prior + K**2 * R_true print(K, reported, actual, actual / reported) # 0.5 0.5 1.25 2.5 # honest choice achieves lower actual variance Kh = prior / (prior + R_true) print(Kh, (1-Kh)**2*prior + Kh**2*R_true) # 0.2 0.8 (< 1.25) # chi-square 1-DOF bands print(stats.chi2.ppf([.025,.975], 1)) # [0.00098, 5.0239] single step print(stats.chi2.ppf([.025,.975], 50) / 50) # [0.6471, 1.4284] 50-run average
The library path is the same idea with a real tracker — filterpy.kalman.KalmanFilter for the filter, scipy.stats.chi2 for the bands. The consistency logging stays ~5 lines of your own numpy either way; that is the whole point — consistency testing is cheap, and there is no excuse not to run it.
Here is the full instrument. Load a preset, or grab the four dials and mismatch the filter yourself. The mismatch badge names the regime; the NEES and NIS lanes catch it. Then flip "Hide truth" to blank the NEES lane — simulating deployment, where only NIS survives.
Top: 2D track (truth, measurements, estimate + 95% ellipse). Below: synchronized NEES and NIS lanes with per-step dots, windowed averages, chi-square bands, and alarm lamps. Mismatch the four dials and read the regime badge. Hide truth blanks NEES — only NIS survives deployment.
The take-home is a tuning workflow the lab teaches by making you feel each failure: (1) fix bias first (innovation mean per channel); (2) fix whiteness with Q (autocorrelation); (3) fix scale with R until windowed NIS sits inside its band; (4) confirm with a NEES Monte Carlo campaign in sim; (5) freeze and regression-gate the consistency dashboard — cross-link regression-testing-ml for step 5's CI machinery.
You have broken a filter every way it can break and watched the instruments catch each one. The last two chapters turn that mastery into two things you can use tomorrow: a drilled interview arsenal, and a map of everywhere this toolkit travels.
Every question in this chapter has been asked, in some form, in perception and estimation interviews at AV, drone, and robotics companies. You now have every tool to answer them — this chapter is the rep work.
Interviewers layer these questions. They start with a metric definition (screening), then inject a contradiction — RMSE good but planner crashing, the Chapter 0 scenario — to test whether you reach for consistency, then push to protocol design (how many runs, which metrics, what bounds) to test staff-level judgment.
Recognize the four archetypes, because every question is one of them: (1) two systems, same metric, different behavior — the answer is always a second metric axis (consistency, RPE vs ATE, SPL vs success); (2) evaluate without ground truth — the answer is innovation-based statistics and their limits; (3) is this improvement real? — the answer is intervals, paired seeds, and n; (4) design the eval — the answer is a protocol: scenarios, randomization, metrics with bounds, pre-registration.
Red flags interviewers listen for: quoting success rates or RMSEs without n or intervals; proposing to tune Q/R "until RMSE is minimized" (straight into the compensating-errors trap); treating NIS-pass as full certification (unobservable states); claiming the CRLB is reachable by a better algorithm.
Calibration language that signals seniority: "necessary but not sufficient," "that metric is blind to X," "the interval, not the point, is the result," "what does the downstream consumer read from this estimator?" Each phrase encodes one of this lesson's chapters.
Let us run a timed micro-drill first. "Our 2-state tracker's average NEES over 100 Monte Carlo runs is 0.4. The bounds sheet says the 95% band for a 100-run average is [1.63, 2.41]. In 60 seconds: diagnosis, consequence, and fix direction."
Locate. 0.4 is far below the lower bound 1.63 — a decisive violation, not noise (the band already accounts for 100-run sampling variability).
Diagnose. Average NEES below band = errors much smaller than P claims = underconfident. Expected NEES for 2 DOF is 2; observed 0.4 means the claimed variance is ~2/0.4 = 5× the actual — P is inflated roughly 5×.
Consequence. The filter converges slower than it could, validation gates are ~√5 = 2.2× wider than needed (letting in clutter and outliers), and downstream consumers over-buffer. In data-association-heavy tracking, wider gates cause more wrong associations, not fewer.
Fix direction. The assumed noise (Q and/or R) is too large in absolute scale. Because RMSE cannot see absolute scale (the compensating-errors trap), shrink both toward honest using NIS scale on real data, re-run the campaign, and require the average to enter [1.63, 2.41].
The assembled 60-second answer: "Underconfident by about 5× in variance — P is inflated. It's outside the band, so it's real, not sampling noise. It costs us convergence speed and doubles our gate widths, which invites misassociation. I'd rescale the noise model down against windowed NIS on log data and re-certify with the same 100-run campaign."
python — the bounds sheet every candidate should be able to generate import pandas as pd from scipy import stats def bounds_sheet(n, run_counts=(1,10,50,100)): # n = DOF (state or meas dim) return {N: stats.chi2.ppf([.025,.975], N*n) / N for N in run_counts} print(bounds_sheet(2)[100]) # [1.6273, 2.4106] <- the drill's band print(2/0.4, (2/0.4)**0.5) # 5.0 variance inflation, 2.24 gate-width factor
Now the arsenal. Attempt each question aloud before opening the answer; the follow-ups are what a strong interviewer actually asks next.
RMSE only grades the point estimates; the filters also output covariance, and downstream consumers (gates, fusion, planners) act on it, so I evaluate consistency next. In simulation with ground truth I run a Monte Carlo NEES campaign — paired seeds, multiple scenario families — and require the run-averaged NEES to sit inside its chi-square band (bounds scaled by run count and state dimension). On real logs without truth I check windowed NIS against its band plus innovation whiteness and per-channel innovation means. Promotion rule: no scenario family shows persistent band violation in either direction, error metrics don't regress beyond their paired-comparison CI, and ellipse containment is near nominal. If one filter is accurate but overconfident, it loses — an honest P is a functional requirement, not a nicety.
Follow-up: Which failure is worse to ship, overconfidence or underconfidence — and is your answer application-dependent?
NEES is the error quadratic form eᵀP⁻¹e. If the filter is consistent, whitening by the covariance turns it into a sum of squared independent standard normals, so NEES is chi-square with DOF equal to the state dimension, expectation equal to that dimension. For 2 DOF the single-sample 95% band is [0.051, 7.38]. Averaging N runs sums N chi-square(2) into chi-square(2N), so the average has band chi2.ppf(α, 2N)/N: for 50 runs, [1.48, 2.59]. Averaging is what gives the test power — single samples outside the band happen 5% of the time by construction. I'd also note a bias makes NEES noncentral chi-square, so persistent inflation can be bias rather than covariance scale, distinguishable by plotting the raw error mean.
Follow-up: Your average NEES is 2.4 for the first half of runs and 6.0 during aggressive maneuvers. What do you conclude?
Everything runs on innovations, which the filter computes anyway. Three monitors per sensor channel: windowed average NIS against chi-square bounds scaled by window length and measurement dimension (variance-scale faults); innovation autocorrelation (time-correlated innovations mean dynamics mismatch, typically Q too small); and innovation-mean drift per channel (bias — miscalibration, misalignment, map staleness). Alarms feed graduated responses: reweight or drop the sensor, widen margins, re-localize. Window length is a tuned latency/false-alarm tradeoff. The blind spot: NIS certifies only the measurement-observed subspace. Weakly-observable states — yaw with position-only sensing, monocular scale, slow IMU biases — can drift with innocent innovations. Those need design-time answers: observability analysis, a sensor that pins the weak state, and state-level NEES certification in simulation before deployment.
Follow-up: GPS jamming starts affecting 5% of your delivery routes. Which of your three monitors fires first, and what does the response ladder look like?
For the PM: every sensor stream contains a fixed amount of information about the state, and no algorithm — present or future — can produce an unbiased estimate more precise than that information allows; the CRLB is that limit, and it is computable. Procedure: compute the bound for the current suite (recursively, in sim, for dynamic systems), run the current filter's Monte Carlo RMSE against it, read the gap. Near the floor (within 10–20%): algorithm work is spent; only more information moves the floor, and the better IMU's noise density plugs into a new, lower floor you can quote before buying. Large gap: the algorithm is leaving purchased information on the table and engineering time is cheaper. Caveats: the bound assumes unbiasedness and may be unreachable in nonlinear regimes, so "near the floor" is the practical criterion; and correlated error sources mean two identical sensors add less than double information, so diversity beats duplication.
Follow-up: The vendor's new IMU halves the noise density. Exactly how much does your position CRLB improve, and why is it not half?
Structure before samples. Define scenario families spanning the flight envelope — hover, aggressive trajectories, wind gusts, GPS-degraded segments, poor initialization — say 8 families, pre-registered with fixed seed ranges so no post-hoc cherry-picking is possible. Use paired seeds: both filters experience identical noise realizations, so per-seed differences remove shared randomness and 25 pairs per family gives tight comparison intervals. Randomize what nature randomizes — noise seeds, initial errors drawn from a specified real-world distribution (never from the filter's own P, which is circular), disturbance profiles. Log full traces. Report per family: RMSE-over-time bands, run-averaged NEES against chi-square bounds, empirical-vs-predicted ellipses, and the paired RMSE-difference CI. Promotion criteria declared before running: new filter regresses in no family beyond the CI, and consistency bounds hold everywhere the legacy filter held them. The whole campaign becomes the CI regression suite afterward.
Follow-up: Your new EKF wins 7 of 8 families but shows average NEES of 4.1 in the wind-gust family. Ship, block, or investigate?
Alignment first: SE(3) or Sim(3)? Fitting scale flatters monocular systems and is meaningless for lidar; the requirement must pin one. Aligned over the full trajectory or the first pose only — full-trajectory alignment absorbs drift into the transform and can halve the number. Which poses: keyframes or all frames, and is the timestamp-association tolerance stated? Then representativeness: how many sequences, what environments, and is 12 cm the mean, median, or best run — ATE is heavy-tailed across environments. Then the missing metric: ATE is blind to local jitter (the zigzag pathology), so I require RPE at per-frame and per-10 m windows plus drift rate. Finally reproducibility: pinned tool (evo with stated flags) so competitors and CI run the identical protocol? A 12 cm claim that survives those questions is a result; one that doesn't is a choice of flags.
Follow-up: The team says loop closure makes their per-frame RPE look worse than the odometry baseline. Is that expected, and how do you compare fairly?
Seventeen of twenty against fourteen of twenty. The Wilson 95% intervals are roughly [64%, 95%] and [48%, 85%] — overlapping across almost their entire ranges, so the 15-point headline gap is well within joint sampling noise; a two-proportion test at these n's is nowhere near significance. The claim as stated is not supported. I'd also probe protocol: were the 20 trials used to select the best checkpoint (winner's curse biases the reported max upward), are retries/resets logged, are the tasks identical and pre-registered, are per-task breakdowns shown or only the favorable aggregate? Constructive ask: report intervals with points, run paired trials on identical task instances (pairing sharpens the comparison as paired seeds do in sim), and either increase n until intervals separate or soften to "comparable, with a favorable point estimate." Small-n overclaiming is the field's reproducibility crisis in miniature.
Follow-up: How many trials per method for a true 85-vs-70 gap to be reliably detectable, and what cheaper design change gets there faster?
The Kalman gain — and therefore the entire estimate sequence — depends on Q and R only through their relative balance: scale both by the same factor and the gain, estimates, and RMSE are unchanged, while reported P scales with the factor. So RMSE constrains a one-dimensional family of (Q, R) pairs and says nothing about which member is honest; a team minimizing RMSE can land anywhere on that ridge. Consistency statistics close the loop because they compare errors against P's absolute scale: NEES (or NIS on real data) reads high when the pair is scaled too small and low when too large, uniquely selecting the honest scale. Workflow: set the ratio for good tracking via innovation whiteness, then set absolute scale via windowed NIS centering, then certify with a NEES campaign. Any tuning story that never mentions a consistency statistic has left P undetermined — and P is what downstream systems consume.
Follow-up: Adaptive filters estimate Q and R online from innovations. What new evaluation problem does that introduce?
The network made the same distributional claim a filter makes, so it gets the same interrogation. On a held-out labeled set, compute pose errors and form the NEES-style normalized quadratic against the predicted covariance per sample: the distribution should match chi-square with DOF equal to the pose dimension, and coverage should be nominal — the 95% ellipsoid containing truth 95% of the time, checked per error regime (object class, occlusion, distance), not just in aggregate, because learned uncertainty is notoriously miscalibrated exactly where errors are largest. If miscalibrated, recalibrate (temperature-style scaling on the covariance, or conformal methods) before fusion — feeding an overconfident covariance into a Kalman-style node poisons the fused estimate and its gates exactly as Chapter 8's arithmetic shows. Then monitor fusion innovations from this "sensor" via windowed NIS in deployment. The mindset transfer is the point: predicted covariance is a claim, and claims get tested.
Follow-up: The network's coverage is 95% overall but 60% on shiny objects. Is a global recalibration acceptable?
Drift rate is accumulated error normalized by distance traveled — terminal or windowed translation error divided by path length, quoted as a percentage (and deg/m for rotation). It's preferred because raw ATE grows with trajectory length, making numbers from different courses incomparable, while drift rate is roughly length-invariant for odometry whose error accumulates linearly-ish; KITTI's translational error is exactly this, averaged over fixed segment lengths. Failure modes: it presumes accumulation — a system with bounded error (loop closures, absolute fixes) shows near-zero long-run "drift" while still having meter-scale local excursions, so drift flatters SLAM against pure odometry; it hides the error's time profile (a cliff at one bad corridor averages into a gentle slope); rotation drift converts to position error quadratically with distance, so the translation number understates yaw problems; and it inherits every protocol sensitivity of Chapter 6. So: headline with drift rate, but always alongside ATE, windowed RPE, and per-segment worst cases.
Follow-up: A competitor quotes 0.2% drift on highway datasets; you measure 0.8% in parking garages. Same system. What is happening, and which number goes in the spec sheet?
Finally, three debate prompts — questions with no single right answer, for practicing the judgment calls interviewers use to separate strong candidates from great ones. State both defensible positions before you pick.
You can now answer any question in this space by reflex — the system made a distributional claim; test the claim; ask who consumes it. The final chapter shows where that reflex travels: to every estimator you will ever ship.
You came in knowing how a Kalman filter works. You leave knowing how to interrogate one under oath. Here is where each instrument goes next.
Back to the prereq with new eyes: revisit kalman-filter's tracking demo — you can now say precisely what "optimal" means (efficient: P achieves the CRLB for linear-Gaussian systems) and what to check when reality is neither linear nor Gaussian (NEES campaigns on the EKF/UKF, where linearization error is exactly what breaks consistency first).
Into sensor fusion: fusion architectures decide which sensors to fuse; this lesson's Fisher-information-adds argument (Chapter 4) is the quantitative backbone of that decision, and the innovation-based health checks (Chapter 3) are how fusion stacks detect a lying sensor at runtime — the 2024 GPS-jamming surge made windowed NIS monitors front-page engineering.
Into SLAM: the systems ATE/RPE (Chapter 6) were built to grade, whose factor-graph marginal covariances deserve the same NEES scrutiny — an overconfident SLAM posterior corrupts every loop-closure decision.
Into robot learning: policy evaluation (success rates, SPL, interventions — Chapter 7) is the front line of the sim-to-real debate in every 2024–2026 manipulation paper; the Wilson-interval discipline is what separates credible claims from demo reels.
Sideways into the family: eval-statistics owns the CI/hypothesis-test machinery we imported; genai-eval applies the same "small-n honesty" mathematics to pass@k and judge agreement; regression-testing-ml turns Chapter 5's campaign and Chapter 8's dashboard into CI gates; metrics-ladder shows how NEES-class engineering metrics roll up into the mission-level numbers executives read.
Here is the whole toolkit as one page. Read it top to bottom: it is the order you apply the instruments.
| Instrument | What it answers | When to use it |
|---|---|---|
| Error metrics (Ch1) | How far is the estimate from truth? | Always, sliced by axis / state / time. Never a single scalar for safety. |
| NEES (Ch2) | Does the error match the claimed P? | When ground truth exists (sim, mocap, RTK). Average over runs for power. |
| NIS + whiteness (Ch3) | Does the innovation stream match its predicted S? | Deployment health check — no truth needed. Windowed, with alarms. |
| CRLB (Ch4) | Is the gap the algorithm's fault or the sensor's? | Deciding tune-vs-buy; proving a requirement infeasible. |
| Monte Carlo (Ch5) | Is the difference real, and how confident? | Comparing filters. Paired seeds, pre-registered scenarios, CIs. |
| ATE + RPE + drift (Ch6) | Global shape vs local motion quality? | Trajectory systems (SLAM, VIO). Report both; pin the alignment protocol. |
| Wilson + SPL + interventions (Ch7) | Is a small-n task claim honest? | Robot task/mission eval. The interval is the result, not the point. |
And the formula cheat sheet — every key statistic in the lesson, with what each symbol means and its acceptance rule:
| Formula | Symbols | Acceptance / use |
|---|---|---|
| NEES = eᵀP⁻¹e | e = state error, P = claimed covariance, n = state dim | ~ χ²(n); healthy avg ≈ n; N-run band chi2.ppf(α, Nn)/N |
| NIS = νᵀS⁻¹ν | ν = z−Hxpred, S = HPHᵀ+R, m = meas dim | ~ χ²(m); deployed monitor; gate at chi2.ppf(0.99, m) |
| CRLB = 1/I, I = N/σ² | I = Fisher info, efficiency = CRLB/var | Floor on unbiased variance; efficiency 1.0 = optimal |
| SE = s/√M | s = across-run std, M = runs | CI width ∝ 1/√M; halving CI = 4× runs |
| ATE = RMSE of aligned poses | after Umeyama alignment | Global consistency; blind to canceling jitter |
| RPE = RMSE of relative-motion error | over step Δ; no alignment | Local accuracy; blind to slow accumulation |
| Wilson center ± half-width | p = k/n, z = 1.96 | Small-n binomial CI; stays in [0,1]; asymmetric near edges |
| SPL = (1/N)∑ success×shortest/actual | path efficiency | Penalizes wander-and-win; embodied-nav standard |
Let us close with a synthesis drill that chains three chapters into one product decision. A delivery robot's VIO drifts at 1% of distance (Ch6). Missions are 200 m. The spec requires 95% of missions to end within 3 m of the goal. Terminal drift error is approximately Gaussian with mean 2 m and standard deviation 0.5 m (from a 60-run campaign, Ch5). Does it meet spec?
Step 1 — expected drift. 1% of 200 m = 0.01×200 = 2.0 m mean.
Step 2 — standardize. The spec asks P(error < 3) ≥ 0.95. z = (3−2)/0.5 = 2.0.
Step 3 — probability. P(error < 3) = Φ(2.0) = 0.9772. 97.7% ≥ 95% — the spec passes, with margin.
Step 4 — stress the margin (the staff move). If drift crept to 1.25%/m, mean = 2.5 m, z = (3−2.5)/0.5 = 1.0, P = 0.841 — FAILS. The pass is fragile to a 25% drift regression; that sensitivity, not the pass, is the finding to report — and exactly what a regression gate should watch.
python — mission-margin calculator from scipy import stats drift_pct, L, sd, radius = 0.01, 200, 0.5, 3 mean = drift_pct * L # 2.0 m print(mean, stats.norm.cdf((radius - mean) / sd)) # 2.0, P(pass)=0.9772 print(stats.norm.cdf((radius - 2.5) / sd)) # stressed: 0.8413 -> fails # drift at which the spec exactly holds: breakeven = (radius - stats.norm.ppf(0.95) * sd) / L * 100 # %/m breakeven
Parting habit: before shipping any estimator, answer three questions in writing — What does it claim (P)? What does it achieve (errors)? Who consumes the claim (gates, planners, humans)? Every incident postmortem in this field is a mismatch among those three. You now have the instruments to catch the mismatch before it ships.
"The first principle is that you must not fool yourself — and you are the easiest person to fool." — Richard Feynman. An estimator's covariance is the easiest thing in the world to fool yourself with. Now you can't.