The Estimation–Learning Merger

Chapter 1: A Filter Is a Computational Graph

To merge filters with learning, we need to see the filter the way a deep-learning framework sees everything: as a computational graph — a sequence of differentiable operations through which gradients can flow. The shocking thing, once you look, is that a Kalman filter already is one. Nobody designed it to be backprop-friendly; it just happens that every step is a smooth function of its inputs.

Recall the one-dimensional Kalman filter — the scalar case, which we'll do entirely by hand so nothing is hidden. The state is a single number x (say, position along a corridor). We carry a belief that is a Gaussian with mean μ and variance P. One full cycle is two steps.

Predict. A motion model says the next state is the old one nudged by control u, with the world adding process noise of variance Q:

Here F is how the state carries forward (F = 1 for "position stays unless moved"), B maps control into state. The hat (̂) marks "predicted, before seeing the measurement." Notice the variance grows — prediction makes you less certain.

Update. A sensor reports z, related to state by H (here H = 1: the sensor reads position directly), corrupted by measurement noise of variance R. We form the innovation y = z − H·μ̂ (how surprised we are), scale it by the Kalman gain K, and correct:

Every symbol there enters through multiplication, addition, or division. There is one division (forming K), and division is differentiable everywhere its denominator is nonzero. So the entire predict–update cycle is a smooth, differentiable map from (μ, P, u, z, and the parameters F, B, H, Q, R) to (new μ, new P). That is the entire conceptual unlock of this chapter: a Kalman filter is a layer.

One cycle, every number shown

Let's run a single cycle with concrete values so the graph is not abstract. Start with prior μ = 0, P = 1. Parameters F = 1, B = 1, control u = 3 (we command a 3-unit move), Q = 2. Sensor H = 1, R = 5, and the measurement comes in at z = 5.

Predict: μ̂ = 1·0 + 1·3 = 3. P̂ = 1·1·1 + 2 = 3. We expected to be at 3, and our uncertainty grew from 1 to 3 because moving is noisy.

Innovation: y = 5 − 1·3 = 2. The sensor says we are 2 units further along than predicted. Innovation covariance: S = 1·3·1 + 5 = 8. Gain: K = 3·1 / 8 = 0.375.

Update: μ = 3 + 0.375·2 = 3.75. P = (1 − 0.375·1)·3 = 0.625·3 = 1.875. The estimate 3.75 sits between prediction (3) and measurement (5), pulled 37.5% of the way toward the sensor. And the new uncertainty 1.875 is smaller than both P̂ = 3 and R = 5 — fusing two noisy numbers always sharpens the belief.

Now the new idea. Suppose ground truth was actually x = 4. Our estimate is 3.75, so the error is 0.25 and the squared loss is L = (3.75 − 4)² = 0.0625. Because the whole cycle is differentiable, we can ask: how would L change if R were slightly different? That derivative ∂L/∂R is a real, computable number — and its sign tells us whether to raise or lower R to fit the data better. Hold that thought; Chapter 2 turns it into learning.

python
# One scalar KF cycle as a pure function — note: NO control flow that
# breaks differentiability. Every line is a smooth op, so autograd works.
def kf_step(mu, P, u, z, F, B, H, Q, R):
    mu_pred = F * mu + B * u          # predict mean
    P_pred  = F * P * F + Q           # predict variance (grows by Q)
    y = z - H * mu_pred               # innovation
    S = H * P_pred * H + R            # innovation variance
    K = P_pred * H / S                # Kalman gain (the one division)
    mu_new = mu_pred + K * y          # corrected mean
    P_new  = (1 - K * H) * P_pred       # corrected variance
    return mu_new, P_new

# Run our hand example — should print 3.75, 1.875
mu, P = kf_step(0, 1, u=3, z=5, F=1, B=1, H=1, Q=2, R=5)
print(mu, P)   # 3.75 1.875  — matches our by-hand arithmetic

Chapter 2: The Differentiable Filter — learning Q and R

Every roboticist has lived this: your filter works in the lab and falls apart in the field, and the fix is to sit there nudging Q and R by trial and error until the trajectory looks right. Those two matrices encode "how much do I trust my model" and "how much do I trust my sensor," and getting them wrong means the filter is either sluggish (ignores good measurements) or twitchy (chases noise). Hand-tuning them is folklore, not engineering.

A differentiable filter replaces that folklore with gradient descent. The idea is exactly the thought we left hanging in Chapter 1: treat Q and R as learnable parameters, run the filter forward over a recorded trajectory for which we know the ground truth, measure how far the filtered estimate lands from truth, and backpropagate that error to update Q and R. Do this over many trajectories and the filter discovers the noise statistics that make it best — no human in the loop.

Formally, we unroll the filter over a sequence of T steps. At each step t the filter outputs μ_t. We have ground truth x_t. The loss is the total squared error along the trajectory:

Because each μ_t depends on μ_t−1, which depends on μ_t−2, and so on, the gradient must flow backward through the whole chain. This is backpropagation through time (BPTT) — the same algorithm used to train RNNs — but applied to the Kalman recursion instead of a learned cell. The framework does it automatically; we just have to make sure R stays positive (a common trick: learn log R and exponentiate, so R = e^ρ is positive for any real ρ).

The gradient, by hand, for one step

Let's actually compute ∂L/∂R for the single cycle from Chapter 1 so the gradient is not magic. We had μ̂ = 3, P̂ = 3, H = 1, z = 5, so y = 2 and S = P̂ + R. The estimate is

With R = 5: μ = 3 + (3/8)·2 = 3.75. The loss against truth x = 4 is L = (μ − 4)². Chain rule: ∂L/∂R = 2(μ − 4) · ∂μ/∂R. First factor: 2(3.75 − 4) = 2(−0.25) = −0.5.

For ∂μ/∂R, differentiate μ = 3 + 6/(3+R) with respect to R: ∂μ/∂R = −6/(3+R)² = −6/64 = −0.09375. Multiply: ∂L/∂R = (−0.5)·(−0.09375) = +0.0469.

The gradient is positive, so to reduce the loss we step R in the negative direction — lower R. That makes sense! Truth (4) is closer to the measurement (5) than to the prediction (3), so we should trust the sensor more, i.e. shrink its noise R and let the gain K grow. The gradient figured that out from one example, with no human intuition required. Run it over thousands of steps and R converges to the value that genuinely fits your sensor.

python
import torch

# A differentiable Kalman filter: Q, R are learnable. We learn log-R so R>0.
log_R = torch.tensor(3.0, requires_grad=True)   # start with a bad guess
log_Q = torch.tensor(0.0, requires_grad=True)
opt = torch.optim.Adam([log_R, log_Q], lr=0.05)

for epoch in range(200):
    mu = torch.tensor(0.0); P = torch.tensor(1.0)
    R = torch.exp(log_R); Q = torch.exp(log_Q)
    loss = 0.0
    for z, u, x_true in trajectory:          # unroll the filter (BPTT)
        mu_p = mu + u;        P_p = P + Q
        y = z - mu_p;         S = P_p + R
        K = P_p / S
        mu = mu_p + K * y;    P = (1 - K) * P_p
        loss = loss + (mu - x_true)**2     # accumulate trajectory error
    opt.zero_grad(); loss.backward(); opt.step()   # grad flows through ALL steps

print(torch.exp(log_R))   # converges to the true sensor variance

That is the whole differentiable-filter recipe. The structure (predict, innovate, gain, update) is frozen Bayesian machinery; only the physically-meaningful scalars Q and R are trained. Crucially, the output is still a calibrated Gaussian — the filter reports a variance P you can trust, which a vanilla regression network cannot give you. We get learning's data-driven tuning and estimation's honest uncertainty.

Chapter 3: Learned Observation Models — a neural net inside the skeleton

So far we kept the filter's machinery classical and only learned the noise scalars. Now we go inside and replace a structural piece with a neural network — but we keep the Bayesian skeleton intact. The piece we replace is the observation model h(x): the function that says "if the state is x, what measurement do I expect?"

Classically h is hand-derived geometry. For a camera, h projects a 3D point through the pinhole equations to a pixel. That works when you can find the 3D point — when there is a crisp corner to track. But what is h when the raw measurement is a full image and the thing you want to estimate is, say, the position of an object you have never written an equation for? You can't hand-derive it. So you learn it.

This is the BackpropKF / differentiable-particle-filter family. A convolutional network takes the image and outputs two things: a pseudo-measurement z (the network's read on where the object is) and a learned, input-dependent measurement noise R. That second output is the quietly revolutionary part. Classical R is a fixed constant you tuned once. Here R is heteroscedastic — it changes per frame, because the network looks at the image and decides how much to trust its own read.

Why is "the network outputs its own uncertainty" such a big deal? Because a plain regression network does not. Ask a vanilla CNN "where is the object?" on an image of pure smoke and it returns a confident, precise, completely wrong coordinate. There is no channel for doubt. By forcing the network to emit R and training it inside the filter, the loss rewards honesty: if the network says "low R" (high confidence) and is then wrong, the filtered estimate is dragged off truth and the loss punishes it. Gradient descent teaches the network to inflate R exactly when its reads are unreliable. This is learned, calibrated uncertainty — and it is the bridge classical filtering needed.

The loss that teaches honesty

To train this, we don't use plain squared error. We use the negative log-likelihood (NLL) of the truth under the predicted Gaussian. For a 1D pseudo-measurement z with predicted noise R, against truth x:

Read the two terms as a negotiation. The second term, (z − x)²/(2R), says "if you were wrong, having claimed small R is expensive" — it pushes R up when errors are large. The first term, ½ ln(2πR), says "but don't just claim infinite R to be safe — that's expensive too" — it pushes R down. The minimum of the two pressures lands exactly at R = (z − x)² in expectation: the network's claimed variance converges to its actual squared error. That is the definition of calibrated.

Let's verify with numbers. Suppose on a class of blurry frames the network's reads are off by about ±3 units (so the true error variance is 9). If the network claims R = 1 (overconfident), the second term is 9/(2·1) = 4.5 per frame — huge. If it claims R = 100 (underconfident), the first term ½ ln(2π·100) = ½ ln(628) ≈ 3.22 — also wasteful. If it claims R = 9 (honest): first term ½ ln(2π·9) = ½ ln(56.5) ≈ 2.02, second term 9/18 = 0.5, total ≈ 2.52 — the lowest. The loss has a basin at the truthful R, so training drives the network there.

python
import torch, torch.nn as nn

class LearnedObs(nn.Module):
    # image -> (pseudo-measurement z, log-variance of that measurement)
    def __init__(self):
        super().__init__()
        self.cnn = nn.Sequential(nn.Conv2d(3,16,3), nn.ReLU(), nn.AdaptiveAvgPool2d(1), nn.Flatten())
        self.head_z   = nn.Linear(16, 1)   # the read
        self.head_lr  = nn.Linear(16, 1)   # log measurement variance (per frame!)
    def forward(self, img):
        f = self.cnn(img)
        return self.head_z(f), self.head_lr(f)

def nll(z, log_R, x_true):                  # the loss that calibrates uncertainty
    R = torch.exp(log_R)
    return 0.5*log_R + (z - x_true)**2 / (2*R)   # up-pressure + down-pressure on R

# Inside the filter, R is no longer a constant — it comes from the image:
z, log_R = obs_net(frame)            # per-frame measurement AND its noise
S = P_pred + torch.exp(log_R)
K = P_pred / S                       # small when the frame is bad -> filter ignores it
mu = mu_pred + K * (z - mu_pred)

Chapter 4: KalmanNet — when the gain itself is learned

Chapters 2 and 3 kept the gain formula K = P̂H^TS⁻¹ sacred and only learned what flows into it. But that formula is only optimal under a strong assumption: that your model is right and your noise is truly Gaussian. In the real world the motion model is approximate (the drone is buffeted by gusts you never modeled) and the noise is fat-tailed. Then the analytic gain is computing the optimal correction for the wrong problem.

KalmanNet takes the boldest step yet: it keeps the predict–update skeleton but replaces the gain computation with a small recurrent network. You no longer need to know Q or R at all. The network looks at the innovation (z − h(x̂)) and its recent history, and directly outputs the gain K to apply. The Bayesian bones — predict the state forward, correct by gain times innovation — remain; the part that required knowing the noise statistics is now learned from data.

Why keep the skeleton at all, instead of going full black-box RNN? Because the skeleton encodes the one thing always true regardless of noise: the corrected state is the prediction plus a gain times the surprise. That structure means the network only has to learn the gain — a small, well-conditioned quantity — rather than re-learn all of state estimation. KalmanNet famously beats the classical Kalman filter precisely in the regime the classical filter was never meant for: model mismatch and non-Gaussian noise, where the analytic K is provably suboptimal.

Where the analytic gain breaks — and the learned one doesn't

Here is the mismatch made concrete. Suppose the true motion has an occasional jump (the drone gets shoved). A classical KF with a small, fixed Q assumes the state moves smoothly, so when a jump happens it treats the resulting large innovation as sensor noise and barely reacts — it lags badly. To fix that you'd raise Q, but then in calm periods the filter becomes twitchy and chases measurement noise. There is no single Q that handles both regimes. The analytic gain is stuck with one trust level.

A learned gain is not stuck. The GRU can recognize the signature of a jump (a large innovation following small ones) and output a big K for that one step — react fast — then return to a small K when calm resumes. It has learned a state-dependent trust policy that no single constant Q can express. That is the precise sense in which learning rescues the classical filter: not by discarding it, but by making the one stubborn knob adaptive.

python
class KalmanNet(nn.Module):
    # Keeps predict/update skeleton; LEARNS the gain from the innovation.
    def __init__(self, dim):
        super().__init__()
        self.gru = nn.GRUCell(2*dim, 64)       # sees innovation + state-change
        self.toK = nn.Linear(64, dim)            # outputs the gain
    def step(self, x, h, z, f_dyn, h_obs):
        x_pred = f_dyn(x)                       # predict (known dynamics)
        y      = z - h_obs(x_pred)              # innovation = surprise
        dx     = x_pred - x                       # how fast things move
        h      = self.gru(torch.cat([y, dx], -1), h)
        K      = self.toK(h)                     # LEARNED gain — no Q, no R, no inverse
        x_new  = x_pred + K * y                    # same update shape as classical KF
        return x_new, h

Chapter 5: Transformers That Implicitly Filter

Chapters 2–4 kept the recursion explicit — there was still a literal predict step and update step in the code. Now comes the strangest part of the merger: a model with no explicit recursion at all can still do filtering. A transformer fed a sequence of noisy observations and trained to output the underlying state learns to perform the estimation in its forward pass. Nobody wrote down predict or update. The filtering emerges.

Recall what a filter fundamentally is: a map from a history of observations z_1:t to a belief about the current state x_t. A transformer's attention mechanism is, by construction, a map from a sequence to a weighted summary of that sequence. Train it on (noisy sequence in, clean state out) pairs and gradient descent discovers which weighting of the past gives the best estimate. That weighting is a learned, soft version of the Bayesian posterior — the model attends more to recent and to low-noise observations, exactly as a Kalman filter weights them by their covariance.

This is the same phenomenon as in-context learning: the transformer is not retrained per sequence: it does the estimation entirely in activations, conditioning on the context it is given. Feed it a longer, cleaner context and its implicit belief sharpens. Feed it junk and it widens. The forward pass is the filter run.

Recall the teacher–student estimator (Ch 6)

This connects directly to the asymmetric setup we met earlier in the course: the teacher–student estimator. A teacher is trained with access to privileged information — in simulation it can see the true state x_t directly, so it learns the ideal mapping with an easy job. But at deployment the robot does not have the true state; it has only raw, noisy observations. So we distill the teacher into a student that sees only z_1:t and must reconstruct what the teacher knew.

What is the student forced to learn in order to match the teacher? It must infer the hidden state from the observation history — it must, in other words, learn to filter. The teacher–student loss is the training signal that turns a generic sequence model into an estimator. The teacher provides the "ground-truth belief" target; the student internalizes the inference. This is why the framing keeps reappearing across the field: privileged-teacher / observation-only-student is the cleanest way to train an implicit filter, because it hands the model the exact posterior it is supposed to approximate.

The practical payoff is enormous: the student inherits priors that a hand-built filter could never encode — it has seen millions of trajectories and knows, for instance, that warehouses have flat floors and that drones don't teleport. It fuses those learned priors with the live observation stream in a single forward pass, no Q or R to tune. The cost is the cost of all learning: it is only as trustworthy as its training distribution, a limitation we confront head-on in Chapter 8.

python
# Teacher–student: distill a privileged estimator into an observation-only one.
teacher = Teacher()      # sees TRUE state x_t in sim -> easy, ideal targets
student = Transformer()  # sees ONLY z_1..z_t -> must infer x_t = learn to filter

for z_seq, x_true_seq in loader:
    with torch.no_grad():
        target = teacher(x_true_seq)        # privileged "belief" target
    pred = student(z_seq)                    # attention over the noisy history
    loss = mse(pred, target)             # student reconstructs what teacher knew
    loss.backward(); opt.step()
# At deployment: student(z_seq) does Kalman-like inference in ONE forward pass.

Chapter 6: Learned Front-Ends on Classical Back-Ends

This is the pattern that ships. Walk into almost any state-of-the-art SLAM or visual-odometry system in 2025 and you find a clean division of labor: a learned front-end that does perception, feeding a classical back-end that does optimization. The two halves of the course, literally bolted together — and it is the most reliable architecture we have.

Start with the failure that motivates it. Classical SLAM front-ends rely on hand-crafted features — corners, blobs (think ORB, SIFT). They work beautifully on textured scenes and collapse on the things real robots actually face: a blank warehouse wall, a glass door, a glossy floor reflecting the lights, a foggy corridor. No corners means no features means no constraints means the whole geometric pipeline starves. This is the drone from Chapter 0.

A foundation depth/stereo model — FoundationStereo and its kin — does not need corners. Trained on enormous, diverse datasets, it produces dense metric depth for every pixel, reading subtle shading and learned priors to estimate distance even on a textureless wall. Where the hand-crafted front-end returned "nothing here," the learned front-end returns a full depth map with a per-pixel confidence. Perception is solved by learning, because perception is exactly where data-driven priors dominate.

But you do not hand that dense depth to a neural network and ask it to output the trajectory. Instead you feed it into a classical geometric back-end: a factor graph or bundle-adjustment problem that takes all the depth/correspondence measurements across all frames and finds the single set of camera poses and 3D points that is most consistent with all of them at once. This is the same machinery from the SLAM chapters — nonlinear least squares on a factor graph, solved on the manifold.

Frozen vs trained, and where the uncertainty goes

Note what is learned and what is not. The front-end is a frozen foundation model — you do not train it on your robot; you inherit it. The back-end has zero learned parameters — it is pure optimization. So the system as a whole has a sharp seam: learning ends where geometry begins. That seam is a feature. It means you can swap in a better depth model next year without touching the back-end, and you can prove things about the back-end (convergence, optimality under Gaussian noise) without reasoning about a black box.

The confidence map is the glue. FoundationStereo emits, per pixel, how much to trust its depth — and the back-end uses that as the measurement weight (the inverse of R, exactly the heteroscedastic idea from Chapter 3, now spatial and dense). A low-confidence pixel on a reflective surface contributes a weak constraint; a high-confidence pixel on a matte wall contributes a strong one. The geometric optimizer down-weights the learned model's uncertain reads automatically. And because the back-end checks global consistency, it can catch a confidently-wrong depth estimate: if one frame's points violate what every other frame agrees on, the residual spikes and a robust cost (or a chi-squared gate, Chapter 8) rejects it. The geometry is a safety net under the learning.

python
# Learned front-end -> classical back-end (the pattern that ships).
depth, conf = foundation_stereo(left, right)   # FROZEN foundation model: dense metric depth + confidence
pts3d = unproject(depth, K_intrinsics)         # per-pixel 3D points

graph = FactorGraph()                          # classical back-end: NO learned params
for i, (p, c) in enumerate(zip(pts3d, conf)):
    graph.add(ProjectionFactor(pose, landmark[i], p,
                              weight=c))           # confidence = inverse measurement noise
graph.add(LoopClosureFactor(...))               # global consistency the learner can't give
poses, map = graph.optimize()                   # nonlinear least squares on the manifold

Chapter 7: The Hybrid Estimator Playground

Here is where every thread of this lesson comes together in one machine you can break. A robot flies a looping path through a 2D world. There is a degraded zone (the red circle — smoke and a blank wall) where measurements get noisy and drop out. You choose the sensor harshness and, crucially, you choose the estimator stack. Watch how each design from the previous chapters survives — or doesn't — when you turn the world against it.

Four stacks, each one a chapter:

Things to try, each one a lesson made tangible. (1) Set degradation high and dropout near zero, then flip between Pure KF and Learned R: the Pure KF chases the noisy in-zone measurements (twitchy); Learned R inflates its covariance and rides the motion model through (smooth). That is Chapter 3 in motion. (2) Now crank dropout up: Pure KF's ellipse balloons but it survives on prediction; the Transformer holds if the gap is short and drifts if it's long (no model to coast on). (3) Inject the worst case — high degradation and the hybrid gate: watch the gate reject the wild in-zone measurements entirely (they flash but don't move the estimate), so the Hybrid line stays nailed to truth. That is the geometric safety net of Chapter 6 catching the learner's hallucinations.

Read the RMSE numbers as you slide. In the clean regime all four are close — on easy data, architecture barely matters, which is exactly why benchmarks on clean data are misleading. The differences explode in the degraded regime. The engineer who only ever tested on clean data ships the Pure KF and watches it diverge in the field; the engineer fluent in both halves ships the Hybrid and sleeps at night.

Chapter 8: Which One, When — and how each one fails

A merger is only useful if you know which combination to reach for. The honest truth is there is no universal winner; there is a decision shaped by your data budget, your tolerance for silent failure, and whether you need a guarantee. Let's lay out the tradeoffs without hype, then build the one safety mechanism that makes learned components trustworthy enough to deploy.

Data efficiency vs flexibility. The more classical structure you keep, the less data you need and the more you can prove — but the more you assume. A pure Kalman filter learns from almost nothing because it assumes almost everything. A transformer estimator assumes almost nothing but needs an ocean of trajectories. Every assumption you bake in is data you don't have to collect and a guarantee you get to keep; every assumption you remove is flexibility you gain and a guarantee you surrender. The merger is the art of choosing which assumptions to keep.

Calibration is not free. A dangerous myth: "the network outputs an uncertainty, so it's calibrated." It usually isn't. Learned uncertainties are typically overconfident — the network says σ = 0.1 when its true error spread is 0.3. On in-distribution data the NLL loss (Chapter 3) calibrates reasonably, but the moment you leave the training distribution the reported covariance becomes fiction. The fix is explicit recalibration: hold out data, measure the true error spread per confidence level, and rescale (temperature scaling is the simplest form). Never trust a raw learned σ on a deployed robot without checking it against held-out reality.

Distribution shift is the real enemy. A foundation depth model fails silently off-distribution: it returns a confident, smooth, completely wrong depth map for a scene unlike anything it trained on, with no error flag. This is the single scariest property of learned front-ends. The saving grace of the merger is that the geometric back-end can catch it — not by understanding the image, but by checking consistency. If the learned depth implies a 3D structure that contradicts what every other frame agrees on, the residual explodes, and a statistical test rejects it. Geometry is the lie detector for learning.

The chi-squared gate, worked by hand

That lie detector has a precise form. Each measurement produces an innovation y (how far it is from prediction) with innovation covariance S. The squared Mahalanobis distance d² = y^TS⁻¹y measures the surprise in units of standard deviations. If the measurement is genuine, d² follows a chi-squared distribution with degrees of freedom equal to the measurement dimension. We gate: accept only if d² is below the chi-squared threshold for our chosen confidence.

For a 2D position measurement (2 degrees of freedom), the 95% chi-squared threshold is 5.99. Let's test two measurements against a prediction, with innovation covariance S = diag(0.0008, 0.0008).

Genuine read: innovation y = [0.03, 0.02]. d² = (0.03² + 0.02²)/0.0008 = (0.0009 + 0.0004)/0.0008 = 0.0013/0.0008 = 1.63. Since 1.63 < 5.99, accept — this surprise is normal. Hallucinated read: y = [0.28, 0.0]. d² = (0.0784 + 0)/0.0008 = 98.0. Since 98 >> 5.99, reject — no honest sensor is that surprising; it's an outlier, drop it. This is exactly the gate firing in the Chapter 7 showcase when you pushed degradation up on the Hybrid stack.

The decision table

Chapter 9: The Dissolving Boundary

We started with two tribes and a drone in the smoke. We end with a single, layered design vocabulary that draws freely from both. Look back at the ladder we climbed: keep the filter but learn its noise (Ch 2); keep the skeleton but learn the observation model (Ch 3) or the gain (Ch 4); drop the explicit recursion entirely and let a transformer filter in its forward pass (Ch 5); or split the system cleanly into a learned perception front-end and a geometric back-end (Ch 6). Every rung keeps some classical structure and learns the rest. The wall didn't fall — it dissolved into a gradient.

The thesis of this capstone, now earned: the boundary between the two halves of this course is gone, and the scarcest, most valuable profile in the field is the engineer fluent in both. The estimation specialist who can't train a network ships a filter that starves on a blank wall. The deep-learning specialist who can't write a factor graph ships a perception model that hallucinates with no safety net. The person who can do both builds the hybrid that degrades gracefully — and that person is rare because the two halves were taught in different buildings by different tribes. You just spent this lesson refusing that separation.

Cheat sheet

Where to go next

Solidify the classical half: Kalman Filter, Extended Kalman Filter, Unscented Kalman Filter, Particle Filter, and Bayes Filter. See the geometry it feeds in Modern VIO, Modern SLAM, and the VNAV course chapters VIO, SLAM & Factor Graphs, and Robust SLAM Frontiers. For the learning half, revisit the Transformer and Vision-Language Models to see the front-ends the merger borrows.

“What I cannot create, I do not understand.” — Richard Feynman.
The merger is creation: you now build the estimator instead of choosing a tribe.