The whole arc of modern robot intelligence — from a 1960 Kalman filter to a 2026 foundation model — and a build ladder of runnable code labs that turns "I read about it" into "I built it." What you cannot create, you do not understand.
Everything a robot does, from an Apollo guidance computer to a humanoid folding laundry, is built on one humble admission: sensors lie a little, motion models lie a little, and the truth is hidden. The entire field is about managing doubt with mathematics, and then — once you know where you are — deciding what to do. This guide walks that whole loop, and unlike a textbook, it makes you build each piece.
This is a companion, not a replacement. Each topic below has a deep interactive lesson elsewhere on the site; this guide threads them into one arc and gives you the thing those individual lessons can't: a build ladder — runnable code labs, right in the page, where you implement the core of every idea yourself.
Hold this mental model. A robot's intelligence is a loop, and this guide walks it from its oldest, most mathematical part to its newest, most learned part:
Most newcomers think modern "AI robotics" replaced the math of state estimation with neural networks. It did not. Inside every Figure, Tesla Optimus, Skydio drone, or Boston Dynamics robot, a Kalman-family estimator is still running at hundreds of hertz underneath the learned policy. The frontier didn't delete the classical stack — it sits on top of it. That is why we start where we start, and why the build ladder starts with a six-line Kalman filter.
People don't learn one way, and the same person doesn't learn the same way for every concept. The deep lessons this guide links to each teach through four parallel channels; know which is your default, and switch deliberately — real understanding usually needs at least two.
| Channel | The learner | Learns by |
|---|---|---|
| Intuition | The Storyteller | Analogies, physical stories, "why would anyone invent this?" |
| Visual | The Experimenter | Live demos, sliders, breaking things on purpose |
| Math | The Prover | Definitions and derivations with nothing hidden |
| Code | The Builder | Working implementations small enough to retype |
This guide leans hardest on the fourth channel. The build ladder is for the Builder in you — but every rung opens with the intuition first, because code you don't understand is code you can't debug.
Expertise isn't binary. Each rung you build moves you along this ladder, and the point of building rather than reading is that it carries you two rungs at a time:
Before you climb the build ladder, walk the whole arc once at altitude. Seven stations take you from "what is a belief?" to "robots that learned from the internet." Each station gives you the one load-bearing idea and a door to the deep lesson where you can live in it. Read this in one sitting — the point is to feel how each idea needs the last.
Walk through your dark house: you blend a fuzzy memory of how you've moved (it drifts) with the occasional touch of a known landmark (it's noisy but doesn't drift), weighted by how much you trust each. That blending is state estimation. The mental upgrade that unlocks the whole field: stop asking "where is the robot?" and start asking "what is the robot's probability distribution over where it is?" That distribution is almost always a Gaussian — a center (your best guess) and a width (your doubt).
Kálmán's 1960 insight, which flew Apollo to the Moon and runs in every phone today: if the models are linear and the noise Gaussian, the intractable Bayes filter collapses into five lines of matrix algebra — and those five lines are provably optimal. The magic of going multivariate: the filter estimates states it never measures (track position, get velocity for free) because correlations in the covariance matrix are conduits for information. And the gain becomes a trust matrix that decides, per dimension, how much of each surprise to absorb.
Reality is rotations and range-bearing sensors and multimodal doubt. Each family member is one strategy for smuggling nonlinearity back into the Gaussian frame. The EKF linearizes — replaces the curve with its tangent line at your estimate — and breaks when the curve bends within your uncertainty. The UKF refuses to touch the function: it pushes a handful of sigma points through the exact nonlinearity and refits, capturing curvature the EKF's tangent misses. And when belief isn't a bell at all — a robot lost among three identical doors — the particle filter abandons Gaussians for a swarm of weighted hypotheses.
Localization needs a map; mapping needs to know where you are. SLAM solves both at once. Dead reckoning drifts unboundedly — until you recognize a place you've seen before. That loop closure is worth a thousand ordinary measurements: it lets you redistribute accumulated error backward through your whole path. Modern SLAM stopped filtering and started optimizing — every pose and landmark a node, every measurement a spring, the whole network relaxed to its minimum-energy shape (a sparse least-squares problem on a factor graph).
An IMU is your inner ear — instant, high-rate, never blinks, but its tiny bias errors compound so viciously that position error grows cubically; useless alone in seconds. A camera is your eyes — barely drifts by re-observing the world, but it's slow, fails in blur and darkness, and can't perceive scale. Each one's weakness is the other's strength: the IMU bridges every camera blink and donates metric scale; the camera anchors the world and lets the filter observe and subtract the IMU's bias. Neither survives a minute; together they run for hours. This is the technology behind every drone hover and AR headset.
Estimation answers "what is the state?" Reinforcement learning answers "given the state, what action maximizes long-term reward?" — with no labels, just trial, error, and a scalar score. The whole field organizes around the value of a state and the recursive insight that the value of now equals reward now plus the discounted value of the best next. And here's the bridge that never leaves: the temporal-difference error — "nudge the estimate by a gain times the surprise" — is structurally the Kalman update. Estimation and learning are the same gesture.
Around 2022 the field pivoted. Pure RL couldn't reach long-horizon, semantic, everyday manipulation — but imitation learning plus internet-scale vision-language models suddenly could. Today's stack: a slow VLM reasoner (System 2) plans in language; a fast action expert (System 1, usually a diffusion or flow policy) emits motor chunks at 50 Hz. The moat isn't the architecture — it's data, manufactured by teleoperation fleets and simulation. And underneath all of it, the classical estimation stack still runs at hundreds of hertz, keeping the body upright so the learned brain can think.
This is the heart of the guide. Eight rungs, each a complete program you run right here in the page — real Python, real numpy, executing in your browser. The ladder is sequenced so each rung rests on the last: rung 8's generative robot policy genuinely stands on rung 1's six-line filter. Press Run to execute, Check to verify your implementation against assertions, Show Solution if you're stuck, and Reset to start over. The first Run loads Python (a few seconds, once).
Everything starts here. A one-dimensional Kalman filter is just two ideas alternating forever: predict (extrapolate your belief and grow your doubt) and fuse (combine prediction with measurement, weighted by trust, which shrinks your doubt). The trust weight K is the Kalman gain. Below, a noisy sensor (sensor noise variance 4) watches a drifting target; watch the filter's RMSE land far below the raw sensor's.
R = 25 (a terrible sensor) and watch the filter lean on its model; set Q = 5 and watch it chase the noise.The real world is nonlinear — a robot's motion uses sin and cos of its heading. The Extended Kalman Filter handles this by linearizing: it replaces the curved motion function with its tangent plane, encoded in a Jacobian matrix of partial derivatives. The single most common source of week-eating EKF bugs is a hand-derived Jacobian with one wrong sign. The professional's defense, which you build here: verify the analytic Jacobian against finite differences.
F_jac (say +v*dt*np.sin(th)) and watch jac_err explode — that's the bug you'll otherwise spend a week chasing in a live filter.The EKF's tangent-plane lie fails when the curve bends sharply within your uncertainty. The Unscented Kalman Filter refuses to linearize: it places a handful of sigma points that encode your Gaussian, pushes each through the exact nonlinear function, and refits. No Jacobians at all. Below, a polar (range, bearing) reading is converted to Cartesian — the textbook nonlinearity. The true distribution bends into a "banana"; watch the UKF's mean land on it while the EKF's stays stuck at the tip.
sigT to 5*np.pi/180 (tiny bearing noise) — the banana straightens and EKF and UKF nearly agree. The UKF's advantage is nonlinearity-times-uncertainty.Some doubt isn't a single bell. A robot in a corridor with three identical doors, having just sensed "I'm at a door," honestly believes it's at one of three places. No Gaussian can say that. The particle filter represents belief as a swarm of weighted hypotheses: move them all by the motion model, weight each by how well it explains the measurement, then resample — clone the believable, kill the rest. Below, 500 particles start uniformly ignorant and collapse onto the true door as motion disambiguates which one it was.
parts = np.random.rand(N) mid-run (a kidnap) and watch whether it recovers — it can't, unless you inject a few random particles each resample. That's why real localizers (ROS's AMCL) do exactly that.Modern SLAM doesn't filter — it optimizes. Every pose is a variable; every odometry step and loop closure is a soft constraint (a "spring"). When a robot loops back and recognizes its start, that loop closure contradicts the accumulated odometry drift, and the optimizer redistributes the error across the whole trajectory. Below you build the real thing in 1-D: Gauss–Newton on a chain of poses with a loop-closure constraint, watching the residual collapse.
An accelerometer integrated twice drifts catastrophically, mostly because of a small, unknown bias. The trick that makes VIO work: put the bias in the state vector and estimate it from occasional position fixes. Below, raw IMU integration explodes while a 3-state Kalman filter — position, velocity, and bias — stays nailed to the truth and recovers the hidden bias.
Now we leave perception for decision. In the cliff-walk, an agent must get from start to goal along a row where one wrong step falls off a cliff. Q-learning (off-policy) learns the value of the optimal policy and hugs the cliff edge — shortest path, reckless while exploring. SARSA (on-policy) learns the value of its actual exploring self and takes the safe detour. One line of code differs between them — and it changes the personality entirely.
Q += al * (target - Q) — it's gain times surprise, the Kalman gesture from Rung 1, wearing a different hat. Experiment: raise eps to 0.4 (clumsier exploration) and SARSA detours even higher, because it's accounting for how often it stumbles.RL is half of decision-making; the other half, and the one that powers today's robot foundation models, is imitation learning — copy an expert's demonstrations. The naive version, behavior cloning, just regresses observations to actions. It has one devastating flaw that defines the modern stack. Human demonstrations are multimodal: at a fork, the expert sometimes goes left around the obstacle, sometimes right — both valid. A network trained to predict the average action splits the difference and drives straight into the obstacle. The fix that swept robotics in 2023 is the diffusion policy: don't predict one action, generate one by iteratively denoising from pure noise — which can represent "left or right" and commit to one. You build that exact mechanism here, in pure numpy.
T to 5 denoising steps and watch the samples land sloppily between the modes — too few steps and the generator can't separate them, which is why flow-matching (fewer, straighter steps) is the current frontier.You now hold the full arc — belief, filter, family, map, fused motion, learned decision, frontier stack — and, if you climbed the ladder, you've built the core of each. That's the difference that matters: not having read about a Kalman filter, but having watched your own six lines beat the sensor they're made of.
Each rung you ran is a weekend project waiting to be made real and yours. Here is the order, with the deep lesson behind each — do them in sequence and you will have implemented modern robot perception end to end:
| Rung | You built | Make it real with |
|---|---|---|
| 1 | Six-line 1-D Kalman filter | Kalman Filter → the 2-D constant-velocity tracker |
| 2–3 | EKF Jacobian + the unscented transform | EKF · UKF → a full unicycle EKF with landmark bearings |
| 4 | Particle filter, global localization | Particle Filter → add kidnap-recovery |
| 5 | Pose-graph Gauss–Newton | Factor Graphs → redo it in GTSAM; run ORB-SLAM3 |
| 6 | 1-D VIO with bias estimation | Modern VIO → OpenVINS on the EuRoC dataset |
| 7 | Q-learning vs SARSA | RL Algorithms → CleanRL PPO on Gymnasium |
| 8 | Diffusion policy — multimodal action sampling | Diffusion Policy · Imitation Learning → train one on PushT |
| 9 | (frontier) | VLAs → LeRobot on your own teleop data |
Estimation & SLAM: Thrun, Burgard & Fox, Probabilistic Robotics (the bible); Barfoot, State Estimation for Robotics (rigorous, free PDF); Solà's error-state / quaternion notes; Dellaert & Kaess, Factor Graphs for Robot Perception; Cyrill Stachniss's SLAM lectures. RL: Sutton & Barto, Reinforcement Learning (free); OpenAI Spinning Up; CleanRL source. Frontier: the ALOHA/ACT, Diffusion Policy, RT-2, π0, and GR00T papers — all readable now that you have the whole stack in your head.
This guide's closing thesis — that the boundary between the classical estimation half and the learned-decision half is dissolving, and the rarest engineers are fluent in both — has its own deep lesson with its own hybrid-estimator playground: The Estimation–Learning Merger. Climb this ladder first; it earns you that one.
“What I cannot create, I do not understand.” — Richard Feynman.
Now close this tab and open a blank file. Rung 1 is six lines long, and you already know all six.