Visual-Inertial Odometry — Fusing a Camera with an IMU

Chapter 1: What the Camera Actually Measures — and How the IMU Resolves Scale

Before we fuse anything we need to be precise about what a camera contributes to the estimate, because "the camera sees" is too vague to build on. A camera does not measure position or depth directly. It measures one thing: where points of the world land on its image plane. Everything visual in VIO — every constraint, every residual — is built from that single primitive. So let's nail it down, with just enough depth to make the fusion concrete, and cross-link the dedicated VIO lessons (classical VIO, modern VIO) for the full treatment.

Features and feature tracks

A feature is a small, distinctive patch of image — a corner of a window, a speck of texture, a high-contrast blob — that the system can recognize again in the next frame. Detect a few hundred of these per image, then match them frame to frame. A single feature followed across many frames is a feature track: a list of pixel coordinates, one per frame, all corresponding to the same physical 3D point seen from a moving camera. That track — the same world point, multiple viewpoints — is the atom of visual measurement. Hold onto it; in Chapter 3 the entire MSCKF is built on the constraint that a single track imposes across many camera poses.

The projection function π(·)

How does a 3D point become a pixel? Through the projection function, written π(·). Take a 3D point expressed in the camera's own frame, p = (X, Y, Z), where Z is the depth straight out of the lens. The pinhole camera maps it to a pixel by dividing out the depth and scaling by the focal length f (plus the image center, the principal point, which we'll drop for clarity):

Stare at that division by Z. That single divide is the source of all the trouble and all the power. It means a point twice as far away (2Z) with twice the lateral offset (2X) lands on the exact same pixel: f·(2X)/(2Z) = f·X/Z. The camera cannot, from one image, distinguish near-and-small from far-and-large. That is monocular scale ambiguity, now visible in the equation. It is also why depth Z is the thing VIO most wants to recover — everything hinges on it.

Reprojection error — the visual residual

Here is how a feature track turns into something an estimator can chew on. Suppose we have a guess for the camera's pose (where it was) and a guess for the 3D position of a landmark. We can then predict where that landmark should appear in the image: take the landmark, transform it into the camera frame using the guessed pose, run it through π(·). Call that predicted pixel π(pose, landmark). The camera actually observed the feature at some measured pixel z. The gap between them is the reprojection error:

where r is a 2-vector (pixels off in u and v), z is the measured pixel, and π(pose, landmark) is the predicted pixel. If our pose and landmark guesses were perfect, the predicted pixel would land exactly on the observed one and r = 0. When they're wrong, r is nonzero, and every VIO system on Earth works by adjusting the poses and landmarks to drive these reprojection errors down. The filter-based methods (MSCKF) treat r as an EKF measurement residual; the optimization methods (VINS) sum r² over all tracks and minimize. Same residual, two machines. This is the visual half of VIO in one equation.

The scale-recovery argument, with numbers

Now the centerpiece. Let's see exactly how the IMU's meters fix the camera's unknown scale, with arithmetic you can do by hand. Suppose the camera, from pure visual odometry over a one-second window, reports that the rig moved along a certain path of unknown size — it says "I moved s units forward," where the shape is known but the unit s is an unknown multiplier (the scale factor). Pure vision gives us, say, a normalized displacement of 1.0 visual-unit over that second, but we don't know how many meters a visual-unit is. Call that unknown conversion λ (meters per visual-unit).

Meanwhile the IMU was measuring acceleration the whole second. Suppose, after subtracting gravity, the net forward acceleration averaged a = 0.4 m/s² and the rig started that second moving at v₀ = 1.6 m/s. Basic kinematics says the real metric distance covered is:

So the IMU says: real distance = 1.8 meters. The camera says: visual displacement = 1.0 visual-units. The scale factor that reconciles them is simply the ratio:

That's it — the scale is recovered. Every future visual displacement now gets multiplied by 1.8 to become real meters. The camera supplied the direction and shape of motion (which it does beautifully and nearly drift-free); the IMU supplied the one number vision lacked — how many meters. Notice it took an accelerating motion for this to work: the ½at² term is what made the IMU's contribution scale-dependent. If the rig had moved at constant velocity (a = 0) with v₀ also unknown, the IMU and camera would both just see "some constant motion" and the scale would stay ambiguous. Hold that thought — it is the deep reason VIO needs acceleration to initialize, which we'll formalize in Chapter 6 as an observability condition.

Below, drag the IMU's measured acceleration and watch the recovered scale snap the wrong-size visual trajectory onto the true one. At a = 0 (constant velocity) the scale is undetermined and the recovered path is garbage; give it some acceleration and the metric path locks in.

Chapter 2: Two Paradigms — Filter the Estimate, or Optimize It

We now know what the camera and IMU each contribute. The remaining question — the one that organizes the entire field — is how to combine them over time. There are two grand answers, and almost every real VIO system is one or the other (or a hybrid). Knowing which camp a system belongs to tells you most of its behavior, so this short chapter is the map you'll navigate the next three chapters with.

The two paradigms are filter-based and optimization-based. They differ in one philosophical choice: when a new measurement arrives, do you incrementally update a running estimate (and throw the measurement away), or do you keep a window of recent measurements and re-solve the whole window from scratch every time?

Why does re-solving help accuracy? Because the visual and inertial measurement functions are nonlinear (that division by Z, the rotations). An EKF linearizes each function once, at the estimate it had when the measurement arrived — and if that estimate was off, the linearization is off, and the error is baked in permanently. An optimizer re-linearizes every iteration around the current best estimate, so as the solution improves, so do the linearizations. It can also revisit an old measurement in light of new ones (you learn where you were a second ago partly from where you are now). The filter cannot look back; the optimizer can. That re-linearization and look-back is precisely why optimization-based VIO is usually more accurate — and precisely why it costs more.

Both paradigms face the same existential problem: the window cannot grow forever, or compute explodes. Each paradigm has its own surgical tool for keeping the window bounded while losing as little information as possible:

The next three chapters take these one at a time: Chapter 3 dissects the MSCKF and its null-space trick; Chapter 4 dissects the VINS-style optimizer and marginalization; Chapter 5 dissects IMU preintegration, the shared enabler that makes the optimization approach tractable at all. Keep this table open in your head — everything that follows hangs on it.

Chapter 3: The MSCKF — Constrain Poses, Eliminate Landmarks

The naive way to fuse a camera into an EKF is to put every 3D landmark into the state vector and estimate it alongside the pose — this is EKF-SLAM. It works, but it has a fatal scaling problem: a thousand landmarks means a thousand 3-vectors in the state, and the covariance matrix is the state size squared, so cost grows quadratically (or worse) with the number of features you track. On a power-limited headset that is a non-starter. In 2007, Anastasios Mourikis and Stergios Roumeliotis published a beautiful trick that gets the constraint information out of the landmarks without ever putting them in the state. It is called the Multi-State Constraint Kalman Filter (MSCKF), and it is the foundation of OpenVISLAM, ROVIO-adjacent filters, and OpenVINS — the filter-based half of modern VIO.

The core idea: a sliding window of poses, no landmarks

The MSCKF state holds the current IMU state (orientation, position, velocity, biases) plus a sliding window of recent camera poses — say the last 10–20 keyframe poses. That's it. No 3D landmarks live in the state. Each time the camera fires, the current pose is cloned into the window; when the window is full, the oldest pose is dropped. Because the state holds only poses (a fixed-size window) and the IMU state, its size is bounded and independent of how many features you track. Track 50 features or 500 — the state is the same size. That is the headline property: bounded computation.

But wait — if landmarks aren't in the state, how does the camera contribute at all? This is the clever part.

One feature, many poses — the multi-state constraint

Recall the feature track from Chapter 1: a single physical 3D point, observed in several frames as the camera moved. Say a landmark L was seen from three poses in the window: poses C₁, C₂, C₃. Each observation gives a reprojection equation:

where each z_i is the measured pixel in frame i, each π(C_i, L) is the prediction from that pose and the (unknown) landmark L, and n_i is pixel noise. Here is the key observation: the same unknown L appears in all three equations. So these three measurements are not independent statements about three different things — they are three views of one thing, and that creates a geometric constraint linking the three poses. Specifically: there must exist a single 3D point L consistent with all three observations. If the three poses are estimated correctly, such an L exists and reprojects cleanly into all three; if the poses are wrong, no single L fits all three and the residuals can't all be zero. That mutual consistency is the constraint the MSCKF feeds to the EKF. The feature track is the glue between poses.

The problem: the landmark is a nuisance — eliminate it

We want a measurement that constrains the poses (which are in the state), but the equations above are contaminated by L (which is not in the state). L is a nuisance parameter — we don't want to estimate it, but it's gumming up the equations. We need to remove it. Here's how, step by step.

Step 1 — triangulate L. Using the current pose estimates, find the 3D point L that best fits all the observations (least-squares triangulation). This gives a working estimate L̂.

Step 2 — linearize. Stack all the reprojection residuals for this track into one tall vector r. To first order, r depends on errors in the poses (via a Jacobian H_X) and on the error in the landmark (via a Jacobian H_L):

where δX is the stacked pose errors (in the state) and δL is the landmark error (the nuisance, not in the state). We must get rid of the H_L·δL term without affecting the H_X·δX term.

Step 3 — null-space projection. This is the heart of the MSCKF. The landmark error δL is a 3-vector, so H_L is a tall, skinny matrix (many rows, 3 columns). A tall skinny matrix has a left null space: a set of directions A such that Aᵀ · H_L = 0. (For m measurement-rows and 3 landmark columns, the left null space has m−3 dimensions — you "use up" 3 rows to pin down the 3-DOF landmark, and the rest are free.) Now multiply the whole linearized equation on the left by Aᵀ:

The landmark term is gone — not approximated away, algebraically annihilated. What remains, r′ = H′_X·δX + n′, is a clean EKF measurement that constrains only the poses in the state. We hand this r′ and H′_X to the standard EKF update (Lesson 7) and the filter corrects the window of poses — with the troublesome landmark never having touched the state. The landmark did its job (linking the poses) and then was projected out of existence.

A worked multi-view example, with numbers

Let's make the constraint concrete with a tiny 1-D version you can verify by hand. A camera moves along a line and observes one landmark from three positions. We'll use a simplified projection where the measured "bearing" of the landmark from camera position x is z = L − x (offset of the landmark from the camera — a stand-in for the pixel measurement; the real one divides by depth, but the algebra of elimination is identical). Three poses, three measurements:

Here the landmark Jacobian is H_L = [1, 1, 1]ᵀ (each residual changes by +1 if L changes by +1 — that "1" is ∂z/∂L). The left null space of [1,1,1]ᵀ is any pair of directions summing to zero across the three rows; a convenient basis is:

Check: AᵀH_L = [1−1+0 ; 0+1−1] = [0 ; 0]. Good — the landmark is annihilated. Now project the residuals:

These two numbers, 0.2 and −0.1, are differences of residuals between poses — they encode "do the poses agree about a single landmark?" and carry no dependence on the unknown L. They are exactly what the EKF needs to correct the relative geometry of C₁, C₂, C₃. The original three residuals contained the landmark; their two independent differences do not. That is the MSCKF measurement, computed by hand. We went from 3 raw residuals (mixed with the nuisance L) to 2 clean pose-only constraints — losing exactly the 1 dimension that was about defining L, as the m−3 count predicts (3 measurements − 1-D landmark = 2).

The MSCKF residual in code

A simplified MSCKF feature update — triangulate, stack residuals, compute the landmark Jacobian, project onto its null space — is remarkably short. Verbose, then compact:

python
import numpy as np

def msckf_feature_residual(poses_x, z, L_hat):
    # poses_x : camera positions where this ONE landmark was seen (length m)
    # z       : the m measured bearings of that landmark
    # L_hat   : the triangulated landmark estimate (the nuisance parameter)
    m = len(poses_x)

    # 1) stack the per-pose reprojection residuals  r_i = z_i - (L_hat - x_i)
    r = np.array([z[i] - (L_hat - poses_x[i]) for i in range(m)])

    # 2) Jacobians.  H_L = dr/dL = +1 per row (column vector of ones here).
    #    H_X (dr/dpose) would be built the same way for the real geometry.
    H_L = np.ones((m, 1))                  # tall & skinny: m rows, 3 cols in real 3-D

    # 3) LEFT NULL SPACE of H_L via the thin QR / SVD: columns of A with A^T H_L = 0
    U, S, Vt = np.linalg.svd(H_L)            # U is (m x m)
    rank = (S > 1e-9).sum()                # = number of landmark DOF used up (1 here, 3 in real)
    A = U[:, rank:]                          # the remaining m-rank columns span the null space

    # 4) PROJECT: r' carries pose-only info; the landmark term A^T H_L is exactly zero
    r_proj = A.T @ r                         # clean EKF residual, landmark eliminated
    return r_proj, A                       # feed r_proj (and projected H_X) to the EKF update

# the worked example above:
msckf_feature_residual([0,2,4], [10.1,7.9,6.0], L_hat=10.0)
# -> r_proj has length m-1 = 2, independent of L_hat (try L_hat=50: same r_proj!)

The line that matters is A = U[:, rank:]: it grabs the columns of the orthonormal basis that span the directions H_L doesn't reach — the left null space — and A.T @ r projects the residual onto them, leaving a measurement that depends on the poses but provably not on L. Change L_hat from 10 to 50 and r_proj is identical: the elimination is exact, not approximate.

python
# compact: the whole elimination is three lines once residuals & H_L are formed
A = np.linalg.svd(H_L)[0][:, np.linalg.matrix_rank(H_L):]  # left null-space basis
r_proj = A.T @ r                                            # landmark-free residual
H_X_proj = A.T @ H_X                                        # landmark-free Jacobian -> EKF update

Chapter 4: VINS-Mono & OKVIS — Sliding-Window Bundle Adjustment

The optimization camp does not maintain a single estimate to update — it maintains a window of recent state (poses, velocities, biases, and recent landmarks) and, each time new data arrives, re-solves the whole window as one big nonlinear least-squares problem. This is sliding-window bundle adjustment, and its flagships are VINS-Mono (Tong Qin, Peiliang Li & Shaojie Shen, 2018) and OKVIS (Stefan Leutenegger et al., 2015). It is the most accurate family of VIO methods, and the structure is worth seeing because it makes the camera+IMU fusion completely explicit.

The cost function: visual factors + inertial factors

The optimizer minimizes a single sum over the window. There are exactly two kinds of terms (called factors), and they are the two sensors:

Read it slowly. X is everything we're solving for: all the camera poses in the window, the velocities, the IMU biases, and the recent landmarks. The first sum is the visual factors: for every (pose i, landmark j) pair where landmark j was seen from pose i, add the squared reprojection error from Chapter 1. The second sum is the inertial factors: for every consecutive pair of states (s_k, s_k+1), add the squared error between "what the IMU said the motion between them was" and "what the current poses say the motion was." Minimize the total and you get the poses, velocities, biases, and landmarks that best agree with both the camera and the IMU at once. That is tightly-coupled fusion, written as one optimization.

The window can't grow forever — marginalization

Each new frame adds a pose (and its factors) to the window. If we never removed anything, the problem would grow without bound and the optimizer would slow to a crawl. The naive fix — just delete the oldest pose and all its factors — is a disaster: it throws away hard-won information about where you were, and the estimate degrades and drifts. We need to remove the variable but keep its information. The tool is marginalization, implemented via the Schur complement.

Here's the idea without the heavy algebra. The least-squares problem, linearized, is a big linear system H · δX = b, where H is the information matrix (how much we know, and how the variables are coupled). Split the variables into the one we want to drop (the old pose, call it α) and the ones we keep (β). The system blocks into:

To drop α while preserving its effect on β, we don't delete its row — we algebraically solve it out, folding its information into the kept block. The result is a smaller system on β alone:

That subtracted term, H_βαH_αα⁻¹H_αβ, is the information the old pose contributed to the kept variables, now baked permanently into them as a prior. The old pose is gone from the state, but everything it taught us about the variables we kept survives. Marginalization is "forgetting a variable without forgetting what it told you." This is the optimizer's analog of how a Kalman filter's covariance carries forward the past — in fact, marginalizing one pose per step is exactly what turns a full-window optimizer into something filter-like.

What VINS-Mono adds on top

VINS-Mono is a full system, not just the optimizer, and three of its extra pieces are worth naming because they're what makes it robust in the wild:

• Online temporal calibration. The camera clock and IMU clock are never perfectly synced (Lesson 15). VINS-Mono estimates the time offset between them as a variable in the optimization, so it self-corrects sync error instead of letting it corrupt the fusion.
• Online extrinsic calibration. The rigid transform between the camera and the IMU (how they're mounted relative to each other) is also estimated online — you don't need a perfect factory calibration, the system refines it as it runs.
• Loop closure. When the camera revisits a place it saw before, VINS-Mono recognizes it and adds a constraint linking now to then, which yanks out accumulated drift — turning odometry (drifts slowly) into something closer to SLAM (globally consistent). This is the bridge to Lesson 9's SLAM material.

The single thing that makes this whole optimization tractable in real time — the reason we can afford to re-solve the window every frame — is the way the IMU factor r_imu(s_k, s_k+1) is computed. Done naively it would be ruinously expensive, and the fix is the subject of the next chapter: IMU preintegration.

Chapter 5: IMU Preintegration — Collapsing Thousands of Samples into One Factor

This chapter explains the single trick that makes optimization-based VIO possible. Without it, the IMU factor r_imu(s_k, s_k+1) from Chapter 4 would have to be recomputed from scratch hundreds of times per optimizer step, and real-time operation would be hopeless. With it, the entire stream of IMU samples between two camera frames collapses into one cheap relative-motion factor. The trick is IMU preintegration, given its definitive on-manifold form by Forster, Carlone, Dellaert & Scaramuzza in 2017. Let's derive why it has to exist.

The naive approach, and why it's prohibitively expensive

Between two consecutive camera keyframes — say 1/30th of a second apart — the IMU has fired maybe 30 times (at 1000 Hz, 33 samples). To form the inertial factor, we integrate those samples: gyro readings integrate to a rotation change, accelerometer readings (minus gravity) integrate twice to a position change. Straightforward. But — and this is the whole problem — the accelerometer and gyro readings must first have their biases subtracted, and the biases are themselves unknowns being optimized.

So here is the catastrophe. The optimizer adjusts the bias estimate. The integration depends on the bias. So every time the optimizer changes the bias — which is every iteration, for every keyframe pair — you must re-integrate all 33 raw IMU samples from scratch. The integration also depends on the starting orientation, which is also being optimized, so it must be redone when that changes too. Multiply: tens of samples × tens of keyframe pairs in the window × several optimizer iterations per frame × 30 frames per second. The re-integration cost explodes. Naive IMU integration inside an optimizer is a non-starter.

The fix: preintegrate relative increments in a bias-independent form

Forster et al.'s insight: instead of integrating in the world frame, integrate in the frame of the first keyframe, producing relative motion increments that depend only on the raw IMU samples (and a fixed bias linearization point) — not on the starting orientation, velocity, or gravity. These increments are:

The magic: ΔR, Δv, Δp depend only on the IMU measurements between the two frames (and the bias values you linearized at) — not on the global pose or velocity at frame k. So you compute them once, when the frame arrives, and they never change as the optimizer wiggles the poses and velocities. The whole IMU stream between two keyframes becomes three stored increments: one rotation, one velocity bump, one position bump. Thousands of samples → one relative-motion factor. That factor is r_imu(s_k, s_k+1): it compares the increments to what the current poses/velocities imply, and the optimizer drives the difference to zero.

But the bias does change — the first-order correction

One loose end: we computed the increments using some bias value, but the optimizer keeps changing the bias. Do we have to re-integrate after all? No — and this is the second half of the trick. Instead of re-integrating, we precompute how the increments would change for a small change in bias: the bias Jacobians, ∂ΔR/∂b, ∂Δv/∂b, ∂Δp/∂b. Then when the optimizer nudges the bias by δb, we correct the stored increment with a cheap first-order update instead of re-integrating:

One matrix-vector product, no re-integration. As long as the bias change δb is small (which it is, near convergence), this first-order correction is accurate. We only re-integrate from raw samples occasionally (a "repropagation") when the bias has drifted far enough that the linearization staled. This is the final piece that makes preintegration cheap and correct under a moving bias estimate.

A worked preintegration + bias-correction example, with numbers

Let's run a tiny 1-D preintegration by hand so the increments and the bias correction are concrete. A rig accelerates along one axis. Three IMU accelerometer samples arrive over an interval, each Δt = 0.1 s apart, all reading the same value (constant acceleration for simplicity). The accelerometer reports a_raw = 2.00 m/s², and our current bias estimate is b = 0.10 m/s². The bias-corrected acceleration is a_raw − b = 1.90 m/s².

Preintegrate Δv (velocity increment) — sum (a−b)·Δt over the three samples:

Preintegrate Δp (position increment) — accumulate the running velocity (simple forward integration, v starting from 0 within this interval):

Now store the bias Jacobian for Δv. Since Δv = Σ(a−b)Δt and we have three samples each of width 0.1 s, a change in bias b changes Δv by:

The optimizer now decides the true bias is a bit higher: δb = +0.05 m/s². Without re-integrating anything, correct Δv with the first-order rule:

Sanity check by full re-integration: with b = 0.15, corrected accel = 2.00 − 0.15 = 1.85, so Δv = 3×(1.85)(0.1) = 0.555 m/s. Identical. The first-order correction reproduced the full re-integration exactly (it's exact here because the model is linear in bias), but it cost one multiply instead of re-summing all the samples. Scale this from 3 samples to 33, and from one factor to a whole window re-evaluated several times per frame, and you see why this single trick is what makes real-time optimization-based VIO feasible.

A preintegration accumulator in code

The accumulator that builds Δv, Δp and the bias Jacobian as samples stream in is short and mirrors the worked example exactly:

python
class Preintegrator:
    # 1-D accelerometer preintegration with a first-order bias correction.
    def __init__(self, bias):
        self.bias = bias            # the linearization point for the bias
        self.dv = 0.0            # velocity increment  Δv
        self.dp = 0.0            # position increment  Δp
        self.dv_db = 0.0         # bias Jacobian  ∂Δv/∂b
        self.dp_db = 0.0         # bias Jacobian  ∂Δp/∂b

    def integrate(self, a_raw, dt):
        a = a_raw - self.bias                 # subtract the (current) bias estimate
        self.dp += self.dv * dt + 0.5 * a * dt * dt   # position uses running Δv
        self.dv += a * dt                     # velocity increment
        self.dp_db += self.dv_db * dt - 0.5 * dt * dt  # chain rule through dp
        self.dv_db += -dt                     # ∂(Δv)/∂b accumulates -dt each step

    def corrected_dv(self, new_bias):
        # FIRST-ORDER correction: no re-integration of raw samples!
        db = new_bias - self.bias
        return self.dv + self.dv_db * db

# the worked example: 3 samples of a_raw=2.0, bias=0.10, dt=0.1
pre = Preintegrator(bias=0.10)
for _ in range(3): pre.integrate(2.0, 0.1)
pre.dv                       # -> 0.57   (matches the hand calc)
pre.corrected_dv(0.15)     # -> 0.555  (bias bumped +0.05, ONE multiply, no re-integration)

The line self.dv_db += -dt is the whole bias-Jacobian story: each sample of width dt contributes −dt to how Δv responds to bias, so after three 0.1 s samples dv_db = -0.3 — exactly the −0.3 we computed. Then corrected_dv applies that Jacobian with one multiply to absorb any new bias the optimizer proposes, never touching the raw samples again. That is the engine room of every modern factor-graph VIO.

Below, accumulate raw IMU samples into a single preintegrated factor, then drag the bias slider to see the cheap first-order correction track the full re-integration — until the bias drifts so far that you'd need a repropagation.

Chapter 6: Initialization & Observability — What VIO Can and Cannot Know

We've built the machinery. Now the question that separates a VIO that works from one that diverges on the launch pad: what can the system actually figure out, and what is forever invisible to it? This is the theory of observability, and it has direct, brutal practical consequences. Get this chapter wrong in a real deployment and your headset thinks it's drifting through the floor.

The four unobservable degrees of freedom

A VIO system, no matter how good, cannot determine four numbers about itself. These are the four unobservable degrees of freedom:

• Global position (3 DOF). VIO is odometry — it measures motion relative to where it started. There is no GPS, no map; "where am I on Earth" is meaningless to it. It only knows displacement from the origin it chose. (That's 3 of the 4.)
• Global yaw (1 DOF). Rotation about the gravity (vertical) axis. The system has no compass, so "which way is north" is invisible. It can measure changes in yaw beautifully (the gyro), but the absolute yaw is a free choice. (That's the 4th.)

That's 3 + 1 = 4 unobservable DOF. Crucially, that's all — and the reason the count stops at 4 is the hero of this chapter: gravity.

Why gravity makes roll and pitch observable

You might expect all three rotation axes to be unobservable, like yaw. They are not — and the reason is that the accelerometer always feels gravity, a constant 9.81 m/s² pointing down. That gravity vector is an absolute reference for "down." When the rig tilts, the felt gravity rotates in the IMU's frame, and the accelerometer reads the tilt directly. So roll and pitch (tilt away from vertical) are observable — gravity pins them. Only rotation about the gravity axis (yaw) leaves the gravity vector unchanged, so only yaw is unobservable. Gravity converts 2 of the 3 rotational DOF from unobservable to observable. Without gravity, VIO would have 6 unobservable DOF (full position + full orientation); with it, only 4.

Initialization: recover scale, gravity, velocity, and bias before you start

The tightly-coupled estimator (MSCKF or VINS) assumes it starts from a decent estimate — it refines, it doesn't conjure an answer from nothing. So before it can run, a separate initialization step (visual-inertial alignment) must recover four things from the first second or two of data:

• Metric scale λ — the meters-per-visual-unit factor (Chapter 1). Without it, every distance is wrong by a constant multiplier.
• Gravity direction — which way is down in the starting frame, so roll/pitch are anchored and gravity can be subtracted from the accelerometer.
• Initial velocity — how fast the rig was already moving when tracking began (the v₀ in the scale-recovery kinematics).
• Gyroscope bias — the slowly-varying offset on the gyro, which otherwise injects a creeping rotation error.

Initialization works by running short windowed vision-only structure-from-motion (which gives the trajectory shape, up to scale) and aligning it with the preintegrated IMU increments (which carry the meters and the gravity direction). Solving that alignment yields scale, gravity, velocity, and bias in one shot. Then, with a good starting estimate, the tightly-coupled filter/optimizer takes over.

Why a bad init dooms the whole run

Here is the part people learn the hard way. The estimator is a local optimizer (or a linearizing filter) — it polishes the estimate it's given, assuming that estimate is in the right neighborhood. If initialization hands it the wrong scale — say it thinks λ = 1.0 when the truth is 1.8 — then every metric quantity is born wrong by 80%, and the filter happily refines a self-consistent but badly-scaled world. Distances, velocities, the map — all stretched. And because scale is barely observable from gentle motion (Chapter 1: you need acceleration to see it), the estimator may take a long time to claw the scale back, or never. A wrong initial gravity tilts the whole world; the system thinks "up" is at an angle and drives the trajectory into the floor. A wrong gyro bias injects a steady yaw drift. Initialization is the foundation; a crack in it propagates through everything built on top.

Below, watch the four unobservable DOF in action: the absolute position and yaw of the estimate are free to drift (you can grab and slide them) while roll/pitch stay pinned by the gravity vector. The shape of the trajectory is locked; only the global frame floats.

Chapter 7: Use Cases & Real Products — Where VIO Actually Runs

VIO is not a research curiosity — it is shipping in hundreds of millions of devices, and it's worth knowing the real systems by name, because "VIO" in the abstract becomes concrete the moment you connect it to the thing in your pocket or on your face. The thread tying every application together: knowing where you are, in real meters, with no GPS, on a small battery-powered device. That is precisely the niche the camera+IMU pair owns.

AR / VR headset tracking — the killer app

The reason VIO became a billion-device technology is inside-out tracking for AR and VR. A Meta Quest headset has no external base stations — it tracks itself purely from onboard cameras and an IMU, which is VIO. Apple's ARKit and Google's ARCore — the frameworks behind every phone-AR app — are VIO engines: that's how your phone anchors a virtual couch to your living-room floor and keeps it there as you walk around it. The IMU's high rate is what makes the tracking feel instantaneous (low motion-to-photon latency); the camera is what keeps it from drifting; the metric scale is what makes the virtual couch the right size. Get the scale wrong and the couch is a doll's couch or a giant's — which is exactly the failure VIO's IMU prevents.

Drones — autonomous flight without GPS

A drone flying indoors, under a bridge, or through a forest canopy has no usable GPS. VIO is how it holds position and navigates anyway. Research and racing drones run VINS-Mono / VINS-Fusion (the optimization-based flagship from HKUST) for exactly this. The metric scale matters enormously here: a drone commanding "move forward 2 meters" must know what 2 meters is, and only the IMU's metric acceleration supplies that — a monocular-only drone would have no idea how far it actually moved.

The named systems — a field guide

Smartphones and beyond

Beyond AR apps, the same VIO machinery stabilizes phone video (the IMU+vision motion estimate drives electronic image stabilization), powers indoor pedestrian navigation, and anchors industrial AR (a technician's tablet overlaying instructions on a machine). Every one of these is the same camera+IMU fusion, tuned for a different platform and power budget — which is exactly the practical choice we turn to next.

Chapter 8: Practical Application — Building a VIO That Works

Knowing the theory is half the battle; deploying VIO means a sequence of engineering choices, each with a real tradeoff. This chapter is the field manual — the decisions you'll actually face, with the reasoning to make them.

Choosing the paradigm: filter vs optimization

The first fork. Use the table you internalized in Chapter 2:

Rule of thumb: if you can afford it, optimization wins on accuracy; if you can't, the MSCKF gives you 90% of the accuracy at a fraction of the compute and a guaranteed bounded cost. Many shipped headsets run a filter for exactly the bounded-latency reason.

Keyframe selection

You can't add every frame to the window — that's wasteful and fills the window with near-duplicate views. Instead, select keyframes: add a new pose to the optimization only when something has changed enough to be informative. Common triggers: the camera has translated/rotated past a threshold, the set of tracked features has changed substantially (lots of new features, or many lost), or enough time has passed. Good keyframe selection keeps the window informative (wide baselines for good triangulation) without bloating it. Too few keyframes and triangulation is weak; too many and you blow the compute budget on redundant views.

The initialization motion you must perform

From Chapter 6: VIO needs excitation to initialize. In practice this means your startup procedure must cause acceleration and rotation. A headset's "look around" prompt, a drone's pre-flight wiggle, a phone-AR app asking you to "move your phone" — these aren't UX flourishes, they're physically necessary to make scale, gravity, velocity, and bias observable. If your device can't move before it needs to localize, VIO is the wrong tool. Design the motion in.

Online vs offline calibration

Two things must be calibrated: the camera intrinsics (focal length, distortion) and the camera-IMU extrinsics (the rigid transform between them) plus the time offset between the two clocks. Offline calibration (a one-time procedure with a checkerboard) is more accurate and is mandatory for intrinsics. But extrinsics and time offset can also be refined online (VINS-Mono does this) — which is essential because a phone bends slightly in your hand, a drone's frame flexes, and the factory calibration drifts. Best practice: calibrate intrinsics offline, then let the system refine extrinsics and time-sync online. (Lesson 15 covers the calibration and time-sync machinery in depth.)

Sensor rates

Run the IMU fast — 200 Hz minimum, often 500–1000 Hz. The IMU is what gives you high-rate pose output between camera frames, so the faster it runs the smoother and lower-latency your tracking. The camera can run slower (20–60 Hz) because each frame is expensive to process and the IMU fills the gaps. The classic architecture: IMU integration produces a pose estimate at IMU rate, and each camera frame corrects that high-rate estimate. This is why your headset feels instantaneous despite a 30 Hz camera — you're seeing the IMU's rate, corrected by the camera.

Handling visual dropout by coasting on the IMU

The most important robustness behavior: when the camera fails — blank wall, motion blur, fast spin, a hand over the lens — do not panic, coast on the IMU. The IMU keeps integrating, so the system keeps producing a pose; it just drifts a little until vision recovers, at which point the visual measurements snap the drift back out. This is the same graceful handoff you saw in Chapter 0's simulation. A well-built VIO detects low feature counts and increases its reliance on the IMU smoothly (via the measurement covariances, à la inverse-variance weighting from Lesson 1) rather than crashing. Short dropouts (a second or two) are survivable precisely because the IMU is a superb short-horizon sensor — the t² drift is tiny over a second.

Chapter 9: Debugging & Failure Modes — When VIO Goes Wrong

VIO fails in characteristic, recognizable ways, and a huge part of being good at it is pattern-matching a symptom to its cause. This chapter is that diagnostic catalog — the failures you will see, what each looks like, and how to detect it.

Bad or failed initialization — the deadliest

Symptom: everything is consistently wrong by a constant factor — the trajectory has the right shape but the wrong size, or the whole world is tilted, or there's a steady yaw creep from the first second. Cause: initialization recovered the wrong scale, wrong gravity, or wrong gyro bias (Chapter 6). Detect: check the initialization's reported scale against a known distance (move a known 1 m and see if it reads 1 m); watch whether the estimated gravity magnitude is near 9.81; flag if the trajectory drives into the floor. Fix: demand more excitation before declaring init complete, and re-initialize if the post-init residuals are large. A bad init is unrecoverable for a local estimator — better to restart than to limp.

Insufficient excitation — scale silently unobservable

Symptom: scale slowly drifts or never converges; the system seems fine then gradually mis-sizes everything. Cause: the motion has been near constant velocity (e.g. a drone in steady cruise, a headset held still) — and from Chapter 1, scale is observable only through acceleration. With no acceleration, scale is unobservable and the estimate of it wanders. Detect: monitor the acceleration excitation; if it's been near zero for a while, raise a "scale may be drifting" flag. Fix: in steady-cruise applications, periodically induce small accelerations, or fall back to other scale sources. This is a famously sneaky failure because the system looks healthy.

Time-sync and extrinsic errors

Symptom: jittery or biased estimates, especially during fast motion; the faster you move, the worse it gets. Cause: the camera and IMU timestamps are misaligned (time-sync error), or the camera-IMU rigid transform (extrinsics) is wrong — so the system attributes IMU motion to the wrong camera instant. Detect: error that scales with motion speed is the tell. Fix: enable online time-offset and extrinsic estimation (Chapter 4); these are covered in depth in Lesson 15. A few milliseconds of sync error is invisible at walking speed and catastrophic at drone speed.

Feature-poor or motion-blurred scenes

Symptom: tracking gets shaky or briefly lost in blank corridors, against white walls, in the dark, or during rapid motion. Cause: too few trackable features (low texture) or smeared features (motion blur, fast rotation). Detect: monitor the tracked-feature count; a sudden drop is the alarm. Fix: coast on the IMU through the gap (Chapter 8), increase IMU reliance smoothly, and add features aggressively when texture returns. Short gaps are survivable; sustained feature-poverty (a long dark tunnel) eventually defeats VIO because the IMU alone drifts.

IMU saturation

Symptom: a sudden, large, unrecoverable error after a sharp impact, drop, or extremely aggressive maneuver. Cause: the accelerometer or gyro hit its physical measurement limit and clipped — it reported its maximum instead of the true (larger) value, so the integrated motion is wrong and there's no way to know by how much. Detect: check whether IMU readings are pegged at their saturation limit. Fix: use an IMU rated for your motion envelope; flag saturated samples and inflate their uncertainty or reject the interval.

Divergence symptoms — how to know it's failing

Across all of these, the universal alarms that the estimator is diverging: the covariance (the filter's own uncertainty) balloons; the measurement residuals (reprojection errors, IMU factor errors) stay large instead of settling near the noise floor; the estimated trajectory does something physically impossible (teleports, accelerates absurdly). A healthy VIO has small, settled residuals and a bounded, sensible covariance. Watch the residuals and the covariance — they tell you the system's health before the output looks obviously broken.

Drift vs loop closure

Finally, remember what VIO fundamentally is: odometry. It drifts — slowly, but without bound — because every estimate is relative to the last. Over a long trajectory the error accumulates. The cure is loop closure (Chapter 4, and Lesson 9's SLAM): when the camera recognizes a previously-seen place, it adds a constraint that snaps the accumulated drift out, turning unbounded odometry drift into a bounded, globally-consistent map. If your long trajectory drifts steadily with no correction, you have odometry; add place recognition and you have SLAM. Knowing which you've built tells you whether steady drift is a bug or just the expected nature of odometry.

Chapter 10: Connections, References & Cheat-Sheet

You've walked the full arc of visual-inertial odometry: why the camera and IMU are perfect partners (each covers the other's blind spots, and the IMU's meters fix the camera's scale), the two paradigms (filter vs optimization), the MSCKF's null-space trick, VINS-style sliding-window bundle adjustment with marginalization, IMU preintegration as the enabler, and the observability theory that explains what VIO can and cannot know. This closing chapter ties it to the rest of the series and hands you a one-page reference.

This whole lesson is the engineering embodiment of the series motto: what I cannot create, I cannot understand, and what I cannot understand, I cannot teach. We didn't just describe VIO — we built its residual by hand, eliminated a landmark by projecting onto a null space we computed, preintegrated three samples and corrected the bias with one multiply, and recovered scale from real kinematics. That is the standard: you should now be able to implement a toy VIO, not just recognize the diagrams.

Where VIO sits in the series

VIO is the culmination of the spatial-fusion arc and the bridge to the modern/learned methods. It is built directly on machinery from earlier lessons:

• Lesson 13: IMU & Inertial Navigation — the inertial half of VIO. The drift, the biases, the integration, and the t² growth all come from here. VIO is "what to do about the IMU's drift": correct it with a camera.
• Lesson 14: INS/GNSS Coupling — the other way to fix IMU drift (with GPS). VIO and INS/GNSS are siblings: same IMU, different drift-correcting partner. Read together to see the pattern: an IMU needs an absolute reference, and the reference can be satellites or a camera.
• Lesson 7: The Extended Kalman Filter — the estimator engine under the MSCKF. The null-space-projected residual feeds a standard EKF update; without the EKF, there's no filter-based VIO.
• Lesson 15: Calibration & Time Sync — the camera-IMU extrinsics and clock offset that VIO depends on (and refines online). The time-sync failure in Chapter 9 lives here.
• Classical VIO & Modern VIO — the dedicated deep-dives on the geometry and the systems, going further than this fusion-focused treatment.

What's next

Lesson 17: LiDAR-Inertial Factor Graphs. VIO fuses a camera with an IMU; Lesson 17 fuses a LiDAR with an IMU. The IMU's role is identical (high-rate, metric, drift-prone, the thing being corrected), but LiDAR replaces the camera's pixels with 3D points — which means no scale ambiguity (LiDAR measures range directly) but a heavier, costlier sensor. You'll see the same factor-graph machinery (preintegration factors, marginalization) carry straight over from this lesson, now with point-cloud registration factors instead of reprojection factors. VIO taught you the camera+IMU pair; Lesson 17 generalizes it to the LiDAR+IMU pair, and the factor-graph view unifies them.

VIO cheat-sheet

References

1. Mourikis, A. I. and Roumeliotis, S. I. "A Multi-State Constraint Kalman Filter for Vision-aided Inertial Navigation." Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), pp. 3565–3572, 2007. The foundational MSCKF paper — sliding window of poses and null-space elimination of landmarks. doi:10.1109/ROBOT.2007.364024
2. Qin, T., Li, P. and Shen, S. "VINS-Mono: A Robust and Versatile Monocular Visual-Inertial State Estimator." IEEE Transactions on Robotics, vol. 34, no. 4, pp. 1004–1020, 2018. The reference optimization-based VIO: sliding-window BA, preintegration, online calibration, loop closure. arXiv:1708.03852
3. Forster, C., Carlone, L., Dellaert, F. and Scaramuzza, D. "On-Manifold Preintegration for Real-Time Visual–Inertial Odometry." IEEE Transactions on Robotics, vol. 33, no. 1, pp. 1–21, 2017. The definitive IMU preintegration formulation with the bias-correction Jacobian. arXiv:1512.02363
4. Leutenegger, S., Lynen, S., Bosse, M., Siegwart, R. and Furgale, P. "Keyframe-based Visual–Inertial Odometry using Nonlinear Optimization." The International Journal of Robotics Research (OKVIS), vol. 34, no. 3, pp. 314–334, 2015. Keyframe-based windowed bundle adjustment fusing stereo vision and IMU. doi:10.1177/0278364914554813