Calibration & Time Synchronization

Chapter 1: Intrinsics vs Extrinsics — Two Very Different Calibrations

Before we hunt the fusion-critical unknowns, we have to name what kind of calibration we are even talking about, because the word "calibration" covers two completely different things and conflating them is a classic confusion. Every sensor has intrinsics — parameters about the sensor itself, in isolation — and a multi-sensor rig additionally has extrinsics — parameters about how the sensors sit relative to each other. Fusion lives and dies on the extrinsics, but you cannot get the extrinsics right until the intrinsics are right, so we recap both.

Intrinsics: a sensor's private parameters

Intrinsic calibration answers: "given this sensor's raw output, what does it mean in physical units?" It is a property of the single device and never mentions any other sensor. The two intrinsics you will meet most are the camera's and the IMU's.

Camera intrinsics. A camera turns 3D rays into 2D pixels. The map is captured by the camera matrix K, a little 3×3 table:

Here f_x, f_y are the focal lengths in pixels (how strongly the lens magnifies, horizontally and vertically), and c_x, c_y are the principal point — the pixel where the optical axis pierces the image, usually near the centre but never exactly at it. On top of K sits lens distortion: real lenses bend straight lines into curves (barrel/pincushion), modelled by a few radial coefficients k₁, k₂, k₃ and tangential p₁, p₂. Intrinsic camera calibration estimates all of these — classically by photographing a checkerboard from many angles (Zhang's method).

IMU intrinsics. An IMU's raw accelerometer and gyroscope outputs are corrupted by three private flaws: a scale factor (a "1 m/s²" true acceleration might read 1.02), a bias (a constant offset even at rest — the very thing that drove the t² drift in Lesson 1), and axis misalignment (the three sensitive axes aren't perfectly orthogonal). IMU intrinsic calibration estimates the scale, bias, and a small misalignment matrix so that raw counts become trustworthy m/s² and rad/s.

Extrinsics: the sensor-to-sensor pose — the fusion-critical part

Now the part this lesson is really about. Extrinsic calibration finds the rigid transform X that takes a point expressed in sensor A's frame and re-expresses it in sensor B's frame. A rigid transform is just a rotation followed by a translation:

where R is a 3×3 rotation matrix (the orientation of A's axes seen from B) and t is a 3-vector (the position of A's origin seen from B). Together (R, t) is the extrinsic, six numbers in all: three for rotation, three for translation. This is exactly the (R, t) you dragged in Chapter 0 — the thing that, when wrong by 2 cm, injected a phantom 2 cm/s.

Why does fusion need this so badly? Because a filter compares a prediction to a measurement, and those two must live in the same frame. The IMU predicts how the rig moved; the camera measures where a feature went. To check the camera against the IMU prediction, you must transport the IMU's prediction into the camera frame — which requires X. A wrong X transports it to the wrong place, and the residual the filter computes is contaminated by the calibration error, not by real sensor disagreement.

The toggle below splits a single feature observation into its frames. Turn intrinsics off and the pixel is mislocated by lens distortion; turn extrinsics off and the camera and IMU frames don't agree on where the feature is. Both must be on for the fused estimate to be trustworthy.

Chapter 2: Spatial Calibration — The Hand-Eye Problem AX = XB

We now know the extrinsic X = (R, t) is the prize. How do we find it without a tape measure and a protractor (which would be hopeless at the precision fusion demands)? The classic answer is one of the most elegant results in robotics, and it falls out of a single observation: if two sensors are rigidly bolted together, then whenever the rig moves, the motion each sensor sees is the same motion, just expressed in its own frame. That constraint, written down, is the famous hand-eye equation.

Deriving AX = XB from rigidity

Let's build it carefully, because the derivation is the understanding. Call the two sensors 1 and 2 (historically "hand" and "eye," from a camera on a robot arm). The rig moves from pose 0 to pose 1. Look at the same physical motion from each sensor's point of view:

• A = the motion as sensor 1 measures it (sensor 1's pose at time 0, transformed to its pose at time 1). You get A from sensor 1's own data — e.g. the IMU integrates its motion, or a camera does visual odometry.
• B = the same motion as sensor 2 measures it, in sensor 2's frame. You get B from sensor 2's own data.
• X = the unknown extrinsic, the fixed transform from sensor 2's frame to sensor 1's frame — the thing we want.

Now trace a single point through the motion two different ways, which must agree because it is one rigid body. Go from "sensor 2 at time 1" back to "sensor 1 at time 0":

Both paths describe the same rigid displacement of the same object, so they must be equal. Setting them equal and renaming gives the celebrated equation (using the convention where A is sensor-1 motion, B is sensor-2 motion, X the extrinsic):

Read it in English: "transforming by sensor-1's motion and then crossing to sensor-2's frame equals crossing to sensor-2's frame and then transforming by sensor-2's motion." It must hold because both sides move one rigid object the same way. The unknown X is squeezed in the middle. One pair of motions (A, B) gives one equation; collect several motions and you can solve for X. Crucially you never measured X directly — you inferred it purely from how the two sensors' self-reported motions relate.

Split the problem: solve rotation first, then translation

AX = XB looks like one equation, but it secretly splits into two, and the split is what makes it solvable. A rigid transform has a rotation part and a translation part. Writing A = (R_A, t_A), B = (R_B, t_B), X = (R_X, t_X) and multiplying out the homogeneous transforms, the equation separates into:

The top equation involves only the unknown rotation R_X. So we solve it first, completely ignoring translation. Once R_X is known, the bottom equation becomes linear in the remaining unknown t_X — rearrange to (R_A − I) t_X = R_X t_B − t_A, a plain linear system you stack over many motions and solve by least squares. Decoupling rotation from translation is the master trick of every hand-eye method; it turns one nasty coupled problem into one rotation problem followed by one easy linear problem.

The rotation solve via the axis-angle (log) trick

How do we solve R_A R_X = R_X R_B for the rotation R_X? Tsai & Lenz's insight: every rotation has a rotation axis (the direction that doesn't move when you rotate). Write each rotation as an axis-angle, i.e. take its "logarithm" to get a 3-vector pointing along the axis with length equal to the angle. Call these α = log(R_A) and β = log(R_B). The matrix equation collapses to a tidy vector statement: R_X must rotate sensor-2's axis onto sensor-1's axis, i.e. α = R_X β, for every motion. Stack a few such pairs and R_X is over-determined and solvable by least squares (the classic form gives a skew-symmetric linear system in a parameter from which R_X is recovered).

Worked hand-eye rotation solve, with actual numbers

Let's do a clean, fully-numeric case so the abstraction lands. Suppose the unknown extrinsic rotation R_X is a 90° rotation about the z-axis — we will pretend we don't know it and recover it. Take two rig motions:

• Motion 1: the rig rotates about its z-axis. Sensor 1 sees axis α₁ = (0, 0, 1) (pointing up, length = the angle). Because sensor 2 is rotated 90° about z relative to sensor 1, it also sees the z-axis: β₁ = (0, 0, 1). (A rotation about the shared z-axis looks the same to both.)
• Motion 2: the rig rotates about sensor-1's x-axis. Sensor 1 sees α₂ = (1, 0, 0). Sensor 2, rotated +90° about z, sees that same physical axis as its own negative-y direction: β₂ = (0, −1, 0).

We seek the R_X satisfying α_i = R_X β_i for both. From motion 1: R_X(0,0,1) = (0,0,1), so R_X fixes the z-axis — it is a rotation about z. From motion 2: R_X(0,−1,0) = (1,0,0), i.e. R_X sends −y to +x. A rotation about z that maps −y→+x is precisely a +90° rotation about z:

Check it: R_X·(0,−1,0) = (0·0 + (−1)(−1) + 0, 1·0 + 0 + 0, 0+0+0) = (1, 0, 0). ✓. And R_X·(0,0,1) = (0,0,1). ✓. We recovered the 90°-about-z extrinsic from two motions, never touching translation, never measuring anything physically — only by comparing how each sensor reported the same rotations. That is hand-eye calibration. (Two motions about non-parallel axes is the bare minimum; in practice you use dozens for robustness, and that requirement — "non-parallel axes" — is the seed of the observability story in Chapter 6.)

Below, feed the solver paired motions one at a time and watch the estimated R_X tighten onto the true extrinsic as the motions accumulate — and watch it stall if you only give it rotations about one axis.

Target-based vs targetless calibration

Where do the motions A and B come from? Two families:

Target-based. Put a known calibration object in view — a checkerboard or an AprilTag grid — whose geometry you know exactly. Each sensor estimates its pose relative to the target, and from those poses you extract clean motions A and B. This is what Kalibr, the standard camera-IMU calibration toolbox, does. The target makes the per-sensor motion estimates precise, so the extrinsic comes out tight. Cost: you need the target and a controlled session.

Targetless (motion/structure-based). No special object. You drive the rig through the world and recover each sensor's motion from its own odometry — the camera does visual odometry, the LiDAR does scan-matching, the IMU integrates. Then AX=XB again. This is how on-the-fly and field recalibration work (and how online self-calibration in Chapter 5 operates). Cost: the per-sensor motions are noisier, so you need more motion and good excitation.

Hand-eye rotation solve in numpy (verbose)

The rotation solve, written out with the log-of-rotation and a least-squares stack, exactly as derived:

python
import numpy as np

def log_rot(R):
    # axis-angle vector of a rotation matrix: direction = axis, length = angle
    angle = np.arccos(np.clip((np.trace(R) - 1) / 2, -1, 1))
    if abs(angle) < 1e-9:
        return np.zeros(3)                 # no rotation -> useless for hand-eye!
    axis = np.array([R[2,1]-R[1,2],
                     R[0,2]-R[2,0],
                     R[1,0]-R[0,1]]) / (2*np.sin(angle))
    return axis * angle                       # the "log" of R

def solve_handeye_rotation(A_rots, B_rots):
    # Tsai-Lenz: stack alpha = Rx . beta as a skew-symmetric linear system.
    M = []
    for Ra, Rb in zip(A_rots, B_rots):
        a = log_rot(Ra)                       # axis of sensor-1 motion
        b = log_rot(Rb)                       # axis of sensor-2 motion
        # the Tsai constraint: skew(a + b) . x = (a - b)
        s = a + b
        skew = np.array([[0, -s[2], s[1]],
                         [s[2], 0, -s[0]],
                         [-s[1], s[0], 0]])
        M.append((skew, a - b))
    H = np.vstack([m[0] for m in M])      # stack all motions (3 rows each)
    d = np.hstack([m[1] for m in M])
    x, *_ = np.linalg.lstsq(H, d, rcond=None)  # least squares over ALL motions
    # recover Rx from the Cayley parameter x (modified Rodrigues)
    theta = 2*np.arctan(np.linalg.norm(x))
    n = x / (np.linalg.norm(x) + 1e-12)
    K = np.array([[0,-n[2],n[1]],[n[2],0,-n[0]],[-n[1],n[0],0]])
    Rx = np.eye(3) + np.sin(theta)*K + (1-np.cos(theta))*K@K  # Rodrigues
    return Rx

Notice the comment on log_rot: a motion with no rotation returns the zero vector and contributes nothing. That single line is why pure-translation motion is useless for the rotation solve — the exact degeneracy Chapter 6 is about. And the compact one-liner heart of it: axis = log(R), stack skew(a+b)·x = (a−b), lstsq, recover R_X.

Chapter 3: Temporal Calibration — Finding the Clock Offset t_d

Space is half the battle. Even with a perfect R and t, your fusion can still diverge if the two sensors disagree about when. Every sensor stamps its data with a timestamp, but those timestamps come from different clocks, through different driver paths, with different latencies. The result is a constant time offset t_d: when you read "image at t = 1.000 s" and "IMU sample at t = 1.000 s," they may actually correspond to instants 8 ms apart. Temporal calibration estimates t_d so you can shift one stream onto the other.

Why a few milliseconds is catastrophic under motion

We saw the headline number in Chapter 0 (5 ms → 2.5 px). Let's understand why the offset turns into error, because the mechanism tells you how to measure t_d. The key fact: the IMU's angular rate and the image's feature velocity are two views of the same motion, and a time offset slides one against the other.

Concretely, when the camera rotates at angular rate ω, a feature at distance f (in pixels) from the centre sweeps across the image at a pixel velocity of roughly v_px = f · ω. The IMU's gyroscope directly measures ω. So at every instant, "feature is moving this fast in the image" and "gyro says we're rotating this fast" are perfectly correlated — if the two streams are aligned in time. If the image stream is delayed by t_d, the feature you see "now" actually corresponds to the gyro reading from t_d ago. The mismatch produces a position error of:

Stare at that product. The error is the time offset times the feature velocity. At rest (ω = 0) it is zero — the offset is invisible. Under fast motion (large ω) it is large. This is the entire signature of a time-sync error: proportional to motion speed, zero at rest. And it is also the lever for measuring t_d: slide the streams against each other and find the shift that minimizes the disagreement between the gyro-predicted feature velocity and the observed feature velocity.

The time-offset cost, worked by hand

Here is how you actually find t_d. Define a cost that measures how well the IMU-predicted motion agrees with the camera-observed motion after shifting the image stream by a trial offset τ:

Sweep τ over a range of candidate offsets; the τ that minimizes C is your estimated t_d. Let's compute it with toy numbers. Say the true offset is t_d = 10 ms, the rig rotates at ω = 2 rad/s, and f = 500 px, so the true per-sample feature displacement the IMU predicts and the camera sees are aligned only when τ = 10 ms. Evaluate the cost at three trial offsets, where each ms of misalignment leaves a residual of f·ω·|τ − t_d| pixels:

The cost is a clean V (a parabola) bottoming out exactly at τ = 10 ms — the true offset. You did not need to know t_d in advance; you found it as the bottom of the bowl. And notice the bowl's steepness: it scales with f·ω. Faster motion makes the V sharper, so the offset is estimated more precisely. Slow motion makes the bowl flat and the offset hard to pin down. Motion is the signal that lets you measure time offset — the more excitation, the sharper the estimate. (At ω = 0 the bowl is perfectly flat: every τ gives zero cost, and t_d is completely unobservable. Another preview of Chapter 6.)

Play with it. Below, two streams — the gyro's rate and the image feature's velocity — slide past each other as you change t_d; the residual is a V that bottoms at the true offset, and the V gets sharper as you increase motion speed.

The time-offset cost in numpy (verbose + compact)

Finding t_d is a 1-D search over the cost we just tabulated:

python
import numpy as np

def time_offset_cost(tau, t_cam, feat_vel, t_imu, gyro_rate, f=500.0):
    # Shift the camera stream by tau, then compare what the IMU predicts
    # the feature velocity should be (f * gyro) against what the camera saw.
    imu_pred = np.interp(t_cam + tau, t_imu, f * gyro_rate)  # predicted px/s
    residual = feat_vel - imu_pred                           # disagreement
    return np.sum(residual**2)                          # total squared cost

def estimate_td(t_cam, feat_vel, t_imu, gyro_rate, f=500.0):
    taus = np.linspace(-0.05, 0.05, 201)        # search +/- 50 ms
    costs = [time_offset_cost(t, t_cam, feat_vel, t_imu, gyro_rate, f)
             for t in taus]
    return taus[np.argmin(costs)]                    # bottom of the V = t_d

# compact: td = argmin over tau of  sum( feat_vel - interp(t_cam+tau, t_imu, f*gyro) )^2

The whole estimator is "shift, interpolate, compare, take the argmin." The np.interp is doing the slide; the squared sum is the V; argmin reads off the bottom. Online estimators (next two chapters) do this incrementally instead of by brute-force sweep.

Chapter 4: Continuous-Time Calibration — One Smooth Trajectory, Any Timestamp

We have a spatial solver (AX=XB) and a temporal solver (the V-cost). The modern batch-calibration tools — above all Kalibr — solve both at once, and they do it with a beautiful idea from Furgale, Barfoot & Sibley: don't think of the trajectory as a list of poses at discrete timestamps; think of it as one smooth continuous function of time, and represent that function with a B-spline.

The problem discrete representations can't solve

Here is the trap continuous-time solves. The IMU samples at, say, 200 Hz; the camera at 20 Hz; and — this is the whole point — their samples don't line up in time, and once you allow an unknown time offset t_d, you literally don't know which instant any measurement belongs to until you've solved for t_d. A discrete estimator that stores "pose at frame 1, pose at frame 2, ..." has no pose to hand you at the exact, off-grid, t_d-shifted instant a measurement actually occurred. You'd have to interpolate ad hoc, and the interpolation interacts with the unknown you're solving for.

The fix is to make the trajectory a function you can evaluate anywhere. Write the rig's pose as a continuous curve T(t), so that for any time — an IMU instant, a camera instant, a camera instant shifted by a trial t_d — you can just ask "where was the rig at time t?" and get an answer. With a continuous T(t), a measurement at any unsynchronized timestamp relates cleanly to one shared trajectory, and t_d becomes just another parameter you optimize, because shifting a measurement in time simply queries T(t) at a slightly different argument.

Why a B-spline

How do you store a continuous trajectory with finitely many numbers? A B-spline: a smooth curve built from a handful of control points (a.k.a. knots). The curve's value at any time t is a weighted blend of the few nearby control points, with weights given by smooth basis functions. Three properties make it perfect for calibration:

• Evaluate anywhere. T(t) is defined for every real t, not just at sample times — exactly what we needed.
• Local support. Each point on the curve depends only on a few nearby control points, so the optimization stays sparse and fast (like a factor graph).
• Built-in smoothness — and free derivatives. A B-spline is differentiable, so you can analytically read off velocity and acceleration. That matters enormously: the IMU measures acceleration and angular rate, i.e. the derivatives of the trajectory. With a spline you predict the IMU reading by differentiating T(t) — no finite differences, no noise amplification.

The control points c_i become the unknowns of one big optimization, together with the extrinsics (R, t) and the time offset t_d. Every measurement — each gyro sample, each accelerometer sample, each detected checkerboard corner — becomes a residual against this single smooth trajectory. Minimize the sum of squared residuals and out pop a smooth trajectory and the calibration, jointly, optimally. This is the continuous-time batch estimation backbone, and it is what makes Kalibr's camera-IMU calibration the gold standard.

Below, see the spline in action: scattered, off-grid sensor samples (different rates) all attach to one smooth curve you can query at any time — including at a shifted time to test a t_d.

Chapter 5: Online Self-Calibration — Making R, t, t_d Filter States

Batch calibration (Chapter 4) is wonderful but assumes you can stop, wave a target around, and run an offline solver. Real systems drift: a phone warms up and its lens shifts microns; a car hits a pothole and the LiDAR mount flexes; a drone's frame bends in flight. The extrinsics and t_d you measured in the lab are stale by the time it matters. The modern answer is online self-calibration: fold the calibration parameters into the estimator's state and let the running filter (or factor graph) refine them continuously, for free, while it does its normal job.

The idea: calibration parameters are just more state

Recall the state vector of a visual-inertial estimator: position, velocity, orientation, IMU biases. The trick is to append the calibration unknowns to that vector:

The estimator already knows how to estimate state from measurements; now the extrinsics and time offset are part of the state, so the same machinery estimates them too. As the system moves, every measurement that depends on (R, t, t_d) — which is every camera-IMU residual — provides a little information about them, and the filter slowly tightens its estimate. This is exactly what VINS-Mono (Qin & Shen) does: it makes t_d a state variable and estimates it online, so the system stays calibrated even as the offset drifts.

Why making t_d a state works — the linearized handle

For the filter to estimate t_d, it needs to know how a measurement changes when t_d changes — the derivative (Jacobian) of the residual with respect to t_d. We already derived it: a feature's image position moves by f·ω per unit time, so its sensitivity to t_d is precisely its velocity in the image. Qin & Shen's insight is to model a feature's position as a linear function of the time offset, using its measured image velocity as the slope:

Now the residual depends linearly on t_d, with a slope (the image velocity) the system already measures by tracking the feature. That makes t_d an ordinary, observable state the filter can correct each step — provided there's motion, because if the features aren't moving, the slope is zero and the filter gets no information (Chapter 6 again).

Filter vs factor graph — two homes for the same parameters

Where you fold the calibration in depends on your estimator architecture. In a filter (EKF/UKF, Lessons 5–8), you literally extend the state vector and covariance matrix, and the calibration parameters get their own rows/columns — their covariance shrinks as motion gives information, grows (slowly) if you let them drift. In a factor graph (the modern VIO/SLAM backbone), the calibration parameters become nodes shared across many factors, and the optimizer refines them jointly with the trajectory. Either way the principle is identical: a parameter you're uncertain about becomes a variable you estimate.

Watch online calibration converge below: start with a deliberately wrong extrinsic and t_d, run the system, and the estimates walk toward truth — fast when the motion is rich, stalling when the motion is poor.

Chapter 6: Observability — You Cannot Calibrate What You Don't Excite

This chapter is the one that turns a junior calibration engineer into a senior one, because it explains a failure that has no error message. You run the calibration, it converges, it reports a confident number — and the number is garbage, silently. The cause is observability: a parameter is only recoverable if the motion you performed actually excited it. Move the wrong way and the calibration solves for a parameter it has no information about, filling it with noise while reporting success.

What "observable" means, concretely

A calibration parameter is observable under a given motion if changing that parameter would change the measurements you collected. If you could change the parameter and no measurement would notice, then no measurement can tell you its value — it is unobservable, and the solver is free to put anything there. We have already met this three times:

• Rotation extrinsic needs rotations about non-parallel axes (Chapter 2). Rotate only about z and you learn nothing about how the sensors differ about x or y.
• Time offset t_d needs motion — the cost bowl is flat at rest (Chapter 3). No motion, no V, no t_d.
• Online states stop converging without excitation (Chapter 5) — the Jacobian (image velocity) is zero when nothing moves.

The degenerate motions, named

Certain motions leave certain parameters unobservable. Memorize these traps:

The cure is the opposite of all of these: full excitation — rotate about all three axes and translate along all three axes, with varying speed (so there's acceleration too). This is exactly why a Kalibr calibration session asks you to wave the rig through a vigorous, varied 3D dance in front of the target rather than slide it smoothly along a rail. The dance isn't superstition; each degree of freedom of motion makes a different calibration parameter observable.

Below, run two motions side by side: a degenerate one (rotation about a single axis) and a full-excitation one. Watch the parameter's uncertainty ellipse: it stays fat under degenerate motion (unobservable) and collapses under full excitation (observable).

Chapter 7: Use Cases & Real Products — Where Calibration Earns Its Keep

This recurring chapter grounds the theory in the actual tools and systems that depend on it. Calibration is invisible when it works and catastrophic when it doesn't, so the products below all treat it as a first-class engineering problem, not an afterthought.

Kalibr — the standard camera-IMU / multi-cam toolbox

Kalibr (from ETH Zurich's ASL, the lab of Furgale, Rehder, Siegwart) is the de-facto open-source toolbox for exactly this lesson. It performs camera intrinsic calibration, camera-IMU extrinsic calibration (R, t), and temporal calibration (t_d) — all jointly, using the continuous-time B-spline batch estimation of Chapter 4. You wave an AprilTag grid in front of the rig with full excitation; Kalibr fits one smooth trajectory and spits out K, distortion, R, t, t_d, and the residuals to judge the fit. It also calibrates multi-camera rigs (camera-to-camera extrinsics) and IMU intrinsics. If you have ever set up a VIO system, you have almost certainly run Kalibr first.

Autonomous-vehicle multi-sensor rigs

A self-driving car carries cameras, LiDARs, and radars, and fusing them demands every pairwise extrinsic: camera-LiDAR (so a LiDAR point lands on the right pixel for semantic labelling), camera-camera (surround-view stitching), LiDAR-LiDAR (merging multiple spinning units into one cloud), and radar-camera (associating a radar return with a visual object). Target-based methods use checkerboards visible to both modalities; targetless methods exploit edges and planes shared between a LiDAR cloud and an image. Time sync is mandated by hardware triggering — a PTP (Precision Time Protocol) or GPS-PPS master clock so the cameras shutter at known instants relative to the LiDAR spin — precisely to drive t_d toward zero rather than estimate it.

AR/VR headsets — factory plus online calibration

A headset (Quest, Vision Pro, HoloLens) is a camera-IMU rig strapped to a moving head, and a wrong extrinsic or t_d makes virtual objects swim against the world — instant nausea. These devices are factory-calibrated (a precise per-unit offline calibration on the production line) and run online self-calibration (Chapter 5) because the plastic frame flexes with temperature and the user's grip. The combination — a tight factory seed refined online — is exactly the offline-plus-online recipe from Chapter 5. The whole comfort of AR rests on t_d staying sub-millisecond under fast head rotation.

Robot hand-eye calibration for manipulation

The original use case, still ubiquitous: a robot arm with a camera (on the wrist, "eye-in-hand," or fixed overlooking the workspace, "eye-to-hand") must know the camera-to-gripper transform to convert "I see the part here in the image" into "move the gripper there." This is the literal hand-eye problem AX = XB (Chapter 2), solved by Tsai & Lenz's 1989 method and its descendants. Every pick-and-place cell, every visual-servoing robot, every bin-picking system runs hand-eye calibration on setup — and re-runs it whenever the camera mount is bumped.

Chapter 8: Practical Application — Actually Running a Calibration

This recurring chapter is the field manual: how you, hands on keyboard, get a good calibration and know it's good. We'll walk a Kalibr-style camera-IMU session end to end, because if you can do that you can do the rest.

Step 1 — intrinsics first, always

Calibrate each sensor's intrinsics before touching extrinsics. For the camera, photograph an AprilTag grid (or checkerboard) from many angles, distances, and orientations — fill the frame, hit the corners, tilt steeply — and fit K + distortion. For the IMU, characterize its noise (the Allan-variance plot gives you the white-noise and bias-instability parameters Kalibr needs). Garbage intrinsics poison the extrinsics, so this is non-negotiable.

Step 2 — the motion pattern: a vigorous, full-excitation dance

This is where most calibrations are won or lost. Hold the rig and move it through a rich 3D trajectory in front of the target: rotate about all three axes (pitch, roll, yaw — nod, tilt, turn) and translate along all three axes (up-down, left-right, in-out), with varying speed so there's acceleration. Keep the target fully in view the whole time. The motion should look almost silly — a fast, varied wobble, not a smooth glide. Each axis of motion makes a different parameter observable (Chapter 6); skip an axis and you ship an unobservable garbage value. A typical session is 30–90 seconds of this dance.

Step 3 — read the residuals: good vs bad

After the solver runs, the verdict is in the reprojection residuals — for each detected corner, the gap between where the calibration predicts it and where it was actually seen, in pixels. What good looks like vs bad:

The pattern matters more than the magnitude. Random sub-pixel scatter means the model fits and the leftover is just noise. Structured residuals — a swirl, a constant bias, anything speed-correlated — mean a systematic error the model didn't capture: a wrong distortion model, a wrong extrinsic, an unestimated t_d. A residual plot that grows with motion speed is the unmistakable fingerprint of a leftover time offset (Chapter 3).

Step 4 — offline once vs online continuously

Decide your regime. Calibrate offline once when the rig is mechanically stable and you can run a target session — a fixed robot cell, a research platform, factory bring-up. Estimate online when the calibration drifts — consumer devices, thermal environments, long missions, anything that flexes. In practice: do a tight offline calibration to get the seed, then enable online self-cal to track drift around it (Chapter 5). Don't run online with no seed — it can wander off if the initial guess is too far.

Step 5 — validate the calibration

Never trust a calibration you haven't validated independently. Three cheap checks:

• Reproject onto a held-out sequence. Run the calibration on data it wasn't fit to; the residuals should stay sub-pixel and unstructured.
• Cross-modal overlay. For camera-LiDAR, project the LiDAR cloud into the image — points should land precisely on the corresponding edges and surfaces. Misalignment of even a few pixels means a bad extrinsic.
• Motion-speed sweep. Deliberately run the rig slow, then fast. If the estimate degrades under fast motion, you have a residual t_d — the calibration looked fine at the gentle speed you tested it at and failed at the speed that matters.

Chapter 9: Debugging & Failure Modes — Reading the Symptoms

This recurring chapter is a field guide to diagnosis. Calibration errors are sneaky precisely because they all look like "the filter is a bit off." The skill is reading the pattern of the error to name its cause. Each failure below has a distinct fingerprint.

Symptom: bad extrinsics

A wrong R or t injects an error with a geometric structure. The two giveaways:

• Constant-bias residual. A wrong translation t leaves a roughly constant offset between the two sensors' views even at low speed — features consistently land a few pixels off in the same direction. The error doesn't vanish at rest the way a time-sync error does.
• Rotation-rate-proportional residual. A wrong rotation R or translation t creates the phantom velocity ω×r from Chapter 0 — the error grows with rotation rate specifically, and points in a direction tied to the rig's geometry. Plot residual magnitude against |ω| and a wrong extrinsic shows a clear slope.

How to confirm: overlay the two modalities (LiDAR-into-image, or one camera's features into the other's frame). A bad extrinsic shows a consistent spatial misregistration you can literally see. Re-run hand-eye with better excitation; if the extrinsic shifts a lot, the old one was wrong.

Symptom: bad time-sync

A wrong t_d has the most diagnostic fingerprint of all, because of its motion dependence:

• Error proportional to motion speed. Residuals are tiny when the rig is slow and balloon when it's fast (the f·ω·t_d law from Chapter 3). Plot error vs speed — a time-sync error is a straight line through the origin.
• Vanishes at rest. Hold the rig still and the residuals collapse to sensor-noise level. This is the single cleanest test: if the error disappears when you stop moving, it's a clock problem, not a sensor problem.

How to confirm: run the motion-speed sweep (Chapter 8). A monotonic error-vs-speed line that hits zero at zero speed is t_d, full stop. Then estimate t_d (sweep the V-cost or enable online estimation) and watch the speed-dependence vanish.

Symptom: thermal / mechanical drift of extrinsics

The calibration was right, then slowly went wrong. The rig heats up (electronics, sun, friction) and the mount expands; vibration loosens a screw; a payload bends the frame. The fingerprint: a calibration that degrades over time or with temperature, fine after a cold start and bad after an hour. How to confirm: log residuals over a long run and correlate with temperature; if error tracks temperature, it's thermal drift. Fix: online self-calibration (Chapter 5) so the estimate tracks the drift, plus better mechanical/thermal design of the mount.

Symptom: insufficient excitation — the silent garbage

The most dangerous, because there's no obvious symptom at calibration time. The solver converged, reported a number, you shipped it — but a parameter was unobservable and got filled with noise. The fingerprint appears only later: the system works in some conditions and mysteriously fails in others (the conditions that exercise the unobservable parameter). How to confirm: inspect the covariance of every calibration parameter, not just the point estimate. A huge variance on a parameter means it was never observed — the point estimate is meaningless. Fix: recalibrate with full excitation (Chapter 6, Chapter 8).

The cardinal error: mistaking calibration error for sensor noise

This deserves its own heading because it wastes more engineer-weeks than any other. Both extrinsic and time-sync errors masquerade as fat sensor noise, so the reflex is to inflate the noise covariances Q and R and move on. This "works" — the filter stops complaining — while hiding a structural bug. The tell that it's calibration, not noise: real sensor noise is random and motion-independent; calibration error is structured and motion-proportional. If your "noise" gets worse when you move faster, or shows a consistent direction, it is not noise — it is calibration, and no covariance tuning will truly fix it.

Chapter 10: Connections, References & Cheat-Sheet

You now hold the silent prerequisite for every other lesson in this series. The filters you built in Lessons 5–10 assumed a common frame and a common clock; this lesson is how that assumption gets earned. Everything spatial that comes after — visual-inertial odometry, LiDAR-inertial odometry, multi-sensor SLAM — begins with the calibration you just learned to find.

Where this sits in the series

This is Lesson 15 — the bridge from the classical filters into the modern spatial-fusion systems. The recurring structure (silent bug → concepts → Use Cases, Practical, Debugging, Connections) is the same one you've walked since Lesson 1.

Related lessons on Engineermaxxing

• Extended Kalman Filter — the filter whose state you extend to carry R, t, t_d for online self-calibration; the Jacobian machinery you need to fold calibration in.
• The Kalman Filter — the optimal complementary fuser that silently assumes a common frame and clock; this lesson earns that assumption.
• EKF (deep dive) — the nonlinear estimation underneath every online-calibration scheme.
• The Bayes Filter — the belief-update foundation; calibration parameters are just more variables in the belief.

Cheat-sheet — the whole lesson on one card

References

1. Furgale, P., Rehder, J., and Siegwart, R. "Unified Temporal and Spatial Calibration for Multi-Sensor Systems." IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 1280–1286, 2013. The paper behind Kalibr: jointly estimates extrinsics and the time offset using a continuous-time formulation. doi:10.1109/IROS.2013.6696514
2. Furgale, P., Barfoot, T. D., and Sibley, G. "Continuous-Time Batch Estimation Using Temporal Basis Functions." IEEE International Conference on Robotics and Automation (ICRA), pp. 2088–2095, 2012. Introduces the B-spline trajectory representation that lets you evaluate sensor poses at arbitrary, unsynchronized times. doi:10.1109/ICRA.2012.6225005
3. Qin, T. and Shen, S. "Online Temporal Calibration for Monocular Visual-Inertial Systems." IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 3662–3669, 2018. Makes the time offset t_d a state variable estimated online, as in VINS-Mono. doi:10.1109/IROS.2018.8593603
4. Tsai, R. Y. and Lenz, R. K. "A New Technique for Fully Autonomous and Efficient 3D Robotics Hand/Eye Calibration." IEEE Transactions on Robotics and Automation, vol. 5, no. 3, pp. 345–358, 1989. The foundational hand-eye (AX = XB) solution, decoupling rotation from translation. doi:10.1109/70.34770