A self-driving car carries three sensors that lie in completely different ways — a camera that sees color but not distance, a LiDAR that knows distance but not color, a radar that feels velocity through fog. The whole game is deciding where inside a deep network to glue them together, and a top-down grid called the Bird's-Eye View turned out to be the answer.
You are in a self-driving car on a wet highway at dusk. A truck ahead kicks up spray; the low sun glares straight into the windshield. Three sensors are watching the road, and right now each one is failing in its own private way.
The camera — the sensor that sees the world the way you do, in dense color and texture — is half-blinded by the sun. Its image is a wash of bloom and lens flare exactly where the truck is. And even when the sun isn't in its eyes, the camera has a permanent handicap: it produces a flat picture. A flat picture has no metric distance baked into it. A camera cannot, from one frame, tell you the truck is 30 meters away rather than 60 — that number simply is not in the image.
The LiDAR — the spinning laser that fires millions of beams and times their echoes — knows distance to the centimeter. It does not care about the sun. But the spray from the truck is scattering its beams, and at the truck's range its points are sparse: a few dozen returns draped over a vehicle the camera renders in a hundred thousand pixels. LiDAR also has no idea what color the truck is, or whether that smear of points is a truck or a billboard.
The radar — the oldest of the three, bouncing radio waves off metal — sails through the spray and glare untouched, and it alone reports something the others cannot measure at all: the truck's velocity, directly, from the Doppler shift of the returning wave. But radar's picture of where the truck is, sideways, is blurry — its angular resolution is coarse, so it can tell you "something metal, closing at 4 m/s, roughly ahead" but not the truck's precise shape or lane.
This lesson is a pivot point in the series. Lessons 1–17 were about fusing state estimates — positions, velocities, scalars and small vectors — with filters: the Kalman filter, the EKF, covariance intersection, data association. That machinery is optimal when each sensor hands you a clean number with a known uncertainty. But a camera does not hand you a number. It hands you a million pixels. A LiDAR hands you a point cloud. The quantity we want — "where are all the objects, and what are they?" — has to be extracted from that raw flood by a deep network, and the new question is: at what point inside that network do we combine the three modalities? That single question — where to fuse — is the spine of modern autonomous-driving perception, and it is the spine of this lesson.
It is worth feeling, concretely, why a camera alone cannot give you distance, because it motivates everything that follows. A pinhole camera projects a 3-D point at distance Z and height Y onto an image at pixel height y through the rule y = f · Y / Z, where f is the focal length in pixels. Watch what that Z in the denominator does to your ability to recover distance.
Take a focal length f = 1000 px. A 1.5-meter-tall pedestrian (Y = 1.5 m) at Z = 30 m projects to a height of y = 1000 × 1.5 / 30 = 50 px. Now take a 3.0-meter-tall object (Y = 3.0 m, say a tall sign) at Z = 60 m: it projects to y = 1000 × 3.0 / 60 = 50 px — the identical pixel height. The camera sees two boxes of the same size and cannot tell which is the near pedestrian and which is the far sign, because the image only ever shows the ratio Y/Z, never Y and Z separately.
This is the scale ambiguity of monocular vision, and it is not a hardware defect to be engineered away — it is the geometry of projection itself. A camera measures direction beautifully and distance not at all. A LiDAR measures distance directly (it times a laser pulse) but at a tiny fraction of the camera's angular density. The whole reason to fuse is that the depth the camera is missing is exactly the depth the LiDAR is overflowing with.
Before we name anything, let's watch the three sensors degrade independently. Below, a top-down driving scene: your car at the bottom, an obstacle ahead. Toggle each sensor's health — sun-blind the camera, drown the LiDAR in rain, leave the radar (which shrugs off both). Watch which information survives.
A top-down view of your car (bottom) and an obstacle ahead. Toggle each sensor's health. The bars on the right show what information remains available: semantics/color (camera), precise depth (LiDAR), velocity (radar). Notice no single sensor covers all three rows — only the union does.
Sun-blind the camera and you lose what the object is (its color, its class) but radar and LiDAR still know it's there and how far. Drown the LiDAR and you lose precise depth but the camera still recognizes it and radar still clocks its speed. The three rows of information are carried by different sensors, and the union is always richer than any single column. That union is what fusion is for.
To fuse sensors well you must know, precisely, what each one measures, in what shape, with what failure modes. Vague intuitions ("the camera is good, LiDAR is better") get you nowhere. This chapter lays out the three modalities as data streams — their physical principle, their output shape and type, and the exact conditions in which they betray you. Memorize this table; it is the reason all three exist on the same roof.
Camera — dense passive imaging. A grid of light-sensitive pixels records the brightness and color of incoming light. Output: a dense array of shape roughly H×W×3 (height × width × RGB) — for a 1920×1080 camera that's about 6 million numbers per frame, 30–60 times a second. It is the only sensor that gives semantics: lane markings, traffic-light color, the text on a sign, whether that shape is a pedestrian or a mailbox. Cheap (a few dollars), high-resolution, and the data the world already understands.
LiDAR — active 3-D ranging. A laser fires pulses and times the echo; distance = (speed of light × round-trip time) / 2. Spinning, it sweeps the scene. Output: a point cloud — a list of (x, y, z, intensity) tuples, typically 100k–200k points per sweep at 10–20 Hz. Each point is a precise metric 3-D location. It works in total darkness (it brings its own light) and gives the geometry the camera lacks. Expensive, mechanically complex, and sparse: those 100k points must cover a whole 360° scene, so a distant car gets only a handful.
Radar — active radio ranging with velocity. A radio wave bounces off objects; the time delay gives range, and crucially the Doppler shift of the returned frequency gives the object's radial velocity directly — how fast it is moving toward or away from you, measured, not inferred. Output: a sparse set of detections, each with (range, azimuth, radial velocity, signal strength), a few hundred per scan. Radio waves are long enough to pass through rain, fog, snow, and dust that scatter both light and laser, so radar is the all-weather sensor. Its catch: poor angular resolution — it can tell you something metal is closing at 4 m/s roughly ahead, but not its precise lateral position or shape.
Read this top to bottom and the design strategy writes itself: each sensor's weakness column is another sensor's strength column.
| Property | Camera | LiDAR | Radar |
|---|---|---|---|
| Physical principle | passive light | active laser (ToF) | active radio (Doppler) |
| Output shape | dense H×W×3 image | sparse (x,y,z,i) cloud | sparse (r,θ,v,rcs) list |
| Semantics / color | excellent | none | none |
| Metric depth | none (ambiguous) | excellent, cm-level | coarse (range only) |
| Direct velocity | no (must infer) | no (must infer) | yes, Doppler |
| Angular resolution | very high | high | low |
| Range | medium | ~50–200 m | long, 250 m+ |
| Night / darkness | fails (low light) | works (own light) | works |
| Sun glare | fails (bloom) | works | works |
| Rain / snow / fog | degrades | scatters badly | works |
| Cost | cheap ($) | expensive ($$$) | moderate ($$) |
Notice that no row is green for all three, and no column is green for all rows. That is the whole point: there is no single best sensor, only a best ensemble. The camera owns semantics and angular resolution; LiDAR owns metric depth; radar owns velocity, range, and all-weather robustness. A perception system that wants to recognize objects (camera), localize them precisely in 3-D (LiDAR), and know their speed in any weather (radar) needs all three, because the union of their green cells covers the road and no single column does.
It is worth pausing on one famous disagreement, because it sharpens what the table means. Tesla's perception stack is camera-only (with radar historically, then removed) — eight cameras, no LiDAR. The argument: humans drive with two eyes (cameras) and no laser, so a sufficiently good network should recover depth from cameras alone, and LiDAR is an expensive crutch. Nearly every other serious AV program — Waymo, Cruise, Mobileye's premium stack, Chinese players like NIO and Xpeng — uses camera + LiDAR + radar. Their argument: the camera's depth ambiguity (Chapter 0) is structural, and on a safety-critical system you want a sensor that measures depth directly rather than inferring it, especially in the glare-and-spray conditions where inference is hardest. We'll return to this in Chapter 7; for now, note that the debate is precisely about whether you can infer the LiDAR's green cells from the camera, or must measure them.
To make the shapes concrete, here is what one frame from each sensor looks like as data your network will consume:
python # CAMERA: a dense pixel grid (H x W x 3), values 0-255 image = np.zeros((1080, 1920, 3), dtype=np.uint8) # ~6.2M numbers # LIDAR: a sparse list of 3-D points (N x 4): x, y, z, intensity (metres) points = np.array([[12.3, -1.1, 0.4, 0.7], # one return ~12 m ahead [30.8, 2.0, 1.2, 0.3], # the truck, sparse at range # ... ~150,000 rows total ]) # RADAR: a sparse list of detections (M x 4): range, azimuth, radial_vel, rcs radar = np.array([[31.2, 0.06, -4.1, 12.0], # 31 m, ~ahead, closing 4.1 m/s # ... ~few hundred rows ])
Three streams, three shapes — a dense grid, two sparse lists. They do not even live in the same coordinate frame yet (the image is in pixels, the clouds in meters), which is exactly the calibration and time-sync problem of Lesson 15. Before any fusion can happen, every stream must be expressed in one common 3-D frame — and the choice of which common frame is the subject of Chapter 4.
We have three streams that must become one scene. A deep perception network is a pipeline: raw data flows in at the left, gets transformed into features in the middle, and emerges as detections (boxes, classes, velocities) at the right. The fusion question is brutally simple to state and surprisingly deep to answer: at which point along that pipeline do we merge the three modalities? There are three canonical answers, and they trade off the same three axes every time.
This is the levels of fusion idea from Lesson 1 (raw / feature / decision), now made concrete for deep perception. The names shift slightly — the field says early, mid (or middle/feature), and late — but the meaning is identical: how far into the processing has each stream traveled before we combine it?
Where: at the very front, before either stream has been processed into features. Data flow: you take raw (or barely-processed) LiDAR points and raw image pixels, register them into a common frame, and feed the combined raw representation into a single network. The canonical example, which we'll build in detail next chapter, is PointPainting: project each LiDAR point into the camera image and "paint" it with the camera's per-pixel semantic scores, so every point now carries (x, y, z, intensity, plus the camera's class probabilities). One enriched point cloud goes into a standard LiDAR detector.
The deal: early fusion preserves the maximum information — nothing has been thrown away yet, so the network can learn the richest possible cross-modal correlations. The price: it is exquisitely sensitive to misalignment. To paint a point with the right pixel, the point's projected image location must be correct to within a pixel or two — which demands near-perfect calibration (the exact geometric transform between LiDAR and camera) and near-perfect time synchronization (both sensors must have captured the same instant; a moving object shifts between a LiDAR sweep and a camera frame taken milliseconds apart). Get either wrong and you paint points with the wrong pixels — the topic of Lesson 15, and a failure mode we'll see in Chapter 9.
Where: in the middle, after each stream has been processed into features but before final detections. Data flow: the camera images go through a vision backbone to produce camera feature maps; the LiDAR cloud goes through a point/voxel backbone to produce LiDAR feature maps; the two feature maps are projected into a shared space (almost always the Bird's-Eye View, Chapter 4) where they spatially align, and then concatenated or summed and processed jointly into detections.
The deal: this is the modern sweet spot, and most state-of-the-art driving detectors live here (BEVFusion and its kin). Features are abstracted enough that small spatial errors blur out rather than corrupt — a feature map is more forgiving of a half-meter misalignment than raw painting is — yet they retain far more information than final boxes. You can also fuse asymmetric sensors gracefully: a dense camera feature grid and a sparse LiDAR feature grid meet as peers in the BEV. The price: it needs more engineering than the other two (you must design the shared space and the projection into it), and it is still somewhat calibration-dependent, though far less brittle than early.
Where: at the very end. Data flow: each sensor runs its own complete detector independently — the camera produces its list of 3-D boxes, the LiDAR produces its list, the radar produces its list — and then a final stage associates and combines these per-sensor detections into one fused track list. This last step is exactly the data-association and covariance-intersection / track-to-track fusion machinery from Lessons 11–12: match the camera's "car at (30, 2)" to the LiDAR's "car at (30.4, 1.8)," decide they're the same object, and fuse their estimates.
The deal: late fusion is the most robust and most modular. Each sensor pipeline is independent, so you can swap, upgrade, or lose a whole modality without retraining the others, and a calibration error only shifts where boxes land, not what gets painted onto what. It degrades gracefully under dropout: lose the camera and the LiDAR detector keeps producing boxes. The price: it throws away the most cross-modal detail. By the time you're combining final boxes, you've discarded the rich pixel-and-point correlations that early/mid fusion could exploit. If the camera alone couldn't confidently see a faint pedestrian, late fusion never gets the chance to combine that faint camera evidence with faint LiDAR evidence into a confident joint detection — each sensor had to decide alone first.
Every fusion-level choice is a point on these three trade-off axes. Read across:
| Axis | Early / data | Mid / feature | Late / decision |
|---|---|---|---|
| Information retained | most (raw, nothing lost) | much (features) | least (final boxes only) |
| Robustness to misalignment | fragile (needs pixel-perfect calib + sync) | moderate (features blur small errors) | robust (only box positions shift) |
| Modularity / swap a sensor | low (one entangled network) | medium | high (independent pipelines) |
| Graceful dropout | poor | good (if trained for it) | excellent |
| Compute / bandwidth | high (raw streams) | medium | low (compact decisions) |
| Where SOTA lives today | niche (PointPainting era) | here (BEV fusion) | safety backup layer |
There is a clean tension here. As you move left to right — early → mid → late — you trade information for robustness and modularity. Early fusion is information-maximal and alignment-fragile; late fusion is information-poor but bulletproof and swappable. Mid fusion sits in the middle by design, which is precisely why it has become the default for production driving stacks: enough information to learn rich correlations, enough abstraction to tolerate the inevitable calibration drift of a car that's been on the road for a year.
Watch the three merge points on a single pipeline. Drag the merge point and see how much of each stream has been processed before they meet.
Two pipelines (camera top, LiDAR bottom) flow raw → features → detections. Click a stage to move the FUSE point. Read the panel: earlier fusion keeps more information but demands tighter alignment; later fusion is more robust and modular but discards cross-modal detail.
Let's make early fusion completely concrete by building the cleanest example of it: PointPainting (Vora et al., CVPR 2020). The idea is so simple it's almost embarrassing, and that simplicity is exactly why it's the perfect teaching example. A LiDAR point cloud is rich in geometry but blind to semantics; a camera image is rich in semantics but blind to depth. PointPainting decorates each LiDAR point with the camera's semantic verdict for the pixel that point lands on. You get a point cloud where every point now knows not just where it is but what it's part of.
The magic is in step 2—3: turning a 3-D point into the pixel it corresponds to. That projection is pure geometry, and we'll do it by hand. Input shapes: a point cloud of shape N×4 and an image segmentation of shape H×W×C (C class scores per pixel). Output shape: a painted cloud of shape N×(4+C). Nothing about the count of points changes; each just grows a few extra columns.
Take a single LiDAR point at X = (8, 1, 0.5) meters in the LiDAR frame — 8 m ahead, 1 m to the left, 0.5 m up. We want the pixel (u, v) it projects to. Projection has two steps: first transform the point into the camera's coordinate frame using the extrinsic calibration, then project onto the image plane using the camera's intrinsics. To keep the arithmetic clean, suppose the calibration has the camera and LiDAR aligned (no rotation) with the camera 0.2 m to the right and the usual camera-frame convention (z forward, x right, y down). The point in the camera frame becomes:
(Here zcam = 8 is the forward distance, xcam = 0.8 is rightward after the 0.2 m offset, ycam = −0.5 because the point is above center and image y grows downward.) Now apply the pinhole intrinsics — focal length f = 1000 px, principal point at (cu, cv) = (960, 540) (image center of a 1920×1080 image):
So our 3-D point lands at pixel (1060, 477). We look up the segmentation network's scores at that pixel — say it returns [car: 0.91, pedestrian: 0.02, road: 0.01, ...] — and we append that vector to the point. The point, formerly (8, 1, 0.5, intensity), is now (8, 1, 0.5, intensity, 0.91, 0.02, 0.01, ...). It is a LiDAR point that knows it's part of a car. Do this for all 150,000 points and the LiDAR detector that consumes the painted cloud can lean on both geometry and the camera's recognition power.
Here is the catch that defines early fusion. The projection above assumed perfect calibration. Suppose the extrinsic is off by a small rotation — the LiDAR is mounted 0.5° rotated from where we think. At zcam = 8 m, a 0.5° angular error shifts the projected pixel by roughly z · tan(0.5°) in meters ≈ 8 × 0.0087 = 0.07 m, which at f = 1000 over 8 m is f · 0.07 / 8 ≈ 8.7 pixels. Eight pixels doesn't sound like much — but if our point was near the edge of the car in the image, an 8-pixel shift can move it onto the road pixels beside the car. Now we paint a genuine car-point with "road" scores. The detector receives a point cloud where the car's near edge is mislabeled road, and it may shrink or miss the box.
And it gets worse for moving objects under poor time sync. If the LiDAR sweep and the camera frame were captured 50 ms apart, a car closing at 20 m/s has moved 1 m between them. The LiDAR point is at the car's true position; the image pixel it projects to shows where the car was 50 ms ago — possibly empty road now. We paint a real car-point with "road." Early fusion's maximum information comes bundled with maximum sensitivity to the exact calibration-and-sync problem of Lesson 15.
Play with it: paint the points correctly, then dial in a calibration offset and watch good points get the wrong color.
LiDAR points (dots) are projected into the camera image and painted with the segmentation color at their landing pixel: green = car, grey = road. The shaded region is the camera's "car" segment. Drag the calibration offset to shift the projection — watch car-points get mis-painted as road (and vice-versa). At zero offset every point is painted correctly.
The whole projection-and-paint loop is short. Watch the 3-D point become a pixel become a color:
python import numpy as np def paint_points(points, seg_scores, T_cam_lidar, K): # points : (N, 4) -> x, y, z, intensity in the LiDAR frame # seg_scores : (H, W, C) -> per-pixel class scores from the image segmenter # T_cam_lidar: (4, 4) -> extrinsic: LiDAR-frame -> camera-frame (calibration) # K : (3, 3) -> camera intrinsics (focal length, principal point) H, W, C = seg_scores.shape N = points.shape[0] painted = np.zeros((N, 4 + C)) # grow each point by C columns painted[:, :4] = points # keep the original x,y,z,intensity for j in range(N): # 1) LiDAR point -> homogeneous -> camera frame (the extrinsic step) p_lidar = np.array([points[j,0], points[j,1], points[j,2], 1.0]) p_cam = T_cam_lidar @ p_lidar # (x_cam, y_cam, z_cam, 1) if p_cam[2] <= 0: # behind the camera -> can't be painted continue # 2) project onto the image plane (the intrinsic step) uvw = K @ p_cam[:3] # (f*x + cu*z, f*y + cv*z, z) u = int(uvw[0] / uvw[2]) # u = f*x/z + cu v = int(uvw[1] / uvw[2]) # v = f*y/z + cv if 0 <= u < W and 0 <= v < H: # inside the image? # 3) PAINT: copy the pixel's class scores onto the point painted[j, 4:] = seg_scores[v, u] return painted # (N, 4+C): geometry + semantics, fused
And the compact version — vectorized, no Python loop, the way it actually ships:
python def paint_fast(points, seg, T, K): P = (T @ np.c_[points[:,:3], np.ones(len(points))].T).T # (N,4) camera frame uvw = (K @ P[:,:3].T).T # (N,3) uv = (uvw[:,:2] / uvw[:,2:]).astype(int) # (N,2) pixels ok = (uvw[:,2] > 0) & (uv[:,0]>=0) & (uv[:,0]<seg.shape[1]) \ & (uv[:,1]>=0) & (uv[:,1]<seg.shape[0]) sc = np.zeros((len(points), seg.shape[2])) sc[ok] = seg[uv[ok,1], uv[ok,0]] # gather scores return np.c_[points, sc] # (N, 4+C)
Mid-level fusion needs a shared space where camera features and LiDAR features can be laid down side by side and spatially aligned. The breakthrough of the last few years — the single idea that reorganized the entire field — is that the right shared space is a top-down grid: the Bird's-Eye View (BEV). Once you see why, modern driving perception clicks into place.
A BEV is exactly what it sounds like: the scene as seen from directly above, like a god's-eye map of the road around the car. Concretely, it's a 2-D grid over the ground plane — say a 100 m × 100 m square around the ego vehicle, divided into cells of 0.5 m × 0.5 m, giving a 200×200 grid. Each cell holds a feature vector describing what's at that patch of ground. The car sits at the center; everything is measured in real meters relative to it.
Three properties make BEV the right canvas, and each one solves a problem that plagued earlier fusion:
1. It is metric. A BEV cell at row 40, column 120 corresponds to a fixed, known patch of ground in meters. Distances and sizes are real and consistent — a car is 2 m × 5 m whether it's near or far, unlike in an image where it shrinks with distance. Detections, planning, and tracking all want metric space, so BEV is the frame they already speak.
2. It is ego-centric and shared. Every sensor can be expressed in the same top-down grid around the car. The LiDAR cloud drops straight into it (just bin the points by their x,y). The camera features get lifted into it (Chapter 5). The radar detections plot directly onto it. All three modalities, all in one grid, in one set of coordinates.
3. Features align spatially. This is the killer property. In the image, a pixel and a LiDAR point at the same world location sit at different coordinates (one in pixels, one in meters) — you can't just stack them. In the BEV, a camera feature and a LiDAR feature describing the same patch of road land in the same grid cell. Now fusion is trivial: concatenate or add the feature vectors in each cell. The hard alignment problem dissolves because the BEV grid is the alignment.
Bonus: no scale ambiguity. Recall the camera's curse (Chapter 0): in the image, near-small and far-large look identical. In the BEV, every object sits at its true ground location and true size — the ambiguity is resolved the moment you've placed features in metric top-down space. (Placing camera features there correctly is exactly the depth problem Lift-Splat solves next.)
A BEV feature map is a tensor of shape (C, H, W) — C feature channels over an H×W spatial grid. For our 100 m range at 0.5 m resolution, H = W = 200. The mapping from a world point (x, y) in meters to a grid cell (row, col) is just an offset-and-scale:
For a point 12 m ahead and 3 m left, with xmin = −50, ymin = −50, cell = 0.5: col = round((12 + 50)/0.5) = round(124) = 124, row = round((−3 + 50)/0.5) = round(94) = 94. The point drops into cell (94, 124). Every sensor's data lands in the grid by this same rule — that shared rule is what makes the cells align.
The LiDAR-to-BEV step is almost trivial: bin every point by its (x, y) into a cell and summarize (max height, point count, max intensity, or a small learned feature). The camera-to-BEV step is the hard one — a pixel has direction but no depth, so which cell does it belong to? That is the entire subject of Chapter 5. Radar plots directly: each detection's (range, azimuth) gives an (x, y), and its Doppler velocity becomes a per-cell feature.
Below, watch three modalities populate one shared BEV grid. Toggle each and see them land in the same coordinate frame, ready to be stacked.
A top-down grid centered on your car (bottom-center). Toggle each modality and watch it drop into the same metric grid: LiDAR occupancy (binned points), camera features (lifted, Chapter 5), radar velocity. Where all are on, the fused cells glow brighter — that's information stacking in a shared cell.
We have our shared canvas (BEV) and the easy modalities drop right in. The one stubborn problem is the camera: a pixel knows its direction but not its depth (Chapter 0), so we don't know which BEV cell along that direction it belongs to. Lift-Splat-Shoot (Philion & Fidler, ECCV 2020) is the elegant answer, and it became the bridge that made camera–LiDAR BEV fusion possible. Its trick is to stop pretending the network knows the depth, and instead let it be honest about its uncertainty.
The name is the algorithm, in three verbs:
A camera pixel corresponds to a ray shooting out into the world; the object could be anywhere along it. Instead of guessing one depth, LSS predicts, for each pixel, a probability distribution over a set of discrete depths — say depths from 4 m to 45 m in 1 m bins, giving D ≈ 41 possible depths. For a pixel, the network outputs a vector like "depth 10 m: 0.6, depth 11 m: 0.3, depth 12 m: 0.1, ..." — it admits it isn't sure, and spreads its bet.
It also computes a feature vector for that pixel (C channels, the semantic content). LSS then forms, at each of the D depths along the ray, a 3-D point whose feature is the pixel feature scaled by that depth's probability. Mathematically this is an outer product: pixel feature vector c (length C) × depth distribution α (length D) gives a (D × C) block — D points along the ray, each carrying a fraction of the feature proportional to how likely that depth is.
Let's lift a single pixel. Say its feature vector (we'll use C = 2 channels for hand-arithmetic) is c = [4, 10], and the network predicts a depth distribution over D = 3 candidate depths {8 m, 9 m, 10 m} of α = [0.2, 0.5, 0.3] (it's most confident the object is at 9 m, but unsure). The outer product α ⊗ c gives the feature deposited at each depth:
| Depth along ray | αd | feature = αd · c = αd · [4, 10] |
|---|---|---|
| 8 m | 0.2 | [0.2×4, 0.2×10] = [0.8, 2.0] |
| 9 m | 0.5 | [0.5×4, 0.5×10] = [2.0, 5.0] |
| 10 m | 0.3 | [0.3×4, 0.3×10] = [1.2, 3.0] |
The pixel's feature has been smeared along its ray: most of it (the [2.0, 5.0] block) lands at 9 m where the network is most confident, with lighter copies at 8 m and 10 m. Note the bookkeeping check: the three feature-blocks sum to [0.8+2.0+1.2, 2.0+5.0+3.0] = [4.0, 10.0] = c — the total feature is conserved, just redistributed by confidence. Now we know the 3-D location of each of these three points (pixel direction × depth), so we can place them in the BEV grid.
Each lifted 3-D point falls into some BEV cell (by the offset-and-scale rule of Chapter 4). Splatting means: for each BEV cell, sum the features of all lifted points that land in it. Many pixels' rays, at many depths, will deposit features into the same cell — splatting accumulates them. The result is the camera's contribution to the BEV feature grid, with shape (C, H, W) — exactly the same shape as the LiDAR's BEV grid, so they can be stacked.
Continuing the example: suppose the 9 m point (feature [2.0, 5.0]) lands in BEV cell (94, 124), and a different pixel's ray also deposited [1.0, 0.5] into that same cell. The cell's camera feature becomes their sum: [2.0+1.0, 5.0+0.5] = [3.0, 5.5]. The grid cell now holds the accumulated camera evidence for that patch of ground. (Efficient implementations use a "cumsum trick" to do this summation fast over millions of lifted points — that engineering is what made LSS practical.)
With camera features now splatted into the BEV grid, you have a top-down camera feature map. "Shoot" is the original paper's term for the downstream planning step (shooting out candidate trajectories in BEV), but for fusion the important thing is simpler: the camera now lives in the BEV grid, aligned with LiDAR and radar, and you fuse by stacking and processing the combined grid — exactly the mid-level fusion of Chapter 4.
Play with the depth distribution. A sharp distribution puts the feature in one cell; a flat one (high uncertainty) smears it across many.
One camera pixel (left) shoots a ray into the BEV (right). The network's depth distribution (the bars) decides how the pixel's feature is spread along the ray and splatted into BEV cells — bright cell = lots of feature. Drag depth confidence: high confidence → a sharp peak → feature lands in one cell; low confidence → flat → feature smears across many cells.
The per-pixel lift (outer product) and the splat (scatter-add into the grid) are the whole algorithm:
python import numpy as np def lift_splat(feat, depth_dist, ray_xy, grid_HW, bounds): # feat : (P, C) -> a C-dim feature for each of P pixels # depth_dist : (P, D) -> a depth probability distribution per pixel (sums to 1) # ray_xy : (P, D, 2)-> the (x,y) BEV position of pixel p at depth bin d # grid_HW : (H, W) -> BEV grid size; bounds = (x_min, y_min, cell) P, C = feat.shape D = depth_dist.shape[1] H, W = grid_HW x_min, y_min, cell = bounds bev = np.zeros((H, W, C)) # the BEV feature grid to fill for p in range(P): for d in range(D): # LIFT: this depth bin gets the pixel feature scaled by its probability # (the outer product alpha_d * c, one row of depth_dist[p] (x) feat[p]) contrib = depth_dist[p, d] * feat[p] # (C,) -- a fraction of the feature # which BEV cell does pixel p at depth d fall into? x, y = ray_xy[p, d] col = int(round((x - x_min) / cell)) row = int(round((y - y_min) / cell)) if 0 <= row < H and 0 <= col < W: # SPLAT: accumulate (sum) into the cell bev[row, col] += contrib return bev # (H, W, C): camera features, now in BEV
And the compact, vectorized core — the outer product as a single einsum, the splat as a scatter-add:
python def lift_splat_fast(feat, depth_dist, cell_idx, H, W, C): # outer product over all pixels at once: (P,D,C) = depth_dist (x) feat lifted = np.einsum('pd,pc->pdc', depth_dist, feat).reshape(-1, C) flat = cell_idx.reshape(-1) # (P*D,) target cell per lifted point bev = np.zeros((H * W, C)) np.add.at(bev, flat, lifted) # scatter-add = the splat return bev.reshape(H, W, C)
Time for the payoff. Everything so far — the three modalities, the three fusion levels, the BEV grid, the depth-honest lift — comes together in one interactive scene. You'll drive a top-down highway with camera, LiDAR, and radar all watching an obstacle ahead, pick your fusion level (early / mid / late), and then drop a sensor (sun-blind the camera, rain-out the LiDAR) to watch how each fusion level degrades. The whole lesson's thesis — that where you fuse decides how your system survives failure — becomes visible.
Top-down scene: your car (bottom) and an obstacle (the box ahead). Pick a fusion level. Then drop sensors — sun-blind the camera or rain-out the LiDAR — and press Run. The detection quality (position, class, velocity) and its failure behavior depend on the level you chose. Early fusion is richest but most fragile; late fusion is coarsest but most robust.
Play with it and the three fusion levels announce themselves through their failure behavior:
The crucial contrast is steps 2 and 3 against each other, and steps 4 and 5. The same failure — a calibration error, or a dead camera — produces opposite outcomes depending purely on where you chose to fuse. Early fusion buys the richest perception with the most fragility; late fusion buys robustness by giving up cross-modal richness; mid fusion in BEV is the engineered compromise that production stacks converged on.
Every lesson in this series ends with the same four chapters — Use Cases, Practical Application, Debugging, Connections — because a concept you can't point at in a shipped product is a concept you don't really own yet. So before any more theory, let's ground multimodal fusion in real autonomous-driving stacks, benchmarks, and the famous sensor debate.
Waymo — camera + LiDAR + radar, fused in BEV. Waymo's vehicles carry a full suite: multiple LiDARs (a tall spinning unit plus close-range units), surround cameras, and radars. Their perception fuses all three, heavily in the BEV/3-D space, precisely because the safety case demands measured depth (LiDAR) and all-weather velocity (radar) on top of the camera's semantics. Waymo is the flagship argument for "measure depth, don't infer it."
Cruise — camera + LiDAR + radar. Like Waymo, Cruise's robotaxis ran a multi-sensor stack with LiDAR at the core for 3-D localization of obstacles. The design philosophy is the same: redundant, complementary modalities so no single sensor failure or condition blinds the car.
Mobileye — two independent subsystems by design. Mobileye's premium approach builds a camera-only perception system and a separate radar+LiDAR system, each capable of driving alone, then cross-checks them — a deliberate late/competitive fusion at the subsystem level for true redundancy (lose one whole subsystem, the other still drives). This is the Lesson-1 competitive-redundancy idea applied to entire perception stacks.
NIO, Xpeng & the Chinese AV wave. The leading Chinese EV makers ship LiDAR + camera + radar fusion in consumer cars, with BEV-based perception networks for highway and urban assisted driving. They are among the largest deployers of mid-level BEV fusion in shipping vehicles.
The sharpest disagreement in the field, and it maps exactly onto this lesson. Tesla runs camera-only (eight cameras, radar removed) on the bet that a good enough network can infer depth from cameras — "humans drive with eyes, so should the car," and LiDAR is an expensive crutch. Everyone else serious — Waymo, Cruise, Mobileye, the Chinese players — uses LiDAR because the camera's depth ambiguity (Chapter 0) is structural, and on a safety-critical system you want to measure depth, not infer it, especially in the glare-and-spray conditions where inference is hardest. The technical core of the debate is precisely the camera depth problem of Chapter 0: can you reliably recover the LiDAR's metric depth from cameras alone? Interestingly, the BEV/lift-splat machinery of this lesson is what makes the strongest camera-only BEV detectors possible — so the same depth-distribution idea powers both camps.
| Benchmark | Sensors | What it measures | Why it matters |
|---|---|---|---|
| nuScenes | 6 cameras, 1 LiDAR, 5 radars | 3-D detection & tracking (NDS metric) | the standard multimodal fusion benchmark; full sensor suite incl. radar |
| Waymo Open | 5 cameras, 5 LiDARs | 3-D detection, motion forecasting | large-scale, high-quality LiDAR; the camera+LiDAR fusion proving ground |
| KITTI | 2 cameras (stereo), 1 LiDAR | 3-D detection (the classic) | where PointPainting and early fusion were first benchmarked |
| Argoverse | cameras + LiDAR | detection, forecasting, HD maps | rich map + sensor data for urban driving |
nuScenes deserves special note: it is the benchmark that includes radar alongside camera and LiDAR, making it the proving ground for genuinely three-modality fusion. The leaderboard's top entries are overwhelmingly mid-level BEV fusion methods — the BEVFusion family and its descendants — which is the empirical reason this lesson spends its weight on BEV.
You don't need a robotaxi to find multimodal fusion — it's in nearly every new car sold. Automatic Emergency Braking (AEB) and adaptive cruise control fuse a forward camera (recognizes that it's a car/pedestrian, reads lane lines) with a forward radar (measures range and closing velocity in any weather). This camera+radar pair is a textbook complementary-plus-competitive fusion: the camera classifies, the radar ranges and clocks velocity, and a detection confirmed by both triggers braking with high confidence. It's the most widely deployed multimodal fusion on Earth, in tens of millions of vehicles.
You know the modalities, the levels, the BEV, and the lift-splat. Now turn it into a procedure you can run when a real driving-perception design lands on your desk. This recurring "now build it" chapter converts "I have these three sensors and a 3-D detection task" into "fuse at this level, in this space, tuned this way, here's why."
The default answer for a modern stack with decent calibration and on-vehicle compute is mid-level fusion in BEV — it's the accuracy/robustness sweet spot, which is why the nuScenes leaderboard is full of it. Reach for late when you need a modular, fault-tolerant safety layer (Mobileye-style), and for early only when your calibration is excellent and you're chasing the last points of accuracy.
If you fuse in BEV, two numbers define your grid and they trade off directly against each other and against compute:
Concrete: a ±50 m range at 0.5 m cells is a 200×200 grid (40,000 cells). Want 0.25 m cells for finer pedestrians? Now it's 400×400 = 160,000 cells — 4× the compute. Want 100 m range for highway? 400×400 again. You can't have huge range and fine resolution and cheap compute — pick two. Most stacks use coarser cells at long range and finer near the car (or a non-uniform grid) to spend resolution where it matters.
LSS's depth distribution can be learned implicitly (only the final detection loss trains it) or explicitly (you also supervise the depth distribution against the LiDAR's true depths). The practical finding: explicit depth supervision helps a lot. Since you have LiDAR on the vehicle anyway, project the LiDAR depths into the camera and use them as ground truth for the depth distribution — the camera branch learns much sharper, more accurate depth, which puts features in the right BEV cells. This is a beautiful instance of the modalities teaching each other: LiDAR supervises the camera's depth, and the fused result beats either alone.
The three sensors don't tick together: a camera at 30 Hz, a LiDAR at 10 Hz, a radar at ~20 Hz, none phase-aligned. Naively fusing the latest frame of each fuses data captured at different instants, and for a fast-moving object that's a real spatial error (Chapter 3's time-sync problem). The fixes: (1) timestamp everything and pick a common reference time; (2) motion-compensate — use ego-motion (and tracked object velocity, which radar gives you directly!) to propagate each sensor's data to the reference time; (3) work at the slowest rate or interpolate. This is the same out-of-sequence / timing machinery the classical chapters of this series formalized — it never goes away just because the fusion is now a neural net.
Here is the single most important robustness technique, and it's cheap. If you train a fusion network with all sensors always present, it learns to lean on the easiest one (usually LiDAR), and when that sensor drops out at inference the whole thing collapses (the modality-imbalance failure of Chapter 9). The fix is modality dropout: during training, randomly zero out a whole modality on some fraction of samples. Forced to sometimes detect with no LiDAR, sometimes no camera, the network learns each modality's independent contribution and degrades gracefully when one fails for real. It's the training-time version of the Chapter 6 dropout sim — you practice the failures so the deployed system survives them.
Knowing how fusion should work is half the battle; the other half is recognizing the specific ways a real multimodal driving system breaks. These failures are rarely loud crashes — they are subtle: a ghost object, a car detected at the wrong range, a network quietly ignoring its weaker sensor. This recurring chapter catalogs the classic failure modes, the symptom each presents, and the test that exposes it.
The signature failure of early fusion. If the LiDAR-to-camera calibration is off, or the sensors are out of time sync, PointPainting projects points onto the wrong pixels — painting real objects with the wrong semantics, and worse, painting empty regions with object semantics. The result is ghost objects: confident detections where there's nothing, and missed or shrunken boxes where there's something.
In a camera-only (or camera-dominant) BEV system, the lift-splat depth distribution is your depth estimate — and if it's wrong, features land in the wrong BEV cell, so objects appear at the wrong distance. A pedestrian the network thinks is at 25 m but is actually at 15 m is a potential collision the planner won't see coming. This is the Chapter-0 ambiguity biting back: the camera guessed depth and guessed wrong.
The most insidious failure, because the system looks great in testing. A fusion network trained with all sensors always present learns to lean on whichever modality is easiest (usually LiDAR for 3-D detection), and quietly ignores the others. You have "fusion" on paper but a single-sensor system in practice — and the moment the dominant sensor drops out, the whole thing collapses, even though two healthy sensors sit right there.
The network was trained mostly on clear daytime data and now faces heavy rain or night. The camera degrades (low light, raindrops on the lens), the LiDAR scatters in rain, and a network that never learned to rely on radar in these conditions detects poorly — in exactly the conditions where reliable detection matters most.
Late fusion combines per-sensor detections, which means it inherits the data-association problem (Lesson 12): you must decide that the camera's "car at (30, 2)" and the LiDAR's "car at (30.4, 1.8)" are the same object before fusing them. Get the association wrong and you either merge two different objects into one (missing a real obstacle) or split one object into two (phantom doubling), both dangerous.
| Check | How | Catches |
|---|---|---|
| Calibration overlay | Project LiDAR points onto the camera image — do they land on objects? | Failure 1 (ghosts) |
| Depth-vs-LiDAR audit | Compare camera-BEV depths against LiDAR / temporal consistency | Failure 2 (wrong range) |
| Per-modality ablation | Kill each modality alone; measure the drop (should be graceful) | Failure 3 (imbalance) |
| Per-condition eval | Split validation by weather/time-of-day, not aggregate | Failure 4 (domain shift) |
| Association audit | Visualize late-fusion matches; check gating thresholds | Failure 5 (mis-association) |
You now hold the modern half of the fusion story — perception-level fusion of camera, LiDAR, and radar, and the BEV revolution that organized it. Here is everything in one place: the cheat-sheet to screenshot, the motto, the real sources, and the cross-links.
Screenshot this:
| Level | What's combined | Info | Robustness | Canonical method |
|---|---|---|---|---|
| Early | raw points + raw pixels | most | fragile (calib/sync) | PointPainting |
| Mid | feature maps in shared BEV | much | good (blurs small errors) | BEV fusion (Lift-Splat) |
| Late | per-sensor detections/tracks | least | most robust + modular | track-to-track (Lessons 11–12) |
This was Lesson 18 of 22 — the pivot from classical state-estimation fusion (filters) to modern perception-level fusion (deep nets). You've seen the three modalities, the three levels, the BEV grid, and the lift-splat construction that bridges camera and LiDAR. Next up, Lesson 19: Transformer / BEV Fusion — where the splat-by-geometry of Lift-Splat is replaced (or augmented) by attention: instead of geometrically projecting features into BEV, a transformer lets each BEV query attend to the relevant image and LiDAR features and pull them in (the BEVFormer / cross-attention lineage). The depth-distribution idea you learned here becomes a learned attention map, and the fusion becomes fully end-to-end.
These existing lessons go deeper on machinery we touched: