RGB-D Scene Classification — When Depth Reveals the Room

Chapter 1: The Task and Its Datasets

Let us state the task precisely. RGB-D scene classification takes a registered pair — a color image of shape (3, H, W) and a depth map of shape (1, H, W) where every depth pixel reports the distance to the surface at the matching color pixel — and outputs one label from a fixed taxonomy of indoor scene categories. The output is a probability distribution over K scene classes, and we read off the most probable one, exactly as in plain RGB scene classification. The new ingredient is the second, geometric channel.

"Registered" is the load-bearing word. It means the two channels are aligned: pixel (i, j) in the depth map describes the same physical point as pixel (i, j) in the color image. Real sensors capture color and depth from slightly different positions, so the raw streams must be warped into alignment before use — the datasets ship pre-registered so you can treat them as a stacked 4-channel image conceptually, even though we will almost never feed them that way.

Two datasets define this field, and you must know both:

NYU Depth v2 (from Nathan Silberman and colleagues at NYU) is the small, beloved starting point: about 1,449 indoor RGB-D pairs with dense per-pixel labels, drawn from categories like bedroom, kitchen, living_room, bathroom, and office. Because it is small, models trained on it alone overfit easily — which is why the from-scratch-depth finding in Chapter 3 was so surprising.

SUN RGB-D (from Song, Lichtenberg, and Xiao at Princeton) is the modern benchmark: about 10,335 RGB-D images captured across four different depth sensors, with a scene-classification split that most papers report on (commonly 19 categories with enough examples). Mixing four sensors is deliberate — it forces models to handle the different noise and resolution characteristics of each device, which is exactly the robustness a deployed system needs.

Notice what these taxonomies have in common: they are indoor. Beaches and forests are easy from color; the hard, depth-hungry problem is telling apart the rooms a robot actually drives through. Let us do a quick worked sanity-check on dataset scale, because the smallness of these sets shapes everything that follows.

The widget below lets you scrub through the two datasets' class lists and see, per class, roughly how many RGB-D pairs you get to train on. Watch how thin the per-class data is — that scarcity is the constraint every design choice in this lesson fights against.

Chapter 2: What Exactly Does Depth Add

It is easy to say "depth helps." Let us be precise about what it adds, because every benefit we name will later justify a design choice. Depth contributes three things that color simply does not contain.

Benefit 1: absolute scale. A color image is scale-ambiguous — a dollhouse kitchen and a real kitchen photograph identically, because color has no units. Depth has units: meters. So a depth map tells you the room is 4 m wide, the ceiling is 2.6 m up, the table is 0.7 m tall. Scale alone separates a cramped hotel room from a spacious master bedroom, a closet from a hallway, a model from the real thing. This is the single biggest thing depth knows that color cannot.

Benefit 2: surface layout and 3D structure. From depth you can recover which pixels are floor, wall, and ceiling, and how surfaces are oriented — their surface normals, the direction each tiny patch of surface faces. A floor faces up, a wall faces sideways, a tabletop faces up at a certain height. This layout — "big horizontal surface low down (floor), two vertical surfaces meeting (corner), small horizontal surface at table height" — is a fingerprint of room type that color textures cannot provide.

Benefit 3: robustness to appearance. Color is hostage to lighting, paint, and clutter. A kitchen at night, a kitchen painted red, a kitchen full of party guests — the colors change wildly, but the geometry (counter height, cabinet layout, the open box of the room) is stable. Depth is invariant to exactly the nuisances that wreck color, so it carries complementary, more stable evidence.

Here is the unifying idea: color tells you what materials and objects are present; depth tells you the shape and scale of the space they live in. Indoor scenes are defined as much by their spatial configuration as by their contents, so depth is not a small bonus — it is half the definition of the room.

Let us turn one benefit into arithmetic, because "depth gives scale" should be concrete. A depth sensor reports distance Z at each pixel. With the camera's focal length f (in pixels) and the pixel coordinates, you back-project a pixel to a real 3D point. The horizontal real-world distance X for a pixel that sits u pixels from the image center is X = Z · u / f. Let us compute it.

The widget makes this tangible: a side-view of a room with adjustable ceiling height and far-wall distance, with the recovered "room box" drawn from those depth-derived measurements. Slide the geometry and watch the inferred scene category change — small box reads "hotel/closet," tall open box reads "open room/lobby."

Chapter 3: Encoding Depth — Raw vs HHA

We have a depth map. How do we feed it to a CNN? This is the most important and most counterintuitive chapter in the lesson, because the obvious answer is the wrong one and the field's clever answer pays off twice.

The obvious answer: raw depth — take the single-channel meters-per-pixel map, maybe normalize it to 0–1, and shove it into a CNN as a grayscale image. This works, but it leaves value on the table. A raw CNN must learn, from scarce data, to compute geometry like "is this pixel floor?" or "how high above the ground is this?" — facts that are not directly visible in the raw distance values. With only ~435 images per class, that is a lot to ask.

The clever answer is HHA encoding (from Saurabh Gupta and colleagues), which turns the one-channel depth map into a three-channel image where each channel is a hand-computed geometric quantity:

The name is the three channels: Horizontal disparity, Height above ground, Angle with gravity. The genius is that HHA pre-computes the geometry the network would otherwise have to learn. It also makes the depth signal look like a natural three-channel image — which means you can initialize the depth-CNN with weights pretrained on ImageNet RGB, transferring useful low-level filters (edges, blobs) into the depth stream.

Now the famous finding, and the reason this chapter exists. Gupta et al. and follow-up work observed something startling on NYU/SUN RGB-D: a depth-CNN trained from scratch (or with HHA) on the depth channel can match or beat a CNN that was fine-tuned from ImageNet on the RGB channel. A from-scratch net beating a pretrained one? Three reasons resolve the paradox:

Reason 1 — the domain gap. ImageNet RGB features are tuned to natural-color textures (fur, sky, foliage). A depth/HHA map has none of that — it is smooth surfaces and geometry. So an ImageNet-pretrained net applied to depth is starting from features that do not transfer well; its head start is mostly irrelevant.

Reason 2 — depth is simpler to learn. Geometry has less variability than color appearance (no lighting, no texture, no paint). With less to learn, a from-scratch depth-CNN can succeed on small data where a from-scratch RGB-CNN would overfit. The HHA channels hand it the hard geometry for free, so it only has to learn the easy combination.

Reason 3 — complementary, cleaner signal. For indoor scenes specifically, geometry is often the more discriminative cue, and it is less corrupted by nuisance variation. A model reading clean geometry can outscore one reading messy, ambiguous color.

Below, toggle between raw depth and the three HHA channels for the same room. Watch how the floor, table, and ceiling pop out in the height channel, and how walls separate from floors in the angle channel — structure that is invisible in the flat raw-depth grayscale.

Chapter 4: Two-Stream RGB + Depth Encoders

We have two channels with two different jobs: color reads materials and objects, depth reads geometry and scale. The natural architecture follows directly from that split — give each its own network. This is the two-stream network: one CNN encoder for RGB, a separate CNN encoder for depth (or HHA), each producing its own scene descriptor, which are then combined.

Why two streams instead of one network eating a stacked 4-channel image? Because, as Chapter 3 showed, RGB and depth have different statistics and different best-pretraining. The RGB stream wants ImageNet-RGB initialization and learns color-texture filters; the depth stream wants HHA input and its own (possibly from-scratch) filters tuned to smooth geometry. Forcing them to share one set of early filters makes each worse at its job. Separate streams let each specialize.

Let us trace the data flow with shapes, because the shapes are the design. The RGB image (3, 224, 224) flows through the RGB backbone to a feature map, which global-average-pools to an RGB descriptor — say a vector of length 512. In parallel, the HHA depth image (3, 224, 224) flows through the depth backbone to its own 512-vector. Now we hold two 512-dim descriptors, one per modality, each summarizing the whole image from its sensor's point of view.

The two descriptors then meet in a fusion step (the whole of Chapter 5), and the fused result feeds a final linear classifier and softmax. The key structural fact: the streams run independently until fusion, so each can be trained, initialized, and even swapped out separately. You can take a strong off-the-shelf RGB scene model as one stream and pair it with a small depth net as the other.

Let us put one number on "two streams beat one" with a toy. Suppose the RGB stream alone gets a class right 70% of the time and the depth stream alone 65%, and — crucially — their errors are somewhat independent because they read different physics. If the two were perfectly independent and we accept a prediction when either is right, the chance both are wrong is 0.30 · 0.35 = 0.105, so "either right" = 89.5%. Real fusion is not that generous (errors are correlated), but the principle holds: complementary, partly-independent streams cover each other's mistakes.

The widget lets you set each stream's solo accuracy and how independent their errors are, and watch the fused accuracy. Crank independence up and fusion soars; set the errors fully correlated and the second stream becomes useless — a vivid lesson in what makes a second modality worth its compute.

Chapter 5: Fusion — Early, Late, and Cross-Attention

The two streams must meet. Where they meet is the fusion strategy, and it is the single most-studied design choice in RGB-D recognition. Three strategies dominate, defined by how early or late the modalities mix.

Early fusion combines the raw inputs (or very early features) before most processing — e.g. stack RGB and depth into a 4-channel (or 6-channel with HHA) image and run one network. It lets the model learn joint low-level patterns from pixel one, but it forces the two modalities to share early filters (the Chapter-4 problem) and to be perfectly registered. Early fusion shines when the channels are tightly aligned and their low-level interaction matters.

Late fusion runs the two streams fully separately to two predictions (two probability distributions, or two descriptors), then combines them at the end — average the probabilities, sum the log-probabilities, or concatenate descriptors before a final classifier. It is simple, robust, lets each stream use its own best pretraining, and degrades gracefully if one sensor fails. Its cost: the modalities never interact at the feature level, so a cue that requires jointly seeing color and depth (e.g. "a shiny surface at table height") is harder to capture.

Cross-attention fusion is the modern middle ground: let the streams talk to each other at the feature level via attention. Each modality's features attend to the other's — depth features can ask "which color regions are at this geometry?" and color features can ask "what is the shape of this textured region?" — producing fused features that are genuinely joint. It is the most powerful and the most data-hungry and compute-heavy.

Because late fusion is the workhorse — the first thing to try and the most robust — let us do it by hand. The clean way to combine two independent classifiers is to add their log-probabilities (equivalently, multiply the probabilities). Why log-prob and not raw average? Multiplying probabilities is the Bayesian "AND of independent evidence": a class wins only if both streams find it plausible, so adding logs lets each stream veto a class the other liked. Let us compute it.

The widget lets you set the two streams' distributions and watch all three fusion rules side by side: simple average, log-prob sum (product), and a learned weighted blend. See how log-prob sum lets a confident stream override a hesitant one, while a naive average sometimes washes the signal out.

Chapter 6: Why RGB-D Is Genuinely Hard

Depth is powerful, but it is not a magic wand, and a competent engineer respects its failure modes. Three forces make RGB-D scene classification hard even with the second sensor — and each one shapes how you build and evaluate the system.

Force 1: indoor scenes share objects and geometry. The same problem that plagues RGB scenes does not vanish with depth. A study room and a small office both have a desk, a chair, and a wall — similar objects and similar geometry. Depth helps most where rooms differ in scale or layout (bedroom vs hotel), but where two rooms share both contents and shape, even RGB-D is stuck. Depth is a strong cue, not an oracle.

Force 2: depth sensors are noisy and incomplete. This is the big one. Consumer depth cameras fail in characteristic ways: missing pixels (holes) where the sensor got no return — shiny, dark, transparent, or distant surfaces reflect or absorb the projected pattern and report no depth. Glass windows, mirrors, black TVs, and anything past the sensor's range come back as zero or garbage. A raw depth map can be 10–30% holes. If your model treats those zeros as "distance = 0 meters" it will hallucinate a wall in your face.

Force 3: depth noise and resolution. Even where it returns a value, depth is noisier than color, and the noise grows with distance — a structured-light sensor's error scales roughly with the square of the range, so a far wall's depth is far less trustworthy than a near table's. Depth maps are also often lower-resolution and blurry at object edges (the "flying pixels" between foreground and background). The geometry you compute (heights, normals) inherits all of this.

The practical upshot is a pipeline step you cannot skip: depth completion / hole-filling. Before HHA or any geometry, you fill the missing pixels — classically with a cross-bilateral filter guided by the color image (colorization-style inpainting), or with a learned depth-completion network. Skip it and your height and angle channels are corrupted exactly where the holes are.

The widget injects realistic sensor failures into a depth map — raise the hole rate and the distance-dependent noise — and shows how the recovered geometry (and the resulting prediction) degrades, then how hole-filling partially rescues it. This is the gap between a clean benchmark depth map and a real robot's sensor.

Chapter 7: Showcase — Toggle Depth, Watch the Confusion Resolve

Everything converges here. This is a working two-stream RGB-D classifier you can drive and break. It holds five candidate indoor scenes — bedroom, hotel_room, kitchen, office, living_room — and two streams: an RGB stream reading color/object cues, and a depth stream reading geometry cues. You set the image's features with the sliders, choose whether depth is fused in, and watch the live distribution and the top-1 call.

The RGB features are the kind of object/material cues color sees: bed-like furniture, cooking surfaces, desk/work setup. The depth features are the geometric cues depth sees: room scale (small→large), ceiling height. No single feature decides a scene; the combination across both modalities does — and depth's job is to break the ties color cannot.

Three things to try, each replaying a chapter:

• Make an ambiguous bedroom/hotel. Set high "bed furniture", and with depth OFF watch bedroom and hotel_room sit in a near-tie (Chapter 0's coin-flip). Now flip depth ON and set "room scale" small with a low ceiling — the depth stream votes hotel and late fusion resolves the tie. You have reproduced the whole premise live.
• Toggle depth on and off repeatedly. Watch the confidence of the top call jump when depth is fused — the gestalt of this lesson in one button.
• Crank the depth-sensor noise. This corrupts the depth stream the way holes and range-noise do (Chapter 6). Watch the fused call degrade toward the RGB-only answer — a vivid demo of why robust fusion must lean on the trustworthy modality.

If you removed this simulation, would you lose understanding? Yes — reading "depth resolves the bedroom/hotel tie" is abstract until you watch the two bars cross the instant you fuse a small-room depth reading, and watch the gain melt as you add sensor noise. The napkin drawing of this whole lesson is exactly this: color in, depth in, fuse, and let geometry break the ties.

Chapter 8: Evaluation — Measuring the Depth Gain

You understand the models; here is how to judge them honestly. RGB-D scene classification has its own evaluation culture, and three ideas cover most of it: mean per-class accuracy, the depth-gain breakdown, and the confusion matrix.

Mean per-class accuracy, not overall accuracy. RGB-D datasets are imbalanced — bedroom and living_room have many images, some classes have very few. Overall accuracy (fraction of all images correct) is dominated by the big classes; a model can ignore rare classes and still score well. So the standard metric is mean class accuracy: compute accuracy within each class, then average those numbers equally. A rare class counts as much as a common one. SUN RGB-D papers report mean class accuracy precisely because of this imbalance.

The depth-gain breakdown is the RGB-D-specific evaluation. The whole reason to add depth is the lift it provides, so you must measure it per class: report RGB-only accuracy, RGB-D accuracy, and the difference, broken down by scene. The pattern is always the same and always instructive — depth gives a large gain on geometry-decided classes (bedroom vs hotel, hallway vs room) and a small or zero gain on classes color already nails (kitchen by its appliances, bathroom by its fixtures). Reading this breakdown tells you exactly where your second sensor is earning its keep.

Here is the mean-class metric and the depth-gain breakdown in code — first from scratch, then the one-liner spirit:

python
import numpy as np

# preds, labels: (N,) predicted / true class indices; K classes
def mean_class_accuracy(preds, labels, K):
    accs = []
    for c in range(K):
        mask = (labels == c)                 # images whose TRUE class is c
        if mask.sum() == 0: continue
        accs.append((preds[mask] == c).mean())  # accuracy WITHIN class c
    return np.mean(accs)                      # average classes equally — imbalance-fair

# depth-gain breakdown: per-class lift from adding depth
def depth_gain(preds_rgb, preds_rgbd, labels, K):
    for c in range(K):
        m = (labels == c)
        a_rgb  = (preds_rgb[m]  == c).mean()
        a_rgbd = (preds_rgbd[m] == c).mean()
        print(c, "rgb", round(a_rgb,2), "rgbd", round(a_rgbd,2), "gain", round(a_rgbd-a_rgb,2))

Notice the metric averages classes, not images — that single choice is what makes the number honest on imbalanced data. The confusion matrix tells you which errors and whether depth fixed them. Compare the RGB-only confusion matrix to the RGB-D one: the off-diagonal mass on geometry-decided pairs (bedroom↔hotel_room) should shrink when you add depth, while pairs color already separated barely move. A good RGB-D model's improvement is visible as specific confusion cells deflating — that is depth doing its job, made visible.

The interactive below shows the per-class depth-gain breakdown. Each row is a scene; the two bars are RGB-only and RGB-D accuracy, and the highlighted gap is the depth gain. Tap a row to read its story — notice the gain is large exactly where geometry decides the class.

Chapter 9: Connections & Where It's Used

RGB-D scene classification is a hub, not an island. The pattern you built — read two complementary channels, encode each well, fuse them, and respect each sensor's failure modes — recurs across vision, robotics, and remote sensing. Naming those connections turns one lesson into a map.

The RGB sibling. Everything here sits on top of plain RGB scene classification: the holistic "name the whole place" task, global average pooling, top-1 vs top-5, and the label-ambiguity ceiling. If any of that felt shaky, see Visual Scene Classification — this lesson is that one plus a depth channel, and the depth channel is the entire story.

Other modalities, same shape. The two-stream-and-fuse pattern is everywhere. Acoustic scene classification asks "is this audio a park, a metro, an airport?" — whole-signal classification with the same top-k culture, over a spectrogram. Remote-sensing scene classification labels satellite tiles (forest, residential, industrial), often fusing optical bands with elevation (a depth-like geometric channel) — the same RGB-plus-geometry fusion, bird's-eye. Each is "this lesson, new sensor."

Beyond depth maps: point clouds. A depth map is a 2.5D view — geometry from one camera position. Lift it to full 3D and you get a point cloud, a set of (x, y, z) points you can classify with point-cloud networks. RGB-D is the gateway to 3D understanding; the same geometry that disambiguated the room here becomes explicit 3D coordinates there. See NeRF & 3D Gaussian Splatting for where dense geometry leads.

The fusion machinery. The attention used in cross-attention fusion is the same mechanism taught in Vision Transformer and The Transformer; the broad two-encoders-one-decision pattern underlies Multimodal Foundation Models and Multimodal RAG; and the convolutional backbones in each stream are the subject of vision architectures generally. RGB-D is where you first feel why multimodal fusion is worth its complexity: a coin-flip becomes a confident call.

Cheat sheet

"Geometry is not true, it is advantageous." — Henri Poincare. In RGB-D scene classification, geometry is exactly the advantage: the meters color cannot see are what tell the bedroom from the hotel room.