A holistic CNN squints at the whole room and guesses "bedroom." But a bedroom and a hotel room are nearly the same pixels. What if we also handed the model an explicit graph of the objects it can see — bed next to nightstand, lamp above desk — and let it reason over the relations? That second view is exactly what pushed MIT Indoor67 past 90%, and we are going to build both branches by hand.
You have built a perfectly respectable scene classifier. It is a convolutional neural network — a network that slides learned filters over the image and builds up features — finished with global average pooling (averaging each feature channel over the whole image) and a softmax. On Places365 it does fine. Then you point it at a stream of indoor photos and the failures start lining up in a suspicious pattern.
It calls a hotel room a bedroom. It calls a bookstore a library. It calls a fast-food counter a cafeteria. Look at the pairs: every confusion is between two scenes that contain almost identical objects. A bedroom and a hotel room both have a bed, a nightstand, a lamp, a window, maybe a desk. The holistic CNN sees the same soup of textures and shapes and cannot reliably tell them apart.
Here is the thing the CNN is throwing away. The objects are not the whole story — their arrangement and co-occurrence are. A hotel room has a bed plus a desk plus a luggage rack plus a TV-on-a-dresser, in a particular tidy, matched configuration. A home bedroom has a bed plus a dresser plus personal clutter. The discriminating signal lives in which objects appear together and how they relate spatially. Global average pooling, by design, blurs that signal into a single 512-number average. It deliberately forgets where things were and what was next to what.
So a question forms, and it is the question this entire lesson answers: what if, alongside the holistic CNN, we explicitly detect the objects, build a graph where each object is a node and nearby objects are connected by edges, and run a second little network over that graph? The CNN reads the texture of the whole place. The graph network reads the structure of the objects. Then we fuse the two. That second branch is a graph neural network (GNN) — a network that operates on nodes and edges instead of a pixel grid.
This is not a hypothetical. Two-branch CNN+GNN scene classifiers are exactly the approach that pushed MIT Indoor67 — the hardest, indoor-only scene benchmark, where every scene shares objects — from the high 80s to above 90% accuracy. The holistic branch alone could not do it. The object graph was the missing view.
Toggle the widget below. The same indoor photo is read two ways: the holistic view (one blurry global descriptor, easily confused) and the object-graph view (explicit nodes and relation edges that disambiguate). Notice that you, a human, used both instinctively — you felt the overall vibe of the room and noticed "oh, there's a luggage rack, that's a hotel." We are going to give the machine both eyes.
Toggle between the holistic CNN view (one global descriptor — bedroom and hotel look the same) and the object-graph view (nodes + relation edges that separate them). Same pixels, two completely different signals.
Over the next nine chapters we will build this two-branch machine from zero. We will recap the CNN branch and its GAP descriptor; detect objects and turn them into graph nodes; build one message-passing step of the GNN branch by hand with real arithmetic; fuse the two descriptors with concatenation and then with a smarter gated fusion; understand precisely why this helps indoor scenes; drive a live two-branch classifier you can break; and face the tradeoffs (you now need an object detector, and it can lie to you). Let us start by making the two views precise.
Let us name the two views precisely, because the whole architecture is just "compute both and fuse." A scene, to this model, is described by two complementary descriptors that capture different kinds of information.
View one: the holistic appearance. This is the global texture, color, layout, and lighting of the whole image — the gestalt, the overall feel. It is what a CNN with global average pooling produces: a single vector summarizing "how much of each visual pattern is present anywhere in the frame." It is excellent at the easy cases (a beach looks nothing like a kitchen) and it is location-blind on purpose. We will call its output the holistic descriptor, a vector we will denote h.
View two: the object-relation structure. This is an explicit list of the objects present and how they relate: "bed connected to nightstand connected to lamp; desk connected to chair." We represent it as a graph — a set of nodes (one per detected object, each carrying a feature vector) and edges (connecting objects that are spatially close or plausibly related). A GNN turns this graph into a second vector, the graph descriptor g, that summarizes the object structure.
Why two views and not one super-view? Because they fail in different places, and a fusion of two differently-failing models is more robust than either alone. The holistic view fails when two scenes share global texture (bedroom vs hotel). The object view fails when the detector misses or there are few objects (an empty hallway). When one is uncertain, the other often is not. This is the same logic as an ensemble, but the two members are deliberately built to be complementary, not just two copies of the same idea.
Concretely, the data flow is two parallel pipes that meet at the end. The image goes into both. Pipe A (the CNN) produces h, say a 512-vector. Pipe B (detector + GNN) produces g, say a 256-vector. A fusion step combines them into one vector, and a final linear-plus-softmax classifier turns that into a scene probability. The genius is in the parallelism: each branch is free to specialize.
Let us put a tiny number on "complementary." Suppose for a hard image the holistic branch alone is right 70% of the time and the object branch alone is right 65% of the time. If their errors were identical, fusing gives you 70% — no gain. But if their errors are independent, the chance both are wrong on the same image is roughly 0.30 × 0.35 = 0.105, so a fusion that defers to whichever is confident can be right up to about 89.5% of the time. That gap — 70% to ~90% — is, almost exactly, the MIT Indoor67 story. Complementary errors are where the accuracy comes from.
The widget below sketches the two-pipe architecture live. Slide "holistic confidence" and "graph confidence" to simulate each branch being sure or unsure on a given image, and watch the fused prediction track whichever branch is more confident — and stay correct even when one branch collapses. This is the entire promise of fusion in one picture.
Drag each branch's confidence in the true class. The fused bar (right) defers toward the more confident branch. Notice fusion stays high even when one branch falls apart — that's complementary failure paying off.
Branch A is the familiar one, and we recap it carefully because its output shape is what the fusion step will eventually consume. The job of this branch is to turn an image into one fixed-length vector that summarizes the whole scene's appearance — the holistic descriptor h.
The input is a tensor of shape (3, 224, 224): three color channels, 224 pixels tall and wide. A backbone like ResNet passes it through blocks of convolutions; each block shrinks the spatial size and grows the channel count. After the last block you hold a feature map of shape (512, 7, 7): 512 learned feature channels, each a tiny 7×7 spatial grid. Each channel is a pattern detector — "vertical edges here", "warm wood texture there", "repeating grid over here" — and the 7×7 grid records where that pattern fired.
Now the move that defines this branch: global average pooling (GAP). Collapse each 7×7 channel to a single number by averaging its 49 values. The (512, 7, 7) map becomes a (512,) vector. Each entry says "how much of pattern c is present anywhere in the image", with the location deliberately discarded. For a holistic scene descriptor this is exactly right: a kitchen is a kitchen whether the stove is left or right. That same location-blindness is also precisely the limitation Chapter 0 complained about — and the reason we will bolt on the graph branch.
Let us watch GAP run on real numbers, because the arithmetic is the whole operation. Take one channel as a 3×3 grid (3×3 instead of 7×7 just to keep it readable):
After GAP, a single fully-connected layer could map h straight to scene logits — that would be a plain CNN classifier. In the two-branch model we instead hold onto h as the holistic descriptor and pass it to the fusion step. So the entire CNN branch is: image → conv backbone → (512, 7, 7) feature map → GAP → h = (512,). That vector is branch A's complete contribution.
The simulation below makes the location-blindness visceral. Paint activation into the feature grid; GAP collapses it to one number; then drag the same activation to a different spot. Watch the GAP value not budge — the descriptor is identical for "pattern in the corner" and "pattern in the center." That invariance is GAP's gift and its curse.
Click cells to light up a channel's activation. The bar shows the GAP value. Use "Move pattern" to slide the lit cells — the GAP value does not change. That's the holistic branch keeping "how much" and discarding "where".
Here is the CNN branch as code — the from-scratch GAP, then the one-line library version — so the shape change is concrete:
python import numpy as np # feature_map: (C, H, W) = output of the conv backbone, e.g. (512, 7, 7) def cnn_branch_descriptor(feature_map): h = feature_map.mean(axis=(1, 2)) # GAP -> (C,) one number per channel return h # the holistic descriptor h; location discarded on purpose
python # Library version: torchvision backbone, GAP, take the pooled vector as h import torch, torchvision.models as models backbone = torch.nn.Sequential(*list(models.resnet18(weights="DEFAULT").children())[:-2]) gap = torch.nn.AdaptiveAvgPool2d(1) # == our global average pool fmap = backbone(image_batch) # (B, 512, 7, 7) h = gap(fmap).flatten(1) # (B, 512) the holistic descriptor
Branch B begins where branch A refuses to look: at the individual objects. Before we can run a graph network, we need a graph, and before we have a graph we need its nodes. The nodes come from an object detector — a network (think YOLO, Faster R-CNN, or DETR) that returns a list of detected objects, each with a class label, a confidence, and a bounding box (the rectangle x, y, width, height locating it in the image).
Each detected object becomes one node in our graph. But a node needs a feature embedding — a vector describing it — not just a class name. Where does that vector come from? Two standard sources, usually concatenated: (1) the detector's own region features (the pooled CNN activations inside that object's box, often called RoI features for "region of interest"), which describe the object's appearance, and (2) a small encoding of the box geometry (its center, size, aspect ratio), which describes where and how big it is. So each node carries both "what this object looks like" and "where it sits."
Now the edges. An edge connects two objects that are related, and the simplest, most common relation is spatial proximity: connect two objects if their boxes are close (or overlapping). Concretely, you compute the distance between box centers and add an edge when that distance is below a threshold, often keeping only each node's k nearest neighbors to avoid a fully-connected mess. The edge can carry its own feature too — the relative offset and distance — encoding the relation "bed is below-left of the nightstand, 40 pixels away."
Let us build one graph by hand, with real coordinates, so "k nearest neighbors over box centers" is not abstract. Suppose the detector returns four objects in a 100×100 image, with these box centers:
Notice what we have manufactured: a compact, explicit data structure that says which objects are present and which are near which — exactly the information GAP destroyed. The "bed adjacent to nightstand adjacent to lamp" chain is the structural fingerprint a hotel room and a home bedroom will differ on, once we add the discriminating objects (luggage rack, mini-fridge).
Play with the graph builder below. Drag objects around the room; edges form to each object's nearest neighbors as you move them. Increase k to add more edges (denser context) or decrease it for a sparser graph. Watch how moving the "luggage rack" next to the bed suddenly wires it into the bed's neighborhood — the graph is the spatial story.
Drag any object. Edges connect each node to its k nearest neighbors by box-center distance. Raise k for denser context. This is exactly how branch B turns a detection list into a graph.
We have a graph: nodes with feature vectors, edges between nearby objects. A graph neural network turns this graph into useful representations through one core operation repeated a few times: message passing. The idea is beautifully simple — each node updates its own vector by mixing in information from its neighbors. After one round, every node "knows about" its immediate neighbors; after two rounds, about neighbors-of-neighbors; and so on. The bed's vector comes to encode "I am a bed, and I sit among a nightstand and a lamp."
One message-passing step has two sub-steps, and naming them keeps the bookkeeping clean. Aggregate: each node gathers its neighbors' current vectors and combines them into one message — the simplest combiner is the mean (average the neighbor vectors). Update: the node forms its new vector from its old vector and that aggregated message, typically by combining them and passing through a small learned transformation and a nonlinearity. Mean-aggregate then update — that pair is the heartbeat of the simplest GNNs (this is essentially a Graph Convolutional Network, or GCN, step).
Let us do exactly one step by hand, with tiny 2-dimensional node features so every number is visible. Take the chain from Chapter 3: bed – nightstand – lamp. Give them 2-D feature vectors and compute the nightstand's new vector (it has two neighbors, bed and lamp).
After a couple of message-passing layers, every node vector is context-aware. The final step of branch B is a readout (also called graph pooling): collapse all the node vectors into one graph-level vector g — again, the simplest readout is to average all node vectors, or to sum them. That single g is the object branch's contribution to the fusion. Note the symmetry with branch A: both branches end by pooling many vectors into one summary — GAP pools spatial locations, readout pools graph nodes.
Here is one message-passing layer plus readout, from scratch — no library — so the aggregate–update–readout story is concrete code:
python import numpy as np # X: (N, d) node features; A: (N, N) adjacency (1 if edge, else 0); W: (d, d) learned def gnn_layer(X, A, W): out = np.zeros_like(X) for i in range(len(X)): nbrs = np.where(A[i] > 0)[0] # indices of i's neighbors msg = X[nbrs].mean(axis=0) # AGGREGATE: mean of neighbor vectors comb = (X[i] + msg) / 2 # UPDATE: blend self with message out[i] = np.maximum(comb @ W, 0) # learned transform + ReLU return out # (N, d) context-aware node vectors def graph_branch_descriptor(X, A, W1, W2): X = gnn_layer(X, A, W1) # layer 1: nodes hear neighbors X = gnn_layer(X, A, W2) # layer 2: nodes hear neighbors-of-neighbors return X.mean(axis=0) # READOUT: one graph descriptor g
The widget animates one message-passing step. Press "Pass message" and watch each node's color shift toward the average of its neighbors — information flowing along the edges. Press it again to run a second round and see context spread further. This diffusion-along-edges is what a GNN does; everything else is learned weights on top of it.
Each node carries a value (its color). "Pass message" replaces each node with the mean of itself and its neighbors — one GNN layer. Run it repeatedly to watch information diffuse across the graph.
We now hold two vectors: the holistic descriptor h from branch A and the graph descriptor g from branch B. Fusion is the operation that combines them into a single representation the final classifier can read. How you fuse is a real design decision with three common answers, in increasing order of cleverness.
Concatenation (late fusion). The simplest: stack the two vectors end to end. If h is 512-dim and g is 256-dim, the fused vector is 768-dim — just h followed by g. A linear layer then maps 768 → number of scenes, and softmax gives probabilities. The classifier learns its own weights on each half, so it can lean on whichever branch is more informative for a given scene. This is the workhorse and a strong baseline; many published two-branch scene models are exactly "concat then classify."
Weighted sum. If you first project h and g to the same dimension, you can add them with weights: fused = α·h + (1−α)·g. This is more compact but forces a shared space and a single global mixing weight.
Gated fusion. The clever one: instead of a fixed mixing weight, learn a gate that decides, per image, how much to trust each branch. A small network looks at both descriptors and emits a gate value z between 0 and 1 (via a sigmoid); the fused vector is z·h + (1−z)·g. On an image with many clear objects, the gate can lean on g; on an empty corridor where the detector found nothing useful, it can lean on h. This is the fusion that best captures "defer to whichever branch is confident this time" — the complementary-failure logic of Chapter 1, made learnable.
Let us run a concat-and-classify by hand so the shapes and the final softmax are concrete. Use tiny descriptors: h = [0.9, 0.1] (holistic says "looks like a bedroom"), g = [0.2, 0.8] (graph found a luggage rack: "hotel-ish"). Concatenate, then classify into two scenes (bedroom, hotel) with a weight matrix.
The widget below lets you mix the two branches live. Set how strongly each branch votes for "bedroom" vs "hotel," and choose the fusion mode (concat-and-classify vs gated). Watch the gate, when enabled, automatically down-weight a branch that has gone vague — the learnable version of "trust whoever is sure."
Set each branch's lean (left = bedroom, right = hotel). Toggle fusion mode. The bars show the fused scene probability; the gate readout shows how much the gated mode trusts each branch this image.
The two-branch idea is not equally useful everywhere. It shines precisely on MIT Indoor67 and indoor recognition in general, and understanding why turns this from a recipe into a principle. The short version: indoor scenes are defined by their objects and the objects overlap heavily between classes, which is exactly the regime where an explicit object graph adds signal that holistic appearance cannot.
Recall the difficulty of indoor scenes from scene classification: low inter-class margin — bedroom vs hotel_room, library vs bookstore, classroom vs conference room sit very close in appearance space. They share objects (chairs, tables, shelves) and global texture (interior lighting, walls, floors). The holistic branch, working from global appearance, has almost nothing to grab onto. Two indoor scenes can be more similar to each other than two photos of the same kitchen.
Now ask: what does separate a library from a bookstore? Not the presence of books — both have books. It is the object co-occurrence and arrangement: a bookstore co-occurs with a cash register, a checkout counter, price displays, retail signage; a library co-occurs with reading desks, study carrels, a catalog station, quiet seating. The discriminating evidence is "which objects appear together", and that is a graph property, not an appearance property. The GNN branch reads it directly; the GAP branch averaged it into oblivion.
Let us make the co-occurrence argument quantitative. Treat each scene as a distribution over which objects appear. Suppose across many images, a library and a bookstore both contain "book" 95% of the time, "shelf" 90%, "person" 60% — nearly identical. But "cash register" appears in 70% of bookstores and 3% of libraries, and "study desk" in 65% of libraries and 8% of bookstores.
This also explains the asymmetry: the two-branch model helps indoor far more than outdoor. A beach, a forest, a highway are defined by global texture and layout — sand and water, trees, road and horizon — with few discrete, detectable objects whose co-occurrence matters. There, the holistic branch already wins and the graph adds little. Indoor scenes are object-dense and appearance-ambiguous; outdoor scenes are object-sparse and appearance-distinct. The graph branch's value is highest exactly where holistic appearance is weakest.
The widget below is a co-occurrence explorer. Pick a discriminating object to "detect," and watch how much it shifts the belief between two object-ambiguous indoor scenes — while detecting a shared object barely moves the needle. This is the engine of the whole chapter: discriminating co-occurrence is the lever the graph branch pulls.
Two object-ambiguous scenes. Click an object to "detect" it and watch the belief shift. Shared objects (book, shelf) barely move it; discriminating objects (cash register, study desk) flip it. The graph branch lives off these discriminators.
Everything converges here into one machine you can drive. The canvas is a room. You place objects into it; the model detects them, builds the k-NN graph, runs message passing, reads out the graph descriptor g, fuses it with a holistic descriptor h, and shows the live scene prediction over four candidates: bedroom, hotel room, library, bookstore. As you add the discriminating objects, watch the prediction shift — the whole lesson, live.
Each object you can place carries a small vote toward scenes (a luggage rack votes hotel; a study desk votes library; a cash register votes bookstore; a bed votes bedroom-or-hotel). The graph branch aggregates these votes through the edges, so an object's influence spreads to its neighbors. The holistic slider sets branch A's baseline guess; the fusion combines both.
Three experiments, each illustrating a chapter:
• Start with just a bed and nightstand. The graph is ambiguous between bedroom and hotel — a near-tie, the Chapter-0 failure. The holistic branch alone cannot break it.
• Now add a luggage rack. One discriminating object, wired into the graph next to the bed, flips the prediction to hotel — the Chapter-6 co-occurrence lever and the Chapter-5 fusion override, together.
• Clear it and build a library (books, shelf, study desk), then drop in a cash register. Watch the prediction slide from library toward bookstore as the discriminating object enters the graph. Then crank "detector noise" and watch a missed or false detection corrupt the graph — a preview of Chapter 8's failure modes.
Click "Add" buttons to drop objects into the room; the k-NN graph and message passing run automatically and the fused scene bars update. Add discriminating objects to flip the prediction; raise detector noise to corrupt the graph.
If you removed this simulation, would you lose understanding? Yes — "the graph branch disambiguates via discriminating co-occurrence" is abstract until you watch the luggage rack flip bedroom to hotel by wiring into the bed's neighborhood. The napkin drawing of this whole lesson is exactly this canvas: objects in, graph built, descriptors fused, one scene out.
The two-branch model is not free lunch. You bought accuracy on object-dense indoor scenes with new dependencies and new ways to fail. An engineer must know the bill before signing.
You now need an object detector. The single biggest cost: branch B requires a detector running at inference, which adds latency, memory, and a whole second model to train, maintain, and keep in distribution. The holistic CNN was self-contained; the two-branch model has a pipeline. On a latency-budgeted robot, running both a backbone and a detector per frame may be infeasible — you might run the graph branch only intermittently, or only when the holistic branch is uncertain.
The graph is only as good as the detections. Branch B inherits every failure of the detector. Three concrete failure modes: missed detections (the discriminating luggage rack is not detected → the graph never learns "hotel" and the scene stays a bedroom); false detections (a hallucinated object adds a spurious node and misleads the message passing); and domain shift (the detector was trained on common-object datasets like COCO, which lack many scene-specific objects — "study carrel", "altar", "lab fume hood" — so on a novel scene the graph is empty or wrong exactly when you needed it most).
Let us quantify the missed-detection failure. Suppose the luggage rack is the only discriminator between bedroom and hotel for a given image, and the detector finds it with probability 0.8 (recall = 0.8). Then 20% of the time the graph branch gets no discriminating signal and the fused model falls back to the holistic branch's coin-flip on this pair. So even a strong 80%-recall detector caps the achievable gain on these images at 80% of the way — the graph branch can only help when the detector cooperates.
Other costs. The graph construction has hyperparameters (k, the distance threshold) that need tuning. The GNN can over-smooth with too many layers (Chapter 4). And on object-sparse scenes the branch contributes nothing — you pay its cost for no benefit unless gated fusion mutes it. None of these is fatal; all are reasons the two-branch model is a targeted tool, not a universal upgrade.
The widget below sweeps detector quality. Drag recall and false-positive rate and watch the fused model's accuracy on the discriminator-dependent pair rise and fall — a live version of the by-hand bound above. Notice the model degrades gracefully toward the holistic baseline rather than collapsing: a robustness property of fusion.
Drag the detector's recall (does it find the discriminating object?) and false-positive rate (does it hallucinate misleading objects?). Watch fused accuracy track detector quality — and degrade gracefully toward the holistic baseline, not to zero.
You have built a complete two-branch CNN+GNN scene classifier from zero: a holistic CNN branch ending in global average pooling, an object branch that detects, builds a k-NN graph, message-passes, and reads out, and a fusion that joins them into one scene decision. The pattern — fuse a holistic view with an explicit structural view — recurs widely, and naming the connections turns one lesson into a map.
The holistic branch lives in the scene-classification world. Everything about GAP, the Places365 / SUN397 / MIT Indoor67 benchmarks, top-1 vs top-5, and the move from CNNs to Vision Transformers is the subject of the broader Visual Scene Classification lesson — this lesson is its object-aware sequel. Read that one for the holistic story in full; read this one for the second eye.
The graph branch lives in the GNN world. Message passing, aggregate-then-update, readout, over-smoothing, and the GCN/GAT family are general graph machinery taught in Graph Neural Networks. The two specific architectures that show up in published two-branch scene models — the GCN (mean-aggregate, the step we built) and the GAT (which learns attention weights over neighbors instead of a plain mean) — are worth studying via the CS224W course series. A GAT branch lets the bed pay more attention to the discriminating luggage rack than to a boring wall, the graph analogue of the gated fusion we built.
The object detector that feeds the graph is its own deep subject — the region features, RoI pooling, and box regression all come from the detection literature; the scene model treats the detector as a frozen front-end, the way our fusion treated each branch as a black box producing a vector.
Same shape, other places. The "holistic + structural, then fuse" template appears far beyond scenes. Visual question answering fuses a holistic image feature with a scene graph of objects-and-relations. Action recognition fuses a holistic video feature with a graph of detected people and objects. Document understanding fuses a page image with a graph of text blocks. Each is "this lesson, new domain": one branch reads the whole, one reads the structure, fusion reads both.
| Approach | How it reads the scene | Best when | Indoor strength |
|---|---|---|---|
| Holistic CNN (GAP) | Global appearance only | Outdoor / object-sparse, texture-defined scenes | Weak on object-ambiguous pairs |
| CNN + GCN two-branch | Holistic + mean-aggregate object graph | Indoor, object-dense, appearance-ambiguous | Strong (>90% MIT Indoor67) |
| CNN + GAT two-branch | Holistic + attention-weighted object graph | Cluttered scenes with a few key objects | Strong; focuses on discriminators |
| Vision Transformer | Self-attention over patches (implicit structure) | Large pretraining data available | Strong with scale; structure is implicit, not explicit |
| Thing | What to remember |
|---|---|
| Two branches | CNN → GAP → holistic descriptor h; detector + GNN → graph descriptor g |
| Holistic branch | conv backbone → (C,H,W) → global average pool → h = (C,); location-blind on purpose |
| Graph build | each detected object = node (RoI features + box geometry); edges = k-NN over box centers |
| GNN step | AGGREGATE (mean of neighbors) → UPDATE (blend self + message, transform, ReLU); readout = mean of nodes |
| Fusion | concat (h then g) + linear + softmax; or gated z·h + (1−z)·g, gate learned per image |
| Why indoor | indoor = object-dense + appearance-ambiguous; discriminating co-occurrence is the missing signal GAP discards |
| Cost / failure | needs a detector; graph quality bounded by detector recall; over-smoothing; useless on object-sparse scenes |
| Result | two-branch CNN+GNN pushed MIT Indoor67 past 90%, where holistic CNNs stalled in the 80s |
"The whole is other than the sum of its parts." — Kurt Koffka. A scene is the whole (the CNN's gestalt) and the structure of its parts (the GNN's object graph). This lesson is the architecture that finally listens to both.