Corners tell a robot where texture is. They never tell it what it's looking at. This is the lesson where geometry becomes semantics: classification, detection, segmentation — and the stop sign that trips the executive's STOP state.
In Lesson 9 you built a corner detector. Point a Harris detector or a SIFT detector at a street scene and it lights up: the corner of a window, the edge of a license plate, the rim of a wheel, a speck of texture on the asphalt. Hundreds of little crosses, each marking a spot where the image gradient is sharp in two directions. It is a beautiful map. And it is, for the question we now need to answer, almost useless.
Because here is what a robot at a four-way stop actually needs to know: is that red octagon a stop sign, or a balloon, or a parked car's tail-light? Is that shape ahead a pedestrian who might step out, or a mailbox that never will? The corner detector found dozens of keypoints on the stop sign — but not one of them said the word "stop." Geometry tells you where interesting structure sits in the image. It is silent about what that structure means.
That silence is the gap this lesson closes. We are making the jump from geometric feature extraction (lines, corners, blobs — the subject of Lessons 7–9) to semantic perception: naming the things in a scene. Stanford's Lecture 10 calls this information extraction, and it sits squarely on the "See" beat of the see-think-act loop from Lesson 1 — the moment raw pixels become words a robot can reason about.
Why is "what" so much harder than "where"? Stanford's notes give two honest reasons. First, the real world is a jumble: objects occlude one another, appear at every scale and pose, and sit under wildly different lighting. Second, there is huge variability within a single class — a "dog" can be a chihuahua or a Great Dane; a "stop sign" can be faded, tilted, half in shadow, or stickered over. A rule like "red octagon = stop sign" breaks the instant the sign is seen edge-on or at dusk. Recognition has to be robust to all of that, which is exactly why hand-written rules lose and learned functions win.
Think about what "red octagon = stop sign" actually demands. You'd need a rule for the sign at every distance (it spans 200 pixels up close, 12 pixels far away), every angle (a circle when edge-on, not an octagon), every lighting (orange at sunset, near-black at night), every occlusion (a branch across one corner), and every defacement (a sticker, graffiti, rust). Each of those is a separate hand-written exception, and the exceptions interact combinatorially. No human can enumerate them. A learned detector, shown thousands of stop signs in all those conditions, discovers the invariant pattern automatically — that is the entire reason the field is "learning-based" and not "rule-based."
Notice the through-line to the rest of this course. Lesson 1 taught that data changes type as it flows around the see-think-act loop: raw pixels become features become a world model become a plan. This lesson owns one specific transformation on that loop — the one that turns geometric features into semantic symbols. Get that transformation right and the executive (Lesson 1) and the planner (Lessons 3–5) have meaningful symbols to reason over. Get it wrong — miss the stop sign, hallucinate a pedestrian — and every layer downstream inherits the error. Perception is where the robot's understanding of the world is born, and where it can first go fatally wrong.
Before we name the tasks, let's see the gap with our own eyes. Below is the same street scene rendered two ways: as a corner map (what Lesson 9 gives you) and as a labeled scene (what this lesson gives you). Toggle between them and ask yourself which one could drive a car.
Toggle the view. The corner map marks high-gradient keypoints — lots of crosses, zero meaning. The labeled scene names each object with a box and a class. Same pixels; one is geometry, the other is semantics.
The corner map is honest about geometry and mute about meaning. The labeled scene answers the question a robot brain actually poses: stop sign, here, 91% sure. That single sentence — a class, a location, and a confidence — is the output of everything we build in this lesson, and it is precisely the input the executive FSM from Lesson 1 has been waiting for.
"Recognize the scene" is not one task — it is a family of tasks that differ by how precise an answer you demand. A robot might want a single word for the whole image, a word plus a box for each object, or a word for every single pixel. These three levels — classification, detection, and segmentation — form the taxonomy of semantic perception, and every model you will ever meet sits at one of them.
Image classification answers the simplest question: what is the single most prominent thing in this picture? In goes an image; out comes one class label, optionally with a probability. "This is a cat (0.97)." "This is a stop sign (0.88)." There is no notion of where — the answer describes the whole frame at once.
The data flow is the cleanest in all of vision. The input is a tensor of shape H × W × 3 (height, width, three colour channels). The output is a vector of length C, one raw score (a logit) per class. A softmax turns those raw scores into probabilities that are all positive and sum to one, and you report the largest. For a robot, classification alone is rarely enough — "there is a person somewhere" doesn't tell you whether to brake — but it is the building block the other two tasks stand on.
Worked softmax, by hand. Suppose a 3-class classifier (stop sign, yield sign, billboard) outputs the raw logit vector z = (2.0, 0.5, −1.0). Softmax exponentiates each score and divides by the total, so the answer is a probability distribution:
The three probabilities sum to 0.786 + 0.175 + 0.039 = 1.000, and the winner is stop sign at 0.786 (79%). Notice that exponentiation amplifies the lead: a logit gap of 1.5 (2.0 vs 0.5) became a probability ratio of about 4.5× (0.786 vs 0.175). That sharpening is deliberate — it pushes the prediction toward a confident single answer — but it is also why a classifier can be confidently wrong: a slightly higher logit for the wrong class still wins decisively. We'll see that bite in Chapter 6.
Object detection answers what and where together. Out comes a list, one entry per object: a class label, a bounding box (four numbers — the box's left, top, width, height, or equivalently two corners), and a confidence score. "Stop sign at box (412, 88, 96, 96), 0.91. Person at box (210, 140, 60, 180), 0.74."
This is the workhorse of robotics. A box plus a class is exactly enough to (a) decide the object matters, and (b) — combined with the camera model from Lesson 8 — estimate how far away and in which direction it is. That last step is what turns "I see a stop sign" into "the stop sign is 8 metres ahead at bearing −3°," which is what the planner and executive actually consume.
Semantic segmentation is the most precise: it assigns a class to every single pixel. Out comes an image the same size as the input, but instead of colours each pixel holds a class id — this pixel is road, that pixel is sky, those pixels are pedestrian. The result is a mask: a dense, pixel-perfect map of meaning.
Segmentation is what you want when the exact shape matters — "where exactly is the drivable road surface?" — not just a coarse box. It is also the most expensive: instead of one label, the model must produce H × W labels. (There is a still-finer cousin, instance segmentation, which separates the two pedestrians that semantic segmentation would lump into one "person" blob; we'll mention it but focus on the semantic version.)
Classification. In: H×W×3. Out: one class (+ probability). Coarsest. "What is this a picture of?"
Detection. In: H×W×3. Out: a list of (class, box, score). Medium. "What objects are here, and where?"
Segmentation. In: H×W×3. Out: an H×W label map. Finest. "What class is every pixel?"
Instance segmentation. Out: a mask per object. Separates the two people a semantic mask would merge.
The three tasks form a ladder of precision — and of cost. Below, switch a toy scene between the three modes and watch the output change from one word, to boxes, to a per-pixel paint job.
Pick a task. Classify gives the whole image one label. Detect draws a box + class per object. Segment paints every pixel by class. Same scene, three answers of increasing precision — and increasing compute.
Before learning, before neural networks, there was the oldest recognition trick in the book, and it is exactly how you find Waldo: you have a little picture of the thing you want (the template), and you slide it over the big image, asking at every position how well does the template line up here? Wherever the alignment is best, that's your match. Stanford's Lecture 10 opens object recognition with precisely this idea.
So we need two things: a source image I and a template T. We score the match with correlation: at each position we lay the template down, multiply each template pixel by the image pixel beneath it, and add up all the products. A big positive sum means "bright where the template is bright, dark where it's dark" — a good alignment.
Raw correlation has a fatal flaw: a region of the image that is simply very bright everywhere produces a huge sum no matter what the template looks like, just because the numbers are big. Brightness, not shape, wins. The classic fix is to normalize. Stanford writes the template-at-a-position as a vector t and the patch underneath as a vector f, and defines the normalized cross-correlation (NCC) as the cosine of the angle between them:
Here f · t is the dot product (sum of element-wise products), and ‖f‖ is the length (Euclidean norm) of the vector. Dividing by both lengths cancels brightness: a patch twice as bright has twice the dot product and twice the norm, so the ratio is unchanged. NCC lives in [−1, +1], and a perfect match scores exactly +1 — that is the whole reason we normalize.
Let's do one match completely by hand. Our template is a 2×2 bright corner — a thing we want to find:
Flattened into a vector, t = (2, 1, 1, 0). Its norm is ‖t‖ = √(4 + 1 + 1 + 0) = √6 ≈ 2.449.
Position A — a matching patch. Suppose under the template we find f = (4, 2, 2, 0) — exactly twice the template (same shape, brighter). Then:
A perfect score of 1.00 — even though the patch was twice as bright. Normalization saw through the brightness to the shape. That is the payoff.
Position B — a non-matching patch. Now suppose under the template we find f = (0, 1, 1, 2) — the bright corner is in the wrong place (bottom-right, not top-left). Then:
A weak score of 0.33. The patch has the same total brightness as the template (both norms are √6) — raw correlation would not distinguish them well — but the bright pixel is in the wrong spot, so the directions disagree and NCC correctly reports a poor match. We slide the template over the whole image, compute this number at every position, and the peak is our detection.
Stanford's notes mention a second similarity measure worth knowing: the sum of absolute differences (SAD). Instead of a dot product, SAD adds up |fk − tk| over all pixels — literally how different the patch is from the template, pixel by pixel. A perfect match gives SAD = 0, and the best position is the smallest SAD (the opposite convention to NCC's largest). For our matching patch at position A — if the patch were exactly the template (2,1,1,0) — SAD would be |2−2|+|1−1|+|1−1|+|0−0| = 0. But against our twice-as-bright patch (4,2,2,0), SAD = |4−2|+|2−1|+|2−1|+0 = 4 — a "bad" score, even though it's the same shape! SAD has no built-in brightness normalization, which is exactly why NCC, with its norm denominators, is usually preferred for matching across lighting changes.
NCC's Achilles heel is geometry. Take the matching template t = (2,1,1,0) (a bright top-left corner) and suppose the object in the image has rotated 90°, so the patch underneath is now f = (1,0,2,1) (the same bright corner, rotated):
The score fell from a perfect 1.00 to 0.67 — for the identical object, just rotated. A real 30° rotation, or a half-size version of the object, drops it further still, often below the detection threshold. The template only matches one exact appearance; it cannot generalize across the poses the same object naturally takes. Hold this number in mind — it is the concrete reason the field abandoned fixed templates for learned, pose-tolerant features (Chapter 5).
First the explicit version — one NCC score at one position, every step spelled out:
python import numpy as np def ncc_at(patch, template): # patch and template are equal-size 2D arrays (the patch is the # image region currently under the template) f = patch.flatten().astype(float) # f = (4, 2, 2, 0) t = template.flatten().astype(float) # t = (2, 1, 1, 0) dot = 0.0 for k in range(len(f)): dot += f[k] * t[k] # 8 + 2 + 2 + 0 = 12 norm_f = (sum(v*v for v in f)) ** 0.5 # sqrt(24) = 4.899 norm_t = (sum(v*v for v in t)) ** 0.5 # sqrt(6) = 2.449 if norm_f == 0 or norm_t == 0: return 0.0 # a flat patch matches nothing return dot / (norm_f * norm_t) # 12 / 12.0 = 1.00
And the idiomatic version — numpy collapses the loops into vector ops, and the answer is identical:
python def ncc(patch, template): f = patch.ravel().astype(float) t = template.ravel().astype(float) denom = np.linalg.norm(f) * np.linalg.norm(t) return 0.0 if denom == 0 else float(f @ t / denom) # f @ t is the dot product; np.linalg.norm is the Euclidean length. # OpenCV does the full slide for you: cv2.matchTemplate(I, T, cv2.TM_CCORR_NORMED)
Now drag the template over the image yourself. The live NCC score updates as you move; watch it spike toward 1.0 only when the template sits exactly on the object it was cut from.
Drag the dashed template box over the scene. The bar shows the live normalized cross-correlation at the current position — near 1.0 on a true match, low elsewhere. Press scan to sweep the whole image and mark the best position.
Before learned features, researchers tried to fix template matching's rigidity with a clever idea Stanford mentions: the bag of visual words. The insight: instead of matching one rigid whole-object template, represent an object as a collection of small distinctive sub-parts ("visual words") — a bike is wheels + frame + handlebars. You build a vocabulary of common patches across many training images, then describe a new image by which visual words appear in it and how often (a histogram), ignoring where they sit. Recognition becomes comparing histograms: an image with eye-patches and a nose-patch probably contains a face.
This was more robust than rigid templates — scrambling the parts' positions doesn't change the histogram, so it tolerates some pose change — but it threw away all spatial structure (a face with the eyes below the mouth scores the same) and still relied on hand-engineered patch descriptors. It was a stepping stone: it accepted that objects are compositions of features, which is exactly the principle a convolutional network would later learn end-to-end, while keeping the spatial layout the bag-of-words discarded.
Template matching gives us a way to find one specific appearance. But suppose we have something stronger: a classifier — a function that takes a fixed-size patch and answers "stop sign? yes/no, with confidence." (We'll build the learned version in Chapters 5–6; for now treat it as a black box that scores a patch.) How do we turn a yes/no classifier, which only judges one patch, into a detector that finds objects anywhere in a big image?
The oldest and most intuitive answer is the sliding window: run the classifier at every position in the image. Crop a window, classify it, slide one step, classify again, slide again — row by row, column by column — and keep every position where the classifier shouts "stop sign!" loudly enough. It is template matching's logic, but the score comes from a trained classifier instead of a fixed correlation.
One window size only finds objects of one size. A classifier trained on 64×64 stop-sign patches will miss a sign that fills 200×200 pixels up close, or a tiny one far away. Stanford's solution, lifted straight from template matching, is the image pyramid: build a stack of progressively smaller copies of the image (each level blurred with a Gaussian then sub-sampled to half size — a Gaussian pyramid), and slide the same fixed window over every level. A big object that overflows the window at full resolution fits perfectly two levels down.
Let's count. An image is roughly W × H positions. Slide a window with a stride (step) of s pixels and you evaluate about (W/s) × (H/s) windows per scale. With L pyramid levels, total classifier calls ≈ L × (W/s)(H/s).
Concretely: a 640×480 image, stride 8, gives (640/8) × (480/8) = 80 × 60 = 4,800 windows. Across 5 pyramid levels that's about 24,000 classifier evaluations for a single frame. At 30 frames per second that is 720,000 classifications every second — and most windows contain nothing but road or sky. The sliding window is conceptually perfect and computationally brutal; almost all of the work is wasted on empty patches.
The cost is dominated by the stride, because it appears squared — halving the stride quarters the gap between windows, quadrupling their count. Here is the same 640×480 frame at five different strides, single scale:
| Stride (px) | Windows = (640/s)×(480/s) | × 5 pyramid levels | Per second @ 30 fps |
|---|---|---|---|
| 4 | 160 × 120 = 19,200 | 96,000 | 2,880,000 |
| 8 | 80 × 60 = 4,800 | 24,000 | 720,000 |
| 16 | 40 × 30 = 1,200 | 6,000 | 180,000 |
| 32 | 20 × 15 = 300 | 1,500 | 45,000 |
From stride 4 to stride 32 the per-frame work drops 64× (8²) — but the coarse grid now steps 32 pixels at a time, and a 40-pixel-wide stop sign far away can fall between windows and never get a clean centered look. That is the trap: every classifier call you save is a chance to miss an object. The grid resolution and the detection rate are tied together by the stride.
python # Sliding-window detection over an image pyramid (sketch). def detect(image, classifier, win=(64,64), stride=8, scales=(1.0,0.75,0.5,0.35)): boxes = [] for sc in scales: # one Gaussian-pyramid level per scale img = resize_blur(image, sc) # blur THEN shrink (anti-alias) wh, ww = win for y in range(0, img.height - wh, stride): for x in range(0, img.width - ww, stride): patch = img[y:y+wh, x:x+ww] score = classifier(patch) # the expensive call, x24,000 / frame if score > 0.5: # map box back to ORIGINAL image coords by dividing by sc boxes.append((x/sc, y/sc, ww/sc, wh/sc, score)) return boxes # MANY overlapping boxes -> needs NMS (Ch 4)
One detail that trips everyone up: coordinate bookkeeping. When you find a hit at pyramid level scaled by sc (say 0.5), the box you found is in the shrunken image's coordinates. To report it on the original frame you must scale back up: divide by sc. A window at (x, y) = (40, 30), size 64×64, found at scale 0.5, maps to original coordinates (40/0.5, 30/0.5) = (80, 60) with size 64/0.5 = 128 — a 128×128 box. The fixed window found a big object precisely because the object was small at that pyramid level. Forget the divide-by-sc and every detection on a shrunken level lands in the wrong place — a classic bug.
Below, a window sweeps a synthetic scene. Watch the classifier-score heat trail it leaves: most positions stay cold, a cluster of warm positions fires around the true object — and notice that a single object triggers many overlapping positive windows. That redundancy is the next problem to solve.
Press sweep: the window marches across the image and the classifier scores each position. The score bar tracks the current window; positions above threshold leave a warm mark. One object → a cluster of overlapping hits. The counter shows total classifier calls — the cost.
The sliding window left us with a mess: a dozen overlapping boxes around one stop sign. We want exactly one box per object. To clean this up we need two tools — a number that measures how much two boxes overlap, and an algorithm that uses it to throw away duplicates.
Intersection-over-Union (IoU) is the standard overlap measure between two boxes. It is exactly what its name says: the area where the boxes overlap (intersection), divided by the area they cover together (union):
IoU runs from 0 (no overlap at all) to 1 (the two boxes are identical). It is scale-free — two big boxes and two small boxes with the same relative overlap get the same IoU — which is what makes it the universal currency of detection. The union is computed as area(A) + area(B) − area(A ∩ B) (you subtract the intersection because it was counted in both areas).
Take two axis-aligned boxes, each written as (x1, y1, x2, y2) (top-left and bottom-right corners):
Step 1 — individual areas. A is 4 wide and 4 tall, so area(A) = 4 × 4 = 16. B is (6−2) × (5−1) = 4 × 4 = 16.
Step 2 — the intersection rectangle. The overlap's left edge is the larger of the two left edges: max(0, 2) = 2. Its top is max(0, 1) = 1. Its right edge is the smaller of the two right edges: min(4, 6) = 4. Its bottom is min(4, 5) = 4. So the intersection box is (2, 1, 4, 4).
Step 3 — intersection area. Width = 4 − 2 = 2, height = 4 − 1 = 3, so area(A ∩ B) = 2 × 3 = 6. (Always clamp width and height at 0 — if the boxes don't touch, one of them goes negative and the intersection is 0.)
Step 4 — union and IoU. Union = 16 + 16 − 6 = 26. Therefore:
An IoU of 0.23 is a modest overlap — the boxes touch but don't agree well. A detection counts as "correct" against ground truth typically at IoU ≥ 0.5; two duplicate detections of the same object usually overlap at IoU > 0.7. That threshold is the dial NMS turns.
Non-max suppression (NMS) is the algorithm that collapses a heap of overlapping boxes into one box per object. The recipe is short and greedy:
Suppose the detector returns four boxes for what is really two objects. We use IoU threshold τ = 0.5:
| Box | Coords (x1,y1,x2,y2) | Score |
|---|---|---|
| P | (0, 0, 4, 4) | 0.92 |
| Q | (1, 0, 5, 4) | 0.85 |
| R | (10, 10, 14, 14) | 0.78 |
| S | (11, 10, 15, 14) | 0.60 |
Pass 1. Highest score is P (0.92) — it becomes a keeper. Now check the rest against P:
Pass 2. Among the survivors {R, S}, highest is R (0.78) — it becomes a keeper. Check S against R:
Output: {P, R} — exactly two boxes for two objects. Four messy detections became two clean ones. That is NMS doing its one job.
python import numpy as np def iou(a, b): # boxes as (x1, y1, x2, y2) ix1, iy1 = max(a[0], b[0]), max(a[1], b[1]) ix2, iy2 = min(a[2], b[2]), min(a[3], b[3]) iw, ih = max(0, ix2 - ix1), max(0, iy2 - iy1) # clamp at 0! inter = iw * ih area_a = (a[2]-a[0]) * (a[3]-a[1]) area_b = (b[2]-b[0]) * (b[3]-b[1]) union = area_a + area_b - inter return inter / union if union > 0 else 0.0 def nms(boxes, scores, tau=0.5): idx = sorted(range(len(scores)), key=lambda i: -scores[i]) # high score first keep = [] while idx: i = idx.pop(0) # the current maximum -> a keeper keep.append(i) # drop every remaining box that overlaps the keeper too much idx = [j for j in idx if iou(boxes[i], boxes[j]) <= tau] return keep
The library one-liner: torchvision.ops.nms(boxes, scores, tau) — same algorithm, vectorized. How fast is it? The sort is O(N log N) for N boxes; the suppression loop compares the current keeper against the rest, which is O(N²) in the worst case (every box compared with every other). Since detectors typically feed NMS a few hundred surviving boxes per image, this is cheap — microseconds. The expensive part was always the detector that produced the boxes, not the cleanup.
A subtlety worth internalizing: NMS runs per class. A stop-sign box and a person box can overlap heavily — a pedestrian standing right under a sign — without suppressing each other, because they are different classes answering different questions. You only suppress duplicates of the same class. (Class-agnostic NMS exists too, but per-class is the default for detection.)
Now turn the dial yourself. Drag the IoU threshold and watch boxes merge or split in real time.
Raw detector boxes (faint) cluster around two objects. Drag the IoU threshold: low τ suppresses aggressively (boxes merge into one per object); high τ suppresses little (duplicates survive). The kept boxes are drawn bold with their scores. The readout reports IoU between the two front-runners.
Here is the unifying idea of the whole lesson, and it is simpler than it sounds. A convolution filter — a small grid of numbers, say 3×3 — is exactly the same operation as template matching: you slide the little grid over the image, multiply-and-sum at every position, and write the result into an output image. Stanford's homework Problem 3 makes this explicit: the correlation operator G = F ⊗ I produces an output whose every pixel is a weighted sum of the input pixels under the filter.
The filter F is a tiny template; the output G is the "match score" of that template at every pixel. The formula, with the filter and image both indexed from the top-left corner:
where (u, v) sweeps over the filter's cells. The single, revolutionary twist that built modern vision: in template matching you hand-design the filter; in a convolutional network the filter's numbers are learned from data. The network discovers, by training, which little templates are worth sliding around.
Convolution is literally a dot product — the bridge back to Chapter 2. Stanford's homework makes the link exact. Flatten the filter into a vector f and the image region under it into a vector tij, and the double sum above collapses to G(i,j) = f · tij — a single dot product. And we already know (Chapter 2) that a dot product measures how aligned two vectors are. So a convolution output is the un-normalized match score between the filter-as-template and the local patch. The only differences from template matching: the "template" is tiny (3×3, not the whole object), the network has many of them, and their weights are learned. Same operation, scaled up and made trainable. That continuity is the quiet, deep idea of the whole lesson.
Let's apply one filter to one patch with real numbers. Take this 3×3 image patch (grayscale values) and a horizontal-edge filter F — Stanford's Problem 3(c):
To compute the output at the center pixel, lay F over the whole 3×3 patch and sum the element-wise products, row by row:
The output is −6. What does that number mean? The filter subtracts the bottom row of pixels from the top row. Where the image is smooth (top and bottom rows similar) the result is near zero; where there's a sharp horizontal edge (top bright, bottom dark, or vice versa) the result is large in magnitude. So this filter is a horizontal edge detector — it lights up at boundaries where brightness changes top-to-bottom. Its rotated cousin [[−1,0,1],[−1,0,1],[−1,0,1]] detects vertical edges instead.
A second position, to feel the slide. Convolution computes one such number at every pixel. Take the vertical-edge filter F′ = [[−1,0,1],[−1,0,1],[−1,0,1]] on the same patch I = [[7,4,1],[8,5,2],[9,6,3]]. It subtracts the left column from the right column:
A big magnitude (−18): this patch has a strong left-to-right brightness drop (left column ~8, right column ~2), and the vertical-edge filter fired hard on it. The same patch gave only −6 to the horizontal filter, because top-to-bottom the columns are nearly flat. Two filters, same patch, completely different answers — each filter is a question, and the output map is the answer at every pixel.
The blur filter, worked. Stanford's Problem 3(d) uses a Gaussian smoothing kernel, F = (1/16)[[1,2,1],[2,4,2],[1,2,1]]. Its weights are all positive and sum to 1+2+1+2+4+2+1+2+1 = 16, so dividing by 16 makes it a weighted average — center pixel weighted most, corners least. On our patch the center response is (1×7+2×4+1×1 + 2×8+4×5+2×2 + 1×9+2×6+1×3)/16 = (16 + 40 + 24)/16 = 80/16 = 5.0 — exactly the center pixel's value, because this patch happens to be smooth. On a noisy patch, the averaging would suppress the noise. Positive weights that sum to 1 = a blur; weights that sum to 0 with opposite signs = an edge detector. This is precisely the Gaussian blur we used to anti-alias the image pyramid in Chapter 3.
One layer of filters turns pixels into an "edge map." But the magic is stacking: feed that edge map into a second layer of filters, and those filters combine edges into corners and curves. A third layer combines corners into textures and object-parts (a wheel, an eye); a fourth into whole objects. Each layer's filters operate on the features the previous layer produced, not on raw pixels. This is the feature hierarchy, and it is why deep networks are deep:
Concretely, picture the receptive field growing. A layer-1 neuron, looking through a 3×3 filter, sees a 3×3 patch of pixels. A layer-2 neuron looks through a 3×3 filter at the layer-1 outputs — but each of those outputs already summarized a 3×3 pixel patch, so the layer-2 neuron effectively sees a 5×5 region of the original image. Stack ten such layers (with downsampling) and a single deep neuron's receptive field spans the whole image — which is how a network that only ever computes local 3×3 dot products ends up recognizing a whole stop sign. The composition of small local views is the global view.
And here is why learned filters win decisively. A hand-coded edge detector is fixed forever. A learned layer-1 filter, trained on millions of stop-sign images, can specialize to "a red-to-grey transition at the octagonal angle" — a feature tuned to the actual statistics of the objects you care about, robust to the lighting and pose variation in the training set. You could never hand-write the thousands of such filters a modern backbone holds; training discovers them. That is the move from Chapter 2's brittle fixed template to a perception system that generalizes.
python import numpy as np def conv3x3_at(I, F, i, j): # output pixel at (i,j): F laid over the 3x3 region with top-left at (i,j) total = 0.0 for u in range(3): for v in range(3): total += F[u][v] * I[i+u][j+v] # multiply-and-accumulate return total # for our example: -6 I = [[7,4,1],[8,5,2],[9,6,3]] F = [[1,1,1],[0,0,0],[-1,-1,-1]] # horizontal-edge filter print(conv3x3_at(I, F, 0, 0)) # -> -6.0
python # Idiomatic: element-wise multiply then sum (numpy), or use scipy/torch for the full slide. I = np.array([[7,4,1],[8,5,2],[9,6,3]], dtype=float) F = np.array([[1,1,1],[0,0,0],[-1,-1,-1]], dtype=float) print((I * F).sum()) # -> -6.0, the center pixel response # Whole-image, every position: # from scipy.signal import correlate2d; G = correlate2d(image, F, mode='same') # or a learned layer: torch.nn.Conv2d(in_ch, out_ch, kernel_size=3)
Pick a filter below and watch it transform an image — edges pop out under an edge filter, the image softens under a blur, sharpens under a sharpen kernel. The single 3×3 grid is doing, at every pixel, exactly the arithmetic we did by hand.
Choose a 3×3 kernel and watch it convolve the source image. Edge kernels light up boundaries; blur averages (smooths); sharpen amplifies differences. The 3×3 grid (right) shows the active filter weights — the numbers a CNN would learn instead of you choosing.
[[1,1,1],[0,0,0],[−1,−1,−1]] is applied across an image. What does it detect?We now have all the pieces. Convolution gives us learned filters (Chapter 5). Stacking them gives a feature hierarchy. IoU and NMS clean up the boxes (Chapter 4). A modern CNN object detector wires these into one trainable pipeline that, in a single forward pass, takes an image and emits the list of (class, box, score) we wanted in Chapter 1. Stanford treats the network internals as a black box ("take AA274B for details") — so here we build the intuition and the data shapes, which is what a robotics engineer actually needs.
Follow the data. It changes shape at every stage — tracking those shapes is how you understand the architecture.
The backbone shrinks space and grows depth. Each "downsample" stage (a strided convolution or a pooling layer) halves height and width while a learned set of filters expands the channel count. Follow a 512×512 RGB image through a five-stage backbone:
| Stage | Operation | Output shape (H × W × C) | Spatial cells |
|---|---|---|---|
| input | — | 512 × 512 × 3 | 262,144 |
| 1 | conv + downsample ÷2 | 256 × 256 × 64 | 65,536 |
| 2 | conv + downsample ÷2 | 128 × 128 × 128 | 16,384 |
| 3 | conv + downsample ÷2 | 64 × 64 × 256 | 4,096 |
| 4 | conv + downsample ÷2 | 32 × 32 × 512 | 1,024 |
| 5 | conv + downsample ÷2 | 16 × 16 × 512 | 256 |
By the final feature map the image has shrunk from 262,144 spatial positions to just 256 (a factor of 1024, since five halvings in each dimension give 25 × 25 = 32 × 32 = 1024), but each of those 256 positions now carries 512 learned features — one number per question the network learned to ask. A detection head sliding over this 16×16 map effectively classifies 256 coarse regions of the original image in a single pass — the slid-template idea of Chapter 3, made efficient and learned. The downsampling is what makes this affordable; the depth is what makes it meaningful.
How does a single feature-map cell predict a box? It doesn't predict box coordinates from scratch — it adjusts a set of pre-defined reference boxes called anchor boxes. At each cell sit several anchors of different shapes (a tall one for pedestrians, a wide one for cars, a square one for signs). The network's job is only to predict small offsets — "shift this anchor 4 pixels left, stretch it 1.2× taller" — plus a class and a confidence. Predicting an offset from a sensible prior is far easier to learn than predicting raw coordinates, and it lets one cell handle objects of several shapes at once.
Two families differ in how they get from feature map to boxes:
Two-stage (Faster R-CNN). Stage one proposes a few hundred candidate regions ("something object-ish is here"); stage two classifies and refines each. Slower, but historically the most accurate — the proposal step focuses compute where it matters.
One-stage (YOLO, SSD, RetinaNet). Skip the proposal step: predict boxes and classes directly from the feature map in a single pass. Faster — often real-time — which is why one-stage detectors dominate on robots, where the perception budget is a few milliseconds per frame.
For a robot the choice is usually one-stage: a TurtleBot or a self-driving car needs detections at 10–30 Hz to feed the loop, and a slow but slightly-more-accurate detector that can't keep up is worse than a fast one that can. The whole point of all this machinery is to turn the brutal 24,000-window sweep of Chapter 3 into one efficient forward pass.
To compare detectors — and to tune the thresholds you'll turn in Chapter 9 — we need to score the output against ground-truth boxes. A predicted box counts as a true positive (TP) if its IoU with a real box of the same class clears a threshold (commonly 0.5). A prediction matching nothing is a false positive (FP); a real object with no matching prediction is a false negative (FN, a miss). From those three counts:
Worked example. A frame has 5 real stop signs. The detector outputs 6 boxes: 4 correctly land on real signs (TP = 4), 2 land on billboards (FP = 2), and 1 real sign is missed (FN = 1). Then precision = 4/(4+2) = 0.67 and recall = 4/(4+1) = 0.80. The detector finds most signs (good recall) but also hallucinates some (mediocre precision).
Here is the crux: the confidence threshold trades precision against recall. Raise it and you keep only confident boxes — fewer false positives (precision up) but more misses (recall down). Lower it and you catch more real objects (recall up) at the cost of more junk (precision down). Sweeping the threshold traces a precision–recall curve; the area under it, averaged over classes, is mean Average Precision (mAP) — the single number papers report. For a robot, the right operating point is a safety decision, not a benchmark one: near an intersection you bias toward recall (never miss a sign, tolerate some false stops); on an empty highway you bias toward precision (don't brake for phantoms).
python import torch # Using a pretrained one-stage-style detector (torchvision). The whole sliding # window + scales + NMS pipeline is now ONE call. from torchvision.models.detection import ssdlite320_mobilenet_v3_large model = ssdlite320_mobilenet_v3_large(weights="DEFAULT").eval() image = load_image("intersection.jpg") # tensor, shape [3, H, W], in [0,1] with torch.no_grad(): out = model([image])[0] # NMS already applied internally # out['boxes'] : [N, 4] (x1,y1,x2,y2) -- the WHERE # out['labels'] : [N] class ids -- the WHAT # out['scores'] : [N] confidences -- HOW SURE for box, label, score in zip(out['boxes'], out['labels'], out['scores']): if score > 0.5 and label == STOP_SIGN_ID: trigger_executive('STOP_SIGN_DETECTED', box, score) # -> Chapter 8
A box says "a stop sign is roughly here." Sometimes a robot needs the exact shape: which precise pixels are drivable road, which are sidewalk, which are the person's silhouette. That is semantic segmentation — classify every pixel — and it produces the densest, most detailed semantic output of all.
Think of it as classification done H × W times, once per pixel, but with the network sharing computation so neighbouring pixels inform each other. In goes H × W × 3; out comes an H × W label map (or, before the final argmax, an H × W × C tensor of per-pixel class scores). Painted by class, that label map is the segmentation mask.
There's an architectural puzzle. A classification backbone (Chapter 6) shrinks the image — an H×W image becomes a tiny H/16×W/16 feature map — because to recognize what an object is you can throw away fine spatial detail. But segmentation needs a full-resolution output: a label for every original pixel. How do you get the spatial detail back?
The standard answer is the encoder–decoder (think of the architecture as an hourglass):
The encoder (the downsampling backbone) figures out what's in the scene. The decoder upsamples step by step back to full resolution, recovering where each thing is. Crucially, skip connections wire high-resolution early-layer features directly across to the decoder — that's how crisp boundaries (where exactly does the road's edge fall?) survive the round trip through the shrunken bottleneck. The famous instances of this design are U-Net (medical imaging) and the segmentation heads in DeepLab/FCN.
How does the H × W × C score tensor become a single label per pixel? By argmax: at each pixel, pick the class with the highest score. Suppose we have 3 classes (road, sidewalk, person) and look at one pixel whose three scores (logits) are (3.1, 1.2, 0.4). The largest is 3.1 (class 0), so this pixel is labelled road. A neighbouring pixel with scores (0.2, 0.5, 2.8) has its max at index 2, so it is labelled person. Run argmax across all H × W pixels and you have the full label map — the mask.
You can apply softmax first if you want a per-pixel confidence (using the Chapter 1 computation): the road pixel's (3.1, 1.2, 0.4) softmaxes to roughly (0.81, 0.12, 0.07) — 81% sure it's road. But for the final mask you only need the argmax; the softmax matters when you want to threshold low-confidence pixels (e.g. mark them "unknown" rather than forcing a guess), which is exactly the safety habit a robot should have at region boundaries.
A segmentation mask feeds the robot's world model directly. The "road" pixels become the free space the planner (Lessons 3–5) searches over. The "person" pixels become a no-go region with a safety margin. Recall from Lesson 1 that this is precisely the semantic map — regions labelled by meaning, not just by occupancy. Learned segmentation is the engine that produces it; the planner consumes it.
python import torch, numpy as np # A segmentation model maps H x W x 3 -> H x W x C (per-pixel class scores). logits = seg_model(image) # shape [C, H, W] -- C class scores per pixel mask = logits.argmax(dim=0) # shape [H, W] -- winning class id per pixel # Turn the mask into things the planner can use: drivable = (mask == ROAD_ID) # boolean H x W: the free-space the planner searches people = (mask == PERSON_ID) # boolean H x W: inflate into a keep-out region print("drivable pixels:", int(drivable.sum()), " person pixels:", int(people.sum()))
Below, a toy segmenter colours a scene region by region. Toggle which classes it shows; this is the dense, pixel-by-pixel map of meaning that a box can only approximate.
Press segment to flood the scene with per-pixel class colours (sky, road, building, sign, person). Toggle a class off to hide it. Unlike a box, the mask traces the exact shape of each region — the road's true edge, the person's true silhouette.
Putting the three localization-aware outputs next to each other makes the precision–cost tradeoff concrete:
| Output | Shape per object | Separates touching instances? | Traces true shape? | Robot use |
|---|---|---|---|---|
| Box (detection) | 4 numbers | yes (one box each) | no (rectangle) | "is there a sign? how far?" |
| Semantic mask | shares an H×W map | no (merges into one blob) | yes (per-pixel) | drivable-surface free space |
| Instance mask | one H×W mask each | yes | yes (per-pixel) | count people, track each one |
Read the "separates instances" column carefully — it is the one genuine weakness of semantic segmentation. Two pedestrians walking shoulder to shoulder become a single connected "person" region; the mask cannot tell you there are two of them. If your robot needs to count people, or follow one specific person, you need instance segmentation (which is, in effect, detection + a mask inside each box). Each rung up this ladder costs more compute and more expensively-labelled data — choose the lowest rung that answers your actual question.
This is the chapter the whole series has been building toward: the moment a perception output changes the robot's behaviour. Stanford's hw3 Problem 4 is exactly this — detect a stop sign, and trigger the robot to stop. Let's trace the full path from photons to a stopped TurtleBot, because every arrow in it is a lesson in this course.
Step 1 — Detect. The detector from Chapter 6 runs on the camera frame and returns the stop sign's bounding box and a confidence. This is the "See" beat of the loop fully realized: raw pixels became a typed, meaningful object.
Step 2 — From box to range and bearing. A box is in pixel coordinates; the planner thinks in metres and angles. The bridge is the pinhole camera model from Lesson 8. Two estimates:
Worked range. Suppose fx = 600 pixels (from calibration), the real sign is Wreal = 0.75 m, and the detected box is w = 96 pixels wide. Then range Z = 600 × 0.75 / 96 = 450 / 96 ≈ 4.7 m. The robot is about 4.7 metres from the sign — a number the planner and executive can act on. Drive forward and the box grows: at w = 192 px the range is 450/192 ≈ 2.3 m — half the apparent size means twice as close, the inverse relationship a pinhole camera enforces.
Worked bearing. Suppose the image is 1280 px wide, so the optical center is about cx = 640 px, and the sign's box center sits at column u = 800 px. Then the horizontal offset is u − cx = 800 − 640 = 160 px to the right, and the bearing is β = atan2(160, 600) ≈ 0.262 rad ≈ 15° — the sign is about 15 degrees to the right of straight ahead. Range 4.7 m, bearing +15°: that pair of numbers is the sign's position in the robot's frame, ready for the planner. The detection became geometry.
Step 3 — Decide. Now the thematic payoff. That detection event is exactly the input the executive FSM from Lesson 1 consumes. Recall the FSM is the tuple (S, I, O, n, o, s0). Here:
This is the complete perception→decision link: a learned function (the detector) produces a symbol (“stop sign”) that becomes an input to a symbolic controller (the FSM) that changes the robot's mode. Pixels became a class became an FSM transition became motor commands. The entire see-think-act loop, closed, with semantics in the middle.
Detectors are fallible (Chapter 6). The executive cannot assume perfect perception, so it is designed to degrade gracefully against two failure modes:
Missed detection (false negative). The sign is there but the detector scored it below threshold (fog, glare, occlusion). If the FSM trusted a single frame, it would blow the stop. Defence: require detection across several consecutive frames before clearing a stop, and lower the confidence threshold near intersections (trade false positives for safety). Other sensors (a map prior that "a stop sign is logged here") back up the camera.
False detection (false positive). A red billboard or a tail-light scores as a stop sign. If the FSM stopped on every flicker, the robot would freeze constantly. Defence: require temporal consistency (the same detection, in a plausibly moving box, over k frames) and geometric sanity (range and bearing consistent with a real approach). A one-frame blip is ignored.
The general principle: the executive treats every detection as evidence, not fact. It integrates detections over time and cross-checks them against geometry and the map before committing to a behaviour change — the same "accumulate evidence, don't trust one reading" logic you saw in the occupancy-grid log-odds update of Lesson 1, and that you'll formalize in the filters of Lessons 13–14.
Everything comes together here. Below is a synthetic intersection scene with several objects — a stop sign, a couple of people, a car. A detector has fired raw boxes (with confidences and IoU clutter), and you hold the two dials that turn raw detections into decisions: the confidence threshold and the IoU threshold for NMS. As you turn them, watch the kept boxes change, the false positives and misses appear — and watch the executive FSM badge flip to STOP the instant a stop sign survives both thresholds.
Faint = raw detector boxes. Bold = kept after threshold + NMS. The badge (top-right) is the executive's state. Drag confidence: too high → the real stop sign drops out (a miss, badge stays DRIVE — dangerous). Drag IoU τ: too high → duplicate boxes survive; too low → nearby people get merged. Watch the badge flip to STOP only when a stop sign passes both gates.
Sweep the confidence dial down from high. At a high threshold, only the most confident detection survives — you'll see the robot miss a faintly-detected real object: the badge stays DRIVE when it shouldn't. That is a false negative, and on a real road it is the dangerous one. Now sweep it low: weak detections (the billboard masquerading as a sign) start surviving — false positives — and the robot freezes for nothing. Somewhere in the middle is the operating point, and there is no value that is right for every scene.
Now the IoU dial. Set τ very high and NMS barely suppresses: you'll see two or three boxes stacked on the single stop sign (duplicates the detector should have collapsed). Set τ very low and NMS gets greedy: stand the two people close together and one of them vanishes — NMS deleted a real object because its box overlapped its neighbour's. The dials interact, the failure modes are real, and a perception engineer spends a lot of time exactly here.
In every one of these, nothing about the detector changed — only the two post-processing dials. That is the lesson: a perception system is the detector plus its thresholds, and the thresholds encode a safety policy you choose, not a fact the network hands you.
You now hold the bridge from geometry to semantics: you can take an image, find what is in it and where, clean up the boxes, estimate range and bearing, and hand a symbol to the robot's executive. Here is where this lesson sits on the autonomy stack and where to go next.
| Concept | What it does | Key number / shape |
|---|---|---|
| Template matching (NCC) | slide a fixed template, score by cosine of patch vs template | NCC = (f·t)/(‖f‖‖t‖) ∈ [−1,1]; perfect = 1 |
| Convolution | slide a (learned) filter; multiply-and-sum at every pixel | G(i,j)=Σ F(u,v)·I(i+u,j+v) |
| IoU | measure box overlap, scale-free | inter / (areaA + areaB − inter) ∈ [0,1] |
| NMS | sort by score, keep max, drop IoU > τ duplicates | one box per object |
| CNN detector | backbone → feature map → head → boxes | img H×W×3 → (class, box, score) list |
| Segmentation | per-pixel class via encoder–decoder | H×W×3 → H×W label map |
Where to go deeper on what we touched: