Computer Vision

Three 3×3 layers give a 7×7 receptive field — the same field of view as a single 7×7 kernel, but with fewer parameters (3×9 = 27 vs 49) and more nonlinearities. This is the key insight behind VGG: stack small kernels to get large receptive fields cheaply.

The stride-2 in layer 1 doubles the "jump" — every subsequent 3×3 kernel covers 2×(3−1) = 4 extra input pixels instead of 2. Compare to three stride-1 layers which give rf=7. One stride-2 layer early on grows the receptive field much faster: 11 vs 7 with the same number of layers.

After just the stem and first residual block, each feature map pixel already sees a 35×35 region of the original 224×224 input. By the end of ResNet-50, the receptive field exceeds the entire input image — meaning each final feature map pixel is theoretically influenced by all input pixels.

Dilated convolutions skip every (d−1) pixels. A 3×3 kernel with dilation=2 samples a 5×5 region but only uses 9 weights. This is how DeepLab achieves large receptive fields in segmentation without increasing parameters or reducing resolution.

Three stacked 3×3 layers use 45% fewer parameters than one 7×7 layer while achieving the same receptive field. Plus they add two extra ReLU nonlinearities, making the function more expressive. This is the VGGNet insight (2014) that killed large kernels.

The full 2×2 output is: [[6, 8], [3, 3]]. Max pooling with 2×2 stride 2 halves each dimension and keeps the strongest activation in each quadrant.

GAP averages over the spatial dimensions (H and W) independently for each channel. Input [7, 7, 2048] → each of the 2048 channels gets averaged over 7×7=49 positions → output [1, 1, 2048]. This eliminates 7×7 − 1 = 48 spatial positions per channel, leaving a single number per channel.

GAP saves ~98.3M parameters — a 49x reduction. This is why every modern classifier uses GAP: it collapses spatial information into a compact vector with zero learned parameters, then lets a tiny FC layer do the classification. Introduced by GoogLeNet (2014), GAP also acts as a regularizer since it forces the network to learn spatially-distributed features.

During training, every layer's activations must be stored for backpropagation. ResNet-50 with batch_size=32 stores ~3 GB of activations across all layers. This is why activation checkpointing (recomputing activations instead of storing them) is critical for training large CNNs on limited GPU memory.

In general, the ratio is approximately K² for large channel counts (here 3² = 9, close to our 8.7). MobileNet uses depthwise separable convolutions throughout, achieving ~8-9x parameter reduction per layer with minimal accuracy loss. The general formula: standard/(depthwise+pointwise) = C_out×C_in×K² / (C_in×K² + C_in×C_out) ≈ K² when C is large.

And this is for a single feature map scale. FPN-based detectors like RetinaNet use 5 feature map levels (P3-P7), with sizes like 100×100, 50×50, 25×25, 13×13, 7×7. Total anchors across all levels: 9 × (10000+2500+625+169+49) = 9 × 13,343 = ~120K anchors per image. The vast majority are negative (no object) — this class imbalance motivated focal loss.

An IoU of 0.143 means only 14.3% overlap. In COCO evaluation, a detection is considered a true positive at IoU ≥ 0.5. The COCO primary metric (AP) averages over IoU thresholds from 0.5 to 0.95 in steps of 0.05.

javascript
function computeIoU(a, b) {
  const ix1 = Math.max(a[0], b[0]);
  const iy1 = Math.max(a[1], b[1]);
  const ix2 = Math.min(a[2], b[2]);
  const iy2 = Math.min(a[3], b[3]);
  const iw = Math.max(0, ix2 - ix1);
  const ih = Math.max(0, iy2 - iy1);
  const inter = iw * ih;
  const areaA = (a[2]-a[0]) * (a[3]-a[1]);
  const areaB = (b[2]-b[0]) * (b[3]-b[1]);
  return inter / (areaA + areaB - inter);
}

Anchors in the "gray zone" (IoU 0.3-0.7) are ignored — they contribute no loss during training. This prevents ambiguous examples from corrupting the classifier. Of the ~9K anchors from a 32×32 feature map, typically only ~256 are sampled for training (128 positive + 128 negative), with the rest discarded.

~84% of anchors come from P3 alone (the finest level). This heavy skew toward small objects is intentional — small objects are harder to detect and need denser coverage. But it also means ~75K of these ~77K anchors are negative, which is the class imbalance problem that focal loss addresses.

Line 6 is the bug. When boxes don't overlap, ix2 - ix1 or iy2 - iy1 can be negative. Two negative numbers multiplied give a positive intersection area, which corrupts the IoU. The fix: clamp both dimensions to zero with Math.max(0, ix2 - ix1) and Math.max(0, iy2 - iy1). Without this clamp, non-overlapping boxes can produce IoU > 1 because the "intersection" is positive while the union is smaller than the sum of areas.

Round 1: Pick A (score=0.9). IoU(A,B)≈0.68 ≥ 0.5 → suppress B. IoU(A,C)≈0 < 0.5 → keep. IoU(A,D)≈0 < 0.5 → keep. IoU(A,E)≈0 < 0.5 → keep. Remaining: {C, D, E}.

Round 2: Pick C (score=0.7). IoU(C,D)≈0.65 ≥ 0.5 → suppress D. IoU(C,E)≈0 < 0.5 → keep. Remaining: {E}.

Round 3: Pick E (score=0.3). No remaining detections. Done.

Survivors: A, C, E — three distinct detections for three distinct regions.

Since IoU (0.55) ≥ threshold (0.5), the lower-scoring box is suppressed. This is a false negative caused by NMS. Solutions: (1) raise the threshold to 0.6 or 0.7, but this lets more duplicate detections through, (2) use Soft-NMS which decays the score instead of removing, or (3) use DETR-style set prediction which avoids NMS entirely by predicting a fixed set of unique objects.

javascript
function nms(detections, iouThreshold) {
  function iou(a, b) {
    const ix1=Math.max(a[0],b[0]), iy1=Math.max(a[1],b[1]);
    const ix2=Math.min(a[2],b[2]), iy2=Math.min(a[3],b[3]);
    const inter = Math.max(0,ix2-ix1)*Math.max(0,iy2-iy1);
    const aa=(a[2]-a[0])*(a[3]-a[1]);
    const ab=(b[2]-b[0])*(b[3]-b[1]);
    return inter/(aa+ab-inter);
  }
  const sorted = [...detections].sort((a,b) => b.score-a.score);
  const keep = [];
  const suppressed = new Set();
  for (let i=0; i<sorted.length; i++) {
    if (suppressed.has(i)) continue;
    keep.push(sorted[i]);
    for (let j=i+1; j<sorted.length; j++) {
      if (!suppressed.has(j) && iou(sorted[i].box,sorted[j].box) >= iouThreshold)
        suppressed.add(j);
    }
  }
  return keep;
}

O(N²) complexity. With N=77K anchors this would be ~3 billion IoU ops. In practice, a confidence threshold (e.g., score > 0.05) pre-filters to a few hundred detections before NMS, making it cheap. GPU-accelerated NMS (torchvision.ops.nms) handles the remaining cost.

1. Raw network output → 2. Decode box offsets (apply anchor transforms to get actual coordinates) → 3. Apply confidence threshold (remove low-score detections) → 4. Per-class NMS (remove duplicates within each class) → 5. Keep top-K (limit final output count)

ResNet-50 total is ~25.6M parameters. Stage 3 (1024-channel blocks) and Stage 4 (2048-channel blocks) dominate because parameter count scales with C². The full count: conv1 (9.4K) + Stage1 (0.2M) + Stage2 (1.2M) + Stage3 (7.1M) + Stage4 (15.0M) + FC (2.0M) ≈ 25.6M.

Note: FLOPs ≈ 2 × MACs (one multiply + one add per MAC). So this layer uses ~3.7 GFLOPs. The key insight: FLOPs scale with spatial area. The same conv layer at 28×28 uses 4x fewer FLOPs than at 56×56. This is why downsampling early is computationally efficient — most FLOPs are spent in the early, high-resolution layers.

Even with per-layer gradients of 0.9 (close to 1!), 100 layers of multiplication shrink the signal to near zero. This is the vanishing gradient problem. ResNet's skip connections bypass this chain by providing an additive gradient path: the gradient through the identity shortcut is always 1, regardless of depth.

Line 3 is the bug. The first conv changes channels from 256 to 512, but the second conv brings it back to 256. While the final shape matches x, the issue is that W1 projects to 512 channels unnecessarily. In a standard ResNet basic block, both convolutions should maintain the same channel count (256→256, 256→256). If you need to change channels, you must add a 1×1 projection shortcut: return out + conv1x1(x, Ws). The code as written doesn't crash on shapes (both are [batch,256,56,56]), but the intermediate 512-channel expansion is wrong for a basic block. For a bottleneck block, the expansion goes 256→64→64→256 (reduce-process-expand), not 256→512→256.

ResNet-50 uses ~4.1 GMACs (~8.2 GFLOPs) for a 224×224 input. The majority of compute is in Stage 2 (56×56, 128 channels) and Stage 3 (28×28, 256 channels). Despite Stage 4 having the most parameters (512 channels), it operates at 7×7 resolution, so its FLOP contribution is small. This is a general principle: parameters concentrate in deep layers, FLOPs concentrate in early layers.

The combination K=4, S=2, P=1 is the standard "doubling" transposed convolution. It exactly doubles the spatial dimension: 7→14, 14→28, 28→56, 56→112. This is the mirror image of stride-2 convolution's halving.

The bottleneck is 16×16. The decoder path mirrors this: 16 → 32 → 64 → 128 → 256, with skip connections concatenating encoder features at each level. The channel counts typically double on the way down (64→128→256→512→1024) and halve on the way up.

The skip concatenation doubles the input channels, which quadruples the conv parameters compared to a conv without skip (256×256×9 = 590K vs 512×256×9 = 1.18M). This is the parameter cost of skip connections, but the boundary quality improvement is dramatic.

The output is [512, 512, 21] — a full-resolution prediction with 21 logits per pixel. After applying argmax along the channel dimension, you get a [512, 512] label map. The final layer is a 1×1 convolution that maps from the decoder's channel dimension to the number of classes. Loss: per-pixel cross-entropy between predicted logits and ground-truth labels.

Bilinear + conv is cheaper and often preferred because transposed convolutions can produce "checkerboard" artifacts when K is not divisible by S. Modern segmentation networks (e.g., DeepLabV3+) typically use bilinear upsampling followed by convolution.

12 pixels of disparity is easy to detect reliably. For comparison, an object at 50 meters would have d = 500 × 0.12 / 50 = 1.2 pixels — much harder, requiring sub-pixel matching algorithms.

A ±1 pixel error at 10m depth produces ~3.4m of uncertainty. Compare to an object at 2m (d=30 pixels): Z_min=60/31=1.94m, Z_max=60/29=2.07m, range=0.13m. Depth precision degrades quadratically with distance: ΔZ ≈ Z²/(fB) × Δd.

For a rectified stereo pair, the epipolar lines are horizontal — every corresponding point has the same y-coordinate. This reduces stereo matching from a 2D search to a 1D search along the same scanline. Rectification is a preprocessing step that warps both images so that epipolar lines become horizontal, using the camera calibration parameters. Mathematically, the fundamental matrix F of a rectified pair has the form F = [0, 0, 0; 0, 0, -1; 0, 1, 0] (up to scale), enforcing y' = y.

A search range of 58 pixels means for each pixel in the left image, you must compare 58 candidate positions in the right image. For a 640×480 image: 640 × 480 × 58 = ~17.8 million comparisons. Modern stereo networks (e.g., RAFT-Stereo) construct a 3D cost volume of this size and process it with 3D convolutions.

This H applies 2x scaling in x (h₁₁=2) plus translation by 1 in x (h₁₃=1). Point (1,0) becomes (2×1+1, 0) = (3, 0). The h₃₁ and h₃₂ entries (both 0 here) create perspective distortion — when they're nonzero, the denominator w' varies across the image, creating foreshortening.

Each point correspondence (x,y) → (x',y') provides 2 equations (one constraining x', one constraining y'). With 8 unknowns, the minimal solution requires 8/2 = 4 point pairs. This is the Direct Linear Transform (DLT) algorithm. In practice, RANSAC picks random 4-point subsets, solves H, counts inliers, and repeats to find the best H robust to outlier matches.

Only 34 iterations needed even with 40% outliers. This is why RANSAC is so practical — even heavily contaminated matches can be handled quickly. With 80% outliers, the same calculation gives N = ln(0.01)/ln(1−0.2⁴) = ~2871 iterations, which is still fast for modern hardware.

1. Detect keypoints (SIFT, ORB, or SuperPoint) → 2. Match descriptors (nearest-neighbor in descriptor space) → 3. RANSAC + homography estimation (robust fitting from matches) → 4. Warp image (apply H to align to reference frame) → 5. Blend (feather/multiband blend for seamless compositing)

A homography maps a single plane. When the scene contains objects at multiple depths and the camera translates (not just rotates), nearby and distant objects shift by different amounts (parallax). No single homography can capture this — you'd need one homography per depth plane, or a full 3D model. This is why panorama stitching works best with pure rotation (all points are "at infinity" relative to the rotation center) or for flat scenes.

The actual YOLOv8-M backbone uses ~8-12 GMACs depending on exact block structure (CSP bottlenecks have fewer MACs than plain convs). The key takeaway: most compute is in the early high-resolution stages. Stage 1 at 320×320 uses as many FLOPs as Stage 5 at 20×20 despite having 12x fewer channels, because spatial area dominates.

Compare to anchor-based Faster R-CNN: 9 anchors per location would give 9 × 8,400 = 75,600 predictions. YOLOv8's anchor-free design reduces predictions by 9x, which dramatically speeds up post-processing (NMS). Each prediction includes: 4 box coordinates + 1 objectness + C class scores = 4 + 1 + 80 = 85 values for COCO.

NMS is negligible at 0.2ms. The bottleneck in a detection pipeline is always the backbone forward pass (5-15ms on GPU), not post-processing. This is why optimizing the backbone (pruning, quantization, TensorRT) gives the biggest latency wins.

Under 1ms for the forward pass — easily supports real-time detection at 30+ FPS even with post-processing overhead. In practice, YOLOv8-M achieves ~2-3ms on RTX 4090 because of memory transfer latency, kernel launch overhead, and some layers being memory-bandwidth-bound rather than compute-bound. TensorRT optimization can bring this closer to the theoretical limit.

Preprocess (~1ms) → Backbone + FPN (~3ms) → Head predictions (~0.5ms) → Decode + NMS (~0.5ms) → Draw + display (~1ms). Total ≈ 6ms, well under 33ms. The backbone dominates at ~50% of total latency, confirming it's the optimization target. With pipelining (process frame N+1 while displaying frame N), you can overlap stages and further reduce effective latency.

For memory-bandwidth-bound layers, throughput scales linearly with model size reduction: ~4x faster. For compute-bound layers, INT8 tensor cores give ~2x the throughput of FP16. In practice, INT8 YOLOv8 achieves ~2-3x end-to-end speedup with minimal accuracy loss (<1% mAP drop on COCO). TensorRT's post-training quantization handles the conversion automatically.