Understand Optical
Flow

Chapter 1: The Flow Field

At every pixel location (x, y) in frame 1, optical flow assigns a displacement vector (u, v). The value u tells how far the pixel moved horizontally, and v tells how far it moved vertically. If a pixel at (100, 50) has flow (3, −2), it moved to (103, 48) in frame 2.

The output of optical flow is a tensor with shape [H, W, 2] — the same height and width as the image, but with 2 channels instead of 3 (RGB). Channel 0 is horizontal displacement u, channel 1 is vertical displacement v. This is a dense flow field: one vector per pixel.

Visualizing Flow: Color Coding

A flow field has two values per pixel, which is hard to show as a grayscale image. The standard visualization uses the HSV color wheel: the direction of motion maps to hue (red = right, cyan = left, green = down, magenta = up), and the magnitude maps to saturation. A stationary pixel is white or black.

Chapter 2: Brightness Constancy

Every classical optical flow method starts with one assumption: a pixel's brightness doesn't change as it moves. If a pixel is bright in frame 1, it's still bright in frame 2 — it just moved to a new location. Formally:

Here I(x, y, t) is the brightness at position (x, y) in frame t. The pixel moved by (u, v), so in the next frame it's at (x+u, y+v). This is the brightness constancy assumption.

Taylor Expansion: One Equation, Two Unknowns

We Taylor-expand the right side around (x, y, t):

Where I_x = ∂I/∂x (horizontal gradient), I_y = ∂I/∂y (vertical gradient), and I_t = ∂I/∂t (temporal change). Substituting the brightness constancy equation, I(x,y,t) cancels on both sides, leaving:

This is the optical flow constraint equation. One equation, two unknowns (u and v). We can't solve it at a single pixel — there are infinitely many (u, v) pairs that satisfy it. This fundamental limitation is called the aperture problem.

Chapter 3: Lucas-Kanade

Lucas and Kanade (1981) had a simple idea: if flow is unknown at one pixel, use the neighbors. Assume that all pixels in a small patch (say 5×5) share the same flow (u, v). Now each pixel contributes one equation I_x·u + I_y·v + I_t = 0, and with 25 pixels we have 25 equations for 2 unknowns. That's an overdetermined system — solve with least squares.

The Math: A 2×2 System

Stack all the pixel equations into a matrix. For a patch of n pixels:

The least-squares solution is:

A^TA is a 2×2 matrix (called the structure tensor). The key: it must be invertible. If the patch is a flat region (no gradient), A^TA is all zeros — no solution. If it's a pure edge, A^TA has one near-zero eigenvalue — ambiguous along the edge. Only at corners (two strong eigenvalues) is the system well-conditioned.

Chapter 4: Horn-Schunck

Lucas-Kanade solves flow locally, patch by patch. Horn and Schunck (1981) took the opposite approach: solve for the entire flow field at once, with a global smoothness constraint. The idea: neighboring pixels should have similar flow, because real objects move coherently.

The Energy Function

Horn-Schunck minimizes a single energy over all pixels:

The first term is the data term: brightness constancy should hold everywhere. The second term is the smoothness term: flow should vary slowly. The parameter α controls the balance — large α means smoother flow (fewer sharp transitions), small α means more faithful to the raw data (noisier).

The minimization leads to a system of equations solved iteratively with Gauss-Seidel or Jacobi iteration. At each iteration, every pixel's flow is updated using its neighbors' flow from the previous iteration. After enough iterations, the flow field converges.

Here ū and v̄ are the local averages of u and v from neighboring pixels. Each iteration nudges the flow toward both the data and its neighbors. α literally controls how much "peer pressure" each pixel feels from its neighbors.

Chapter 5: FlowNet — Learning to See Motion

Classical methods are elegant but slow and brittle. In 2015, Dosovitskiy et al. asked: can a convolutional neural network learn optical flow end-to-end? The answer was FlowNet — the first CNN to predict dense flow directly from an image pair.

FlowNetSimple: The Brute Force Approach

Stack both frames along the channel dimension: input is [6, H, W] (3 RGB channels from each frame). Feed this into an encoder-decoder (contracting path with conv+stride, expanding path with deconv). The output is [2, H, W] — predicted flow.

FlowNetCorr: Explicit Matching

Instead of stacking the images, process each through a separate encoder to get feature maps, then compute a correlation volume that measures how similar each feature in image 1 is to features in image 2 within a search window. This gives the network an explicit matching signal.

Training: Synthetic Data

Where do you get ground-truth flow to supervise training? You render it. The FlyingChairs dataset pastes random chair images onto backgrounds and moves them — the renderer knows exactly how every pixel moved. The loss is simply:

End-point error (EPE): the Euclidean distance between predicted and ground-truth flow, averaged over all pixels. FlowNet achieved ~2.7 pixel EPE on FlyingChairs while running at 10 FPS — orders of magnitude faster than classical methods.

Chapter 6: RAFT — The Modern Standard

RAFT (Recurrent All-pairs Field Transforms, Teed & Deng 2020) is the dominant optical flow method. It won the ECCV 2020 Best Paper Award and every method since builds on its ideas. The core insight: don't predict flow in one shot — iteratively refine it by repeatedly looking up correlations.

Architecture: Three Stages

Stage 1: Feature Extraction. A shared CNN encoder processes both images independently, producing feature maps at 1/8 resolution:

Stage 2: Correlation Volume. Compute the dot product between every feature in f₁ and every feature in f₂. The result is a 4D tensor:

This is the all-pairs correlation volume. Entry C(i,j,k,l) measures how similar the feature at position (i,j) in image 1 is to the feature at (k,l) in image 2. Build it once, look it up many times.

Stage 3: Iterative Update (GRU). Starting from zero flow, a Gated Recurrent Unit refines the estimate over 12 iterations. At each iteration:

Training Details

RAFT supervises at every iteration, not just the last. The loss weights later iterations more heavily:

This encourages each iteration to improve, not just the final output. L1 loss (not L2) because it's less sensitive to outliers at motion boundaries.

Chapter 7: Training on Synthetic Data

Ground-truth optical flow for real video is nearly impossible to obtain. You'd need to know exactly where every single pixel moved — sub-pixel precision for millions of pixels. Instead, the entire field relies on synthetic datasets where the renderer provides perfect ground truth for free.

The Datasets

The Training Schedule

RAFT and its successors follow a specific curriculum. Start simple, increase complexity:

The Generalization Gap

Models trained on synthetic data struggle with real-world challenges that don't appear in training:

• Textureless surfaces — white walls, clear skies (synthetic data has rich textures)
• Reflections and transparencies — glass, water, mirrors break brightness constancy
• Non-Lambertian materials — specular highlights shift with viewpoint
• Motion blur — fast-moving objects smear across the frame
• Fog and rain — atmospheric effects change appearance between frames

Chapter 8: Applications

Optical flow isn't just an academic exercise. It's a core primitive that powers dozens of real systems. Here's how flow enables each application:

Video Stabilization

Shaky handheld video has erratic global motion. Compute optical flow between consecutive frames → estimate the dominant (camera) motion → subtract it → smooth the residual. The result: steady footage as if shot on a gimbal. Every phone camera and YouTube's video stabilizer use this pipeline.

Action Recognition (Two-Stream Networks)

Simonyan & Zisserman (2014) showed that feeding precomputed optical flow as a separate stream alongside RGB dramatically improves action recognition. The spatial stream sees appearance (who/what), the temporal stream sees motion (doing what). Flow compresses temporal information into a single tensor that CNNs can process.

Video Frame Interpolation

Given frames at t=0 and t=1, generate the frame at t=0.5. Compute forward flow (0→1) and backward flow (1→0), then warp both frames to the midpoint and blend. This is how slow-motion video is created from standard 30fps footage.

Autonomous Driving

Optical flow helps self-driving cars detect independently moving objects. The ego-vehicle's motion creates a predictable flow pattern (expansion from the focus of expansion). Anything deviating from this pattern is an independent mover — a pedestrian, cyclist, or other car.

Video Editing

Remove an object from video by computing flow, propagating the inpainting mask across frames along flow trajectories, then filling the hole with temporally consistent content.

Chapter 9: Connections & Beyond

Optical flow is one way to understand motion, but it has clear limitations: it's 2D (ignores depth), pairwise (two frames only), and dense (one vector per pixel, even where it's ambiguous). Modern research addresses each of these.

What Optical Flow Cannot Do

The Broader Landscape

Cheat Sheet