Learning the Stack: Imitation, VLAs, Splats & World Models

Chapter 1: Imitation Learning — Learn the Policy by Copying

Hand-tuning a controller for a hard task is painful. So here is the most direct learning idea in all of robotics: just watch an expert do it, and copy. A human teleoperates the robot through the task a few hundred times. We record, at every instant, what the robot saw and what the human did. Then we train a function to reproduce the human's action from the observation. That function is the robot's new policy.

Let's name the pieces precisely. At each timestep the robot has an observation o (a camera image, a LiDAR scan, a joint-angle vector — whatever the sensors give) and takes an action a (a steering angle, a target velocity, a gripper command). A policy is a function that picks the action from the observation:

The subscript θ (theta) is the policy's parameters — for a neural network, the millions of weights. Training means searching for the θ that best mimics the expert. The expert hands us a dataset of demonstrations: pairs of (what was seen, what the expert did),

where the star on a* marks "the expert's action" — the label we want to predict. With observations and labels, this is now an ordinary supervised learning problem, identical in form to image classification. We minimize the average mismatch between the policy's action and the expert's:

Here ‖ · ‖² is squared error (for continuous actions like steering); for discrete actions you'd use a cross-entropy loss instead, exactly as in classification. Minimize ℒ by gradient descent and you have a policy. This recipe is called behavior cloning (BC) — the robot clones the expert's behavior.

From scratch: behavior cloning in fifteen lines

To see there is no magic, here is BC written as plainly as possible — a tiny linear policy trained on (observation, expert-action) pairs:

python
import numpy as np

# Demonstrations: each row o_i is an observation, a_i is the expert's action.
O = np.array([[1.0, 0.2], [0.9, 0.4], [0.3, 0.8], [0.1, 0.9]])   # N x d_obs
A = np.array([0.7, 0.6, 0.2, 0.1])                                # N (expert actions)

# Linear policy: a_hat = O @ theta. (A neural net just replaces this line.)
theta = np.zeros(O.shape[1])
lr = 0.1
for step in range(2000):
    a_hat = O @ theta                       # predicted actions, shape (N,)
    err   = a_hat - A                        # residual = policy minus expert
    grad  = (2/len(A)) * O.T @ err          # gradient of mean-squared error
    theta = theta - lr * grad                # gradient-descent step

print("learned policy weights:", theta.round(3))
print("on a NEW observation [0.5, 0.6] -> action", round(float(np.array([0.5, 0.6]) @ theta), 3))

That's the entire engine. A modern visuomotor policy swaps the line a_hat = O @ theta for a convolutional encoder plus a few dense layers, and the action becomes a vector, but the loop — predict, compare to expert, descend — is byte-for-byte the same.

One gradient step, by hand

Let's grind a single step so there is no mystery. Take just two demonstrations, each with a one-number observation and a one-number expert action: (o₁, a₁*) = (2, 1) and (o₂, a₂*) = (4, 3). The policy is the single-weight linear map π_θ(o) = θ·o, starting at θ = 0. Learning rate η = 0.05.

• Predict. With θ = 0: â₁ = 0·2 = 0 and â₂ = 0·4 = 0.
• Residuals (predicted − expert). r₁ = 0 − 1 = −1, r₂ = 0 − 3 = −3.
• Gradient of mean-squared loss. ∂ℒ/∂θ = (2/N) Σ_i r_i o_i = (2/2)[(−1)(2) + (−3)(4)] = −2 − 12 = −14.
• Descend. θ ← θ − η(−14) = 0 + 0.05·14 = 0.70.

After one step θ = 0.70. Check it improved: now â₂ = 0.70·4 = 2.8, much closer to the target 3 than the 0 we started with. Repeat the loop and θ climbs toward the value that best fits both demos (here ≈ 0.70, since the two demos pull slightly differently). That is all gradient descent does: nudge the weight in the direction that shrinks the average squared mismatch with the expert. Replace the scalar θ with millions of weights and the scalar o with an image, and you have a deep visuomotor policy — same three steps, predict-residual-descend.

The idiomatic version is one PyTorch call:

python
import torch, torch.nn as nn
policy = nn.Sequential(nn.Linear(d_obs, 256), nn.ReLU(), nn.Linear(256, d_act))
opt = torch.optim.Adam(policy.parameters(), lr=1e-3)
for o, a_star in demo_loader:                 # mini-batches of (obs, expert action)
    loss = ((policy(o) - a_star)**2).mean()      # the same squared-error objective
    opt.zero_grad(); loss.backward(); opt.step()

Watch it learn below. An expert (teal) traces a curving lane. The BC policy (warm) starts knowing nothing; each demonstration it sees nudges its weights toward matching the expert's steering, and you can watch its predicted trajectory snap onto the expert's as the training set grows.

Data flow, concretely. A real image-based BC policy takes an observation of shape 3×224×224 (a color image), runs it through a frozen-or-fine-tuned vision encoder to a feature vector of length, say, 512, then through a small action head (two dense layers) to an action vector of length, say, 7 (a 6-DoF arm pose plus a gripper). The label is the human's recorded 7-vector at that frame. Loss is mean-squared error per dimension. Nothing here is exotic — it is image classification with a continuous, multi-dimensional label.

Why not just use reinforcement learning?

A fair question, since RL also produces a policy. The difference is what you must supply. RL needs a reward function (a number scoring every state) and exploration — the robot tries things, mostly fails, and slowly discovers what earns reward. For a delicate manipulation task, hand-writing a reward that captures "did it fold the towel nicely" is often harder than the task itself, and exploration on real hardware is slow and dangerous.

Imitation sidesteps both: the expert is the reward signal, implicitly. You never write down what "good" means; you just show it. The cost is that you can never be better than your expert (BC caps at human performance) and you inherit the distribution-shift problem. The honest framing:

In practice the frontier blends them: imitation to bootstrap a competent policy from demos, then RL to polish it past the expert — but the workhorse for "get a robot doing a new task this week" is imitation, because demonstrations are the cheapest signal a human can give a machine.

Chapter 2: The Distribution-Shift Problem — Why Copying Drifts

Behavior cloning trains the policy on the states the expert visited. But the expert is good — the expert stays in the center of the lane, never near the ditch. So the dataset contains almost no examples of "you are halfway into the ditch, here is how to recover." And here is the trap: the moment the clone makes even a tiny error, it nudges the robot slightly off the expert's path — into a state the policy never saw in training. There, it has no idea what to do, so it errs again, drifting further off, into states even more foreign. The error compounds.

The worked compounding-error estimate

Let's put numbers on it. Suppose the trained policy makes a mistake with small probability ε (epsilon) on any single step — say ε = 0.01, a 1% per-step error rate. The robot runs for a horizon of T steps. How much total error do we expect over the episode?

The key fact: once the policy makes its first mistake, it has fallen off the training distribution, and BC was never trained to recover — so we pessimistically assume it stays off and pays a cost on every remaining step. Step through it:

• The first mistake happens, on average, around step 1/ε. With ε = 0.01, that's around step 100.
• A mistake at step t leaves roughly (T − t) steps of accumulated off-distribution cost after it.
• Sum the expected cost over all steps where a first mistake could occur. The probability of a first slip at step t times the (T − t) steps it ruins, summed, grows like the area of a triangle — quadratic in T.

The rigorous result (Ross & Bagnell, 2010) is that the expected total cost of a behavior-cloned policy scales as:

Read that carefully. It is not ε·T (one mistake per step, what you'd naively expect). It is ε·T² — the horizon enters squared, because each early mistake poisons all the steps after it. Plug in numbers:

Notice the killer in row two: doubling the horizon roughly quadruples the error. A policy that's 99% reliable per step is not 99% reliable over a long task — it is essentially guaranteed to drift off and never come back, because nothing in the data taught it how to come back.

Watch the quadratic appear, in code

You don't have to take the T² on faith. Simulate it. Run a fleet of cloned robots; each step, with probability ε, a robot slips off-distribution and — because BC can't recover — pays a unit cost on every remaining step. Sum the costs and compare horizons:

python
import numpy as np
rng = np.random.default_rng(0)

def bc_total_cost(eps, T, fleet=5000):
    off = np.zeros(fleet, dtype=bool)        # has each robot fallen off-distribution?
    cost = 0.0
    for t in range(T):
        # a robot still on-distribution slips this step with prob eps
        slip = (~off) & (rng.random(fleet) < eps)
        off = off | slip                      # once off, stays off (no recovery in the data)
        cost += off.mean()                    # every off robot pays 1 this step
    return cost / fleet * fleet               # expected total cost (already a mean over fleet)

eps = 0.01
for T in [30, 60, 120]:
    c = bc_total_cost(eps, T)
    print(f"T={T:3d}  total cost={c:6.1f}   cost/(eps*T^2)={c/(eps*T*T):.2f}")
# T= 30  total cost=   4.2   cost/(eps*T^2)=0.47
# T= 60  total cost=  16.8   cost/(eps*T^2)=0.47   <- 4x the T=30 cost
# T=120  total cost=  67.0   cost/(eps*T^2)=0.47   <- 4x again

The ratio cost / (eps*T^2) stays roughly constant across horizons — that constant is the hidden proof that the cost really is proportional to εT². Double T and the cost quadruples, every time. The simulation has no neural network in it at all; the blowup is a property of the feedback structure (acting chooses your next state), not of any particular model.

The fix: DAgger (Dataset Aggregation)

DAgger attacks the root cause directly. The problem is that the policy visits states the expert never demonstrated. So: let the policy drive, collect the off-distribution states it actually wanders into, and ask the expert what they would have done there. Add those labels to the dataset and retrain. Iterate.

Now the dataset covers the recovery states. The policy learns "when I'm halfway into the ditch, steer back." Because the training distribution now matches the distribution the policy induces, the compounding feedback is broken, and the error scales linearly:

Here is DAgger as a loop — notice that the only structural change from BC is that the policy, not the expert, generates the states we collect, and the expert only supplies the labels:

python
D = collect_expert_demos()           # start with ordinary BC demos
policy = train_bc(D)
for i in range(num_iterations):
    # 1. let the CURRENT policy drive -> it visits its own (off-distribution) states
    states = rollout(policy)         # the states BC never saw
    # 2. ask the expert what THEY would do at each visited state
    labels = [expert_action(s) for s in states]
    # 3. aggregate the new (state, expert-label) pairs and retrain
    D = D + list(zip(states, labels))
    policy = train_bc(D)             # now trained on the states it actually visits

The subtle, important detail: the labels come from the expert, but the states come from the policy. That is what aligns train and test distributions. After a few iterations the policy's rollouts stop wandering into unlabeled territory, because that territory got labeled. The O(εT2) triangle collapses to a thin O(εT) line.

The horizon enters to the first power, not squared. The cost of this fix is real: DAgger needs an interactive expert — a human (or a slower, known-good controller) available to label new states during training. That's expensive, and HG-DAgger and other variants reduce the human burden, but the principle stands.

Below, run both policies side by side. The expert's path is fixed; tune the per-step error ε and the horizon T and watch the BC robot drift off quadratically while the DAgger robot — trained on recovery states — stays on track.

Chapter 3: Modern Visuomotor Policies — From Images to Actions

BC's recipe is sound; the hard part is the function π_θ when the observation is a raw image and the action is a delicate sequence of arm motions. A linear map won't cut it. Modern visuomotor policies — "visuo" (vision) + "motor" (action) — are deep networks that map pixels to motor commands, and two ideas make them work where naive BC fails: diffusion action heads and action chunking.

The data flow, with shapes

A visuomotor policy is a pipeline of typed tensors. Let's trace one frame through it:

The vision encoder is often pretrained on internet images (so it already knows edges, textures, objects) and then fine-tuned on robot frames; the action head is trained from scratch on the demonstrations. Freezing vs. fine-tuning the encoder is a real design knob: freeze it and you keep its broad visual knowledge but can't adapt it to the robot's odd camera; fine-tune it and you adapt but risk overwriting that knowledge with a few hundred demos.

Why diffusion for the action head?

A plain regression head predicts one action and minimizes squared error. But expert demonstrations are multimodal: faced with an obstacle, a human might go left or right, both fine. Squared-error regression averages them — and the average is "go straight into the obstacle." This is the mode-averaging disaster.

Diffusion policies fix it by borrowing the trick from image generation (the Diffusion lesson). Instead of predicting the action directly, the network learns to denoise a random action back into a good one. Training: take an expert action a*, corrupt it with Gaussian noise to get a_k = a* + noise at noise level k, and train the network to predict the noise that was added, conditioned on the observation:

where ε is the actual noise added and ε̂_θ is the network's prediction of it. At run time you start from pure noise and run the denoiser a handful of steps; the action emerges. Because denoising can settle into either the "go left" basin or the "go right" basin depending on the random start, it captures both modes instead of averaging them.

A single denoising step, by hand

Denoising is just "take a small step from your current guess toward a cleaner one." With a current noisy action guess a_k and the network's noise prediction ε̂, one step subtracts a scaled fraction of the predicted noise:

Suppose the true target action is a* = 0.50. We start from a noisy guess a₃ = 0.90 (way off). The network, conditioned on the observation, predicts the noise direction; with step size α = 0.5 each step roughly halves the gap:

Four cheap steps converge the random guess onto the clean action. The number of steps trades speed for quality; robot diffusion policies typically use a handful (real-time inference can't afford the hundreds that image generation uses).

From scratch: a denoising sampler in a few lines

Here is the sampling loop with a stand-in for the learned denoiser, so the control flow is visible. The real network eps_net takes the current noisy action, the observation, and the noise level, and predicts the noise to remove:

python
import numpy as np

def sample_action(eps_net, obs, n_steps=6, d_act=7):
    a = np.random.randn(d_act)               # start from PURE NOISE (a random action)
    for k in range(n_steps, 0, -1):       # walk noise level k from high -> 0
        noise_pred = eps_net(a, obs, k)      # network: what noise is in this guess?
        alpha = 1.0 / k                       # step size shrinks as we near the clean action
        a = a - alpha * noise_pred           # subtract a fraction of the predicted noise
    return a                                 # a clean, valid action chunk

# Different random starts -> different valid modes. That is the whole point:
# run sample_action twice on the SAME obs and you may get "reach left" once,
# "reach right" the next -- both correct, never the bad average of the two.

The training side is the mirror image: corrupt an expert action with known noise, ask the network to predict that noise, and minimize the squared error of the prediction (the loss above). At run time you never have the clean action to start from — so you start from noise and let the trained eps_net carve it back to a valid action. The randomness in the starting noise is precisely what lets the policy commit to one mode instead of averaging.

Action chunking

The second idea: don't predict one action and re-plan every frame. Predict a chunk — a short sequence of the next H actions (say the next 8–16 steps) — and execute several of them open-loop before re-querying. This is the trick behind ACT (Action Chunking with Transformers). Why it helps:

• Fewer decision points. If you only decide every 8 steps, you have 8× fewer chances to make a compounding mistake — chunking directly shrinks the effective horizon in the T² law.
• Smoother, more committed motion. Predicting a whole trajectory avoids the jitter and indecision of re-deciding every frame (the "should I go left or right" dithering at the obstacle).
• Handles non-Markovian demos. Humans demonstrate with intent over time; a chunk can capture "reach, then grasp, then lift" as one coherent unit.

Put the first point in numbers, using Chapter 2's law. Suppose a task is 240 control steps long with per-step error ε = 0.01. Decide every step and you have T = 240 decision points, so the compounding cost scales like ε·240² = 0.01·57600 ≈ 576. Execute chunks of 8 and you only make a fresh, error-prone decision every 8 steps — T_eff = 240/8 = 30 decision points — so the cost scales like ε·30² = 0.01·900 = 9. Chunking shrank the decision horizon 8×, which cut the squared-horizon cost roughly 64×. (You don't get this for free: within a chunk you're acting open-loop, so very long chunks stop reacting to surprises — the same open-loop-vs-closed-loop tension from Lesson 1. Chunk length is a real, tuned knob.)

Chapter 4: Vision-Language-Action Models — One Model, Any Instruction

A diffusion policy trained on one task does one task. To get a robot that responds to "put the cup in the sink" today and "fold the towel" tomorrow — with no retraining — we need the policy to understand language and to generalize to objects it has never manipulated. That is the promise of Vision-Language-Action models (VLAs): one model that takes a camera image and a natural-language instruction and outputs robot actions.

The recipe: a VLM backbone with an action decoder

Strip a VLA to its skeleton and it is two parts:

• A pretrained VLM backbone. A large transformer (the kind behind image-chat assistants) that ingests the image (tokenized into visual tokens) and the instruction (tokenized into word tokens) and produces a rich joint representation. This is pretrained on web-scale image+text and carries all the semantic knowledge.
• An action decoder. The piece that turns that representation into robot actions. Two flavors exist, and the distinction is the whole story of the field's evolution.

Flavor 1 — actions as text tokens (RT-2)

RT-2 (Google, 2023) had the audacious idea: treat actions as just another language. Discretize each action dimension into, say, 256 bins, and assign each bin a token — reusing rarely-used tokens from the VLM's existing vocabulary. Now "move the gripper +2cm in x" is literally a word the model emits, exactly like emitting "cat." The VLM is fine-tuned to, given image + "pick up the bag of chips," output a string of action-tokens that decode to motor commands.

The payoff: because actions live in the same token space as language, the model can co-train on internet vision-language data and robot data simultaneously, and its web knowledge bleeds into its actions. RT-2 could pick up "the extinct animal" (a toy dinosaur) it never saw labeled in robot data — the VLM knew what extinct meant.

Action tokenization, by hand

How does a continuous motor command become a discrete token? Bin it. Take one action dimension — say end-effector velocity in x, which ranges over [−1, +1] m/s — and chop that range into 256 equal bins. The bin index is the token. To encode v_x = +0.30 m/s:

• Normalize to [0, 1]: (0.30 − (−1)) / (1 − (−1)) = 1.30 / 2 = 0.65.
• Scale to 256 bins: floor(0.65 × 255) = floor(165.75) = 165.
• Token = bin 165 (mapped onto one of the VLM vocabulary's least-used word tokens).

Decoding reverses it: token 165 → 165/255 = 0.647 → 0.647×2 − 1 = +0.294 m/s. The round-trip lost a sliver (0.30 vs 0.294) — that's quantization error, the price of pretending a continuous action is a word. A 7-DoF action becomes 7 tokens, emitted one at a time. This is why tokenized VLAs are smooth-enough but not buttery: 256 bins per dimension is coarse, and autoregressive emission is slow. That coarseness and latency are exactly what π₀'s continuous action expert removes.

Flavor 2 — a continuous action expert (OpenVLA, π₀)

Tokenizing continuous actions is lossy and slow (you emit one token per dimension, autoregressively). OpenVLA (open-source, 2024) keeps the discretized-token approach but on an open Llama-based backbone. π₀ (pi-zero, Physical Intelligence, 2024) goes further: it attaches a continuous action expert — a small flow-matching/diffusion head (Chapter 3's idea) — to the VLM, so it emits smooth, high-frequency action chunks directly instead of discrete tokens. This gives dexterous, 50 Hz control while keeping the VLM's semantic generalization.

The data flow, end to end

The generalization is the magic. Train on "pick up the apple" with apples, and at test time say "pick up the banana" with a banana the robot never manipulated — it often works, because the VLM grounds "banana" the same way it grounds "apple," and the action-grounding transfers. This is generalization to new objects and new instructions, the thing the classical stack could never do.

The forward pass, with shapes

To make "data flow" precise, here is one inference call with tensor shapes annotated — the contract a VLA implements:

python
img    = camera()                          # (3, 224, 224)  RGB frame
instr  = "put the cup in the sink"           # a string

vis_tok = vision_encoder(img)              # (256, 1024)  256 visual tokens, dim 1024
txt_tok = tokenizer(instr)                 # (7, 1024)    7 word tokens, same dim
tokens  = concat(vis_tok, txt_tok)         # (263, 1024)  one sequence, vision + language

h = vlm_backbone(tokens)                   # (263, 1024)  joint representation (FROZEN/fine-tuned VLM)
# --- two ways to turn h into actions ---
# RT-2 / OpenVLA:  action = decode_tokens( lm_head(h) )  -> 7 discrete bins -> (7,)
# pi-0:            action_chunk = flow_expert(h)          -> (H, 7) continuous chunk
motor_command = action[:6], gripper = action[6]   # 6-DoF pose + 1 gripper

Read off the design from the shapes. The image becomes 256 visual tokens; the instruction becomes a handful of word tokens; they are concatenated into one sequence so the transformer can attend across vision and language jointly (that cross-attention is how "cup" the word binds to the cup the pixels). The VLM backbone is the pretrained, mostly-frozen part — its weights carry the web knowledge. Only the action machinery at the end is trained heavily on robot demos. Same backbone, two decoder choices: discrete tokens (RT-2/OpenVLA) or a continuous chunk (π₀).

Tap the stages below to expand each one's input/output and see what's frozen vs. trained.

Chapter 5: 3D Gaussian Splatting — A Scene as a Cloud of Blobs

Switch modules. A robot also needs a map — Lesson 1 gave us occupancy grids, feature maps, semantic maps. The frontier adds a photorealistic, learned one. 3D Gaussian Splatting (3DGS), from a landmark 2023 paper, represents a whole scene not as a mesh or a grid but as millions of little 3D Gaussian blobs floating in space — and renders novel views of it in real time, photorealistically, learned purely from a set of photos.

What one Gaussian is

Each splat is a fuzzy 3D ellipsoid carrying four things it learns from the photos:

• Position μ — where the blob's center sits in 3D, a vector (x, y, z).
• Covariance Σ — a 3×3 matrix giving the blob's shape: how stretched and oriented the ellipsoid is (a flat splat tiles a wall; a round one fills a corner).
• Color c — usually view-dependent (so a shiny surface can look different from different angles), encoded with spherical harmonics.
• Opacity α — how solid vs. transparent the blob is.

A single Gaussian's "intensity" at a 3D point x is the classic bell curve, peaking at its center and fading with distance, shaped by Σ:

Worked example: one 2D Gaussian's contribution to a pixel

Let's compute exactly how much one splat paints a pixel. Take an isotropic 2D Gaussian (round, for simplicity) centered at pixel μ = (10, 10) with "spread" σ = 3 and opacity α = 0.8. How much does it contribute to the pixel at p = (12, 10)?

• Distance from center: p − μ = (2, 0), so squared distance d² = 2² + 0² = 4.
• For a round Gaussian, Σ = σ²I, so the exponent is −½ · d²/σ² = −½ · 4/9 = −0.222.
• Gaussian falloff: G = e^−0.222 = 0.80.
• Weighted by opacity: α · G = 0.8 × 0.80 = 0.64.

So this one splat contributes a weight of 0.64 of its color to that pixel. A pixel right at the center (d = 0) would get the full α × 1 = 0.8; a pixel 9 units away (d² = 81) gets 0.8 × e^−4.5 ≈ 0.009 — almost nothing. The blob fades smoothly, which is exactly why splats blend seamlessly instead of showing hard edges.

Why a full covariance, not just a radius

We used a round splat (one number, σ) for the worked example, but real splats are anisotropic — stretched ellipsoids. That's the job of the 3×3 covariance Σ. A flat, wide splat tiles a wall in one or two splats; a round one would need dozens. To keep Σ a valid covariance (symmetric, positive-definite) while it's optimized, 3DGS factors it as a rotation times a scale:

where S is a diagonal scale matrix (how long the ellipsoid is along each of its own axes) and R is a rotation (how the ellipsoid is tilted in space). The optimizer adjusts the three scales and the rotation, and this product is always a legal covariance — a small but crucial trick, because directly optimizing nine matrix entries would let Σ drift into nonsense.

To render, each 3D Gaussian is projected to the image plane, where it becomes a 2D Gaussian (the splat's footprint). The "spread" you see on screen is that projected 2D covariance — which is why a splat looks bigger when it's near the camera and smaller when far, exactly like real geometry under a pinhole camera (our Lesson 8 projection math). The rasterizer then sorts these 2D footprints by depth and runs the alpha-composite below.

Differentiable rasterization — how the pixel gets its final color

Many splats overlap a pixel. We sort them front-to-back and alpha-composite — the same over-operator as transparent layers in an image editor. Each splat lays down its color times its (Gaussian-weighted) opacity, times however much light is still un-blocked by closer splats:

The crucial word is differentiable: every operation here — the Gaussian falloff, the opacity, the blend — is smooth, so we can take the gradient of "rendered image minus real photo" with respect to every splat's position, shape, color, and opacity. Then gradient descent nudges the splats until the rendered views match the photos. That is the entire training loop: render, compare to photo, backprop into the blobs.

Render one pixel, from scratch

Here is the alpha-compositing of three overlapping splats onto a single pixel — the inner loop of the entire renderer, in plain code:

python
import numpy as np
# three splats over one pixel, already sorted front (near) to back (far):
#   color c (RGB 0..1), Gaussian falloff G at this pixel, base opacity alpha
splats = [
    {'c': np.array([0.9, 0.2, 0.2]), 'G': 0.80, 'a': 0.6},   # front red blob
    {'c': np.array([0.2, 0.8, 0.3]), 'G': 0.50, 'a': 0.7},   # middle green blob
    {'c': np.array([0.2, 0.3, 0.9]), 'G': 0.90, 'a': 0.9},   # back blue blob
]
C = np.zeros(3)        # accumulated pixel color
T = 1.0               # transmittance: how much light is still un-blocked
for s in splats:                       # front to back
    w = s['a'] * s['G']              # this splat's effective opacity at the pixel
    C += s['c'] * w * T               # lay down color, dimmed by remaining light T
    T *= (1.0 - w)                    # less light passes to the splats behind
print("pixel RGB =", C.round(3), "  light left =", round(T, 3))
# front blob (w=0.48) dominates; the blue blob behind contributes almost nothing,
# because the front two already blocked most of the light (T fell fast).

Trace it by hand. The front red blob has effective opacity w = 0.6×0.80 = 0.48, so it contributes 0.48 of its color and leaves transmittance T = 1−0.48 = 0.52. The green blob (w = 0.7×0.50 = 0.35) contributes 0.35×0.52 = 0.18 and drops T to 0.52×0.65 = 0.34. The blue blob, however solid, only gets 0.34 of the light. Front splats occlude back ones — exactly how a real surface hides what's behind it — and because each step is a smooth multiply, the gradient flows back through all of it to every splat.

Where the splats come from: densification

Training doesn't start with millions of splats — it starts with a sparse point cloud (from structure-from-motion, our Lesson 9 idea: match features across photos to triangulate 3D points). Then, as gradient descent runs, the optimizer densifies: where the rendered image is still wrong, it splits large fuzzy splats into smaller ones and clones splats into under-covered regions; where splats are nearly transparent, it prunes them. The cloud grows from thousands to millions, sculpting itself to fit the photos. This adaptive density is why 3DGS captures fine detail without wasting splats on empty air.

3DGS vs. NeRF

The predecessor was NeRF (Neural Radiance Fields), covered in our NeRF & 3DGS lesson. NeRF stores the scene implicitly in a neural network's weights and renders by shooting rays and sampling that network hundreds of times per ray — gorgeous, but slow (minutes per frame to train-render). 3DGS stores the scene explicitly as the splat cloud and rasterizes with a fast GPU sort-and-blend — real-time rendering (100+ FPS) at comparable quality.

Chapter 6: World Models — Let the Robot Dream

Trying an action on real hardware is slow, expensive, and sometimes destructive (a real robot arm that learns by smashing into things doesn't last). What if the robot could learn an internal simulator of its world, run thousands of imagined rollouts inside its own head, and only act in reality once it has a good plan? That internal simulator is a world model — a learned function of the environment's dynamics.

Formally, a world model learns the transition distribution: given the current state and an action, what state comes next?

This is exactly the MDP transition model from our MDP and Model-Based RL lessons — but now learned from data instead of hand-specified. With a learned dynamics model, the robot can do model-based RL: plan or train a policy entirely on imagined trajectories, then deploy. The MBRL lesson covers the spectrum; here we focus on the modern, latent flavor.

Latent dynamics — dream in a small space, not in pixels

Predicting the next raw camera image is wasteful — most pixels are irrelevant background. Modern world models (the Dreamer family) instead learn a compact latent state z — a small vector (say length 32–512) that captures only the task-relevant gist of an observation — and learn the dynamics in that latent space. Three learned pieces:

• Encoder z_t = enc(o_t) — squashes the image (3×64×64) into a latent vector (length 32).
• Latent dynamics z_t+1 = f(z_t, a_t) — predicts the next latent from the current latent and action. This is the "dream" step, and it's cheap: it never touches pixels.
• Decoder / reward head — reconstructs the image (to force the latent to keep real information) and predicts the reward (so we can score imagined plans).

The three pieces, with shapes, so the "dream is cheap" claim is concrete:

python
# --- learned at training time (all three trained jointly on real rollouts) ---
z      = encoder(obs)            # (3,64,64) image  ->  (32,) latent     [the expensive step, once]
z_next = dynamics(z, action)     # (32,) + (act,)   ->  (32,)            [the cheap step, repeated]
img    = decoder(z)              # (32,) -> (3,64,64)   [training crutch: forces z to keep info]
r      = reward_head(z)          # (32,) -> scalar      [so we can score dreams]

# --- at PLAN time: dream entirely in the 32-dim latent, never touch the decoder ---
z = encoder(obs)                 # encode ONE real frame
for t in range(15):                # 15 imagined steps, each just a 32-dim matmul
    a = policy(z)
    z = dynamics(z, a)           # no image ever rendered -> thousands of dreams/sec

Encoding an image is the costly operation; you pay it once at the start of a dream. Every subsequent imagined step is a tiny operation on a 32-vector. That asymmetry — encode once, dream cheaply — is the entire reason latent world models are practical, and why a robot can evaluate thousands of imagined futures in the time it takes to move once for real.

The latent rollout, with shapes

Here is one imagined plan, traced. The robot encodes its current real observation once, then rolls forward in latent space for an imagined horizon H, scoring each step's predicted reward:

python
z = encode(obs)                 # real image (3x64x64) -> latent z (len 32)
total = 0.0
for t in range(H):              # dream H steps WITHOUT touching the real robot
    a = policy(z)               # pick an imagined action from the latent
    z = dynamics(z, a)          # z_{t+1} = f(z_t, a_t)  -- pure latent step, cheap
    total += reward_head(z)     # accumulate predicted reward of the dream
# total = the imagined return of this plan; train the policy to maximize it,
# OR compare many such dreamed plans and execute the best first action for real.

One dream, with numbers

To make the rollout concrete, collapse the latent to a single number z (think "distance to goal," normalized) and let the learned dynamics be the simple rule z_t+1 = z_t + 0.3·a_t, with reward r = 1 − z² (highest when z = 0). The robot encodes its real observation to z₀ = 0.9 and the policy proposes action a = −1 (drive toward the goal). Dream three steps:

The dreamed return is 0.64 + 0.91 + 1.00 = 2.55, and it climbed because driving toward the goal was the right call — all computed without moving the real robot once. Because every step here is differentiable, we can ask "would a slightly different action have scored higher?" and nudge the policy that way. That is training the policy inside the dream: ascend the gradient of the imagined return with respect to the policy's parameters. The Dreamer family does exactly this with a neural dynamics model instead of our toy rule.

Why it transfers to reality: the dynamics model was fit to real data, so a policy that scores well in the dream tends to score well in the world — up to the model's error. DreamerV3 famously learned to collect diamonds in Minecraft from scratch, with no human data, by dreaming. A cousin idea, JEPA (LeCun), predicts in latent space but skips pixel reconstruction entirely, predicting future representations directly — arguing that forcing the model to reconstruct every pixel wastes capacity on irrelevant detail. And video-generation world models (Genie, world-simulator lines) push the same idea toward learned, playable environments you can literally drive an avatar through.

Watch a latent rollout below. The robot encodes one real frame, then "dreams" forward; the imagined trajectory (warm) explores possible futures cheaply in latent space, while the occasional real step (teal) corrects any drift between dream and reality.

Chapter 7: Putting It Together — Learned Blocks in the Classical Stack

We now have five learned ingredients. The engineering question is how much of the Lesson 1 stack to replace. There is a spectrum, and where you land is a deliberate tradeoff between generalization and verifiability.

The spectrum, module by module

Take the four-module stack from Lesson 1 and ask, for each, "hand-built or learned?"

Modular-learned vs. fully end-to-end

Two philosophies sit at the ends of the spectrum:

Safety and verification of learned policies

A hand-built PID controller can be analyzed: you can prove stability margins. A learned policy is a black box — how do you trust it near people? This is an open and active area, and a few practical patterns:

• Safety filters / shields. Wrap the learned policy in a hand-built guard (a control-barrier function, a reachability check) that vetoes any action that would leave a certified-safe set. The learned policy proposes; the verified filter disposes. This is the executive FSM from Lesson 1, upgraded.
• Runtime monitoring. Detect when the policy is out of distribution (its input doesn't look like training data) and hand control back to a conservative fallback — directly addressing Chapter 2's distribution shift.
• Sim-and-stress testing. Because you can't enumerate the open world, you adversarially search a (3DGS-built, Chapter 5) simulator for failure cases before deployment.

The safety filter is small enough to write down. The learned policy proposes; a hand-verified guard checks the proposal against a safe set and substitutes a known-safe fallback if it fails:

python
def act(obs, state):
    a = learned_policy(obs)                  # the black-box VLA / diffusion policy proposes
    if out_of_distribution(obs):             # monitor: does input look like training data?
        return conservative_fallback(state)   # e.g. brake, hold pose -> hand control back
    if not safe(state, a):                  # shield: would 'a' leave the certified-safe set?
        return safe_projection(state, a)     # nearest action that keeps the robot safe
    return a                                 # proposal is in-distribution AND safe -> allow it

Both guard functions — safe() (a control-barrier or reachability check) and out_of_distribution() — are hand-built and verifiable. The learned policy supplies competence and generalization; the verified wrapper supplies the guarantee. You get most of the upside of learning with a hard floor on behavior.

A concrete design decision

Make it real. You're shipping a warehouse picker. Pose the question per module: Perception? Learned — clutter and novel SKUs make hand-coded detectors hopeless, and you have lots of bin images. Localization? Hand-built EKF — the warehouse is mapped, the math is provable, no reason to learn it. Planning? Hand-built A* on the known floor plan — you must prove the robot avoids the racking. Control/grasp? Learned diffusion policy — dexterous grasping across thousands of object shapes is exactly where imitation shines. Executive? A hand-built FSM that safety-filters the learned grasp policy and falls back to "drop and re-approach" when the grasp monitor flags trouble. That hybrid — learned where messy, hand-built where it must be certified — is the 2025 production answer, not "all learned" or "all classical."

Chapter 8: The Frontier — Honest Open Problems

It would be dishonest to end a Stanford course pretending these methods are solved. They are not. Here are the real, unglamorous walls the field is hitting in 2025 — the problems your generation gets to work on.

1. Data scarcity

The internet has trillions of words and billions of images, which is why language and vision models are so good. There is no internet of robot actions. Collecting demonstrations requires a physical robot and a human teleoperator, in real time — orders of magnitude more expensive than scraping text. The Open X-Embodiment effort (pooling robot data across labs) and large teleoperation campaigns help, but robot data remains the binding constraint. This is why VLAs lean so hard on pretrained vision-language knowledge: they borrow web data to compensate for scarce robot data.

2. Generalization

A VLA trained in one lab kitchen often fails in a different kitchen with different lighting, a different table height, a slightly different gripper. The generalization that's so impressive in demos is real but narrow — semantic generalization ("banana" vs "apple") far outpaces physical generalization (a counter 5 cm higher). Bridging that gap is open.

3. Sim-to-real gap

Training in simulation is cheap and safe, but a policy trained in sim rarely transfers cleanly to reality — physics, friction, sensor noise, and lighting never match exactly. Domain randomization (train across randomized sims so reality is "just another variation") and high-fidelity 3DGS reconstructions of real scenes (Chapter 5) help, but the gap is never zero. This is the single most cited reason a paper's robot works and yours doesn't.

Domain randomization is worth seeing concretely, because it is a direct application of Chapter 2's lesson. The insight: if you train on one sim, reality is an out-of-distribution input and the policy fails. So widen the training distribution until reality falls inside it — randomize friction, mass, lighting, textures, camera angle, sensor noise on every episode:

python
for episode in range(N):
    sim.friction = rng.uniform(0.5, 1.5)      # real friction is somewhere in here
    sim.mass     = rng.uniform(0.8, 1.2) * nominal
    sim.lighting = rng.uniform(0.3, 1.0)
    sim.cam_pose = nominal_cam + rng.normal(0, 0.02, 6)
    sim.obs_noise= rng.uniform(0.0, 0.05)
    rollout_and_train(policy, sim)            # policy must work across ALL of these
# A policy that survives this range treats the REAL robot's true
# friction/mass/lighting as "just another sample it already handles."

The policy can't memorize one physics, so it learns a strategy robust across the whole range — and reality, sitting inside that range, is no longer out-of-distribution. It is the same fix as DAgger (cover the states you'll actually encounter), applied to the sim-to-real gap. The catch: you must randomize the right things over the right ranges, and if reality drifts outside your ranges, you're back to a failing OOD input.

4. Evaluation

How do you even measure a generalist robot policy? Each real-world trial is slow and noisy; success rates have huge variance; there is no standard benchmark the way ImageNet standardized vision. Two labs reporting "70% success" may have tested utterly different conditions. Reproducible, fair evaluation of embodied policies is genuinely unsolved.

5. Safety and guarantees

We can prove a PID controller is stable. We cannot, today, prove a billion-parameter VLA will never do something catastrophic. The compounding-error and distribution-shift problems from Chapter 2 are not just training nuisances — they are why a learned policy near a human is hard to certify. Formal verification of learned controllers is a frontier of its own.

Chapter 9: Showcase — Watch Imitation Fail, Then Watch DAgger Fix It

Here is the lesson's payoff, and it dramatizes the single most important idea in modern imitation learning: why naive copying drifts, and what fixes it. An expert traces a winding lane. We launch a fleet of cloned robots. Each has a tiny per-step error ε. Watch the BC fleet fan out into a widening plume of failure — the O(εT²) blowup made visible — while the DAgger fleet, trained on recovery states, hugs the lane.

What you are watching is the difference between memorizing and understanding. The BC fleet memorized the expert's nice center-lane states; the instant a robot leaves them, it has nothing. The DAgger fleet was shown how to recover — it saw the off-lane states and was told what to do there — so its training distribution matches the states it actually visits. Same network, same expert, same ε. The only difference is which states the data covered, and that difference turns a quadratic catastrophe into a linear, survivable error.

Now push it: set ε tiny (0.005) and T huge (160). The BC plume still eventually fans out — a 99.5%-reliable per-step policy is still doomed over a long horizon, because the T² always wins eventually. There is no amount of "make the policy a little better per step" that escapes the compounding trap. Only changing the data distribution does. That single insight — that imitation's failure is about the data distribution, not the model — is the most important thing in this lesson.

Chapter 10: The Capstone — All 15 Lessons, One Last Time

You started this series, fifteen lessons ago, with a single picture: the see-think-act loop and a four-module autonomy stack. Every lesson since has been a deep dive into one band of that stack. Here, at the very end, we map all fifteen lessons back onto it — one final "you are here," now complete.

Below is the full autonomy stack with every lesson placed on its module. Tap a band to see which lessons built it and what you can now create. This is the diagram from Lesson 1, finally filled in.

The whole series on one page

The series cheat-sheet

Go deeper on Engineermaxxing

This capstone necessarily moved fast. These full lessons go all the way down on each frontier idea:

• Lesson 1: The See-Think-Act Loop — the stack we just finished filling in; re-read it now and watch how much more you see.
• Imitation Learning — behavior cloning, compounding errors, DAgger and HG-DAgger in full, with the diffusion-policy math.
• Vision-Language-Action Models — RT-2, OpenVLA, π₀, action tokenization, cross-embodiment, sim-to-real.
• World Models — Dreamer v1→v3, latent dynamics, JEPA, video-prediction world simulators, planning in imagination.
• NeRF & 3D Gaussian Splatting — volume rendering, radiance fields, the full splat rasterizer, generative 3D, 3D in robotics.
• Model-Based RL — the planning-with-a-learned-model spectrum behind world models.
• Diffusion Models — the denoising machinery the diffusion action head borrows.