The robot learning stack.

00Classical robotics — why we need learning

Before you can appreciate what learning buys you, you need to understand what classical control gives you for free — and where it breaks.

The robot as a dynamical system

A robot arm is a chain of rigid bodies connected by joints. Each joint has a motor. The motor applies a torque $\tau$, which accelerates the joint angle $q$. The arm's links have mass, they are spinning, and gravity is always pulling them down. Every controller must fight all three forces simultaneously.

In plain English: a robot arm is heavy, it is moving, and gravity pulls it down. The controller has to fight all three at every instant.

The equation of motion for a robot with $n$ joints:

• $q \in \mathbb{R}^n$ — the joint angle vector. For a 7-DOF Franka, $q = (q_1, \ldots, q_7)$. Each $q_i$ is the angle of one revolute joint.
• $\dot{q}, \ddot{q}$ — the joint velocity and acceleration. $\dot{q}$ is how fast each joint is currently moving; $\ddot{q}$ is the acceleration we are trying to produce.
• $M(q) \in \mathbb{R}^{n \times n}$ — the mass (inertia) matrix. Encodes how heavy and how far from the joint each link is. It is symmetric, positive-definite, and configuration-dependent: the same torque produces different accelerations depending on the arm's pose. When the arm is stretched out, the effective inertia is large; when folded close, it is small.
• $C(q, \dot{q})\dot{q} \in \mathbb{R}^n$ — the Coriolis and centrifugal forces. These are velocity-dependent terms that arise because the arm is a system of coupled rotating bodies. When joint 2 spins fast, it creates a centrifugal force that tries to fling joint 3 outward. These forces are zero when the arm is stationary and grow quadratically with speed.
• $g(q) \in \mathbb{R}^n$ — the gravity torque vector. The torque that gravity exerts on each joint, computed from the mass distribution and the current configuration. A horizontal arm at full extension feels maximum gravity torque; a vertical arm feels almost none.
• $\tau \in \mathbb{R}^n$ — the commanded joint torques. This is what the controller sends to the motors. The equation says: the torques must equal the sum of inertial, velocity-coupling, and gravitational loads.

This equation is exact for rigid bodies. If you know $M$, $C$, and $g$ perfectly, you can compute the exact torque needed for any desired motion using computed torque control: $\tau = M(q)\ddot{q}_{\text{des}} + C(q, \dot{q})\dot{q} + g(q)$, which cancels all nonlinearities and reduces the system to $\ddot{q} = \ddot{q}_{\text{des}}$. This is the gold standard of classical control — and it requires a perfect model.

PID control — the classical workhorse

When you do not have a perfect model (you never do), you fall back to PID control. Define the error as $e(t) = q_{\text{des}}(t) - q(t)$, the difference between where you want the joint and where it is. The PID controller computes a torque command as:

• $K_p \, e(t)$ — the proportional term. Push harder when the error is larger. If the joint is 10 degrees off, push 10 times harder than if it is 1 degree off. The gain $K_p$ sets the stiffness.
• $K_i \int_0^t e(s) \, ds$ — the integral term. Accumulates past error. If there is a persistent offset (say, gravity pulling the joint down by 2 degrees that $K_p$ alone cannot overcome), the integral term slowly builds up torque until the steady-state error vanishes. Too much $K_i$ and the system winds up and oscillates.
• $K_d \, \dot{e}(t)$ — the derivative term. Damps oscillation by opposing rapid changes in error. As the joint approaches the target and the error shrinks quickly, $\dot{e}$ is large and negative, so $K_d$ acts as a brake. Without it, a stiff ($K_p$-heavy) controller overshoots and rings.

Where classical control hits a wall

PID + computed torque + gravity compensation can produce beautiful motion in controlled environments. Industrial robots in factories prove this every day. But four failure modes are structural — no amount of gain tuning fixes them:

1. Contact. When the robot touches something, the dynamics change discontinuously. The mass matrix $M(q)$ now includes the object. The constraint forces are impulsive. No smooth model covers the transition from free motion to rigid contact to sliding to release. Classical controllers either go stiff (dangerous) or go compliant (imprecise).
2. Deformable objects. Cloth, rope, dough, cables — these have effectively infinite-DOF dynamics. You cannot write $M(q)$ for a towel. The state space is so large that analytical models are intractable. Every fold pattern is a different dynamical regime.
3. Partial observability. Classical controllers assume full state knowledge: joint angles, velocities, and the full world model. But a real robot cannot see behind objects, cannot measure the mass of an unknown payload until it lifts it, cannot sense whether a drawer is locked or just stiff. The controller needs information that no sensor provides.
4. Task diversity. Each task requires its own controller, its own gain schedule, its own state machine with hand-coded transitions. A pick-and-place controller cannot pour water. A pouring controller cannot fold cloth. Scaling to 100 tasks means 100 hand-engineered controllers — 100 person-months of tuning that transfer nothing between them.

Interactive: PID tuning

Drag the P, I, and D sliders to control a simulated 1-DOF joint tracking a step setpoint. Watch how the gains trade off rise time, overshoot, and steady-state error. The amber line is the setpoint; the blueprint line is the actual joint angle; the terracotta fill shows the error.

Notice the tradeoffs. Crank $K_p$ to 200 and the joint overshoots violently. Add $K_d = 30$ and it damps, but at the cost of sluggish response. Add $K_i$ and the steady-state error vanishes, but the system winds up on sudden changes. Now imagine doing this for 7 joints simultaneously, with a changing payload, on a task the controller was never designed for. That is why we need learning.

01The problem

A robot policy is a function from sensors to motor commands. The interesting part is everything that word "function" hides.

Formally, a robot operates inside a partially-observed Markov decision process. The world has a true state $s_t$; the robot sees an observation $o_t$ that is a noisy, lossy projection of it. At each tick the robot picks an action $a_t$, the world transitions to $s_{t+1}$ via dynamics $p(s_{t+1} \mid s_t, a_t)$, and (sometimes) emits a reward $r_t$. A policy $\pi(a_t \mid o_t, h_t)$ — possibly stateful via history $h_t$ — maps observations to actions. The goal is a policy that achieves the task, defined either by a reward function the policy maximizes (RL) or by a dataset of demonstrations the policy imitates (BC), or both.

Unpacking the POMDP

Each of those symbols hides an enormous amount of engineering reality. Let us make them concrete for a 7-DOF robot arm picking up a mug from a table.

State $s_t$ is the full physical truth of the world at time $t$. For our mug task it includes: the 7 joint angles and 7 joint velocities of the arm (14 numbers), the end-effector 6-DOF pose (6 numbers), the gripper finger width (1 number), the mug's 6-DOF pose (6 numbers), the mug's mass and friction coefficients, the positions of every other object on the table, contact forces at every contact point, and the deformation state of any soft objects. In principle the state also includes air currents, table vibration, and the thermal expansion of the links. In total, $s_t$ could be hundreds or thousands of dimensions. No sensor suite measures all of it.

Observation $o_t$ is what the robot actually sees. Typically: a 480×640×3 RGB image from a camera (or two), plus a proprioception vector $q_t \in \mathbb{R}^{14}$ (joint positions and velocities) and a gripper width scalar. The camera image is a lossy 2D projection of the 3D scene — it loses depth, cannot see behind objects, is affected by lighting and reflections, and compresses the mug's full 6-DOF pose into a pattern of pixels that a vision encoder must learn to decode. Objects that are occluded by the robot's own arm simply vanish from the observation. This is why we say the process is partially observed: the observation is a strict subset of the state.

Action $a_t$ is the motor command the policy sends to the robot at each control tick. Common choices include: target joint positions $q_{\text{target}} \in \mathbb{R}^7$ (the low-level PD controller handles the torques), end-effector pose deltas $\Delta T \in \mathbb{R}^6$, or raw joint torques $\tau \in \mathbb{R}^7$. The choice of action space is a structural commitment that cascades through the entire stack (section 02).

That's the textbook frame. The reasons robot learning is hard are not in the textbook:

• Compounding error. A small mistake at step $t$ moves the robot to a state $s_{t+1}$ slightly outside the training distribution, so the next action is worse, and so on. The error grows in the worst case as $O(T^2)$ in horizon $T$ for naive behavior cloning — a result that is the original sin of the field.
• Multimodality. Humans demonstrating the same task move differently. Averaging two valid trajectories produces an invalid one (think: two ways around an obstacle, averaged, hits the obstacle). A policy that regresses to the mean is broken.
• Non-stationary distributions. The robot's actions shape its future observations. This is the difference between supervised learning, where the data is fixed, and any learning paradigm where the policy is a participant in data generation.
• The reality gap. Simulators are fast, free, and wrong. Real robots are slow, expensive, and right. Bridging the two is the central engineering problem of the field.
• Tight latency budgets. A 200ms control loop is luxurious; many tasks need 50ms or less. Architectures compete on inference latency, not just success rate.

Scale of the challenge

To put these numbers in perspective: a table-top pick-and-place runs ~100 steps at 10Hz (a 10-second episode). The action space is 7-dimensional (6 joint targets + gripper). Two cameras produce 480×640 RGB images per step. A single demonstration yields ~100 observation-action pairs.

To train a reliable policy, you need 50–200 demonstrations of one task. At 5 minutes per demonstration (setup, teleop, reset), that is 4–16 hours of human time. A general-purpose robot needing 100 tasks requires 5,000–20,000 demonstrations. This is why data efficiency is the bottleneck, and why pre-trained VLAs that share representations across tasks are so important.

Stated bluntly: a 99%-accurate per-step naive BC policy on a 200-step task fails reliably, because the second-order growth term dominates. Action chunking, DAgger-style correction, and chunked / receding-horizon control are all attacks on the same root cause — they break the feedback loop between policy errors and state distribution shift.

Derivation: the $O(\epsilon T^2)$ compounding-error bound

This result, due to Ross and Bagnell (2010), is foundational. Let us derive it step by step.

Step 1: define the distribution shift. Let $d_t^*$ be the state distribution at time $t$ under the expert, and $d_t^{\hat\pi}$ under the learned policy. At $t=0$ they are identical (same start state). At each subsequent step, a mistake by $\hat\pi$ can move the state off the expert's distribution.

Step 2: bound the deviation per step. At step $t$, the total variation distance between the two state distributions satisfies:

This follows by induction. At $t=0$ the distance is zero. Each step, the policy either follows the expert (no additional shift) or deviates (shift grows by at most $\epsilon$). After $t$ steps, the cumulative shift is at most $t\epsilon$.

Step 3: sum the cost over the horizon. The expected cost (number of mistakes) at step $t$ has two components: (1) the probability $\epsilon$ of a direct mistake even when on-distribution, and (2) the additional cost from being off-distribution, which is at most $t\epsilon$ (since on unfamiliar states, the policy might always err). Thus:

The $\epsilon T$ term is what you would get if distribution shift did not exist — just per-step error accumulated linearly. The $\epsilon T^2 / 2$ term is the compounding penalty. For $T = 200$ and $\epsilon = 0.01$, the linear term gives 2 expected mistakes but the quadratic term gives 199 — completely dominating.

Code: compounding error in 1D

The abstraction becomes visceral with a concrete example. Here is a minimal simulation: the "expert" traces a sine-wave trajectory, and a naive BC policy has been trained to imitate it with small Gaussian noise. Watch how a tiny per-step error ($\sigma = 0.02$) compounds into total failure over 200 steps.

import numpy as np

T = 200                        # horizon
dt = 0.05                      # timestep
noise_std = 0.02               # per-step action noise (1% of range)

# Expert trajectory: sine wave
expert_pos = np.sin(np.arange(T) * dt)
expert_vel = np.cos(np.arange(T) * dt) * dt  # actions = velocity

# Naive BC: predict expert action + small noise
bc_pos = np.zeros(T)
for t in range(T - 1):
    # Policy sees its OWN state (not expert's) — this is the key
    action = expert_vel[t] + np.random.randn() * noise_std
    bc_pos[t + 1] = bc_pos[t] + action  # integrates from bc_pos, not expert_pos

# By step 200, |bc_pos - expert_pos| ≈ 0.02 * sqrt(200) ≈ 0.28
# But it's worse: the policy was trained on expert states, not its own.
# On unfamiliar states, errors are correlated, not random — they compound.
print(f"Final drift: {abs(bc_pos[-1] - expert_pos[-1]):.3f}")
print(f"Max drift:   {np.max(np.abs(bc_pos - expert_pos)):.3f}")

The critical detail is on line 10: the policy integrates from bc_pos[t], not expert_pos[t]. At training time, the observation always came from the expert's trajectory. At test time, the policy sees its own drifted state — a state it never trained on. The error is not just additive noise; it is systematic because the policy's predictions become increasingly unreliable on out-of-distribution states.

The control loop: where policy meets physics

A robot policy does not operate in isolation. It is one component in a real-time feedback loop with hard timing constraints:

The total loop latency determines what tasks are feasible. A pick-and-place on a static table can tolerate 200ms policy latency (5 Hz control). Catching a thrown ball requires < 20ms (50+ Hz). Diffusion policies, which need 10–100 denoising steps, push toward the slow end; this is why DDIM acceleration, consistency distillation, and flow matching (which needs fewer steps) are active research areas.

The POMDP in numbers

The observation-to-state ratio captures partial observability: from ~1M pixels the policy must infer ~1000 state dimensions. The camera is a massive lossy compressor. Modern vision encoders (SigLIP, DINOv2) trained on internet-scale data make this compression tractable.

Why this is not supervised learning

The non-stationarity point deserves emphasis. In standard supervised learning (image classification, language modeling), the data distribution is fixed: the model's predictions do not change the inputs it will see next. A cat classifier that misclassifies a dog does not somehow cause more dogs to appear.

In robot learning, a policy that drifts left at step 10 will see a different scene at step 11 than it would have seen if it had gone right. The policy's outputs change its future inputs. This creates a feedback loop between the model and its data distribution — a distribution shift that grows over time. This is the fundamental reason why per-step accuracy (the metric supervised learning optimizes) is a poor predictor of task success (the metric that matters). A 99% per-step accuracy policy can have a 0% task success rate, as the worked example above demonstrates.

Roadmap: a taxonomy of responses

Every major idea in the field attacks one or more of the five failure modes above:

The rest of this article walks through each response, starting from the foundations and building toward the full 2026 stack.

Interactive: error growth curves

02Spaces of action and observation

Choose the action space carelessly and no architecture will save you.

Action representations

The choice of action space is a structural commitment that propagates through the entire stack. Five common options:

The relative-EE space is quietly the most important shift of the last three years. A relative-pose policy that drifts can recover by issuing a corrective $\Delta T$; an absolute-pose policy that drifts is permanently confused. Relative actions also factor out the absolute pose of the demonstration, which means the same demo collected at any table works.

Forward and inverse kinematics

To understand why joint-space and task-space (end-effector) representations are fundamentally different, consider the simplest possible robot: a 2-link planar arm with link lengths $l_1, l_2$ and joint angles $\theta_1, \theta_2$.

Forward kinematics (FK) maps joint angles to end-effector position. It is always unique and always differentiable:

• $l_1, l_2$ — link lengths (fixed constants of the robot)
• $\theta_1$ — the shoulder angle, measured from the positive x-axis
• $\theta_2$ — the elbow angle, measured relative to link 1 (not the world frame)
• $(x, y)$ — the end-effector position in world coordinates

Inverse kinematics (IK) goes the other direction: given a desired end-effector position $(x, y)$, find joint angles $(\theta_1, \theta_2)$. This is harder for three reasons:

1. Multiple solutions. For the 2-link arm, there are generically two configurations that reach the same point — elbow-up and elbow-down. For a 7-DOF arm, there is a continuous manifold of solutions (a 1D null space).
2. No solution. If $(x, y)$ is beyond reach ($\sqrt{x^2 + y^2} > l_1 + l_2$), no joint angles work.
3. Singularities. When the arm is fully extended ($\theta_2 = 0$), the Jacobian loses rank and small task-space motions require infinite joint velocities.

This is why policies that output joint positions avoid the IK problem entirely — the FK is computed only for monitoring, not for control. Policies that output end-effector poses must solve IK at every step, introducing a numerical solver into the control loop.

import numpy as np

def fk_2link(theta1, theta2, l1=1.0, l2=1.0):
    """Forward kinematics for a 2-link planar arm."""
    x = l1 * np.cos(theta1) + l2 * np.cos(theta1 + theta2)
    y = l1 * np.sin(theta1) + l2 * np.sin(theta1 + theta2)
    return x, y

# Verify worked example
x, y = fk_2link(np.radians(30), np.radians(45))
print(f"EE position: ({x:.3f}, {y:.3f})")  # (1.125, 1.466)

# The Jacobian: how EE velocity relates to joint velocity
def jacobian_2link(theta1, theta2, l1=1.0, l2=1.0):
    """2x2 Jacobian: [dx/dθ1, dx/dθ2; dy/dθ1, dy/dθ2]"""
    s1, c1 = np.sin(theta1), np.cos(theta1)
    s12, c12 = np.sin(theta1 + theta2), np.cos(theta1 + theta2)
    return np.array([
        [-l1*s1 - l2*s12, -l2*s12],
        [ l1*c1 + l2*c12,  l2*c12]
    ])

Rotation parameterization

Never regress to Euler angles or raw quaternions. Both have discontinuities or double-cover problems that confuse gradients. The accepted choice is the 6D continuous representation from Zhou et al. — predict the first two columns of the rotation matrix and Gram–Schmidt them. It has no discontinuities and trains cleanly.

Why Euler angles fail: gimbal lock

Euler angles parameterize rotation as three sequential rotations: $R = R_z(\psi) \, R_y(\theta) \, R_x(\phi)$ (yaw-pitch-roll). When the pitch angle $\theta = \pm 90\degree$, the yaw and roll axes align — two of the three degrees of freedom collapse into one. Formally, the derivative $\partial R / \partial \psi$ and $\partial R / \partial \phi$ become linearly dependent at $\theta = \pm 90\degree$, so the Jacobian from Euler angles to the rotation matrix drops rank from 3 to 2. This means:

• A neural network predicting Euler angles near $\theta \approx 90\degree$ faces a discontinuity in the mapping: a tiny change in the target rotation requires a huge jump in the predicted angles.
• The gradient signal becomes degenerate — the network cannot tell which direction to move.
• This is not a rare edge case. A robot reaching forward and down regularly passes through pitch = $\pm 90\degree$.

Gram-Schmidt orthogonalization: step by step

The 6D representation predicts a raw $3 \times 2$ matrix (6 numbers) and converts it to a valid rotation matrix $R \in SO(3)$ via Gram-Schmidt. The steps are:

1. Normalize column 1: $\hat{c}_1 = c_1 / \|c_1\|$
2. Orthogonalize column 2: subtract its projection onto $\hat{c}_1$: $c_2' = c_2 - (\hat{c}_1 \cdot c_2) \hat{c}_1$, then normalize: $\hat{c}_2 = c_2' / \|c_2'\|$
3. Cross-product for column 3: $\hat{c}_3 = \hat{c}_1 \times \hat{c}_2$

The resulting $R = [\hat{c}_1 \mid \hat{c}_2 \mid \hat{c}_3]$ is guaranteed to be a valid rotation matrix ($R^\top R = I$, $\det R = +1$), and the mapping from 6D input to $SO(3)$ is continuous everywhere. This continuity is why gradient-based learning works cleanly.

import torch

def rotation_6d_to_matrix(d6: torch.Tensor) -> torch.Tensor:
    """Convert 6D rotation to 3x3 matrix. d6: (..., 6)."""
    c1, c2 = d6[..., :3], d6[..., 3:]
    # Step 1: normalize column 1
    c1_hat = c1 / c1.norm(dim=-1, keepdim=True).clamp(min=1e-8)
    # Step 2: orthogonalize column 2 via Gram-Schmidt
    dot = (c1_hat * c2).sum(-1, keepdim=True)
    c2_orth = c2 - dot * c1_hat
    c2_hat = c2_orth / c2_orth.norm(-1, keepdim=True).clamp(min=1e-8)
    # Step 3: cross product for column 3
    c3_hat = torch.cross(c1_hat, c2_hat, dim=-1)
    return torch.stack([c1_hat, c2_hat, c3_hat], dim=-1)

# Verify: random 6D input → valid rotation matrix
R = rotation_6d_to_matrix(torch.randn(6))
print("R^T R ≈ I:", R.T @ R)      # identity
print("det(R) ≈ 1:", torch.det(R)) # +1.0

This function is differentiable everywhere, so gradients flow cleanly from the action loss back through the rotation representation to the network weights. Every modern manipulation policy that outputs rotations (ACT, Diffusion Policy, $\pi_0$) uses this or an equivalent representation.

Observations

The modern observation tuple is some subset of:

• RGB images from one or more cameras — fixed scene cams, wrist cams, fisheye on a handheld stick. Wrist cams are extraordinarily helpful for fine manipulation; one wrist camera often beats two scene cameras.
• Proprioception — joint positions, velocities, gripper width, end-effector pose. Cheap and informative.
• Force / torque — six-axis F/T sensors at the wrist. Crucial for contact-rich tasks; usually ignored because data is hard to collect.
• Tactile — DIGIT, GelSight, pressure arrays. Promising; not yet load-bearing in flagship policies.
• Language — instruction strings, encoded by a frozen language encoder (T5, CLIP text, an LLM).
• Goal images — used by Octo and others to specify task without language.

The observation stack that works

For most manipulation tasks in 2026, the winning observation is:

1. One wrist camera (the most informative single sensor for contact-phase manipulation).
2. One scene camera (for spatial context and reaching).
3. Joint positions + gripper width (proprioception).
4. 2-step observation history (captures short-term dynamics without causal confusion).
5. A language instruction or goal image (for multi-task policies).

Force/torque is the next modality to add when it's available; it consistently helps on contact-rich tasks. Tactile is promising but not yet load-bearing in flagship systems.

03Three paradigms

Imitation, reinforcement, and the rapidly growing hybrid in between.

Every modern robot policy lives somewhere on a triangle whose vertices are imitation learning, reinforcement learning, and model-based / world-model methods. The dominant practical paradigm in 2026 is imitation, often pre-trained at scale, sometimes fine-tuned with RL. The pure-RL vertex remains alive in locomotion, dexterous in-hand manipulation, and any setting where a simulator is faithful enough to train in.

The triangle has interesting interior points. Offline RL trains a value-aware policy on demonstration data without further interaction — useful when you have demos but no simulator. Residual RL trains an RL correction on top of a frozen BC base. HIL-SERL blends online RL with human interventions for sample-efficient real-world learning. The big new entrant — VLA fine-tuning with RL — uses a pre-trained vision-language-action backbone and a small amount of task RL to specialize.

Derivation: BC as maximum likelihood

Behavior cloning looks like "just regress to actions," but it is maximum likelihood estimation in disguise. Understanding this connection reveals why MSE is the natural loss and what assumptions it bakes in.

Setup. We have a dataset of expert demonstrations $\mathcal{D} = \{(o_i, a_i)\}_{i=1}^{N}$. We model the expert's action distribution as a Gaussian conditioned on the observation:

In plain English: We're assuming the expert's action at any given observation is "the right action, plus some random wobble." The neural network predicts the right action; the wobble is a fixed-width bell curve around it. This is the simplest possible model of expert behavior — one answer per situation, with noise. It's also the model that makes MSE loss "the correct thing to do" (as we'll derive below). The hidden cost: it assumes there's only ONE right answer per observation. When there are two valid ways to do something, this model averages them.

• $f_\theta(o)$ — the neural network's predicted mean action for observation $o$
• $\sigma^2$ — a fixed noise variance (assumed isotropic and constant)
• $d$ — the action dimensionality (e.g. 7 for a 7-DOF arm)

Step 1: write the log-likelihood. The log-likelihood of the dataset under this model is:

Step 2: maximize. The second term is constant with respect to $\theta$. So maximizing the log-likelihood is equivalent to minimizing:

This is MSE loss, scaled by a constant. The derivation reveals the hidden assumption: the expert's actions are unimodal Gaussian around a single mean. When this assumption fails — when the distribution is bimodal, skewed, or heavy-tailed — MSE is the wrong loss, and you need the expressive action heads of section 05.

Derivation: the Bellman equation

Reinforcement learning is built on a single recursive identity. The value of being in state $s$ under policy $\pi$ is the reward you get now plus the discounted value of where you end up:

In plain English: "How good is it to be here?" equals "what I get right now" plus "how good is where I'll end up, slightly discounted because the future is less certain." It's a recursive definition: the value of THIS state depends on the value of the NEXT state. Think of it like asking "how much is this house worth?" — it's the rent you collect this month, plus the discounted value of all future rent. The Bellman equation is the backbone of every RL algorithm ever written.

• $V^\pi(s)$ — the value function: expected cumulative discounted reward starting from state $s$ and following policy $\pi$ forever
• $r(s, a)$ — the immediate reward for taking action $a$ in state $s$
• $\gamma \in [0, 1)$ — the discount factor: how much we value future reward relative to present reward. $\gamma = 0.99$ means reward 100 steps away is worth $0.99^{100} \approx 0.37$ of today's reward
• $p(s' \mid s, a)$ — the transition dynamics: the probability of landing in state $s'$ after taking action $a$ in state $s$
• The outer expectation is over the policy's action choice; the inner expectation is over the stochastic dynamics

The Bellman equation says: the value of a state is self-consistent. If you know $V^\pi$ for all next states, you can compute $V^\pi$ for the current state. This recursive structure is the foundation of every RL algorithm — policy gradient methods estimate $V^\pi$ to compute advantages, and value-based methods iterate the Bellman equation directly to convergence.

The Q-function extends the value function to state-action pairs: $Q^\pi(s, a) = r(s,a) + \gamma \, \mathbb{E}_{s'}[V^\pi(s')]$. The optimal policy simply picks $a^* = \arg\max_a Q^*(s, a)$. Deep RL algorithms like SAC (used in HIL-SERL) approximate $Q^*$ with a neural network and derive the policy from it.

The Bellman equation also reveals RL's core computational challenge: credit assignment. In our mug-picking example, the robot receives reward +1 only when the mug is lifted. But the action that caused success was the approach trajectory 30 steps earlier. The Bellman recursion propagates this +1 reward backward through time via the discount factor: the approaching state gets value $0.9^{30} \approx 0.04$ — a faint signal that the network must detect amidst noise. This is why sparse rewards make RL hard (the signal is too faint) and why reward shaping (adding intermediate rewards) helps but introduces its own problems (the policy may exploit the shaped reward without solving the actual task).

• $A^\pi(s, a)$ — the advantage: how much better action $a$ is compared to the average action the policy would take in state $s$. Positive advantage = better than average; negative = worse.
• Policy gradient methods (PPO, section 14) use the advantage to update the policy: increase the probability of actions with positive advantage, decrease those with negative advantage.

The hybrid spectrum

No modern system sits at a pure vertex. The practical spectrum includes:

• Pure BC — ACT, Diffusion Policy with only demo data. Fast to train, limited by data quality.
• BC + RL fine-tuning — the "pre-train then adapt" recipe. $\pi_0$ fine-tuned with RLHF. BC provides a strong prior; RL corrects its distribution shift.
• Offline RL — run RL algorithms on a fixed dataset without further interaction. IQL fits here. Useful when you have demos but no simulator.
• Residual RL — freeze a BC base policy, train an RL "residual" $a = \pi_{\text{BC}}(o) + \pi_{\text{res}}(o)$. BC handles coarse motion; RL learns fine adjustments.
• Human-in-the-loop RL — HIL-SERL. A human intervenes during RL rollouts, dramatically improving sample efficiency.
• World model + planning — Dreamer, UniSim. Learn dynamics from data, plan inside the model. Most sample-efficient; model errors compound during planning.

The practical takeaway: start with BC, upgrade as needed. For a new task, collect 50–100 demonstrations and train a Diffusion Policy. If the success rate plateaus below target, the bottleneck determines the next step. If the policy fails on novel object positions (distribution shift), try RL fine-tuning or more diverse demonstrations. If the policy is multimodal (inconsistent actions), check that the action head is expressive enough. If the policy succeeds in simulation but fails on the real robot, the problem is the sim-to-real gap.

Paradigm comparison

Interactive: the paradigm triangle

04Why naive BC fails

A regression model that loses to physics.

The simplest behavior cloning recipe: collect demonstrations $\{(o_i, a_i)\}$, train a network $\pi_\theta(o)$ to minimize $\sum_i \| \pi_\theta(o_i) - a_i \|^2$, deploy. It works on toy problems and fails on real ones for three reasons that compound.

Compounding error

Train-time observations come from the expert. Test-time observations come from the policy. Even if the policy is $\epsilon$-accurate per step, after $T$ steps it has wandered $O(\epsilon T)$ from the demonstration distribution; the loss bound on the expected number of mistakes is $O(\epsilon T^2)$. This is Ross & Bagnell, 2010. It means a 99%-accurate per-step BC policy fails reliably on a 200-step task.

Multimodality

The expert distribution $p(a \mid o)$ is often multimodal. Mean-squared regression converges to the conditional mean, which can be a bad action: imagine demonstrations split between going around an obstacle on the left and on the right; the mean goes through the obstacle.

In plain English: If you ask a neural network "minimize the squared error against ALL the demonstrations," the mathematically optimal answer is the AVERAGE of all demonstrated actions for each observation. The network isn't being dumb — it's doing exactly what you asked. The problem is that the average of two valid actions can be an invalid action, the same way the average of "turn left" and "turn right" is "go straight into the wall."

Two valid trajectories average to one invalid trajectory. The squared-error minimizer is the conditional mean, and the mean of two perfectly-good modes can sit exactly where neither one ever went. The remedy is not better optimization — it is an output distribution that can represent a bimodal answer.

Derivation: why MSE yields the conditional mean

Proof. Fix an observation $o$. The inner optimization is:

Expand the squared norm:

Take the gradient with respect to $c$ and set it to zero:

The result is immediate: the MSE-optimal prediction for observation $o$ is the conditional mean $\mathbb{E}[a \mid o]$. When $p(a \mid o)$ has two modes at $a_L$ and $a_R$ with equal probability, the conditional mean is $(a_L + a_R)/2$ — which is exactly the midpoint between the modes. If those modes go around an obstacle, the midpoint goes through it.

Interactive: bimodal action distribution

Causal confusion

A policy with too much information can learn shortcuts that don't generalize. The classic example: a self-driving model with access to a brake-light indicator learns to brake when the brake light is on — which works perfectly on demonstration data and catastrophically when deployed, because it's predicting its own past action rather than reading the road.

In manipulation, causal confusion often appears when the policy is given proprioceptive history (past joint positions). The policy can learn to "replay" the demonstration trajectory by attending to where the joints were one step ago, rather than looking at the camera to see where the object is. This works perfectly on the training distribution (because the joints were always in the same place at the same point in the demo) and fails completely on new object positions.

DAgger: fixing distribution shift

Dataset Aggregation (Ross et al., 2011) directly attacks the distribution shift problem. The insight: if the policy's mistakes lead it to unfamiliar states, get expert labels for those states and add them to the training set. After enough rounds, the policy has seen its own failure modes and can recover from them.

The algorithm has five steps, repeated in a loop:

1. Train initial policy

   Train $\hat\pi_1$ on the initial expert dataset $\mathcal{D}_0 = \{(o_i^*, a_i^*)\}$ using standard supervised learning (MSE or your preferred loss).

2. Roll out the learned policy

   Deploy $\hat\pi_n$ in the environment (or simulator). Collect the observations $\{o_1, o_2, \ldots, o_T\}$ that the policy visits — not the expert. These are the states where the policy actually operates, including the out-of-distribution ones where it fails.

3. Query the expert

   For each policy-visited observation $o_t$, ask the expert: "What would you do here?" Record the expert's action $a_t^* = \pi^*(o_t)$. This is the expensive step — it requires a human or an optimal controller to label novel states.

4. Aggregate the dataset

   $\mathcal{D}_n = \mathcal{D}_{n-1} \cup \{(o_t, a_t^*)\}$. The training set now contains both the original expert demonstrations and expert corrections for the policy's mistakes.

5. Retrain

   Train $\hat\pi_{n+1}$ on the aggregated dataset $\mathcal{D}_n$. Go to step 2.

Why it works. After $N$ rounds of DAgger, the policy has been trained on states drawn from its own distribution (not just the expert's). The distribution shift vanishes, and the error bound drops from $O(\epsilon T^2)$ to $O(\epsilon T)$ — linear in the horizon, not quadratic. The quadratic term disappears because the policy no longer encounters states it hasn't been trained on; every state it visits is (approximately) in the training set.

def dagger(env, expert, policy, initial_data, n_rounds=10, n_rollouts=5):
    """DAgger: Dataset Aggregation for imitation learning."""
    dataset = initial_data.copy()       # D_0 = expert demos

    for round_i in range(n_rounds):
        # Step 1 & 5: train on aggregated dataset
        policy.train(dataset)

        # Step 2: roll out the learned policy
        new_data = []
        for _ in range(n_rollouts):
            obs = env.reset()
            for t in range(env.max_steps):
                action = policy.predict(obs)  # policy's action
                # Step 3: query expert for the CORRECT action at this state
                expert_action = expert.label(obs)
                new_data.append((obs, expert_action))
                # Execute the POLICY's action (not the expert's)
                obs, _, done, _ = env.step(action)
                if done: break

        # Step 4: aggregate
        dataset.extend(new_data)
        print(f"Round {round_i}: dataset size = {len(dataset)}")

    return policy

The critical detail: on line 14, we execute the policy's action (so the rollout visits the policy's distribution), but we record the expert's action (so the policy learns to correct its own mistakes). This is what closes the distribution gap.

DAgger in practice: limitations and variants

Pure DAgger is rarely used in modern manipulation for a practical reason: step 3 requires an expert to label arbitrary states, including states the robot reaches through failures. For a human teleoperator, this means watching the robot fail and retroactively answering "what would I have done at each frame?" — which is cognitively demanding and error-prone. Several practical variants address this:

• HG-DAgger (Human-Gated DAgger): The human watches the robot execute the policy and intervenes only when necessary, taking over control. The intervention data is added to the dataset. This is psychologically easier than labeling every state.
• IWR (Interventionist Learning with Recovery): When the human intervenes, the system records the transition from the policy's state back to the expert's trajectory. This "recovery" data teaches the policy to correct its own mistakes.
• RL fine-tuning as implicit DAgger: Running RL on top of a BC policy has the same effect: the policy encounters its own failure states and learns to handle them, but through reward signal rather than expert labels. This is why BC + RL fine-tuning has largely replaced DAgger for high-stakes manipulation.

The fixes

Each failure has a class of solutions:

• For compounding error: action chunking, receding-horizon control, history conditioning, DAgger-style on-policy correction, RL fine-tuning.
• For multimodality: expressive output distributions — Gaussian mixtures, categorical heads, energy-based models, diffusion, flow matching, VQ.
• For causal confusion: careful observation design, dropout on proprioception during training, and refusing to feed the policy the previous action.