Beyond the Controller: SMPC Data for Offline-to-Online Loco-Manipulation

Chapter 1: The Reward Is Three Numbers

Here is the entire reward function of the paper. Not a summary of it — the whole thing, for every task, on both robots.

with γ = 0.99. There is no term for end-effector distance, no term for contact force, no term for base tilt, no weight to balance against another weight. "Crashed" is defined by two thresholds on the state: the torso height or tilt exceeds pre-defined limits. That is the whole contract between the engineer and the agent.

Three numbers deserve three separate derivations, because each one is doing precise work.

Why −1 every step

A constant negative reward per step is a time penalty: the only way to stop accumulating it is to reach the goal, and the only way to accumulate less of it is to reach the goal sooner. The agent is not told how to be fast. It is told that lateness is the only thing that costs.

Work out what that implies for the return. The discounted return from the start of an episode that reaches the goal after exactly T steps is a finite geometric sum:

Put γ = 0.99 in and compute three cases by hand. Recall 0.99¹⁰ = 0.904382, and each further factor of ten in the exponent just multiplies by that again:

the time penalty, γ = 0.99# 0.99^20 = 0.817907    0.99^30 = 0.739700    0.99^40 = 0.668971
# 0.99^50 = 0.605006    0.99^80 = 0.447523

T = 30 steps :  G = -(1 - 0.739700) / 0.01 = -26.030
T = 50 steps :  G = -(1 - 0.605006) / 0.01 = -39.499
T = 80 steps :  G = -(1 - 0.447523) / 0.01 = -55.248

# never reaching the goal at all, and never crashing:
T = inf      :  G = -(1 - 0) / 0.01 = -100.000

Read the last line carefully, because it is the anchor for everything that follows. An agent that wanders forever without ever succeeding and without ever falling collects exactly −100. That is the value of infinite competent stalling, and it is the number the crash penalty is designed against.

Why the crash penalty is exactly −2/(1−γ)

The paper gives the reason in one clause: the penalty is "chosen such that agents prefer the constant negative reward over crashing." Unpack that into an inequality.

Crashing terminates the episode. Termination means no future rewards accumulate, so the return of an episode that crashes immediately is simply the crash penalty itself, call it −c. The alternative, stalling forever, returns −1/(1−γ) = −100. For the agent to prefer stalling we need:

Any c above 100 satisfies it. The paper picks c = 2/(1−γ) = 200, exactly twice the stalling value. That choice buys two things beyond the inequality. First, margin: a crash is not marginally worse than stalling, it is decisively worse, so the preference survives function-approximation error in the critic. Second — and this is the part that pays off in Chapter 6 — it makes the return of the whole MDP live inside a closed interval whose endpoints you can write down: the best possible return is 0 and the worst possible is −200.

Now check a crash that happens partway through, because that is the case the agent actually faces. Crash at step 20, having collected −1 for steps 0 through 19:

crash at step 20 vs stalling foreverprefix  = -(1 - 0.99**20) / 0.01 = -(1 - 0.817907) / 0.01 = -18.209
penalty = 0.99**20 * (-200)                        = -163.581
                                                    -------
G(crash at 20)                                      = -181.790

G(stall forever)                                    = -100.000
G(reach goal at step 20)                            =  -18.209

The ordering is exactly what you want: succeeding > stalling > crashing, with real distance between each. And notice that discounting softens a late crash relative to an early one — crashing at step 200 costs 0.99²⁰⁰·200 = 26.80 in present value instead of 200. This is not a bug. It correctly encodes that a robot which survives for a long time before failing has done something more useful than one that tips over immediately.

The catastrophe hiding in this reward

Sparse rewards are clean and they are also, on their own, unlearnable in this setting. It is worth deriving why, because the derivation explains the exact shape of the fix.

Consider an agent early in training that has never reached a goal and never crashed. Every transition it has ever seen carries r = −1 and is non-terminal. Ask what the Bellman equation says the Q-function should be. For any state s and any action a:

Every reward on the right is the same constant. The unique fixed point of that recursion, if no transition in the data ever ends the episode or pays anything different, is the constant function:

A deterministic policy like TD3's is improved by ascending the critic with respect to the action: ∇_aQ(s, a). If Q is constant in a, that gradient is exactly zero. The actor receives no signal at all. Not a weak signal, not a noisy signal — a mathematically zero one. Training does not converge slowly; it does not begin.

Play with the landscape

Press flat critic to see the failure mode from the derivation above: with no goal transitions anywhere in the data, the whole surface collapses onto a single horizontal line at −1/(1−γ), and the actor gradient dies. Then move γ toward 0.998 and notice how the curve near small T flattens out — a very high discount makes finishing at step 30 versus step 50 nearly indistinguishable in return, which is one reason 0.99 rather than 0.999 is the right neighbourhood for a task whose horizon is tens of steps, not thousands.

One more consequence: the reward scale

Look ahead at the hyperparameter table for a second, because one entry belongs to this chapter. The paper uses a reward scale of 0.01, and explains why in a single sentence: "To keep the value predictions close to 0, we scale our rewards." Do the arithmetic:

reward scaling and the value rangeunscaled:   step reward -1     crash -200     Q in [-200, 0]
scale 0.01: step reward -0.01  crash -2.0    Q in [ -2,  0]

A neural network initialised in the usual way outputs numbers of order one. Asking it to regress targets around −100 means the first thousand gradient steps are spent inflating the output layer instead of learning the shape of the value function. Multiply the reward by 0.01 and the targets land in [−2, 0], where the network already lives. Nothing about the optimal policy changes — scaling a reward by a positive constant leaves the argmax untouched — but the optimisation gets much easier. Remember the interval [−2, 0]; Chapter 6 will bolt it into the architecture.

Why 0.99 and not 0.999 — the effective horizon

The discount factor is usually presented as a knob for "how far ahead the agent cares." Make that precise. The quantity 1/(1−γ) is the effective horizon: the number of steps beyond which future reward has been discounted into irrelevance, and simultaneously the magnitude of the stalling return we just computed. Because this paper's high level runs at 5 Hz, every horizon converts directly into seconds:

Verify the last row rather than trusting it. At γ = 0.999, 0.999³⁰ = 0.970431 and 0.999⁵⁰ = 0.951206, so G(30) = −29.569 and G(50) = −48.794. As a fraction of the stalling value of −1000, those sit at 2.96% and 4.88% — two numbers a function approximator is being asked to separate near the bottom of its dynamic range. At γ = 0.99 the same pair is −26.03 and −39.50 against a stall of −100: 26% and 39%, comfortably distinguishable.

What "at goal" actually means, and why termination is load-bearing

One more piece of the reward specification deserves attention, because it is invisible in the equation and decisive in the implementation. Reaching the goal does not merely pay 0 — it terminates the episode. So does crashing.

Termination is what makes the Bellman target for those transitions a bare number rather than a bootstrap:

the Bellman target, with terminationy = r + gamma * (1 - done) * Q_target(s_next, pi(s_next))

# goal transition:  r = 0,    done = 1  ->  y = 0.0     exactly
# crash transition: r = -200, done = 1  ->  y = -200.0  exactly
# ordinary step:    r = -1,   done = 0  ->  y = -1 + 0.99 * Q_target(...)

Only the first two lines contain information that did not come from the critic itself. Everything else is the network regressing onto a slightly shifted copy of its own opinion. That is why Chapter 5 will care so much about how many goal-terminal transitions are in each batch: they are the only entries in the entire training signal that are grounded in the environment rather than in the network's current beliefs.

Chapter 2: The Policy Does Not Move Joints

A recurring beginner mistake when reading legged-robot papers is to assume the policy outputs torques, or at least joint positions, for all the joints. This one does not, and the reason it does not is load-bearing for everything else.

The architecture is hierarchical: two policies stacked, with a fixed interface between them. The paper describes the arrangement as decoupling "task-level navigation and manipulation from base maneuvering", and adds a detail that is easy to skim past but is the reason the whole pipeline works: "This structure is shared during data collection and deployment, allowing seamlessly swapping between the SMPC expert or the RL agent for the high-level policy."

The high-level action, dimension by dimension

The high-level policy emits

and each piece is a delta, not an absolute target. Spelled out:

Two of those are morphology-dependent, and the paper says exactly how. "For Spot, we keep the torso height and pitch fixed and do not command these values." On the G1 humanoid they are commanded, "to enable strategies relying on lowering the upper body." So the same architecture yields a smaller action space on the quadruped (3 + n_a) and a larger one on the humanoid (3 + n_a + 2), which is the only structural change between the two robots.

Why deltas and not absolute targets

This is the paper's answer to a safety problem that sparse rewards create. Read the reasoning in their words first: "Because policies trained with sparse objectives seek optimal solutions that push the physical limits of the platform, we must ensure these behaviors remain safe and deployable."

Think about what an absolute-target policy could do. If the action is the desired base velocity, then the policy can emit +2.0 m/s on one control step and −2.0 m/s on the next. Nothing in the action space forbids it. The only thing standing between you and a 4 m/s velocity reversal in 200 ms is the hope that the reward discouraged it — and we just deleted every shaping term that could have.

With deltas, the bound is structural. Suppose the action range for Δv is scaled to ±0.1 m/s per high-level step, and the high-level runs at 5 Hz:

acceleration limits from the action spacemax |dv| per step      = 0.1 m/s
high-level period      = 1 / 5 Hz = 0.2 s
=> max acceleration    = 0.1 / 0.2 = 0.5 m/s^2          # enforced by the action SPACE

# and the running command is clamped to a speed envelope:
v_cmd <- clip(v_cmd + dv, -v_max, +v_max)        # no reward term needed

The policy is now incapable of commanding an unsafe transient. Constraints that live in the action space cannot be traded away by an optimiser, which is precisely what you want when the optimiser has been told that only speed matters.

Count the dimensions, then count what the hierarchy saved

Put concrete numbers on the action space so the saving is visible. Spot's arm is a six-joint design, so n_a = 6 and, with torso height and pitch held fixed, the high-level action is nine-dimensional. (The paper does not state n_a; six is the standard Spot arm and we use it only to make the arithmetic concrete.) Compare that against the flat alternative — a single policy commanding every joint directly — on two axes at once.

The dimension saving is worth more than the factor of two suggests, because exploration volume is exponential in dimension. Ask what fraction of the action space a small region of side 0.1 (in normalised units, where the full range is 2.0) occupies:

halving the dimension is not a factor of 2fraction of an 18-dim action cube inside a 0.1-wide window:
    (0.1 / 2.0)**18 = 0.05**18 ~= 3.8e-24

fraction of a  9-dim action cube inside the same window:
    (0.1 / 2.0)**9  = 0.05**9  ~= 1.95e-12

ratio = 1.95e-12 / 3.8e-24 ~= 5e11       # five hundred BILLION times easier to hit

And the credit-assignment saving stacks on top. In the flat formulation, a sparse reward delivered at the end of a 10-second episode must be assigned across 500 decisions; in the hierarchical one, across 50. The discount factor already told us the effective horizon is 100 steps — at 50 Hz that would be 2 seconds and the goal would be beyond the horizon entirely.

The low-level controller, and the word "frozen"

Underneath sits a whole-body maneuvering controller trained with the ReLIC framework of Zhu et al. (CoRL 2025), adapted here to run inside mjlab. Its job:

It "strictly tracks the commanded arm motion and base velocity while dynamically adjusting the leg joints to maintain balance." The arm and base commands are honoured; the legs are spent on staying upright. And then the crucial sentence: the low-level policy "is subsequently frozen for all downstream task learning."

Freezing has three consequences worth separating.

The first two rows are why an off-policy method can eat this data at all. If the low-level controller were being fine-tuned online, every transition in the replay buffer would become stale the moment it was written, and the "offline" half of offline-to-online would be replaying a world that no longer exists.

Trace one command all the way down

Follow a single high-level decision through the stack, with rates attached. The high-level policy runs at 5 Hz; the low-level controller and the physics run at 50 Hz.

What the two networks actually see

One more implementation detail that matters for sim-to-real, stated in the appendix: the setup is an asymmetric actor-critic. "The actor receives noisy observations, while the critic receives exact observations from the simulation."

The asymmetry is not a trick, it is a consequence of where each network will live. The actor is going to be shipped to a robot whose state estimate comes from motion capture plus onboard sensing, so it must be trained on inputs that look like that — noisy. The critic is only ever used during training, inside the simulator, where the exact state is free. Handing the critic the exact state gives it a cleaner regression target, which reduces value-estimation variance, which produces a better gradient for the actor. You get the benefit of privileged information without ever depending on it at deployment.

Alongside it: domain randomisation over "the object mass, its size, and friction." Note what is randomised — not the robot, the object. The manipulated thing is the part of the world the simulator knows least well, and it is the part whose properties change every time someone rolls a different tire onto the floor.

What must be in the observation, derived rather than listed

The paper does not enumerate the observation vector, but the design forces most of its contents and you can reconstruct them from first principles. The rule is the Markov property: the observation must contain everything the optimal action depends on. Walk the action components and ask what each one needs.

Notice that the first two rows are entirely a consequence of choosing deltas. Absolute-target actions would not need the current command in the observation — but they would need a shaping term to stay safe. The delta parameterisation moved the cost from the reward function into the observation vector, which is a much better place for it: an observation entry is a design decision you make once, while a reward weight is a design decision you re-litigate every training run.

Chapter 3: The Teacher Is a Random Number Generator

"Sampling-based model predictive control" sounds like it needs a chapter of convex optimisation. It does not. The version this paper uses — the authors say a "streamlined SMPC formulation, similar to" predictive sampling (Howell et al., 2022), "with random sampling and a warm-start strategy typical of MPC algorithms, is sufficient to solve our tasks without additional complexities such as Model Predictive Path Integral Control" — is four lines of pseudocode. Here is the whole algorithm in words:

No gradients. No model linearisation. No contact schedule. The "model" is the simulator itself, queried as a black box. That is exactly why the authors can point it at a quadruped with an arm and at a bipedal humanoid without rewriting anything: "SMPC allows for rapid, intuitive tuning of complex behaviors and can be straightforwardly applied to any robotic morphology."

The numbers, and what they mean in seconds

Table 2 of the paper fixes the planner for every task and both robots. Convert each entry into a physical quantity, because the raw integers hide the design:

Why splines, not 50 independent samples

This is the single most important design choice in the sampler, and the paper gives the reason in half a sentence: "To generate smooth behaviors and prevent high-frequency control noise, the SMPC samples actions as splines rather than independent, step-wise commands." Let us make that concrete, because the difference is enormous.

Suppose the action has d dimensions. Sampling independently at every step means drawing

Sampling a spline means drawing 10 knots and interpolating:

Two things improve at once. First, coverage. Random search in a 50d-dimensional box with 256 samples is hopeless — the samples are all essentially the same distance from everything. In a 10d-dimensional box, 256 samples actually explore. Second, smoothness: an independently-sampled sequence is white noise, which for a robot arm means chattering at the controller's Nyquist rate, which the low-level tracker will either filter out (wasting the sample) or faithfully reproduce (shaking the machine). A spline through 10 knots has bounded curvature by construction.

python — sampling one batchdef sample_splines(warm_start, sigma_min, sigma_max, n_ctrl, horizon, K, d):
    # warm_start: (horizon, d) — last iteration's elite, already shifted forward
    knots_ws = warm_start[::horizon // n_ctrl]            # (n_ctrl, d) = (10, d)

    # one noise scale per candidate, spread over [sigma_min, sigma_max]:
    # some candidates stay near the warm start, others explore far.
    sig = np.linspace(sigma_min, sigma_max, K)[:, None, None]   # (K, 1, 1)

    knots = knots_ws[None] + sig * np.random.randn(K, n_ctrl, d)   # (K, 10, d)
    seq   = interpolate(knots, horizon)                          # (K, 50, d)
    return np.clip(seq, -1.0, 1.0)                              # actions are normalised

Notice the noise range rather than a single σ. With σ spanning 0.5 to 1.0, the batch contains both conservative candidates that refine the current plan and aggressive ones that could find a different plan. That is an exploration schedule folded inside a single batch, and it costs nothing.

One replan, by hand, with four candidates

Shrink every dimension until the arithmetic fits on a page, then do the whole thing manually. One action dimension, horizon T = 4, two spline knots, four candidates, γ = 0.99, and a target action profile x = [0.2, 0.4, 0.6, 0.8] that the dense cost is implicitly asking for. The cost of a candidate is the discounted squared tracking error:

Four candidates are drawn around a warm start of [0, 0]. Expand each into its four-step action sequence:

step 1 — knots to action sequencesdiscount weights: gamma^t = [1.000000, 0.990000, 0.980100, 0.970299]
target          : x     = [0.200, 0.400, 0.600, 0.800]

c0  knots [ 0.10, 0.30]  ->  a = [ 0.1000,  0.1667,  0.2333,  0.3000]
c1  knots [ 0.30, 0.90]  ->  a = [ 0.3000,  0.5000,  0.7000,  0.9000]
c2  knots [-0.20, 0.50]  ->  a = [-0.2000,  0.0333,  0.2667,  0.5000]
c3  knots [ 0.25, 0.75]  ->  a = [ 0.2500,  0.4167,  0.5833,  0.7500]

Now score them. Take candidate 3 in full, then the rest by the same recipe:

step 2 — cost of candidate 3errors  = [ 0.0500,  0.0167, -0.0167, -0.0500]
squares = [ 0.002500, 0.000278, 0.000278, 0.002500]
weighted= [ 0.002500, 0.000275, 0.000272, 0.002426]
                                                --------
C(c3)   =                                        0.005473

and the full scoreboardC(c0) = 0.438245
C(c1) = 0.039404
C(c2) = 0.489328
C(c3) = 0.005473   <-- ELITE (argmin)

Candidate 3 wins. Now do what the algorithm does next, which is the part people forget:

step 3 — commit, shift, discardcommit  : store a[0] = 0.2500 and a[1] = 0.4167    # N_e = 2 of 4 steps
discard : a[2], a[3] are thrown away — they were never real
warm    : next warm start = elite knots shifted forward = [0.75, 0.75]
store   : the two committed transitions get the SPARSE reward, not C

Two lessons are visible in numbers this small. First, the elite is not the optimum. A perfect tracker would have cost 0; the best of four got 0.005473. The planner is not solving the optimisation, it is sampling near it — which is exactly why the RL policy is later free to beat it.

Second, check the Chapter-3 order statistic against this batch. The four costs have mean 0.2431 and standard deviation 0.2217, so the elite sits

against a predicted 0.84σ for K = 4. The prediction is in the right neighbourhood on a batch of four; on a batch of 256 it is tight. That estimator is how you decide, before running anything, whether your sampling budget is adequate.

The warm start is where the information lives

If you deleted the warm start and sampled around zero every iteration, the planner would be a memoryless random search and would solve nothing. The warm start is what turns 256 samples per replan into a sequence of increasingly good plans.

The mechanics, from Algorithm 1: after picking the elite A*, the planner sets the next warm start to ShiftTimeForward(A*, N_e). In plain terms: the elite covered steps 0…49; we executed 0…9; so steps 10…49 of the elite are still a valid plan for the new state, shifted to positions 0…39, and positions 40…49 are filled in. The next batch is sampled around that.

There is a companion line right after it — ResetStaleWarmStarts — and it is doing something specific. Environments whose episode just ended, or that were reset to a new tile state, hold a warm start describing a task that no longer exists. Sampling around a stale plan is worse than sampling around zero, so those are cleared.

The dense cost you actually write

The paper says the planner is "guided by easily formulated dense cost functions rather than sparse rewards" and does not print them, because they are task-specific and, by the paper's own argument, not the interesting part. But you will have to write one, so it is worth being concrete about what such a cost looks like and, more importantly, what tuning it feels like when the feedback loop is seconds instead of hours.

python — a dense cost (illustrative)def smpc_cost(state):
    c  = 3.0 * dist(state.tire_xy, state.goal_xy)          # make progress
    c += 1.0 * dist(state.ee_xyz, state.tire_contact_xyz)  # keep the arm on the tire
    c += 0.5 * abs(state.torso_tilt)                       # stay upright
    c += 0.2 * norm(state.action_rate)                     # do not chatter
    return -c                                             # SMPC maximises reward = -cost

Four terms and four weights — the same shape as the reward function Chapter 0 said was so expensive to tune. The difference is entirely in the loop, and it is worth naming the individual moves that become available when a design edit costs seconds:

That last row is the one to hold on to. It is the reason Chapter 9's multimodality problem was diagnosable at all.

Why argmax and not a weighted average

The paper explicitly declines to use Model Predictive Path Integral control, which is the best-known sampling-based MPC variant. MPPI differs in one place: instead of taking the single best candidate, it forms an exponentially weighted average of all K sampled trajectories, weighting each by exp(−cost/λ). That usually helps — it uses information from every sample instead of throwing away 255 of them, and it smooths the plan across iterations.

The paper's stated reason for skipping it is sufficiency: random sampling with a warm start "is sufficient to solve our tasks without additional complexities." But there is a second reason worth noticing yourself, and it foreshadows Chapter 9.

Best-of-K: what the sample count actually buys

The paper's Q3 ablation reduces the number of sampling environments and measures the downstream damage. To predict what that does, model the trajectory scores of the K candidates as draws from some distribution with mean μ and spread σ. The elite is the maximum of K draws, and for roughly normal scores the expected maximum is well approximated by

where Φ⁻¹ is the inverse standard normal CDF. Evaluate it:

elite quality vs K, by order statisticsK = 256 :  Phi_inv(1 - 1/257) = Phi_inv(0.99611) ~= 2.66 sigma
K =  64 :  Phi_inv(1 - 1/ 65) = Phi_inv(0.98462) ~= 2.16 sigma
K =  16 :  Phi_inv(1 - 1/ 17) = Phi_inv(0.94118) ~= 1.57 sigma
K =   4 :  Phi_inv(1 - 1/  5) = Phi_inv(0.80000) ~= 0.84 sigma

# 256 -> 64 costs only 0.50 sigma. 256 -> 16 costs 1.09 sigma PER REPLAN.
# A 10-second task at 5 Hz is 50 replans, so losses compound along the
# trajectory: a slightly worse plan leaves you in a slightly worse state,
# from which the next best-of-K is drawn from a worse distribution.

Two predictions fall out of this, and Chapter 9 will check both against the paper's data. Prediction one: the loss from shrinking K is sublinear, so tasks with slack should barely notice. Prediction two: the loss compounds along the trajectory, so tasks that need many consecutive near-perfect decisions should fall off a cliff. The paper finds exactly that: "most tasks are relatively robust to a low number of sampling environments… However, for tire rolling, which requires the most coordination, we see a drop in successes."

Watch the planner think

Turn the warm start off and keep pressing replan. The committed trace stops improving, because every iteration starts its search from scratch instead of from the accumulated knowledge of the previous ones. That is the difference between a planner and a slot machine.

Chapter 4: One Million Samples Per Hour

A planner that solves the task is not yet a dataset. To bootstrap a sparse-reward agent you need millions of transitions, and the paper reports a throughput that makes the whole approach viable: one million samples per hour on a single GPU, where a sample is one (observation, action, reward, next observation) tuple. The hardest task needed four million of them, "collected within 4 GPU hours."

That number is only achievable because of one structural trick, and the trick is worth understanding in full because it explains an odd line in the paper's Figure 3 caption: "Note the different x-axes scales for a task's trajectory steps and total simulation steps."

The tiling idea

A single SMPC replan needs K = 256 rollouts from the same state. A GPU simulator like MuJoCo Warp gives you thousands of independent environments running in lockstep. So: partition the environments into tiles of K environments each. Every tile is one planning problem; every environment inside a tile is one candidate.

The paper describes the loop precisely: "Each tile broadcasts its initial task to all its environments upon reset, runs sampled actions, selects the best trajectories, and adds them to our expert buffer." With M tiles you are solving M independent SMPC problems in parallel, using M × K simulated robots.

The reason this is even expressible is a property of the simulator that the paper calls out explicitly: "Because MuJoCo Warp relies on a functional programming paradigm with a single explicit state representation, it lends itself easily to sample-based implementations that need full control over the state." The whole physical state is a tensor you can read, write, and broadcast. SetStates is an assignment, not an API dance.

Algorithm 1, with shapesD      = []                                # expert dataset
A_bar  = zeros(M, T, d)                     # warm-start actions, one per TILE
s_ckpt = GetStates(envs, M)                 # M tile states — the "real" states

while len(D) < target:
    s_full = Repeat(s_ckpt, K)              # (M, ...) -> (M*K, ...)
    SetStates(envs, s_full)               # broadcast: every candidate starts here

    A = SampleSplines(A_bar, s_min, s_max, N_c)   # (M, K, T, d)
    O, R_smpc, R_sparse, s_Ne = SimulateHorizon(envs, A)  # roll ALL of them T steps

    R_tau = sum(gamma**i * R_smpc[:, :, i] for i in range(1, T+1))  # (M, K)
    j     = ArgmaxPerTile(R_tau)                            # (M,) elite index

    A_star = A[arange(M), j]                # (M, T, d)  the elite plans
    s_ckpt = s_Ne[arange(M), j]               # (M, ...)   elite states after N_e steps
    A_bar  = ShiftTimeForward(A_star, N_e)
    ResetStaleWarmStarts(A_bar)

    tau = ExtractCommittedSteps(O, A, R_sparse, j, N_e)  # (M, N_e) transitions
    D  += tau                                            # SPARSE reward is what gets stored
    if any_episodes_finished():
        DiscardFailedEpisodes(D)
return D

Read that pseudocode twice, for two details

Detail one: two reward streams. SimulateHorizon returns both R_smpc and R_sparse. The planner ranks candidates using R_smpc, the hand-tuned dense cost. But ExtractCommittedSteps stores R_sparse into the dataset. The dense cost is a search heuristic that never leaves the data-collection loop. Every transition in the four-million-sample buffer carries the same three-valued reward from Chapter 1. This is the mechanism by which "we tuned a dense cost" and "the agent never sees a shaped reward" are both true.

Detail two: DiscardFailedEpisodes. The dataset is filtered, not raw. When an episode finishes badly, its transitions are removed. So the offline buffer is a set of successful demonstrations — which matters enormously for Chapter 5, because the entire job of the offline data is to supply goal-terminal transitions that break the flat-Q symmetry.

The arithmetic that makes the throughput number sensible

Here is the calculation that explains the "different x-axes" caption. Per tile, per iteration:

simulated steps vs stored samplessimulated per tile per iteration = K * T   = 256 * 50 = 12800 env-steps
stored    per tile per iteration = N_e      =            10 transitions

ratio = 12800 / 10 = 1280 simulated steps per stored sample

# For the hardest task's 4,000,000 stored samples:
total simulated env-steps = 4e6 * 1280 = 5.12e9      # 5.12 BILLION
wall clock                = 4 h = 14400 s
throughput                = 5.12e9 / 14400 ~= 356000 env-steps / s

Notice that M cancels: the ratio is a property of K, T and N_e, not of how many tiles you run. Adding tiles buys you wall-clock, not efficiency. And that 1280:1 ratio is the real price of a sampling-based planner: to write down one decision you must imagine 1,280 steps of physics that never happened.

It also tells you where the compute goes, which the paper confirms: "data collection computation time is almost exclusively spent running the forward simulation." Hence the caching optimisation — "we save computation by caching the simulation state after N steps and then resetting all simulations to the state of the most successful sample after the full rollout" — which avoids re-simulating the committed prefix on the next iteration. (Our 5.12e9 figure ignores that saving, so treat it as an upper bound.)

What the buffer costs in bytes

The replay buffer holds 2²⁴ = 16,777,216 transitions, which is a suspiciously round number — a power of two is what you choose when the constraint is memory and indexing, not statistics. Size it. Assume an observation of about 60 floats (the command accumulators, object pose and velocity, torso height and tilt, goal-relative terms, and proprioceptive summary from Chapter 2's derivation) and the nine-dimensional Spot action:

replay buffer footprint (our estimate)per transition: obs 60 + act 9 + next_obs 60 + reward 1 + done 1 = 131 floats
                131 * 4 bytes (fp32) = 524 bytes

full buffer  : 16777216 * 524 B  = 8.79 GB
expert set   : 4000000  * 524 B  = 2.10 GB      # 23.8 % of capacity
one batch    : 8192     * 524 B  = 4.29 MB      # trivial to move per step

Two readings of that table. First, 8.8 GB is a deliberate number: it fits alongside a policy, a simulator and its states on a single large GPU, or comfortably in host memory. Second, the expert set is a fifth of the buffer and yet supplies half of every batch during the bootstrap phase — expert transitions are being replayed roughly four times as often as their share of storage. That over-sampling is the whole mechanism, and it is why the phase-out in Chapter 7 is a change to the sampler, not a deletion from the buffer.

Three ways to get demonstrations, priced

The middle column of the last row is the whole argument. Fifty-five hours of demonstrated behaviour per hour of wall-clock, on a machine no human can puppet, for the price of four cost terms.

A dataset of only successes — is that cheating?

It is worth stopping on DiscardFailedEpisodes, because filtering a dataset by outcome is exactly the kind of move that produces a biased estimator, and in most of statistics it would be a mistake. Here it is not, and the reason is specific enough to be worth stating.

The concern would be real if the demonstrations were being used to estimate something about the world — a dynamics model, say, or the probability that a push succeeds. Deleting failures from a dynamics dataset gives you a model that thinks pushes always work, which is survivorship bias in its purest form.

But nothing here estimates dynamics. The transitions are used for one thing: to supply Bellman targets for a value function whose targets are computed from the stored reward and the stored next state, both of which are exactly what the simulator produced. A transition is not a claim that this outcome is typical. It is a claim that this state, under this action, went to that state and paid that reward — which remains true regardless of what happened to the rest of the episode.

There is a second-order cost, though, and it is the honest caveat. The buffer contains no crash transitions, so the crash penalty of −200 — a full half of the value function's dynamic range — is never demonstrated by the expert data. The critic learns about crashing entirely from the agent's own online experience.

Sanity-check the dataset size against real time

Four million high-level samples, at one sample per 0.2 seconds of simulated behaviour (Chapter 2):

Produced in four hours on one GPU. Now compare against the alternative the paper is arguing away from. Collecting 9.26 days of real teleoperated whole-body loco-manipulation on a Spot with an arm would take — assuming an operator who never sleeps, a robot that never needs a battery swap, and a tire that never rolls out of the workspace — nine and a quarter days. In practice, months. And it would have to be repeated for the G1.

Chapter 5: Offline-to-Online, Fifty-Fifty

We now have a four-million-transition buffer of successful, sparse-labelled, on-interface demonstrations. This chapter is about the learner that consumes it.

The base algorithm is TD3 (Fujimoto et al., 2018), configured following FastTD3 (Seo et al., 2025) for massively parallel simulation, with the offline-to-online recipe of Ball et al. (2023) layered on top. Three ideas, each of which we should be able to state from first principles.

TD3 in one pass, with the paper's actual settings

TD3 is a deterministic actor-critic method for continuous control. The actor π_φ(s) outputs an action directly — no distribution — and the critic Q_θ(s, a) scores it. The actor is improved by gradient ascent on the critic:

which is the equation Chapter 1 showed to be identically zero under a flat critic. TD3 adds three fixes to its predecessor, and the paper's hyperparameters name all of them:

That Polyak factor has an interpretable meaning. Each update does θ⁻ ← (1−τ)θ⁻ + τθ, so an old value decays by (1−τ) per update. Its half-life:

how long the target network remembersn = ln(0.5) / ln(1 - 0.005) = (-0.693147) / (-0.00501254) = 138.3 updates

# So the regression target the critic chases is roughly a 138-update-old
# average of itself. Slow enough to be stable; fast enough to track.

The FastTD3 shape: enormous batches, many gradient steps

Table 1 of the paper looks unusual if you are used to single-environment RL:

The batch size is not a memory knob here, it is a variance argument. With thousands of parallel environments feeding the buffer, consecutive transitions are highly correlated; a large batch drawn across the whole buffer decorrelates the gradient. And with 8 gradient steps per environment update, the ratio of learning to acting is high — which is only safe because the buffer is huge and, crucially, because the critic is bounded (Chapter 6).

The one line that makes it offline-to-online

Here is the entire modification, in the paper's words: "During the initial phases of learning, we replace 50% of the transitions in the replay buffer with pre-collected expert data."

Not "pretrain on the offline data, then fine-tune." Not "initialise the actor by behaviour cloning." Every sampled batch is half expert, half self-collected, from step zero. Concretely:

python — the offline-to-online switchdef sample_batch(online_buf, expert_buf, N_b, expert_ratio):
    if expert_ratio > 0:
        n_e = int(N_b * expert_ratio)          # 8192 * 0.5 = 4096
        n_o = N_b - n_e                        # 4096
        return concat(expert_buf.sample(n_e), online_buf.sample(n_o))
    return online_buf.sample(N_b)

# in the training loop, once per update:
ratio = 0.5 if success_rate < eta else 0.0      # eta = 0.1 — see Chapter 7
for _ in range(8):                             # N_grad = 8
    batch = sample_batch(online, expert, 8192, ratio)
    td3_update(batch)

Four thousand and ninety-six expert transitions in every batch. Why does that fix the flat-Q problem from Chapter 1? Because a fraction of those 4,096 are goal-terminal transitions carrying r = 0 and a done flag. For those, the Bellman target is not −1 + γQ′ but simply 0. That single different number propagates: states one step from a goal learn Q ≈ −1, states two steps away learn −1.99, and a gradient in a appears wherever the critic can distinguish actions that shorten the path from actions that lengthen it.

The whole update, with every paper setting in place

Assemble the pieces into the loop that actually runs. Every constant below is from Table 1 or from the appendix; nothing is invented.

python — one TD3 update, as configuredGAMMA, TAU, SIGMA_S = 0.99, 0.005, 0.05
POLICY_DELAY, ACT_REG, R_SCALE = 2, 5e-4, 0.01
Q_MIN, Q_MAX = -2.0, 0.0                 # derived in Chapter 6

def td3_update(batch, step):
    obs, act, rew, nobs, done, obs_exact, nobs_exact = batch
    rew = rew * R_SCALE                          # -1 -> -0.01, -200 -> -2.0

    # ---- critic ---------------------------------------------------
    with torch.no_grad():
        na = actor_target(nobs)                  # actor sees NOISY obs
        na = (na + (torch.randn_like(na) * SIGMA_S).clamp(-0.5, 0.5)).clamp(-1, 1)
        q1t = critic1_target(nobs_exact, na)     # critic sees EXACT obs
        q2t = critic2_target(nobs_exact, na)
        y   = rew + GAMMA * (1 - done) * torch.min(q1t, q2t)

    q1, q2 = critic1(obs_exact, act), critic2(obs_exact, act)
    critic_loss = mse(q1, y) + mse(q2, y)
    opt_step(critic_opt, critic_loss)         # lr 1e-4 (5e-4 on G1)

    # ---- actor, every OTHER step ----------------------------------
    if step % POLICY_DELAY == 0:
        a = actor(obs)
        actor_loss = -critic1(obs_exact, a).mean() + ACT_REG * (a ** 2).mean()
        opt_step(actor_opt, actor_loss)       # lr 3e-4
        polyak(actor_target, actor, TAU)      # tau 0.005
        polyak(critic1_target, critic1, TAU)
        polyak(critic2_target, critic2, TAU)

Six details in that block repay a second look, because each one is a decision rather than boilerplate.

Watch the value propagate, backup by backup

The claim that "one goal transition breaks the symmetry" deserves to be watched happening rather than asserted. Take the unscaled reward for legibility and follow the Bellman target through six rounds of updates, starting from a critic that predicts a constant −100 everywhere — the flat fixed point from Chapter 1.

value propagating back from one goaly = r + gamma * (1 - done) * Q_target(s_next)      gamma = 0.99

round 1  state AT the goal        done=1  y = 0                        = 0.000
round 2  one step from the goal  done=0  y = -1 + 0.99*(  0.000)   = -1.000
round 3  two steps away          done=0  y = -1 + 0.99*( -1.000)   = -1.990
round 4  three steps away        done=0  y = -1 + 0.99*( -1.990)   = -2.970
round 5  four steps away         done=0  y = -1 + 0.99*( -2.970)   = -3.940
round 6  five steps away         done=0  y = -1 + 0.99*( -3.940)   = -4.901

# meanwhile every state NOT on a path to a goal is still pinned at -100.
# -4.901 vs -100 is a gradient. That difference IS the learning signal.

Read the last comment carefully. The critic now assigns wildly different values to states five steps from a goal and states nowhere near one. The actor's ascent direction ∇_aQ is no longer zero, because moving toward the goal region changes Q by tens of units. From nothing, a landscape.

Now put the throughput next to it. Value information travels backwards at most one step per gradient update, so how fast does the useful region grow? With 8 gradient steps per environment update, the frontier of grounded value expands by up to 8 steps of the MDP per update cycle — and 8 MDP steps is 1.6 seconds of robot behaviour at 5 Hz. A few hundred update cycles is enough to reach the start states of every episode in the dataset.

Why this is not behaviour cloning, in one table

That second row is the paper's central claim in the making: "allowing RL to outperform the expert demonstrations." An imitation method cannot make that claim in principle. A value-based method can, because the value function is defined by the reward, not by the data.

Chapter 6: The Critic Cannot Lie Upward

This chapter is about one architectural detail that most readers skim, and it is one of the most transferable ideas in the paper. It takes three sentences to state and a page to justify.

The sentences: "because our sparse reward formulation defines strictly known maximum and minimum possible returns, we structurally bound the critic network outputs to these limits. Unbounded overestimation in Q-functions is strongly associated with learning instability, and bounded critics have been successfully applied in advanced algorithms. Empirically, we observe that enforcing these bounds at the architectural level significantly stabilizes training, particularly when hyperparameter configurations are still suboptimal."

Derive the bounds

Under the Chapter 1 reward, what is the largest return an episode can possibly earn? The best case is reaching the goal immediately: reward 0, episode ends. So the upper bound on Q is 0. No policy, no state, no action can do better, because there is no positive reward anywhere in the MDP.

The lower bound takes one line of care. There are two candidate worst cases:

The paper writes exactly this expression and appends the reason the second term does not compound: "taking into account that all robot crash penalties immediately terminate the episode and thus do not accumulate any future rewards." You cannot crash twice. So the floor is a single crash at step 0, and the whole value function is confined to

How to enforce it in the network

The paper's recipe: "We enforce this by adding a tanh activation function after the critic's final linear layer and then shifting and scaling it to the desired interval. We also add a constant bias to the prediction before applying tanh, thereby shifting the network output towards zero during initialization." Written out:

python — a critic head that cannot escapeclass BoundedCritic(nn.Module):
    def __init__(self, obs_dim, act_dim, hidden, q_min=-2.0, q_max=0.0, b0=2.0):
        self.trunk = mlp(obs_dim + act_dim, hidden, 1)
        self.q_min, self.q_max, self.b0 = q_min, q_max, b0

    def forward(self, obs, act):
        u = self.trunk(cat([obs, act], -1)) + self.b0   # bias BEFORE tanh
        t = tanh(u)                                   # t in (-1, 1)
        # affine map (-1, 1) -> (q_min, q_max):
        return self.q_min + (t + 1.0) * (self.q_max - self.q_min) / 2.0

Check the map by hand at the three points that matter. With q_min = -2 and q_max = 0:

verifying the affine mapt = -1  ->  -2 + (-1+1)*2/2  = -2.0       # the crash floor
t =  0  ->  -2 + ( 0+1)*2/2  = -1.0       # the "stall forever" value, -1/(1-g)*0.01
t = +1  ->  -2 + ( 1+1)*2/2  =  0.0       # at the goal

# with b0 = 2.0 and a near-zero trunk output at init:
u = 0 + 2.0 = 2.0,  tanh(2.0) = 0.9640  ->  Q_init = -2 + 1.9640 = -0.036

The bias does what the paper says: at initialisation the critic predicts a value close to 0 rather than close to the middle of the interval. Starting optimistic under a purely negative reward is deliberate — every observed transition then pushes the estimate down toward the truth, which is a well-conditioned direction to learn in and gives untried actions a slight optimism bonus early on.

Why unbounded critics break, mechanically

The failure mode the paper is guarding against has a name: the interaction between bootstrapping, function approximation and off-policy data, sometimes called the deadly triad (van Hasselt et al., 2018). Trace the loop once:

Now insert the bound. Step 3 becomes impossible: the network cannot output anything above 0, so the inflation has nowhere to go. It also cannot output anything below −2, so a pathological pessimism cannot run away either. The tanh saturates and the gradient through it shrinks as the estimate approaches a bound, which acts as an automatic brake exactly where the trouble starts.

Put a number on how bad unbounded overestimation gets

The word "compounds" is doing heavy lifting in the argument above. Make it arithmetic. Suppose each Bellman backup injects a small positive bias δ — from function approximation, from the max in the target, from stale target networks. The biased recursion is

which has the same geometric structure as the return itself. The accumulated bias at the fixed point is:

how a per-backup bias becomes a value errortotal bias = delta / (1 - gamma) = delta * 100        # gamma = 0.99

delta = 0.01  -> bias  1.0   on a true range of 200   (0.5 %)  survivable
delta = 0.05  -> bias  5.0                            (2.5 %)  noticeable
delta = 0.20  -> bias 20.0                            ( 10 %)  the policy now
                                                       prefers a fantasy
delta = 1.00  -> bias 100.0  -> Q above 0            # IMPOSSIBLE by construction

The same 1/(1−γ) factor that set the effective horizon in Chapter 1 also amplifies bias a hundredfold. That is the deadly-triad loop in one line: any systematic optimism is multiplied by the effective horizon before it reaches the policy.

The last row is the point. With a bounded head, a bias large enough to matter would have to push the output past 0, and it cannot. The bound does not eliminate small biases — it eliminates the runaway, which is the failure that shows up as a training curve falling off a cliff and slowly climbing back.

The bound is also an automatic brake, and you can see it in the derivative

There is a second effect of the tanh head that the paper does not spell out and that is worth deriving, because it explains why the bound helps before anything actually hits the ceiling.

The derivative of tanh is 1 − tanh²(u), so the gradient reaching the trunk is scaled by how close the output already is to a bound:

how the tanh head throttles its own gradientt = tanh(u)      dQ/du = (Q_max - Q_min)/2 * (1 - t**2) = 1.0 * (1 - t**2)

Q = -1.00  (mid-range, stalling)   t =  0.00   scale = 1.000   full speed
Q = -0.50                          t =  0.50   scale = 0.750
Q = -0.20  (nearly at the goal)    t =  0.80   scale = 0.360
Q = -0.05                          t =  0.95   scale = 0.098   10x slower
Q = -1.95  (nearly a certain crash) t = -0.95   scale = 0.098   10x slower

Read that as a control system. In the middle of the range — where most states live and where the interesting distinctions are — the head is transparent and learning proceeds at full rate. As an estimate approaches either extreme, the effective learning rate for that sample falls automatically, by a factor of ten before the bound is even close. The runaway loop from the flow diagram above needs each round to move the estimate further; the tanh makes each successive round move it less.

This is why the paper can honestly report that bounded critics are "still well-behaved" at eight times the default critic learning rate. A global learning rate that is too large is dangerous only where it produces large steps; the tanh shrinks precisely the steps that would run away, and leaves the rest alone.

Three ways the field bounds critics, and where this one sits

Note that these are complementary, not competing: the paper uses TD3's twin critics and the structural bound. The first reduces the frequency of optimism, the second caps its magnitude.

The ablation, and what it actually claims

The authors are careful not to overclaim. Their test raises the critic learning rate to 8 × 10⁻⁴ — eight times the default of 1 × 10⁻⁴ — on Spot box pushing, and compares:

So the claim is not "bounded critics learn better." It is: "bounded critic networks increase the range of hyperparameters that still yield successful runs, thereby making tuning easier." The bound widens the basin of settings that work.

Chapter 7: The Teacher Must Be Fired

Everything so far has argued that the expert data is indispensable. This chapter argues that it becomes harmful, and that the transition happens surprisingly early — at a 10% success rate.

The mechanism from Chapter 5: "To prevent the policy from continually relying on these demonstrations, we use a curriculum that phases out the expert data once the agent achieves a sufficient empirical success rate, shifting to pure online learning." The threshold η is 0.1. Once one episode in ten succeeds, the expert fraction drops from 50% to 0% and never comes back.

The result that makes this a chapter rather than a footnote

The appendix sweeps the threshold across all Spot tasks. The authors describe the outcome as surprising, and it is:

Read the 0.5 row once more, because it contains the cleanest possible causal evidence. The same run, same seeds, same everything, crawls while the expert data is in the buffer and immediately accelerates the moment it is removed. That is not a correlation. The expert data was the brake.

Why good data goes bad

The paper's explanation: "while the SMPC data is necessary to guide initial exploration, it is too far from the policy to achieve the fine-grained improvements required by the tasks, and it becomes harmful once the policy is narrowed to a successful strategy with a shifted data distribution."

Unpack that into the machinery. Once the agent is succeeding 10% of the time, its own trajectories occupy a narrow, specific region of state-action space — its strategy. The expert trajectories occupy a different region: overlapping, but centred elsewhere and, by Chapter 8's evidence, slower. Now consider what half a batch of expert data does to the critic:

Do the arithmetic in that last box, because it is the crux. Using the Chapter 1 formula with γ = 0.99:

a late-stage improvement, in returnG(40) = -(1 - 0.99**40)/0.01 = -(1 - 0.668971)/0.01 = -33.103
G(38) = -(1 - 0.99**38)/0.01 = -(1 - 0.682555)/0.01 = -31.745
delta = 1.358 out of 33.1 -> 4.10 %

# after the 0.01 reward scale, that is a Q difference of 0.0136
# on a critic whose whole output range is 2.0 units wide.
# The signal the actor must climb is 0.68% of the critic's range.

Two thirds of one percent. Every unit of critic capacity spent representing a distribution the policy has left is capacity not spent resolving that 0.68%. That is why removal helps, and why the effect is strongest on the hardest tasks — tire uprighting and tire rolling, which need the most fine-grained coordination and never converge with a 0.5 threshold.

Implementing the trigger without introducing a new bug

"Once the agent achieves a sufficient empirical success rate" is a sentence that hides an estimator, and estimators have failure modes. Three questions the implementation has to answer, each with a wrong answer that is easy to write.

python — a phase-out trigger that behavesETA, WINDOW = 0.1, 200

class PhaseOut:
    def __init__(self):
        self.recent, self.fired = deque(maxlen=WINDOW), False

    def on_episode_end(self, reached_goal, from_online_policy):
        if from_online_policy:                    # NEVER count expert episodes
            self.recent.append(float(reached_goal))

    def expert_ratio(self):
        if self.fired:
            return 0.0                            # latched — never comes back
        if len(self.recent) == self.recent.maxlen and np.mean(self.recent) >= ETA:
            self.fired = True
            return 0.0
        return 0.5

Note the third line of expert_ratio: the window must be full before the trigger can fire. Without that guard, a run whose first two episodes happen to succeed reports a 100% success rate and phases out on a sample of two.

And note what the trigger changes. It sets the ratio the sampler uses; it does not delete anything. The four million expert transitions stay exactly where they are, they simply stop being drawn. That distinction matters if you ever want to study what the expert data was doing — and it is why "phase out" is the right word rather than "discard."

The other dial: how much expert data, before the phase-out

The companion ablation varies the expert ratio rather than the threshold. The paper fixes it at 50% "following the recommendations of" Ball et al. (2023), and checks whether that transfers to SMPC-generated data:

That last row deserves attention because it looks paradoxical. The expert data is gone after the phase-out, so how can having had more of it still help afterwards? The authors offer a hypothesis worth taking seriously: "we posit that exposing the critic to more expert data before improves value estimation around successful states and thereby helps convergence once the policy becomes successful."

In other words the benefit is a legacy in the critic weights. A critic that saw many successful states early has a better-shaped value landscape around the goal region, and that shape persists after the data source is cut. The demonstrations are not just a bridge; they leave the far bank better mapped.

Why the trigger is a success rate and not a step count

A detail that is easy to miss: the curriculum fires on an empirical success rate, not on a step budget. That is a deliberate choice with a real consequence, and it is worth arguing through.

The success-rate trigger is self-scaling: it defines the switch in terms of the only thing that actually matters, namely whether the online buffer has begun producing its own goal transitions. One threshold generalised across five tasks and two robots without per-task adjustment. That is why η can be a constant in Table 1 rather than a per-task entry.

And notice that η = 0.1 is a low bar in a specific sense. At a 10% success rate and, say, 4,096 online samples per batch drawn from recent experience, roughly one in ten of the episodes represented contains a goal transition — enough for the online data to ground the critic by itself. The threshold is set at the earliest point where the lifeline becomes redundant, not at the point where the agent is good.

Drive the curriculum yourself

Set the threshold to its maximum (never phase out) on tire rolling and watch the curve stall in the same way the paper reports. Then drop the expert ratio to 0% on any task and watch the flat-critic failure from Chapter 1 reappear: with no goal-terminal transitions anywhere, nothing ever moves.

Chapter 8: The Student Outgrew the Controller

This is the result that gives the paper its edge. A policy trained entirely on trajectories produced by a sampling-based controller ends up faster than that controller on the task it was demonstrating.

The measurement, from Q1: the authors compare average task-completion time between the trained policy and the SMPC expert on all tasks. Their finding: "our sparse-reward approach consistently outperforms SMPC. Some tasks see improvements above 50%. Notably, learned policies show increased consistency, indicated by a 11-45% reduction in standard deviation of task duration."

How the comparison was made, and why the framing is fair

Read Figure 5's caption for the statistics, because a "we beat the teacher" claim lives or dies on them: "SMPC statistics are computed over the complete dataset of 4M samples, policy statistics are averaged over 50k simulated episodes without exploration noise."

One honest caveat you should notice yourself: the SMPC statistics come from a filtered dataset (Chapter 4 deleted failed episodes), so both sides are being scored on their successes. And both are evaluated in simulation, not on hardware — a hardware race would be confounded by the fact that the SMPC cannot run in real time at all.

Why the student can be better — three converging reasons

Reason one: the objectives differ, and only one of them is the task. The SMPC maximises the dense cost you wrote. The RL policy maximises the sparse return, which by Chapter 1 is a pure monotone function of time-to-goal. The paper puts it directly: "Sparse rewards eliminate the need for manual tuning and remove the biases introduced by surrogate reward shaping. This enables the discovery of task-optimal behaviors."

Concretely: your dense cost probably included a term keeping the end-effector near a nominal pose, or penalising base velocity, because without those the planner produced ugly motion. Those terms are now permanently in the teacher's objective. The student has no such term. If leaning further and moving faster gets to the goal sooner, the student takes it and the teacher never will.

Reason two: the teacher is a local, sample-limited optimiser. The paper attributes the expert's variance to exactly this: "We attribute SMPC's variance to suboptimal trajectories resulting from the limited number of parallel environments used for exploration." Recall the Chapter 3 order statistics — with K = 256 the elite sits about 2.66σ above the mean of the sampled candidates, but that is 2.66σ above the candidates it happened to draw, around a warm start that may already be in a mediocre basin. The planner re-solves from scratch every 0.2 s with no memory beyond one shifted trajectory.

Reason three: the student amortises across the entire dataset. A neural policy trained on 4M transitions has, in effect, seen the outcome of billions of simulated futures and compressed them into weights. At inference it does not search — it recalls. The paper's framing: pairing "the rapid tuning capabilities of optimal control with the fast execution speeds of neural networks and robustness of RL policies trained with domain randomization."

Work the improvement in return units

The paper reports times and standard deviations, not returns. Convert, so that the improvement is expressed in the currency the agent actually optimises. Take an illustrative task where the SMPC needs 8.0 s on average and the policy achieves the reported ">50%" improvement, finishing in 4.0 s. At the 5 Hz high-level rate:

a 50 % time gain, in return (illustrative)SMPC   : 8.0 s * 5 Hz = 40 steps
policy : 4.0 s * 5 Hz = 20 steps

G(40) = -(1 - 0.668971) / 0.01 = -33.103
G(20) = -(1 - 0.817907) / 0.01 = -18.209

improvement in return = 14.894          # 45 % of the teacher's return magnitude

# and the consistency gain, taking the reported 11-45 % std reduction
# on a teacher with std 2.0 s:  policy std = 1.78 s  ..  1.10 s

The 8.0 s and 2.0 s are ours, chosen to make the conversion legible; the 50% and the 11–45% band are the paper's. What is not illustrative is the direction and the mechanism: shaving steps is the only thing the sparse return rewards, so any consistent improvement in completion time is, by construction, an improvement in the objective.

Why the variance drop is the more interesting number

An 11–45% reduction in the standard deviation of task duration is easy to skim past, and it is arguably the better evidence for the paper's thesis. Consider what each system does when it meets a slightly unusual initial condition:

The student is not merely faster on average; it has removed a source of randomness that is intrinsic to the teacher's algorithm. And for a system that will run on real hardware, predictability is worth as much as speed.

Push the improvement to zero and the two distributions coincide — that is the behaviour-cloning ceiling from Chapter 5, the best an imitation method could ever reach. Everything to the left of that is territory only a value-based method can enter.

What "surpasses the teacher" does not mean

Claims of this shape get overread, so it is worth fencing the result carefully. Three things the paper does not establish, two of which it says so itself.

It does not mean the policy is globally optimal. The limitations section is explicit: "the achieved behavioral optimality remains local. Because the agent initially relies on SMPC demonstrations for successful samples, its policy is tied to the dataset's distribution… not expected to discover globally optimal strategies that fundamentally diverge from the provided trajectories." The student climbs the hill it was placed on, faster and more reliably than the teacher did — and it does not go looking for a different mountain. Chapter 9 makes this sharper still, since the dataset is deliberately narrowed to a single mode.

It does not mean SMPC is a bad controller. The comparison is between a planner that gets 256 samples and 0.2 seconds of commitment per decision, and a network that has absorbed the outcomes of billions of simulated futures. Give the planner 10,000 samples and a longer horizon and the gap would narrow — at a compute cost that puts it even further from real time. The teacher is not being beaten at its own game; it is being beaten at a game with a different scoreboard.

It does not mean imitation is obsolete. What it means is that imitation was the wrong consumer for this data. The same trajectories, fed to a behaviour-cloning objective, would have produced a policy that is by construction no faster than the demonstrations. The trajectories did not get better; the objective did.

How to check a claim like this in your own work

If you build this pipeline, you will want to make the same comparison, and there are three ways to fool yourself. Guard against each:

Chapter 9: How Much, How Good, How Many Ways

Three ablations, three questions an engineer would actually ask before adopting this pipeline. Two of them behave the way you would guess. The third does not, and it is the most instructive result in the paper.

Q2 — how much data do you need?

The authors scale the dataset "from hundreds of thousands to millions of samples" per task and report a clean pattern: "a clear correlation with task complexity: while easier tasks can be successfully solved with little data, more complex whole-body manipulation tasks require significantly larger datasets. On our most difficult task, agents require four million samples collected within 4 GPU hours to converge."

Their explanation is a statement about the critic, and it is the right one: "We posit that this effect is caused by the policy gradient estimate. While critics trained on little data yield sufficiently accurate gradient estimates for easy tasks, value estimates for complex tasks with a wider state distribution require training on a dataset with greater coverage."

And note the practical framing the authors give it: "While our method's data generation can be parallelized across multiple GPUs, a small dataset is generally desirable for efficiency." Four million samples is four GPU-hours. Nobody is claiming that is free; they are claiming it is the cheapest known way to get it.

Q3 — how good does the data have to be?

The quality knob is the one we predicted in Chapter 3: "Reducing the number of sampling environments during SMPC data generation leads to less exploration of the cost landscape, reducing the likelihood of sampling a well-performing trajectory, and thus resulting in lower-quality expert datasets."

Result: "Surprisingly, most tasks are relatively robust to a low number of sampling environments. However, for tire rolling, which requires the most coordination, we see a drop in successes. We thus conclude that the required data quality is highly correlated with the required task precision."

Both of our Chapter 3 predictions land. The order statistics said the loss from shrinking K is sublinear (256 → 64 costs only 0.5σ), which explains the robustness; and the loss compounds along a trajectory, which explains why the one task needing the longest chain of near-perfect decisions is the one that breaks.

Q4 — the ablation that should change how you think

Here is the setup. The tire-rolling task admits several genuinely different solutions. In the paper's own list: "kicking the tire into the goal with Spot's legs, pushing it with the shoulder, or stepping into the tire." All of them work. An SMPC that has not been told otherwise will find whichever one its random draw stumbles into on that particular episode.

So the authors built two datasets. One multi-modal, containing all of those strategies mixed together. One uni-modal, obtained by "stricter rewards" — specifically, "introducing terms for leg and body contacts generates solutions that use the arm instead" — and, they note, "these terms can be tuned interactively in just a few minutes."

Then the result, which is worth reading twice: "the policy trained on the multi-modal dataset completely fails to learn, even though the demonstration success rate is slightly higher, indicating that enforcing a single behavioral mode in the demonstration data is required for successful offline initialization."

Derive why, because the mechanism is completely general

The paper names the structural cause: "RL policies rely on the Markov property and output uni-modal action distributions, whereas SMPC is inherently multi-modal and not observation-dependent." Two separate problems are stacked in that sentence. Take them one at a time.

Problem one: mode averaging. TD3's actor is deterministic — one observation in, exactly one action out. Suppose two demonstrations pass through nearly the same observation s and take opposite actions: a₁ = +1.0 (swing the arm right and push) and a₂ = −1.0 (swing the leg left and kick). Both succeed, so the critic learns roughly equal values at both:

a deterministic actor, two equal modesQ(s, +1.0) = -25        # arm push: 25 steps to goal
Q(s, -1.0) = -25        # leg kick: 25 steps to goal
Q(s,  0.0) = ?          # NO DATA HERE. The network interpolates.

# A smooth network fitted to two equal peaks at +-1 with nothing in
# between typically produces a surface that is flat or even PEAKED at 0.
# Actor gradient at a = 0:
dQ/da |_(a=0) = 0        # by symmetry — the pulls cancel exactly

# So the actor sits at a = 0: arm half-raised, no contact, no kick.
# It is the AVERAGE of two good actions and it does nothing at all.

This is the classic mode-averaging failure of deterministic and uni-modal policies, and it is why the field reaches for diffusion policies and mixtures when it wants to keep multi-modal demonstrations. It is also worse than it looks here, because TD3 adds target policy smoothing — it deliberately makes Q smooth in a, which is exactly the property that fills the valley between two modes with a plausible-looking plateau.

Problem two: the teacher is not a function of the observation. This is the Chapter 3 warm-start property coming due. The SMPC's action depends on the state and on the initial guess it inherited from previous iterations. So the same observation, in two different episodes, can legitimately produce different actions purely because the planner's internal history differed. From the student's point of view, the mapping it is asked to value is not a function at all. Even a stochastic policy class would struggle; a deterministic one has no chance.

Detect the problem before you spend four GPU-hours on it

The multimodality failure is invisible in every summary statistic you would normally check. Demonstration success rate: fine, higher even. Dataset size: fine. Reward distribution: fine. Training curve: flat, which looks exactly like a learning-rate problem, a network-size problem, or a bug. You could burn a week on it.

So measure it directly, before training. The test follows straight from the mechanism: find groups of transitions with similar observations and ask whether their actions agree.

python — a modality auditimport numpy as np
from sklearn.neighbors import NearestNeighbors

def modality_score(obs, act, k=32, n_probe=5000):
    """Ratio of local action spread to global action spread.
    ~0.0 = the data is a function of the observation (uni-modal).
    ~1.0 = knowing the observation tells you nothing about the action."""
    nn = NearestNeighbors(n_neighbors=k).fit(obs)
    idx = np.random.choice(len(obs), n_probe, replace=False)
    _, nbr = nn.kneighbors(obs[idx])          # (n_probe, k)

    local  = act[nbr].std(axis=1).mean(axis=0)  # spread among neighbours
    glob   = act.std(axis=0)                  # spread over the whole set
    return (local / (glob + 1e-8)).mean()

# Reading the number, per action dimension as well as averaged:
#   < 0.3   a deterministic actor can fit this. Proceed.
#   0.3-0.6 suspicious. Plot the per-dimension histogram of act[nbr]
#           and look for TWO PEAKS rather than one wide blob.
#   > 0.6   your planner is solving the task more than one way.
#           Go add cost terms until it stops. Minutes, not hours.

The distinction in the middle band matters: a wide unimodal blob is noise, which a deterministic actor handles fine by learning the conditional mean, because the mean of a single mode is a member of that mode. Two peaks are modes, and the mean of two peaks is a member of neither. Always look at the histogram, never only the standard deviation.

The three ablations as a checklist

Chapter 10: On Real Robots, Honestly

Everything so far happened in MuJoCo Warp. This chapter is about what left the simulator, what the paper admits it cannot yet do, and where the idea goes next.

The five tasks and the two machines

The object specifications matter because they set the scale of the physics. The box is 0.5 × 0.5 × 0.5 m at 1.2 kg — light, so pushing it is about contact and geometry rather than force. The tire is 0.33 m in radius, 0.34 m wide, and 14.3 kg — roughly twelve times the box's mass and enough that the reaction force genuinely threatens the robot's balance. That is the jump in difficulty between rows two and three.

The deployment setup, stated plainly

Both of those deserve a moment of engineering honesty. Off-board inference over WiFi adds latency and a failure mode that would not exist onboard; the authors flag it as convenience, and the G1 result shows the policy is small enough to run onboard when they choose to. Motion capture is the bigger asterisk: it means the robot is not perceiving the tire, it is being told where the tire is by an external system in an instrumented room.

The headline result stands on its own terms: "the policies trained exclusively on sparse rewards can be reliably deployed on the physical hardware across all five tasks and both robotic platforms. This demonstrates that our hierarchical architecture and offline-to-online training pipeline generalize robustly to real-world dynamics without requiring platform-specific reward engineering." No per-platform reward tuning, on two morphologies, on hardware. That is the claim, and it is well supported.

The limitations, in the authors' own order

The paper's limitations section is unusually candid. Four items, each with a cause you can now trace back through this lesson.

Where this sits in the landscape

Three families of prior work converge here, and the paper positions itself against each.

Against MPC for loco-manipulation. Whole-body inverse-dynamics MPC, hierarchical MPC for non-prehensile manipulation, nonlinear MPC with manipulator dynamics — all deploy the optimiser on the robot and pay its computational cost forever. This paper deploys the optimiser in the datacentre, once.

Against demonstration-guided RL. Adding human demonstrations to a replay buffer, augmenting behaviour cloning losses, bootstrapping with imitation, informed reset distributions — the paper's objection is uniform and practical: those works study "tabletop end-effector manipulation tasks", and "obtaining human demonstrations for loco-manipulation tasks, especially on non-humanoid robots, is a restrictive limitation."

Against planner-generated data. Closest of all is Jacta, which uses a graph-based motion planner with task-specific heuristics to generate dexterous manipulation data for RL. The distinction the authors draw: "Our approach focuses on dynamic, whole-body loco-manipulation rather than stationary dexterity. Furthermore, instead of relying on tree search and discrete heuristics, we utilize a unified SMPC framework that readily scales to a wide variety of loco-manipulation tasks and embodiments."

And the forward-looking claim, which is the one most likely to outlive the specific results: "By demonstrating that SMPC can be visually tuned in near real-time and massively parallelized for data generation, we position it as a scalable utility for the RL community." Not a controller. A utility — a thing you keep in the toolbox next to your simulator and reach for whenever a sparse-reward task refuses to explore.

Build it yourself, in the order that will not waste your time

Every ordering decision below is one the paper's own ablations justify. Follow it and the failure modes announce themselves early, when they are cheap.

The cheat sheet

References worth reading next

1. Schuck, M., Sorokin, M., Manni, S., Ta, D., Schoellig, A. P., Hutter, M., Le Cleac’H, S., Brüdigam, J. "Learning Loco-Manipulation From SMPC Demonstrations With Sparse Offline-to-Online RL," 2026 — arXiv:2608.12063. The paper this lesson is built on.
2. Howell, T., Gileadi, N., Tunyasuvunakool, S., Zakka, K., Erez, T., Tassa, Y. "Predictive Sampling: Real-time Behaviour Synthesis with MuJoCo," 2022 — reference [8]. The sampling-based planner this one is modelled on.
3. Fujimoto, S., van Hoof, H., Meger, D. "Addressing Function Approximation Error in Actor-Critic Methods" (TD3), ICML 2018 — reference [6]. Twin critics, delayed updates, target smoothing.
4. Seo, Y., Sferrazza, C., Geng, H., Nauman, M., Yin, Z., Abbeel, P. "FastTD3: Simple, Fast, and Capable Reinforcement Learning for Humanoid Control," 2025 — reference [18]. The massively-parallel TD3 configuration.
5. Ball, P. J., Smith, L., Kostrikov, I., Levine, S. "Efficient Online Reinforcement Learning with Offline Data," ICML 2023 — reference [1]. The source of the 50% expert-ratio recommendation.
6. Zhu, X., Chen, Y., Sun, L., Niroui, F., Le Cleac’h, S., Wang, J., Fang, K. "Versatile Loco-Manipulation through Flexible Interlimb Coordination" (ReLIC), CoRL 2025 — reference [24]. The frozen low-level controller.
7. Brüdigam, J. et al. "Jacta: A Versatile Planner for Learning Dexterous and Whole-body Manipulation," CoRL 2024 — reference [4]. The closest prior work on planner-generated RL data.
8. van Hasselt, H. et al. "Deep Reinforcement Learning and the Deadly Triad," 2018 — reference [7]; Bhatt, A. et al. "CrossQ," ICLR 2024 — reference [2]. The instability the bounded critic guards against, and prior art for bounding it.
9. Zakka, K. et al. "mjlab," 2026 — reference [22]; Todorov, E. et al. "MuJoCo," IROS 2012 — reference [19]. The simulator stack that makes tiled sampling possible.