Learning to act
while waiting

Suppose the robot's previously trained policy already knows several plausible ways to approach the bag. We want reinforcement learning to favor the useful movements without rebuilding that entire skill from scratch. A R L I builds on a method called diffusion steering through reinforcement learning, or D S R L.

A diffusion or flow policy starts its action generation from a noise vector. Guided by the observation, a learned generator turns that vector into a sequence of robot commands. Different initial noise can produce different coherent movements. D S R L trains a small actor to choose that initial noise using reward. The actor learns which starting points tend to produce useful behavior through the existing generator.

The large vision and language backbone stays frozen during this reinforcement learning stage. The action expert that converts noise into actions also stays frozen. The small actor and its value estimators are the parts being trained. The real base policy's earlier supervised adaptation is a separate stage.

In these experiments, the actor selects one latent vector and repeats it along the action chunk's time dimension. Repeating the noise does not mean repeating one robot command. The action expert still uses its learned temporal structure to generate a changing sequence of movements.

This is also different from residual control. A residual controller adds a correction to an action that already exists. Latent steering chooses an input before the action generator runs. It inherits the generator's useful structure, but its intervention must arrive before that generator needs the noise. That deadline will determine how fresh the actor's information can be.

Imagine a much simpler robot that moves along a line. Its old observed position is zero. In the first episode, it has already committed to two movements to the right, each of size one. In the second episode, it has committed to two equally large movements to the left. At the instant a new correction can begin, the first robot will be at positive two and the second at negative two.

Both old observations say zero. But returning to the center requires opposite corrections. The first robot needs a correction of negative two. The second needs positive two. A learning policy that sees only the old observation cannot distinguish those cases. Averaging the corrections would produce zero, which solves neither one.

This illustrates a state aliasing problem. Different decision situations have been compressed into the same input. In a Markov decision model, the state should contain the information needed to describe how the next outcome depends on the chosen action. The physical environment may satisfy that description while the small controller's chosen input does not. An old image alone can leave out the action queue that is already shaping the future.

This line robot is a constructed teaching example, not a reported benchmark. Its simple dynamics expose bookkeeping that is harder with a moving gripper and a soft bag.

Notice that the missing information is not secret. The controller has already selected those pending movements. It can read them from its queue. The first modification in A R L I is therefore to give that queue to the learner.

Return to the two line robots. This time, the actor receives both the old position and the pending action sequence. It can distinguish zero followed by two rightward movements from zero followed by two leftward movements. Those different inputs can now support different corrections.

A R L I applies this idea to the real inference window. The small actor receives the actions from the previous chunk that will be executed while the large model computes the next chunk. These commands are already committed. Including them does not let the actor change the past or replace commands after they have been sent to the robot.

The implementation described in the paper flattens the intermediate action array and joins it to the other features before the small network's final layers. The actor learns how those commands affect the circumstances in which its noise will be used, without a separate world model.

There is an important distinction from real time chunking, or R T C. R T C also uses committed actions, but primarily to encourage continuity between the old and new motion. Giving the same information to a reward trained actor lets it learn how the pending motion affects task success. Continuity and reward directed adaptation are related goals, but they are not the same computation.

Now break our simple example. Keep the pending commands unchanged, but imagine someone moving the target while inference is running. The queue can explain the robot's planned motion. It cannot reveal a disturbance that was absent from the old observation. In a real scene, objects can slip, contact can change, and another object can enter the path. To react to those events, the learner needs genuinely newer information, not only a better interpretation of the old frame.

The useful opportunity is inside the large model's computation. Many vision, language, and action models have a vision and language backbone followed by a smaller action expert. The backbone processes the image and instruction. The expert uses the resulting features and the noise input to generate actions.

Noise is needed when the action expert starts. A R L I therefore runs its small actor later, with another observation, the original observation, and committed actions. This adds information without a second expensive backbone pass.

Be precise about what updates. The intermediate image enters the small actor. It does not replace the earlier features already produced by the frozen vision and language backbone. The actor uses the new information to choose noise that steers the existing action expert.

Here is the real setup expressed on one clock. Start the large inference call at zero milliseconds. The first new action is scheduled about one hundred sixty seven milliseconds later. The remaining delay for the small actor and action expert together is seven control steps, or about one hundred seventeen milliseconds. That places the extra observation fifty milliseconds after the first one. It is fresher, but still not current when execution begins.

Move a disturbance to just before that extra observation in the interactive timeline. Move the disturbance just after the observation, and this noise choice cannot respond to it. The method has shortened an information gap, not abolished causality. Its benefit depends on which events become visible before the intervention deadline and whether the frozen generator can express a useful response.

A useful correction can still produce an awkward handoff between chunks. The old plan moves the gripper in one direction. A newly generated plan begins somewhere incompatible. Abruptly switching between them can create a jump in the commanded motion.

Real time chunking addresses this continuity problem. During generation, it encourages the beginning of the new chunk to agree with the actions that are already committed. A softer overlap region can connect that fixed prefix to a freely generated continuation. A R L I can combine this guidance with the noise selected by its learned actor.

Keep two clocks separate. The prefix consists of robot commands indexed by physical control time. The action expert also takes numerical steps while it turns noise into a trajectory. Ten committed robot commands are not necessarily ten denoising iterations. The paper lists inference delay and denoising step count as different settings. Confusing them produces the wrong mask and the wrong execution schedule.

When a chunk is ready, the controller skips the part whose physical time has already passed. It does not execute those prefix commands again. In the real preset, the prediction contains fifty actions. Ten correspond to the elapsed prefix, and the controller executes twenty actions from the usable continuation before replanning. The remaining predicted tail is not automatically executed.

Continuity guidance also creates a tradeoff. If old actions are strongly enforced, the new policy has less freedom to respond immediately. This can improve smoothness while limiting reactivity. The authors explicitly identify that limitation. Their experiments often benefit from R T C, but that does not make it universally helpful under every disturbance.

The learned actor and continuity guidance have distinct roles. Next, follow how their combined experience becomes training data.

The robot sends many low level commands during one action chunk, but its noise actor makes one decision for that chunk. Training data must preserve this distinction. Otherwise the learner could be credited with decisions it never made, or paired with observations that belong to the wrong point in the timeline.

The paper treats sampling and executing a chunk as one transition in a latent decision process. The action stored for the small controller is its chosen noise. The environment still evolves at the faster physical control rate. When the simulation plots show environment steps, those are the original control steps, not the number of entries in the latent replay buffer.

The observation pipeline is concrete. The real setup has a front camera and one wrist camera on each arm. The small network uses images resized to sixty four pixels square. Three color views contribute nine channels for one time point. Joining old and intermediate views gives eighteen channels under a channels last description. Joint measurements are joined separately. Pending actions are flattened and added before the final multilayer network.

An actor proposes latent noise. A group of ten critics estimates its value. The actor is rewarded for noise that receives higher value while retaining an entropy incentive for exploration. The critics learn from reward and a discounted estimate of what follows. Terminal masks prevent a finished episode from borrowing value from a nonexistent future.

Replay discount must match the decision interval. Upstream D S R L raises its per step discount to the query horizon. That synchronous reference is not the unreleased A R L I integration. The theorem also uses a different stride from the real execution horizon. Keep these settings distinct.

Suppose a critic could evaluate every relevant action chunk perfectly. There would still be a cost to deciding from delayed information. An unexpected event may make the best continuation different from the one chosen earlier. The theory asks how that cost accumulates when each delayed decision has a bounded mismatch from an ideal reference.

The policy in the appendix preserves the actions already committed and chooses the best remaining completion according to a chunk value function. Its next backup advances by the number of actions left after the delayed prefix. That specific interval determines how much the next error is discounted.

Imagine the largest value error at one decision. It is bounded by a local mismatch plus a discounted copy of the future error. Repeating the argument gives a geometric series. If the discount is below one and the completion advances time, the series has a finite sum. A larger local mismatch produces a larger bound. A discount closer to one accumulates more of the future mismatch.

The assumptions do substantial work. The data must obey the environment's transition dynamics. It must have the required open loop consistency. Optimal chunks must lie within the data's support, and so must optimal continuations for the committed prefixes being considered. A critic cannot be declared correct for an unseen completion simply because its neural network produces a confident number.

The paper's theoretical schedule is not identical to every practical setting. Fifty predicted actions minus ten delayed actions leaves forty, while the real controller executes twenty before replanning. That is why the bound should not be presented as a numerical certificate for the deployed actor. It also does not account for every finite data or optimization error. Its useful message is conditional: if delay creates only a controlled oracle mismatch and the learning assumptions hold, that mismatch need not grow without bound.

The simulation study tests whether these information changes help reinforcement learning recover under delay. It includes three dynamic tasks from a physics benchmark and a simulated bimanual cube transfer task. The physics tasks use compact symbolic observations and a previously trained flow policy. The cube transfer experiment uses images and joint state with a much larger pie zero policy.

The main comparisons include naive asynchronous D S R L, the same method with real time chunking, A R L I with and without real time chunking, and residual reinforcement learning. These methods intervene at different places and receive different information. A fair comparison must keep those differences visible.

In the reported curves, A R L I generally learns more effectively under delay than naive latent steering. Adding continuity guidance helps in several cases. The car launch task is an important exception: several methods perform strongly there. The evidence does not say ordinary reinforcement learning fails in every environment that has latency.

The simulation curves average three random seeds. Their shaded uncertainty corresponds to one standard error. Evaluations use fifty rollouts. Those are different sources of counting: training seeds, evaluation attempts, and original environment steps along the horizontal axis. A smooth line is not an infinite sample, and its final plotted point should not be read as an exact universal performance level.

Our interactive comparison uses the reported experimental settings and qualitative curve evidence. It does not manufacture missing raw measurements or display a synthetic learning curve as if it came from the authors. The next question is whether the same idea helps on physical hardware, where a bag can bend and a connector can catch on its pins.

The real experiments use a bimanual robot with two arms. One task inserts a power connector into an inverter box. Another places a shoe into a narrow bag. The third lifts and positions a bag with both arms. Each task begins from a pie zero point five policy adapted using human demonstrations.

The paper reports improvement from roughly forty percent starting success to near perfect success windows after around one hundred to one hundred twenty five online episodes across these tasks. Read that statement together with the graph. Each point summarizes the most recent ten training episodes. Individual task baselines differ, and later windows can dip. Near perfect windows are evidence of improvement, not a guarantee that all future attempts succeed.

There is also work before those online episodes. The base policy receives supervised adaptation from task demonstrations. The real reinforcement learning setup begins with twelve hundred latent transitions and twenty four thousand offline training updates. Comparing only the online episode count can hide that initial investment.

Success rate is not the only practical measurement. A slow policy that succeeds often may complete fewer useful tasks than a faster one with similar reliability. The authors estimate successes per hour using successful episode durations from the last twenty episodes. They assume a failed attempt takes twice the average successful duration before failure is declared.

Try a constructed example. If successful attempts take twelve seconds and success probability is eighty percent, that timeout assumption gives an average attempt length of fourteen point four seconds. The resulting estimate is two hundred successes per hour. Those are teaching inputs, not a measured A R L I throughput result. Change the timeout assumption and the estimate changes.

Ablations ask which parts of the mechanism actually matter. The authors remove the intermediate action information, remove the fresher observation, and compare different amounts of remaining delay. The pattern is informative without being uniform across tasks.

Using both kinds of intermediate information generally works better than using only one in the tested settings. In the real shoe task, the actions only variant is competitive with the full method. In bag placement, that reduced variant struggles to maintain equally strong learning. This suggests that predictable committed motion and newly observed events can contribute differently depending on the task.

The swimmer experiments also move the intermediate observation through the inference window. A smaller remaining delay gives the actor newer information. The plots and appendix show that this generally helps. Without real time chunking, the observation must be fresher to produce the same clear learning improvement. The main text contains an inconsistent phrase about the direction of this effect; our explanation follows the timing definition, the plotted conditions, and the appendix.

A separate diagnostic examines how the learned noise distribution changes during training. It replays four held out shoe episodes, two successful and two failed, through saved model checkpoints. This analysis uses the actions only model because those saved trajectories did not contain the intermediate observation needed by the full method.

The resulting heatmaps show changes that depend on the trajectory and phase of the movement. Failed examples show more displaced and sometimes narrower noise distributions around difficult alignment portions. The authors treat this conservatively. Four replayed trajectories do not establish a population level failure detector. They provide a way to inspect what the noise steering network changes, rather than assuming it merely rescales randomness by the same amount everywhere.

We can now place several robot learning methods on one timeline. Real time chunking maintains continuity while a base policy computes asynchronously. A R L I chooses latent noise before the action expert runs, using information that arrives later than the original backbone observation. A stepwise residual controller changes individual actions after generation. These intervention points come with different information deadlines and different constraints.

The parallel paper on Real Time E X P O fine tuning intervenes after action proposals have been generated. It edits and selects from generated action proposals using a more recent observation at a replanning boundary. Its base policy also receives behavior cloning updates from successful experience. That is a different training design from A R L I's frozen generator and latent steering. The two papers use different tasks and experimental setups, so their separate headline numbers do not establish a direct winner.

For a new system, first ask when the needed input becomes available. If the generator requires its noise immediately and cannot separate vision computation from action generation, A R L I cannot obtain the same late observation opportunity. Its committed action conditioning can still be useful. If a disturbance arrives after the actor has selected noise, that particular decision cannot react to it. If the frozen generator cannot express a successful continuation, better information alone may not create one.

The contribution is a precise correction to the learning problem: tell the small controller what is already happening and give it the freshest observation its intervention deadline permits. Then train under that same schedule.

When you explain A R L I to someone else, draw the queue before the network. Mark the old image, the later image, the noise deadline, and the first new action. Once those times are visible, the architecture has a reason to exist. The robot can keep moving while it learns what waiting means.