Dream in Structure, Then Act

Chapter 1: Four Ways to Couple Imagination and Action

Imagine two WAMs that predict exactly the same futures, from exactly the same data, with exactly the same network. One reaches 75% average success in simulation. The other reaches 34%. Nothing about what they predict differs. The only difference is when each piece gets generated and which pieces are allowed to look at which.

That is not a hypothetical. It is one column pair of the paper's Table I, at 250 demonstrations per task (ModAR at 75%, Independent-noise at 34%). This chapter builds the vocabulary to see why such a small-sounding choice can be worth forty points.

A one-paragraph primer on denoising generation

Every model in this chapter generates its outputs by denoising. Picture a sculptor who starts with a block of static and, in a handful of passes, chips it into a statue. Each pass looks at the current rough block and predicts the finished statue, and then moves the block a little toward that prediction. After enough passes, the block is the statue.

The formal name for the version ModAR uses is flow matching (Chapter 5 builds it from zero). The progress of one pass is tracked by a number the paper calls the flow timestep τ. In the paper's convention, τ = 0 means pure noise and τ = 1 means the clean result. Generation walks τ from 0 to 1 in a fixed number of steps. ModAR uses 8 Euler steps per generated stream.

One more word you will see constantly: a stream is one set of tokens being generated together. The future point tracks are one stream. The future DINO features are another. The action chunk is another. A WAM that predicts four future modalities plus actions has five streams to generate. The formulations below are five different answers to "how do those five streams relate while they are being generated?"

Formulation (b): Unified, or joint generation

The most common answer is to generate everything at once. All streams start as noise together, share one flow timestep, and are refined in lockstep. At every refinement step, every stream can attend to every other stream, so the action tokens can read the half-finished future tokens and vice versa.

Analogy: a comic artist sketching all six panels of a page simultaneously, glancing across the page while each panel is still a rough pencil sketch. The paper labels this Unified and uses it to represent joint-generation WAMs such as DreamZero and Cosmos Policy, which "denoise future-observation and action streams together" with cross-stream attention. In the paper's controlled implementation, Unified uses one shared flow timestep drawn from a logit-normal distribution with (μ, σ) = (−1, 1).

Notice the subtle consequence. At step 3 of 8, the action stream is being refined while looking at futures that are themselves only three-eighths clean. The action prediction conditions on partially noisy predicted futures. The paper returns to this asymmetry with a clever control experiment in Chapter 6.

Independent-noise: same inference, different training

A close cousin of Unified changes only the training procedure. During training, instead of drawing one shared τ for all streams, it draws a separate τ for each stream, independently. One training example might have the tracks at τ = 0.9 (almost clean) while the actions sit at τ = 0.2 (mostly noise). At inference it does exactly what Unified does: all streams denoised simultaneously on one shared schedule.

The paper attributes this design to Unified World Models and Flex-π. One way to see its appeal (this is our gloss, not a claim from the paper): a stream sampled near τ = 1 behaves almost like clean conditioning for the others, so a single network gets exposed to many different "who conditions on whom" situations during training.

The paper's from-scratch experiments, however, find Independent-noise "performs poorly." Its explanation is a train/test mismatch. At test time, all streams are always at the same noise level. During training, independently sampled noise levels "rarely match the synchronized test-time denoising schedule," so the model "receives insufficient training signal near its inference regime." It spends most of its practice on situations it will never face.

Formulation (c): Disjoint, as in Fast-WAM

The third answer cuts the connection. Disjoint predicts each target independently, "without attention between future-observation and action targets." The futures and the actions share the backbone that reads the current observation, but the action tokens never look at the predicted future tokens, and the future tokens never look at the action tokens.

Why would anyone want that? Because then the future prediction is purely a training-time auxiliary objective. At deployment you can skip generating the future entirely and save its cost. This represents Fast-WAM, whose paper title asks the question directly: do world action models need test-time future imagination? The future-prediction loss still shapes the shared representation during training. It just never feeds the action at run time.

Formulation (d): Action-only

The baseline predicts actions directly from observations and predicts no future at all. The paper describes it as "a typical flow-matching behavior cloning policy." Because it has no future-prediction objective, it has nothing to learn from actionless demonstrations, so it trains only on action-labeled data. That is why, in the results tables, Action-only has a number only at the smallest data scale: adding actionless data changes nothing for it.

Formulation (a): ModAR, futures first and one at a time

A separate lineage (UniPi and VERA) already reversed the coupling: they "fully denoise visual futures and then infer actions from the resulting frames, rather than co-denoising both streams." ModAR builds on that futures-then-actions ordering and extends it across modalities. It denoises future modalities one at a time, each fully, each conditioned on the ones already finished, and denoises actions last.

That gives ModAR's final step a clean interpretation. An inverse dynamics model (IDM) answers a detective's question: given what the world looks like now and what it will look like next, which action caused the change? If you see a drawer closed and then open, the answer is "someone pulled it." The paper notes that ModAR's final action-prediction step "acts as an inverse dynamics model," mapping the fully generated future to the actions that would produce it.

The simulation below animates the table. Each lane is one stream. A lane fills from left (noise) to right (clean) as it is denoised. The arrows show which streams the currently active stream can read. Switch to the training view to see how each formulation samples noise levels during training, and resample a few times to feel why Independent-noise so rarely practices the synchronized situation it meets at test time.

Spend a moment on the training view for Independent-noise. With five streams each drawing its own τ, the chance that all five land close together is small. Chapter 6 puts a toy number on "small." The point to carry forward: the inference procedure of Unified and Independent-noise is identical, so any gap between them in the results is caused purely by how their training noise was sampled.

What stays fixed across all five

A comparison is only as good as its controls. The paper implements every formulation "within one controlled implementation": all methods "share the same backbone, action-labeled data, and optimization budget." The same six simulation tasks, the same demonstration counts, the same 1.2M optimizer steps. Only the coupling changes.

Why ordering could matter at all

It is worth pausing on why generating streams in sequence might help, before the evidence arrives. When you generate everything jointly, every stream starts from noise with nothing solid to lean on. When you generate in sequence, the second stream starts with a finished first stream in hand.

The paper cites two image-generation results that motivate this. Latent Forcing generates image latents before pixels, "allowing the latents to serve as a semantic scratchpad for generating fine-grained appearance." Modality Forcing adapts a pretrained image generator to jointly generate depth and finds that image models contain priors that improve depth accuracy. The paper's summary of the lesson: "generating easier-to-predict modalities earlier can expose structure that simplifies subsequent predictions."

There is also a plain probabilistic reason, which Chapter 4 makes exact. Any joint distribution over several variables can be written as a product of conditionals, in any order, with no loss. So sequential generation does not restrict what the model can represent. It only changes what each generation step has to figure out on its own, and that is precisely the thing the order controls.

One decision, traced through two samplers

Abstract tables hide the thing that matters, so let us trace one decision. The robot is at decision time t. It must output 16 actions. Both models below predict the same futures (tracks, DINO, depth, RGB) plus actions, and both use 8 Euler steps per unit of work.

In Unified, one shared clock ticks 8 times. At tick k (k = 0 through 7) every stream sits at τ = k/8 and is moved to τ = (k+1)/8. (We assume evenly spaced steps throughout this lesson; the paper specifies eight Euler steps but not their spacing.) So when the action stream takes its first step, the futures it reads are pure noise (τ = 0). When it takes its last step, the futures it reads are at τ = 7/8 = 0.875, still carrying one eighth of their noise.

In ModAR, the clock ticks 8 times for tracks alone, then the finished tracks are frozen and re-embedded as context. Then 8 ticks for DINO, reading clean tracks the whole time. Then depth, then RGB. Only after 32 ticks does the action stream start its own 8 ticks, and on every one of them it reads four finished futures.

Here are the two sampling loops side by side, as pseudo-PyTorch. It is a sketch of the control flow the paper describes, not the authors' code. The key line is the one that decides what the network is allowed to read.

# ---- Unified: one shared clock, every stream refined together ----
streams = {m: torch.randn(shape[m]) for m in ["tracks", "dino", "depth", "rgb", "act"]}
for k in range(8):
    tau = k / 8
    x_hat = net(obs, streams, tau={m: tau for m in streams})   # joint attention, all noisy
    for m in streams:
        v = (x_hat[m] - streams[m]) / (1 - tau)
        streams[m] = streams[m] + v / 8

# ---- ModAR: one stream at a time, finished streams become context ----
context = {}
for m in ["tracks", "dino", "depth", "rgb", "act"]:
    y = torch.randn(shape[m])
    for k in range(8):
        tau = k / 8
        x_hat = net(obs, context, target=(m, y), tau=tau)  # reads only FINISHED blocks
        y = y + (x_hat - y) / (1 - tau) / 8
    context[m] = y          # re-embedded as clean context for every later block
actions = context["act"]

The ModAR loop is the paper's inference paragraph turned into code: "We generate one modality at a time... The resulting clean tokens become context for generating the next modality, and we condition the final action chunk on the complete generated future." The paper adds one engineering detail the sketch leaves out: keys and values for the current observation and all previously generated modalities are cached, so they are not recomputed at every denoising step (Chapter 4 explains why that is safe).

What each formulation does with a human video

Return to the data argument from Chapter 0. A human video of cup stacking has future frames but no robot actions. Each formulation handles that example differently, and the difference is the reason data scaling behaves so differently across them in Chapter 6.

The paper states the ModAR rule plainly: "For actionless examples, we omit action prediction and supervise only the available future targets." And the loss has an explicit switch for it: the action term is included "only for examples with action labels."

What happens when an imagined future is wrong

Every formulation has a failure mode, and they are different.

In Disjoint, a wrong future cannot hurt the action at run time, because the future is never generated at run time. The flip side: a right future cannot help it either. The paper also sees a subtler cost: as more actionless data is added, Disjoint's performance falls, and the authors suggest the shared representation becomes "increasingly shaped by the future-observation prediction objective, causing negative transfer to the action prediction objective."

In Unified, streams are refined together, so no stream ever commits early. That limits how much any single early mistake can dominate. But it also means the action never gets to read a committed future.

In ModAR, commitment is the point, and so is the danger. If the tracks come out wrong, DINO is generated to agree with wrong tracks, depth with both, and the action with all three. The paper names this directly: modality-autoregressive generation "is susceptible to cascading error, as imperfections in earlier generations become inputs to all later predictions." Its fix, context noise, gets a full treatment in Chapter 5, and removing it is the single most damaging ablation in Chapter 7 after removing tracks.

The formulations as answers to two questions

A compact way to hold all five in your head is to ask each of them two questions: are futures generated before actions or alongside them? and does the action ever read a predicted future?

Read the table as a map of what each design bets on. Disjoint bets that futures are useful only as a teacher for the shared representation. Unified bets that futures and actions should be negotiated together. ModAR bets that the future should be settled first, in a particular order, and that the action should then be chosen with that settled future in view.

One way to test that you really own the map: pick any cell and say what the model would do with a human video of cup stacking. Unified and Independent-noise learn the future streams from it and hope joint attention carries the lesson to the actions. Disjoint learns the future streams from it and then never generates them at run time. ModAR learns its future blocks from it, and at run time its action step reads blocks of exactly that kind. Action-only cannot use the clip at all.

Chapter 2: Targets, Horizons, and Tokens

Before any architecture, pin down the contract. At one decision moment, what exactly goes into ModAR, and what exactly comes out? If you cannot write both down with shapes, you cannot build it. This chapter writes both down, using the paper's own notation and its own implementation numbers.

What the model is given

The paper's goal is "to jointly model future multimodal observations and robot actions conditioned on the current observation, robot configuration, and task label." Each of those three conditioning pieces has a symbol.

Here t is the decision time, the control step at which the robot stops and plans. o_t is the multimodal observation at that moment: what the single camera sees, in the modalities the model uses. q_t is the robot's proprioceptive configuration, its sense of its own body: "joint positions and gripper opening." g is "the learned embedding associated with a discrete task label," a trainable vector looked up from a task ID such as stack bowls.

Two details are worth underlining now, because they come back as limitations. There is one camera, not several: models "condition on a single-camera 168×224 observation." And the task is a discrete label, not a sentence: this is not a language-conditioned model. The paper lists that second point among its limitations.

What the model must predict: futures

For every modality m in the set M = {rgb, depth, dino, tracks}, the model predicts a short list of future snapshots.

Read it symbol by symbol. y^m_t+Δ is one future frame of modality m, taken Δ control steps after the decision. Δ is the dynamics stride: how far apart, in control steps, consecutive predicted frames sit. J is how many future frames are predicted. H is the prediction horizon, the last predicted moment, and it equals J times Δ by construction.

What the model must predict: actions

Actions are predicted differently. Not every Δ steps, but at every single control step up to the horizon.

a_t is the command sent to the robot at step t. A_t is a block of H consecutive commands, called an action chunk: rather than choosing one command, looking again, and choosing the next, the policy commits to a whole short sequence at once (the idea popularized by ACT). The paper spells out how the two timelines line up: "the final prediction target corresponds to the next decision point t+H, while A_t contains the H actions executed between t and t+H."

In other words, the model imagines where the world will be at the moment it next gets to think, and it chooses the commands that fill the gap.

Worked example 1: the timeline, with the paper's numbers

The implementation section gives the values: "horizon H = 16 and dynamics stride Δ = 8, yielding J = 2 sparse targets at t+8 and t+16, together with a dense H-step action chunk; policies replan every H steps." Let us check every piece of that sentence, with nothing skipped.

Step 7 is our arithmetic, but it matters later: ModAR's measured latency (147.9 ms per query) is paid once per chunk, not once per control step. The paper does not state the control frequency, so we will not convert that into a duty cycle.

One more fact about the actions: they are absolute joint configurations, not changes. Each a_t says "put the joints here," which is also the form of q_t. So the conditioning and the targets live in the same 14-dimensional space.

Turning each modality into tokens

A transformer consumes a sequence of vectors, called tokens, all of the same width. Four very different kinds of data have to become that one kind of thing. The paper's "Modality tokenization" paragraph does it in five moves.

Two terms need definitions. To patchify is to chop an image into non-overlapping squares and treat each square as one token, the same trick a Vision Transformer uses. A point track is the path of one physical point through a video: where it is in each frame, and whether it is visible or hidden behind something. CoTracker3 is the tracker that produces those paths here.

Notice the neat alignment. The paper puts RGB, depth and the track queries on one 12×16 grid, with the track queries at the patch centers. DINOv2 ViT-S/14 also works in 14×14 patches. If it is run on the same 168×224 view (our inference; the paper does not state DINO's input size), it too yields a 12×16 grid. In that case all four visual modalities describe the same 192 locations, and token (row 5, column 9) means the same patch of table in every modality. The token counts below assume that alignment.

Worked example 2: counting every token

The paper gives the grid and the horizon. The counts below are our arithmetic from those numbers. The one place where we have to read between the lines is the exact layout of a track token; the paper says it encodes displacement and visibility, and we read that as two displacement numbers plus one visibility number.

Step 9 is the quiet argument of the whole paper, in numbers. Future RGB is 225,792 raw values to get right, about 196 times the size of the track future (225,792 / 1,152 = 196). The tracks, which Chapter 7 will show are the most costly modality to remove, are the smallest target of all. Size and usefulness are not the same thing.

Once projected, every token has the same width, 384 (the shared transformer width you will meet in Chapter 3). If each projection is a single linear layer with a bias, as "we linearly project" suggests, then the RGB projection alone holds 588 × 384 + 384 = 225,792 + 384 = 226,176 parameters, and the track projection only 3 × 384 + 384 = 1,152 + 384 = 1,536.

Try the Flex-π timing and watch the visual token budget double while the action chunk stays at 16. Halving the stride doubles J, and every visual stream grows with J. This is one reason the future side of a WAM, not the action side, dominates its compute.

Position: axial rotary embeddings

A transformer is blind to order unless you tell it where each token sits. ModAR tells it with rotary position embeddings (RoPE): before attention compares a query with a key, both are rotated by an angle proportional to their position. Because the rotations compose, the similarity between two tokens ends up depending on how far apart they are, not on where they both are. (The RoFormer lesson derives this in full.)

Here is the smallest possible example, with illustrative numbers. Take a 2-number slice of a query, (1, 0), and a frequency of 0.5 radians per time step. A token at time 1 is rotated by 1 × 0.5 = 0.5 rad. A token at time 2 is rotated by 2 × 0.5 = 1.0 rad. Their dot product is cos(1.0 − 0.5) = cos(0.5) = 0.8776. Shift both to times 5 and 6 and the angles become 2.5 and 3.0 rad, but the dot product is still cos(3.0 − 2.5) = cos(0.5) = 0.8776. Only the gap survives.

Axial RoPE applies that idea along several axes at once by splitting each attention head's dimensions into groups, one group per axis. For ModAR's visual tokens the axes are time, row and column, so a token at (future frame 2, row 5, column 9) is rotated by three independent sets of angles. Action tokens only have a time axis. The paper does not say how the dimensions are divided among the axes; the idea is what matters here.

RoPE says where a token is. It does not say what kind of token it is: a DINO token and a depth token at the same (time, row, column) get identical rotations. That is the job of the learned modality embedding, a trainable vector per modality added to every token of that modality, like a colored tag on each card in the deck.

Frozen versus computed offline versus trained

Three different things happen to the three "helper" networks, and it is worth being precise.

On the real robot, a fixed third-person ZED stereo camera "provides RGB observations and stereo depth for both robot and actionless human demonstrations." So depth there is measured by stereo matching rather than rendered. A stereo estimate has its own errors, and whatever errors it has become part of the depth targets; the paper does not quantify them.

What the tokens do when the world misbehaves

Each representation degrades differently when the input gets hard, and the paper's design already encodes one of those cases.

Occlusion. When a gripper passes in front of a tracked point, the point's position becomes unknowable from the image. That is exactly why the track token carries a visibility value alongside its displacement: the model can say "this point went behind something" instead of being forced to invent a position.

Lighting and texture changes. These rewrite the RGB future, as Sim 1 showed, while the paper describes DINO features as "robust to appearance variation" and tracks as encoding motion "independently of appearance."

A single viewpoint. With one camera, depth is the only modality that carries distance explicitly. That is the paper's stated reason depth helps: it "makes scene geometry explicit and provides direct supervision for the spatial reasoning required for predicting 3D actions."

The tokenizer, as code

Here is the whole tokenization for one training example as pseudo-PyTorch, with every shape annotated. The shapes follow the paper's implementation numbers; the function names, and the 3-number track layout, are ours.

# One example, batch B. Shapes from the paper: 168x224 image, 12x16 grid of 14x14 patches,
# J = 2 future frames (t+8, t+16), H = 16 actions, 14-D joint configurations.
obs_rgb    # (B, 3, 168, 224)          current camera image
obs_depth  # (B, 1, 168, 224)          current depth
fut_rgb    # (B, 2, 3, 168, 224)       frames at t+8 and t+16
fut_depth  # (B, 2, 1, 168, 224)
q          # (B, 14)                   joint positions + gripper opening, both arms
task_id    # (B,)                      discrete task label
actions    # (B, 16, 14)               absolute joint targets; absent for actionless data

def patchify(x, p=14):                       # (B, T, C, 168, 224) -> (B, T, 12, 16, C*p*p)
    B, T, C, Hh, Ww = x.shape
    x = x.reshape(B, T, C, Hh // p, p, Ww // p, p)
    return x.permute(0, 1, 3, 5, 2, 4, 6).reshape(B, T, Hh // p, Ww // p, C * p * p)

dino = load_dinov2_vits14().eval().requires_grad_(False)   # FROZEN target extractor
with torch.no_grad():
    fut_dino = dino.patch_tokens(fut_rgb.flatten(0, 1)).view(B, 2, 12, 16, 384)

# Tracks: CoTracker3 follows the 192 patch-centre queries through the demo video.
disp, vis = cotracker3_targets(video, queries=patch_centres(12, 16), times=[8, 16])
fut_tracks = torch.cat([disp, vis], dim=-1)          # (B, 2, 12, 16, 3): dx, dy, visible

D = 384                                                  # common token width
proj = nn.ModuleDict({"rgb": nn.Linear(588, D), "depth": nn.Linear(196, D),
                      "dino": nn.Linear(384, D), "tracks": nn.Linear(3, D),
                      "act": nn.Linear(14, D)})
mod_emb = nn.Embedding(5, D)                             # learned "which modality" tag

def visual_tokens(name, grid):                         # grid: (B, 2, 12, 16, c)
    tok = proj[name](grid) + mod_emb(ID[name])          # (B, 2, 12, 16, 384)
    pos = axial_ids(t=2, rows=12, cols=16)             # (384, 3): (time, row, col) for RoPE
    return tok.flatten(1, 3), pos                          # (B, 384, 384)

tok_rgb, pos_rgb = visual_tokens("rgb", patchify(fut_rgb))
tok_act = proj["act"](actions) + mod_emb(ID["act"])       # (B, 16, 384)
pos_act = torch.arange(16)                                 # time-only RoPE for actions

A single track token, made concrete

Take the patch at row 5, column 9. Its pixel square spans y = 70 to 83 and x = 126 to 139, so its center, where the track query sits, is at

Now suppose (illustrative numbers) CoTracker3 reports that the physical point under that query is at x = 151, y = 70 at t+8 and is visible, and at t+16 it has passed behind the gripper. The two track tokens for that query would then carry:

Two honest gaps: the paper does not say whether displacements are normalized (for example divided by the image size) and does not say what displacement value is stored for an occluded point. A reimplementation has to choose. What the paper does fix is the meaning: displacement "from its initial position" plus "its visibility."

What one training example contains, by data source

Chapter 3: One Trunk, Five Specialists

You now have five kinds of tokens: tracks, DINO, depth, RGB and actions, all 384 numbers wide. They need to talk to each other, because the whole point is that tracks inform DINO, DINO informs depth, and all of them inform the action. But they also need room to be themselves, because a depth patch and a joint command are very different objects. ModAR's backbone is designed around exactly that tension.

The newsroom analogy

Picture a newsroom putting out tomorrow's paper. First, everyone sits in one editorial meeting: the sports desk hears what politics is running, the photo desk hears what the lead story needs. Then each desk goes back to its own room and writes its own section, in its own style, without the other desks talking over it. Finally each section goes to its own layout template.

ModAR's network is that newsroom. The shared meeting is a stack of transformer blocks every stream attends through. The desks are small modality-specific expert stacks, where attention is restricted to one stream. The layout templates are per-modality linear output heads. The paper's one-sentence summary: "cross-modal information is fused in the shared blocks before within-modality specialization in the expert blocks."

The diffusion transformer, briefly

The backbone is a diffusion transformer (DiT), the architecture Peebles and Xie introduced for image diffusion (the DiT gleam builds it from zero). A DiT is an ordinary transformer that takes noisy tokens in and predicts something about their clean version, with the noise level injected into every layer. Nothing in it is specific to images, which is why it transfers so easily to tracks, depth and actions.

Each transformer block does two things, each wrapped in a normalization and a residual connection. Self-attention lets every token gather information from the tokens it is allowed to see: each token emits a query, every visible token offers a key and a value, and the token receives a weighted mix of values, weighted by how well its query matches each key. Then an MLP (a small two-layer network) transforms each token on its own.

The paper's exact backbone

The implementation section lists it in one sentence, which we unpack line by line below. "The shared DiT contains six width-384 transformer blocks with six attention heads (ViT-S), followed by two width-384 expert blocks for DINO, depth, and RGB, one width-128 block for tracks, and two width-128 blocks for actions."

We read "two width-384 expert blocks for DINO, depth, and RGB" as two blocks per modality, since the paper describes the expert stacks as modality-specific. With six heads on a width of 384, each head works in 384 / 6 = 64 dimensions.

Why are the tracks and action experts narrower? The paper does not say, but the token sizes from Chapter 2 suggest a reason: a track token starts life as about three numbers and an action token as fourteen. A 384-wide specialist for a 3-number target would be mostly empty space. Narrow experts spend parameters where the targets actually have detail.

"Causally available" in the shared-trunk row is doing a lot of work. It means a stream can only attend to what would exist at that point of generation: the current observation, the finished earlier modalities, and itself. Chapter 4 turns that phrase into an exact attention mask.

Conditioning every layer: adaLN

The network also has to know three things that are not tokens: the robot's joint configuration q_t, the task embedding g, and how noisy each stream currently is (its flow timestep τ). The paper feeds all three through adaLN: "The robot configuration, task embedding, and flow timesteps for each modality condition the transformer through adaLN."

Start with plain layer normalization. Given a token vector x, subtract its mean, divide by its standard deviation, then multiply by a learned scale γ and add a learned shift β.

In plain layer norm, γ and β are fixed learned constants. In adaptive layer norm (adaLN), a small network computes γ and β from the conditioning. The analogy: a mixing engineer's fader. The song (the token) is the same, but the fader position (γ, β) is set by who is listening, how noisy the room is, and which song this is.

Worked example: adaLN on a 4-number token

The numbers here are illustrative. The point is to see one normalization and two different conditionings, with every step written out.

Same token, same weights, different behavior. In this toy, the conditioning turns the response down for the noisy stream and up for the clean one; a trained adaLN learns whatever mapping helps, and the paper does not describe what its learned mapping looks like. The general point stands either way: this is how one set of weights can serve every noise level and every task.

One subtlety deserves a flag. ModAR's streams are at different noise levels in the same forward pass (finished context is clean, the block being generated is noisy). "Flow timesteps for each modality" suggests the timestep part of the conditioning is applied per stream, so tokens of different streams get different γ and β in the same layer. That is our reading of the sentence; the paper does not show the adaLN wiring. The robot configuration and task embedding are described as "global adaLN conditioning to each DiT layer."

Worked example 3: where do 30.1 million parameters live?

First, which model does that number describe? The paper gives 30.1M once, for one specific variant: "our trained-from-scratch tracks–DINO–depth ModAR variant," the one it compares with Flex-π. That variant predicts no RGB, so it has no RGB expert. The paper does not report a parameter count for the four-modality model.

The paper gives the total and the block layout. It does not itemize the rest. So here is a back-of-envelope budget for the tracks–DINO–depth variant, using the standard rule of thumb that one transformer block with a 4× MLP holds about 12d² parameters (4d² in the four attention projections, 8d² in the MLP), ignoring biases and norms. The 4× MLP ratio is our assumption (it is the ViT-S standard); the paper does not state it.

What could fill that 11.8M? Some pieces we can size exactly from Chapter 2, and the next worked example does. The rest belongs to parts the paper does not size: the adaLN conditioning networks, the task-embedding table, and whatever connects the width-128 experts to the width-384 trunk.

The leftover is large enough to be interesting. The original DiT computes a separate scale-and-shift regression in every block; at full width that costs roughly 6d² per block. For this variant's ten width-384 blocks that is 10 × 6 × 147,456 = 8,847,360, and for its three width-128 blocks 3 × 6 × 16,384 = 294,912, together 8,847,360 + 294,912 = 9,142,272, about 9.1M. That fits inside the 11.8M, so the budget is at least consistent with a DiT-style per-block adaLN regression. It does not prove one: the paper does not show the wiring, and the lesson will not pretend otherwise.

Worked example: the projections and heads, sized exactly

Under the single-linear-layer reading of "we linearly project each modality" and "a linear output head," and assuming the track and action heads read from their width-128 experts, the in and out layers can be counted exactly (weights plus biases; our arithmetic). The table lists all five streams for reference; the sums below it are for the tracks–DINO–depth variant, which has no RGB head.

So most of the unexplained 11.4M must sit in the conditioning pathway and the adapters, the two parts the paper describes only in words. If you reimplement the tracks–DINO–depth ModAR and land far from 30.1M, those are the places to look.

The four-modality model is larger (our estimate)

Since 30.1M belongs to the tracks–DINO–depth variant, the four-modality model of Tables I and II must be bigger: adding an RGB target adds an RGB expert and an RGB output head. The paper does not report that model's size, but the same rule of thumb estimates the difference.

The real-world WAMs "observe and predict only tracks, DINO, and depth," the same modality set as the 30.1M variant. So the real-robot model is plausibly close to 30.1M too, though the paper does not report its size either.

Not a mixture of experts in the router sense

"Experts" can mislead. In large language models, a mixture of experts uses a learned router that sends each token to a few of many experts. ModAR has no router. Every DINO token goes to the DINO expert, always; the assignment is fixed by modality. This is closer in spirit to the separate action expert in π₀, where a smaller set of weights is dedicated to one kind of token, than to a routed sparse model.

What is trained, and why nothing is pretrained

Every weight in the trunk, the experts, the adaLN pathway, the embeddings, the projections and the heads starts from random initialization. The only pretrained network anywhere in the pipeline is the frozen DINOv2 encoder that defines DINO targets (plus CoTracker3 producing track targets). That is not a limitation of engineering effort; it is the experimental design. With no pretrained trunk, a gap between two ModAR variants can only come from the variants.

It also means the model is small enough to be trained many times. The paper trains one multitask model per method and per data scale, for 1.2M optimizer steps each (Chapter 5). Presumably (our inference; the paper gives no reason) that cost is also why the 6B Flex-π appears in Chapter 8 as a single system-level comparison rather than across every configuration.

What the split buys when a stream goes wrong

Suppose the depth stream is currently a mess, early in its denoising. In the shared blocks, depth tokens are being generated and can read the observation, tracks and DINO, but nothing generated later can read them yet. In the depth expert blocks, the mess stays inside the depth stream. So during depth generation, no other stream's representation is being perturbed by half-finished depth. The block-causal mask (Chapter 4) guarantees the first property; the expert restriction guarantees the second.

The price of this isolation is that the only place modalities meet is the six shared blocks. If cross-modal reasoning needed more depth than that, the design would bottleneck there. The results in Chapters 6 and 7 suggest six was enough for these tasks; the paper does not ablate the trunk depth.

Tracing one DINO token through the network

Concept plus realization: here is the path a single future-DINO token takes, with shapes, while ModAR generates the DINO block at inference (four-modality setting, batch size B). The shapes of the input and output are fixed by Chapter 2; the attention sizes use our assumed 576 observation tokens.

The action stream takes the same path with different sizes: 16 tokens of 14 numbers are projected to width 384; in the shared blocks those 16 queries read 576 + 1,536 + 16 = 2,128 keys; then the width-128 action expert runs, and a head maps back to 14 numbers per step. Going from width 384 to 128 requires some projection between the trunk and the narrow experts. The paper does not describe it, so treat that part of the diagram as a necessary but unspecified adapter.

What each conditioning signal tells the network

Worked example: attention work per query, ModAR versus Unified

Sequential generation needs five times more network passes than Unified. Does it need five times more work? Count the attention scores computed per head in the shared blocks, summed over all denoising steps of one query (our arithmetic, four-modality setting, 576 observation tokens assumed, observation and finished blocks cached).

By this proxy, ModAR computes roughly half the shared-block attention of a Unified query, because each of its many passes only pushes one block's queries through the network while the rest sits in the cache. So why does the paper say sequential generation increases latency? Because 40 small passes must run one after another, while 8 larger passes can use the GPU's parallelism more fully; for a 30M-parameter model, per-pass overhead is plausibly a large share of the time. That explanation is ours. The paper's statement, and its measured 147.9 ms, are the facts.

Why not simply one bigger shared trunk?

The paper does not ablate this, so what follows is design reasoning rather than evidence. A fully shared stack forces every layer to serve every modality at once: a depth map, a feature map and a joint-command chunk all compete for the same weights all the way to the output. Experts give each target a few private layers to turn the shared, fused representation into its own peculiar output format. The block-causal mask means cross-modal fusion must happen where attention can reach earlier blocks, which is exactly the shared trunk, so the design puts its sharing where the conditioning happens and its specialization where the formats diverge.

Chapter 4: Generating in Blocks, in Order

A novelist writing a murder mystery does not write every page at once. They decide who did it first. Then the clues get written to agree with that decision. Then the red herrings, which only make sense once the real clues exist. The last chapter, the reveal, is written knowing everything. Each stage is easier because the earlier ones are settled.

ModAR generates the future the same way, and this chapter makes that precise. First with probability, where the ordering turns out to cost nothing. Then with an attention mask, which is how the ordering is enforced inside a transformer. Then with a training trick that supervises every stage in a single forward pass.

The chain rule: ordering is free

Any joint probability of two things can be split into "the first thing" times "the second thing given the first."

This is the chain rule of probability, and it holds exactly, for any variables and in either order. Nothing is approximated. So a model that generates a first and then b given a can represent exactly the same joint distribution as a model that generates both at once.

A tiny example with illustrative numbers shows why the order still matters in practice. Let M be "the cup moves this chunk" (a fact tracks would reveal) and S be "the cup's patch region shifts" (a fact DINO would reveal). Suppose the joint probabilities are p(M=1, S=1) = 0.45, p(M=1, S=0) = 0.05, p(M=0, S=1) = 0.05, p(M=0, S=0) = 0.45.

The joint is identical either way. What changes is the difficulty of each step. Once the motion is settled, predicting where the cup's patches go is nearly deterministic. That is the whole "scratchpad" intuition in five lines of arithmetic.

The paper's factorization

Now scale the idea up. Pick an ordering (m₁, …, m_K) of the modalities you predict (all four, or a subset), and write Y_t for their future targets in that order. The paper models the joint distribution of futures and actions as its Equation (1):

Symbol by symbol. p_θ is the model's distribution, with all network weights collected in θ; there is one network, used for every factor. Y_t^m_k is the future of the k-th modality in the order. Y_t^<k is shorthand for all the modalities before it, (Y^m₁, …, Y^m_k−1); for k = 1 it is empty. The product ∏ multiplies the K future factors, and the leading factor is the action, conditioned on the observation and on every predicted future.

Written out for the three-modality setting the paper uses on real robots, with the order tracks, DINO, depth:

Read the last factor as a job description. Given where things were (c), where they will be (Y), and what they will look like, output the commands that make it happen. That is the inverse dynamics model from Chapter 1, and the paper uses the same words: the final action-prediction step "acts as an inverse dynamics model (IDM), mapping the generated future observations to the actions that induce the predicted transitions."

Blocks, not tokens

Language models are autoregressive token by token: word 1, then word 2, then word 3. ModAR is autoregressive block by block: "we generate all tokens of one modality jointly before moving to the next modality." Within a block, all 384 tokens of, say, the DINO future are denoised together, in parallel, over 8 flow steps. Across blocks, the order is strict.

Why not token by token? The paper does not argue this, but the numbers make it obvious: four modalities at 384 tokens each would be 1,536 sequential generation steps, each a network pass. Block-wise generation needs 8 passes per block instead, and the within-block structure (every patch of a depth map depends on its neighbors) is handled by the denoiser, which is good at exactly that.

The order: tracks, DINO, depth, RGB

The paper fixes one order for all main experiments: tracks → DINO → depth → RGB, with actions last. The stated intuition: "this orders the targets from compact, structured representations that are easier to predict toward increasingly high-dimensional and detailed representations."

Compare that with the raw target sizes from Chapter 2: tracks 1,152 numbers, DINO 147,456, depth 75,264, RGB 225,792. Depth has fewer raw numbers than DINO but comes after it. So the order is not "smallest first" in a literal sense; it is "most structured first." A DINO future is a map of what is where, coarse and semantic; a depth future asks for metric detail on every patch.

The paper is candid about how much it tested this. It compares the chosen order with its reverse (Chapter 7), and "We leave a full systematic comparison of modality orderings to future work." Its limitations section adds that the best order "may depend on the task or specific scenario."

Inference: denoise, re-embed, move on

At test time the paper's procedure is three verbs: "we therefore denoise one block, re-embed the completed prediction as context, and then denoise the next block."

Re-embed means the finished prediction is treated like an input: it goes through the same linear projection and modality embedding as a clean target would, and becomes a context block. Every later block attends to that context.

The paper adds a speed trick: "We cache keys and values for the current observation and all previously generated modalities to avoid recomputing them at every denoising step." A key-value (KV) cache stores each token's attention keys and values so they need not be recomputed. It is exact here, not approximate, because of the block-causal structure: an earlier block never attends to a later one, so nothing that happens during later denoising can change an earlier block's keys and values.

Worked example 4: what the cache saves

For the arithmetic we need a guess about the observation tokens. Figure 2 of the paper shows the current observation embedded from DINO, depth and RGB, so we assume 3 × 192 = 576 observation tokens in the four-modality simulation setting. Every other count comes from Chapter 2. This is our arithmetic, counting "tokens pushed through the network," which is a rough proxy for compute.

The exact wall-clock saving depends on details this proxy ignores (attention cost grows with the number of visible tokens; the experts only process their own stream). The direction is unambiguous, and it is why sequential generation is affordable at all: ModAR's measured 147.9 ms includes 40 denoising steps.

Training: every stage in one pass

Training has a problem inference does not. At inference you generate the blocks one after another. In training you have the true futures for every block, and you want to teach all K + 1 conditionals of Equation (1) at once, from one forward pass, without letting any block cheat.

The paper's solution: "we supervise every stage in one pass by creating a clean context copy and a noisy prediction copy of each target modality." Every target modality appears twice in the training sequence. The context copy holds the true future (clean, apart from the context noise of Chapter 5), playing the role a finished block plays at inference. The prediction copy holds a noised version of the same future, which the model must denoise.

Then one block-causal mask decides who reads whom. In the paper's words, it "lets the prediction copy for m_k attend only to the observation and context copies of m₁, …, m_k−1, preventing target leakage." (Each prediction copy also attends within itself, since its own tokens are denoised jointly.)

Target leakage is what happens when a model can see the answer it is graded on. If the noisy DINO copy could read the clean DINO copy, the network would learn to copy it, the loss would drop toward zero, and nothing about predicting DINO would be learned. If it could read the clean depth copy (a later modality), it would learn to rely on information that will not exist yet at inference. The mask forbids both.

What do the context copies themselves attend to? The natural reading, mirroring inference where each finished block was embedded after its predecessors, is that a context copy reads the observation, earlier context copies, and itself. The paper states the rule only for prediction copies, so treat that as our reading.

Worked example 5: one pass versus five

And beyond compute, the single pass has a quieter benefit: all K + 1 losses in Equation (4) come from the same example at the same time, so the gradient for "predict DINO from tracks" and "predict actions from everything" are computed on identical context.

Three patterns are worth finding in the matrix. The prediction-copy rows have a staircase of allowed context cells: each one reads one more context block than the row above. The context-copy columns are read by later prediction copies only, never by the prediction copy of the same modality. And the action row reads every context copy, which is Equation (1)'s last factor drawn as a mask.

The mask, as code

ORDER = ["tracks", "dino", "depth", "rgb", "act"]
SIZE  = {"obs": 576, "tracks": 384, "dino": 384, "depth": 384, "rgb": 384, "act": 16}

# the training sequence: observation, clean context copies, noisy prediction copies
blocks = [("obs", "obs")] + [("ctx", m) for m in ORDER[:-1]] \
                         + [("noisy", m) for m in ORDER]

def allowed(q, k):
    (qk, qm), (kk, km) = q, k
    if kk == "obs":   return True                  # everyone reads the current observation
    if qk == "obs":   return False                 # the observation reads only itself
    if q == k:        return True                  # a block's own tokens are denoised jointly
    if kk == "ctx":   # clean context: only STRICTLY EARLIER modalities
        return ORDER.index(km) < ORDER.index(qm)
    return False                                   # never read another block's noisy copy

def token_mask(blocks):                            # (L, L) boolean, L = 3,664 here
    spans, s = [], 0
    for b in blocks:
        spans.append((s, s + SIZE[b[1]])); s += SIZE[b[1]]
    M = torch.zeros(s, s, dtype=torch.bool)
    for i, q in enumerate(blocks):
        for j, k in enumerate(blocks):
            if allowed(q, k):
                M[spans[i][0]:spans[i][1], spans[j][0]:spans[j][1]] = True
    return M                                        # used in the SHARED blocks; experts see only their own span

Two implementation notes connect this to the rest of the model. The mask applies in the shared blocks; in the expert blocks, each stream's attention is already restricted to its own stream (and, for context versus prediction copies, the same no-leak rule must hold there too). For an actionless example, the action block has no target, so the action loss term is simply dropped, as the paper specifies.

Two sequences, one set of weights

There is a quiet correctness property hiding in this chapter, and it is worth stating plainly. At inference, the DINO block is computed while reading exactly {observation, finished tracks}. In training, the noisy DINO copy is computed while reading exactly {observation, tracks context copy}. Same set of blocks, same attention pattern, same weights. The only differences are that training context is a (lightly noised) ground truth instead of a generation, and that training handles every stage in one pass.

This is the block-level version of teacher forcing, the standard trick for training autoregressive models: during training, feed the model the true previous outputs rather than its own guesses, so every step can be supervised in parallel. Teacher forcing has a well-known weakness: at test time the model sees its own imperfect outputs, which it never practiced on. Context noise (Chapter 5) is ModAR's answer to exactly that weakness. The mask makes training and inference structurally identical; context noise makes them statistically closer.

One training example, stage by stage

Put the mask and Equation (1) side by side for a single four-modality training example. Every row below is computed in the same forward pass.

For an actionless example, the last row simply contributes no loss. Everything above it still trains, which is how a human video teaches "given this motion, what will the features and depth look like?" without ever saying anything about joint commands.

Notice what the table implies for a labeled example: all five rows train at once, and each reads a different amount of context. The tracks row learns to imagine from the observation alone, the hardest possible starting point, while the action row sees the most. If you mask the table differently (for example, letting the depth row read nothing but the observation), you are no longer training Equation (1), and the inference loop, which always hands depth the finished tracks and DINO, would meet inputs the network never learned to use.

The reverse order, written out

The ablation in Chapter 7 reverses the future order to RGB → depth → DINO → tracks, with actions still last. Its factorization is equally exact:

The difference is the first job. In the forward order, the first block to be generated from nothing but the observation is the compact track field (1,152 raw numbers). In the reverse order, it is a full RGB future (225,792 raw numbers), with every lighting and texture detail, and every later block conditions on whatever mistakes that hardest-first step made. Same joint distribution on paper; very different sequence of difficulties in practice. The measured cost is 10 points (75% to 65%).

Chapter 5: The Flow Objective and Context Noise

Chapter 4 said each block is "denoised over 8 flow steps" and moved on. This chapter opens that box. By the end you will be able to write ModAR's training loss, its sampler, and its defense against cascading errors, with every symbol accounted for.

Mixing noise into a target: the interpolant

Take one future modality's true target Y (say, the DINO future: 384 tokens of 384 numbers). Draw a same-shaped block of Gaussian noise ε. Pick a mixing level τ between 0 and 1. The paper forms the noisy version with a straight-line blend, its Equation (2):

Y_t^m is the clean target of modality m. ε^m is noise drawn from a standard normal, N(0, I), independently for each modality. τ_m is the flow timestep, also drawn independently per modality in ModAR. Ỹ_t^m (read "Y tilde") is the noisy input the network sees. At τ = 1 it is the clean target; at τ = 0 it is pure noise. This straight-line path is the linear flow-matching interpolant of Lipman et al. (the flow matching gleam derives it).

One number, to make it concrete (illustrative): a target value Y = 0.8, a noise draw ε = −1.2, and τ = 0.25 give Ỹ = 0.25 × 0.8 + 0.75 × (−1.2) = 0.2 − 0.9 = −0.7. Three quarters noise, one quarter signal.

Three things the network could predict

Given Ỹ and τ, a denoiser can be trained to output one of three equivalent-looking quantities.

Given Ỹ and τ, any one of the three determines the other two, so they look interchangeable. They are not interchangeable for learning. ModAR trains "every output stream with a JiT-style x-prediction objective": the output head directly predicts the clean target, written Ŷ_t^m (read "Y hat"). JiT is the paper "Back to Basics: Let Denoising Generative Models Denoise" by Li and He.

The paper gives both an observation and a reason. The observation: "replacing x-prediction with velocity (v) prediction is often unstable and can cause training to diverge." The reason: "Velocity prediction can struggle in high-dimensional spaces; predicting the clean sample is well-conditioned when the data lie on a low-dimensional manifold, as with images, depth, and other visual representations."

Unpack "low-dimensional manifold" with an analogy. Every possible 384-token DINO map is a point in an enormous space, but the maps that real scenes produce occupy a thin, structured sheet inside it, the way all real faces occupy a tiny corner of the space of all pixel grids. A clean target always lies on that sheet, so predicting it asks the network to land on something structured. Noise and velocity are spread across the whole space, so predicting them asks the network to reproduce full-dimensional randomness. The paper applies the same x-prediction parameterization to robot actions, citing prior VLA and WAM work that does likewise.

From a clean guess to a velocity

A sampler needs a velocity to move along, but the network outputs a clean guess. The paper converts: v_θ = (Ŷ − Ỹ) / (1 − τ). Here is where that comes from, in three lines.

The loss, and why it has that odd denominator

For each supervised modality the paper minimizes its Equation (3):

E is the average over training examples, noise draws and timesteps. The numerator is the squared error of the clean guess. The denominator divides by (1 − τ) squared, except that it never lets that quantity fall below δ_m, which the paper says "stabilizes the loss near the clean endpoint." The paper sets δ_m = δ_act = 0.05.

Why divide at all? Because that denominator turns the clean-guess error into a velocity error. Here is the derivation (ours, but it follows directly from the paper's two formulas).

So the network outputs a clean guess (the well-conditioned target), but it is graded in velocity units, which is what the sampler actually uses. That combination, x-prediction with a velocity-space loss, is the JiT recipe the paper cites. The δ clamp exists because as τ approaches 1 the factor 1/(1 − τ)² explodes, and tiny errors on almost-clean inputs would dominate every batch.

Worked example 6: how the weight moves with τ

Near the clean end, the loss insists on precision; near the noisy end, it forgives. That matches intuition: when the input is almost clean, a good model has no excuse for being wrong.

The total loss

Each modality's loss and the action loss are simply added, the paper's Equation (4), with unit weights:

L_act has the same form as Equation (3), with the action chunk A_t, its prediction, its own timestep τ_act and δ_act in place of the modality versions. The action term is "include[d] only for examples with action labels," which is how actionless human or simulated demonstrations contribute: they supply every future term and nothing else.

Where on the path to practice: logit-normal timesteps

Training needs a τ for every stream in every example. Uniform sampling would spend equal effort everywhere on the path. ModAR instead samples logit-normal timesteps: draw n from a normal distribution N(μ, σ), then squash it, τ = 1 / (1 + e⁻ⁿ). μ slides where the practice concentrates; σ sets how spread out it is. The paper uses a different (μ, σ) for each stream.

The last two columns are our arithmetic. One of them in full: sigmoid(−2) = 1 / (1 + e²) = 1 / (1 + 7.389) = 1 / 8.389 = 0.1192. So half of all DINO training examples sit at τ below 0.12: the DINO stream practices mostly in the noisy regime, where the big structural decisions happen. Tracks practice centered on the middle of the path, and depth's larger σ spreads its practice widely. The paper reports these settings without explaining how they were chosen, so we will not invent a rationale.

For the baselines: "Unified uses one shared flow timestep with (μ, σ) = (−1, 1); Independent-noise samples each stream's modality-specific distribution independently." That second sentence is the concrete version of the mismatch from Chapter 1: at inference all streams share one τ, but in training a DINO stream near 0.12 and a tracks stream near 0.5 are the typical case.

The sampler: eight Euler steps per stream

At inference, each stream starts as pure noise at τ = 0 and is integrated to τ = 1 "using the velocity implied by its clean prediction." Euler integration is the simplest way to follow a velocity: take the current point, add velocity times step size, repeat. With 8 steps the step size is 1/8 = 0.125:

Worked example 7: one value, eight steps

Illustrative numbers: the true clean value is 0.8 and the starting noise is −1.2. First, a perfect network (it always guesses 0.8).

A perfect x-predictor walks a straight line at constant speed, which is what "linear interpolant" promised. Now an imperfect network whose guess is biased high by 0.3 × (1 − τ), a toy error that shrinks as the input gets cleaner.

Look at the last row. With evenly spaced steps, the final Euler step multiplies (Ŷ − y) by 0.125 / (1 − 0.875) = 1, so the output lands exactly on the network's last clean guess, 0.8375. The final error (0.0375) is whatever the network gets wrong at τ = 0.875, which is precisely where Equation (3) weights errors 64 times more heavily than at τ = 0 (1 / 0.125² = 64). The loss and the sampler are pulling in the same direction.

Cascading error, and the context-noise fix

Now the danger from Chapter 1, in this chapter's language. During training, the context copies are the true futures. During inference, the context blocks are the model's own generations, carrying errors like the 0.0375 above. A model trained only on perfect context has never seen imperfect context, so it may trust a slightly wrong track block completely and build a wrong DINO block on top of it, then a wrong depth block, then wrong actions. The paper: "imperfections in earlier generations become inputs to all later predictions."

The fix, "following Latent Forcing," is to make the training context imperfect on purpose. When predicting m_k, for every context block m_j and every training example, independently sample

U(1 − β, 1) is a uniform draw between 1 − β and 1. β is the maximum context-noise level; the paper sets β = 0.5, so every context block is somewhere between half noise (τ = 0.5) and perfectly clean (τ = 1), with an average mixing level of 0.75. This noise is applied "only during training, not during inference."

The optimization recipe

Everything above is trained with a conventional, fully specified recipe. Every item in this table is from the paper's implementation section; the right column's arithmetic is ours.

An EMA (exponential moving average) keeps a slowly updated copy of the weights, each step moving it 0.1% toward the live weights; evaluating the EMA copy smooths out step-to-step noise. The equal-sized batch rule matters more than it looks: at D = 1,250, actionless demonstrations outnumber labeled ones 1,200 to 50, yet each optimizer step still sees half labeled data, so the action loss is never starved.

Training and sampling, as code

def logit_normal(mu, sigma, n):
    return torch.sigmoid(mu + sigma * torch.randn(n))

TS    = {"dino": (-2, 1), "tracks": (0, 1), "depth": (-1, 1.6), "rgb": (-1, 1), "act": (-1, 1)}
BETA, DELTA = 0.5, 0.05

def training_step(batch):
    B = batch.size
    ctx, noisy, taus = {}, {}, {}
    for m in ORDER:                                          # tracks, dino, depth, rgb, act
        Y = batch.target[m]                                    # (B, N_m, c_m) clean future
        if m != "act":                                          # context copy + CONTEXT NOISE
            t_c = torch.empty(B, 1, 1).uniform_(1 - BETA, 1)
            ctx[m] = t_c * Y + (1 - t_c) * torch.randn_like(Y)
        tau = logit_normal(*TS[m], B).view(B, 1, 1)          # independent per stream
        noisy[m] = tau * Y + (1 - tau) * torch.randn_like(Y)        # Eq. (2)
        taus[m] = tau

    x_hat = net(batch.obs, batch.q, batch.task, ctx, noisy, taus,
                mask=BLOCK_CAUSAL)                                 # one pass, every stage
    loss = 0.0
    for m in ORDER:
        w = 1.0 / (1 - taus[m]).clamp_min(DELTA) ** 2              # Eq. (3) weight
        err = ((x_hat[m] - batch.target[m]) ** 2).sum((1, 2))
        per_ex = w.view(-1) * err
        if m == "act":
            per_ex = per_ex * batch.has_actions                    # actionless rows: no action term
        loss = loss + per_ex.mean()                              # Eq. (4), unit weights
    return loss

@torch.no_grad()
def act(obs, q, task, steps=8):
    cache = net.encode_context(obs, q, task)                    # KV for the observation
    for m in ORDER:
        y = torch.randn(SHAPE[m])                                   # tau = 0: pure noise
        for k in range(steps):
            tau = k / steps
            x_hat = net.denoise(m, y, tau, cache)                   # reads cached earlier blocks
            y = y + (x_hat - y) / (1 - tau) / steps              # Euler on v = (x_hat - y)/(1 - tau)
        if m == "act":
            return y                                                 # (16, 14) joint targets
        cache = net.append_context(cache, m, y)                     # re-embed, no context noise

Two details in that sketch are ours and two are the paper's. Ours: the shapes of the noise tensors and the exact cache API. The paper's: context noise is sampled independently per context block and per example, and it is switched off at inference.

Chapter 6: Formulations × Data Scale

Five formulations, three data scales, six tasks, fifty trials each. That is 5 × 3 × 6 × 50 = 4,500 potential evaluation episodes behind one figure; Action-only exists only at the smallest scale, which removes 2 × 6 × 50 = 600 of them, leaving 3,900. This chapter reads that figure, its per-task table, and the two control experiments the authors ran to defend it against the obvious objections.

The simulation testbed

The benchmark is RoboTwin 2.0, a simulated bimanual manipulation benchmark with strong domain randomization. The paper picks "six representative RoboTwin tasks": dump bin, pick bottles, place bread, put bottles, stack bowls, and turn switch.

The data design is the clever part. Each task uses D ∈ {50, 250, 1250} total training demonstrations. Exactly 50 of them keep their action labels. The remaining D − 50 are used as actionless demonstrations: their actions are withheld, so they can only supervise futures. The number of action-labeled demonstrations therefore never changes. Only the amount of "video-like" data grows.

"For each method and data scale, we train one multitask model for all six tasks," and every model is evaluated "on 50 held-out initial conditions per task." So each overall number is an average over 6 × 50 = 300 episodes. Because the labeled count is fixed, any improvement from D = 50 to D = 1,250 is attributable to actionless data. Action-only cannot use actionless data, so it has a result only at D = 50 (the figure draws it as a flat dashed line at 0.46).

Table I, in full

Success rates by formulation and scale, copied from the paper. Bold marks the highest value in each row.

Notice what the paper claims and what it does not. It claims ModAR "achieves the highest average success rate at every data scale," which the overall rows confirm. It does not claim ModAR wins every task: Unified is ahead on pick bottles at 50 and 1,250 and on stack bowls at 250. The biggest ModAR margins are on place bread (0.74 vs 0.50 at 250) and put bottles (0.82 vs 0.62 at 250).

Worked example 9: rebuilding the overall row

The overall number is the plain mean of the six task numbers. Checking two cells by hand confirms the table is internally consistent and shows how much each task moves the average.

That last line is the kind of reading the paper's summary sentence hides. ModAR's average advantage is real, but it is concentrated in tasks where, plausibly, a precise sense of where objects will end up matters most; the paper itself makes the general point that "the best modality set naturally varies across tasks."

Reading the controls as a skeptic

A good habit with any controlled study is to list every difference between the compared systems and check which ones were equalized. Here is that list for ModAR versus Unified (the controlled items are from the paper; the "not separately tested" items are our observations).

The untested row is worth noticing without over-weighting: per-modality timestep distributions are also used by Independent-noise, which performs worst, so they are clearly not sufficient on their own for good results.

Why the labeled count is held at 50

Holding the action-labeled demonstrations fixed is what makes Table I a clean test of actionless data. If labeled data grew with D, any formulation would improve simply from more action supervision, and the scaling column would mix two effects. With 50 labeled demonstrations everywhere, the only way a formulation can improve with D is by turning actionless futures into better actions, which is precisely the WAM promise under test.

One consistency check links the two tables: ModAR's row in Table I (0.66, 0.75, 0.76) is identical to the last column of Table II, because both are the same four-modality model. When two tables in a paper agree like this, you can use either as a cross-check on the other.

Finding 1: ModAR is best at every scale

Overall: 0.66, 0.75, 0.76 at D = 50, 250, 1,250, against Unified's 0.63, 0.67, 0.64. The paper's explanation is the scratchpad hypothesis: "early modalities act as scratchpads for later ones: generating coarser or easier targets first provides structured context for more detailed targets."

Finding 2: ModAR turns actionless data into success

"ModAR benefits most from additional actionless data: adding 1,200 actionless demonstrations improves the average success rate from 66% to 76% (10 percentage points), compared with a mere 1% improvement for Unified." From the table: ModAR 0.66 → 0.76 is +0.10; Unified 0.63 → 0.64 is +0.01. Most of ModAR's gain arrives by D = 250 (+0.09); the next 1,000 actionless demonstrations add one more point.

The paper reports this gain without explaining it. One plausible reading (ours, not the paper's) connects it to the IDM framing from Chapter 4. Actionless data trains the future blocks, and in ModAR the action step reads the observation, configuration and task plus every finished future block. If actionless data makes those blocks better, the action step has better evidence to work from, while Unified's action stream only ever reads partially noisy futures. Control 2 below is consistent with that reading, since it shows ModAR's futures are themselves more useful for acting, but it does not test the mechanism directly.

Finding 3: Disjoint gets worse with more data

Disjoint beats Action-only on average (0.55 vs 0.46 at D = 50), suggesting that future prediction can help even as an auxiliary loss (by the rough error bars later in this chapter, a 9-point gap on 300 episodes is about two standard errors). But then it declines: 0.55, 0.55, 0.48. The paper's hypothesis is negative transfer: "with more actionless data, the shared representation becomes increasingly shaped by the future-observation prediction objective," to the detriment of the action objective, which in Disjoint never gets to read the futures that representation was shaped for. The steepest drops are place bread (0.48 to 0.22) and dump bin (0.92 to 0.78).

Finding 4: Independent-noise collapses

Independent-noise is the worst formulation at every scale (0.37, 0.34, 0.33), below even Action-only, and it also gets worse with more data. On stack bowls it falls from 0.36 to 0.04. Recall the paper's hypothesis: independently sampled noise levels rarely match the synchronized schedule used at test time, so the model gets "insufficient training signal near its inference regime."

How rarely? Here is a toy estimate (ours, not the paper's). Call a training example "inference-like" if all five streams' timesteps fall within 0.1 of each other. For five independent uniform draws, the probability that max − min ≤ w is 5w⁴ − 4w⁵.

The paper does not quantify this, and "within 0.1" is our arbitrary threshold. But the orders of magnitude make the hypothesis concrete: the Independent-noise model almost never practices the exact situation it faces at every inference step. The training view of Sim 2 in Chapter 1 shows the same thing one draw at a time.

Control 1: is ModAR winning only because it samples longer?

ModAR uses 8 Euler steps per stream: 40 in total across four futures and actions. Unified, Independent-noise and Disjoint use 8 in total. So the paper re-evaluated each of those three at D = 250 with 40 Euler steps, "matching ModAR's total number of sampling steps."

"Additional steps do not close the gap." Unified actually gets worse with more steps. The paper does not explain that drop; a finer integration grid means the model is evaluated at τ values it saw with different frequency during training, which is at least one candidate explanation, but it is our speculation. The conclusion stands either way: ModAR's advantage is not a sampling-budget artifact.

Control 2: are ModAR's futures actually better?

The second objection is subtler. ModAR's action step reads finished futures; Unified's reads half-finished ones. Perhaps ModAR's futures are no better than Unified's, and it only wins because its action predictor has the easier job.

The test: train a separate IDM "to map ground-truth future observations to actions," with context noise on its inputs during training. Then take the best ModAR and Unified checkpoints, let each generate its futures and actions normally at every prediction step, "discard its native action prediction, pass its predicted futures to the separate IDM, and execute the resulting actions." Now both systems use the same action predictor, and the only difference is the quality of the futures they hand it.

(Values read from the bar labels of the paper's Figure 5.) "ModAR still outperforms Unified with the separate IDM, suggesting that its predicted futures are themselves more useful for predicting actions." Notice also that the gap widens with actionless data, echoing Finding 2. Both systems score a little lower with the separate IDM than with their native action heads (for ModAR, 0.69 vs 0.75 at D = 250). Our explanation, not the paper's: the native heads were trained jointly with the futures they read, while the separate IDM was trained on ground-truth futures.

How much should you trust a 3-point or 8-point gap?

The paper reports no confidence intervals, so here is a rough binomial estimate of our own, treating each of the 300 episodes as an independent coin flip (it ignores task structure and best-checkpoint selection, so read it as a scale, not a verdict).

This is exactly why the authors word the Flex-π result as "slightly higher observed," and why the formulation claim rests on consistency across three scales, two controls and a real-robot replication rather than on any single cell.

Six tasks, six stories

The overall row is an average of six quite different curves. Reading them one by one (numbers from Table I, interpretations ours) shows where each formulation's character comes from.

Dump bin is the easy task: even Action-only reaches 0.82. ModAR climbs to 0.98 at D = 1,250. Disjoint is the only formulation that drops sharply here (0.92 to 0.78), a first sign of its negative-transfer problem.

Pick bottles is Unified's task: 0.58, 0.60, 0.62 against ModAR's 0.48, 0.60, 0.58. It is the clearest case in the table where joint generation is at least as good.

Place bread is ModAR's biggest win: 0.74 and 0.72 at the larger scales, against 0.50 and 0.44 for Unified. Placing an object precisely is a plausible place for a finished geometric future to pay off, and Chapter 7 shows depth-containing models do especially well here.

Put bottles is the cleanest data-scaling story: ModAR goes 0.64, 0.82, 0.82, while Independent-noise sits at 0.20, 0.20, 0.18.

Stack bowls is where Independent-noise collapses (0.36, 0.18, 0.04) and where Unified edges ModAR at D = 250 (0.76 vs 0.68) before ModAR retakes the lead at 1,250 (0.80 vs 0.68).

Turn switch shows ModAR peaking at D = 250 (0.74) and falling back at 1,250 (0.64), a reminder that "more actionless data" is not guaranteed to help on every task even for the best formulation.

Worked example: what a collapse looks like in episodes

These are the per-task realities behind the averages: for two of the baselines, feeding in more actionless data makes the robot fail more often, while for ModAR it mostly makes the robot succeed more often.

Keep the same caution you applied to the averages. With 50 episodes per task, one rate carries a standard error of up to √(0.5 × 0.5 / 50) = √0.005 = 0.071, about 7 points. A 16-success collapse (Independent-noise on stack bowls, 18 to 2) is far outside that; a 2-point wobble on a single task is not. Read the per-task stories as patterns to check, not as verdicts.

A caution about generalizing the Independent-noise result

The paper is careful to say that Independent-noise "performs poorly in our from-scratch experiments." Keep that qualifier. Flex-π, which also samples noise levels independently per stream during training, reaches 72% on the same data in Chapter 8, starting from a pretrained video model. So the collapse in Table I is evidence about training this formulation from scratch at this scale, not a verdict on independent noise sampling in general. The train-test mismatch hypothesis may simply matter much less when the backbone already knows how to denoise video.

Figure 4(a), in words

The paper's figure draws each formulation as a colored bar at each of the three scales, with Action-only as a dashed horizontal line at 0.46. The visual story: the ModAR bars rise from left to right and are the tallest in every group; Unified is flat; Disjoint sags at the right; Independent-noise sits below the dashed line throughout. Sim 7 above redraws the same numbers as lines so the trends are easier to compare.

Chapter 7: Which Futures Matter

Chapter 6 fixed the modalities and varied the formulation. This chapter does the opposite: it fixes the formulation (ModAR) and varies what it predicts. It is the chapter that decides whether the headline "RGB is not the right thing to imagine" survives contact with data, and whether each structured modality earns its place.

Go back to the prediction you wrote at the end of Chapter 0. Which single modality gives the best policy alone at 50 demonstrations, and which one hurts most to remove? Both answers are in this chapter.

Two ways to ask "does this modality help?"

There are two natural experiments, and the paper runs both.

The first is additive: start from one modality and add others one at a time, following the generation order (tracks, then DINO, then depth, then RGB). Alongside, train a variant for each modality predicted alone. That is Figure 4(b) and Table II, run at all three data scales.

The second is leave-one-out: start from the full four-modality model and remove exactly one modality. That is part of Figure 6, run at D = 250. The two experiments can disagree, and when they do, the disagreement is informative.

Table II, in full

Every row is one task and one scale; every column is a set of predicted modalities, all trained as ModAR. Bold marks the best value in each row. The RGB column is, in the paper's words, "the typical WAM setting of predicting only future RGB and actions."

The last column equals ModAR's column in Table I (0.66, 0.75, 0.76), as it must: it is the same four-modality model.

Reading the single-modality columns

Answer to the first prediction: at D = 50, DINO is the best single modality by a wide margin (0.60, against 0.46 for depth, 0.43 for RGB and 0.41 for tracks). It stays the best single modality at 250 (0.65) and 1,250 (0.68). A semantic representation of "what is where" seems to be the most useful single thing to imagine.

RGB alone is weak: 0.43 at D = 50, which is below Action-only (0.46). Imagining the future in pixels, with this small a model and no pretraining, does not help at all at the smallest scale; it only pulls ahead of Action-only once actionless data is added (0.54, then 0.58).

Tracks alone are the weakest single modality at D = 50 (0.41), with some striking task-level failures: 0.12 on pick bottles, 0.10 on place bread. Keep that in mind for the leave-one-out result below.

Worked example 10: the value of each addition

Following the cumulative columns and subtracting neighbors (our arithmetic on the overall rows):

Three things jump out. Adding DINO to tracks is the biggest single step at every scale. Depth's contribution grows with data (0.00, 0.03, 0.05). And RGB's contribution is +0.03, +0.00, −0.01: in the paper's words, "additionally predicting future RGB on top of the first three modalities provides no consistent gain."

The paper's summary of the additive experiment: "Across data scales, performance generally holds or increases with each added modality, showing that ModAR can effectively combine the benefits of predicting multiple modalities." And it is careful about the exceptions: "the best modality set naturally varies across tasks because different features define each task, and some modalities represent those features better than others; as a result, additional modalities do not always help."

You can see those exceptions in the table. Put bottles at 1,250 peaks at 0.90 with tracks, DINO and depth, and drops to 0.82 when RGB is added. Stack bowls at 250 peaks at 0.86 with only tracks and DINO. Dump bin at 1,250 reaches 1.00 with only tracks and DINO.

Why would RGB not help?

The paper's hypothesis is short: "predicting future RGB introduces high-variance appearance details while adding little information beyond the more structured targets in our setting." Unpack the two halves.

High variance. RoboTwin 2.0's own title advertises "strong domain randomization." The paper does not say which randomization settings its six tasks used, but wherever lighting, texture or clutter varies between episodes, the RGB target is where that variation lands, and the loss charges the model for failing to guess it. Little additional information. By the time RGB is generated, the model already has motion, semantics and geometry. What RGB adds on top is mostly appearance, which, on the paper's hypothesis, adds little beyond the structured targets for these tasks. Recall from Chapter 2 that RGB is also the largest raw target (225,792 numbers per example), so it is expensive to predict and, per this table, not worth it for these tasks.

The ablations: Figure 6

Figure 6 takes the full four-modality ModAR at D = 250 (average success 0.75) and changes one thing at a time. Values are read from the figure's bar labels and match the numbers stated in the text.

Answer to the second prediction: removing tracks hurts most (0.75 to 0.61), followed by DINO (0.65), depth (0.70), and RGB (no change). The paper's conclusion: "predicting RGB contributes the least of the four modalities to success rate in this setting."

On order, the paper says reversing "lowers the average success rate to 65%, supporting our choice to generate compact, structured representations before increasingly detailed ones." On context noise: "context noise during training is critical for preventing errors from compounding across successive modalities."

Worked example: which ablations are clearly outside the noise?

Figure 6 reports six-task averages over 300 episodes each. Converting to episodes and applying the same rough binomial check as Chapter 6 (our arithmetic):

On this rough scale, removing tracks, removing context noise, removing DINO and reversing the order are all well outside run-to-run noise; removing depth is suggestive but weaker; removing RGB changes nothing. That ranking matches the paper's prose exactly, including its choice to call context noise "critical."

The disagreement, and what it suggests

Put the two experiments side by side for tracks. Alone, tracks are the weakest modality (0.53 at D = 250). Removed from the full set, they cause the largest drop (−0.14 at D = 250).

One reading, consistent with the paper's scratchpad hypothesis, is that tracks are valuable less as a final answer and more as a first draft: compact motion context that makes every later block easier. A caveat the paper does not discuss, and which you should keep: removing tracks also changes which modality goes first (DINO becomes the first block), so the w/o-tracks ablation mixes "no tracks" with "a different first block." The reverse-order ablation points the same direction (putting tracks last costs 10 points), but neither experiment isolates position from content. The paper leaves "a full systematic comparison of modality orderings to future work."

What the modality study does not say

It is worth being as careful as the paper is. The finding is that RGB gives "no consistent benefit" in this setting: small from-scratch models, six simulated tasks, a single camera. It is not a claim that RGB futures are useless in general. A model initialized from a video generator that already understands appearance might extract more from an RGB target, and the paper explicitly frames its conclusion as "a promising alternative to relying solely on future RGB prediction," not a replacement for it.

Similarly, the per-task variation is large. Anyone deploying this on a new task should expect the best subset to differ, and should budget for a small ablation of their own.

Per-task patterns worth noticing

The paper's general remark, that different tasks are defined by different features, is easy to see in specific cells (numbers from Table II; the interpretations are ours).

Put bottles at D = 50: DINO alone reaches 0.64 while depth alone reaches 0.12. With little data, knowing what is where seems far more useful than knowing how far for this task.

Place bread at D = 250: depth alone reaches 0.64 while tracks alone reach 0.26, and the best model (0.76) is the one that adds depth to tracks and DINO. Placement seems to reward explicit geometry.

Pick bottles: tracks alone go from 0.12 to 0.48 to 0.42 as data grows. Motion prediction on its own looks data-hungry.

Stack bowls at D = 1,250: DINO alone reaches 0.84, the best value in that row, above every multi-modality model.

Worked example: is "RGB hurts on put bottles" real?

At D = 1,250 on put bottles, adding RGB takes the model from 0.90 to 0.82. Tempting to conclude RGB hurts. A quick binomial check (ours) with 50 episodes per rate:

A single-task gap of about one standard error is weak evidence either way. That is exactly why the paper phrases its RGB finding as "no consistent gain" across tasks and scales, rather than "RGB hurts." The honest reading is about the pattern: +0.03, 0.00, −0.01 on average, and zero change in the leave-one-out ablation.

Gains are real, but not strictly additive

Compare DINO's value in the two experiments at D = 250. Added on top of tracks, DINO is worth +0.19 (0.53 to 0.72). Removed from the full model, it costs 0.10 (0.75 to 0.65). If modality contributions simply added up, those two numbers would match. They do not, because what a modality contributes depends on what else is already predicted; plausibly, in the full model, depth and RGB partly cover for a missing DINO block. The paper's summary that tracks, DINO and depth "provide additive gains" is best read as "each one adds something on average," not as a claim that their values sum exactly.

Figure 4(b), in words

The paper's figure groups bars by scale. Within each group: single-modality models (RGB, depth, tracks, DINO), then the cumulative sets (tracks+DINO, +depth, +RGB), with Action-only as a dashed line at 0.46. The caption singles out the RGB bar as the one that "matches the typical WAM setting of predicting only future RGB and actions." In every group, the cumulative bars stand clearly above the RGB bar.

The scratchpad hypothesis: an evidence ledger

The ledger is a fair summary of how the paper words it: the results "support" the hypothesis; they do not prove the chosen order is optimal.

Two design details of the modality study

The cumulative sets follow the generation order. The additive columns grow as tracks, then tracks + DINO, then + depth, then + RGB: the same order the model generates them in. So each cumulative model is the previous one with one more block appended at the end of the sequence, just before the actions. That keeps the experiment clean: adding a modality never changes the position of the ones already there.

The leave-one-out ablations are run at D = 250 only. Figure 6 is explicitly "with 250 total demonstrations per task and 50 action-labeled demonstrations," the scale at which the full model reaches 75%. The paper does not say why that scale was chosen; the additive study in Table II is the one that covers all three scales.

One last habit from this chapter: whenever a paper reports "modality X helps," ask measured how. Alone, added, or removed? At which data scale? Averaged over which tasks? Table II and Figure 6 answer all three, which is why their conclusions can be stated so precisely.

Chapter 8: A 6B Rival, and Real Arms

Everything so far compared ModAR with its own siblings: same size, same data, same budget. Two questions remain that a practitioner will ask immediately. Is a 30-million-parameter model trained from scratch even competitive with the large video-pretrained WAMs people actually use? And does any of this survive a real robot, real lighting, and human videos filmed by people rather than rendered by a simulator?

The rival: Flex-π

Flex-π is concurrent work by G. Yan, J. Liu, Y. Fan, L. Cai, M. Liao, J. Zhang and D. Fox, titled "A Multi-Stream World-Action Model with Compute Flexibility." It represents the pretrained-video-backbone approach Chapter 0 described, extended (as Chapter 0 also mentioned) to predict several future modalities, RGB latents, 3D pointmaps and DINO features, jointly with actions. It has about 6B parameters.

The paper fine-tunes it on the same RoboTwin data at D = 250, following Flex-π's own recipe as closely as the setting allows.

The paper's reading: "on these in-distribution tasks, a compact WAM trained from scratch can perform as well or better than a much larger video-model-initialized WAM using substantially less training compute." And, in the same paragraph, the caveat: "this is a system-level comparison rather than a controlled architectural comparison, since the models differ in scale, pretraining, target modalities, and training recipe." Chapter 6's binomial aside puts the 3-point gap at under one standard error, which is why "as well or better" is the right phrase.

One more asymmetry, which the table's "final checkpoint" note hints at: ModAR's 75% is its best evaluated checkpoint (the protocol from Chapter 5, best of twelve), while Flex-π's 72% is its final checkpoint after 30,000 steps. Picking the best of several evaluations can only raise a score, so this difference tilts a 3-point comparison toward ModAR. The paper states both protocols; it does not say how Flex-π would score under best-checkpoint selection.

Note also the phrase in-distribution. The evaluation uses held-out initial conditions of the same six tasks the models were trained on. It does not test the thing video pretraining is usually credited for, generalization to new objects, scenes and instructions. The paper's limitations section says as much.

Worked example 11: checking "about 20× fewer FLOPs"

The paper states the ratio (approximately 20×) but not its accounting. We can still check that it is plausible from numbers the paper does give. A FLOP is one floating-point operation; training compute is commonly estimated as roughly 6 × (parameters) × (tokens processed), the "6ND" rule of thumb.

So the reported ratio is the right order of magnitude, and the remaining factor is exactly what the rule of thumb cannot see: how many tokens each model processes per sample, that frozen components (VAE, text encoder, DINOv3) only cost a forward pass, and that Flex-π's action expert runs at a smaller width than its video backbone. We cannot reconstruct the paper's exact accounting, and this example does not pretend to. What it does show is the shape of the trade: ModAR trains on almost seven times more samples, but each sample costs roughly two hundred times fewer parameter-operations.

Inference speed

"On a single NVIDIA GeForce RTX 5090 GPU, end-to-end inference for ModAR takes 147.9 ms (6.76 Hz) when generating all four future-observation modalities and actions."

The paper does not report the latency of the three-modality real-robot configuration or the robot's control rate, so we stop there. The honest summary is the paper's own: sequential generation "increases inference latency relative to simultaneous or action-only generation."

The real-world setup

Three "challenging tabletop tasks," performed with bimanual YAM arms: stacking cups, folding a crumpled towel, and placing an object in a drawer and closing the drawer. For each task the authors collect three kinds of data.

A fixed third-person ZED stereo camera "provides RGB observations and stereo depth for both robot and actionless human demonstrations." EgoDex is a large-scale egocentric video dataset of human dexterous manipulation; the paper calls its demonstrations "out-of-domain" relative to the robot's setting. As decided in Chapter 7, all real-world WAMs "observe and predict only tracks, DINO, and depth." Evaluation uses "30 rollouts per task, with varied initial object poses," and again one multitask model per method and data mixture.

Two practical details the paper does not spell out, which you would need to decide in a reimplementation: what the robot-configuration input q_t is for human videos (which have no robot), and how depth is obtained for the EgoDex clips. We flag them rather than guess.

It helps to see what one human clip contributes, in the terms of Chapter 4. Its frames give an observation and targets at t+8 and t+16, so every future row it has targets for receives a loss (the paper's phrase is "the available future targets": tracks, DINO, and depth wherever depth exists for that clip). The action row receives nothing, because there is no action chunk to compare against. So every human clip is pure future supervision, and whatever it teaches has to reach the robot's arms through the futures the action step later reads.

Why vary the initial object poses across the 30 rollouts? Because a policy evaluated from one fixed start could succeed by replaying a memorized trajectory. Varying where the cups, towel or object begin forces the model to use what it sees, which is exactly where imagined motion, semantics and geometry should matter. (The paper states the protocol; the rationale is the standard one, and ours to spell out.)

Result 1: formulations on real hardware

Figure 7(a) compares Action-only (100 robot demos per task, since it cannot use actionless data) with Unified and ModAR (both with the 100 robot demos plus 200 in-domain human and 1,000 EgoDex demos). Per-task values are read from the figure's bar labels.

"ModAR achieves the highest success rate on all three tasks," and the ordering matches simulation: ModAR above Unified above Action-only.

Result 2: learning from human videos

Figure 7(b) trains three separate ModAR models: robot demos only; plus the 200 in-domain human demos; plus those and the 1,000 EgoDex demos. Human data supervises "future-observation predictions, but not action prediction."

The paper's conclusion: ModAR "can benefit from actionless data collected across embodiments and domains."

Worked example 12: from rates back to rollouts

With 30 rollouts per task, every rate is a count out of 30. Recovering the counts (our arithmetic) makes the results tangible and checks the overall percentages exactly.

Two comparisons across the panels are worth making (both numbers are from the paper, the juxtaposition is ours). First, the like-for-like data comparison with Action-only: both trained on robot demos only, ModAR reaches 70.0% against 52.2%, a gap of 17.8 points without any human data at all. Second, ModAR without human data (70.0%) already edges out Unified with all of it (66.7%).

And the diminishing return from EgoDex (+2 successes from 1,000 out-of-domain demos, versus +10 from 200 in-domain ones) is a useful calibration for anyone planning data collection. With 90 trials per configuration, a 2-success difference is well within noise, so the fair reading is "in-domain human video helped clearly; out-of-domain video did not hurt and may have helped a little."

Why the real-robot model does not even observe RGB patches

The paper's wording is precise: the real-world WAMs "observe and predict only tracks, DINO, and depth." So RGB is dropped on both sides, as a target and as an observation stream. The camera image is still used, because DINO features are computed from it by the frozen encoder; it simply never enters the transformer as raw patches. This saves the largest input projection and 192 observation tokens per query, and it follows directly from the simulation finding that RGB added no consistent benefit "while increasing training and generation costs."

A scale marker for training compute

The paper reports only the ratio (~20×), not absolute FLOPs. For a feel of the magnitude, suppose (our assumption) that the compared ModAR variant processes about 3,000 tokens per training example. Then the 6ND rule gives:

Both absolute figures rest on our token assumption and the crude rule; only the ratio is the paper's.

A latency sketch for the real-robot configuration

The paper reports 147.9 ms for 40 passes (four futures plus actions). The real-robot model runs 32 passes (three futures plus actions). A crude proportional estimate, ours, and ignoring that the dropped RGB block is one of the larger ones:

Treat that as a ballpark only; the paper measured one configuration on one GPU, and it lists the added latency of sequential generation among its limitations.

What would change the real-world conclusion

Three things the study does not rule out, worth keeping in mind (ours): the ranking could shift on tasks where appearance itself matters (reading a label, sorting by color), since the real-robot models do not predict RGB at all; a larger number of rollouts could shrink or widen the Unified gap, which is about 2.6 standard errors on our rough scale; and the human-video benefit was measured only for ModAR, so it is unknown how much Unified would gain from the same videos in isolation.

What the real-world study does and does not show

It shows that the simulation ranking of formulations reproduces on hardware, on three contact-rich bimanual tasks. It shows that in-domain human video clearly improves ModAR (70.0% to 81.1%). And it shows that adding 1,000 out-of-domain EgoDex clips per task gave a further 2 points (83.3%), which the paper reads as benefit "across embodiments and domains" and which, by our rough estimate below, is within noise.

It does not show generalization to unseen tasks or objects (three tasks, discrete labels, varied initial poses only), and each configuration rests on 90 rollouts. The paper does not report how Unified or Action-only would fare with human data ablated in the same way, beyond the configurations listed. These are exactly the limits Chapter 9 collects.

Per task: where ModAR's real-world margin comes from

Using the recovered counts (out of 30 rollouts per task), the gaps become concrete.

Stacking cups is where Unified struggles most and ModAR's margin is largest. And the human-video gains per task, from Figure 7(b): fold towel 20 → 24 → 25, place in drawer 22 → 24 → 25, stack cups 21 → 25 → 25. In-domain human video helps every task; EgoDex adds at most one more success on any task.

Worked example: how solid are the real-world gaps?

A rough binomial check with 90 rollouts per configuration (our estimate, treating rollouts as independent):

So, on this rough scale: ModAR over Action-only is very clear, ModAR over Unified is reasonably clear, and the EgoDex increment is indistinguishable from noise. The paper's wording matches: the formulation result "mirror[s] the simulation results," and human data "improves" ModAR, with the big step coming from in-domain video.

What "interpolating to a smaller hidden size" means

Flex-π's recipe initializes its action expert from the video backbone even though the action expert is narrower. The paper describes this as interpolating the pretrained weights "to a smaller hidden size for the action expert." The general idea is to resample each pretrained weight matrix down to the narrower width so the action expert starts from something video-shaped rather than from random numbers. The exact procedure belongs to Flex-π's own paper; the point for us is the contrast: Flex-π's video backbone and action expert start from pretrained video weights, while every parameter of ModAR starts from scratch. (The paper does not say whether Flex-π's smaller input and output layers, for example for pointmaps or actions, are also pretrained.)

What the Flex-π comparison can and cannot tell you

One more difference hides in the table above: ModAR's DINO targets come from a frozen DINOv2 encoder, while Flex-π keeps a frozen DINOv3 encoder. The two models are not even predicting the same feature space, which is one more reason the paper calls this a system-level comparison.

Chapter 9: Limits, Connections, and What to Read Next

You started with a robot that imagined its future in pixels, paying for lamp glints and tablecloth patterns. You now know a design that imagines motion first, then meaning, then distance, and only then acts, and you have checked its evidence line by line. This last chapter draws the boundary of what that evidence covers, connects ModAR to the rest of the site, and points to what to read next.

The whole argument, in six lines

The limitations the paper states

The paper's limitations section is short and specific. Here it is, point by point, with what each one means for how far you can carry the results.

And the future work it proposes: "larger-scale experiments with broader task diversity, generalization evaluation, and training on heterogeneous internet-scale actionless data," plus exploring "the best order for generating modalities, which may depend on the task or specific scenario."

Caveats stated elsewhere in the paper

Three more boundaries appear in the body text rather than the limitations section, and they matter just as much.

The Flex-π comparison is system-level. The models "differ in scale, pretraining, target modalities, and training recipe," so the comparison says a compact from-scratch model can keep up on these tasks, not that sequential generation beats a pretrained backbone in general.

Order was tested only against its reverse. "We leave a full systematic comparison of modality orderings to future work." The chosen order beat its reverse by 10 points; whether some third order would beat both is unknown.

The RGB conclusion is scoped. RGB gave "no consistent benefit in our experiments," with from-scratch models. The conclusion frames the result as "a promising alternative to relying solely on future RGB prediction."

Open questions we would ask next

These are ours, not the paper's. Each follows from something the paper leaves unmeasured.

• Does ordering still help with a pretrained trunk? Flex-π is pretrained and joint; ModAR is from scratch and sequential. The missing cell is pretrained and sequential.
• How sensitive is context noise? Only β = 0.5 is reported, and removing it costs 12 points. A sweep would say whether 0.5 is a sweet spot or a safe default.
• Position versus content. The w/o-tracks ablation also changes which block goes first. Putting tracks second, or DINO first with tracks kept, would separate "tracks are informative" from "the first block should be compact."
• Could the order be chosen per task? Table II shows the best subset changes by task. A model that picks its own order, as the limitations section hints, would be the natural next architecture.
• What does a partially generated future buy? Disjoint skips futures entirely at inference; ModAR generates all. Dropping late blocks at run time (a latency knob) was not evaluated.
• Heterogeneous human video at scale. EgoDex added 2 successes out of 90 on top of in-domain video. Whether internet-scale, uncurated video helps more, less, or not at all is one of the paper's stated future-work items.

Where ModAR sits among world models

The Causal World Modeling paper in that table (LingBot-VA on this site) is one of the works ModAR cites as predicting future RGB as image latents from frozen video VAEs, which makes it a good contrast read.

How the simulations in this lesson relate to the paper

Where to go next on this site, by goal

• Shore up the generative machinery: the flow matching gleam, then the DiT gleam, then the RoFormer lesson.
• Understand the targets: DINOv2 and CoTracker3 (the two frozen or offline tools ModAR leans on), then Depth Anything V2 for learned depth.
• See the policy side: Diffusion Policy and ACT for action chunks, then π₀ for a flow-matching VLA with an action expert.
• See other world models: LingBot-VA for a video-first WAM, PointWorld for 3D point futures, World-In-World for closed-loop evaluation.

Lessons on this site that connect

Papers to read next (from ModAR's reference list)

Where each part of the paper lives in this lesson

Questions to ask the next world-action model paper you read

ModAR's controlled design gives you a checklist for reading its neighbors critically.

• What exactly does it imagine? RGB latents, structured features, 3D points? And is there an ablation showing each target earns its cost?
• How are futures coupled to actions? Jointly, independently, or in sequence? Does the action ever read a finished future?
• Does it scale with actionless data? Is the number of action-labeled demonstrations held fixed while actionless data grows, as in Table I?
• Are the baselines controlled? Same backbone, same labeled data, same optimization budget? Or is it a system-level comparison?
• Is the sampling budget matched? Sequential and joint samplers can differ fivefold in network passes.
• What is evaluated? Held-out initial conditions of trained tasks, or genuinely new tasks, objects and scenes?
• How many trials? With 50 or 30 trials per task, single-cell differences of a few points are within noise.

Common mistakes when explaining ModAR

ModAR in three sentences, for three readers

For a practitioner: if you have few robot demonstrations and lots of task video, a small WAM that predicts point tracks, DINO features and depth in that order, then actions, is a strong, cheap recipe; skip RGB unless your task depends on appearance, and do not skip context noise.

For a researcher: the paper cleanly separates formulation, target representation and actionless-data scale in a from-scratch study, finds sequential ("modality-autoregressive") generation best at every scale with two controls, and leaves open ordering search, pretrained sequential models and broad generalization.

For a student: a robot does better when it first imagines where things will move, what they are and how far away they are, one step at a time, and only then decides how to move its arms.

Experiments the paper invites (ours)

Reimplementation checklist

If you set out to rebuild ModAR from this lesson, here is every specification, split by where it comes from. The right column is honest about what you would have to decide yourself.

The right column is short, which is a compliment to the paper: almost everything that determines the results is written down.

Glossary

Where the field is heading, per the paper's related work

The paper situates itself in a fast-moving 2026 literature, and its reference list is a good map of the open directions.

Richer structured futures. WAM4D uses spatial register tokens for fast 4D prediction; 3PoinTr lifts tracks to 3D; ST-WAM targets semantic-temporal features for robustness to visual distribution shift. Each is a different bet on which structure a policy should imagine.

Human video at scale. EgoWAM trains beyond pixels on in-the-wild egocentric human data; EgoDex supplies large-scale egocentric manipulation video. ModAR's small EgoDex gain is one data point in a question these works are actively probing.

How much imagination is needed at run time. Fast-WAM asks whether test-time future imagination is needed at all. ModAR's results suggest that, in its setting, generating the right futures in the right order improves success, at a latency cost the paper lists as a limitation.

Pretrained generators. DreamZero, Cosmos Policy, VERA and Flex-π all build on video models. The natural synthesis, a pretrained model that also generates structured modalities in sequence, is the experiment this paper leaves for someone to run.

A final self-test

Try these three without looking back; each takes one or two sentences.

1. A colleague proposes generating RGB first "because it contains everything." Which experiment in the paper speaks to this, and what did it find? Check: the reverse-order ablation (RGB → depth → DINO → tracks) dropped success from 75% to 65%.

2. Why can ModAR learn from a human video but Action-only cannot? Check: ModAR supervises its future blocks on actionless data and drops only the action term; Action-only has no future objective, so an actionless example gives it nothing to learn.

3. What single number best summarizes why context noise matters? Check: without it, success at D = 250 falls from 75% to 63%, below Unified's 67%.