A video is a very expensive way to store a world. Marionette stores the world in 276 numbers, computes the geometry with a program that has no weights at all, and lets the neural network do only the one thing networks are actually good at — making it look real.
You are playing a game that does not exist. There is no engine underneath it, no collision mesh, no skeleton, no scene graph. There is a neural network that has watched an enormous amount of gameplay footage, and every time you press a button it produces the next frame of video.
For the first five seconds it is astonishing. The monster lunges, your character rolls, dust kicks up, the light on the armour is correct. Then something small goes wrong. The monster's tail passes half a metre into a rock. Nothing catches it, because there is no rock — there are only pixels that looked like a rock.
Ten seconds later the tail is longer than it was. Twenty seconds later the monster is a different monster.
This is not a bug in a particular model. It is the shape of the whole approach, and it is worth naming precisely before we fix it.
Autoregressive generation means the model's own output becomes its next input. Frame 1 is produced from your seed image, frame 2 from frame 1, frame 3 from frame 2, and so on for as long as you keep playing. Nothing else carries information forward. Whatever the world needs to remember, it must remember by being visible in the last frame, or by being held in the network's own activations.
Marionette's opening sentence names the consequence exactly: interactive game world models “autoregress visual observations directly in pixel or latent space, forcing structured properties such as pose, geometry, and occlusion to be implicitly maintained by the same generative sequence.”
Read that list again — pose, geometry, occlusion. Those are not aesthetic properties. They are bookkeeping. A game engine maintains them with a few hundred lines of linear algebra that are correct by construction. A video generator maintains them the way a very good improviser maintains a story: mostly, plausibly, and with no mechanism that can ever say no, that is not where the tail is.
Here is the observation that reframes everything. A video is a high-dimensional observation of a low-dimensional world state.
Make that concrete with numbers we will use all lesson. One frame of Marionette's RGB output is 704 × 1280 pixels with three colour channels:
And the state that determines what is happening in that frame — where both characters are, how every joint is bent, which way each is facing — is 276 numbers. The ratio:
Roughly ten thousand pixels carry the information of one state number. Learning the distribution over videos directly means learning, in that 2.7-million-dimensional space, a structure that lives on a 276-dimensional sheet inside it — and learning it entangled with lighting, texture, motion blur, and the particular grain of the capture.
So posit the state. Call it s — the physical configuration of the scene, written down explicitly. If you have it, then how the world looks and how the world evolves become conditionally independent, and the single hard problem splits into two well-posed ones:
The left factor answers what does this configuration look like? The right factor answers given what has happened and what the player pressed, what happens next?
This decomposition is old — it is the classical dynamics-and-observation split of control theory, and of Ha and Schmidhuber's world models. What Marionette does with it is new: it insists that the state be explicit, metric, and human-readable, and it puts a program, not a network, in the middle.
The interesting question is not whether to factorize but where the line falls. Marionette's answer is a statement about what each side is good at.
Neural generative models are extraordinary at appearance and perceptual plausibility. They are weak at exact bookkeeping over long horizons and at discrete logic. The paper picks a beautifully sharp example: Minecraft's redstone — an in-game system of logic circuits that behaves like digital wiring. Whether a redstone lamp turns on is determined by the logical state of the circuit, not by what the nearby pixels look like. A model trained to produce the next frame can render one plausible frame while completely ignoring that state.
The analogy the authors reach for is the one you already use every day: a language model with a calculator. You do not ask the network to generate the product of two eight-digit numbers. You give it a tool that is right by construction and let it decide when to call it.
Hold on to the parameter counts, because they tell the story on their own. The dynamics model — the part that decides what actually happens in the world — is about 27.5 million parameters. The observation model is 5 billion. The bridge is zero.
Half a percent of the parameters decide the world. The other 99.45% decide how it looks. That is the factorization made physical.
You could object: a latent world model also has an intermediate state. Why does it matter that this one is written in metres?
Three things become possible, and none of them is available to a latent.
First, whole failure modes stop being representable. The bridge draws a fixed skeleton topology. A monster cannot become a different species. It cannot gain a limb. Not because a loss discourages it — because there is no number in the state vector that could express it.
Second, the state can be checked against an external structure between steps. The terrain is a fixed, known object. If the predicted root ends up inside a hillside, you can project it back onto the surface at the frame it appears, and because the terrain features are re-sampled from the corrected root, the next step is conditioned on a legal configuration. That is a per-step contraction. A purely generative rollout has no equivalent operation, because there is nothing to project onto.
Third, violations are measurable in absolute units. Penetration depth in metres. Foot slide in metres per second. Bone-length variation as a percentage. You can detect drift and, crucially, attribute it — which is how the paper is able to report, in Chapter 9, a smooth and healthy-looking video-quality curve sitting on top of a state-layer catastrophe that the curve does not register at all.
Both loops feed their own output back in. The left one recurs on an appearance latent; the right one recurs on a metric state that a rule can inspect between steps. Turn the projection rule on and watch what happens to the error curve — and notice that the rule is not available at all on the left, because there is nothing there to project.
Three things to notice while it runs. First, both curves start identically — for the first second or two there is no visible difference, which is exactly why this failure ships. Second, the unconstrained curve is a random walk with drift, and drift wins: the error grows roughly linearly, not as the square root of the step count. Third, the projected curve is bounded, and it is bounded by a rule with no learned parameters, applied after the network and before the render.
Chapter 0 asserted that a fight between a hunter and a monster fits in 276 numbers. This chapter opens the vector and reads every slot, because the layout is not an implementation detail — almost every design decision in the paper is visible in it if you know how to look.
Here is the state s, in full. Read it as a concatenation, left to right, of seven blocks:
M is the monster. N is the hunter — the player character. Now the table, with the arithmetic you should check yourself:
| Indices | Component | Dim | Running total |
|---|---|---|---|
[0:3] | Monster root displacement (local, per-frame) | 3 | 3 |
[3:162] | Monster root-relative joint positions — 53 × 3 | 159 | 162 |
[162:165] | Hunter root displacement (local, per-frame) | 3 | 165 |
[165:258] | Hunter root-relative joint positions — 31 × 3 | 93 | 258 |
[258:264] | Weapon points, hunter-relative — 2 × 3 | 6 | 264 |
[264:270] | Monster root rotation (6D) | 6 | 270 |
[270:276] | Hunter root rotation (6D) | 6 | 276 |
All positions are in metres. The world frame is Y-up and right-handed. There is no normalisation, no learned scale, no unit ambiguity: if two numbers differ by 1.0, the two points are one metre apart, and you can walk over and measure it.
The monster skeleton has 54 points. The hunter has 32. The weapon has 2. But the state stores 53 and 31 joint positions. Where did the missing joint go?
It went into its own block. The root — the joint at the base of the skeleton that carries the character's overall position in the world — is stored separately, in the three-dimensional δ block, and it is stored as a displacement since the previous frame, not as a position. Every other joint is stored relative to the root.
This is a two-line change to a data format and it is doing an enormous amount of work. Let us take it apart.
Suppose you store joint positions in absolute world coordinates. Now the hunter performs the identical crouch twice: once at world position (12.0, 0.0, 40.0) and once at (86.0, 3.5, −12.0). Every single one of the 31 joint vectors differs between the two recordings, in all three coordinates, by exactly the difference in root position.
The model must now learn that these two very different-looking 93-dimensional vectors mean the same thing — and it must learn it separately for every position on the map it ever sees. That is a colossal waste of capacity spent memorising a translation it could have been handed for free.
Store the joints relative to the root and the crouch is one vector. The same pose produces the same numbers everywhere on the map. The paper puts it plainly: the representation becomes “invariant to global position and heading.”
Heading matters as much as position. The joints are stored in the root's rotated frame, so a crouch facing north and a crouch facing south are also the same 93 numbers. Without that, the model would additionally have to learn every pose at every yaw angle.
The root itself cannot be root-relative — that would be identically zero. So it is stored as a per-frame local displacement: how far did I move since last frame, expressed in my own body frame?
Three consequences follow, and the third is the one that bites later.
The numbers stay small and stationary. At 20 fps a running character covers maybe 12 centimetres per frame. So δ lives in roughly [−0.3, 0.3] regardless of whether the fight is at the map origin or 400 metres away. An absolute position, by contrast, would range over the whole stage — a distribution that shifts under the model as the character moves.
The same walk cycle is the same numbers. Walking north-east at 2.4 m/s and walking south at 2.4 m/s produce identical δ sequences, because the displacement is expressed in the body frame. The heading is carried entirely by the 6D rotation block.
Errors integrate. Absolute position is recovered only by summing the deltas — and a sum has no memory of being wrong. Get one frame's δ wrong by a centimetre and every subsequent world position is off by that centimetre, forever. Chapter 9 is about exactly what that costs.
Every block is drawn proportional to its width in the vector. Tap a block to read its index range, its dimension, what it physically means, and — the part that matters in Chapter 2 — which of the two transformers is responsible for writing it.
Six numbers — two 3D points — stored relative to the hunter. That is the entire weapon.
It is worth pausing on how little that is, and why it is enough. A weapon in this game is a rigid object held in a hand. Two points on a rigid object pin down its position and its axis; what they cannot pin down is its roll about that axis. The authors accepted that. For a great-sword swing, where the blade goes matters and how the blade spins about its own length mostly does not — and the diffusion model, which has seen hundreds of hours of this weapon, will paint a plausible roll anyway.
This is the offloading principle applied at the smallest scale in the paper: put in the state exactly what must be exact, and let appearance handle the rest.
Each entity's root orientation takes six numbers. A rotation in 3D has three degrees of freedom, so six is double-counting. That is deliberate, and it is one of the most consequential small choices in the paper.
The 6D parameterization stores the first two columns of the 3×3 rotation matrix. Column three is recovered as their cross product. The representation is due to Zhou et al. (2019), and the argument for it is about continuity.
| Parameterization | Numbers | What goes wrong |
|---|---|---|
| Euler angles (yaw, pitch, roll) | 3 | Gimbal lock. At pitch = 90° yaw and roll become the same axis, and the mapping is not continuous. A network regressing yaw across that boundary has to jump |
| Quaternions | 4 | Double cover. q and −q are the identical rotation, so the regression target is ambiguous, and a network smoothly interpolating between q and −q passes through zero — which is not a rotation at all |
| Full rotation matrix | 9 | Continuous and unambiguous, but the network must maintain nine numbers on a six-dimensional manifold, and its raw output will never be exactly orthonormal |
| 6D (two columns) | 6 | Continuous, no double cover, and any six numbers whose first two 3-vectors are not parallel project onto a valid rotation. The network can output whatever it likes and the bridge fixes it up |
That last row is the point. The network is never asked to emit an orthonormal frame. It emits six unconstrained numbers, and a closed-form procedure — Gram–Schmidt, which we will run by hand in Chapter 3 — turns them into an exact element of SO(3). The constraint is enforced by the tool, not learned by the network. That sentence is the whole paper in miniature.
Let us make the abstraction physical. Take a single frame, mid-fight, and write down what is in memory.
The monster's root has moved δM = (0.00, 0.00, 0.09) — nine centimetres straight ahead in its own body frame. Its 53 root-relative joints are 159 numbers describing a lunge: the head forward and low, the tail counterweighted back. Its 6D rotation says it is facing 42° off world-north.
The hunter's root has moved δN = (0.14, 0.00, −0.06) — sidestepping right and slightly back. Its 31 joints are 93 numbers in a mid-roll crouch. The weapon's two points sit low and behind, because the great-sword is sheathed on the back.
Now the two questions that separate this from a latent.
How far apart are the characters? You cannot read it off the state directly — the roots are deltas. You integrate both root streams from the start of the rollout, and subtract. The answer comes out in metres, and Chapter 9 reports that exact quantity going from 4.9 m to 21.2 m over sixteen seconds.
Is the monster's foot inside the ground? Reconstruct the world-space joint position (Chapter 3), query the terrain height under it (Chapter 5), subtract. The answer is a signed distance in metres. The paper reports it as the fraction of frames in which any of 11 key joints sits below the surface by more than a 0.15 m margin.
Neither question can be asked of a latent vector, at all. That is not a matter of the latent being smaller or less interpretable; it is that there is no operation on it that returns metres.
Something has to fill the 276 slots, one frame at a time, in response to what the player is doing. That something is the dynamics model, and Marionette splits it into two transformers that do very different jobs.
Before the architecture, the motivation — because the split is not an optimisation, it is a claim about the problem.
Watch a real fight and describe what happens in the next tenth of a second. There are two answers, at two completely different scales.
The first is a decision: the hunter is going to dodge-roll to the right. That is one choice from a menu. It is discrete, it is low-dimensional, it is the thing a player's thumb actually causes, and it is the thing you would write in a strategy guide.
The second is an animation: over the next 12 frames, the hunter's 31 joints move through a specific continuous trajectory that constitutes a dodge-roll from this starting pose. That is 93 continuous numbers per frame, and no player has ever thought about it.
Ask one network to do both and you have coupled a 173-way (or 689-way) categorical choice to a 258-dimensional continuous regression, in the same weights, with the same loss balance. Marionette separates them.
Here is the detail that made this architecture click for me, and it falls straight out of Chapter 1's table.
ActionGPT is conditioned on, and regresses, the 18-dimensional root and rotation stream. Where does 18 come from? Two entities, each contributing a 3-dimensional root displacement and a 6-dimensional rotation:
PoseGPT consumes and predicts the 258-dimensional body state. Where does 258 come from? The monster's joints, the hunter's joints, and the weapon:
And:
The two stages partition the state vector exactly, with no overlap and no gap. ActionGPT owns where the bodies are and which way they face. PoseGPT owns what shape the bodies are in. Every slot has exactly one author. (Try the “colour by owner” toggle on the Chapter 1 simulation and the partition is visible as two contiguous regions.)
This is why the paper can say the interface between the two stages “is a state sequence and nothing more” — and why it can note that a different animation model, such as ARDY or Kimodo, could be dropped in to supply the PoseGPT half without touching anything else.
ActionGPT is a causal transformer with sinusoidal position encodings and a bounded attention window of block size 512. At 20 fps, that window is:
Per frame it consumes a deliberately thin summary:
| Input | What it is | Why it is there |
|---|---|---|
| 18D root/rotation stream | Both entities' root displacement and 6D rotation | Where everyone is and which way they face — the only spatial facts a decision needs |
| Learned action embeddings | Per-entity embedding of recent action ids | What each character is currently doing. A dodge cannot start in the middle of a dodge |
| Animation-progress scalars | How far through the current animation each entity is | Actions have duration. This is the difference between “can act now” and “locked in recovery” |
| Weapon sub-state | The 6 weapon numbers | Whether the weapon is drawn, and roughly where it is pointing |
| Terrain features (optional) | Egocentric height patch + per-joint clearances, through a 2-layer MLP to 32 dims | Where you can go depends on what is walkable. Chapter 5 |
Notice what is not in that list: the 258 body-pose numbers. ActionGPT never sees a joint. It decides what to do from position, heading, what is currently playing, and the ground — which is roughly the information a player has from the camera.
From these it produces, per frame per entity: a categorical distribution over that entity's action vocabulary, plus a regressed next root displacement and 6D rotation.
The action ids are not invented by a labelling process. They come from the game's own animation-bank and motion identifiers — the engine's internal name for the animation it actually played on that frame. That gives a frame-accurate, zero-offset label, which the paper is right to emphasise: an action id here is the animation the engine played, not a guess from a vision model.
| Entity | Vocabulary | Skeleton | Root-relative joints in state |
|---|---|---|---|
| Monster | 173 action ids | 54 points | 53 → 159 numbers |
| Hunter | 689 action ids | 32 points | 31 → 93 numbers |
| Weapon | — | 2 points | 6 numbers (hunter-relative) |
The asymmetry is worth reading. The hunter has four times the action vocabulary of the monster (689 vs 173) on a skeleton with fewer joints (32 vs 54). That is what a player-controlled character looks like: an enormous menu of context-dependent moves, weapon-specific combos, and cancels, expressed through a comparatively simple humanoid rig. The monster is the reverse — a big elaborate body running a smaller, more stereotyped behaviour repertoire.
PoseGPT is the larger model — 8 layers, width 512, about 25 million parameters — and its job is narrower than its size suggests. It consumes the recent 258-dimensional body state and the (possibly overridden) action ids, and predicts the next-frame 258-dimensional body pose.
The paper's own phrase for it is exactly right: it “acts as an animation player, mapping the chosen action to the resulting body configuration.”
One detail matters for implementation: the action input is aligned to the frame being animated, a one-step shift relative to the decision stream. ActionGPT decides at frame t what will happen at frame t+1; PoseGPT animates frame t+1 with that decision already in hand. Get the shift wrong and you have built a model that animates the previous decision — which will still train to a plausible-looking loss, and will feel unmistakably laggy.
Here is the whole autoregressive step, in order. This is what runs every 50 milliseconds.
| # | Step | Shapes |
|---|---|---|
| 1 | ActionGPT reads the recent 18D root/rotation stream, action embeddings, progress scalars, weapon sub-state, terrain features | context → logits over 173 and 689, plus 18 regressed numbers |
| 2 | Control override (optional): replace the sampled action id with a target id, and/or overwrite the root displacement and rotation | one tensor assignment |
| 3 | PoseGPT reads the recent 258D body state and the (possibly overridden) action ids | context → 258 numbers |
| 4 | Assemble: interleave the 18 root/rotation numbers and the 258 body numbers into the state layout of Chapter 1 | → st+1 ∈ R276 |
| 5 | Terrain collider projects an infeasible root back onto the ground surface (Chapter 5) | 276 → 276 |
| 6 | Append to context, re-sample terrain features from the corrected root, repeat | window bounded at 512 |
Step 2 is the entire control interface, and it deserves its own section.
In most controllable generative systems, “control” means an extra conditioning pathway: an encoder for the control signal, a cross-attention block, a training objective that teaches the model to respect it, and a guidance scale at inference that trades control strength against quality.
Marionette has none of that, because the action is already an explicit token in the model's own output stream. To make a character perform an action you overwrite the sampled id with the one you want:
No retraining. No auxiliary network. No gradient. The paper is explicit that this is the same operation a player's button press performs — the button press is an action id, and the model already speaks that language because the ids came from the engine's animation bank in the first place.
There is a second control channel, and it controls a different quantity. ActionGPT regresses the root displacement and rotation as continuous outputs, and PoseGPT consumes that root stream rather than deriving it from the body. So the root is an input to animation, and it too can be overwritten at a frame.
| Channel | What writing it does | What it does not do |
|---|---|---|
| Action token (discrete id) | Says what the character is doing. The natural interface for button-like commands | It steers rather than pins — the pose a token produces still depends on the context it is injected into |
| Root (3D displacement + 6D rotation) | Pins locomotion exactly. A displacement fixes travel speed and direction; a rotation fixes heading. Expressed in metres and radians, and reproduced exactly in world space because the bridge integrates it, not the network | Says nothing about the limbs. A body whose held action has finished animating will translate without stepping unless the action channel is driven too |
The authors use them separately and deliberately: movement demonstrations script the root and leave the action stream free; action demonstrations do the converse. Both are single tensor assignments on the model's own output stream.
For the controllability experiments the model decodes deterministically — temperature 0. That is not a quality choice, it is an experimental one: with greedy decoding and a fixed seed context, three rollouts that differ only in the injected token stream are identical up to the injection point, frame for frame. The paper's Figure 6 relies on exactly this: three rollouts share a seed, remain pixel-identical until the 5-second mark, and diverge only after the token switch.
If you build this, keep that property. Being able to say “these two rollouts differ in exactly one integer” is what turns a demo into evidence.