MotionBricks:
Real-Time Motion from a Modular Latent Brain

Chapter 1: The Key Insight

Here is the move that makes everything else possible. Instead of asking the model to "generate a walk" or "generate a vault," MotionBricks asks it a much more humble, much more general question: given where the character is now (context) and where it needs to be soon (target), fill in the motion in between. That is the classic animation problem of in-betweening.

Why is this the right abstraction? Because every form of control collapses into the same currency: a keyframe. Want the character to walk north at 1.5 m/s? That is a target root keyframe. Want its hand to land on a ledge? That is a target pose keyframe. Want a specific stylized strut? Drop a style keyframe. Navigation and object interaction, locomotion and acrobatics — they all reduce to "here are some context frames, here are some target frames, generate the bridge." One interface, infinitely many behaviors.

Manufacturing the need: why naive in-betweening fails

You might think: if I have a start pose and an end pose, can't I just interpolate? Linearly blend joint angles from A to B? Try it in the widget below. Drag the scrubber and watch a naive linear interpolation between two real poses. The character's feet skate across the floor, limbs pass through impossible configurations, and the timing is robotic — there is no sense of weight, no push-off, no gait. Linear interpolation has no idea how bodies actually move.

The "learned in-between" toggle shows the goal: a model that has seen hundreds of thousands of real motions and therefore knows that to get from a left-foot-forward stance to a right-foot-forward stance, the body must shift weight, the back leg must push off, the arms must counter-swing. It fills the gap with plausible physics it absorbed from data, not a straight line through joint space.

Chapter 2: Background — the landscape MotionBricks fights in

To appreciate the design choices ahead, you need a map of what came before. There are three broad camps, and each one is strong on one axis and weak on another. The whole art is winning on every axis at once.

Camp 1: Traditional methods (graphs & motion matching)

Animation graphs replay clips inside states and blend during transitions. Motion matching (Clavet 2016, used widely in AAA games) is the clever successor: instead of pre-wiring transitions, it keeps a big database of motion frames and, every few frames, searches for the database frame whose pose and trajectory best match the current state and the user's command, then plays forward from there. This frees you from rigid node-based graphs and feels wonderfully responsive. But it is still fundamentally retrieval: it can only ever produce motion that already exists in the database, it needs careful feature engineering, and it still does not overcome the underlying scalability wall as skill counts grow.

Camp 2: Deterministic neural models

Phase-Functioned Neural Networks (PFNN, Holden 2017) and its descendants predict the next pose autoregressively from controls and a phase variable. They are fast and were a genuine breakthrough. But deterministic regression to a single "average" next pose causes well-known artifacts — over-smoothing (the model averages plausible futures into mush), foot sliding, and object penetration — and they tend to be specialized for a narrow task set.

Camp 3: Generative models (diffusion & tokens)

Motion Diffusion Models (MDMs) brought genuinely diverse, high-quality, text-controllable motion. Their curse is inference speed: iterative denoising takes many network passes — seconds to minutes per clip. Tricks like DDIM, flow matching, and autoregressive diffusion claw back speed but still trade quality for it and barely reach real time. Token-based models (the camp MotionBricks belongs to) discretize motion into tokens — like words — with a VQ-VAE, then predict tokens with a transformer, often via iteratively masked prediction (MaskGIT-style). This is fast, but prior token methods used a single codebook without root–pose separation, and had not yet reached industry-grade quality, scale, and interactivity simultaneously.

The widget plots every camp on the two axes that matter for production — throughput (how many frames per second you can generate) and quality (how real the motion looks, inversely tracking FID). Toggle the methods. Notice the empty top-right corner: fast and high-quality. That corner is the prize.

Chapter 3: The Four-Stage Pipeline

Before diving into any single component, let's get the data flow exact. MotionBricks's runtime is a four-stage pipeline. Each stage produces something concrete that the next consumes. Getting these shapes and roles straight now makes every later chapter click into place.

Click each stage in the diagram to trace what flows through it — the actual inputs, outputs, and tensor shapes.

The four stages in words

• Stage 0 — Smart Primitives. Takes the current character state S and user commands / game events C. Produces a set of target keyframe constraints 𝐓 = {𝐓₁, 𝐓₂, 𝐓₃} — constraints on local root, global root, and pose respectively. This is the only human-facing stage.
• Stage 1 — Root Module ℱ. Reads the constraints and predicts how many in-between frames T there should be, plus an initial root trajectory {r} (where the pelvis goes, and which way it faces, over those frames).
• Stage 2 — Pose Module 𝐏. Conditioned on the root trajectory and the keyframes, predicts the discrete pose tokens {z_q} — a compact code for the full-body motion (limbs, spine, everything except the root). One token per 4 frames.
• Stage 3 — Token Decoder. Decodes the pose tokens together with the root trajectory and keyframes into the final continuous motion — joint positions, rotations, velocities, and contact labels, frame by frame.

The runtime loop & the masking trick

The system runs autoregressively: it generates a buffer of future motion, pops one frame per tick to drive the character, and replans only when the control signal changes or the buffer is running low. Replanning lazily (not every frame) is what keeps it cheap — and on robots, it avoids fighting the low-level controller for compute.

One more crucial detail that recurs everywhere: flexible constraints via masking. Different tasks need different keyframe density. Navigation might supply just 1–2 keyframes (don't over-constrain a walk!); precise object interaction might supply dense hand positions. So every module accepts an arbitrary subset of constraints. When a constraint is absent, MotionBricks substitutes a learnable mask embedding — a trained "this is missing" token — so the network degrades gracefully instead of crashing. The same masking mechanism is shared across the root module, pose module, and decoder.

Chapter 4: The Structured Multi-Head Tokenizer (showcase)

This is the cornerstone of MotionBricks, and the single most important idea to absorb. The job: turn continuous motion into a compact sequence of discrete tokens (so a transformer can model it like language), and do it in a way that scales to a 350,000-clip dataset and degrades gracefully when a token is wrong. The two design decisions that make it work are root–pose disentanglement and multi-head quantization.

First idea: encode pose separately from root

The motion state at each frame is a tuple (r_g, r_l, p, q, v, c): global root values (pelvis position + heading), local root values (pelvis linear/angular velocity), joint positions p, joint rotations q, joint velocities v, and contact labels c. Crucially, the encoder ignores the root and encodes only the local pose:

Why throw away the root? Because pose intent and root intent are largely separable. A person can walk the same gait at slightly different speeds, or follow the same footstep pattern along a curved vs. straight path. The shape of the motion (the gait) is one thing; where it travels (the root) is another. By tokenizing only the pose, the same pose tokens can be reused across many different root trajectories — which, as you'll see in the widget, lets you warp a motion to a new path without re-generating the body. The encoder progressively downsamples (rates 2 then 4) via a 1D conv U-Net (or a transformer), so T frames become T/4 latent vectors.

Second idea: quantize with K codebooks, not one

Now the continuous latent z_e must become discrete. The naive approach (VQ-VAE) uses one big codebook: snap the whole latent vector to its nearest entry. MotionBricks instead splits the feature dimension into K parts and quantizes each part against its own codebook {e₁, …, e_K}:

Each frame is now described by K token indices instead of 1. This is the difference between describing a person with one giant word from a vocabulary of millions, versus describing them with K independent attributes ("head-tilt = 3, left-arm = 71, stance = 12, …"). The authors deliberately do not hand-partition by body part — they let the network learn its own decomposition, fully data-driven. Two payoffs follow, and they are the heart of the paper:

• Capacity that actually scales. K heads with C entries each gives an effective vocabulary of C^K combinations — exponential in the number of heads — without a single unwieldy codebook. (Chapter 9 shows the single-head baseline plateaus while this keeps improving up to a billion effective tokens.)
• Graceful degradation. If the pose module mispredicts one token, only one head's slice of the latent is wrong; the other K−1 heads still carry correct information, so the decoded motion wobbles instead of exploding. A single-head codebook has no such safety net — one wrong token corrupts the whole pose.

A worked example you can do by hand

Take K = 2 heads. Say the encoder outputs a 4-dim latent z_e = [0.9, 0.2, −0.6, 0.1], split into head-1 = [0.9, 0.2] and head-2 = [−0.6, 0.1]. Each head has a tiny 3-entry codebook:

• Head-1 codebook E₁: e₁₀ = [1.0, 0.0], e₁₁ = [0.0, 1.0], e₁₂ = [−1.0, 0.0].
• Head-2 codebook E₂: e₂₀ = [−0.5, 0.0], e₂₁ = [0.0, −0.5], e₂₂ = [0.5, 0.5].

Head-1: distances² to [0.9,0.2] are  to e₁₀: (0.1)²+(0.2)²=0.05;  to e₁₁: (0.9)²+(0.8)²=1.45;  to e₁₂: (1.9)²+(0.2)²=3.65. Nearest = e₁₀ → index 0. Head-2: distances² to [−0.6,0.1] are  to e₂₀: (0.1)²+(0.1)²=0.02;  to e₂₁: (0.6)²+(0.6)²=0.72;  to e₂₂: (1.1)²+(0.4)²=1.37. Nearest = e₂₀ → index 0. So this frame's tokens are (0, 0), and the reconstructed latent is the concatenation [1.0, 0.0, −0.5, 0.0]. Now imagine the pose module mispredicts head-1 to index 2 ([−1.0,0.0]) — head-2 is still correct, so half the latent survives. That's the safety net, made concrete.

See it, break it

The showcase has three views. Encode animates a motion through downsampling into K codebook heads. Disentangle shows the killer feature: fix the pose tokens, change the root trajectory, and watch the same body motion warp onto a new path (this is Figure 4 of the paper, made live). Perturb corrupts a fraction of tokens and contrasts multi-head vs. single-head degradation — drag the corruption slider and watch which one falls off a cliff.

Training the tokenizer

The tokenizer is trained with the standard VQ-VAE objective: a codebook loss that pulls each codebook entry toward the encoder output, and a commitment loss (weighted by β) that pulls the encoder output toward its chosen entry, using the stop-gradient operator sg(·) to route gradients correctly past the non-differentiable argmin:

In practice they use a running-mean codebook update (more stable than gradient updates) and note that an FSQ tokenizer (finite scalar quantization — no learned codebook at all, just rounding to a fixed grid) gives similar performance. During training the decoder is fed a random 0–10 keyframe constraints at arbitrary positions, forcing it to learn robust, generalizable features rather than memorizing a fixed constraint pattern.

Chapter 5: The Neural Backbone — root & pose modules

The tokenizer gives us a vocabulary. The backbone is the language model that generates tokens from constraints. It is split into two transformers that run coarse-to-fine: the root module (50M parameters) decides timing and the coarse path, then the pose module (150M parameters, the largest network) fills in the full-body tokens.

The root module: a two-step transformer

Step 1 — how many frames? Before you can draw a trajectory you need to know its length. Going from a start pose to a target two meters away takes more frames at a stroll than at a sprint. So step 1 predicts a distribution over in-between frame counts (at 4-frame resolution). Its inputs are three kinds of embedding:

• 16 learnable frame-slot embeddings {h¹_t} — one query per temporal location, covering up to 64 frames after 4× downsampling.
• A global constraint embedding f(𝐓₁, 𝐓₂, 𝐓₃) — a learnable 3-layer MLP encoding all the keyframe constraints.
• A timing embedding g(T₁) — encodes a ground-truth frame count when one is supplied, or a learnable mask embedding otherwise.

From the output distribution we sample (or argmax) to get the predicted frame count T₂. A neat control hook: you can apply bit-wise masking over the valid frame counts to constrain timing at inference (e.g. "make this take exactly 20–24 frames").

Step 2 — where does the root go? The intermediate frame-slot embeddings from step 1 are passed to step 2, with slots beyond the predicted count T₂ masked out. A second temporal transformer then predicts the initial global root trajectory. Formally:

Because of 4× downsampling, each output embedding decodes into 4 consecutive frames of root trajectory.

The pose module: masked token modeling

Now the big one. The pose module models the distribution of pose tokens, conditioned on the root trajectory (from the root module) and keyframes. It uses masked token modeling, the same idea behind MaskGIT and modern masked image generators: start with all tokens masked, predict them all in parallel, keep the most confident predictions, re-mask the rest, and repeat. Each step "freezes" the tokens the model is surest about and lets later steps fill the harder ones with that context.

Here φ embeds and combines the three input types (local root, global root, current pose tokens) via an MLP; 𝐏 is the transformer; the final hidden state is partitioned along the feature dimension into K slices, and each slice h^t_p,k is projected by a linear head f_k and softmaxed to give that head's token distribution. (See the multi-head structure paying off again — one transformer, K parallel classification heads.)

Step through both modules in the widget. The root view shows the timing distribution forming and then the root path being drawn; the pose view animates MaskGIT-style iterative token reveal — watch confident tokens lock in first and the rest fill around them.

Chapter 6: Smart Locomotion & the spring model (showcase)

We have a frozen backbone that turns keyframes into motion. Now: where do the keyframes come from? A player pushes a thumbstick — that is a velocity and a heading, not a keyframe. Smart locomotion is the primitive that bridges raw navigation commands into target keyframes the backbone can consume, while solving two nasty problems: making natural motion from arbitrary velocity/heading commands (including impossible ones — the "control dead zone"), and supporting many styles with smooth transitions.

Step 1: a critically damped spring for the initial path

You can't feed a raw thumbstick command straight to the backbone — it's jittery and may point somewhere infeasible. First you need a smooth, physically reasonable guess at where the root should go. The classic tool from motion matching (Clavet 2016) is the critically damped spring: imagine the character's pelvis attached by a spring to a moving target; tune the spring so it converges to the target as fast as possible without overshooting or oscillating. That "fastest without overshoot" condition is exactly critical damping.

MotionBricks first computes a naive target r_g,1 by linearly extrapolating the current state with the commanded velocity and heading over a 1.0 s horizon. Then it smooths the path from the current root r₀ toward that target with the critically damped spring solution:

where γ is the damping coefficient and r₀, v₀ are the current position and velocity. Evaluating at t = 1.0 s yields the spring-smoothed target r_g,2. The whole transient lives under the exponential envelope e^−γt — that's what guarantees no overshoot.

Worked example: plug in real numbers

Say the character starts at r₀ = 0 with velocity v₀ = 0.5 m/s, the extrapolated target is r_g,1 = 2.0 m, and damping γ = 4.0. Then:

• r₀ − r_g,1 = −2.0
• v₀ + γ(r₀ − r_g,1) = 0.5 + 4.0×(−2.0) = −7.5
• bracket at t=1: −2.0 + (−7.5)(1.0) = −9.5
• e^−4.0 ≈ 0.0183, so r(1.0) = 0.0183×(−9.5) + 2.0 ≈ 1.83 m.

The spring has carried the root smoothly from 0 to ≈1.83 m, closing in on the 2.0 m target without overshooting it. Crank γ up and it converges faster (snappier, more responsive); drop it and the approach is lazy and soft. Play with exactly this in the widget.

Steps 2–4: progressive neural refinement

The spring path is smooth but mechanistic — it doesn't know about footsteps, style, or the body. So MotionBricks refines it in stages. This is the progressive root refinement, summarized as four named stages:

The neural root refinement (stage 3) is where the magic is. Feeding the spring path plus keyframes into the root module produces trajectories with realistic detail instead of mechanical smoothness, and its timing prediction safeguards against control dead zones — if a user commands an infeasible velocity, the network produces a natural motion anyway rather than teleporting or stuttering. Remarkably, identical commands produce different root paths depending on the chosen style (a "happy" walk curves differently than a "stealth" crouch), because the style keyframe shapes the refinement.

Chapter 7: Smart Object — interaction without animation graphs

Smart locomotion handles getting around. Smart object handles interacting with the world — climbing ledges, vaulting fences, sitting, picking up swords, opening doors. Traditionally each interaction needs its own pre-canned clip tied to a specific object position, plus careful entry/exit blends. That is the animation-graph curse all over again. Smart object dissolves it with two lightweight ingredients: intent keyframes and an interaction binding.

Intent keyframes & the drop-frame dial

An interaction is defined by a few essential poses. A ledge climb, for instance, needs a contact pose (hands reach the wall edge) and an exit pose (standing on the ledge). You author these with standard translate/rotate/IK gizmos, or pull them from a motion library. The key control is the drop-frame attribute τ on each keyframe set, which decides how strictly the pose must be hit:

• τ = 0 → hard constraint: the motion must fully reach the target pose before proceeding. Essential for precise contact — hands must actually land on the ledge.
• τ > 0 → soft guidance: the keyframe shapes intent, but the model may transition to the next phase up to τ frames early without nailing the exact pose. Perfect for preparation poses (getting hands/feet ready for a vault) where strict enforcement would look stiff and over-constrained.

Interaction binding: detection, sockets, anchoring

The binding glues interaction logic to the object and manages three things: (1) Detection — an interaction mesh derived from the object's geometry acts as a trigger volume; a box-collision trace decides if the character is in range. (2) Object sockets & placement — for portable objects, sockets define where the object attaches (e.g. the hand) and placement logic snaps it to a valid surface when released. (3) Keyframe anchoring — for scene/static objects, the mesh's world transform is a pivot: keyframes flagged with a boolean ω = 1 are rotated around that pivot at runtime.

That anchoring flag is what delivers the headline benefit: the same keyframe definition works from any approach angle or object placement. Author a ledge-climb once; the pivot rotation lets the character climb that ledge approaching from the north, the east, or a diagonal — and lets identical "sit" keyframes adapt to chairs at different positions and orientations. Combined with the backbone's generative adaptation, one smart object yields automatic motion variations (falling adapts to height, pickups adapt to angle) without any extra authoring.

In the widget: place an object, set τ to feel hard vs. soft constraint enforcement, and rotate the approach angle to watch pivot anchoring re-target the same keyframes. The character's reach either snaps exactly to contact (low τ) or transitions early and loosely (high τ).

Chapter 8: Experiments & Results

A production claim demands production-scale evidence. MotionBricks is evaluated on four datasets of very different scales, against six recent baselines, across seven categories of metric. The headline: it wins on essentially everything that matters, including the things that usually trade off against each other.

The datasets

How motion quality is even measured

You can't just eyeball 350k clips. The seven metric families each capture a different failure mode:

• Speed — FPS & latency (ms).
• Distribution — MMD (maximum mean discrepancy) and FID (Fréchet distance in a learned motion-autoencoder latent): does the distribution of generated motion match real motion? Lower = more realistic.
• Human eval — 40 participants pick a winner among 3 methods; win rate + 1–5 Likert score.
• Diversity — variability across 20 generations (joint-level and latent-level).
• Smoothness — joint & root jitter (acceleration magnitude); lower = smoother.
• Physical plausibility — foot skate (sliding during contact), penetration (into the ground), foot-contact accuracy.
• Precision — context/target keyframe & root error, and reaching success (% reaching the target within 5 cm and 15°). Notably they use max per-joint error, not mean, because max better matches human perception of a glitch.

The 350k scoreboard

Explore the comparison below. Pick a metric and see every method's bar; MotionBricks ("Ours") is highlighted. The story repeats across metrics: deterministic methods (Cond. In-betweening, Two-stage, Delta-interp) get okay precision but terrible FID and human scores; diffusion methods (CondMDI, Closd-DiP) get good FID but weaker constraint satisfaction and slower speed; the prior token method (MMM) is decent but not dominant. Ours sits at or near the top of all of them, simultaneously.

The same pattern holds across LaFAN1-G1 (robotics), HumanML3D (public), and Bones-70k: MotionBricks leads on distribution quality, smoothness, and precision. On the two small datasets, baselines occasionally match or edge ahead on raw diversity — unsurprising, since a high-capacity model's advantage shows most when there's enough data to learn from. There is no significant trade-off between diversity and precision or speed.

Chapter 9: Scaling Laws & Ablations

The most important experiments aren't the leaderboard — they're the ablations that prove why the design works. Three results matter most: tokenizer scaling, the head/codebook trade-off, and dataset scaling.

Scaling: multi-head keeps climbing, single-head plateaus

This is the paper's central scientific claim. Take the structured multi-head tokenizer and a standard single-head VQ-VAE baseline, and grow the codebook capacity. The multi-head tokenizer's reconstruction loss keeps improving as you scale the effective vocabulary all the way up to ~10⁹ tokens. The single-head baseline plateaus almost immediately — throwing more codebook entries at it does nothing. Drag the slider and watch the two curves diverge.

Heads vs. tokens-per-head: a coherence trade-off

Fix the total capacity (~10⁶) and vary the split — many heads with small codebooks (e.g. 20 heads × 2 entries) vs. few heads with big ones (2 heads × 1000 entries). Under token perturbation (simulating prediction errors), a clear trade-off appears: too many tokens per head hurts NPSS (temporal coherence — the motion gets jittery), while too few tokens per head hurts FID and keyframe precision. The paper's recommendation: 128–256 tokens per head, with total capacity around 10⁹.

Dataset scaling & a counterintuitive FID

Train on 10% → 100% of the 350k data. MotionBricks's keyframe precision consistently improves with more data while the baseline's degrades (its limited capacity gets overwhelmed). One honest surprise the authors flag: FID sometimes worsens with more data for both methods. Their hypothesis — and this is a great lesson in reading metrics critically — is that models trained on smaller data overfit and produce less diverse output that happens to hug the training distribution, yielding artificially low FID. More data → more diverse, genuinely better motion → slightly higher FID. The metric that matters (precision, and human eval) keeps improving.

The engineering that ships it

Training: 32 GPUs (4 nodes × 8), 2M updates, batch 256/GPU, AdamW at lr 5×10⁻⁵ with cosine decay. Tokenizer is a 23.5M-param U-Net (1024 channels); pose module 150M (16 layers, 16 heads, dim 1024); root module 50M. Roughly 7 days (tokenizer) + 3 days (root) + 7 days (pose) on H100s. Inference: RTX 5090, 2 ms / 15,000 FPS, exported via ONNX + TensorRT into a native C++ UE5 plugin. On the Unitree G1 robot it runs on a Jetson Orin (5 ms latency), feeding kinematic targets to a physical tracking controller, replanning lazily at 10 Hz to avoid starving the low-level control loop.

Chapter 10: Connections & cheat sheet

MotionBricks is a beautiful synthesis of ideas you've met elsewhere. Seeing the lineage makes the whole thing feel inevitable rather than magical.

Where each idea comes from

• Discrete latents / codebooks → VQ-VAE. The multi-head twist is the paper's contribution, but the quantize-then-model-with-a-transformer recipe is the same one behind discrete image and audio generation. See Self-Supervised Learning and Generative Models.
• Masked token modeling → MaskGIT / masked image generators, themselves descendants of BERT's masked language modeling. See BERT and Attention Is All You Need.
• The speed problem it's escaping → diffusion models. See Diffusion for why iterative denoising is slow.
• Critically damped springs → classic control / signal smoothing, the same family as the filters in state estimation and trajectory smoothing in Trajectory Optimization.
• Sim-to-real & humanoid control → the robotics deployment leans on a learned tracking policy; see Sim2Real and Policy Gradients.

The cheat sheet — every key equation & symbol

Limitations the authors name honestly

• Dataset coverage — 350k is huge but still thin on rare motions (one clip for a 0.5 m vault); object physics isn't explicitly modeled.
• Visual planning — smart object relies on privileged sim info (ground-truth object poses); a real robot only has noisy RGB-D/SLAM, so visually-driven kinematic planning is future work.
• Physical awareness — a kinematic planner can request physically impossible motions (self-collisions, unsafe flips); co-training planner + tracking policy could fix this.
• Retargeting — runtime retargeting is fast but low-quality; offline is accurate but takes months. Bridging this remains open.