How do you write the physics of a blob of dough or a pile of coffee beans? You don't. You represent the world as particles plus relations and let a graph network learn how neighbors push each other — the right structure as a powerful inductive bias.
Guest lecturer Yunzhu Li opens with a humbling question: how far are we from human-level robot manipulation? A person can roll dough into a snake, cut it, press it into a circle, and never once thinks about it. The robot cannot. And the reason cuts to the heart of this lecture.
When a robot picks up a rigid mug, an engineer can write the physics by hand: a mug is one rigid body with a position and an orientation, six numbers, and Newton's laws move it. That is the world of Lectures 12–14 — analytical, physics-based models, and they are excellent when the object obeys clean equations.
Now hand the robot a fistful of dough. There is no "the position of the dough." It squishes, folds, sticks, tears. A pile of coffee beans is worse: a thousand little grains that flow like a liquid one moment and jam like a solid the next. No tidy rigid-body equation describes either. The classical toolbox simply has nothing to write down.
Here is the key bet, the thesis of the entire lecture. Instead of treating the dough as one inscrutable blob, represent it as thousands of tiny particles, and learn a single rule for how one particle pushes its neighbors. Apply that same local rule everywhere, every step, and the global behavior — flowing, folding, flattening — emerges. The right structure (a graph of particles) is a powerful inductive bias: a built-in assumption that makes learning generalizable and sample-efficient.
This is a pile of particles. Nobody wrote "dough equations" — each particle only knows to gently push the neighbors touching it. Press Poke to shove a region; watch a global ripple emerge from purely local rules. Slide stiffness to make it floppy (dough) or springy (granular).
That is the entire philosophy in one picture. The robot never models "dough" as a thing. It models a particle and its neighbors — the simplest possible interaction — and trusts that complex behavior is just that interaction, repeated. By the end of this lesson you will be able to build the message-passing step that learns it.
Before committing to particles, Li lays out the menu. To plan, a robot runs a loop: observe the world, imagine the effect of an action with its model, pick the best action, act, observe again. Everything hinges on that model. There are three families, and each makes a different trade.
Write the equations by hand. Pros: excellent generalization — physics is physics, so it works far beyond any dataset. Cons: it requires the full state (every particle's exact position) and it only covers systems you can actually write down. Dough? Beans? You're stuck. There's also a hidden tax: state and parameter estimation — you must measure all those positions and physical constants from messy sensors.
Encode the camera image into a single latent vector z, roll it forward with a learned function, decode back to a prediction. This is the world model of MuZero and friends. Pros: learns from partial observations, handles unknown systems. Cons: the latent is unstructured — it's an opaque bag of numbers with no built-in notion of objects, parts, or where things are — and so its generalization is limited. Train it on five beans and ask about fifty: the vector has no idea what "more beans" even means.
Keep the scene as an explicit set of particles wired into a graph, and learn the local interaction. Pros: learns from partial observation, handles unknown systems, and — because it bakes in the structure of "stuff made of interacting parts" — it generalizes excellently and is sample-efficient. This is the lecture's choice.
| Model | Learns from pixels? | Handles unknown systems? | Structured? | Generalization |
|---|---|---|---|---|
| Analytical physics | × (needs full state) | × | ✓ | Excellent (where it applies) |
| Latent-vector learned | ✓ | ✓ | × | Limited |
| Particle + graph (this lecture) | ✓ | ✓ | ✓ | Excellent |
Notice the recurring formal skeleton Li keeps redrawing. A perception step turns observations into a state: state from observations and past actions. A dynamics step predicts the next state from the current state and action. A control step picks an action from the state. Three boxes. This lesson lives almost entirely in the dynamics box — learning the function that says "given the scene and the push, here's the next scene."
Why is the dynamics box the hard one? Perception you can borrow from vision; control you can borrow from the rest of the course (MPC, CEM, MPPI). But the dynamics of dough and beans is the part with no off-the-shelf answer — and getting its representation right is what unlocks the other two. A good structured dynamics model gives control something accurate to plan through, and tells perception what to even estimate.
So we choose particles. Why particles, and not, say, a mesh or a voxel grid? Because particles are the finest level of abstraction and the most general and flexible one. A liquid splashing, a pile collapsing, dough folding over itself — all three are just clouds of points moving. The same representation covers all of them. (This idea comes straight from particle-based graphics simulators; Li credits Macklin and colleagues.)
Concretely, the scene becomes a set of N particles. Each particle i carries a small feature vector — the data the model actually consumes:
Stack those for all N particles and you get the node feature matrix — a table of N rows by D columns. If position and velocity are 3 numbers each and there are a few attribute flags, D might be 8 or 10. That table is the state. There is no hidden "dough object"; the dough is literally these rows.
(N, D). Rows = particles. Columns = position, velocity, and material attributes. The tool the robot holds is just more particles, flagged as rigid and driven by the robot's commanded motion. The model never sees an image of "dough"; it sees this numeric table.
Why does going this fine-grained matter? Because at the particle level, the rules of physics are local and universal. Two touching fluid particles repel a bit and get dragged along together; two touching dough particles do almost the same thing with more stickiness. The complexity of "dough behavior" is not in any single particle — it's in the collective. Get the local rule right and the collective takes care of itself.
One honest question: the robot sees camera pixels, not particles. How does a depth camera image become this clean table of positions? That's the perception step — the box upstream of dynamics. In the original work, particles are sampled from a 3D reconstruction of the scene (point clouds from a depth camera, or a learned 3D representation). Each visible surface gets seeded with particles; the material flag comes from knowing what object you're manipulating. It's not free, and Chapter 10 points to the modern tools (Segment Anything, descriptor fields) that make it more robust. But for this lesson, assume perception hands us the particle table; our job is the dynamics that move it forward.
The other thing the table must include is the tool. The robot's gripper, roller, or knife is also represented as particles, flagged as rigid and driven directly by the robot's commanded motion. That's how an action enters the model: the action moves the tool particles, the tool particles push the dough particles through the same message-passing rule, and the dough deforms. The next chapter wires all of them together so that local rule has something to act through.
A pile of unconnected particles can't interact — we need to say who pushes whom. The answer is a graph. The graph has two pieces, and Li writes the state as exactly this pair: the set of nodes V and the set of edges E.
Why a radius? Because physical contact is local. A grain of rice doesn't push a grain on the other side of the bowl — it only nudges the grains it's touching. So we connect each particle only to its near neighbors. This keeps the graph sparse (each node has a handful of edges, not N of them) and encodes a true physical fact: forces propagate through contact, neighbor to neighbor.
One more subtlety: for a liquid, particles are constantly rearranging, so who is whose neighbor changes every frame. That means the graph isn't fixed — you rebuild the edges from the current positions at each step. Li calls this dynamic graph building, and it's exactly what fluids and granular piles need. For a rigid object, neighbors stay put, so the edges can stay fixed. The structure adapts to the material.
Each dot is a particle. We draw an edge between any two within the interaction radius (the faint circle around the highlighted particle). Slide r: too small and neighbors disconnect; too large and the graph saturates. The count of edges is shown live.
Now we have the full state object Li writes: a graph (V, E). Nodes are particles, edges are within-radius relations. The next chapter is the engine that turns this graph one step into the future.
Here is the engine. The Graph Neural Network (GNN) takes the current graph and predicts every particle's next state in three moves. Li writes the two learned functions explicitly: an edge function and a node function. Walk through it with the data flow in hand.
For every edge connecting a sender particle to a receiver particle, compute a message. The message is a small learned function (a multilayer perceptron, an MLP) of the two particles' features:
Here a is the sender's feature vector and b the receiver's. The message is a vector — think of it as "how hard, and in what direction, does a want to nudge b." Its dimension is whatever the MLP outputs, say 64 numbers.
Each receiver particle collects all the messages arriving on its incoming edges and sums them. The sum is the particle's total "felt push" from its neighborhood:
Why sum, and not average or concatenate? Because a sum is permutation-invariant — it doesn't care what order the neighbors arrive in, and it works for any number of neighbors. Three neighbors or thirty, the aggregation is the same operation. Hold that thought; it's the secret to the whole generalization story in Chapter 5.
Finally, a second learned MLP takes the particle's own features plus its aggregated message and predicts its next state:
Do this for every node and you've produced the entire next graph. Repeat the three steps to roll the simulation forward in time.
fedge and fnode. That's it. They are shared across every edge and every node — the same tiny network runs at particle 1 and at particle 9,000. Training minimizes the squared error between the predicted next positions and the true next positions, summed over time: a simple supervised regression, but on a structured prediction.
Let's make the data flow fully concrete. Say the scene has N = 500 particles, each with a D = 8-dimensional feature (3 position, 3 velocity, 2 material flags). The radius graph produces, say, E = 3,000 directed edges. Walk the tensors:
500 × 8 matrix in.fedge, get a message — result is a 3000 × M matrix, with say M = 64.500 × 64 matrix of aggregated "felt push" — one row per particle, regardless of how many edges fed it.500 × 72), run fnode, predict the next position/velocity — back to 500 × 8.Notice the only place the number 3,000 appears is the message stage; it vanishes at aggregation. And nowhere does 500 appear inside the MLPs — they operate row by row. That's the whole trick, visible in the shapes.
Let's build the heart of it — the build-the-graph-and-aggregate step. You'll construct the neighbor edges from positions and sum each node's incoming messages. This is exactly Steps 1–2 above, the part where structure does its work.
This is the payoff chapter — the reason the lecture bets on graphs. The claim is bold: a model trained on small scenes works on much larger ones it has never seen. In the paper, a fluid model trained on a few hundred particles predicts a splash with twice as many particles, accurately. Nothing about the architecture changed. Why does that work?
Two architectural facts do all the heavy lifting, and both come from Chapter 4.
The edge MLP and node MLP are shared. There is no "particle 7's parameters." Every edge uses the identical fedge; every node uses the identical fnode. So the model learns a local interaction rule, not a specific scene. Add a thousand more particles and the rule still applies — there's simply more places to apply it. The number of learned parameters does not depend on N at all.
Because aggregation is a sum over whatever edges exist, a node with more neighbors is handled by the exact same code path as a node with fewer. The model is permutation-equivariant: relabel or reorder the particles and the predictions just come back relabeled the same way — the physics doesn't care which particle you called "number 1."
Contrast this directly with Lecture 15's latent-vector world model. There, the entire scene — however many objects — is squeezed into one fixed-length vector z, say 256 numbers. Three things break.
The latent model isn't wrong — it's the right tool when you can't get particles (raw pixels, abstract games). But when the world really is a pile of interacting stuff, throwing away that structure throws away the very thing that makes the problem learnable from little data.
The same two facts buy sample efficiency. Because the model only has to learn one small local rule — not a giant lookup over whole scenes — it needs far less data. In RoboCook, each tool's dynamics is learned from just about 20 minutes of offline interaction. That's the difference between a robot you can deploy and one you can't.
Let's prove permutation-equivariance to ourselves. We'll run the message-passing update on a small particle set, then shuffle the particle order and run it again. If the architecture is doing its job, the second result is just the first one shuffled the same way — bit for bit.
The same message-passing engine has to handle a rigid box, a sloshing fluid, and a lump of plasticine. They behave nothing alike. Li's insight is that the engine stays the same — only the graph structure and a few modeling choices change per material. This is the DPI-Net paper (Learning Particle Dynamics, ICLR 2019).
| Material | Graph structure | Motion character |
|---|---|---|
| Rigid objects | Hierarchical modeling; particles share a fixed body | Global motion (the whole body moves together) |
| Deformable (plasticine, dough) | Hierarchical modeling | Local motion (parts move relative to each other) |
| Fluids / granular | Dynamic graph building (rebuild edges each frame) | Local motion, constantly rearranging |
Two ideas earn a closer look.
For a rigid body, every particle moves identically — but message passing only spreads information one neighbor-hop per step. To tell a particle on the left edge that the right edge just moved, the signal would need many steps to crawl across. The fix: add a hierarchy. Group particles, route messages up to a coarse "summary" node and back down. Now global information (the body's overall motion) propagates in a couple of hops instead of dozens. It's the difference between whispering down a line and using a loudspeaker.
For a rigid object the neighbor relationships never change, so build the edges once. For a fluid they change every instant, so rebuild them every step from the current positions (Chapter 3's dynamic graph building). Same engine, different graph-construction policy — chosen to match how the material actually moves.
There's a subtlety in how the tool meets the material. The graph actually carries different relation types: particle-to-particle edges within the dough (how the material holds together) and tool-to-particle edges (how the rigid tool pushes the material). Each relation type can have its own learned edge function, but the aggregation and node update stay shared. So the same message-passing machinery models "dough sticking to itself" and "press shoving dough down" without any special-case code — just a flag on the edge telling fedge which interaction it is.
This is the deep reason one architecture spans rigid, fluid, and deformable: the differences between materials live in the graph (which edges exist, how often they're rebuilt, whether there's a hierarchy) and in cheap per-edge type flags — not in the engine. The engine is always: message, sum, update.
Let's feel how a learned local rule rolls forward in time. We'll run a short particle simulation where each step applies a simple neighbor-interaction (repel if too close, attract if a bit far — a stand-in for the learned fedge) and check the collective settles into a stable, spread-out structure rather than collapsing or exploding.
A dynamics model alone is just a predictor. The point is control: use the learned simulator to choose actions. The trick is closed-loop, and it's the same model-predictive control (MPC) idea from the rest of the course — now powered by a learned GNN instead of hand-written physics.
Two details make this work. First, the learned model is naturally differentiable — it's a neural network — so you can optimize the action by gradient descent through the predicted rollout, not just random search. Second, because the model is never perfect, you execute only the first action, then re-sense and re-plan. Errors don't compound because you keep correcting.
There are two broad ways to find the best action, and the lecture uses both. Sampling-based MPC draws many random action sequences, rolls each through the model, and weights them by how good their predicted outcome is — this is the family of the Cross-Entropy Method (CEM) and Model-Predictive Path Integral (MPPI), and a more recent variance-optimal variant, CoVO-MPC. Gradient-based MPC instead differentiates through the learned rollout to directly descend on a better action. The crucial enabler for both is the GPU: it runs thousands of imagined rollouts (or gradient steps) in parallel, fast enough to re-plan in the control loop. The learned simulator is just a batched forward pass, so this parallelism is natural.
| If you plan… | Open-loop (commit to the whole plan) | Closed-loop MPC (replan each step) |
|---|---|---|
| Model is perfect | Works fine | Works fine (overkill) |
| Model is slightly off | Drifts: small errors accumulate over the rollout | Stays on target: each step re-grounds in the true state |
| World gets disturbed | Fails — never sees the disturbance | Absorbs it — re-senses and corrects |
The flagship demo, RoboCook (CoRL 2023, Best Systems Paper), manipulates real dough with a toolkit of fifteen 3D-printed tools — grippers, rollers, presses, a knife, a circle cutter. Two nested decisions:
It's worth seeing exactly how the low-level model learns, because it's the same regression from Chapter 4 with one twist. Take the current dough state and a particular tool. Sample a random action for that tool. Run the GNN forward to predict the next dough state. On the real robot, actually apply the action and observe the true next state. The loss is the difference between predicted and true particle positions, and you backpropagate it into fedge and fnode. The twist: the model is tool-conditioned — the tool's identity and geometry are part of the input, so one model handles many tools, and a roller pushes differently than a press because their particles differ. Just 20 minutes of this random poking per tool is enough.
There's a quiet question we've dodged: how many particles should represent the scene? Li argues the most effective representation is not fixed — it must be dynamic. The testbed is object-pile manipulation: push a pile of granola, rice, carrots, or candy into a target shape.
Represent the pile with too few particles and the model is ineffective — it's too coarse to capture the shape you need. Too many and it's inefficient — planning over thousands of particles is needlessly slow. A good resolution is both minimum (for efficiency) and sufficient (for the task).
The fix (Dynamic-Resolution Model Learning, RSS 2023): pick the abstraction level that maximizes task reward, found via Bayesian optimization — a sample-efficient search that tries a few candidate resolutions, sees which plans best, and homes in on the sweet spot. A resolution regressor reads the current scene and the goal and proposes how fine to go; then the usual GNN forward-prediction and trajectory optimization run at that resolution. In the real world this pushes piles of granola, rice, carrots, and candy into letter shapes.
Time to see the whole engine breathe. Below is a small particle scene wired into a contact graph. Press Run one step and watch the three moves of message passing animate, particle by particle: edges light up as messages flow, each node glows as it aggregates its felt push, then every particle nudges toward where the rule sends it. Hold the controls and break it.
Each dot is a particle; lines are contact edges. A step computes a message on every edge, sums them at each node (the node briefly glows by how much it felt), and moves each particle by its update. Crank the radius up and the graph saturates; crank coupling up and it gets jittery.
The most instructive thing to notice: nothing in this loop knows the scene's size or shape. The exact same per-edge and per-node operations run whether there are twelve particles or twelve thousand. That invariance — visible right here in the animation — is the generalization story of Chapter 5, made concrete.
You now own the structured-world-model recipe end to end: represent the scene as particles, wire them into a contact graph, learn a local interaction rule with a GNN, roll it forward to simulate, and plan actions through it with MPC — choosing tools and motions to reshape dough, fluids, and piles.
Be honest about what's hard. The structured model assumes you can get particles — a clean 3D representation of the scene — which leans on perception that can be brittle in clutter or occlusion. Long horizons still accumulate prediction error. And building the right graph (radius, hierarchy, edge types) is hand-engineering, not learned. Li's "future directions" slides target exactly these gaps:
| Box | Open problem | Direction Li points to |
|---|---|---|
| Perception | More generalizable particle/keypoint extraction from pixels | Visual foundation models (Segment Anything, DINOv2); descriptor & mask fields (D³Fields); 3D keypoint tracking |
| Dynamics | Better sample efficiency & generalization; physics-informed structure | Physics-integrated representations (PhysGaussian = physics on 3D Gaussians) |
| Control | Guarantees on safety and optimality | Sparse neural dynamics for tractable, provable model-based control; CoVO-MPC |
The throughline: keep the structure (it's what gives generalization and efficiency) while making the perception that feeds it more robust and the control that uses it more principled.
An unstructured black box can learn dynamics, but it must rediscover, from data alone, that the world is made of consistent interacting parts. The structured model assumes that and cannot violate it — so it generalizes from a few hundred particles to thousands and learns from minutes of data. The right structure is not a detail; it is the inductive bias that does the heavy lifting.
This lesson is part of the CMU 16-831 series. Continue with the neighbors that bracket it, and the companion Gleams that expand each thread:
"Don't model the dough — model a particle, and let the dough emerge."