A world model lets an agent test actions in its head before it commits. A robot must fetch a cup without toppling a tower: reacting knocks it over, imagining first does not.
Learn what a world model is, what it must represent, predict and learn from interaction, and why agents that face new situations need one.
Press Run with the robot set to Reacts. Then switch to Imagines first, press Run again, and compare what each one does about the tower. Then we build, piece by piece, everything underneath: what the robot must remember, how it predicts, and how acting brings back evidence.
You need no prior knowledge of machine learning. We explain every word the first time we use it, and we build the rest from zero.
Based on Lecture 1 of CIS 6280: World Models (Fall 2026, University of Pennsylvania), taught by Jiatao Gu with teaching assistants Mutian Tong, Yong-Hyun Park, Enxin Song and Xinyue Ai. Lecture slides (PDF). Every idea is re-taught here in our own words and devices. This is the first lesson of a series that follows the course.
A toy scene made for this lesson, in centimetres, not the lecture’s robot. The arm and the tower are each 8 cm wide. A straight reach clears the tower from 8 cm off the line. The arc around needs 15 cm of room on one side of the table, and moving the tower first always works. The three plans take 3, 5 and 7 steps.
Chapter 0
See why one wrong action costs a robot far more than a software agent, and meet the nine words the whole course uses
A coding agent is working in a software project. It edits a file and runs the tests. A red failure appears within seconds. The agent reads the message, undoes its edit with one command, and tries again. Nothing is lost.
Now picture a robot arm in a kitchen. It reaches for a glass and closes its gripper a little too hard. The glass breaks. No command brings the glass back.
Both agents do the same kind of thing: act, look at the result, act again. But one wrong action costs them very different amounts. That gap is the reason this course exists.
A world model is a learned model of how a world works. An agent asks it a question such as “what happens if I do this?” and gets an answer before it acts. Chapter 2 gives the lecture’s own definition. First we need the words the syllabus uses, because the rest of the lesson leans on them.
The lecture opens with a short story, and the logistics wait for Chapter 10. In November 2025 the instructor wrote a proposal for a new course on generative models, which are models that produce new images, text or video. Two excellent courses on that topic already exist at Penn: Deep Generative Models (Fall 2025) and Discrete Generative Models (Fall 2026).
Meanwhile the phrase “world models” kept appearing everywhere. So the slide asks: why not go straight there? The lecture then shows an AI assistant searching for a comparable course at other universities, and the punchline is a meme caption: “Fine, I’ll do it myself.” The question the lecture really starts with comes next: what is a world model?
The lecture plays a clip of a coding agent at work inside a software project, the OpenAI Codex app. It asks you to watch for four things: what is visible, what can be changed, what gives feedback, and what can be undone. The slide names the four answers.
The state is legible: the files are text, so the agent can read all of them, and the observation almost equals the state. The tools are explicit: each action is a named command. The tests, which are small programs that check the code, are fast: feedback arrives in seconds. And many edits are reversible: version control, a tool that keeps every past copy of a file, restores any earlier one.
The lecture then shifts to the physical world, with a demo of a robot from V-JEPA 2 (Meta AI, 2025) that acts from a current scene toward a goal. Its four words are the mirror image. The robot sees partial views, because a camera sees only part of the scene. It meets contact, where touching things changes them in ways that are hard to see. It lives with delay, because sensing and moving take time. And it faces real consequences, because a dropped glass stays dropped.
Try the difference with a toy task of eight steps. Three of the steps contain a mistake: steps 3, 5 and 7. Run it in each world and read the counters.
Each tile is one step of a task. Choose a world, press Run, and watch what each mistake leaves behind. Then run the other world.
Illustrative toy, not data from the lecture. The mistakes sit at steps 3, 5 and 7. In the software world each mistake is caught by a fast test and undone at the price of one extra step. In the robot world no step can be undone. The four property words under the tiles are the lecture’s own.
Count what the run gave you. In the software world, 3 of the 8 steps were mistakes and all 3 were undone. The task took 11 steps (8, plus 1 for each undo) and left nothing broken. In the robot world the same 8 steps finished in 8, which looks faster. But 3 mistakes are still in the world.
This is the failure mode, in three beats. The symptom: an agent that works well in software is moved onto a robot and keeps breaking things. The measurement: do not count steps, count lasting mistakes, which here is 3 of 8. The fix: move the trial and error to a place where it can be undone.
The rest of the lecture asks three questions about such a model: what it must represent, what it must predict, and how interaction brings back evidence. Chapters 5 to 8 take them in order, after four chapters on why the whole idea is needed.
Chapter 1
Weigh five claims about what a learner must discover, then test the last one yourself by choosing which details of a falling bottle to predict
Someone knocks a bottle off a shelf. Before it lands, your hand is already moving. You did not work out the pattern of every drop that will splash out. You worked out where it will be, and when, and you put your hand there.
That small act hides a big question. How does a learner come to know what will happen next, in a situation it has never seen exactly before? The lecture poses it as five hypotheses, which are claims we hold up for testing. Each one comes from a short clip of a well known researcher. The lecture summarises them in one table, and each claim has a plain meaning.
| Speaker | The claim, as the lecture states it | In plain words |
|---|---|---|
| Ilya Sutskever | Prediction may reveal the process behind data. | To guess well, a learner must work out the rule that made the data. |
| Jitendra Malik | Web data omits much embodied experience. | A body learns things that nobody writes down or films. |
| Richard Sutton | Interaction reveals what actions actually do. | To learn what an action does, you must act and look. |
| Fei-Fei Li | Physical action depends on spatial structure. | You must know where things are and how they touch. |
| Yann LeCun | Useful prediction need not reproduce every detail. | A good guess keeps what matters and drops the rest. |
Now take them one at a time. The first clip is from a 2023 NVIDIA GTC conversation. Sutskever argues that prediction can be a form of compression, which means writing something down in a shorter form that still lets you rebuild it. Think of the last page of a detective novel. To predict the next word there, you must have worked out who did it. The shortest description of a story is the plot that made it. So a learner that predicts well may be forced to discover the hidden process behind the data.
The second clip is Malik, in a 2023 Berkeley talk. His point is about what is missing. The web holds text and pictures that people chose to put there. A child’s first years are different. The child grabs, drops, tips and tumbles. Most of that experience was never digitized, so a model trained on the web never sees it.
The third is Sutton, on a 2025 podcast. He stresses learning from the consequences of action. Watching a million videos of people pouring water does not tell you what happens when you tilt the jug a little too far. Only trying it does.
The fourth is Fei-Fei Li, in a 2024 TED talk, on geometry and 3D relations. Before a glass falls, you already use spatial structure: the glass sits at the table’s edge, with nothing under it, above a hard floor. Those are relations between things, not pictures of things.
The fifth is LeCun, at a 2026 summit, with the bottle-drop test. The slide asks you to choose one consequence of dropping a bottle, and then to name what you did not need to predict. That last question is the one we can play with.
Here is the bottle, with eight details a predictor could try to get right. Each detail has an effort cost, in made-up units that grow with how hard the detail is to compute. The three tasks need different details. Switch the task, then switch details off until the prediction is just enough.
Teal bars are the details your prediction keeps. A pink diamond marks a detail the task needs. Start with everything switched on, then cut.
Illustrative: the effort units (2, 1, 2, 6, 24, 1, 4 and 3, adding to 43) and the choice of which details each task needs are ours, made up to show the idea. They are not measurements, and not the lecture’s numbers. The test itself, “what did you not need to predict?”, is the lecture’s.
Work the first task by hand. To catch the bottle you need two details: where it lands and when, and how it is tilted. Those cost 2 + 1 = 3 units. A predictor that keeps all eight details spends 43 units, which is 14.3 times as much. And 40 of those 43 units, about 93%, go to details the hand never uses.
Now try “decide to mop”. It needs whether the bottle breaks and roughly how far the liquid spreads: 2 + 6 = 8 units. Where it lands is not needed for this task. The droplet positions, which cost 24 units alone, still do not matter.
This is the failure mode, in three beats. The symptom: a model draws a beautiful, exact splash but cannot tell you whether to catch the bottle. The measurement: count the share of effort spent on details the task never uses, here 93%. The fix: let the task decide what to keep. Chapter 5 returns to this idea as “a useful representation keeps the differences that change predicted consequences”.
The lecture closes the set with one question: what lets an agent act beyond familiar situations? Each claim answers a part. Prediction forces the learner to find the process (Sutskever). Interaction supplies experience the web lacks (Malik) and shows what actions do (Sutton). Spatial structure is what the prediction is built from (Li). And the best predictions are selective (LeCun). The course’s answer is a learned world model. The next chapters build it up.
Chapter 2
Predict first, find the rule your eyes used, and meet the lecture’s one-sentence definition of a world model
Four toy blocks are stacked on a table. Each one sits a little to the side of the one below. You have not touched them. Will the stack stand or fall?
The lecture asks exactly this, with a video of a block stack and a rule for the room: pause first, predict one outcome, and name the visual relation you used. The clip comes from research on how people judge whether a stack is stable (an eLife paper from 2024). Most people can also say why: “the top is leaning too far out”.
The relation is support: does what is above sit on top of what is below? Physics gives support a precise form, and it needs one idea, the centre of mass. It is the balance point of an object. Lay a ruler on one finger and slide it until it balances. The spot over your finger is the ruler’s centre of mass.
For a group of equal blocks, the balance point is just the average of their centres. Two blocks centred at 2 cm and 8 cm balance at 5 cm. The rule for a tower follows: the balance point of everything above a joint must sit over the face of the block below it. If it sits past the edge, that part of the tower tips off.
Here is the worked example with the default tower below. Each block is 10 cm wide. The block centres sit at 0, 2, 5 and 9 cm, measured along the table from the middle of the bottom block. We check each joint from the top down.
So this tower falls, although every upper joint holds. A failure low in the stack carries everything above it. That is why the check runs from the top down and stops at the first failing joint.
Set how far each block sits from the one below, then commit: press It stands or It falls. The balance points appear only after you commit.
Equal blocks, each 10 cm wide, with offsets measured from the block below. The rule is the one worked above: the balance point of the blocks above a joint must lie within 5 cm of the middle of the block below. Real towers add friction and wobble, which this toy leaves out.
Try a few towers. Notice how often your eye was right, and how often it missed a tower that was fine at every joint but one. Press New tower to deal a random one. The score in the chip counts your commits.
This is the whole rule, as a small function. It takes the block centres, a list of numbers of shape (n,), and returns the first failing joint, or nothing.
pythondef first_failing_joint(xs, half_width=5): # xs: centres of the blocks, bottom to top, in cm for i in range(len(xs) - 1, 0, -1): # joints, top down above = xs[i:] # every block above joint i com = sum(above) / len(above) # their balance point if abs(com - xs[i - 1]) > half_width: # past the face below? return i return None print(first_failing_joint([0, 2, 5, 9])) # 1 (the bottom joint fails) print(first_failing_joint([0, 1, 2, 3])) # None (it stands)
Look at what this function is. A list of numbers goes in: the positions, which describe the world now. A verdict comes out, which describes how the world may change. That small shape, now in and change out, is the lecture’s whole definition.
Read it in two halves. To represent is to hold the world as information the model can use, here the four block positions. To predict is to say how that information changes. The last four words matter most for agents: including the consequences of interaction. If the agent pushes the top block, the world changes in a way it would not on its own. Chapter 4 and Chapter 8 return to this.
The tower rule is a tiny hand-written world model. Real world models are learned from experience, because nobody can write rules like this for every scene. But the pattern is the same, and it is worth keeping in mind as the cases get bigger.
The failure mode, in three beats. The symptom: you say “stands” because the stack looks neat, and it falls. The measurement: the margin at each joint. Here the bottom joint is 0.33 cm over its limit, small enough that the eye cannot see it. The fix: check every joint, top down, against the one rule.
Chapter 3
Learn why fast feedback can replace prediction for some tasks, and measure the exact moment it arrives too late
Picture a walking robot with many legs on a rough floor. Suppose it has no map of the room and no picture of the future. Each time a leg bumps a ledge or slips, a quick reflex in that leg adjusts the next step. The robot can still get across.
The lecture shows exactly this, in an MIT archive clip of a many legged robot from Rodney Brooks’s work. The slide asks you to watch for behaviour produced by fast feedback instead of a complete replica of the world inside the machine. Brooks’s lesson: robust behaviour can come without a central model of the world.
Feedback is information the world sends back after you act. A thermostat shows the idea. It reads the temperature, turns the heater on if the room is cold, and reads again. It never predicts tomorrow’s weather. It only reacts: it compares what it sees now with what it wants, and corrects. The slide draws it as three words: observe, react, observe again.
As code, a reactive controller is one line. It takes what the sensor shows now, the error, and turns it into an action.
pythondef react(position, target, gain=0.5): return gain * (target - position) # act on the error you see now
The slide gives three conditions under which this is enough: the same objective (the target does not change), rapid feedback (the answer comes back quickly) and cheap correction (a wrong move costs little to fix). A walker like this meets all three. The goal is always “keep walking”. The leg feels the ledge at once. A stumble costs one swing.
Now the lecture sets a new task: retrieve the cup without knocking over the tower. Split it into two parts, as the slide does. The goal is to retrieve the cup, and the robot has done that before. The constraint is to keep the tower standing, and this exact arrangement of cup, tower and arm is new. The slide’s own summary: the objects are familiar, but this combination of arrangement, goal and constraint is new.
The robot reaches. Its arm touches the tower. The camera and the arm’s sensors report contact. The controller decides to stop. All of that takes time. The slide spells out the order of events: reach, contact, collapse, feedback. By the time the robot gets feedback, the tower is already falling. In the lecture’s words, feedback can improve the next attempt, but it cannot undo this one.
So two of the three conditions fail. The objective is new, and correction is not cheap, because a toppled tower stays toppled. The third, rapid feedback, is a matter of numbers, and we can measure it. Suppose the arm moves at speed v toward the obstacle. At the moment of contact, the sensors need a delay d before the controller can react. During that delay the arm keeps pushing. So the arm travels v × d past the point of contact.
The obstacle has a margin, the distance you may push it before something breaks. Reacting works only if v × d is no more than the margin. That one inequality is the whole chapter.
The warm line is how far the arm has pushed past contact, over time. It stops when feedback arrives. The dashed line is the margin. Choose an obstacle, then drag speed and delay until the arm stops in time.
Toy numbers made for this lesson. The arm moves at a steady speed until feedback arrives, then stops at once. The margins (12 cm for a bump, 1.5 cm for the tower) are illustrative. The inequality, travel past contact = speed × delay, is exact for this setup.
Work it by hand. At 20 cm/s with a 300 ms delay, the arm travels 20 × 0.3 = 6 cm past contact before it can react. The sturdy bump has 12 cm of margin, so 6 cm is fine: reacting is enough. The tower has 1.5 cm, so 6 cm is four times too much. The tower tips after 1.5 / 20 = 75 ms, long before feedback at 300 ms.
Now find the fix a reactive robot would have. The safe speed is the margin divided by the delay: 1.5 / 0.3 = 5 cm/s. Slow enough, and the arm stops in time. But a robot that creeps at 5 cm/s through every task is not a good robot, and cutting the delay to zero is not possible. A real sensor, a real computer and a real motor all take time.
This is the failure mode, in three beats. The symptom: the robot fetched cups perfectly in every trial without the tower, and the first trial with the tower ends in a collapse. The measurement: travel past contact against margin, here 6 cm against 1.5 cm. The fix: do not wait for contact. Predict that the reach will touch the tower, and choose a different plan first. That needs a model of consequences, which is the subject of Chapter 4.
Chapter 4
See why a learned reflex cannot cover every combination of scene, goal and constraint, and why reusable consequences can
You stand at a kitchen shelf. Your goal is the top cup. A tower of blocks your child built stands in front of it. You have three ways to act. You can lean around the tower and reach the cup from outside. You can reach straight over the tower. Or you can slide the tower aside first.
You do not need to try all three to know what each does. You already know that a hand over a tower tends to knock it down. The lecture uses the same scene with a robot and a cup, and it makes the point on one slide: the future depends on what the robot does.
| Action | What happens | Result |
|---|---|---|
| Reach outside | The cup moves. The tower stays. | Cup reached, tower untouched |
| Reach across | The arm makes contact. The tower falls. | The constraint breaks |
| Move the tower | The layout changes. The path opens. | A new, clear path to the cup |
The same scene unfolds three ways. This is the first thing to take from the chapter. A single picture of the scene cannot say what happens next. It must be paired with an action.
A fair question follows, and the lecture asks it directly. Why not skip the model of the world and train a learned response: a function that takes the current scene and the task and outputs the safe reach? Roboticists call such a function a policy. You feed it many examples of scenes and good actions, and it learns to produce good actions for scenes like them. The ability to give good answers on cases it has not seen exactly is called generalization.
The slide’s answer is honest. In principle, yes. With enough representative experience, a learned response may generalize to a situation it has not seen exactly before. But training cannot densely cover every combination of scene, goal and constraint.
That word combination carries the weight. The cup-and-tower task has three things that can vary: the arrangement of the objects, the goal, and the constraint. If each can take a handful of values, the number of whole situations is a product. And products grow fast. Four things with five values each give 5 × 5 × 5 × 5 = 625 situations. Six things with eight values each give 86 = 262,144.
The lecture then draws two columns. On the left, a familiar task: the same situation repeats. The robot recognizes the task and reuses the action that worked before. That is fine, and it is what a learned response does well.
On the right, a new goal or constraint: the useful action may change. Now the robot reuses known consequences instead. It remembers what a reach over a tower does, and what pushing a block does. Then it chooses for the new goal. The slide puts it in one line: the model predicts, and the agent chooses again.
Here is why this scales. The consequences are pieces. There are only a few of them (k things, m values each, so k × m facts), and every demonstration teaches several at once. A new situation is a new arrangement of pieces the robot already knows. The toy below counts both. It is a simplified model with a strong assumption, which we state in its fine print.
The grid has one square for an equal share of all situations; filled squares were met in a demonstration. The bars show how often a brand-new situation is covered. Raise the number of things that vary.
A toy with a strong assumption, made for this lesson and not from the lecture. Each demonstration is a situation drawn at random, every situation equally likely, and consequences are independent facts that compose. Real learned responses also generalize to situations that are close to ones they have seen, which this count ignores. The formulas are exact for the assumption: the chance a new situation was met is 1 − (1 − 1/mk)N, and the chance every fact is known is (1 − (1 − 1/m)N)k.
Work the default by hand. With k = 4 and m = 5 there are 625 situations. After N = 100 demonstrations, the chance that a fresh situation was one of them is 1 − (1 − 1/625)100 = 1 − 0.852 = 15%. Now count facts. There are only 4 × 5 = 20 of them. Each particular fact is missing from all 100 demonstrations with probability (4/5)100, which is about 2 in 10 billion. So essentially every fact is known, and a new situation made of known facts is covered about 100% of the time.
Now push the sliders. Raise k to 6 and m to 8. The count of situations jumps to 262,144, and 100 demonstrations cover under 0.04% of them. Even 10,000 demonstrations cover only about 4%. The facts, still only 48, stay fully known. The gap between the bars is the lecture’s argument in numbers.
The failure mode, in three beats. The symptom: a robot that scores 99% on its test set fails on the very first new combination, such as this tower with this cup. The measurement: the share of test situations that were exact repeats of training ones. The fix: learn the consequences of actions as reusable parts, and choose among actions at run time. That is what the lecture means by “a world model supports reasoning about consequences”.
The last slide of this part names the three questions that sit inside the cup-and-tower task, and they organise the rest of the lecture. Representation: what about the scene must the robot keep track of? Prediction: what happens after each possible action? Interaction: how does acting reveal new evidence, and change the next step?
A world model links all three. The lecture separates them only to look at each one. Chapter 5 takes the first, Chapters 6 and 7 the second, and Chapter 8 the third.
Chapter 5
Learn why a camera frame is not the world, how history fills the gap, and why prediction teaches a model which differences matter
The lecture’s first thread is representation, and it opens with a question: what can experience teach the model to carry forward? Start with what the robot actually receives. It does not receive “the whole world”. It receives one camera image of its table.
The slide labels what that image is made of: the appearance of things, the viewpoint they are seen from, the visible geometry, and the image boundary where the picture ends. In the lecture’s words, this image is one representation of the scene, shaped by its sensor and its viewpoint. A representation is a stand-in for the world, written in a form a model can use.
The next slide lists what that single frame leaves out. It cannot show motion, because a still picture has no speed. It cannot show contact history: did the gripper touch something a moment ago? And it cannot show what lies outside the view. The slide’s fix is to combine the current frame with history, into a working representation supported by history. This is the memory from Chapter 0.
Here is the simplest case of why history helps. A ball moves along a line, 3 cm each step. The camera sees it with a small error of its own, about 0.8 cm. You must say where the ball will be 3 steps from now, when it is behind a wall.
With one frame you only know where the ball is, not where it is going. The best you can do is guess it stays. With two frames you can draw a line through the two positions and extend it. Take this run. The ball is truly at 0 cm one step ago and at 3 cm now, and it will be at 12 cm in 3 steps. The camera reports 0.2 and 3.4.
More frames average out the camera’s error. Slide the history in the device and read the error for each length.
Blue dots are camera frames, with the camera’s error. The warm line is fitted through the frames you keep and extended behind the wall. The red segment is the miss. Drag the history length, then press New noise.
A toy made for this lesson. The ball truly moves 3 cm per step; each camera frame adds a random error of 0.8 cm. One frame predicts that the ball stays put. Two or more frames use a least-squares line through them. The bars are the mean miss over 600 simulated runs with fixed seeds.
Read the bars. One frame misses by about 9 cm, because it cannot see the speed. Two frames cut the miss to about 3 cm. By six frames the miss is under 1 cm, because the camera’s errors have averaged out. The cost is one more frame to remember each time. That trade, memory against accuracy, shows up in every world model.
For a ball on a line, we chose the variables by hand: position and speed. For a robot in a kitchen we cannot. The slide says it directly: we cannot hand-design every variable, so the representation must be learned. Representation learning turns experience into information the model can reuse in a new scene. The model learns it from many observations instead of from a person’s list.
But learning normally needs labels, which are correct answers supplied by people, such as “this is a cup”. Labels are slow and costly. The lecture’s next idea removes them: prediction can train a representation without manual labels. Think of a busy street crossing seen from above. To predict the next frame of the crowd, a model must track who is where, and how each person is moving. The slide’s key line: the future supplies the answer, which is what actually happened. The next frame is the label, and it arrives for free.
Now the lecture asks: what part of the future should the model predict? It gives two options. The model can predict the next observation, which is the future camera frame, and compare it with what the camera actually saw. Or it can predict a future representation: features computed from that future frame, compared in a learned feature space. A feature is a number that summarises some property of the scene, such as how far a block overhangs. The lecture’s summary: it can predict the next image, or a learned representation of that image. (Our JEPA lesson follows the second option all the way, and the course covers it in Lectures 5 and 6.)
Which one is right? The lecture’s answer is that either way, prediction teaches the model what matters. Its example: two towers with different support have different outcomes. For this tower, support is what separates “fall” from “stay”. And the follow-up slide makes the point sharper: change the colour of the blocks, and the geometry is unchanged, so the colour need not change the prediction. A useful representation keeps the differences that change predicted consequences.
Test that claim on towers. The device holds 60 stored towers with their outcomes, and 300 new towers to predict. For a new tower it finds the most similar stored one, in the representation you choose, and copies its outcome. Switch between three ways of measuring similarity and read the accuracy.
The left tower is new. The right tower is the most similar stored one, in the representation you pick. The right tower’s outcome becomes the guess. Press New tower a few times under each representation.
A toy made for this lesson. Each tower has three blocks with random colours and random sideways offsets; it falls by the balance rule of Chapter 2. Half of the 60 stored towers and half of the 300 new ones fall. “Raw pixels” measures distance over three colour numbers and two offsets, each scaled to the same range. The support margin is a hand-picked feature: how far past an edge the worst joint’s balance point sits. Real models learn such a feature from data instead of being given it.
Half of the towers fall and half stand, so a coin flip scores 50%. The colour-only run sits near that, at 54%. The raw-pixel run does better, at 68%, because the offsets are in there, but the three colour numbers keep pulling the wrong neighbours. The support margin gets every one of the 300 towers right, since it keeps exactly the one difference that changes the outcome and drops the rest.
This is the failure mode, in three beats. The symptom: a model that predicts the future well on towers it trained on, and badly when the blocks are painted differently. The measurement: accuracy under a change that should not matter, here from the support margin’s 100% down to 54% once colour dominates. The fix: a representation trained by prediction, so that colour, which never helps predict, is pushed out.
Chapter 6
Separate watching from intervening, see why one push can lead to several outcomes, and learn why that makes prediction a generative problem
A weather forecaster looks at a satellite map and says where the storm will move. Nobody on the ground needs to do anything. Now picture a child poking the top block of a tower and asking what happens if she pushes a little harder. These are both predictions, and the lecture’s second thread, prediction, begins by telling them apart.
The first kind asks what may happen next? The world changes by itself. The lecture’s examples: where will the storm move, what will the person in the video do next, will the motion of this object settle or continue. The system only watches. No action from our agent is needed.
The second kind asks what may happen next if the agent does this? The agent intervenes. The examples: what if the robot pushes, what if the vehicle accelerates, what if we apply a force in an experiment. The action becomes part of the information used to predict. The lecture’s summary is short: both are prediction, and only the second depends on an action.
As data flow the difference is one extra input. A watching model maps a state to the next state. An action-conditioned model maps a state and an action to the next state. “Conditioned on” just means “given”.
python# watching: state (D,) -> next state (D,) # intervening: state (D,) + action (A,) -> next state (D,) def predict(state, action=None): # D numbers describe the world now; A numbers describe what the agent does ...
The next slide complicates the picture. Take one tower, one visible history, one light push. The lecture shows three different pictures of what happens next: the tower barely moves, it falls left, it falls right. Same scene, same push, three outcomes. The slide’s explanation: the action changes the possibilities, but hidden factors can still leave several outcomes plausible. A hidden factor is something that affects the outcome but that the camera cannot see, such as how sticky the table is, or a slight lean in the top block.
So a single answer is the wrong shape for the question. Suppose the model must give one future. If it picks the most likely one, it is wrong whenever another outcome happens. If it averages the futures, it can produce something that never happens: half a tower leaning left and half leaning right, blurred into one picture.
The lecture’s answer: when several futures are possible, prediction becomes a generative modeling problem. A generative model does not return one answer. It returns a sample, which is one draw from the set of plausible outcomes, with likely ones drawn more often. Run it again on the same query and you may get a different draw.
The slide gives a name to one run: a rollout. One rollout is one sample. Repeat the same query, same scene and same push, and you reveal the plausible futures the model has learned. The slide also warns about what repetition can show. It reveals “the impossible futures it invents” too: in its example, a run where the blocks change shape.
Try it on a toy. A hidden stickiness of the table and a hidden lean decide what a push does. You choose the push, and you can see the real split of outcomes by running many rollouts.
Each press of Run once is one rollout: one sample. Filled bars are your tallies. Tick marks are the true probabilities. The row of towers shows your last rollouts.
A toy made for this lesson. A hidden stickiness h is drawn uniformly from 0 to 1, and the top block leans left or right with equal chance. The push knocks the tower over when push × (1.2 − h) reaches 4.5. The true probabilities come from that rule, exactly. “Imperfect model” makes the sampler invent an impossible tower (blocks that change shape) in 8% of rollouts, to show the failure the lecture names.
Work the default by hand. The push is 6. The tower falls when 6 × (1.2 − h) ≥ 4.5, which means 1.2 − h ≥ 0.75, which means h ≤ 0.45. Stickiness is drawn between 0 and 1, so it lands at or below 0.45 with probability 45%. When it falls, it goes left or right with equal chance, so each direction gets 22.5%. The tower stays upright the other 55%.
Now compare the two kinds of model. A model that gives one answer picks “stays”, the likeliest, and is right 55% of the time. Push up to 10 and the real split becomes 25% stays, 37.5% left, 37.5% right. The best single answer is now right only 37.5% of the time. A model that samples can reproduce the whole split, and only a model that samples can show you that the tower might go either way.
The failure mode, in three beats. The symptom: a planner trusts one predicted future, and the tower goes the other way. The measurement: run the same query many times and count the split, as the device does. The fix: model the whole distribution of futures, and then check samples for the impossible ones before you plan with them. Later lectures of the course (7 to 10) build the generative models that do this.
Chapter 7
Measure how tiny one-step errors pile up over a long rollout, and find how far ahead a model must look for the goal to matter
You give a friend directions by phone. Turn left at the bakery, then walk 200 steps. You are off by one degree at the first turn, which is nothing. But your friend now stands in the wrong place, and your next direction starts from where they are, not from where you meant them to be. Ten turns later, they are on another street.
That is a rollout with a mistake in it. A rollout runs a model forward step by step, and each predicted step becomes the input to the next. The lecture shows the effect in a clip from the 2018 paper World Models by Ha and Schmidhuber. A generated rollout of a driving game slowly leaves the road. The slide asks you to find the first error, then watch it become part of the generated history. Its caption: one-step accuracy is not enough when future predictions depend on earlier predicted frames.
Take the simplest case, a car on a gently curving road. The model predicts how the heading changes at each step. It is almost perfect: off by a bias of δ = 1 degree per step. Move the car 1 cm per step.
Check it the way a test set does: start from the true state, predict one step, compare. The heading is off by 1 degree, so the car lands 2 sin(0.5°) × 1 cm = 0.017 cm from where it should. Nothing could look better.
Now roll it out. Step 1 ends 0.017 cm off. Step 2 starts from that wrong place, with a heading that was already 1 degree wrong, and adds another degree of error: the heading is now 2 degrees off. After t steps the heading is t × δ off. And every step moves the car in a slightly wrong direction, so the position errors add up. Add up the lateral error 1 cm × (δ × 1) + 1 cm × (δ × 2) + … and you get about s × δ × t2 / 2, where s is the step length and δ is in radians. The error grows with the square of the steps.
Blue is the true road. Warm is the model’s own rollout, each step starting from its last prediction. The chart below shows the gap at every step. Drag the bias and the number of steps.
A toy made for this lesson. The true road’s heading is 0.35 sin(2πt/50) radians and each step is 1 cm long. The model knows every heading change exactly and adds a constant bias to each. The “one step” line starts from the true state each time. The exact errors come from the simulation; the t² formula is the approximation worked above.
Read the chart. The one-step line sits at 0.017 cm, flat, the whole time. The rollout curve bends up like a parabola. Halve the bias and the final error halves too, but double the steps and it quadruples. Sixty steps with a 1 degree bias cost 30 cm.
The failure mode, in three beats. The symptom: a model scores superbly on a one-step test, and its generated video leaves the road. The measurement: evaluate the gap along the rollout and not only at one step, here 13.9 cm at step 40 against 0.017 cm at step 1. The fix: train and test on rollouts, and keep the model from trusting its own mistakes. Later lectures (3, 11 and 13) return to this.
The lecture adds a second reason to roll out far: prediction must extend far enough for the goal and constraint to matter. The slide compares two plans for the cup-and-tower task. Grab now makes early progress, and the task fails. Move the obstacle first has a slower first step, and the goal and constraint are satisfied. The slide’s summary: the relevant horizon is long enough to reveal whether the task actually succeeds.
A horizon is how many steps ahead a planner looks. Give each plan a score at every step: progress is positive, and breaking the tower costs 8. A planner adds up the scores inside its horizon and picks the plan with the larger total.
Each cell is one step. Blue marks progress, red marks a broken constraint. The teal line is how far the planner looks. Drag it from 1 to 12 steps.
Illustrative scores made for this lesson, not from the lecture. Grab now: +1 for each of steps 1 to 4, then −8 at step 5 when the arm hits the tower, then nothing. Move first: 0 for steps 1 to 3 while the tower is moved, +1 for each of steps 4 to 7, then +5 at step 8 when the cup is fetched with the tower standing.
Work it by hand. Looking 4 steps ahead, grab now scores 1 + 1 + 1 + 1 = +4 and move first scores 0 + 0 + 0 + 1 = +1. The planner grabs. Looking 5 steps ahead, grab now adds −8 and drops to −4, while move first reaches +2. The planner switches. So the planner must look at least 5 steps ahead to see the collision at all.
Put the two ideas together. A longer horizon reveals the truth about the task, but a longer rollout piles up more error. Good world models need both: accurate long rollouts, and a horizon matched to the task.
Chapter 8
Treat a prediction as a proposal, update a belief when evidence returns, and see how many such updates teach the model better consequences
You press the button for the lift. You expect a light to come on. Three things can follow. The light comes on, and your expectation is confirmed. Nothing happens, and it is contradicted. Or an alarm sounds, and you are surprised. The lecture uses exactly these three words, and they are the third thread of the course.
The thread is interaction. The slide puts it in one line: prediction proposes a result, and the world returns evidence. Evidence is any new observation that makes a belief stronger or weaker. Interaction compares an expected consequence with an observed one.
The lecture frames it as two questions before an action and two after. Before play: what should change, and what should stay the same? After play: what actually changed, and where did the expectation fail? The match between those answers is the evidence.
The next slide repeats the split from Chapter 6. When the agent only watches, it asks what may happen next. When it can act, the current scene is not enough: it must ask what may happen next if the robot moves the tower. The world model predicts the consequence. The agent, with its goal, chooses the action. And the action changes which consequences matter without fixing one outcome.
That is why the lecture calls a predicted consequence “only a proposal”. Take “move the tower, and the path should open”. It is not yet observed. The robot executes the move and gets a new observation, which will confirm, contradict or surprise. In the slide’s words: prediction offers possibilities, and interaction supplies evidence.
The slide for the thread has three verbs, and the loop is simple. The robot acts. It observes what came back. It chooses again, and the observation becomes the starting point for the next prediction. If the path is open, it reaches for the cup. If the path is still blocked, it moves again or tries another side.
There is a number hiding in “the path should open”. Call it q, the model’s belief that the path opens, between 0 and 1. The camera is a noisy witness. When the path is open, it shows a gap with probability r, and when the path is blocked, it shows no gap with probability r. Say r = 0.9. How should the belief change after the camera speaks? Bayes’s rule answers: weigh each possibility by how well it explains what you saw.
Work it with a prior of q = 0.6, which is the belief you hold before the camera speaks, and a gap seen. The belief after the camera speaks is the posterior.
Now the opposite observation, no gap. Open and no gap: 0.6 × 0.1 = 0.06. Blocked and no gap: 0.4 × 0.9 = 0.36. The posterior is 0.06 / 0.42 = 0.14. The belief falls to 0.14, and the robot looks again instead of reaching.
This is the lecture’s “one interaction updates the situation”: observe, update, choose. And the slide adds a key detail: no retraining is required for this immediate update. The model’s weights stay put. Only the belief about this episode changes.
One episode is five cards, left to right. Below, the line is the model’s own estimate that moving the tower opens the path, as episodes pile up. The dashed line is the hidden truth.
A toy made for this lesson, not the lecture’s robot. In the hidden world, moving the tower opens the path 70% of the time. The model starts with no experience (an estimate of 0.50) and, after each episode, sets its estimate to (opens + 1) / (episodes + 2). The robot reaches for the cup when its belief after the camera reading is at least 0.80. The camera shows a gap with probability r when the path is open and no gap with probability r when it is blocked.
Press Run one episode and follow the cards. The first prior is 0.50. After one reading, the belief jumps to 0.90 or drops to 0.10 (with r = 0.9). Then press Run 20 more and watch the line. It wanders at first, because ten episodes tell you little, and it settles near the hidden 70%.
The last slide of the thread separates two time scales. This episode: update what is happening now, with no retraining. Across experience: learn better consequences. Its examples: a soft push shifts the tower, a hard push topples it, and building support first keeps it upright. Accumulated trajectories can improve future predictions.
Count it by hand with the device’s rule. After 10 episodes in which the path opened 7 times, the estimate is (7 + 1) / (10 + 2) = 0.67. After 40 episodes with 28 opens it is (28 + 1) / (40 + 2) = 0.69. That number then becomes the prior for episode 41. The per-episode update used no learning at all. The cross-episode update is learning, and it is what turns a vague prior into a good one.
The failure mode, in three beats. The symptom: the robot expects the path to open, never checks, and keeps reaching into a blocked path. The measurement: how often the observation contradicts the expectation, and whether a belief of 0.9 is right about 90% of the time. The fix: observe before choosing again, and let the evidence change the belief.
Chapter 9
Tour six real systems, from game worlds to weather and cells, and ask the same three questions of each
Open a weather app, and a forecast for ten days appears. Step into a video game, and the street ahead exists only because you turned the corner. Ask a driving team whether the car should slow down, and an answer comes from a replay of the same moment played two ways. These are all uses of the same idea. The lecture ends its theory with a tour of six of them, then returns to the three questions.
For each system we keep to what the lecture says and shows. The slides give names, organisations and years, and a one-line claim. We add plain explanations of the words, and nothing more about how any system is built.
Genie 3 (Google DeepMind) is the first. The slide says: build an interactive world, and step inside. As the user moves, what changes, and what stays coherent? Its key sentence: the application is not one video. It is a world that continues in response to the user. A video is fixed once made. A world answers back.
Marble (World Labs) turns an image into a persistent 3D world. Persistent means the world stays the same when you look away. The slide asks two tests: turn around, does the same room remain? Edit an object, does the change persist? A spatial world model, it says, can support navigation, editing, design and simulation.
GAIA-4 (Wayve, 2025) tests a driving decision before the car goes on the road. It starts from the same recorded context and plays two actions, “slow down” and “speed up”, and you watch where they begin to diverge. The slide calls this action-conditioned counterfactual sensor futures. A counterfactual is what would have happened if you had acted differently. Action-conditioned futures can expose consequences before a real vehicle commits.
GraphCast (Google DeepMind, Science 2023) forecasts the planet and not just the next video frame. The slide reports a forecast horizon of 10 days and a runtime of under 1 minute for one forecast. Its lesson is about representation: the “world” can be physical variables over time rather than pixels.
AIDO Cell 1.0 (GenBio AI, August 2026) is shown as a virtual-cell world-model prototype. The slide asks you to test a gene edit or a drug before the wet lab, which is the real laboratory. A virtual cell can be perturbed, observed, cloned and perturbed again. The ambition: a persistent cell state whose response can be read across biological scales.
DayDreamer (Wu, Escontrela, Hafner and colleagues, CoRL 2022) lets a robot practice inside a learned world. It learned world models and behaviours from imagination on four real robots: a quadruped walking, two arms doing visual pick and place, and a small ball robot navigating. The slide’s warning matters: imagined practice can reduce hardware trials, but only if the learned model is accurate enough.
Pick a world in the device. For each one it gives the lecture’s three questions in plain words, and it shows whether the agent can act or only watches.
Choose a system. The diagram shows whether an action flows from the agent into the world. Below it are the three answers, taken from the lecture’s slide for that system.
The three answers restate what the lecture’s slides say about each system, in our words. Nothing here describes how a system is built inside. The 10 days and the under 1 minute for GraphCast are the figures the slide reports. The watch-or-act split follows the lecture’s distinction from Chapter 6.
Count the arrows. Five of the six worlds let the agent act: move, edit, slow down, perturb, behave. Only GraphCast watches. That matches the lecture’s own early split: forecasting the weather needs no action from our agent, while a car, a robot or a cell experiment asks “what if we do this?”.
Now the lecture goes back to the robot. It asks you to think in pairs for 60 seconds and give one answer per question. What should this robot remember, predict and check? Here is one good answer for each, using the chapters you have read.
The next slide generalises the exercise. In every world the three questions ask the same thing in different words. Representation: what does the model observe, and what must it remember? Prediction: what could happen next, and how does an action change the future? Interaction: what can the agent change, and what feedback returns? The lecture says the course will ask them about videos, robots, weather, cells and software agents.
The failure mode, in three beats, comes from the DayDreamer warning. The symptom: a robot is superb in its imagined practice and clumsy on real hardware. The measurement: take the same action sequence, run it in the model and in the world, and compare the outcomes. The fix: keep some real trials running, refresh the model with them, and trust imagination only where the model has checked out.
Chapter 10
Keep the whole lecture on one page: the numbers, a checklist, the core as code, the course plan, and where to go next
A world model is the ability to represent a world and predict how it may change, including the consequences of interaction. Agents need one when the task is new, the feedback is slow or the mistakes cannot be undone. A reactive reflex then arrives too late, and a learned response cannot cover every combination of scene, goal and constraint. Three questions organise the idea. Representation: what must the model carry forward, learned from experience, and kept only where it changes the future? Prediction: what happens after each action, as a set of possible futures that a model samples, and that must be rolled out far enough for the goal to matter without letting errors pile up? Interaction: how does acting bring evidence that confirms, contradicts or surprises, and updates the next choice now and the model over many episodes? The same three questions fit videos, robots, weather, cells and software agents.
| What | Number or rule | Why it matters |
|---|---|---|
| The definition (lecture) | represent a world, predict how it may change, including interaction | One sentence for the whole course |
| Three questions (lecture) | representation, prediction, interaction | They organise every later lecture |
| GraphCast (lecture) | 10 day forecast, under 1 minute per forecast | A world need not be pixels |
| Tower rule (toy) | balance point 5.33 cm from the middle, limit 5 cm | A tiny hand-written world model |
| Reaction budget (toy) | speed × delay = 6 cm past contact, margin 1.5 cm | When feedback is too late |
| Combinations (toy) | 625 situations, 100 demos meet 15%, all 20 facts | Consequences are reusable, whole answers are not |
| History (toy) | 1 frame misses 9.0 cm, 2 frames 3.2 cm, 6 frames 0.85 cm | Memory buys accuracy |
| Representation (toy) | colour 54%, raw pixels 68%, support margin 100% | Keep what changes the future |
| Futures (toy) | push 6: 55% stays, 22.5% left, 22.5% right | Prediction is sampling |
| Compounding error (toy) | 1° bias: 0.017 cm at step 1, 13.9 cm at step 40 | Error grows with steps squared |
| Horizon (toy) | grab now wins to 4 steps, move first from 5 | Look far enough for the goal to show |
| Belief (toy) | 0.60 → 0.93 after a gap, 0.14 after none | Evidence updates the next choice |
Here is the hero of this page as a few lines. It predicts every plan first, keeps the ones that survive, and takes the cheapest. The reflex acts first. After it, a belief update, the number from Chapter 8.
pythonPLANS = {"reach across": 3, "reach around": 5, "move tower first": 7} # steps each plan takes def predict(plan, tower_offset_cm, table_half_width_cm): """The world model: does this plan reach the cup and keep the tower up?""" if plan == "reach across": return tower_offset_cm >= 8 # arm 4 cm + tower 4 cm clear the line if plan == "reach around": return table_half_width_cm >= 15 # the arc needs 11 cm + arm 4 cm return True # push the tower to the lane edge first def react(offset, half_width): return "reach across" # the reflex: act first, look afterwards def imagine_then_act(offset, half_width): ok = [p for p in PLANS if predict(p, offset, half_width)] # simulate every plan first return min(ok, key=PLANS.get) # cheapest plan that survives for offset, half in [(12, 20), (0, 20), (0, 14)]: print(offset, half, "|", react(offset, half), "|", imagine_then_act(offset, half)) # 12 20 | reach across | reach across # 0 20 | reach across | reach around # 0 14 | reach across | move tower first def update(prior, saw_gap, r=0.9): like_open = r if saw_gap else 1 - r like_blocked = 1 - r if saw_gap else r return prior * like_open / (prior * like_open + (1 - prior) * like_blocked) print(round(update(0.6, True), 2), round(update(0.6, False), 2)) # 0.93 0.14
CIS 6280 meets for 24 lectures in Fall 2026 at the University of Pennsylvania, taught by Jiatao Gu with teaching assistants Mutian Tong, Yong-Hyun Park, Enxin Song and Xinyue Ai. The grade has four parts: in-class attendance, three coding assignments, one major project, and a written final exam with a one-page cheat sheet; the exact weights will be announced separately. Each of the three assignments isolates one part of the agent and world loop. First, design your own environment, with its rules, observations, actions and feedback. Second, learn a world model, a representation that predicts how the world changes. Third, build a policy, which uses the model or feedback to choose actions. The project asks teams of one to three to design a world model for a shared environment. Teams compare results on a leaderboard, and the work moves through a proposal, a checkpoint, a presentation, and a report with code. The posted dates run as follows. Assignment 1 is open from 9 to 23 September. Assignment 2 runs from 30 September to 19 October. The proposal falls on 14 October. Assignment 3 runs from 21 October to 18 November. The checkpoint is on 16 November, the presentation on 3 December, and the report and code on 14 December. The course site is the source of truth, and the lecture says topics and order may change.
This lesson is the first of a series that follows the course lecture by lecture, and each new lesson will link back here. The tentative schedule, grouped by us into parts, is below.
| Part | Lectures |
|---|---|
| Foundations | 02 Probabilistic Formulation, 03 Environments, Simulators and Rollouts, 04 State-Space Models |
| Representation learning | 05 Self-Supervised Foundations, 06 Self-Supervised II: JEPA |
| Generative foundations | 07 Latent-Variable and Adversarial Models, 08 Autoregressive Models, 09 Diffusion and Flow Matching, 10 Normalizing and Autoregressive Flows |
| Sequences and planning | 11 Sequential Generative World Models, 12 Model-Based RL, Planning and Control |
| Video and space | 13 and 14 Video World Models, 15 and 16 Spatial World Models, 17 Neural Physics |
| Robots, language and agents | 18 and 19 World Models for Robot Learning, 20 LLMs as World Models, 21 Reasoning Models, 22 World Models for Digital Agents |
| Evaluation and synthesis | 23 Evaluating World Models, 24 Course Synthesis and Open Problems |
Now press Present or Teach and explain the cup-and-tower task out loud, from memory: why reacting is too late, what the robot must represent, which futures it must predict, and how acting brings back evidence. Then go back to the robot at the top, move the tower to 5 cm off the line on a narrow table, and say which plan the world model picks and why.
Credit: this lesson re-teaches Lecture 1 of CIS 6280, World Models, Fall 2026, University of Pennsylvania, by Jiatao Gu (instructor) with teaching assistants Mutian Tong, Yong-Hyun Park, Enxin Song and Xinyue Ai. Read the original lecture slides (PDF) and the course site. The slides are not copied here; every figure, device and toy is ours, and any number marked toy is made up for this lesson.