History, Foundations and Probabilistic Formulation

A world model answers one question: given what I have seen, what could happen if I do this? Push a block and see why an honest answer is a spread of futures, not one guess.

Learn where the prediction question came from, in 80 years of cognition, control, planning and learning, and how the course writes it as one precise formula.

Choose a push, press Push 20 times, then flip between a point prediction and a distribution. Watch which one the real pushes embarrass. Then we build, piece by piece, the history, the neural networks and the probability underneath.

You need a rough idea of a probability (a number from 0 to 1 for how likely something is) and a little algebra. We build the rest from zero.

Based on Lecture 2 of CIS 6280: World Models (University of Pennsylvania, Fall 2026), taught by Jiatao Gu. Lecture slides. Reading: Loghman Samani, a history of model-based RL. Every idea is re-taught here in our own words and devices; it follows Lecture 1, the world models overview.

Push the blue block

Illustrative toy world, not the lecture’s data. The true outcome chances (light push: upright 62%, leaning 28%, sliding 9%, collapsing 1%; hard push: 10%, 20%, 38%, 32%) are made up. The point model puts 97% on its one guess and 1% on each other outcome. Every push is a real random draw in your browser, and both scores are the average of −ln of the probability each model gave the outcome that happened.

Chapter 0

Where the Prediction Question Came From

Trace one question, what could happen next, back through five fields that each found a piece of it

You hold a full cup near the edge of a table. You are about to nudge it. Before your hand moves, you already know what will happen: a gentle nudge slides it a little, and a hard one tips it over.

You did not try both pushes. You ran them in your head. That quiet rehearsal is the idea this whole course is about, and Lecture 2 asks where the idea came from and how to write it down exactly.

Housekeeping, in one paragraph

The first slides cover logistics. Course capacity rises from 49 to 70 seats. A student who is still waiting should wait for an official seat while enrolment settles. A student who wants to audit should email the TAs, who add auditors to Canvas later. That is all the admin there is.

Three questions, and the one for today

Lecture 1 ended with three questions that kept coming back across very different worlds. Representation asks what information must persist. Prediction asks what could happen next. Interaction asks what the system can change, and what feedback returns. Our overview lesson covers that lecture.

Today is about the middle one. Where did the prediction question come from, and how can we state it precisely? The answer has two halves. First a history, then a formula.

A convergence, not a family tree

The lecture opens its story with a warning. Modern world models are a convergence. Five ideas came from different fields: internal surrogates (a stand-in for the world), predictive state (a summary that carries the past forward), action and consequence models, learned dynamics (rules fitted from data) and imagination (practice inside the model). The 2018 paper called World Models packaged the idea in a memorable way. It did not invent it.

So the slide on the roadmap says to read it as a convergence, not one family tree. Each field added one piece.

FieldWhat it addedWhere we meet it
CognitionInternal surrogates and flexible representationChapter 1
ControlPredictive state, feedback and uncertaintyChapter 2
AI and system identificationExecutable or learned transitionsChapter 3
Reinforcement learningPlanning, and learning from imaginationChapters 4 and 5
Deep learningPixels, latent variables and generative futuresChapter 5

The assigned reading tells the same story from the reinforcement learning side. Loghman Samani’s history of model-based RL starts with a plain problem. Model-free methods learn by trial and error, and they waste data, sometimes needing millions of tries. A model-based agent first learns a model of its environment, then uses it to plan or to make practice data. Samani follows 26 papers from Sutton’s Dyna in 1990 to DeepMind’s Genie 3, and says the tension between sample efficiency and model accuracy drove most of the field’s progress. We use his thread in Chapters 4 and 5.

Why spend half a lecture on history?

There are three reasons. First, it stops us from mistaking a recent paper for a definition. Second, each field leaves us a test we can apply to any new model. Does it predict the consequence of an action? Does it carry state forward? Does it say how unsure it is? Third, the old methods still run inside the new ones. The reading’s TD-MPC, a modern latent model, plans with model predictive control, an idea from 1978. Knowing the old piece tells you what the new system is doing.

See the pieces arrive

The device below holds the lecture’s milestones, one dot per paper or idea, in five rows. Drag the year from 1943 forward. Watch which rows are lit before 2018, when the famous package arrived.

Scrub through 80 years

Each dot is a milestone from the lecture. A bright dot has already happened at the year you chose. Drag the slider and read what each row had contributed by then.

1990

The milestones and years are the ones on the lecture’s slides (some span a few years; we place each at its slide year). Which row a milestone sits in is our own grouping, following the roadmap slide.

Notice what the slider shows. By 1960 two rows are already busy, and no neural network exists. By 1990 the idea of practising inside a learned model is on the board. The deep learning row fills late. That is the convergence: deep learning made the old pieces work on pixels, it did not start from nothing.

One more rule from the lecture is worth keeping now. A world model is judged by what it enables, not by how literally it copies the world. We return to this in the next chapter, and again in the last.

The road

  1. Internal models: Craik, Tolman and latent learning.
  2. Feedback, state and uncertainty: Wiener, Bellman, Kalman, Conant and Ashby.
  3. Writing the rules or fitting them: Shakey, STRIPS, system identification.
  4. Choosing and imagining: MPC, Dyna, forward and inverse models.
  5. Maps to modern world models: PILCO to Genie and Marble.
  6. The question as data: observation, action, history, horizon, target.
  7. Neural networks: MLP, CNN, RNN, transformer.
  8. Probability and learning: distributions and the NLL loss.
  9. The formulation: latent state and one conditional distribution.
  10. The worked case: one-step versus rollout objectives.
  11. Field guide: numbers, terms, checklist, code.
The lecture calls modern world models a convergence. What does that mean?

Chapter 1

Internal Models: Try Before Acting

See how a small stand-in for the world lets an animal or a machine rehearse, and what that stand-in must keep

A rat sits at the start of a maze. Yesterday it learned the way to the food: left, left, right, straight. Today a guard slides a board across the second corridor. What does the rat do?

If it only remembered a chain of turns, it keeps pushing at the board. If it remembered the shape of the maze, it turns around and finds another way. The difference between those two rats is the difference between a reflex and a model. The first era of our history, 1943 to 1948, is about that difference.

Craik: try before acting (1943)

Kenneth Craik was the first director of the MRC Applied Psychology Unit in Cambridge. In 1943 he made a simple claim. An organism that carries a small-scale model of the world in its head can try out actions inside the model, and so avoid some costly trial and error in the real world. The lecture sums it up as internal rehearsal.

The slide on the small-scale model adds a warning. A model is a selective surrogate (a stand-in that keeps only some features), not a miniature photograph. It keeps three things.

  1. Situation. Retain the conditions that matter.
  2. Possible action. Represent alternatives that could be tried.
  3. Consequence. Preserve the differences that guide a choice.

The lecture gives two examples. A bridge model keeps forces. A game model keeps legal moves. Neither needs every surface detail. Craik’s surrogate also included the organism’s own possible actions, which is exactly what a modern action-conditioned model needs.

Chess shows the point. A model that keeps legal moves can tell you what the board looks like after e4, with no idea what colour the pieces are or how the room is lit. A photograph of the board keeps the colour and the light, and still cannot tell you which moves are allowed. For deciding what to do, the first model is the better one. Craik’s word surrogate means a stand-in chosen for its use, not for its likeness.

Tolman: find a new route (1948)

Edward Tolman watched rats run mazes and noticed behaviour that a chain of memorized turns could not explain. When the familiar path was blocked, rats took a new route that pointed roughly at the goal. The lecture shows his sunburst maze: many corridors fan out from a hub, and the familiar one is blocked.

Tolman called what the rats seemed to hold a cognitive map. Its key property is that the knowledge is relational. It records how places connect, so it can be recombined when the old route fails. A response chain records only what to do next, and breaks when one link breaks.

Try it. In the maze below, the memorized chain is the straight corridor. Block it, or tap any cell to add your own blocks, then run both agents.

Block the familiar route

Pink walks the memorized chain of moves. Purple holds the map and plans again with a breadth-first search. Press Block the route or tap cells to toggle blocks, then press Run both.

Toy maze, 11 by 7 cells, four-way moves, one step per cell. The map agent runs a real breadth-first search over the free cells it knows. It sees the whole maze, which keeps the toy simple; real animals and robots must build the map from experience.

With nothing blocked, both walkers take the same 10 steps. The chain is not worse, only more brittle. After a block the chain stops at the first cell it cannot enter, while the map walker takes a detour through another gap in the wall. The map did not know the detour in advance. It knew how cells connect, and could work the detour out.

Latent learning: knowledge before reward

The lecture then revisits a classic experiment from 1930, read through Tolman’s 1948 eyes. Three groups of rats ran the same maze each day. One group (HR) found food at the end from the start. A second (HNR) never got food. A third (HNR-R) got no food for ten days, then found food at the end of Day 11.

You might expect the third group to improve slowly from Day 11. It did not. Its errors collapsed on Day 12, and its performance quickly approached the group rewarded from the start. During the ten empty days it had been learning the maze. The reward only changed what the behaviour revealed.

The slide puts it in one line: learning is not the same as current performance. A model of the world can grow without any reward, and a reward can show it off later. This matters for us. Many modern world models learn from raw experience with no task signal at all.

The takeaway of the first era

The lecture closes the era with two words: purpose and representation. The purpose is to rehearse before acting, comparing alternatives without trying every one in the world. The representation keeps useful structure: relationships that support flexible prediction and behaviour, not every visible detail. And again: a world model is judged by what it enables, not by how literally it copies the world.

Rats in the delayed-reward group found food for the first time at the end of Day 11, and their errors collapsed on Day 12. What does that show?

Chapter 2

Feedback, State and Uncertainty

Learn why correcting after the error is not yet a world model, and how predict-then-correct fixes the gap

A thermostat reads 19 degrees. The target is 22. The heater turns on. When the room reaches 22, the heater turns off. The thermostat never wonders what the room will do next. It only reacts to the error it sees now.

That loop is powerful, and it is not yet a world model. This chapter explains the gap, and the three ingredients that the control field added to close it: feedback, state and uncertainty.

Wiener: act, measure, correct (1948)

Norbert Wiener put communication and control, in animals and in machines, into one shared language. His word was cybernetics, and its core is a loop: act, measure what happened, correct the next action, and repeat. The loop is called closed because the result of each action returns as the next input.

The lecture draws the line carefully. A feedback controller corrects after an error. It can react to the present error without simulating several future temperature paths. A world model compares before acting: from the current state it predicts what action A would lead to, and what action B would lead to. The model adds an internal prediction of consequences under candidate actions. A feedback controller may be purely reactive. A world model adds a way to look ahead.

Bellman: one action, two effects (1957)

Richard Bellman asked how to choose actions when each one changes both the payoff now and the situation later. The lecture shows his original recurrence for the equipment replacement problem. The state is the machine’s current condition. The action is keep or replace. The choice changes today’s return and tomorrow’s condition.

fN(i)
The best total return from state i when N stages remain.
max over q
Try every action q and keep the best one.
bi(q)
The payoff now, for taking action q in state i. The slide marks this part “now”.
∑j aij(q) fN−1(j)
The future. Each next state j has chance aij(q), the slide’s “dynamics”, and is worth fN−1(j) with one fewer stage left.

The world-model piece is the chance aij(q) = P(st+1 = j | st = i, at = q): the probability of the next state given the state and the action. This is a transition model. Here is a toy run with made-up numbers. A machine is Good or Worn. Keeping a Good one pays 10 and keeps it Good with chance 0.8 (Worn with chance 0.2). Keeping a Worn one pays 4 and it stays Worn. Replacing costs 5, and the new machine then earns its 10 this stage, so replacing nets 5. After a replacement the machine is Good with chance 0.8 and Worn with chance 0.2, the same chances as for a kept Good machine.

f1(Good) = max(10, 5) = 10    f1(Worn) = max(4, 5) = 5 (replace)

f2(Good), keep: 10 + 0.8×10 + 0.2×5 = 19    replace: 5 + 0.8×10 + 0.2×5 = 14

f2(Worn), keep: 4 + 1.0×5 = 9    replace: 5 + 0.8×10 + 0.2×5 = 14

With two stages left, a Good machine should be kept (19 beats 14) and a Worn one replaced (14 beats 9). Every number came from comparing consequences across stages. The lecture adds the limit: Bellman did not explain how to learn the transition model from pixels.

Kalman: observation is not state (1960)

Now a harder problem. A camera sees a ball at position 3. Is that the whole truth? No. One image shows position and hides velocity, friction, contact, or a cause that is out of view. The lecture says it directly: observation is not state. The state is the hidden quantity that carries forward everything the future depends on.

Rudolf Kalman’s model has two lines. One is the dynamics, which predicts. The other is the sensor, which observes. The letters A, B and C stand for fixed tables of numbers (matrices) that say how the pieces combine.

Dynamics, predict
The latent dynamic state x moves by the rule A, is pushed by the control input u (through B), and is nudged by process noise w, a small random shock the rule cannot explain.
Sensor, observe
The noisy, incomplete observation o reads part of the state through C, plus measurement noise v.

The recipe is two moves repeated. Predict with the dynamics, then correct with the new observation. Our Kalman filter lesson derives the exact blend. For now the device shows the idea. A ball moves and the sensor reads only its position, with noise. Drag the noise up, then hide the sensor for a stretch and watch the confidence band.

Predict, then correct

Blue dots are noisy position readings. The thin line is the truth. The purple line is the filter’s estimate, and the purple band is one standard deviation of its confidence.

2.5

Toy track: a ball whose speed drifts between 0.3 and 0.9 units per step, 60 steps, a two-number state (position and speed), a sensor that reads position only. The noise draws are fixed by a seed, so the same slider value always gives the same dots. The filter is the standard predict and correct update, with a small process noise.

Two things happen. When the noise is high, the purple line is far smoother and closer to the truth than the dots, because the filter trusts its own prediction more. And when you hide the sensor, the filter keeps going on prediction alone, but the band grows. That growing band is the third ingredient: a useful estimate carries confidence, not only a point guess.

Conant and Ashby: what does “model” mean? (1970)

The Good Regulator Theorem sets a floor. The simplest optimal regulator must preserve the system distinctions that change successful regulation. If hot and cold require different actions, the regulator must tell hot from cold. Here “model” means a task-relevant mapping from system situations to regulating responses. It does not imply a full simulator, a neural module, a complete copy of the world or an arbitrary rollout engine. Keep that in mind when a later slide says a model need not reconstruct every pixel.

The lecture ends this era with the three ingredients. A closed loop: actions change the observations that arrive next. A predictive state: the current measurement may not be enough. And uncertainty: a useful estimate carries confidence.

A camera frame shows a ball at position 3. Why is the frame not the ball’s state?

Chapter 3

Two Ways to Get the Rules: Write Them or Fit Them

Compare a hand-written action model with dynamics estimated from data, and run both on a tiny example

Suppose you want a robot to bring a box from room B to room C. You do not want to tell it every wheel turn. You want to say box in room C and have it work out the steps. To do that, the robot needs rules for what each action does to the world.

Where do the rules come from? The lecture shows two answers that grew side by side from the late 1960s. In one, a person writes the rules. In the other, a machine fits them to data. Modern neural world models take the second path, and the first one still shapes how we think about states and actions.

Shakey: a goal without the motor steps (1966 to 1972)

Shakey was a robot built at SRI, the Stanford Research Institute. It combined perception, a description of the world, planning and execution in one working system. The slide’s point is that a goal could be given without prescribing every motor step. Planning connected the acting system to an explicit symbolic model of what actions do.

STRIPS: a world state written as symbols (1971)

The planner’s formalism was STRIPS, the STanford Research Institute Problem Solver, by Richard Fikes and Nils Nilsson. It has three parts. A state is a set of symbolic facts, such as ROBOT_AT(A) and BOX_AT(A). An operator is an action rule. The next state is the updated set of facts.

The lecture asks whether a symbolic state counts as a world state. Its answer is yes, with one addition: the world model also includes the rules that predict how actions transform those symbols. An operator has three lists.

  1. Preconditions must already hold. For PUSH(box, A to B): ROBOT_AT(A) and BOX_AT(A).
  2. Delete lists the facts made false: ROBOT_AT(A) and BOX_AT(A).
  3. Add lists the facts made true: ROBOT_AT(B) and BOX_AT(B).
successor
The set of facts that holds after the action.
current state
The set of facts that holds now.
delete list
Facts the action makes false. Remove them.
add list
Facts the action makes true. Put them in. The symbol ∪ means union.

PUSH is an abstract action rule, not a motor command. The operator predicts the consequence. A separate planner searches for a sequence of operators that reaches the goal. Prediction and search are two jobs. Play with both below. The box starts in room B, and the goal is BOX_AT(C).

Apply an operator

Pick GO or PUSH and the rooms, then press Apply. The box under the picture checks each precondition. Or press Plan for me and watch a search find the steps.

A toy STRIPS world with three rooms, one robot and one box, so the whole state space is 3 × 3 = 9 states. The operators GO and PUSH are written exactly as the lecture writes PUSH. The planner is a breadth-first search over those 9 states.

Try a wrong move first: PUSH from A to B while the box sits in B. The precondition BOX_AT(A) fails, so the operator does not apply. That is the whole value of the formalism. The model refuses what cannot happen. A good plan here is two steps: GO from A to B, then PUSH from B to C. The planner finds it by trying operators on states, not by driving.

The other road: estimate the dynamics from data

The lecture sets symbolic AI and system identification side by side. Symbolic AI writes the transition rules: facts and operators define how one situation becomes another. System identification estimates the dynamics from data: apply inputs, measure responses, and the responses reveal a numerical predictive model. The lecture gives four steps.

  1. Choose a model class. State what kind of dynamics can be represented.
  2. Design informative inputs. Excite the system and record input and output paths.
  3. Fit the dynamics. Estimate the unknown parameters from the measured responses.
  4. Validate on new data. Test prediction beyond the paths used for fitting.

A tiny worked case, with made-up readings. The model class is x′ = a·x + b·u: the next position x′ depends on the position x and a push u. We do not know a or b. We try two experiments. Set x = 1 and u = 0, and read x′ = 0.9. Then set x = 0 and u = 1, and read x′ = 0.5. The first says a = 0.9. The second says b = 0.5. Now validate on a pair we never fitted: x = 3, u = 1. The model predicts 0.9×3 + 0.5 = 3.2. The plant gives 3.1. The error is 0.1, which is small, so we keep the model.

Why did we pick those two experiments? Imagine the push u is always zero. Then every reading is x′ = a·x, and b can take any value without changing a single data point. The symptom is a fit that looks perfect while b is wild. The measurement is that data with u = 0 never vary with b. The fix is the second step of the recipe: design inputs that make every unknown matter. The lecture’s historical point follows. Neural world models scale this recipe to pixels, tokens and latents. They did not invent learned dynamics.

In STRIPS, the robot is at A and the box is at B. Can PUSH(box, A to B) run?

Chapter 4

Choosing and Imagining with a Model

Use a model to pick actions, to practise offline, and to pass a learning signal back to a controller

Your phone plans a whole drive. You take the first turn. The phone checks where you really are, and plans again. It never trusts the old plan for long. That habit has a name, and it is the first way to put a world model to work.

This chapter covers seven slides of 1978 to 1992. They answer one question: once you have a model, what do you do with it? The answers are to choose actions, to practise, and to teach a controller.

MPC: choose with a world model (1978)

The world model predicts. Model predictive control (MPC) uses those predictions to choose the next action. The lecture gives it three steps.

  1. Candidates. Try several short action paths.
  2. Predict and score. Use the model to compare where each path may lead.
  3. Act and replan. Choose one path, execute only its first step, then look again.

In short: MPC = model + goal + look-ahead choice + feedback. The horizon H is how many steps ahead each candidate looks. The reading adds a detail from Nagabandi and colleagues: they modelled the next state as st+1 = st + fθ(st, at), a small learned change added to the old state, and said that replanning every step makes up for model errors.

The device below is a small robot with a round obstacle between it and the goal. It plans with a model that moves it one fixed step in the chosen direction. Seventy-seven candidate paths, each heading for a different waypoint and then the goal, are scored by where they end after H steps and whether they touch the obstacle. Change the horizon. Turn on the wind, which the model does not know about. Turn off the look-again step.

Plan, step, look again

The teal line is the path the robot really took. Orange lines are the plans it made. Press Run MPC, then change the horizon, the wind, or whether it looks again after each step. The read-out counts bumps and arrivals.

8

Toy robot on a 1 by 0.6 floor, 0.045 per step, a round obstacle of radius 0.1, 77 candidate paths per plan: head for one of a grid of 77 waypoints, then straight for the goal, cut off after H steps, so each setting always gives the same run, at most 80 steps. Each plan is scored by its closest approach to the goal plus 0.3 times its final distance, plus a penalty if it touches the obstacle. The model knows the obstacle and the step size and knows nothing about the wind, which nudges the robot 0.014 down each step.

Run it with a few settings. Two things show up. First, with the wind on and the look-again switch off, the robot drives blind for H steps at a time, and the read-out counts bumps into the obstacle (horizon 12 or 16 gives 5 bumps each). Switch look-again on and each new look corrects the drift: horizons 6, 8 and 14 then reach the goal with 0 bumps. That is why only the first step is executed.

Second, looking again does not fix everything, and the result does not improve in a straight line with the horizon. With the wind on, horizons 4 and 10 jam the robot against the obstacle: the read-out shows 69 bumps, one for every step it spends pushing at it. Our score only asks how close a plan ends to the goal, so for some horizons the robot hovers in front of the obstacle and the run ends with “Did not arrive”. That is a real failure of MPC with a short-sighted cost. The usual fix is a cost-to-go, a number that says how good the end of the plan is. The reading’s TD-MPC adds exactly that, a learned value for the end of the horizon, and Chapter 5 returns to it.

Johnson-Laird: compare possible situations (1983)

Philip Johnson-Laird, in Mental Models, brought cognition into the story. People reason by building possible situations. A conclusion is necessary only if it survives every situation the premises allow. One counterexample refutes it. Take the premises “the ball is left of the cup” and “the cup is left of the bell”. In every layout that fits, the ball is left of the bell, so that conclusion is necessary. “The ball is next to the cup” is not: one layout puts the bell between them, so it is refuted. The slide notes these possibilities are not yet probabilities, dynamics over time or action-conditioned futures.

Neural system identification (1990)

The 1990 slide repeats the Chapter 3 recipe with a neural network as the model class. Apply commands to an unknown plant, observe how it changes, and fit a neural surrogate that predicts the response. The lecture calls this neural and adaptive control, and not yet the modern deep-learning era.

Dyna: a learned model supports imagination (1990)

Richard Sutton’s Dyna turned the model from something to fit into something an agent uses inside. It connected real experience with hypothetical experience made by an internal model. In the reading’s words, each step does three things. The agent updates from the real transition, updates its model, and then runs K more planning updates on transitions the model invents, treating them just like real ones.

Does the extra practice help? Run it. The toy world is a small maze. The reward is 1 at the goal and 0 elsewhere. Compare K = 0, 5 and 50 planning steps.

Practise in the model

The picture shows how many steps each episode took, averaged over 10 runs, on a log scale. Lower is better. Choose how many imagined updates the agent makes per real step.

Tabular Dyna-Q on a 9 by 6 maze with walls (start at the left, goal at the top right), learning rate 0.1, discount 0.95, 10% random moves, a deterministic remembered model, 30 episodes, averaged over 10 seeded runs. The results are computed live in your browser, not taken from the lecture or the reading.

With K = 0 the agent only learns from real steps, and the first episodes wander for hundreds of steps. With K = 50 the agent replays what the model remembers between real steps, so a single success spreads backward through the maze almost at once. This is the lecture’s historical takeaway: models entered the learning loop. We return to planning mechanics and Dyna algorithms later in the course.

Making the world differentiable (1990)

That same year, Jürgen Schmidhuber used two recurrent networks. A controller produces an action. An environment model predicts the sensory input and reinforcement that follow. To train the controller, you backpropagate the task error through the frozen model (its numbers are held fixed, only the controller learns) into the controller. The real world need not be differentiable. The learned surrogate gives an approximate path for the learning signal.

Forward and inverse models

The lecture uses one arm example to separate two models. A forward model is a consequence predictor: given the arm state xt and a motor command at, predict the next hand pose, ŷt+1 = fθ(xt, at). An inverse model is an action selector: given xt and a desired hand pose y*, produce the command at = gφ(xt, y*). The forward direction is the core of an action-conditioned world model. The inverse can be ambiguous, because several commands can reach the same target, such as an elbow up or elbow down.

Distal teacher: a forward model teaches a controller (1992)

Michael Jordan and David Rumelhart asked how to train the inverse model when the target says where the hand should end up, not which command was right. Their answer chains the controller to a frozen forward model and measures the outcome error.

∂L/∂a
How the outcome error changes if we nudge the command a. This is the learning signal for the controller.
∂ŷ/∂a
The forward model’s Jacobian: how the predicted outcome moves when the command moves. It turns outcome error into a command signal.
∂L/∂ŷ
How the error L = ‖ŷ − y*‖² changes with the predicted outcome.

A one-number worked case, with made-up values. Let the forward model be ŷ = 2a, the target y* = 6, and the controller currently output a = 1. The prediction is ŷ = 2, so L = (2 − 6)² = 16. Then ∂L/∂ŷ = 2(ŷ − y*) = −8 and ∂ŷ/∂a = 2, so ∂L/∂a = −16. A step of 0.1 downhill moves a to 1 + 1.6 = 2.6. Now ŷ = 5.2 and L = 0.64. The loss fell from 16 to 0.64 in one step, and no one ever said that a = 3 was the right command.

The slide ends with a limit. This is a learned, differentiable credit-assignment path for a low-dimensional motor task. It is not yet a general high-dimensional generative world model.

In the distal teacher idea, what does the frozen forward model do?

Chapter 5

From Maps to Modern World Models

Compare what each famous model chose to predict, and find the one question they all share

Imagine a row of famous models laid on a table. One draws video. One draws a 3D room. One never draws anything and only guesses a number called a value. Which of them is a world model?

The lecture’s answer is careful. Each model chose a different thing to predict. That choice is the design. This chapter walks twelve slides, from a map made of probabilities to a world you can walk through, and sorts the models by what they predict.

Maps that persist: occupancy grids and SLAM (1985 to 2006)

A robot sees a wall through a noisy sensor, turns, and loses sight of it. Where is the wall now? An occupancy grid, as in Elfes’ 1989 work, answers with a map of cells marked free, occupied or uncertain. SLAM estimates the robot’s pose while it updates the map. The enduring idea is a world-centred state that survives after the camera moves, and supports localisation, prediction and planning.

A worked cell, with a made-up sensor that is right 70% of the time. A cell starts at 0.5. One “occupied” reading gives odds 0.7 to 0.3, so the chance is 0.70. A second reading multiplies the odds again: (0.7/0.3)² = 5.4, so the chance is 5.4/6.4 = 0.85. A “free” reading divides the odds back, so one hit and one miss return to 0.50. Evidence accumulates, and doubt stays visible.

PILCO: learn the dynamics, then improve the policy (2011)

PILCO uses a probabilistic learned model to get more value from every real trial. It fits a Gaussian process (a model that predicts a change in state together with how unsure it is), then imagines rollouts by pushing a whole distribution forward, not one guessed path, and improves the policy to minimise expected long-run cost. It learned the real swing-up task from only 17.5 seconds of interaction. The reading explains why uncertainty matters: a single deterministic model commits with false confidence outside its data, and a planner then exploits its mistakes. The slide adds that the experiments were still low-dimensional, not pixels.

Pixels or latents? (2015 and 2016)

Two ideas arrived the same year. Oh and colleagues generated Atari frames conditioned on the chosen action: predict the visible future. E2C encoded raw images and learned action-conditioned transitions in a compact space built for control: predict a compact state. The lecture notes that latent prediction was explicit in 2015, well before JEPA. PlaNet later pushed latent dynamics to harder image tasks.

Finn, Goodfellow and Levine then showed the test that defines the field. Hold the visual history fixed. Change the action. The predicted future should change. Oh showed it in Atari, and Finn in physical interaction. The slide adds that robotics is evidence, not the definition.

World Models, packaged (2018)

Ha and Schmidhuber named three parts. V (vision) turns each high-dimensional frame into a compact vector zt. M (memory) predicts how the internal situation changes over time. C (controller) picks actions from the learned internal situation. In the reading, C is a single linear map at = Wc[zt, ht] + bc, where ht is M’s memory. The idea that earned its fame is training the controller “inside the dream”: the model’s imagined rollouts replace the real world. The lecture calls it a memorable neural package, not the origin of internal models. Its dream echoes Dyna.

Latent worlds you can train in (2019 and 2020)

PlaNet and Dreamer made the latent state the working surface. The reading gives the design that PlaNet introduced, the recurrent state space model (RSSM), which splits the state into a deterministic memory h and a stochastic part s.

ht
The deterministic memory: an RNN update from the last memory, the last stochastic state and the last action. It carries long-range information.
st
The stochastic state, drawn from a distribution given the memory. It holds the uncertainty about what happens next.
ot
The observation, drawn given both. It lets the model be trained against real frames.

PlaNet plans in that latent space. Dreamer goes one step further and learns behaviour from imagined latent trajectories. The slide’s summary is the line to keep: a learned world model became not only a place to look ahead, but a place to train behaviour.

Different targets for different jobs (2018 to 2025)

GQN (2018) compresses a few posed images into one scene state and answers “what should I see from this new viewpoint?” It does not model action-conditioned dynamics. MuZero (2020) predicts only reward, value and policy-relevant quantities, uses the learned dynamics inside look-ahead search and discards visible detail that does not affect the decision. A useful model need not reconstruct every pixel, and the lecture notes this is one famous design choice, not the only definition. JEPA (2022) matches an abstract target in representation space instead of generating every visible detail; the context is encoded, a predictor infers the representation of what is missing or future, and the comparison happens in feature space. NeRF (2020) puts a static scene in a neural field: colour and density as a continuous function of 3D position and viewing direction. By itself it does not predict how the world changes after an action.

Genie (2024 and 2025) generates the next sensory experience while the user acts inside it: it conditions on history, prompt and the current action, and keeps enough consistency to stay interactive. The reading adds that Genie 3 runs at 720p and 24 frames per second. Marble generates a persistent 3D world from text, image, video or coarse geometry that you can navigate and export. The slide’s stronger test still asks: how do state and observations change with action and time?

Sort them yourself. Switch on the properties you care about, and see which models are left.

Sort the models

A filled dot means the model has that property. Press the buttons below to require a property. Rows that fail go dim.

The properties follow the slide text and the reading, with our own tidy grouping. “Takes an action” means the model is conditioned on the agent’s action; GQN is queried by a camera pose, JEPA by a context, and neither is an action. “Carries uncertainty” is marked only where the slide or the reading says the dynamics are probabilistic. Treat a blank as “not stated”, not “never”.

Turn on only Takes an action as input. Eight of the thirteen stay lit. Then add Predicts pixels: only Oh, Finn and Genie remain. Neither a famous name nor a pretty video decides membership. The lecture’s test is the conditional question: given what I have, and what I do, what comes next? The next chapter makes that question exact.

Which statement matches the lecture about MuZero and JEPA?

Chapter 6

Turning the Question into Data

Write down observations, actions, history, horizon and target, and count the numbers each one holds

A robot arm hovers over a table with a blue block, a green block and a yellow block stacked in a tower. Someone asks it: what happens if you push the blue one? Before the arm can answer, three things must be pinned down. What does it know right now? What exactly would it do? What does it want to predict?

The history is over. From here the lecture turns to a precise formulation, and it begins by asking those three questions in the most plain way it can. It says world models are not the same as robotics. Games, software and science can all ask the same conditional-future question. The slide draws it as three boxes in a row: available evidence, then candidate action, then possible future.

The plan for the rest of the lecture

Four foundations come next, in order. Data: observations, actions and time indices. Neural networks: functions that map inputs to predictions. Probability: distributions over possible futures. Formulation: the inputs, the targets and the training loss. This chapter does the first. Chapters 7, 8 and 9 do the other three.

Tensors: the shapes numbers come in

Everything a computer sees is a block of numbers, and the shape of the block says what it means. A tensor is the general name for such a block, and its shape lists the size along each axis. The lecture lists four common shapes.

NameShapeWhat it holds
Scalar1One number
Vectordd numbers
ImageHimg × Wimg × CHeight, width and channels (for colour, C = 3)
VideoT × Himg × Wimg × CTime, then an image at each time

The slide’s caption: the tensor shape records the feature, space, channel and time axes. A small colour image of 64 by 64 pixels holds 64 × 64 × 3 = 12,288 numbers. Four of them in a row make a video tensor of 4 × 64 × 64 × 3 = 49,152 numbers. Keep these counts in mind. Every choice later has a price in numbers.

Observation and action at time t

Two letters carry most of the lecture. The observation ot is what the system can measure at time t: pixels, sensor readings, text or a screen. The action at is an input applied to the world: a push, a motor command, a game move or a click. Time moves in whole steps t = 1, 2, 3, and so on.

History: what we have seen and done

To predict, the model needs everything that might matter. The lecture packs it into the history ht.

ht
The history at time t: all the evidence we hold when we must predict.
o≤t
Every observation up to and including now: o1, o2, up to ot.
a<t
Every action strictly before now: a1 up to at−1. The action at is left out.

Why leave at out? The slide answers: at stays separate because we want to predict its consequence. If it were hidden inside the history, we could never ask what a different push would do. So the history holds t observations and t − 1 actions, and the candidate action arrives as its own input.

The history grows by one frame and one action at each step. At t = 1 it holds one observation and no actions. At t = 4 it holds four observations and three actions. At t = 100 it holds a hundred frames, and no model wants to read all of them each time it predicts. That growth is the reason Chapter 9 introduces a latent state: a fixed-size summary that stands in for a history of any length.

Horizon and target

Two more choices complete the question. The horizon H says how far ahead to look: one step, or a segment through time t + H. The target Yt+1:t+H says what information to predict: pixels, pose, contact, task signals or a learned representation. The notation t+1:t+H means “from step t+1 to step t+H”.

Build a question yourself below. Slide the time and the horizon. Choose how big a frame is and what the target is. Watch what the model must take in, and what it must give back.

Count the numbers in the question

Blue boxes are observations, teal boxes are actions, and orange dashed boxes are the targets to predict. Purple brackets the history ht. Dashed teal boxes are the candidate actions you supply.

4
3

Illustrative sizes. A colour frame has 3 channels. The action is 3 numbers (push direction x and y, and strength). A block pose is 7 numbers (3 for position, 4 for orientation), and a learned vector is 32 numbers; both are our own examples, not the lecture’s.

The point of the exercise is the contrast. With 224 by 224 frames and a horizon of 3, asking for pixels means predicting more than 450,000 numbers, while asking for a block pose means predicting 21. The lecture is not telling you which to pick. It says to specify the target, because the target is what the whole model, the data and the loss will be built around.

Why does the lecture keep at out of the history ht?

Chapter 7

Neural Networks as Prediction Machines

Build a layer, see why it needs a bend, and match each network type to the part of the world it reads

You have a history of frames and a push. You want the next pose of the block. No one can write the rule for that by hand across every table, light and block. So we build a function with many adjustable numbers, and let data set them. That function is a neural network.

This chapter builds one from a single layer, in eleven slides. Our aim is not to cover deep learning. It is to know what each architecture reads, so that later we can say which one a world model uses for which part of its input.

One layer

A layer takes a vector u of din numbers and returns a vector v of dout numbers. It does that in two moves: an affine map W u + b (a matrix times the vector, plus an offset), then a nonlinearity σ (a bend applied to each number).

v
The output vector, dout numbers.
σ
The nonlinearity, such as ReLU, which keeps positive numbers and turns negatives into 0.
W
The weight matrix, with dout rows and din columns. Learned from data.
u
The input vector, din numbers.
b
The bias: dout numbers added after the matrix. Also learned.

A shape check shows the data flow. Say u holds 6 numbers: a 3-number state and a 3-number action. A layer with 64 outputs has W of shape 64 × 6 (384 numbers) and b of shape 64, so 448 learned numbers. A second layer from 64 down to 3 has W of shape 3 × 64 and b of shape 3, so 195 more. Together that is 643 learned numbers, and the whole network maps 6 numbers to 3.

Why the bend matters

The lecture makes a sharp point: two affine layers are still one affine map. Without a nonlinearity, W2(W1u + b1) + b2 can be regrouped into (W2W1)u + (W2b1 + b2), which is one matrix and one bias. A thousand such layers collapse to one.

A worked case with one input. Let W1 = [2, −1]⊤, b1 = [1, 0], W2 = [3, 1] and b2 = 0.5. Without a bend, the output is 3(2u + 1) + 1(−u) + 0.5 = 5u + 3.5. That is a straight line, and W2W1 = 3·2 + 1·(−1) = 5 and W2b1 + b2 = 3.5 agree. Now add ReLU after the first layer and set u = −1. The hidden numbers [−1, 1] become [0, 1], so the output is 3·0 + 1·1 + 0.5 = 1.5, while the straight line says −1.5. The bend changed the answer, and with the bend a network can draw curves.

See it, and fit it. The device trains a one-input network by gradient descent to match a wavy target. Toggle the bend, change the width, and press Train.

Straight line or curve?

The dashed line is the target, a wave. The solid line is the network. Press Train to run 300 steps of gradient descent. Then switch the bend off and train again.

6

A real network trained live in your browser: one input, one hidden layer, one output, 40 points on the target y = sin(2u) for u from −2 to 2, plain gradient descent with learning rate 0.05 on the mean squared error. The starting weights are seeded, so every visit begins the same.

With no bend, the best this network can ever do is the best straight line through the wave, whatever its width. With the bend and six units, the error falls steadily as the line learns to turn. That is the whole reason for σ.

Depth, loss and the update

A deep network composes many layers: fθ = fθL(L) ∘ … ∘ fθ2(2) ∘ fθ1(1). The symbol ∘ means “feed the output of one into the next”. Learning follows four steps, and the device just ran all four. (1) Predict: run the data through the model. (2) Loss: compare the output with the target and get one number. (3) Gradient: compute ∇θL, the direction in which each learned number should move to cut the loss; backpropagation computes it. (4) Update: θ ← θ − η∇θL, a small step downhill of size η.

Four kinds of layer, four kinds of input

MLP (multi-layer perceptron). Fully connected: every output unit sees every input number. It suits fixed-length vectors such as state and action vectors, and small output heads. Its limit is no built-in sense of space or time. Flatten an image and the grid is gone.

ConvNet (convolutional network). It first restricts connectivity, so each output sees only a small patch. It then shares weights, so the same detector slides over many places. A worked count: mapping a 64 × 64 image to a same-size map with a fully connected layer needs 4,096 × 4,096 = 16,777,216 weights. A shared 3 × 3 filter needs 9. In a world model a CNN encodes the spatial evidence in each frame, and the lecture warns that something else must still connect time and actions.

A frame encoder does not model time. Run a shared CNN over frame t−2, t−1 and t and you get three feature maps. Motion needs a mechanism that combines multiple frames and actions. The encoder gives you where things are, not where they are going.

RNN (recurrent network). It reuses the same function at every time step and carries a hidden vector forward, so earlier inputs can affect later outputs. The current memory is read out as the prediction.

Transformer. A token is one input item written as a vector: an image patch, a sensor reading or an action. All tokens enter together, and the output is one updated vector per token. Position and time encodings mark where and when each token occurs. Its key operation is self-attention: compare each token with all tokens, turn query-key scores into attention weights, and take a weighted sum of values. Convolution uses a fixed local neighbourhood; attention lets each token look anywhere. Our transformer lesson builds it in full.

ArchitectureInputOperationCommon use here
MLPfixed-length vectorfully connected layersstate or action model, output head
CNNimage or spatial gridshared local filtersframe encoder or decoder
RNNsequencerecurrent memory updatetemporal model
Transformertoken sequenceself-attentionlong or multimodal context
You stack ten layers that are all W u + b with no activation function. What function does the stack compute?

Chapter 8

Probability and Learning

Turn one guess into a distribution, then score it with the loss that rewards honest odds

You flick a puck across a table. You flick it the same way ten times. Three times it sticks. Five times it slides a little. Twice it slides far. The table has friction you cannot see, and the contact is never exactly the same.

Now someone asks a model: what will the puck do? A single answer cannot be right ten times out of ten. This chapter shows how to write a model whose answer is a spread, and how to teach it with a loss that rewards honest odds.

The four roles, in one table

The lecture first pins down what a world model takes in and gives out. We met the first three in Chapter 6. The fourth is new.

RoleSymbolMeaningExample
Historyhtpast observations and actionsframes and past controls
Actionat … at+H−1candidate future inputsa push or motor command
TargetYt+1 … Yt+Hfuture quantity to predictpixels, pose or contact
Outputpθ(Y | h, a)a conditional distributionprobabilities or a density

The deterministic model gives one answer

The simplest world model is a function: ŷt+1 = fθ(ht, at). History and action go in, one predicted next target comes out. The Greek letter θ stands for all the learned numbers inside, the same ones as in Chapter 7. A function gives one answer for one input. Look at what that does when the world does not.

The lecture’s picture shows one history and one push leading to three futures: slide far, slide a little, stick. Unknown friction and contact conditions change the result. Take a simpler table, where the puck either sticks (0 metres) or slides far (0.6 metres), half the time each. A model trained to be close, on average, to the recorded distances learns the mean: 0.5×0 + 0.5×0.6 = 0.3 metres. That is a distance the puck never travels. The symptom is a confident prediction that is never right. The measurement is the gap: 0.3 metres away from every real outcome, every time. The fix is to predict the spread.

Random variable and observed value

To talk about spread, the lecture separates two things with a capital and a lowercase letter. Before we look, the next target is a random variable Yt+1: a quantity with several possible values, each with a chance. After we look, one value yt+1 is recorded in the dataset. Capital means possible, lowercase means happened.

The model’s whole output is then written pθ(Yt+1 | ht, at). Read it left to right: the probability of the next target, given the history and the action, under the parameters θ. The bar means “given”.

What the network outputs

A network cannot output a distribution as an abstract thing. It outputs the numbers that define one. For a discrete target such as stick, slide or topple, it outputs logits (raw scores), and the softmax turns them into probabilities that add to 1. For a continuous target such as position or velocity, it outputs a mean μ and a spread σ, and those define a Gaussian density. The two cases read fθ(ht, at) → ℓ → softmax(ℓ), and fθ(ht, at) → (μ, σ).

Scoring a distribution: negative log likelihood

How do we tell a good distribution from a bad one? Look at the probability it gave to what really happened. The negative log-likelihood (NLL) of an observed outcome y is −log pθ(y | h, a). The slide shows two models scoring the same recorded outcome, “the puck slides far”. One assigns it 0.10, so −log(0.10) = 2.30, a large loss. The other assigns 0.70, so −log(0.70) = 0.36, a small loss. Higher probability, lower NLL. (We use the natural logarithm here, as the slide’s numbers imply.)

Training sums this over the whole dataset D = {(h(i), a(i), y(i))} of N examples.

L(θ)
The training loss: the average surprise of the model at what really happened. Lower is better.
1/N
Average over the N training examples.
∑
Add up the loss of every example, from i = 1 to N.
pθ(y(i) | h(i), a(i))
The probability the model gave to the outcome that was recorded in example i, given that example’s history and action.

Now play the part of the model. The device holds the ten recorded pushes from the story above, with outcomes stick, a little and far. You set three logits, and the softmax turns them into probabilities. Watch the loss.

Be the model, score the odds

Orange bars are your model’s probabilities. Blue bars are how often each outcome really happened in the ten recorded pushes. The gauge shows your average NLL, and the green mark is the lowest score any model can get on this data.

0.0
0.0
0.0

Toy dataset: ten pushes of one history and one action, with outcomes 3 stick, 5 a little, 2 far (our made-up story, not the lecture’s data). Probabilities come from a real softmax of your three logits, and the NLL is the average of −ln p over the ten outcomes.

Press Match the data. The probabilities become 30%, 50% and 20%, and the loss reaches its floor. Then press Be sure it slides far. The model now gives 97% to an outcome that happened only twice in ten, and gives almost nothing to the eight pushes that stuck or slid a little. The loss jumps, because each of those eight is scored by −log of a tiny number. A model that is overconfident is punished hard. A model that knows nothing (a third each) scores −ln(1/3) = 1.10 on every push. Honest odds win.

For a continuous target the same score applies to a density, which is the height of the bell curve at the value that happened. A made-up example: the puck slides y = 0.42 m. Model A says μ = 0.40, σ = 0.05. Its density at 0.42 is 7.37, so its NLL is −ln 7.37 = −2.00. Model B says μ = 0.40, σ = 0.5, a density of 0.80 and an NLL of 0.23. A density can be bigger than 1, so a good continuous NLL can even be negative. What matters is that A, which is both close and confident, scores lower.

A model gives probability 0.70 to the outcome that was recorded, and another gives 0.10. Which scores the lower negative log-likelihood, and by roughly how much?

Chapter 9

The Formulation: Encode, Predict, Read Out

Write the whole world model as one conditional distribution, with a latent state that stands in for the history

A video shows a ball at one spot. One frame cannot say whether it is rolling left, rolling right or sitting still. Two frames can. Ten frames can say how fast, and how sure we are. The same facts apply to any evidence about the world. The formulation in this chapter is a careful way to say how much evidence we hold, and what we ask of it.

Every earlier chapter feeds in. History, action, target and horizon come from Chapter 6. The network comes from Chapter 7. The distribution and the loss come from Chapter 8. Here they snap together.

One step

The smallest question is the one-step prediction.

pθ
The model, with learned numbers θ.
Yt+1
The random next target.
ht
The history of observations and past actions.
at
The one candidate action.

It predicts a distribution over the next target from history and one candidate action. That is Chapter 8, line for line.

Many steps: a distribution over a trajectory

Often we need a whole stretch of the future. A trajectory is a sequence of states or targets over time. The multi-step form gives the model a series of candidate actions at to at+H−1 and asks for a joint distribution over the next H targets.

pθ(Yt+1:t+H | ht, at:t+H−1)

The notation t+1:t+H stands for the whole block of targets. The key word is joint. The model assigns a chance to whole paths, so it can say that two outcomes tend to go together, for example that if the block starts leaning at step 2, it probably collapses by step 5.

Observation, history and latent state

Now the hard part. The lecture draws a chain of four boxes. The physical world has a complete state, usually not fully observed. A measurement gives the current observation ot. The available evidence is the history ht, observations plus past actions. And a learned summary, the latent state, compresses that evidence: zt = Eθ(ht). The word latent means hidden: no sensor reads it, the model infers it.

This is partial observability, which you may know from our POMDP lesson: the observation does not reveal the full state. The latent state is the model’s answer. It is a compact vector that keeps what the prediction needs from the whole history, so the model need not re-read every frame.

The latent form of the model replaces the history with the summary: pθ(Yt+1:t+H | zt, at:t+H−1). It is built from three parts, which the lecture draws as a pipeline.

  1. Encode. An encoder Eθ reads the history ht and writes zt.
  2. Predict under action. A predictor takes zt and the candidate actions at:t+H−1 and computes what the future looks like.
  3. Read out. A distribution head turns that into the output distribution, such as the softmax or the (μ, σ) of Chapter 8.

These may be separate modules or one shared network. The formulation does not care. It only fixes what goes in and what comes out.

Try it. The device below follows a cart whose speed you cannot see. It reads only position. You choose how many frames the model has seen, how far ahead to look, and which push you will make. Orange lines are possible futures drawn from the model. The more history, the tighter the fan.

Sample the futures

Blue dots are position readings in the history. Orange lines are 30 sampled futures from the model. The blue line is what really happens. Change the history, the horizon or the push.

2
6

Toy cart: position moves by its speed each step, a push changes the speed by 0.06 per step, and the speed gets a small random nudge. The sensor reads position with noise 0.15. The model is the exact linear-Gaussian estimate of the position and speed from the frames seen, then 30 futures drawn from it (seeded). The true speed is 0.3.

With one frame, the cart’s speed is a guess from nothing, and the fan is wide. Add frames, and the speed estimate firms up and the fan narrows. Lengthen the horizon, and the fan widens again, because small errors in speed pile up over steps. Switch the push, and the whole fan shifts: that is the model answering a different question for a different action. The history shrinks the uncertainty about now. The horizon grows the uncertainty about later. Only the first of these can be bought with more evidence.

World modelling is a capability, not one model

The lecture broadens the view with one more idea. The same trained system may answer three different questions. An action-free interface predicts the world alone, pθ(Yworldt+1:t+H | ht), as a video predictor does. A supplied-actions interface predicts the consequences of actions you give, the canonical controllable query developed today. A joint continuation models the world and the action together, pθ(Yworldt+1:t+H, At:t+H−1 | ht), the style of recent world-action models. Lowercase a means supplied; uppercase A means modelled as random.

The joint splits by the chain rule of probability, one factor for the actions and one for the consequences.

pθ(Y, A | ht) = pθ(A | ht) × pθ(Y | ht, A)

Read the two factors as two queries. The action query, p(A | h), asks what tends to be done: a behaviour or policy interface. The world query, p(Y | h, A), asks what follows this action. The joint query asks how both co-evolve. The slide ends with two cautions. Predicting only actions does not expose world consequences. And modelling behaviour jointly does not by itself choose the best action; choosing is a later job.

In the device, you add more frames to the history and the fan of futures gets narrower. What does the extra history reduce?

Chapter 10

One Worked Case: One-Step and Rollout Objectives

Specify the block-push model from start to end, then compare training on one step with training on a whole segment

Back to the robot arm and the tower of three blocks. The lecture now takes one question, what happens after we push the blue block?, and specifies a model for it from start to end: history, latent state, action, target, uncertainty and objective. We follow the slides in that order. This is the same toy world as the hero at the top of the page.

Step 1: the evidence

One image of the tower shows where the blocks are. It does not show whether they are moving, or whether the blue block is already pressing against the next one. Three recent frames, joined by what the robot did between them (a previous push, a contact change), show motion that one image hides. So the evidence is the history ht = (o≤t, a<t), and the encoder compresses it into zt = Eθ(ht). That is Chapter 9’s first two boxes, now with a real tower in them.

Step 2: candidate actions

A push needs three numbers: direction, magnitude and duration. The lecture compares two candidates. Candidate A is left, light and short: a brief, low-magnitude push on the same blue block. Candidate B is right, hard and long: a longer, higher push in the opposite direction. The slide stresses that the action sequence is supplied to the model. Selecting an action belongs to a later decision system.

Step 3: fix the query

A vague question gets a vague model, so the lecture fixes three things. The target is tower pose plus contact: predict whether blocks move, touch, separate or begin to fall. The horizon is the next H steps, following the early response through possible loss of stability. The uncertainty is acknowledged up front: friction, mass, contact, occlusion and execution are only partially known, so several outcomes remain. In symbols, Yt+1:t+H is the set of future pose and contact variables.

Step 4: each push makes a different future distribution

With the same latent state zt, the two actions give two distributions: pθ(Yt+1:t+H | zt, aAt:t+H−1) and pθ(Yt+1:t+H | zt, aBt:t+H−1). The slide describes them in words. After the light push left, most of the probability may stay near upright, with some chance of leaning. After the hard push right, probability can shift toward sliding, separation or collapse. This is what you played at the very top of the page, with made-up numbers. The model does not pick the better push. It only answers, for each push, what the future looks like.

Step 5: two ways to build it

The formulation fixes the interface, not the machine inside. The lecture shows two builds that fit it. A modular visual sequence model: frames go through a shared CNN, an RNN keeps memory and takes the action, and a distribution head reads out pθ(Y | zt, a). A fused token model: frame patches, action tokens and a time signal all enter one Transformer, followed by a distribution head, giving pθ(Y | ht, a). Both take history and action in, and both give a distribution over Y out. This is Chapter 7’s table put to work.

Step 6: one-step and sequence losses differ

Training needs a score, and the same model can be scored two ways. The one-step loss supervises one immediate target per training window. The sequence loss scores a whole observed segment.

L1 = −log pθ(yt+1 | ht, at)    versus    LH = −log pθ(yt+1:t+H | ht, at:t+H−1)

In LH, one model call scores the chosen future segment. The lecture adds a careful warning: a sequence target does not by itself imply recursive rollout or an autoregressive factorisation. A model may predict all H steps at once.

There is also a third way, the one the syllabus names: a rollout objective. Run the model for H steps by feeding its own prediction back in as the next input, and score the result against the real future. This is the way a planner or an imagination loop uses a model, so it is the way we should often measure it. The two losses can disagree, and the device shows how. The true dynamics here are a curve. The model is a straight line, x′ = αx + β, which cannot be exact. Choose how many steps to roll out during training and search for the line that does best.

Train one step, test many

Top: the true next-step curve (blue) and the model line (orange). Bottom: how far off the model is after 1 to 20 steps of rolling its own predictions forward. The grey dashed curve is the model trained on one step only.

10

Toy world: x′ = 1.2·x·exp(−x²/6), which settles near 1.05, from 20 start points between 0.2 and 3.0. The model is the line x′ = αx + β. Each line is found by a real grid search over α and β for the smallest mean squared error after the chosen number of rolled-out steps. The grid search runs in your browser, with no noise in the data.

At length 1 the line is the best single-step fit, and it is off by 0.24 on average after one step, which is the best a straight line can do there. Slide the length up. The line changes its slope and its offset, and at length 10 it is wrong by 0.68 after one step but only 0.03 after twenty. It has given up the first step to get the long run right. At length 20 it becomes almost a flat line that predicts the resting value. None of these is the “correct” model. Each is the best answer to the question its loss asks. So the loss must match how the model will be used: a one-step loss for one-step predictions, a rollout loss for planning and imagination.

Step 7: the same recipe in other worlds

The last slide of this part is the synthesis. The same conditional distribution, pθ(Yfuture | available history, candidate actions), fits many domains.

DomainHistoryCandidate actionFuture target
Physical systemframes and sensorsforce or motor commandpose, contact or pixels
Gamescreens and past movesjoystick or discrete movescreen, event or value-relevant feature
Software agentpage state and interaction logclick, type or commandnext interface state or task event
In the device, the line trained on 10-step rollouts is worse after one step than the line trained on one step, but much better after twenty. What does that show?

Chapter 11

Field Guide

Keep the numbers, the terms, the checklist and the core code, and see where Lecture 3 picks up

Here is the whole lecture in one paragraph. The idea of a world model is old: a selective stand-in for the world that lets you try before acting. Control added feedback, predictive state and uncertainty. AI added written rules, and system identification added fitted ones. Reinforcement learning put the model inside the learning loop. Deep learning scaled it to pixels and latents. All of it reduces to one question, pθ(Yfuture | history, candidate actions), which needs data, a network, a distribution and a loss to answer.

The numbers worth keeping

NumberWhat it isWhere
1943Craik: the small-scale model, a selective surrogateChapter 1
Day 11, Day 12Food first found, then errors collapse: latent learningChapter 1
1948, 1957, 1960, 1970Wiener, Bellman, Kalman, Conant and AshbyChapter 2
1971STRIPS: successor = (state − delete) ∪ addChapter 3
1978, 1990MPC; Dyna and two recurrent networksChapter 4
17.5 secondsReal interaction PILCO needed for the swing-up taskChapter 5
12,288 numbersOne 64 × 64 colour frame (our example)Chapter 6
2.30 and 0.36−log of 0.10 and of 0.70, the slide’s two lossesChapter 8
L1 and LHOne-step and whole-segment lossesChapter 10

The terms, in one line each

TermOne line
Internal modelA selective stand-in for the world, used to rehearse before acting.
Observation otWhat the system can measure at time t.
Action atAn input applied to the world.
History htObservations up to t and actions before t.
Latent state ztA learned summary of the history, zt = Eθ(ht).
Horizon H and target YHow far ahead to look, and what to predict.
NLL−log of the probability the model gave to what happened.
Forward and inverse modelPredict the outcome of a command, or pick the command for an outcome.
RolloutFeed the model’s own prediction back in to look several steps ahead.

A checklist for specifying a world model

  1. Say what you observe. Write the shape of ot: pixels, sensors, text or screens.
  2. Say what you do. Write the shape of at, and who supplies it.
  3. Say what the history holds. Put o≤t and a<t in ht, and leave at out.
  4. Say what you predict. Choose the target Y and the horizon H.
  5. Say where the state lives. Choose an encoder, so that zt stands in for the history.
  6. Say what the output is. Choose a distribution head, categorical or Gaussian.
  7. Say how you score it. Pick L1, LH or a rollout loss to match the way you will use it.

The core as code

This is the formulation in about thirty lines of Python. It is a sketch to read, with the shapes in the comments. It uses a Gaussian head, a recurrent latent and the two training losses of Chapter 10.

pythonimport numpy as np

class WorldModel:
    """p(Y_next | z_t, a_t) as a Gaussian, with a recurrent latent state z."""
    def __init__(self, d_obs, d_act, d_z, rng):
        s = 0.1
        self.We = rng.normal(0, s, (d_z, d_z + d_obs + d_act))  # encoder: (z, o, a_prev) -> z
        self.Wp = rng.normal(0, s, (d_z, d_z + d_act))          # predictor: (z, a) -> next z
        self.Wmu = rng.normal(0, s, (d_obs, d_z))               # head: z -> mean of Y
        self.logsig = np.zeros(d_obs)                            # head: log scale of Y

    def encode(self, z, o, a_prev):                          # fold one new observation into the latent
        return np.tanh(self.We @ np.concatenate([z, o, a_prev]))

    def step(self, z, a):                                    # predict under a candidate action
        z2 = np.tanh(self.Wp @ np.concatenate([z, a]))
        return z2, self.Wmu @ z2, np.exp(self.logsig)         # next latent, mean, scale

def nll(y, mu, sigma):                                       # -log N(y; mu, sigma^2), summed over dimensions
    return np.sum(0.5 * ((y - mu) / sigma) ** 2 + np.log(sigma) + 0.5 * np.log(2 * np.pi))

def loss_one_step(m, z, a, y):                               # L_1: z comes from the REAL history
    _, mu, sg = m.step(z, a)
    return nll(y, mu, sg)

def loss_rollout(m, z, acts, ys):                            # only the FIRST z comes from real data
    total = 0.0
    for a, y in zip(acts, ys):
        z, mu, sg = m.step(z, a)                                 # feed the model's own latent forward
        total += nll(y, mu, sg)
    return total / len(ys)

Where it sits in the field

This lecture is the spine of the course. Lecture 1 asked the three questions. Lecture 2, this one, defined the prediction question. Lecture 3 asks where the experience that trains such a model comes from: environments, simulators and rollouts, with the reset and step calls, trajectories and experience collection. Our lessons on world models, model-based RL, Markov decision processes and POMDPs cover neighbouring ground, as do the Kalman filter, the Bayes filter, backpropagation, loss functions, the transformer, JEPA and NeRF and 3D Gaussians. The Genie paper lesson goes deeper on interactive video worlds.

The takeaway. A world model is judged by what it enables, not by how literally it copies the world. Written down exactly, it is one conditional distribution over futures, given a history and candidate actions, trained so that the real outcome gets high probability. History, state and uncertainty came from control. Rules and their fitting came from symbolic AI and system identification. Practising inside the model came from reinforcement learning. Scale came from deep learning.

Credits

This lesson follows Lecture 2 of CIS 6280: World Models at the University of Pennsylvania, Fall 2026, taught by Jiatao Gu. The slides are the course’s own, and every idea here is re-taught in our own words and devices. The history of model-based RL in Chapters 0, 4 and 5 draws on Loghman Samani’s essay. Toy numbers in the devices are marked illustrative where they appear.

Which pair names the next lecture’s question and its topics?

Now press Present or Teach and explain from memory how one question grew from Craik’s rehearsal to a conditional distribution: what each field added, what the history holds, why the latent state exists, and why a one-step loss and a rollout loss can disagree. Then go back to the block at the top, push it 20 times with a point model and with a distribution, and say why honest odds score better.

CIS 6280 Lecture 2: History, Foundations and Probabilistic Formulation
The course site · Lecture slides · Samani, a history of model-based RL · Back to Gleams