A simulator turns one push into a next observation: it moves the pusher for ten tiny steps, finds where the pusher touches the T, and turns that touch into motion.
Learn how an environment hides a physics engine behind two calls, reset and step, and how a policy collects experience from it.
Pick where the pusher hits the T, then press Run 8 actions. A high hit and a low hit spin the T in opposite ways. Then we build, piece by piece, everything underneath: the controller, the clocks, the contact and the rollout.
You need a little Python and the idea of a speed (distance per second). We build the rest from zero.
Based on Lecture 3 of Penn CIS 6280: World Models (Fall 2026), taught by Jiatao Gu, and its reading, the Gymnasium basic usage guide. Lecture slides. Every idea is re-taught here in our own words and devices.
A small JavaScript stand-in for the lecture’s Pymunk world, not Pymunk itself. The lecture’s numbers are kept: action scale 100, gains kp = 100 and kd = 20, ten physics steps of 10 ms per action, friction 0.85, rest hit e = 0. The 512 by 512 arena (the usual PushT size), the ground drag on the T (6 per second) and the T’s mass of 1 are our own illustrative choices.
Chapter 0
Follow a single push from the action you choose to the pixels you see next
Picture a small robot arm at a table. A blue pusher sits next to a grey block shaped like the letter T. You tell the pusher: move right. A moment later the pusher touches the T. The T slides, and it also turns a little. What does the camera above the table show next?
Somebody must compute that answer. In the real world, nature does it for free. In a computer, a program must do it. This lecture is about that program: how it is built, how a learner talks to it, and where it stops being true.
CIS 6280 is a course about world models: programs that predict what a world does next. Lecture 3 starts with the oldest and simplest kind, a physics simulator, where a person writes the rules down by hand. The running example is a puzzle called PushT. Every later chapter takes one piece of it apart.
Lecture 2 left us with one idea to carry forward: many futures fit the same past. The T could slide left, slide right or spin. To ask about one of them, you must say four things.
First, what to predict, written Y: the T’s position, or the next camera image. Second, the history you have so far, written ht. Third, the actions you will take, written at onward. Fourth, the horizon, written H: how many steps ahead to look. The lecture packs all four into one line.
Try it. The device below holds the history fixed: the pusher at the left, the T at the right. You choose the actions and the horizon. The world answers with one future.
The faint T is where it sits now. The solid T is the answer for your push after H actions. The three faint outlines are the answers to the other pushes. Change the push, then drag the horizon.
Each future is computed live by this page’s small PushT engine, with a push strength near 0.6 for the three moving choices. One action lasts 0.1 s, so H = 6 looks 0.6 s ahead. The numbers are the engine’s, not the lecture’s.
A simulator is the special case where the model, pθ, is a program that gives one sure answer. Later lectures replace it with learned models that give probabilities. The query keeps the same four parts. Only the machine that answers it changes, and that is why we start with the machine we can read line by line.
Look at what the device did. Same history, four different futures. The question has an answer only after you pick the actions. A question that names actions is an action-conditioned question. World models for robots live on this kind.
The lecture draws PushT with four roles. A static boundary is the fixed wall of the arena. It never moves. The goal overlay is a T outline that only scores the task: nothing collides with it. The two bodies that matter are the pusher and the T.
The kinematic pusher follows commanded motion. A kinematic body does what its controller says, and a contact does not change that. The dynamic T-block responds to contact. A dynamic body moves according to the forces on it. Think of a car you steer, and a shopping cart you shove. The car follows you. The cart follows physics.
Both are rigid bodies. A rigid body keeps its shape while the whole body slides and turns. Real blocks bend a tiny bit. The simulator ignores that, and Chapter 10 shows what it costs.
Now the key move of the lecture. The T moved, and a camera saw it. But the camera did not decide where the T went. Something deeper did.
Inside the simulator lives the state, written st: every number the physics needs. For PushT the state holds the pusher’s position and velocity, and the T’s position, angle, velocity and spin. The agent never sees these numbers directly. It sees an observation, written ot. In PushT the observation is a picture, a grid of pixels. The simulator makes it in two separate jobs.
This split matters for everything that follows. The physics step is the part a person writes down by hand, from mechanics. The observation step is a choice of what to reveal. You can swap the second without touching the first. Show the learner pixels, or show it the raw numbers, or add camera noise. The world underneath stays the same.
A picture shows where things are. It can hide how fast they move, and it can hide which body is which. A single frame of the T tells you its pose, not its velocity. Two frames that look the same can start two different futures. Chapter 1 puts a number on that.
The lecture has five movements. First comes the PushT world and the environment object that wraps it (Chapters 1 to 3). Then the engine inside one step: control, time and contact (Chapters 4 to 7). Then the leap to robots (Chapter 8). Then rollouts, the experience a simulator makes (Chapter 9). Last come the limits (Chapter 10). The reading, Gymnasium’s basic usage guide, sits in Chapter 2.
reset, step and what each returns.Chapter 1
See why one push can slide the T and spin it at the same time
Put a paperback on a desk. Press one fingertip against the middle of its edge and push. The book slides straight. Now press near a corner with the same push. The book slides and also swings round. Same finger, same push, different result.
The T in PushT does the same thing. The lecture shows one push and asks one question: why does the T both translate and rotate? This chapter answers it with three ideas. Pose and motion say what the T is doing. Force and torque say what changes it. Impulse says how a touch delivers the change.
The pose of the T is where it is and which way it points: a position (x, y) and an angle. The motion of the T is how the pose is changing right now: a linear velocity (how fast the position changes) and an angular velocity, written ω (how fast the angle changes, in radians per second). The pusher has a position and a velocity too.
The lecture states the rule in one line: the same pose now, with a different velocity, gives a different next pose. So the simulator’s state stores both pose and motion. A photograph holds only the pose.
Take a worked case. The T sits at x = 300 px with angle 0. In case A it rests. In case B it moves right at 60 px per second and turns at 1 radian per second. Wait 0.1 s. Case A is still at x = 300. Case B is at x = 300 + 60 × 0.1 = 306, and its angle is 1 × 0.1 = 0.1 radian, about 5.7 degrees. Both photographs at time zero were the same.
What changes a velocity? A force, a push or a pull. The lecture gives two laws, one for each half of the motion. For sliding, F = ma: a force F gives a body of mass m an acceleration a (a change of velocity per second). Mass resists the change, so a heavy block speeds up slowly.
For turning, τ = Iα. Torque, written τ (tau), is the turning effect of a force. Rotational inertia, written I, resists turning, the way mass resists sliding. α (alpha) is the angular acceleration, the change of spin per second.
Torque depends on where the force lands. The distance from the T’s center to the line of the force is the lever arm, written r. A force aimed straight through the center has no lever arm: it gives no torque and only slides the T. A force that misses the center has a lever arm, so it slides the T and turns it. This is the paperback again. The corner has a long lever arm.
In a real contact the force is not steady. The surfaces squeeze, push back, and let go, all in a few thousandths of a second. The force climbs, peaks and falls. Tracking that curve costs a lot of computing.
The simulator uses a shortcut. It adds up the force over the whole touch and calls the total an impulse, written J. The lecture writes J as the integral of F dt, which means “force times time, summed over the tiny slices of time.” Then it applies the whole impulse at one instant. The velocity jumps.
The lecture is honest about this step. A real contact deforms, vibrates and makes sound and heat. The rigid-body model keeps the total impulse and drops the rest. A velocity jump is a good guess, not the truth.
Now play with the lever arm. The impulse below always points right and always has the same size. Only the height of the hit changes.
The pink arrow is one impulse of 100 units, always pointing right. Drag the height of the hit. The dashed T shows the pose 0.3 s later. The purple bar shows the spin.
Illustrative units from this page’s engine: mass 1, rotational inertia 1,932, heights in pixels from the T’s center (negative is up). Dividing J = 100 by the mass gives the slide; the lever arm times J, divided by the inertia, gives the spin. The dashed T ignores ground drag.
Check the device by hand at a height of 30 px above the center. The lever arm is 30 px. The cross product is r × J = 30 × 100 = 3,000, with the sign that means clockwise. Divide by the rotational inertia, 1,932: the spin jumps by 3,000 / 1,932 = 1.55 rad/s. The slide jumps by 100 / 1 = 100 px/s. After 0.3 s the center has moved 100 × 0.3 = 30 px, and the T has turned 1.55 × 0.3 = 0.47 radian, about 27 degrees. Those are the numbers the dashed T shows.
Three things stand out. The slide is the same at every height: it is 100 divided by the mass. The spin changes with height. At height zero the lever arm vanishes and the spin vanishes. And the sign flips: a hit above the center spins the T one way, a hit below spins it the other way. Your hero at the top of the page did exactly this with the upper, center and lower hit.
One more detail. The T’s center of mass is not at the middle of its bounding box, because the bar on top holds more of the weight than the stem. So the “center hit” of the hero is a hit at the center of mass, not the middle of the picture. Where the contact lands decides the story.
Chapter 2
Wrap a world in reset and step, so a policy can talk to any simulator the same way
Think of a game console and a game. You do not need to know how the console draws the screen. You press a button and the screen changes. Two things matter to you: how to start a new game, and how to press the next button.
A learner wants the same deal from a world. It should not need to know about physics engines. It should call two things, one to start over and one to act. The lecture builds that deal in PushT, and the course reading, the Gymnasium basic usage guide, describes the same two calls for every environment.
First a short lesson in Python. A class is a recipe for a kind of thing. An object (also called an instance) is one thing made from the recipe. A method is a function that belongs to the object. The object keeps its own data, so each object has its own memory.
Slide 11 of the lecture uses this to define an environment. The class PushTEnv extends Gymnasium’s gym.Env. It owns a world in self.state. It has two methods.
pythonimport gymnasium as gym class PushTEnv(gym.Env): def reset(self, *, seed=None, options=None): self.state = make_initial_state() return observe(self.state), {} # (observation, info) def step(self, action): self.state = transition(self.state, action) # the physics obs = observe(self.state) # the pixels return obs, reward(self.state), False, False, {} # obs, reward, terminated, truncated, info env = PushTEnv() # one instance = one world env.step(action) # acts on THIS world
Read the code like a story. reset builds a fresh world and returns the first observation. step takes an action, runs the physics to get a new state, draws an observation and returns five things. The call env.step(action) changes the world that env owns, and no other. Make a second instance and you have a second, separate world.
The reading names the five returns. Observation: what the agent can see. Reward: a number that scores the step, positive for good outcomes and negative for bad ones. Terminated: true if the task ended by itself, by success or failure. Truncated: true if we cut the episode short, for example at a time limit. Info: extra data for debugging. One action and the reply to it make one timestep. A run from reset to the end is an episode. The reading ends the loop when terminated or truncated is true.
The environment also declares what it accepts and returns. env.action_space and env.observation_space describe the valid shapes. PushT’s action is two continuous numbers, which fits a Gymnasium Box, the space for bounded numbers of any shape. env.action_space.sample() draws a random valid action, which is how the reading’s first loop plays.
Try it. Two worlds sit side by side. Choose which one receives your calls. Reset with a seed, then step.
Each call goes to the chosen object only. Watch the other world stay put. The box below the picture shows the call and exactly what it returned.
A toy environment on this page’s engine. Its observation is five numbers (pusher x and y, T x and y, T angle) instead of pixels. The reward is the negative distance from the T to the goal at (400, 256), divided by 100. Terminated means the T is within 20 px of the goal and within 15 degrees of upright. Truncated means 50 steps. These rules are illustrative, not the lecture’s.
Two details are worth catching. A new reset(seed=7) always gives the same first state: the seed makes the start repeatable. And a step that moves nothing still returns a full tuple. The shape of the reply never changes. That is the whole point of the interface.
The reading’s own example shows the loop on CartPole, an environment that asks you to balance a pole on a moving cart. Its reward is +1 for every step the pole stays upright. It truncates the episode at 500 steps and terminates it when the pole falls too far. A random policy that drops the pole after, say, 22 steps (an illustrative number) scores 22. A perfect policy scores 500. The sum of rewards over an episode is its return, and Chapter 9 uses it to compare policies.
Inside transition lives the physics. Slide 13 gives it a name worth remembering: a physics simulator is a world model. It is predictive: it answers what happens after an action. It is executable: you can call it again and again to generate a future. It is physics-based: its rule comes from mechanics, geometry and numerical methods, not from data. And it is still a model: rigid bodies, friction, time steps and parameters only approximate reality.
Why run the world in software? The lecture gives four reasons, and each one is something a real robot cannot easily do. Control: choose the initial state and repeat exactly the same condition, even a hidden velocity and a random seed. Intervention: branch from one state and try another action to compare two futures. Scale: run many trials, faster, cheaper and more safely than on a robot. Inspection: read every variable, including pose, velocity and contact forces that a camera hides.
Slide 15 stops us from confusing a tool with a role. Three roles appear in every setup. The policy reads an observation and chooses an action. The environment interface defines actions, observations, reward, reset and termination. The transition mechanism advances the internal state.
This lecture fills them with a Python policy, a Gymnasium-compatible PushT, and the Pymunk rigid-body library. Another course could use another interface or a robot bridge, and another engine or the real world, without changing the roles. Gymnasium and Pymunk are today’s choices. They are not the definition of an environment.
Slide 12 previews where we go next: one env.step(action) hides a seven-part chain from state and action to the returned tuple. The next chapters open that chain one link at a time.
env_a = PushTEnv() and env_b = PushTEnv(), then call env_a.step(action). What happens to env_b?Chapter 3
Turn two numbers into a place for the pusher to go, and meet the three objects of the physics engine
Hold a joystick and push it a little to the right. The character does not teleport to the edge of the screen. It starts heading right. How hard you push says how eager it is. The stick gives a direction and an eagerness, never a place.
The pusher in PushT takes its orders the same way. The policy hands the environment two numbers. The environment turns them into a spot that the pusher should head for. This chapter shows that translation, then opens the toolbox that the physics engine uses to hold the world.
The lecture writes the action as at = (ax, ay). It has four properties, and each one prevents a mistake.
Two components. One number says how far to go sideways, the other how far to go up or down. Dimensionless. The values are scaled by the environment. They are not pixels, not Newtons and not an angle, so you cannot read ax = 1 as “1 pixel”. Relative. The target starts from where the pusher is now, not from the corner of the arena. Held fixed. One target stays in place for all ten physics steps of this action.
The lecture’s worked case. The pusher sits at pt = (273, 195). The action is at = (1, 0), a full push to the right. Then p* = (273, 195) + 100 × (1, 0) = (373, 195). The target lies 100 pixels to the right on the same row. A half push, at = (0.5, 0), would put it at (323, 195).
Why make the target relative? Suppose the pusher moves 16 px during the first action, and the policy sends the same action again. The new target lies 62 px from the new position, so the target travels with the pusher. A policy that always says “half a push to the right” keeps pushing right, wherever the pusher is. If the action named an absolute spot, the policy would first have to read the pusher’s coordinates to say “keep going”.
Why unitless? A learned policy usually outputs small numbers between −1 and 1, which a neural network produces easily. The environment, not the policy, knows that 1 means 100 px. Change the scale to 50 and the same policy takes half-size steps with no change to the policy. The interface hides the units. In Chapter 8 the same slot holds a joint target or a tool target for a robot, and the rest of the pipeline does not notice.
One more fact is easy to miss. The target is a reference, not a promise. The pusher heads there, but the action lasts only a tenth of a second. Does the pusher arrive? Run it below.
The pusher starts at (273, 195), the lecture’s example. Drag the two numbers of the action. The dashed ring is the target p*. Then press Run one action and watch how far the pusher gets.
The scale 100, the start (273, 195) and the first action (0.62, 0) come from the lecture’s slides. Positive ay points down the screen, as the pixels do. The distance the pusher covers comes from this page’s engine with the lecture’s gains (kp = 100, kd = 20).
The pusher covers only about three tenths of the way in one action: 18 of the 62 px at the default setting. It is still speeding up when the action ends. The next action sets a new target from the new position, and the pusher keeps its speed. So a push is something that builds across actions. Chapter 4 explains the controller that causes this.
Now the toolbox. Pymunk, the 2D physics library behind PushT, stores a world in three kinds of object.
A body holds motion state: position, angle, linear velocity, angular velocity, mass and inertia. A shape holds collision geometry: a circle or a polygon, plus friction and elasticity (how bouncy it is). A shape attaches to a body. A space holds bodies and shapes. Its method space.step(dt) advances the motion and resolves contacts.
Why split body and shape? Motion and geometry are different questions. The body answers “how does it move?” The shape answers “where does it touch?” You can change one without the other.
| Part of PushT | Body | Shape | Who decides its motion |
|---|---|---|---|
| Blue pusher | kinematic | a circle | the controller prescribes it |
| Gray T-block | dynamic | polygons | contact determines it |
| PushT world | one space holds both | every shape is registered | space.step(dt) advances all of them |
The instructor builds this world in a Colab notebook in four steps. The code is short enough to read whole.
pythonimport pymunk space = pymunk.Space() # 1. the 2D simulated world pusher_body = pymunk.Body(body_type=pymunk.Body.KINEMATIC) # 2. motion prescribed by control pusher_shape = pymunk.Circle(pusher_body, 15) t_body = pymunk.Body(mass=1.0, moment=moment) # 3. motion determined by physics t_shape = pymunk.Poly(t_body, vertices) space.add(pusher_body, pusher_shape, t_body, t_shape) # 4. register before stepping
Think of a toy theatre. The space is the stage. Each body is an actor with a position and a speed. Each shape is the costume that decides where the actor can bump into the others. The director calls space.step(dt), and every actor moves one beat.
Step 4 is the one beginners forget. The engine can only update objects that live in the space. A body that was never added is a ghost: it exists in your code and not in the world. Also notice moment, Pymunk’s word for the rotational inertia of Chapter 1. You must supply it, because the engine cannot guess it from a shape.
This page’s T is a stand-in with the same job: a bar 120 px wide and 30 px tall on top of a stem 30 px wide and 90 px tall, two rectangles glued into one body. The lecture does not give its polygon sizes, so the numbers here are ours.
Chapter 4
Build the PD controller that turns a distance into a push, and tune its two gains
You drive toward a stop sign. You do not plan every inch of pedal in advance. You look at how far away the sign is, and how fast you are going, and you adjust. Far and slow: press the gas. Close and fast: brake. You repeat this many times a second.
That habit is called feedback: measure where you are, compare it with where you want to be, correct, and measure again. The pusher in PushT needs it. A target 62 px away does nothing by itself. Something must turn the remaining distance into motion, again and again.
The lecture draws the loop as four boxes (slide 21). Observe: read the pusher’s position p and velocity v. Compare: compute the error, using the latest position. Correct: update the motion from the error and the velocity. Advance: step the world, which gives a new p and v. Then read the new state and repeat.
One thing never changes inside the loop: the target p*. One action holds p* fixed. Only the feedback is recomputed from the newest state.
The position error is a vector from the pusher to the target.
The simplest correction pulls the pusher toward the target, harder when it is farther away. P stands for proportional: the correction is proportional to the error. In symbols, aP = kp e. A small error gives a small pull and a large error gives a large pull. The number kp is a gain, a dial that sets how aggressive the pull is. Think of a spring that links the pusher to the target. The gain is the stiffness.
A spring alone is a problem. Pull a ball on a spring and let go: it overshoots, swings back, overshoots again, and never settles. We need something that drags on the motion.
D stands for derivative, which is the rate of change. The target is fixed, so as the pusher moves toward it, the error shrinks at the pusher’s speed: the rate of change of e is −v. The D term pushes against that velocity: aD = kd × (rate of change of e) = −kd v. A fast pusher gets a strong brake. A slow pusher gets a weak one. This is damping, like pushing a hand through water. The gain kd sets the strength of the water.
Add the two. The lecture’s PushT code uses this one line.
The lecture’s first step: the pusher is 50 px from the target and at rest, so e = 50 and v = 0. Then acmd = 100 × 50 − 20 × 0 = 5,000 px/s². With no speed yet, only the pull speaks. PD is a structure, not a recipe: other robot controllers keep the same two terms and command a torque, a velocity or a force instead of an acceleration.
Now watch what the two gains do. The device simulates one fixed target 100 px away, starting at rest, with the lecture’s time step of 0.01 s.
The purple curve is the pusher’s position over 1.5 s. The dashed line is the target. Try the lecture’s gains first, then set the brake kd to zero.
The simulation is exact for this one-dimensional pusher: velocity first, then position, with dt = 0.01 s, as in the lecture. The target of 100 px is illustrative. “Settled” means within 5% of the target and staying there.
With the lecture’s gains, the pusher glides to the target and stops, with no overshoot. That is not luck. With kp = 100, the matching brake is kd = 2 × √100 = 20, the balance point where the spring and the water just cancel the swing. Less brake overshoots. More brake is slow.
Here is the failure in the order you meet it. Symptom: the pusher swings past the target and back, again and again. Measurement: with kd = 0, the position climbs to 200 px, twice the target, and never settles. Fix: raise kd until the overshoot is gone, and no higher.
The device hides arithmetic worth seeing once. The pusher is at 0, the target is at 50, the gains are the lecture’s, and each step updates velocity first, then position (Chapter 5 explains why).
| Step | Error e | Speed v before | acmd = 100e − 20v | New v | New p |
|---|---|---|---|---|---|
| 1 | 50.000 | 0.0 | 5,000.0 | 50.0 | 0.500 |
| 2 | 49.500 | 50.0 | 3,950.0 | 89.5 | 1.395 |
| 3 | 48.605 | 89.5 | 3,070.5 | 120.2 | 2.597 |
The pull shrinks as the error shrinks, and the brake grows as the speed grows. After 0.03 s the pusher has covered only 2.6 of 50 px. It starts slowly because it starts from rest. That is exactly why Chapter 3’s pusher got only about three tenths of the way in one action.
One last point from slide 25. The pusher is a kinematic body in Pymunk. Its motion is prescribed, so the course code updates its velocity directly: pusher.velocity += a_cmd * dt. That skips solving for acceleration from a force with F = ma. The gray T is a dynamic body. Pymunk finds its response from mass, inertia, its current motion and the contact impulses of Chapter 1.
So the PD command never touches the T. The chain is: PD command, then pusher velocity, then contact impulse, then T motion. The pusher must reach the T before the T moves.
Chapter 5
Advance the world with semi-implicit Euler, and separate the control clock from the physics clock
A film looks like smooth motion. It is really 24 still pictures every second, and your eye fills the gaps. A simulator works the same way. Nature moves continuously. A computer cannot. It can only jump from one frame to the next, and the size of each jump is a choice.
This chapter is about that choice. It explains how the pusher’s motion is advanced one small step at a time, and why one action from the policy hides ten of those steps.
Physics describes the pusher between frames with two relations. Velocity is the rate of change of position. Acceleration is the rate of change of velocity. The lecture writes them with a dot, which means “change per second”.
A computer cannot follow a curve exactly. It does numerical integration: it takes a small time step, written dt, and moves along the current slope for that long. Each step has two updates.
Use the lecture’s first step. The error is 50 px and the pusher is at rest, so acmd = 100 × 50 − 20 × 0 = 5,000 px/s². With dt = 0.01 s, the velocity jumps by 5,000 × 0.01 = 50 px/s. In the course code you see only the velocity line, pusher.velocity += a_cmd * dt. The position advance happens inside space.step(dt).
The order matters more than it looks. The lecture compares two schemes (slide 28). Forward Euler updates the position first, using the old velocity, then the velocity. The acceleration changes v now, but the position only responds on the next substep. Semi-implicit Euler, also called symplectic Euler, updates the velocity first and then moves the position with the new velocity. The course uses semi-implicit, because the new velocity moves the pusher at once, and it is usually more stable for mechanical motion at the same dt.
By hand, from rest with acmd = 5,000: forward Euler ends step 1 with p = 0 + 0 × 0.01 = 0 and v = 50. Semi-implicit ends step 1 with v = 50 and p = 0 + 50 × 0.01 = 0.5 px. Half a pixel apart after one step. Does a half pixel matter? The lecture says it can: crossing a contact boundary switches the impulse J from zero to non-zero, which changes the torque and the rest of the rollout. A one-step difference in when the pusher touches the T can change where the T ends up.
Check it. The device below runs one action of the pusher toward a target 62 px away, and puts a wall in its way.
The solid purple curve is semi-implicit Euler, the course’s choice. The dashed warm curve is forward Euler. The pink line is a wall. Move the wall, then change the number of physics steps K inside one 0.1 s action.
One-dimensional pusher, target 62 px away, lecture gains kp = 100 and kd = 20, one action of 0.1 s split into K steps of dt = 0.1 / K. The wall is illustrative. The lecture’s own setting is K = 10, dt = 10 ms.
At K = 10 the two schemes differ by one substep in when they first reach the wall. Now drag K down to 1. A single big step of 0.1 s throws the pusher 62 px in one jump, so the wall is first noticed with the pusher far inside it. This is the usual failure of a time step that is too big. Symptom: objects pass through each other or sink in. Measurement: the depth of overlap at first detection, 54 px at K = 1 here. Fix: more substeps, so each jump is small.
The environment runs two clocks. The control clock is how often the policy chooses an action: 10 times a second, so one action lasts 100 ms. The physics clock is how often the world is advanced: dt = 10 ms. So one action holds K = 10 physics steps. In each one the controller runs, then the world steps.
The 100 ms is an interface choice, not a law of physics. A shorter interval lets the policy react more often. A longer one gives each action a longer effect and changes the rollout. Pick the control rate to match what the policy can decide in time, and the physics rate to match what the contacts need.
Slide 30 draws the substep as four owners. The controller reads p and v and computes acmd from the latest pusher state and the fixed target. The course wrapper integrates it: pusher.velocity += a_cmd * dt. The Pymunk space calls space.step(dt): it advances every body, finds contacts and changes the dynamic T through impulses. The environment reads the next state, where both the pusher and the T have a new pose and motion.
One misreading to avoid: space.step(dt) does not update only the T. Every substep has one controller update and one complete space step, and the space step moves everything it holds.
Chapter 6
Find where two shapes touch, and decide whether they are closing or separating
You stand in a crowded room and must find out whether anyone is touching. Checking every pair of people with a tape measure takes forever. So you do it in two passes. First you ask a cheap question: who stands close at all? Then you measure only those pairs.
Physics engines do exactly this. This chapter follows the lecture to the moment of first contact. It freezes the world at that moment and asks what the engine must find out: where the shapes touch, which way the surface faces, and whether the shapes are coming together or moving apart.
Slide 31 shows the loop of one action in ten substeps, and it stops the loop at the first substep that has a contact. In the course code that is the line if first_contact(): break. The simulated clock pauses on that frame. Everything below happens inside one substep, with time stopped.
Slide 32 lets you run the real thing: one persistent world, one 0.10 s action at a time, with editable settings for action scale, substeps, block mass, friction, damping and the two PD gains. The device in Chapter 3 and the hero on this page are our small versions of it.
What must the engine know at the frozen frame? The lecture says three things. Geometry: the exact circle and polygons and their current poses, which tell it where contact can occur. Motion: the linear and angular velocity of both bodies, because touching points may be moving together, apart or along the surface. Physical properties: mass, inertia, friction and elasticity, which set how strongly each body responds once the contact is found.
Slide 34 gives the two-pass idea in three steps. First, wrap each shape in a simple outer box. Second, keep only the pairs whose boxes overlap. An overlap is a possible contact, not proof. Third, check the real boundaries of those pairs only. The lecture gives each pass a name worth recognizing. The first is the broad phase, which finds possible pairs. The second is the narrow phase, which tests the exact shapes and outputs the local contact geometry.
The point is cost. Most boxes do not overlap, so the engine spends exact geometry work only on the few pairs that might touch. In PushT there are few bodies and the saving is small. In a robot scene with hundreds of parts it decides whether the simulator runs fast or crawls.
The narrow phase hands back three things (slide 35). The contact point c is where the two exact boundaries meet. The normal n is the local direction that separates the surfaces. Its sign depends on the order of the two bodies; the course defines it to point from the pusher to the T. The tangent t is a direction along the touching surface, perpendicular to n.
One warning from the lecture: the normal is a geometry direction. It does not say where either body is moving.
To decide whether the contact needs a response, the engine needs velocities. The point on the T that touches the pusher moves with both the T’s sliding and its spin: vT,c = vT + ωT × r. Here r is the arm from the T’s center to the contact point. In two dimensions, ω × r = (−ω ry, ω rx). Subtract the pusher’s velocity to get the relative velocity at the contact.
The sign decides. If vn < 0, the shapes are closing: the gap is shrinking and a normal response is needed. If vn ≥ 0, they are separating: the gap is growing, and a contact does not pull them back together. Contacts can only push.
A worked case with the device’s defaults. The pusher touches the T’s bar 30 px above the center, so r = (−60, −30). The normal points right, n = (1, 0). The pusher moves right at 100 px/s. The T is still but spins at 0.5 rad/s. Then ω × r = (−0.5 × (−30), 0.5 × (−60)) = (15, −30). So vrel = (0 + 15 − 100, 0 − 30 − 0) = (−85, −30). Its part along n is vn = −85. Negative: closing.
The dashed boxes are the broad phase. The pink dot is the contact point, the pink arrow the normal and the thin pink line the tangent. Move the hit point, the pusher speed and the T’s spin, and watch the sign of vn.
Computed live from this page’s engine with the T at rest at its centre and the pusher just touching its left face. Speeds are in pixels per second. Positive spin turns the T clockwise on screen. The grey box at the right stands for a far wall that the broad phase skips.
Slide through three cases. At the defaults the contact is closing: vn = −85. Pull the pusher speed to −100 and the pusher backs away: vn becomes positive, the chip says Separating, and the engine does nothing. Now leave the pusher at rest and spin the T fast the other way. The T’s own surface can run into a still pusher. Spin alone can make a contact close.
Chapter 7
Compute the impulse that stops the shapes closing, then add friction and several contacts
Roll a marble straight into a wall. It stops, or it bounces back. The wall did not push on it for a long time and then stop. It pushed for a blink, with exactly the strength needed to undo the marble’s inward speed, and no more.
Chapter 6 found the contact and told us the shapes are closing at some speed vn. This chapter answers the next question: how big is the push that stops the closing, and what does it do to the T? Then it adds sideways friction, and then the case where one T is touched twice.
Start with the simplest case, a center hit on the T (slide 40). The closing speed is vn = −2 m/s, the T’s mass is m = 2 kg, and there is no rebound, e = 0. The impulse must remove the 2 m/s of inward speed. A mass of 2 kg moving at 2 m/s has 4 units of push in it, so j = −m vn = 2 × 2 = 4 N·s.
Now the general rule. Two things change in an off-center hit. A hit with a lever arm can also turn the T, so some of the response goes into spin, and the T gives way more easily at that point. The lecture calls the result the effective mass at this contact: how easily the touching point can move along the normal. Its inverse sits in the denominator.
A worked off-center case, with illustrative numbers: m = 2 kg, vn = −2 m/s, e = 0, lever term r × n = 0.1 m and rotational inertia I = 0.04 kg·m². Then 1/m = 0.5 and (r × n)²/I = 0.01 / 0.04 = 0.25. The sum is 0.75, so j = 2 / 0.75 = 2.67 N·s. This is smaller than the 4 N·s of the center hit, because the T can turn out of the way. With e = 1 the numerator doubles, and so does j.
Then the T update is the one from Chapter 1: Δv = j n / m, and Δω = (r × j n) / I. Try it on the engine below.
The pusher touches the T and moves toward it. The pink arrow is the impulse the engine applies. The two bars compare the normal impulse and the friction impulse, with a notch at the friction limit. Change the hit, the speeds, the rebound and the friction.
The same contact solver as the hero: mass 1, rotational inertia 1,932, the T at rest. Impulses are in mass times pixels per second. The lecture’s friction value is 0.85. A positive surface speed moves the pusher down the screen along the T’s face.
Three things to try. Set the hit height to zero and watch the spin vanish while j rises to its largest value, because the T can no longer turn out of the way. Raise e from 0 to 1: j doubles at a center hit. Drag the speed into the T to zero: j drops to zero, because nothing is closing.
So far the pusher only presses straight in. Real pushers also slide along the surface, and surfaces grip. Slide 41 shows the same diagonal pusher motion on two surfaces. With low friction the pusher slips along the T’s face and the T gets little sideways push. With higher friction the surface motion transfers to the T.
The rule is Coulomb friction. The tangent t is the direction of relative sliding at the contact, and friction acts against it. The friction impulse jt may not exceed μ times the normal impulse: |jt| ≤ μ jn. If the impulse that would stop the sliding fits under that limit, the surfaces stick. If not, the contact slides, the friction impulse sits at the limit, and some motion remains. In the device the chip says which one you have. Friction changes how much of the pusher’s motion the T receives, so it changes the whole future trajectory.
Slide 42 adds a wall. The T now touches the pusher and the wall at once. You might solve each contact alone, then add the answers. That fails. The two contacts share one T, so they share its velocity and spin. Impulse J1 changes the T’s v and ω. Now the wall contact sees a different relative velocity. Impulse J2 then changes v and ω again, and the pusher contact sees a different velocity. The two answers are coupled through the T.
Pymunk handles this with repeated passes (slide 43). In one physics substep, space.iterations = 10 sets ten solver passes. Pass 1 uses the current velocities. Pass 2 reads the velocities that pass 1 updated, and so on. All ten passes happen inside the same interval from t to t + dt. They add no physical time.
Why do the passes converge? Take a toy with illustrative numbers. Each contact starts 1 unit short of a fix, and fixing one makes the other worse by half of the correction. Pass 1 fixes A, which leaves B at −1.5, then fixes B, which leaves A at −0.75. Pass 2 leaves A at −0.19. Pass 3 leaves −0.047. Each pass cuts the leftover error by a factor of 4. After ten passes it is about 0.000003. This is why the solver is cheap and good enough.
Keep two words apart. A physics substep calls space.step(dt) and moves time forward by dt. A solver iteration refines the contact impulses inside that same step.
The lecture ends this part with a habit worth copying. Freeze an upper-contact case. Before running, predict three signs: closing or separating, impulse zero or positive, clockwise or counter-clockwise. Then read the trace in causal order. The course’s teaching trace for that case reads: contact at (384, 195), normal (+1, 0), vn = −0.82 m/s (closing), J = (+0.91, +0.06) N·s, Δv = (+0.91, +0.06) m/s and Δω = −0.37 rad/s (clockwise). The lecture is careful to say this is a trace around Pymunk’s outputs, not its internal source code.
Read it as a chain: geometry locates the contact, vn confirms closing, J prevents overlap, and Δv and Δω update the state. The same chain runs in the device above.
Chapter 8
Carry the same pipeline to three dimensions, joints and motor torque
Picture a robot arm in a lab, pushing a block across a table. Strip the picture down to what matters. A tool moves. The tool touches an object. The object moves in response. That is PushT, seen from above.
So is PushT just a toy? The lecture says it is a 2D view of a 3D task, and it is careful about what carries over and what does not. Still present: action, then tool motion, then contact, then object response. Added in 3D: height, full orientation, links, joints and actuators. The pusher is the robot’s tool, and the T is the object.
A flat object has a pose of three numbers: x, y and one angle. A 3D object needs more. Its position is p = (x, y, z). Its orientation says which way its own axes point in the world. Orientation is stored as a rotation matrix R or as a quaternion, a four-number code for a rotation. The lecture only names these two forms and derives neither, so we will not either. All you need now: a 3D pose is a position plus an orientation, and together they are the object’s configuration.
Motion gets richer too. Linear velocity v = (vx, vy, vz) is the direction and speed of sliding. Angular velocity ω = (ωx, ωy, ωz) holds an axis of rotation and a rotation rate: how the orientation is changing now. So a free 3D rigid body has a state of configuration (p, R) and motion (v, ω). Same idea as the T’s (x, y, θ) and (vx, vy, ω), with more room.
A robot arm is not one rigid body. It is several, chained together. A link is one rigid segment. A joint is a rule for relative motion between two links, such as rotate or slide. An articulated robot is links connected by joints, and the tool moves with the whole chain.
Two joint kinds are enough for now. A revolute joint rotates, and its value qi is an angle in radians or degrees. A prismatic joint slides, and its value qi is a distance in meters. Collect them all: q = (q1, …, qn) is the robot’s current joint configuration, and q̇ says how fast each value is changing. The lecture points out that PushT already used the idea: the T’s configuration (x, y, θ) and its motion (vx, vy, ω) are joint coordinates of a free body, with θ as the angle.
The robot’s state lives in joint angles, but the task lives at the tool. The step from one to the other is forward kinematics, written tool pose = FK(q): pure geometry from joints to tool. The lecture asks for recognition only, with no derivation. For a flat arm with two links it is short enough to do by hand.
Take links of 140 px each and a base at (40, 120). Set the shoulder angle q1 = 30° and the elbow q2 = 40° (angles measured down from the right, as on a screen). The tool sits at x = 40 + 140 cos 30° + 140 cos 70° = 40 + 121.2 + 47.9 = 209.1, and y = 120 + 140 sin 30° + 140 sin 70° = 120 + 70.0 + 131.6 = 321.6. The second link turns by the sum of both angles, because it rides on the first.
The arm’s tool is the pusher. Pick what you control: the joint angles, or the tool position. Push the T with the tool and see how differently the two interfaces feel.
A flat two-link arm on this page’s PushT engine: links of 140 px, base at (40, 120), angles in degrees measured down from the right, the tool is a kinematic pusher with the same contact rules as before. Link lengths and the base are illustrative. The inverse geometry in tool mode picks the elbow-bent solution.
In joint mode the tool follows an arc. A straight push needs both sliders to move together. In tool mode the push is easy to ask for, but the joint angles do odd things behind the scenes. Neither interface is wrong. The lecture’s point is that the action interface chooses what kind of target the policy specifies, and the controller below it does the rest.
A real robot cannot teleport its tool. Slide 50 draws the chain. The policy gives either a joint target q* or a tool target x*tool. A controller compares that target with the measured robot state (the joint state q, q̇, and the tool pose from FK). Its output is a motor torque τ, the physical effort at the joints. The torque plus contact makes the robot move. A common choice is joint PD control, the Chapter 4 controller applied at every joint.
Note the contrast the lecture draws. In PushT the controller output is an acceleration command for a kinematic pusher. In a dynamic robot it is a motor torque, and the dynamics do the rest.
Moving from 2D to 3D keeps one contact normal per contact. But sliding now has two independent directions, t1 and t2, which together span the tangent plane. Tool, object, table, links and joints all interact. Orientation and angular velocity are fully 3D. It is the same simulation logic with a richer state and contact problem.
The lecture names three engines. Pymunk is for 2D rigid bodies and a good fit for PushT. MuJoCo is for articulated 3D robot simulation. Genesis is another articulated 3D engine.
Chapter 9
Chain policy and environment into trajectories you can repeat, branch and run in parallel
A student pilot crashes a plane in a flight simulator. She climbs out, laughs, and starts the flight again from the runway. She does this a hundred times a day, and nothing breaks. A real plane would not allow it.
That is the second half of the lecture’s argument for simulators. The first half was how a simulator works. This half is what we do with one: we let a decision maker act in it, over and over, and collect what happens.
Slide 52 separates two functions that people often blur. The policy asks “what should I do now?” It reads the current observation and selects an action: at drawn from πφ(· | ot). The simulator asks “what happens if I do it?” It advances the physical state: st+1 = Fsim(st, at). The environment then returns the resulting observation and reward.
Read the two lines together: the policy decides, the simulator responds, the policy observes. One map takes an observation to an action. The other takes a state and an action to a next state. Mixing them up is a common source of confusion later in the course, so fix them now.
Chain the two maps and you get a rollout: a decision, a consequence, a decision, and so on. The policy observes o0 and chooses a0. The call env.step(a0) returns o1 and r0. The policy chooses a1. Repeat until the environment terminates or a chosen horizon is reached. The whole record is one trajectory: τ = (o0, a0, r0, o1, a1, r1, …). The sum of its rewards is its return.
The lecture lists four things a simulator gives that the real world rarely does (slide 54). Reset: start a new rollout from a controlled initial condition, with reset(seed=7). Branch: return to the same state and compare different actions, which gives two futures from one state. Repeat: collect many episodes across starts, actions and policy versions. Parallelize: advance many independent simulator instances at once, env 1 to env N. Together they turn policy interaction into a controllable data-generation process. The same experiment can be repeated, varied and scaled.
Try all four in the device. Two policies play PushT. The rushed policy always pushes right at a fixed rate. The aimed policy first lines the pusher up with the T’s center height, then pushes. Pick a seed, run one rollout, then compare all six seeds.
Choose a policy and a seed, then press Run rollout. After a run, Branch at action 4 returns to the saved state and tries a different action there, drawing the original ending as a faint T. Compare 6 seeds fills the table below.
A toy task on this page’s engine. The seed sets the pusher’s starting height. Reward per action is minus the distance from the T to the goal at (400, 256) divided by 100. An episode terminates on success (T within 20 px and 15 degrees of the goal) or failure (T leaves the 512 by 512 arena, our size), and is cut off at 40 actions. Both policies are hand-written for this page.
Look at what the table does. Same simulator, same six seeds, two policies. The aimed policy finishes all six with returns near −8. The rushed policy succeeds on one seed and fails on the rest, mostly because its push twists the T. This table is the shape of every policy comparison in the course: the numbers come from rollouts, and the seeds make them fair, because both policies face the same six starts.
Count the cost. The aimed policy’s six rollouts hold 60 actions in all, which is 6.0 seconds of simulated robot time. Your browser ran them in far less than a second. A real robot needs 6 seconds of its own time, plus the time to put the T back after each trial. And a real robot cannot rewind to action 4 and try another action. That is the practical value of reset and branch. It is also why simulators are the first stop for any policy that needs many trials.
Slide 55 gives the lecture’s code for this comparison. It is short enough to remember.
pythondef compare_runs(policy, seeds): returns = [] for seed in seeds: obs, _ = env.reset(seed=seed) total = 0.0 for _ in range(HORIZON): action = policy(obs) obs, reward, terminated, truncated, _ = env.step(action) total += reward if terminated or truncated: break returns.append(total) return returns
Every line has a job. reset(seed=seed) makes the start repeatable. The inner loop is the rollout. The reward is added to a running total. The break is the Gymnasium rule from Chapter 2: stop when terminated or truncated. The function returns one return per seed.
The lecture then names three stages of learning from simulated experience. The simulator supplies experience: repeated transitions produce observations, actions, rewards and next observations. The reward supplies preference: returns tell us which simulated outcomes better satisfy the task. Learning updates the policy: an algorithm changes the policy parameters so that future actions get better returns. A later lecture builds the algorithm. Today the point is where the data comes from.
There is a quiet, important sentence on the slide: the simulator is already the transition model, so policy learning need not fit an additional learned world model. The simulator plays that role. It also has a catch, and the slide says it plainly: if the target is reality, simulation mismatch matters. That is the sim-to-real gap, and it is the next chapter.
reset(seed=seed) let you do?Chapter 10
Find where a simulator and reality drift apart, and what closes the gap
You train a driving policy in a simulator. In the simulator it never crashes. You put it on a real car, and the car drifts toward the curb. Nothing is wrong with the policy. Nothing is wrong with the simulator either, by its own rules. The two disagree about the world.
The lecture ends its main argument here. A simulator is wonderful, and it is still a model. This chapter lists the places where the model and the world part ways, and what engineers do about each one.
Slide 56 puts the idea in one sentence: a physics simulator is an oracle for its specified world. An oracle gives the exact answer for the rules it was given. The sim-to-real gap is the difference between that specified world and the physical target. Give the simulator and the real robot the same visible start and the same pusher command. The simulator follows its model exactly. Reality follows the mechanisms and timing of the real world. The two futures drift apart.
The lecture lists where the gap can enter: state sensing and estimation, parameters (mass, friction, gains, delay), missing physical mechanisms, timing and numerical contact, and coverage and computation.
Inside a simulator the engine stores its exact state: pusher position and velocity, the T’s pose and velocities, the contact cache and the chosen physical parameters. A teaching environment may expose all of it, or render an image from it. A physical robot gets noisy, partial observations: camera frames, joint encoders, force or tactile readings, timestamps and missing data. Velocity, contact state, friction and hidden geometry often have to be inferred over time.
So the useful state of a real system is built from history: observation history, then an estimated predictive or latent state, then future observations and outcomes. This is the predictive-state problem from Lecture 2, back again.
System identification means choosing the parameters whose simulated measurements best explain real experiments. The lecture’s warning is that matching one rollout does not uniquely identify the world. Several mass, friction and damping settings can give almost the same path in one experiment. A different experiment then splits them.
Here is a small case you can check with a pencil. A block slides on a floor and is pushed by a constant force F. Its acceleration is a = F/m − μg, with g = 9.81 m/s². Setting 1 is m = 1 kg and friction μ = 0.30. Setting 2 is m = 2 kg and μ = 0.045. Push with F = 5 N. Setting 1 gives a = 5 − 2.94 = 2.06 m/s². Setting 2 gives a = 2.5 − 0.44 = 2.06 m/s². The same path. After 0.5 s from rest, both blocks have slid 0.26 m. Now push with F = 10 N. Setting 1 gives a = 7.06 and setting 2 gives a = 4.56. After 0.5 s the blocks have slid 0.88 m and 0.57 m. The first experiment could not tell the settings apart. The second can. These numbers are illustrative. The lesson is real: choose informative experiments.
Parameters are numbers inside a model. The model form is the list of mechanisms the model has at all. The rigid-body PushT model has one pose, one linear and angular velocity, and an instant contact impulse. The T keeps its shape exactly. A compliant or deformable model has many local displacements and internal forces, so it can show compression and bending. It also has many more degrees of freedom, a material law, and a smaller time step for fast elastic motion.
The slide’s rule is worth memorizing: changing parameter values cannot create a mechanism that the model form lacks. If the real block bends and your model is rigid, no setting of friction fixes that. Richer physics also costs more computing.
A real loop runs on a physical clock. The camera captures an image, transport and compute and queueing take time, and only then does the actuator start. The lecture puts it as 50 to 100 ms of delay, against zero in the simulator. During the wait the T kept moving, so contact time and location shift. At a T speed of 100 px/s, an 80 ms delay is 8 px of motion before the command lands.
The lecture asks you to keep four times apart. Physics dt: one numerical integration step. Control interval: time between policy decisions. Delay: capture-to-effect latency. Wall-clock speed: how fast the computation finishes. Delay belongs to the interaction model when it changes which state receives the action.
A contact mode is the answer to three questions: which contacts exist, and whether each one is separating, sliding or sticking. In each mode the active equations are different. Move the action a tiny amount and the simulation may cross a mode boundary. There the result may jump, and a gradient (the slope a learning algorithm follows) may jump, vanish or turn unstable. Finite dt, collision tolerances and a finite number of solver iterations can also shift which mode the numerical simulator picks. Measure it yourself.
Each point on the curve is one full rollout: eight pushes of strength 0.5, with the pusher starting at the height on the horizontal axis. The vertical axis is how far the T turned. Drag the height across a jump, then change the number of physics steps per action.
Computed live from this page’s engine: 141 full rollouts per curve. A jump is any step of more than 12 degrees or 25 px between neighbouring heights. The curve is the engine’s, not the lecture’s, but the effect, a sharp change at a mode boundary that moves with dt, is the one the lecture describes.
Read the curve. At the low heights the pusher passes below the T’s corner and the T never moves: a flat zero. Then, over a gap of a single pixel, the T turns more than 30 degrees. In this page’s engine, the push that first catches the corner begins at a height of 203 px with 10 steps per action, and at 207 px with 5. A policy that learns by following slopes sees a cliff there.
A rollout may need a contact solve, many bodies, rendering and sensing. Higher fidelity can reduce throughput, and rare events and long horizons may still get little coverage. One alternative is a learned surrogate world model: a neural network trained to predict the next state. It can imagine many action-conditioned futures at once, with fast batched inference.
The speed comes from approximation. A learned surrogate can omit rare contacts, extrapolate poorly, or confidently imagine a future that neither the physics simulator nor reality would produce.
Slide 63 gives four moves and a loop. One: sensing and state estimation. Match what the real system can observe: model cameras, encoders, noise, occlusion, timestamps and the estimator. Two: calibration and variation. Use system identification for likely values. Use domain randomization, which means training across many random settings of the parameters, when the target varies or calibration is uncertain. Three: a richer physical model. Add the mechanisms that matter for the task, when tests show the simple model misses them. Four: real data and hybrid models. Mix simulated and real rollouts, and keep the trusted physics as a backbone while a learned part corrects the residual error.
These four sit in a loop: simulate or train, test on the real system, measure the mismatch, update the model and the data. The lecture adds a warning. Planners and policies can exploit errors in both physics simulators and learned world models. They find the cracks and fall in. So hold out real tests, search for failure cases, and keep the improvement loop open.
Chapter 11
Keep the numbers, the checklist, the core code and the way onward
You can now follow one call to env.step(action) from top to bottom. The lecture closes (slide 64) with that chain on one slide, and it is worth holding in your head as a single sentence. A current state and an action become a relative pusher target. A PD controller turns the target into a controlled velocity update. space.step(dt) integrates that update and detects contact. The contact changes the T’s velocity and spin. That gives the next physical state. Gymnasium then returns the observation, the reward and the episode signals.
Around that chain the lecture hangs three claims. A simulation is an executable world model: specified physics and a numerical solver answer transition queries. Policy learning uses rollouts: if the simulator is the target world, a policy can learn directly from it. And the reason for the rest of the course: the target world exceeds the simulator. Use target data, learned components and reality checks where the specification fails. The closing line is the one to keep: understand the simulator deeply enough to know both what it can generate, and what it cannot.
| What | Number | Why it matters |
|---|---|---|
| Action to target | p* = p + 100 a; (273, 195) + 100 (1, 0) = (373, 195) | The action is unitless and relative to the pusher |
| PD gains | kp = 100 s−2, kd = 20 s−1; first step 5,000 px/s² | Pull toward the target, brake against the speed |
| First integration step | dt = 0.01 s; Δv = 5,000 × 0.01 = 50 px/s | Velocity first, then position (semi-implicit Euler) |
| Two clocks | 10 Hz control = 100 ms per action; dt = 10 ms; K = 10 | One action holds ten physics steps with one fixed target |
| Solver passes | space.iterations = 10 inside one physics step | Coupled contacts need repeated passes, and no time passes |
| Center-hit impulse | m = 2 kg, vn = −2 m/s, e = 0: j = 4 N·s | The impulse cancels the closing speed and no more |
| Upper-contact trace | J = (+0.91, +0.06) N·s; Δω = −0.37 rad/s, clockwise | Geometry, vn, J, then Δv and Δω, in causal order |
| Gymnasium step | 5 values: observation, reward, terminated, truncated, info | One interface for every environment |
| Real-loop delay | 50 to 100 ms against 0 in the simulator | Delay shifts contact time and place |
| Term | Plain meaning |
|---|---|
| State | Every number the physics needs. It holds pose and motion. |
| Observation | What the agent gets to see, drawn or sensed from the state. |
| Kinematic body | A body that follows its commanded motion. Contact does not change it. |
| Dynamic body | A body that moves under forces, impulses and contact. |
| Impulse | The contact force added up over a very short time, applied at one instant. |
| PD controller | Pull toward the target, brake against the speed. |
| Semi-implicit Euler | Update the velocity first, then move the position with the new velocity. |
| Broad phase, narrow phase | A cheap box test for possible pairs, then an exact test for those pairs. |
| Closing, separating | The sign of the normal speed vn: negative closes, non-negative separates. |
| Rollout, return | One run from reset to the end, and the sum of its rewards. |
| Sim-to-real gap | How far the specified simulated world is from the physical one. |
| Model form | The mechanisms a model contains. Parameters cannot add a missing one. |
This sketch is assembled from the lecture’s own snippets (slides 11, 30 and 31). The names pusher, space and observe are the course’s. Every line is one box in the chain above.
pythonclass PushTEnv(gym.Env): def step(self, action): target = pusher.position + 100 * Vec2d(*action) # Ch 3: p* = p + 100 a, fixed for this action dt = 0.10 / 10 # Ch 5: 100 ms action, ten 10 ms substeps for step in range(10): error = target - pusher.position # Ch 4: feedback uses the latest state a_cmd = kp * error - kd * pusher.velocity # Ch 4: PD command (kp=100, kd=20) pusher.velocity += a_cmd * dt # Ch 5: velocity first space.step(dt) # Ch 6, 7: positions, contacts, impulses obs = observe(self.state) # Ch 0: pixels drawn from the state return obs, reward(self.state), terminated, truncated, {} # Ch 2: the five returns def compare_runs(policy, seeds): # Ch 9: rollouts, one return per seed returns = [] for seed in seeds: obs, _ = env.reset(seed=seed); total = 0.0 for _ in range(HORIZON): obs, reward, terminated, truncated, _ = env.step(policy(obs)); total += reward if terminated or truncated: break returns.append(total) return returns
Lecture 2 asked which futures could follow a history. This lecture built the machine that answers for one specified world. The next lecture, State-Space Models, turns to how a learner keeps a belief about a state it cannot see, which is the problem of Chapter 10’s sensing section.
On this site, three lessons go deeper on neighbours of today’s ideas. Task Environments covers the reset, step and reward contract at scale, for agents that train on tasks. microMDP builds the formal decision problem that a policy solves in an environment. Agent Environments shows the same interface on a computer screen instead of a table.
The reading is the Gymnasium basic usage guide. The lecture slides are on the course site.
About the course. CIS 6280 meets Tuesdays and Thursdays 12:00 to 1:29 PM at the University of Pennsylvania, in AGH 105A and 105B. Lecture 3 was given on September 1, 2026. The slides show Assignment 1 releasing Monday, September 7, and the course site lists it as due September 24. Every idea in this lesson is re-taught from Jiatao Gu’s lecture; the figures and devices are ours.
Now press Teach and explain, from memory, what happens inside one env.step: the target, the ten feedback steps, the contact, the impulse and the new state. Then go back to the pushing hero and predict where each hit will send the T before you run it.