A representation is only as good as what a simple readout can recover from it. Rewrite the same points in polar coordinates, and a line that scored 50% scores 100%.
Learn what a world model should keep from what it sees, how to test that with a probe, and how reconstruction, masking, next-patch prediction and contrastive matching each teach an encoder.
Switch the coordinates, then press Fit the readout. Blur the rings and see how far the new form carries the same line. Then we build, piece by piece, everything underneath: probes, pretext tasks, masked autoencoders, and the contrastive loss.
You need a feel for a vector (a short list of numbers) and a little Python for the code. We build the rest from zero.
Based on Lecture 5 of Penn CIS 6280: World Models (Fall 2026), taught by Jiatao Gu, and its reading, Lilian Weng’s Contrastive Representation Learning. Lecture slides. Every idea is re-taught here in our own words and devices; none of the course’s slide images appear on this page.
Toy data, seeded so it is the same on every visit: 60 points on an inner ring (radius 0.5) and 60 on an outer ring (radius 1.0). The readout is logistic regression, 400 steps of gradient descent on standardized features. This is the lecture’s coordinate example (its Cartesian and polar picture), rebuilt as a live fit; the accuracies are measured here, not copied.
Chapter 0
See why the form you give a scene decides which questions are easy to answer about it
A robot arm pushes a flat T-shaped block across a table. A camera watches from above. Before each push, you may write down a few facts about the scene. After the push, the block turns one way or the other. Which facts must you write down to call the turn right?
Your first idea is a name: “a pusher and a T-shaped block.” That sentence is true, and it is useless. It reads the same before every push. It cannot tell you which way the block will turn.
Better facts are about where things are. How high is the pusher? How high is the block? How is the block turned? The device below tests this. It makes 3,000 test pushes, then predicts each turn from only the facts you keep.
Left: one push. The pusher moves right and hits the block. The dot marks the block’s balance point. Right: how often your kept facts call the turn right. Switch facts on and off. Press Show another push to see a new scene.
Illustrative toy, not the course’s PushT simulator. The pusher always comes from the left. A push turns the block counterclockwise if it hits below the balance point, clockwise if above. The balance point sits 0.35 units from the block’s middle along its stem. Each fact is cut into 8 bins; the guess for each bin is the majority outcome of 6,000 training pushes, scored on 3,000 fresh ones.
With the name alone you are right 50% of the time, a coin flip. Add the pusher’s height and you reach about 73%. Add the block’s height and you reach about 89%. Only when you also record the angle does the score climb to about 96%. Each fact you drop costs you part of the answer.
Now give the idea its name. The camera gives an observation: the raw thing the sensor reports, here a grid of colored pixels. A representation is a description of that observation, written in a form we choose. The lecture writes it as a function at work:
The lecture lists three common forms. Each one keeps different facts and costs a different amount of space.
| Form | What it is | Size in our toy |
|---|---|---|
| Pixels | An array of red, green and blue values | A 96 × 96 frame is 96 × 96 × 3 = 27,648 numbers |
| Measured coordinates | The pusher’s position and the block’s pose (where it is and how it is turned) | 5 numbers: pusher x and y, block x and y, angle |
| Learned features | Numbers an encoder produces after training | Your choice, for example 256 numbers |
The frame size and the feature size are our own round numbers, to show the shapes. The lecture’s point is the last line of its slide: learning the representation changes which relationships are easy to compute.
Go back to the instrument at the top of the page. It holds two rings of points. The inner ring is one class and the outer ring is another. The class is the same in both coordinate systems. Only the way we write each point down changes.
In Cartesian form, each point is (x, y): how far right and how far up. No straight line separates two rings that share a center. Any line leaves about half of each ring on the wrong side. In polar form, each point is (r, θ): its distance from the center and its angle. Now the inner ring has r near 0.5 and the outer ring has r near 1.0. One threshold on r separates them.
The lecture states this plainly: the class information is unchanged, and a threshold on the radius becomes sufficient. Nothing was added to the data. We only moved it to a form where the readout we can afford, a straight line, is enough.
Lecture 4 introduced the state of a world model: the variable that the model evolves forward in time. The lecture draws the two ideas side by side, so the difference matters.
A representation may use history. The lecture writes it as zt = Eφ(ot; o<t, a<t): the encoder sees the current frame, the earlier frames o<t and the earlier actions a<t. Nothing forces it to be Markov, which means that the present summarizes the past completely. A state, in Lecture 4’s model, is assumed to be Markov: given the current state and action, the earlier states do not change the next-state distribution.
p(st+1 | s0:t, a0:t) = p(st+1 | st, at)
Think of a diary and a snapshot. A representation is allowed to be a diary: it can hold what happened before. A Markov state is a snapshot that already contains everything the diary would add. The slide’s warning follows: using temporal context does not automatically make a representation a Markov state.
This is Lecture 5 of CIS 6280, World Models, at the University of Pennsylvania, taught by Jiatao Gu on September 8, 2026. This week has two parts. Today covers data targets and view matching. Thursday covers teachers, feature targets and distribution constraints. Slides and readings live on the course site. The reading for today is Lilian Weng’s post on contrastive representation learning, and we weave it into Chapters 8 and 9.
Chapter 1
Freeze an encoder, fit a tiny readout on top, and let the score tell you how easy the information is to reach
A friend hands you a sealed box. Inside is an encoder that someone trained on thousands of pictures of the pushed block. You cannot open the box. You want to know one thing: does it know how the block is turned?
You can ask the box for its description of any picture. That description is a list of numbers. Your plan is simple. Fit a tiny gauge on top of those numbers, and see whether the gauge can read the angle. If the gauge reads it well, the angle is easy to reach. This gauge is called a probe.
The word frozen matters. A frozen encoder keeps its weights fixed, so training the probe cannot change the description. If the probe scores well, the encoder deserves the credit. If you let the encoder learn too, you would be testing your training run, not the box.
The lecture’s example reads the object’s orientation. The probe is the simplest model there is, a linear probe: multiply the feature vector by a matrix and add a vector.
Training has two steps. First, fit: train only W and b on labeled images, and minimize the squared error between the answer and the known orientation. Second, test: predict the angle in scenes the probe has never seen, and report the angular error. If the task is a category instead of an angle, the probe predicts one score per class and trains with cross-entropy, a loss that is small when the right class gets a high score.
Here is one probe answer by hand. Say an image gives the feature e = (0.8, 0.6, 1.0), and the fitted W is [[1, 0, 0], [0, 1, 0]] with b = (0, 0). Then ŷ = (0.8, 0.6). The angle of that point is atan2(0.6, 0.8) = 36.9°. If the true angle is 40°, the angular error is 3.1°. Average that over many unseen images and you have the probe’s score.
The lecture gives two rules. First, compare encoders fairly: use the same labels, the same way of extracting features, the same probe type and the same test scenes. Change one thing at a time. Second, read the answer with humility. A successful linear probe shows that the information is easy to access. A failed probe does not prove the information is missing. A stronger probe might still find it.
What does the encoder hand back? There are two common answers. A global representation is one vector for the whole image, such as (0.2, −0.7, …). A local representation is one vector for each image region, a grid e1, e2, e3 over e9 for a 3 × 3 split. Local features keep their spatial arrangement. They can also use whole-image context, because each region’s vector was computed while the model looked at the other regions. A vision transformer makes exactly this kind of output: one feature per patch.
As shapes, a local representation is a table of K × d numbers: K patches, each with d numbers. A global one is a single 1 × d row. Our example size is K = 196 patches of d = 768 numbers; that is 150,528 numbers instead of 768.
Both probes freeze the encoder and start from the same patch features e1 to eK. They differ in how they squeeze K vectors into one. A linear probe uses a fixed pooling, usually the mean, then a linear head. Every patch gets weight 1/K, whatever it shows. An attentive probe learns how to weigh the patches. A learned query (a vector the probe trains, which asks “does this patch matter?”) scores each patch. A softmax turns those scores into positive weights αk that sum to 1. The weights depend on the input.
Why does the choice matter? Picture the block filling one patch out of 64. The mean gives that patch weight 1/64. Its signal is nearly drowned by the 63 patches that show only table. The attentive probe can give the block patch most of the weight. Try it below.
Top: K patches. One holds the block, with its direction drawn. Below: how much weight each probe gives each patch. Bottom: the average angle error of each probe on 300 unseen images. Drag K up, then raise the noise.
Illustrative toy with seeded data. Each patch feature is 3 numbers: the block’s (cos, sin) and a presence flag, plus Gaussian noise; empty patches hold only noise. Both heads are fitted by ridge regression on 400 images. The attentive probe’s query is fixed to the presence flag (a trained query would learn this); its weights are softmax(4 × presence). Not a measured result from the course.
At K = 4 the two probes are fairly close, 18° against 10° with the default noise. At K = 16 the mean-pooling probe misses by 40° while the attentive probe stays near 9°. The signal did not vanish. Mean pooling spread it thin and added the noise of every empty patch. The lecture’s summary says it well: attentive probing learns a richer readout, so you must report the probe type when you compare scores.
Chapter 2
Decide which details to keep and which to ignore, and see why the answer depends on the question you ask
The robot pushes the T-block from the left. Where should the pusher touch it? If the pusher hits the stem, the block slides one way. If it hits the crossbar, the same push turns the block another way. The scene looks almost identical in both cases. The result does not.
So a world model that wants to predict the push must keep some details and may throw others away. This chapter asks which ones. The answer is never “everything” and never “the name.” It depends on what you will do with the features next.
Here is where representations sit in a world model. The encoder reads the observations o0 to ot and writes features z0 to zt. A predictor, another network Fθ, reads those features and the actions the robot took, and guesses the next observation.
The lecture splits the work in time. This week we learn the encoder and test its features. Later, we learn the predictor from sequences. So everything in this lesson asks one question: when a predictor reads these features later, will the facts it needs be there?
The lecture shows one concrete choice from robotics. Instead of one big vector, represent an image as a set of keypoints: K learned visual points, each with a position, written as e = (x1, y1, …, xK, yK). That is 2K numbers. For K = 16, that is 32 numbers instead of 27,648 pixels. The idea comes from the Deep Spatial Autoencoders paper by Finn and colleagues (ICRA 2016).
The reason is the push. Same T, same push to the right, but different contact point: the stem in one case, the crossbar in the other. Different contact can lead to different motion. So for prediction, preserve the positions and orientations that affect contact.
A nuisance variation is a change in the image that does not matter for your goal. Lighting is the usual example. But the lecture’s slide makes a sharper point with a three-row table. Lighting is not a nuisance in itself. It is a nuisance for some questions.
| Downstream question | Effect of a lighting change | What you want |
|---|---|---|
| Where is the object relative to the pusher? | Same geometry, different illumination | Keep geometry easy to read |
| What will the next RGB frame look like? | Brightness and shadows change the pixels | Retain or model appearance |
| What object is present? | Many lighting changes keep the identity | Stable category information |
Read the middle row twice. If your world model must draw the next frame, it must know the brightness. A code that threw lighting away cannot draw it. So “nuisance” names a relationship between a variation and a task. Try it with a toy.
Each scene has a geometry value g (where the object is), a lighting value b and a shape cue. Choose which of the three your code keeps. Each bar scores one question from the table, using only the kept facts. Slide the lighting to see one frame change.
Illustrative toy, not measured on real images. 3,000 scenes with g and b drawn uniformly from −1 to 1 and a shape cue equal to the object class (−1 or +1) plus noise. The next pixel is 0.5 + 0.3 g + 0.5 b. Scores are the share of variance a least-squares fit explains (questions 1 and 2) or the accuracy of a threshold on the fit (question 3).
Start with geometry only. Question 1 scores 100%: the code kept exactly what it needs. Question 2 scores about 28%: the code cannot draw the brightness. Question 3 sits near 50%, a coin flip, because shape was dropped too. Turn lighting on and Question 2 rises to 100%, while Questions 1 and 3 do not change. Turn the shape cue on and Question 3 reaches about 95%. No code wins every row until it keeps all three facts, and a real code has a limited size.
Two words describe how an encoder should react when the input changes by a transformation T, such as a rotation. An invariant encoder ignores T: E(To) ≈ E(o). A rotated cup is still a cup, so the features do not move. An equivariant encoder tracks T in a predictable way: E(To) ≈ ρ(T) E(o), where ρ(T) is a known matrix that says how the features must change.
Take a worked case. The block’s angle is stored as the point e = (1, 0), which means 0°. Rotate the block by 90°. An equivariant code moves to (0, 1): the point turned by the same 90°, so the angle is still readable. An invariant code stays at (1, 0): the angle is gone, and a probe for angle can only guess. For “which object is this,” the invariant code is ideal. For “how is it turned,” you need the equivariant one.
A world model needs both promises, each in the right place. It wants identity to survive a push. It wants pose to change in a way the predictor can follow. That is why the next chapters keep asking, for each training method, what it makes the features keep.
Here is a failure you will meet. The symptom: a world model draws crisp T shapes but gets the contact point wrong, so its predicted pushes turn the wrong way. The measurement: probe the code for the contact point, and compare the probe error with a probe for the shape. If the shape probe scores high and the contact probe scores low, the code kept what the block is and dropped where it touches. The fix: keep the coordinates that decide contact, for example keypoints, or add a training signal that needs them.
Chapter 3
Follow the training signal from labels to rewards to raw data, and meet the idea of a target the data makes for itself
Picture a network trained to do one plain job: look at a photo and name its category. Nobody tells it to build a good description of the photo. It does so anyway, because the last layers cannot name the category unless the layers before them have organized the picture well.
That is the first surprise of this chapter. Features are often a side effect of solving some other task. The lecture shows four examples, and each one names its training signal: the thing the model is scored against.
Class labels. In AlexNet (2012), the training signal is a human category label for each image. After training, the hidden features of two images are close when the images are related, so nearby features retrieve related pictures. Training updated both the output head and the layers that produce its features.
Next-word prediction. A native multimodal model such as Kimi K3 (2026) learns its visual features from scratch. The image goes through a vision encoder, called MoonViT-V2 in the lecture. The visual features and the text then predict the next text token. One loss trains the vision encoder, the projector that connects it to the language model, and the language backbone together.
Game outcomes. The original AlphaGo learns features from the board. Its training signal starts with expert moves, then adds self-play and game results. It combines supervised learning and reinforcement learning. The features serve a policy (which move to pick) and a value (who is winning).
The next observation. A world model reads a history of observations and actions, builds features, and predicts the future. Its training signal is what is actually seen next. If the encoder is trained jointly with the predictor, the prediction error also updates the encoder.
The lecture sums it up in one line: representation learning happens in supervised learning, in reinforcement learning, and in predictive modeling. Useful features can be learned while solving another task.
A world model needs more than object names. What matters changes over time: pose, motion, contact, occlusion (one thing hiding another) and the results of actions. To label all of that by hand, in every frame, costs too much. Rewards can be sparse, with a signal only once in a long task, or tied to a single task.
But experience already holds relationships. Neighboring moments look alike. Different views show the same thing. Actions have outcomes. The lecture’s question is whether a model can learn from those relationships without anyone labeling each moment. It adds a caution: human learning also uses language, instruction and social feedback, so the picture of a child at play motivates the question and does not name one learning algorithm. This week, then, we learn reusable representations from data without manual labels for every training target.
Self-supervised learning (SSL) is the name for this. The recipe has three beats. Change the data in a way you control. Ask the network to recover what you did, or what was hidden. Score it against the answer you already have, because you made the change.
The lecture’s first example is a rotation. Take an image o. Apply a rotation T90 that turns it 90 degrees. Send the turned image through the encoder Eφ, then through a small rotation head qψ that outputs a score for each possible angle. The sampler knows the answer, 90°, because it chose the angle.
After pretraining, throw the rotation head away. Keep the encoder, and fit a new readout for the angle or the geometry you really want. The rotation task was only a way to make the encoder learn. The lecture later names such a task a pretext task.
The lecture organizes the field into three families. They differ in what supplies the training answer. Use the same four questions on each one: what is the visible input, what is the target, what is the loss, and which features do we reuse afterward?
| Family | What supplies the training answer? | Examples |
|---|---|---|
| Reconstruction | Recover the original content or structure from partial or altered data | Rotation, Jigsaw, DAE, MAE, AIM |
| Contrastive learning | Which other view belongs with this example? | SimCLR |
| Match learned targets | A teacher feature, an assignment, or another view’s embedding | DINO, data2vec, JEPA |
Today covers the first two. The third is Thursday’s topic. The device below runs the four questions on five different training methods, so you can feel how the same encoder gets very different homework.
Pick a training method. A pulse walks from the input to the reused features, one stage at a time. Watch who or what provides the target.
Sketches of the methods in the lecture’s map, simplified: RotNet (4 angles), masked autoencoders (the lecture’s 4 of 16 patch example) and SimCLR (three images, six views). Counts come from those examples; the stage text is our own summary.
Notice the third box in each row. In the first row, a person fills it. In every other row, the data does. That is the whole trick, and the reason the chapters ahead can use as much unlabeled video and imagery as exists.
Chapter 4
Build a puzzle whose answer the data knows, and watch an easy shortcut spoil what the network learns
You want a network to understand pictures, and you have a million photos with no labels. Here is a trick. Cut each photo into pieces, shuffle them, and ask the network to put them back. Nobody has to label anything. You made the shuffle, so you know the answer.
Such a made-up job is called a pretext task: a training task built only so that the network learns features worth reusing. The puzzle itself is not the goal. This chapter walks through the classic ones, and then shows how a puzzle can fail.
The lecture groups these under reconstruction: alter or hide some information, then predict the original content or its structure. It shows three pairs of alteration and target.
The first two recover structure with an angle or layout label. The third recovers data, meaning pixels or tokens. In all three the picture is: altered input, then encoder, then features, then a prediction of the original information.
Doersch and colleagues (2015) take two patches from one image: a reference patch and a query patch. A shared encoder reads both. A pair classifier then chooses among eight possible offsets: the query was above, below, left, right or at one of the four corners of the reference. The sampler records the offset, so it knows the answer.
A guess by luck is right 1 time in 8, which is 12.5%. To do better, the network must recognize parts and their arrangement, such as that a window sits above a door. The lecture then asks a sharper question. What did this task teach? It runs a new task, retrieval: take a query patch and fetch its nearest neighbors in feature space. Random features fetch random textures. Features trained for position fetch patches of the same kind of thing. A task built for one purpose revealed structure beyond it.
RotNet (Gidaris and colleagues, 2018) generates the target itself: turn the image by 0°, 90°, 180° or 270°, and ask which one was used. A lucky guess is right 25% of the time. To do better, the encoder must keep cues that tell the turns apart: the sky is up, a face has eyes above a mouth, a tree has its roots below.
The lecture compares where features respond in two networks, one trained on object recognition and one trained on rotation prediction. The maps suggest object structure in both. The lecture is careful: a map only suggests. A separate probe must test the variable you actually need. It adds a distinction that matters later: predicting a rotation and matching rotated views make different demands on the encoder.
Not every puzzle teaches. The lecture’s thought experiment uses a shape, either a T or an L, and asks the network to restore it. Look at three versions.
Here is why useful features can emerge from the middle case. When structure helps predict the hidden content, lowering the error rewards features that encode that structure. The design rule: remove the copying shortcut, and keep enough evidence that inference remains useful.
Noroozi and Favaro (2016) turn spatial layout into a target. Cut an image into tiles, apply a permutation, and ask which permutation was applied. The task should reward understanding how regions relate. But tiles that touch can expose a shortcut. If two tile edges continue the same line of color, you can reassemble them by matching pixels along the cut, with no idea what the picture shows.
Try this yourself. The device below builds smooth random pictures, cuts each into 2 × 2 tiles with a gap of g pixels between them, and shuffles the tiles. The solver is dumb on purpose. It tries all 24 orderings and picks the one whose touching edges match best. It looks only at edge pixels.
Left: a picture cut into four tiles. The pink bands are pixels thrown away between tiles. Right: the solver’s score as the gap grows, over 150 pictures. Drag the gap from 0 to 8.
Toy: smooth random pictures of 64 × 64 pixels, tiles of 28 × 28, the solver compares only the pixels along each shared edge. This is our own illustration of the shortcut; the accuracy numbers are measured here and are not the paper’s.
With no gap, the solver restores the layout almost every time, using nothing but edge pixels. At a gap of 4 it falls to about half (56%). At 8 it drops to 9%, not far above the 4% of a random guess among 24 orderings. A network trained on the no-gap puzzle can win the same way, by reading edges.
The paper ran a controlled version of this on real networks. With normalization and color jitter fixed, it compared two settings. With no gap between tiles, jigsaw accuracy was 98% and detection on the VOC benchmark scored 47.7 mAP. With a gap, jigsaw accuracy fell to 88% but detection rose to 52.6 mAP. Here mAP is mean average precision, a score for how well a detector finds and names objects; higher is better. Removing an easy cue hurt the puzzle score and improved the detection result. A better score on the pretext task does not mean better features.
| Task | Constructed target | Possible useful evidence |
|---|---|---|
| Relative position | Neighbor class | Part layout and scene geometry |
| Rotation | Applied angle | Object orientation and upright structure |
| Jigsaw | Patch permutation | Spatial arrangement |
| Colorization | Missing color channels | Object and material context |
| Inpainting | Missing image region | Shape and contextual completion |
The last two rows are the next chapter. Notice the habit this table builds: for any pretext task, name its target, and then ask what evidence the network would need to hit that target.
Chapter 5
Fill in colors, holes and noisy values, and see why the loss you pick decides what kind of guess you get
Hand a child a black-and-white photo of a toy car and a box of crayons. What color is the car? She might pick red. She might pick blue. The photo does not say. It only says “car”, and cars come in many colors.
Now ask a network to do this job, millions of times, and use the job to teach it what cars, grass and skies look like. This is the idea of predicting missing data: remove part of the input, and train the network to put it back. The next chapters use it at scale. First we need to see what the choice of loss does to the answer.
Zhang and colleagues (2016) trained a network to color gray photos. The training pairs cost nothing: take any color photo, drop its color channels to get the input, and keep the original colors as the target. The lecture shows good results and then a failure. A gray object can honestly be several colors. The same gray evidence can support several plausible colors.
To learn the object and its material, the network must use context: grass is probably green, sky is probably blue, a banana is probably yellow. That is the pretext task’s gift. The lecture lists the likely evidence as object and material context.
Pathak and colleagues (2016) cut a hole out of the middle of a photo and ask a context encoder to fill it. The missing center must be inferred from the surrounding evidence: the shape of the car edge on both sides, the lines of the street. The lecture shows two outputs of the same system, one trained with a squared-error (L2) loss and one trained with L2 plus an adversarial loss, a second network that tries to tell fills from real photos. Its comment is brief and important: the loss changes which errors the reconstruction is encouraged to avoid.
Why would the loss matter? Think of the car again, with two possible colors. The device below makes the problem concrete.
The true car is orange or blue. Slide the chance that it is blue. Each row is one way to guess the color: the average (what a squared-error loss learns), the likelier color, or a random draw from the chances.
Toy with one number for color: 0 is orange and 1 is blue; the car is blue with the chance you set. Expected squared error is q(1−q) for the average, min(q, 1−q) for the pick, and 2q(1−q) for a draw. A guess “is a real color” when it lies within 0.15 of one of the two true colors.
At 50% the squared-error guess is a muddy color halfway between orange and blue. It has the lowest average error (0.25 against 0.50 for the others). Yet it matches no real car. Squared error is minimized by the mean of the possible answers, so it prefers a blend that is never correct over a pick that is sometimes wrong. That is why L2 fills tend to look blurry. The adversarial loss punishes a fill that looks fake, so it pushes toward one sharp, real-looking answer.
The oldest recipe is the denoising autoencoder (DAE) of Vincent and colleagues (2008). Corrupt the input x by removing some values, to get x̃. Encode it, y = fφ(x̃), to get features. Decode them, z = gθ(y), to predict the clean input. Compare against the clean x, not the corrupted one.
Here is a shape-level trace. Take an image x of 28 × 28 = 784 values. Zero out about a third of them at random to get x̃, also 784 values. The encoder maps 784 numbers to, say, 128 features. The decoder maps 128 back to 784. The loss compares those 784 outputs with the clean x. Copying the corrupted values is not enough, because a third of them are wrong. The network must lean on dependencies in the data: neighboring pixels of a stroke agree, so a missing pixel can be guessed from its neighbors.
pythondef dae_step(x, enc, dec, p=0.33): keep = (rand_like(x) > p) # 1 where we keep the value x_tilde = x * keep # corrupt: some values removed z = dec(enc(x_tilde)) # features, then a guess at the clean input loss = ((z - x) ** 2).mean() # compare with the CLEAN x return loss
One more check from the lecture. A world model can decode its features back into a picture, which shows what the code kept. The lecture’s example, from Ha and Schmidhuber (2018), puts an observed frame next to a reconstruction made through the visual code. The two look alike, but fine details are gone. Its challenge is practical: pick one task-relevant detail, then propose a measurement beyond visual similarity. For the pushed block, that detail might be the contact point. A measurement could be a probe that reads it from the code, as in Chapter 1.
Here is the failure in the field. The symptom: colorized photos look washed out, with brown grass and gray skies. The measurement: compare each predicted color with the colors that really occur for that object; if the distance is large for every real color, the model has predicted a blend. The fix: change the loss so a sharp answer wins, for example an adversarial term, or let the network predict a distribution over colors and then draw from it.
Chapter 6
Hide most of the image, encode only what remains, and score the guess on the hidden part
Imagine a jigsaw puzzle with three quarters of the pieces missing. You can still tell what the picture is: a clock face, a pair of animals, a street. Your eyes use the few pieces you have to guess the whole. A network can learn to do the same, and in doing so it must learn what clocks, animals and streets look like.
This is the masked autoencoder (MAE) of He and colleagues (2022). It combines two older ideas from the last chapter: hide part of the input, and reconstruct the hidden part. Its trick is in who sees what.
First, cut the image into a grid of patches: equal square pieces. The lecture uses a grid of 16 patches (4 by 4) and shows 4 of them visible. The method then has three steps.
The lecture’s note on cost is the key design choice. Mask tokens enter the lightweight decoder after the expensive encoder. Why does that matter? A token is one patch as the encoder sees it. The encoder compares every token with every other token, a cost that grows with the square of the token count. With 4 tokens that is 4 × 4 = 16 comparisons. With all 16 it would be 16 × 16 = 256. Hiding most of the image makes the encoder 16 times cheaper in that step.
As shapes, the visible tokens are a table of 4 × d numbers going into the encoder. The decoder sees a table of 16 × d: 4 real feature rows plus 12 copies of one learned mask-token row, each with its position added. It outputs a table of 16 × p, where p is the pixel count of one patch.
pythondef random_mask(tokens, ratio=0.75, rng=np.random): n = tokens.shape[0] # 16 patches keep = int(n * (1 - ratio)) # 4 stay visible perm = rng.permutation(n) vis, hid = np.sort(perm[:keep]), np.sort(perm[keep:]) return tokens[vis], vis, hid # encoder gets only tokens[vis]
Which patches count in the loss? Only the hidden ones. The lecture works a one-number-per-patch example to make this clear. Four patches have target values 1, 2, 3, 4. The encoder sees patches 1 and 3. The decoder predicts all four values. The scored set is T = {2, 4}.
Now put numbers in. Say the decoder predicts (1.1, 1.8, 3.2, 4.5). Patches 1 and 3 are visible, so they are not scored, even though the decoder is close on them. Patch 2: (2 − 1.8)² = 0.04. Patch 4: (4 − 4.5)² = 0.25. The loss is (0.04 + 0.25) / 2 = 0.145. The visible patches supply context. The masked patches supply the scored answers. Scoring the visible ones too would let the decoder win points by copying.
The mask ratio is the fraction of patches hidden. It is a design choice you test, not a given. The lecture notes that its picture examples use 80% masking, while the main training recipe uses 75%. It then plots two curves against the ratio: one when the whole encoder is fine-tuned on the task, and one when the encoder is frozen and a linear readout is fit. It asks where the useful range is and why the two curves differ. Its lesson in one line: a strong result after fine-tuning does not imply equally accessible frozen features.
The geometry of the mask matters too. The lecture compares a scattered random mask at 75%, one missing block at 50%, and a regular grid at 75%. The same fraction of hidden pixels can leave very different spatial evidence. A block can remove a whole object. A grid leaves a hint everywhere. Try three masks on a toy picture.
Left: the picture, with pink patches hidden. Right: a simple rebuild that fills each hidden patch by blending nearby visible pixels. Pick a mask style, then change how many of the 16 patches are hidden.
Toy: a 24 × 24 picture of a smooth background and a bright T, cut into 4 × 4 patches of 6 × 6 pixels. The rebuild is a distance-weighted blend of visible pixels, not a trained decoder, so the error numbers show how much evidence a mask leaves, not what MAE scores. The 16-patch grid and the 16-versus-256 count follow the lecture’s example.
At 12 hidden, the random mask leaves four scattered patches, and the blend rebuilds the picture with an error of about 0.18. The block style leaves the visible patches bunched in one corner, so the far side has nothing near it, and the error doubles to about 0.38. The grid style spreads the four visible patches evenly and lands near 0.20. Same count, very different evidence. What counts is not how many pixels you hide, but how much evidence remains near each hidden one.
The lecture ends this part with a warning. Look at triplets of masked input, reconstruction and original: the reconstruction often looks right and still differs from the original in detail. Recovering a plausible scene and preserving the exact observed detail are different tests. For a world model, the second test is the one that decides whether the contact point survives.
Chapter 7
Swap the decoder for a task head, then meet image models that learn by guessing the next patch
Training is over. You hold an encoder that learned from millions of masked pictures. Nobody has asked it a real question yet. A product team arrives and says: “Tell me what is in each photo, where it is, and which pixels belong to it.” How do you put the encoder to work?
A downstream task is the task we want to solve after pretraining. The recipe is short: image, then pretrained encoder, then a task head (a small new network), then the task’s prediction. The reconstruction decoder is gone. In its place stands whatever head the task needs.
| Task | What does the head predict? |
|---|---|
| Classification | An image category |
| Object detection | Object locations and categories |
| Segmentation | Labels or masks for image pixels |
There are two ways to train the head. Frozen encoder: keep the encoder fixed and train only the head. This is the linear or attentive probing of Chapter 1. Fine-tuning: train the head and also update the encoder. Fine-tuning usually scores higher, because everything adapts. Frozen scores tell you more about the features themselves. The lecture’s rule: replace the reconstruction decoder with the task head, and evaluate the task you need.
MAE hides a random subset of patches. A second family follows the idea behind language models. It reads patches in a fixed order and predicts each next patch from the ones before. AIM, from El-Nouby and colleagues (2024), is an image-only autoregressive model: it predicts its own next piece, one step after another.
Read the lecture’s figure from the bottom up. Split the image into patches and choose an order: 1, 2, 3, and so on. The input is the patches in that order. The target is the same list shifted by one position. The feature at patch i predicts patch i + 1. A small prediction head, an MLP, turns that feature into the next patch’s pixel values.
One rule keeps the game fair: the scored target must stay hidden from its predicting position. If patch i could see patch i + 1, it would copy it. A causal mask enforces this: each position may look only at itself and earlier positions. AIM adds one wrinkle, a prefix. The first S patches may see each other, like the context in a question. Their next-patch targets are then visible to them, so those predictions are not scored. The device shows the mask for 5 patches.
Left: the scan order on a 3 × 3 image. The 5th patch visited is highlighted, with the one before it. Right: the mask for 5 tokens. Filled cells are what each output row may see. Slide the prefix S.
The mask rule and the S = 2 table follow the lecture: rows up to S see the whole prefix, later rows see themselves and earlier tokens, and a row’s target is scored only if it is hidden from that row. The raster and spiral orders follow the lecture’s picture; the checkerboard and random orders are our own examples.
At S = 2, rows 1 and 2 see {x1, x2}. Row 1’s target is x2, which it can see, so that row is excluded. Row 2’s target is x3, hidden, so it is scored. Rows 3 and 4 are scored too, and row 5 has nothing left to predict. That gives 3 scored rows out of 5.
The scan order is a training choice. Raster reads left to right, top to bottom. A spiral works outward from the center. A checkerboard visits alternate cells first. A fixed random order shuffles them. Ask of each: which patch comes before patch 5? The answers differ. The lecture’s reminder is plain: autoregressive order in an image is not physical time. Nothing in a photo happened first.
AIMv2, from Fini and colleagues (2024, CVPR 2025), adds text. One sequence holds the image patches first and the caption tokens after them. At image positions the model predicts the next patch’s pixels, scored with squared error. At text positions it predicts the next caption token, scored with cross-entropy. The two losses add, with a weight α on the image part.
L = Ltext + α Limg
The image comes first so that text prediction can use its visual features. Both losses train the vision encoder that is used downstream. The lecture also reads an ablation. A caption-only baseline, called Cap, is compared with the version that also predicts image patches, with the encoder frozen and an attentive probe on top. The advice is the lesson of the whole chapter: compare objectives within a controlled setup before you compare headline scores.
The lecture closes the family with three honest doubts.
The training target is missing pixels. The downstream target might be an object’s category or location. Reconstruction alone does not evaluate its features. So reconstruction can learn strong features, but its objective does not directly name every downstream need. That leads to the next family, which asks a different question: which views should have similar features?
Chapter 8
Turn “which view belongs to which image” into a classification problem, and read its loss number by number
You are shown a photo of a tower, then two more photos: a second shot of the same tower from another angle, and a giraffe. Which one belongs with the first? You answer at once. You never needed a label. You only needed to know that two of the photos share a source.
That is contrastive learning. The training answer is not a category or a missing pixel. It is the pairing: which views belong together. SimCLR, from Chen and colleagues (2020), is the cleanest version, and Lilian Weng’s post is a good map of its relatives.
The lecture states the mechanism in three beats. First, construct the pairing: sample two views (altered copies) of one example, and call them an anchor A1 and a positive A2. Second, encode and compare: map each view to an embedding (a vector that stands for it) and compute a similarity. Third, favor the positive over the negatives. A negative is a view from a different image, such as B1 or C1. We want the matched view to score higher than every mismatch.
s(A1, A2) > s(A1, B1), s(A1, C1)
Notice what is missing: no pixels are rebuilt. The loss compares embeddings, not reconstructed pictures. Here are the data-flow shapes for one view: an image goes through the backbone Eφ to a feature e, then through a projector gψ to a vector u, and the loss compares u. The lecture adds a note that is easy to miss: the loss compares u, but downstream readouts commonly use e. The projector is a small helper that lives only during pretraining.
Two crops of one image supply a positive pair. How you alter the views decides what the encoder must learn. The main SimCLR recipe combines crop-and-resize, color distortion and blur. (The lecture’s figure surveys more candidates; the main recipe does not use every one.) Weng’s summary of the paper agrees: the composition of random cropping and random color distortion is crucial.
Why color distortion? The lecture builds a small trap. Suppose you only crop. The anchor is a tower with a blue sky. Candidate 1 is another crop of the same tower. Candidate 2 is a giraffe in front of a green field. A rule that says “pick the blue candidate” gets this right without recognizing the tower. Now change each crop’s colors separately. The same tower crop turns orange, and the giraffe crop turns blue. The same color rule now picks the wrong one. Independent color changes make color alone an unreliable clue, so the network has to look at shapes.
A caution from the lecture’s own question: which transformation would harm an orientation-sensitive or color-sensitive task? If you must read the block’s angle, a rotation augmentation tells the encoder to throw the angle away. The views you choose are the facts you ask the encoder to ignore.
Take three source images A, B and C. Two views each give six views: A1, A2, B1, B2, C1, C2. Anchor A1 now chooses among five candidates: A2, B1, B2, C1, C2. A1 itself is excluded. Exactly one candidate, A2, is the positive. Four are negatives.
Lay the six views along both sides of a 6 × 6 table. The diagonal is the self-pair, crossed out. Each row has one positive, from the same source, and four negatives. Both A1 to A2 and A2 to A1 count as training terms, so the table holds six five-way problems. In general, N images give 2N views, and each anchor has 2(N − 1) negatives, as Weng writes.
The cosine similarity of two vectors is their dot product after each is scaled to length 1. It runs from −1 (opposite directions) through 0 (unrelated) to 1 (the same direction). We turn a row of similarities into probabilities with a softmax (the step that makes scores positive and sum to 1, as in a language model) and then take the negative log of the paired view’s probability.
This is a classification task whose correct class comes from how we sampled the views. (The SimCLR paper names this loss NT-Xent, a normalized temperature-scaled cross-entropy.) Now work one row by hand, using the lecture’s example. The positive has similarity 1 and the four negatives have similarity 0, at temperature τ = 1. The positive’s weight is e1 = 2.718. Each negative’s weight is e0 = 1. The sum is 2.718 + 4 = 6.718. So p+ = 2.718 / 6.718 = 0.405, each negative gets 1 / 6.718 = 0.149, and the loss is −ln 0.405 = 0.905. The device runs this exact row. Move the knobs.
One anchor, one positive, four negatives. Bars show each candidate’s probability. The number under each bar is its gradient: the push on its similarity. Start at the lecture’s setting, then lower the temperature and raise the similar negative.
The lecture’s fixed example: positive similarity 1, four negatives at 0, τ = 1, giving probabilities 0.405 and 0.149 and a loss of 0.905. The sliders move the positive and negative 1 only; the other three negatives stay at 0. The gradient is the lecture’s formula, (pk − 1[k = +]) / τ; a negative number means the similarity is pushed up, a positive number means it is pushed down.
Read the gradient row. The positive carries p+ − 1 = −0.595 (at τ = 1), a push up. Each negative carries its own probability, 0.149, a push down. Higher-probability negatives get a stronger push, which is why a negative that looks too much like the anchor is pushed hardest. Now set τ to 0.1, the positive’s similarity to 0.6 and the similar negative to 0.9. That negative takes about 95% of the row and the loss jumps past 3. Lower τ sharpens whatever the current preference is, even when it is wrong. These gradients flow back through the projector and the encoder, which is how the features change.
Here is a second row with made-up scores, to practice. At τ = 0.5, say the positive scores 0.9 and the negatives score 0.1, 0.2, −0.1 and 0.0. The weights are e1.8 = 6.05, e0.2 = 1.22, e0.4 = 1.49, e−0.2 = 0.82 and e0 = 1.00. Their sum is 10.58. So p+ = 0.572 and the loss is 0.559.
pythonfrom scipy.special import logsumexp def nt_xent(u, tau=0.5): # u: (2N, d); rows 2k and 2k+1 are two views of image k u = u / np.linalg.norm(u, axis=1, keepdims=True) s = (u @ u.T) / tau # (2N, 2N) scaled cosine scores np.fill_diagonal(s, -np.inf) # a view never competes with itself pos = np.arange(len(u)) ^ 1 # partner of row i: 0<->1, 2<->3, ... logp = s - logsumexp(s, axis=1, keepdims=True) return -logp[np.arange(len(u)), pos].mean()
Chapter 9
See why the loss needs a push as well as a pull, what goes wrong with false negatives, and how two big systems cope
Suppose a lazy student gets this rule: “Make two views of the same picture look alike.” The student finds a perfect answer in one second. Ignore the picture and write the same word on every page. All views look alike. The score is perfect, and the student has learned nothing.
A network can do the same. This chapter is about the part of the loss that stops it: the negatives. It also covers what goes wrong with them, what they cost at scale, and how CLIP and SigLIP, the two big image-and-text systems in the lecture, use them.
Take the simplest loss that agrees with “two views should match”. It measures how far apart the two views’ vectors are: ℓpair = ∥u(1) − u(2)∥2. The encoder can reach ℓ = 0 by sending every input to the same vector c. That failure is called collapse. The loss is perfect and the features say nothing, because every picture gets the same description.
Now ask the positive to beat four negatives instead. If the encoder collapses, all five candidates tie. Each gets probability 1/5, and the loss is −log (1/5) = log 5 = 1.61. Pushing the negatives away from the anchor, so that the positive stands out, lowers the loss. So the negatives give the loss a reason to keep different pictures apart. The device trains a tiny encoder both ways, so you can see collapse happen.
Eight pictures, each a colored point in a plane. A tiny encoder (a 2 × 2 matrix plus an offset) places them. Choose the loss, then press Train. The line below traces how far apart the eight points sit.
Toy, trained live in your browser with finite-difference gradients: 8 inputs spread around a circle, 2 noisy views each, embedding u = W x + b, 300 steps at rate 0.3. “Pull only” is the squared distance between the two views’ vectors. “Pull and push” is the contrastive loss of Chapter 8 at τ = 0.5, on cosine similarity. The spread is the average distance between the 8 clean points. Not a measured result from the course.
In the pull-only run the points slide together until the spread reaches 0.01. Eight different pictures now have one description. In the push-and-pull run the spread stays near 1.5, the loss settles at about 1.5, and the pictures stay apart. The lecture puts it in one line: simply deleting the negatives is insufficient. It also says that learning without negatives is possible, because there are other ways to prevent collapse (Thursday’s subject, with BYOL as the famous example).
In SimCLR the negatives are the other pictures in the same batch, the group of examples one training step handles together. That is cheap, but it rests on an assumption: “a different image ID means a different thing.” The lecture breaks that assumption with a robot example. A batch holds two nearby frames of one video. The anchor is frame A. The positive is another crop of A. A negative is frame B, a moment later, where the T-block is in nearly the same place.
The training rule says: lower A’s similarity to B, because B has a different image ID. But what we wanted was the opposite. Similar object configurations should stay easy to recognize as similar. A false negative is a pair that the sampler separates even though the pair shares information we want to keep. More negatives mean more comparisons, but also more false negatives, more score computation and more communication. Weng’s post meets the same problem under the name of sampling bias, and describes debiased and hard-negative-weighted losses as fixes.
More negatives usually help. The lecture shows how a big run gets them. Take 100 GPUs, 256 images on each, and 2 views of each image: 100 × 256 × 2 = 51,200 vectors. Each GPU holds 256 × 2 = 512 of them.
The exchange carries feature vectors, not images. For a sense of size, suppose each vector has 128 numbers stored in 2 bytes each (an assumption, not the lecture’s figure). Each GPU receives 50,688 vectors from its peers, about 13 megabytes per step. That is the “communication” cost in the lecture’s list.
CLIP (Radford and colleagues, 2021) applies the same idea to two kinds of data. The positive is an observed image and its caption. The other captions in the batch act as negatives. Two encoders, one for images and one for text, form an image-by-text similarity matrix. A softmax along each row picks the right caption for an image. A softmax along each column picks the right image for a caption. Weng describes it as maximizing the N correct pairs and minimizing the N(N − 1) incorrect pairs of a batch, and notes the 400 million image and text pairs it was trained on.
After training you can classify without class labels. Write each class as a prompt, such as “a photo of a dog”, and pick the prompt closest to the image. The lecture’s caution for a world model: text supplies semantic supervision at scale, but captions may omit the geometry or motion that a world model needs. A caption says “a T on a table” and not “the stem is 12 degrees clockwise”.
CLIP’s softmax compares one caption with all competing captions in its row. That needs the whole batch at once. SigLIP (Zhai and colleagues, 2023) changes the loss: score every image and text pair on its own, as a yes-or-no question.
For an image of a tower beside a river, the caption “A tower beside a river” has label +1, so the loss encourages a higher score. The caption “A pair of giraffes” has label −1, so the loss encourages a lower score. There is no batch-wide softmax denominator, so pair losses can be computed and added up block by block. The lecture keeps the honesty: SigLIP still uses negative pairs. Cross-device negatives still need communication, and false negatives remain.
Chapter 10
Keep the numbers, the checklist and the core code, then carry the open question into Lecture 6
You can now reproduce the first half of the week from memory. A representation is a description of an observation in a chosen form. A probe tests it with a tiny readout on a frozen encoder, and a probe score belongs to the pair of encoder and probe. Good features keep what the later question needs, and what that is depends on the question. Self-supervised learning lets the data make its own target. Reconstruction hides or alters the input and recovers it: rotation, jigsaw, colors, holes, noise, masked patches, next patches. Contrastive learning asks which views belong together and needs negatives to avoid collapse. Each method makes the features keep different facts.
| What | Number | Why it matters |
|---|---|---|
| Linear probe | y = W e + b; train only W and b | The encoder stays frozen, so the score tests the features |
| Mean pooling | weight 1/K per patch | Dilutes one useful patch among K |
| Relative position | 8 offsets, chance 12.5% | The sampler knows the answer |
| Rotation | 4 angles, chance 25% | Forces orientation cues into the features |
| Jigsaw, gap or no gap | 98% puzzle, 47.7 mAP vs 88% puzzle, 52.6 mAP | A harder puzzle transferred better |
| MAE patches | 4 of 16 visible; 75% in the main recipe, 80% in the examples | Encoder cost falls with the visible count |
| MAE loss | mean over masked patches only | Visible patches would allow copying |
| AIM prefix S = 2 | 3 of 5 rows scored | A target must be hidden from its predicting position |
| AIMv2 loss | L = Ltext + α Limg | One sequence trains one encoder on two targets |
| SimCLR toy batch | 3 images, 6 views, 5 candidates per anchor | Each row is a classification problem |
| One loss row at τ = 1 | p+ = 0.405, loss 0.905 | Gradient (pk − 1[k = +]) / τ |
| All candidates tied | p+ = 1/5, loss log 5 = 1.61 | What collapse costs once negatives exist |
| Big-batch negatives | 100 × 256 × 2 = 51,200 vectors; 51,198 negatives per anchor | Negatives come from the batch, shared by all-gather |
The lecture ends its map with one habit: for any method, name the visible input, the target, the loss and the reused features. Here is the table, filled in.
| Method | Visible input | Target | Loss | Reused |
|---|---|---|---|---|
| RotNet | Turned image | The angle | Cross-entropy, 4 classes | Encoder |
| Jigsaw | Shuffled tiles | The permutation | Cross-entropy | Encoder |
| Denoising autoencoder | Corrupted input | Clean input | Reconstruction error | Encoder |
| MAE | Visible patches | Pixels of masked patches | Squared error on masked | Encoder |
| AIM | Earlier patches | Next patch pixels | Squared error | Encoder |
| SimCLR | Two crops | Which view matches | Softmax over candidates | Backbone feature e |
| CLIP, SigLIP | Images and captions | Which caption matches | Softmax or sigmoid | The encoders |
pythonimport numpy as np from scipy.special import logsumexp # 1. A linear probe on frozen features e: (n, d) -> angle labels as (cos, sin) def fit_probe(e, angle, lam=1e-3): y = np.stack([np.cos(angle), np.sin(angle)], 1) # (n, 2) X = np.hstack([e, np.ones((len(e), 1))]) # add the bias column return np.linalg.solve(X.T @ X + lam * np.eye(X.shape[1]), X.T @ y) # (d+1, 2): W and b # 2. The MAE loss: score only the masked patches def mae_loss(pred, target, masked): # pred, target: (16, p); masked: indices return ((pred[masked] - target[masked]) ** 2).mean() # 3. The SimCLR loss (NT-Xent): u is (2N, d), rows 2k and 2k+1 are views of image k def nt_xent(u, tau=0.5): u = u / np.linalg.norm(u, axis=1, keepdims=True) s = (u @ u.T) / tau np.fill_diagonal(s, -np.inf) pos = np.arange(len(u)) ^ 1 logp = s - logsumexp(s, axis=1, keepdims=True) return -logp[np.arange(len(u)), pos].mean() # 4. The SigLIP loss: every image-text pair is its own yes or no def sigmoid_loss(zi, zt, a, b): # zi, zt: (n, d), unit length; a > 0 s = zi @ zt.T # (n, n) cosine scores y = 2 * np.eye(len(zi)) - 1 # +1 on the diagonal, -1 elsewhere return np.logaddexp(0, -y * (a * s + b)).mean() # -log sigmoid(z) = log(1 + e^-z)
The lecture closes by asking what each family leaves out, using the world-model test: does the target expose position, contact and motion?
| Reconstruction | Contrastive learning | |
|---|---|---|
| The training answer | Original content or structure (DAE, MAE, AIM, AIMv2) | Which examples should match (SimCLR, CLIP, SigLIP) |
| What can go wrong | Easy clues solve the task; pixel targets reward irrelevant detail or have several plausible answers | Positive views can erase useful differences; false negatives separate similar cases; large pools cost computation and communication |
| The world-model test | Does recovering the target expose position, contact and motion? | Do the pair labels preserve the differences later predictions need? |
Thursday’s lecture, Lecture 6, turns to self-distillation and prediction in feature space. DINO matches a teacher’s outputs. data2vec predicts contextual features. The JEPA family predicts features of hidden image regions or video. The question it inherits: if the targets are learned too, how do we keep them informative without negatives?
Credit: this lesson re-teaches Lecture 5 of CIS 6280 World Models (University of Pennsylvania, Fall 2026), taught by Jiatao Gu, with Lilian Weng’s post on contrastive representation learning as the reading. Read the course’s own slides and the course site. The explanations, devices and toy numbers here are ours, and each toy is labelled as one.
Now press Present or Teach and explain one training method out loud, from memory: what is visible, what the target is, who makes it, and what the encoder must keep to score well. Then go back to the two rings at the top, set the ring blur to 0.6, and say why polar coordinates still label about 96% right.