Representation Learning I

A representation is only as good as what a simple readout can recover from it. Rewrite the same points in polar coordinates, and a line that scored 50% scores 100%.

Learn what a world model should keep from what it sees, how to test that with a probe, and how reconstruction, masking, next-patch prediction and contrastive matching each teach an encoder.

Switch the coordinates, then press Fit the readout. Blur the rings and see how far the new form carries the same line. Then we build, piece by piece, everything underneath: probes, pretext tasks, masked autoencoders, and the contrastive loss.

You need a feel for a vector (a short list of numbers) and a little Python for the code. We build the rest from zero.

Based on Lecture 5 of Penn CIS 6280: World Models (Fall 2026), taught by Jiatao Gu, and its reading, Lilian Weng’s Contrastive Representation Learning. Lecture slides. Every idea is re-taught here in our own words and devices; none of the course’s slide images appear on this page.

Fit a straight-line readout

0.20

Toy data, seeded so it is the same on every visit: 60 points on an inner ring (radius 0.5) and 60 on an outer ring (radius 1.0). The readout is logistic regression, 400 steps of gradient descent on standardized features. This is the lecture’s coordinate example (its Cartesian and polar picture), rebuilt as a live fit; the accuracies are measured here, not copied.

Chapter 0

What a Representation Is

See why the form you give a scene decides which questions are easy to answer about it

A robot arm pushes a flat T-shaped block across a table. A camera watches from above. Before each push, you may write down a few facts about the scene. After the push, the block turns one way or the other. Which facts must you write down to call the turn right?

Your first idea is a name: “a pusher and a T-shaped block.” That sentence is true, and it is useless. It reads the same before every push. It cannot tell you which way the block will turn.

Better facts are about where things are. How high is the pusher? How high is the block? How is the block turned? The device below tests this. It makes 3,000 test pushes, then predicts each turn from only the facts you keep.

Choose what to record

Left: one push. The pusher moves right and hits the block. The dot marks the block’s balance point. Right: how often your kept facts call the turn right. Switch facts on and off. Press Show another push to see a new scene.

Illustrative toy, not the course’s PushT simulator. The pusher always comes from the left. A push turns the block counterclockwise if it hits below the balance point, clockwise if above. The balance point sits 0.35 units from the block’s middle along its stem. Each fact is cut into 8 bins; the guess for each bin is the majority outcome of 6,000 training pushes, scored on 3,000 fresh ones.

With the name alone you are right 50% of the time, a coin flip. Add the pusher’s height and you reach about 73%. Add the block’s height and you reach about 89%. Only when you also record the angle does the score climb to about 96%. Each fact you drop costs you part of the answer.

A representation is a description in a chosen form

Now give the idea its name. The camera gives an observation: the raw thing the sensor reports, here a grid of colored pixels. A representation is a description of that observation, written in a form we choose. The lecture writes it as a function at work:

e
The representation: a list of numbers that describes the observation.
Eφ
The encoder: a function that turns an observation into a list of numbers. The small φ stands for its settings, which training can change.
o
The observation: the raw pixels, or whatever the sensor gives.

The lecture lists three common forms. Each one keeps different facts and costs a different amount of space.

FormWhat it isSize in our toy
PixelsAn array of red, green and blue valuesA 96 × 96 frame is 96 × 96 × 3 = 27,648 numbers
Measured coordinatesThe pusher’s position and the block’s pose (where it is and how it is turned)5 numbers: pusher x and y, block x and y, angle
Learned featuresNumbers an encoder produces after trainingYour choice, for example 256 numbers

The frame size and the feature size are our own round numbers, to show the shapes. The lecture’s point is the last line of its slide: learning the representation changes which relationships are easy to compute.

The same information can be easy or hard to use

Go back to the instrument at the top of the page. It holds two rings of points. The inner ring is one class and the outer ring is another. The class is the same in both coordinate systems. Only the way we write each point down changes.

In Cartesian form, each point is (x, y): how far right and how far up. No straight line separates two rings that share a center. Any line leaves about half of each ring on the wrong side. In polar form, each point is (r, θ): its distance from the center and its angle. Now the inner ring has r near 0.5 and the outer ring has r near 1.0. One threshold on r separates them.

The lecture states this plainly: the class information is unchanged, and a threshold on the radius becomes sufficient. Nothing was added to the data. We only moved it to a form where the readout we can afford, a straight line, is enough.

A representation is not yet a state

Lecture 4 introduced the state of a world model: the variable that the model evolves forward in time. The lecture draws the two ideas side by side, so the difference matters.

A representation may use history. The lecture writes it as zt = Eφ(ot; o<t, a<t): the encoder sees the current frame, the earlier frames o<t and the earlier actions a<t. Nothing forces it to be Markov, which means that the present summarizes the past completely. A state, in Lecture 4’s model, is assumed to be Markov: given the current state and action, the earlier states do not change the next-state distribution.

p(st+1 | s0:t, a0:t) = p(st+1 | st, at)

Think of a diary and a snapshot. A representation is allowed to be a diary: it can hold what happened before. A Markov state is a snapshot that already contains everything the diary would add. The slide’s warning follows: using temporal context does not automatically make a representation a Markov state.

The week in one paragraph

This is Lecture 5 of CIS 6280, World Models, at the University of Pennsylvania, taught by Jiatao Gu on September 8, 2026. This week has two parts. Today covers data targets and view matching. Thursday covers teachers, feature targets and distribution constraints. Slides and readings live on the course site. The reading for today is Lilian Weng’s post on contrastive representation learning, and we weave it into Chapters 8 and 9.

The road

  1. How probes test a representation: freeze the encoder, fit a tiny readout, read the score.
  2. What features must keep: nuisance depends on the question; invariance versus equivariance.
  3. Where features come from: labels, rewards, next tokens, and targets the data makes.
  4. Pretext tasks: rotation, relative position and jigsaw, and the shortcut that spoils them.
  5. Predicting what is missing: colors, holes and denoising.
  6. Masked autoencoders: hide most patches, encode the rest.
  7. Reusing the encoder: downstream heads, and next-patch prediction.
  8. Contrastive learning with SimCLR: the loss, one number at a time.
  9. Negatives, CLIP and SigLIP: collapse, false negatives, and scale.
  10. Field guide: numbers, checklist, code, and Thursday’s question.
Two rings of points share a center. A straight line fails on their (x, y) values but works on their (r, θ) values. What changed?

Chapter 1

How Probes Test a Representation

Freeze an encoder, fit a tiny readout on top, and let the score tell you how easy the information is to reach

A friend hands you a sealed box. Inside is an encoder that someone trained on thousands of pictures of the pushed block. You cannot open the box. You want to know one thing: does it know how the block is turned?

You can ask the box for its description of any picture. That description is a list of numbers. Your plan is simple. Fit a tiny gauge on top of those numbers, and see whether the gauge can read the angle. If the gauge reads it well, the angle is easy to reach. This gauge is called a probe.

Freeze the encoder, train only the gauge

The word frozen matters. A frozen encoder keeps its weights fixed, so training the probe cannot change the description. If the probe scores well, the encoder deserves the credit. If you let the encoder learn too, you would be testing your training run, not the box.

The lecture’s example reads the object’s orientation. The probe is the simplest model there is, a linear probe: multiply the feature vector by a matrix and add a vector.

ŷ
The probe’s answer: two numbers.
W, b
The only things we train. W is a small grid of numbers, 2 rows by as many columns as the feature has numbers. b is a list of 2 offsets.
e
The frozen feature vector from the encoder. We never change it.
(cos α, sin α)
The true answer, written as a point on a circle. We use two numbers instead of the raw angle because angles wrap around. 359° and 1° are 2° apart, but as plain numbers they look 358 apart.

Training has two steps. First, fit: train only W and b on labeled images, and minimize the squared error between the answer and the known orientation. Second, test: predict the angle in scenes the probe has never seen, and report the angular error. If the task is a category instead of an angle, the probe predicts one score per class and trains with cross-entropy, a loss that is small when the right class gets a high score.

Here is one probe answer by hand. Say an image gives the feature e = (0.8, 0.6, 1.0), and the fitted W is [[1, 0, 0], [0, 1, 0]] with b = (0, 0). Then ŷ = (0.8, 0.6). The angle of that point is atan2(0.6, 0.8) = 36.9°. If the true angle is 40°, the angular error is 3.1°. Average that over many unseen images and you have the probe’s score.

Compare fairly, and read the result with care

The lecture gives two rules. First, compare encoders fairly: use the same labels, the same way of extracting features, the same probe type and the same test scenes. Change one thing at a time. Second, read the answer with humility. A successful linear probe shows that the information is easy to access. A failed probe does not prove the information is missing. A stronger probe might still find it.

Global and local features

What does the encoder hand back? There are two common answers. A global representation is one vector for the whole image, such as (0.2, −0.7, …). A local representation is one vector for each image region, a grid e1, e2, e3 over e9 for a 3 × 3 split. Local features keep their spatial arrangement. They can also use whole-image context, because each region’s vector was computed while the model looked at the other regions. A vision transformer makes exactly this kind of output: one feature per patch.

As shapes, a local representation is a table of K × d numbers: K patches, each with d numbers. A global one is a single 1 × d row. Our example size is K = 196 patches of d = 768 numbers; that is 150,528 numbers instead of 768.

Linear probing and attentive probing

Both probes freeze the encoder and start from the same patch features e1 to eK. They differ in how they squeeze K vectors into one. A linear probe uses a fixed pooling, usually the mean, then a linear head. Every patch gets weight 1/K, whatever it shows. An attentive probe learns how to weigh the patches. A learned query (a vector the probe trains, which asks “does this patch matter?”) scores each patch. A softmax turns those scores into positive weights αk that sum to 1. The weights depend on the input.

Linear probe
Average all K patch features, then apply the head. Train only W and b.
Attentive probe
The weights αk come from a learned query that scores each patch; vk are projected copies of the patch features. Train the attention pooling and the head together.

Why does the choice matter? Picture the block filling one patch out of 64. The mean gives that patch weight 1/64. Its signal is nearly drowned by the 63 patches that show only table. The attentive probe can give the block patch most of the weight. Try it below.

Mean pooling or attention

Top: K patches. One holds the block, with its direction drawn. Below: how much weight each probe gives each patch. Bottom: the average angle error of each probe on 300 unseen images. Drag K up, then raise the noise.

16
0.20

Illustrative toy with seeded data. Each patch feature is 3 numbers: the block’s (cos, sin) and a presence flag, plus Gaussian noise; empty patches hold only noise. Both heads are fitted by ridge regression on 400 images. The attentive probe’s query is fixed to the presence flag (a trained query would learn this); its weights are softmax(4 × presence). Not a measured result from the course.

At K = 4 the two probes are fairly close, 18° against 10° with the default noise. At K = 16 the mean-pooling probe misses by 40° while the attentive probe stays near 9°. The signal did not vanish. Mean pooling spread it thin and added the noise of every empty patch. The lecture’s summary says it well: attentive probing learns a richer readout, so you must report the probe type when you compare scores.

A frozen encoder scores badly under a linear probe on object angle. What may you conclude?

Chapter 2

What a World Model’s Features Must Keep

Decide which details to keep and which to ignore, and see why the answer depends on the question you ask

The robot pushes the T-block from the left. Where should the pusher touch it? If the pusher hits the stem, the block slides one way. If it hits the crossbar, the same push turns the block another way. The scene looks almost identical in both cases. The result does not.

So a world model that wants to predict the push must keep some details and may throw others away. This chapter asks which ones. The answer is never “everything” and never “the name.” It depends on what you will do with the features next.

Features are the input to a future predictor

Here is where representations sit in a world model. The encoder reads the observations o0 to ot and writes features z0 to zt. A predictor, another network Fθ, reads those features and the actions the robot took, and guesses the next observation.

ôt+1
The predicted next observation. The hat means “a guess”.
Fθ
The predictor network, with trainable settings θ.
z≤t
All features so far. The predictor may look back at earlier ones when the newest feature does not hold enough.
a≤t
The actions taken so far, which the predictor always knows.

The lecture splits the work in time. This week we learn the encoder and test its features. Later, we learn the predictor from sequences. So everything in this lesson asks one question: when a predictor reads these features later, will the facts it needs be there?

Robots need to know where things are

The lecture shows one concrete choice from robotics. Instead of one big vector, represent an image as a set of keypoints: K learned visual points, each with a position, written as e = (x1, y1, …, xK, yK). That is 2K numbers. For K = 16, that is 32 numbers instead of 27,648 pixels. The idea comes from the Deep Spatial Autoencoders paper by Finn and colleagues (ICRA 2016).

The reason is the push. Same T, same push to the right, but different contact point: the stem in one case, the crossbar in the other. Different contact can lead to different motion. So for prediction, preserve the positions and orientations that affect contact.

Whether lighting matters depends on the target

A nuisance variation is a change in the image that does not matter for your goal. Lighting is the usual example. But the lecture’s slide makes a sharper point with a three-row table. Lighting is not a nuisance in itself. It is a nuisance for some questions.

Downstream questionEffect of a lighting changeWhat you want
Where is the object relative to the pusher?Same geometry, different illuminationKeep geometry easy to read
What will the next RGB frame look like?Brightness and shadows change the pixelsRetain or model appearance
What object is present?Many lighting changes keep the identityStable category information

Read the middle row twice. If your world model must draw the next frame, it must know the brightness. A code that threw lighting away cannot draw it. So “nuisance” names a relationship between a variation and a task. Try it with a toy.

Keep or drop a fact

Each scene has a geometry value g (where the object is), a lighting value b and a shape cue. Choose which of the three your code keeps. Each bar scores one question from the table, using only the kept facts. Slide the lighting to see one frame change.

0.0

Illustrative toy, not measured on real images. 3,000 scenes with g and b drawn uniformly from −1 to 1 and a shape cue equal to the object class (−1 or +1) plus noise. The next pixel is 0.5 + 0.3 g + 0.5 b. Scores are the share of variance a least-squares fit explains (questions 1 and 2) or the accuracy of a threshold on the fit (question 3).

Start with geometry only. Question 1 scores 100%: the code kept exactly what it needs. Question 2 scores about 28%: the code cannot draw the brightness. Question 3 sits near 50%, a coin flip, because shape was dropped too. Turn lighting on and Question 2 rises to 100%, while Questions 1 and 3 do not change. Turn the shape cue on and Question 3 reaches about 95%. No code wins every row until it keeps all three facts, and a real code has a limited size.

Invariance and equivariance make different promises

Two words describe how an encoder should react when the input changes by a transformation T, such as a rotation. An invariant encoder ignores T: E(To) ≈ E(o). A rotated cup is still a cup, so the features do not move. An equivariant encoder tracks T in a predictable way: E(To) ≈ ρ(T) E(o), where ρ(T) is a known matrix that says how the features must change.

Take a worked case. The block’s angle is stored as the point e = (1, 0), which means 0°. Rotate the block by 90°. An equivariant code moves to (0, 1): the point turned by the same 90°, so the angle is still readable. An invariant code stays at (1, 0): the angle is gone, and a probe for angle can only guess. For “which object is this,” the invariant code is ideal. For “how is it turned,” you need the equivariant one.

A world model needs both promises, each in the right place. It wants identity to survive a push. It wants pose to change in a way the predictor can follow. That is why the next chapters keep asking, for each training method, what it makes the features keep.

Here is a failure you will meet. The symptom: a world model draws crisp T shapes but gets the contact point wrong, so its predicted pushes turn the wrong way. The measurement: probe the code for the contact point, and compare the probe error with a probe for the shape. If the shape probe scores high and the contact probe scores low, the code kept what the block is and dropped where it touches. The fix: keep the coordinates that decide contact, for example keypoints, or add a training signal that needs them.

A code keeps object position but drops lighting. For which question does that hurt?

Chapter 3

Where Features Come From

Follow the training signal from labels to rewards to raw data, and meet the idea of a target the data makes for itself

Picture a network trained to do one plain job: look at a photo and name its category. Nobody tells it to build a good description of the photo. It does so anyway, because the last layers cannot name the category unless the layers before them have organized the picture well.

That is the first surprise of this chapter. Features are often a side effect of solving some other task. The lecture shows four examples, and each one names its training signal: the thing the model is scored against.

Four ways a network ends up with features

Class labels. In AlexNet (2012), the training signal is a human category label for each image. After training, the hidden features of two images are close when the images are related, so nearby features retrieve related pictures. Training updated both the output head and the layers that produce its features.

Next-word prediction. A native multimodal model such as Kimi K3 (2026) learns its visual features from scratch. The image goes through a vision encoder, called MoonViT-V2 in the lecture. The visual features and the text then predict the next text token. One loss trains the vision encoder, the projector that connects it to the language model, and the language backbone together.

Game outcomes. The original AlphaGo learns features from the board. Its training signal starts with expert moves, then adds self-play and game results. It combines supervised learning and reinforcement learning. The features serve a policy (which move to pick) and a value (who is winning).

The next observation. A world model reads a history of observations and actions, builds features, and predicts the future. Its training signal is what is actually seen next. If the encoder is trained jointly with the predictor, the prediction error also updates the encoder.

The lecture sums it up in one line: representation learning happens in supervised learning, in reinforcement learning, and in predictive modeling. Useful features can be learned while solving another task.

Why this lecture turns to unlabeled experience

A world model needs more than object names. What matters changes over time: pose, motion, contact, occlusion (one thing hiding another) and the results of actions. To label all of that by hand, in every frame, costs too much. Rewards can be sparse, with a signal only once in a long task, or tied to a single task.

But experience already holds relationships. Neighboring moments look alike. Different views show the same thing. Actions have outcomes. The lecture’s question is whether a model can learn from those relationships without anyone labeling each moment. It adds a caution: human learning also uses language, instruction and social feedback, so the picture of a child at play motivates the question and does not name one learning algorithm. This week, then, we learn reusable representations from data without manual labels for every training target.

The data makes its own target

Self-supervised learning (SSL) is the name for this. The recipe has three beats. Change the data in a way you control. Ask the network to recover what you did, or what was hidden. Score it against the answer you already have, because you made the change.

The lecture’s first example is a rotation. Take an image o. Apply a rotation T90 that turns it 90 degrees. Send the turned image through the encoder Eφ, then through a small rotation head qψ that outputs a score for each possible angle. The sampler knows the answer, 90°, because it chose the angle.

L
The loss: one number that is large when the guess is bad. CE is cross-entropy, the usual loss for picking one of several classes.
qψ
The rotation head: a small network that turns features into four scores, one for each of 0°, 90°, 180° and 270°.
Eφ
The encoder from Chapter 0. This is the part we want to keep.
T90 o
The image o, turned by 90°.
90°
The answer. No person wrote it. The code that did the turning knows it.

After pretraining, throw the rotation head away. Keep the encoder, and fit a new readout for the angle or the geometry you really want. The rotation task was only a way to make the encoder learn. The lecture later names such a task a pretext task.

A map of three families

The lecture organizes the field into three families. They differ in what supplies the training answer. Use the same four questions on each one: what is the visible input, what is the target, what is the loss, and which features do we reuse afterward?

FamilyWhat supplies the training answer?Examples
ReconstructionRecover the original content or structure from partial or altered dataRotation, Jigsaw, DAE, MAE, AIM
Contrastive learningWhich other view belongs with this example?SimCLR
Match learned targetsA teacher feature, an assignment, or another view’s embeddingDINO, data2vec, JEPA

Today covers the first two. The third is Thursday’s topic. The device below runs the four questions on five different training methods, so you can feel how the same encoder gets very different homework.

Who supplies the answer

Pick a training method. A pulse walks from the input to the reused features, one stage at a time. Watch who or what provides the target.

Sketches of the methods in the lecture’s map, simplified: RotNet (4 angles), masked autoencoders (the lecture’s 4 of 16 patch example) and SimCLR (three images, six views). Counts come from those examples; the stage text is our own summary.

Notice the third box in each row. In the first row, a person fills it. In every other row, the data does. That is the whole trick, and the reason the chapters ahead can use as much unlabeled video and imagery as exists.

In rotation pretraining, where does the target “90°” come from?

Chapter 4

Pretext Tasks: Rotation, Position and Jigsaw

Build a puzzle whose answer the data knows, and watch an easy shortcut spoil what the network learns

You want a network to understand pictures, and you have a million photos with no labels. Here is a trick. Cut each photo into pieces, shuffle them, and ask the network to put them back. Nobody has to label anything. You made the shuffle, so you know the answer.

Such a made-up job is called a pretext task: a training task built only so that the network learns features worth reusing. The puzzle itself is not the goal. This chapter walks through the classic ones, and then shows how a puzzle can fail.

Alter or hide, then recover

The lecture groups these under reconstruction: alter or hide some information, then predict the original content or its structure. It shows three pairs of alteration and target.

The first two recover structure with an angle or layout label. The third recovers data, meaning pixels or tokens. In all three the picture is: altered input, then encoder, then features, then a prediction of the original information.

Relative position: where was this patch?

Doersch and colleagues (2015) take two patches from one image: a reference patch and a query patch. A shared encoder reads both. A pair classifier then chooses among eight possible offsets: the query was above, below, left, right or at one of the four corners of the reference. The sampler records the offset, so it knows the answer.

A guess by luck is right 1 time in 8, which is 12.5%. To do better, the network must recognize parts and their arrangement, such as that a window sits above a door. The lecture then asks a sharper question. What did this task teach? It runs a new task, retrieval: take a query patch and fetch its nearest neighbors in feature space. Random features fetch random textures. Features trained for position fetch patches of the same kind of thing. A task built for one purpose revealed structure beyond it.

Rotation: which way is up?

RotNet (Gidaris and colleagues, 2018) generates the target itself: turn the image by 0°, 90°, 180° or 270°, and ask which one was used. A lucky guess is right 25% of the time. To do better, the encoder must keep cues that tell the turns apart: the sky is up, a face has eyes above a mouth, a tree has its roots below.

The lecture compares where features respond in two networks, one trained on object recognition and one trained on rotation prediction. The maps suggest object structure in both. The lecture is careful: a map only suggests. A separate probe must test the variable you actually need. It adds a distinction that matters later: predicting a rotation and matching rotated views make different demands on the encoder.

A useful task hides the answer but keeps the clues

Not every puzzle teaches. The lecture’s thought experiment uses a shape, either a T or an L, and asks the network to restore it. Look at three versions.

  1. The entire shape is visible. Copying is enough. The error can be low with no understanding.
  2. The middle is hidden. The visible parts and their arrangement give clues. Structure helps fill the gap.
  3. The entire shape is hidden. The input cannot tell a T from an L. More difficulty now means more guessing, not better features.

Here is why useful features can emerge from the middle case. When structure helps predict the hidden content, lowering the error rewards features that encode that structure. The design rule: remove the copying shortcut, and keep enough evidence that inference remains useful.

Jigsaw: a shortcut you can measure

Noroozi and Favaro (2016) turn spatial layout into a target. Cut an image into tiles, apply a permutation, and ask which permutation was applied. The task should reward understanding how regions relate. But tiles that touch can expose a shortcut. If two tile edges continue the same line of color, you can reassemble them by matching pixels along the cut, with no idea what the picture shows.

Try this yourself. The device below builds smooth random pictures, cuts each into 2 × 2 tiles with a gap of g pixels between them, and shuffles the tiles. The solver is dumb on purpose. It tries all 24 orderings and picks the one whose touching edges match best. It looks only at edge pixels.

Beat the edge shortcut

Left: a picture cut into four tiles. The pink bands are pixels thrown away between tiles. Right: the solver’s score as the gap grows, over 150 pictures. Drag the gap from 0 to 8.

0

Toy: smooth random pictures of 64 × 64 pixels, tiles of 28 × 28, the solver compares only the pixels along each shared edge. This is our own illustration of the shortcut; the accuracy numbers are measured here and are not the paper’s.

With no gap, the solver restores the layout almost every time, using nothing but edge pixels. At a gap of 4 it falls to about half (56%). At 8 it drops to 9%, not far above the 4% of a random guess among 24 orderings. A network trained on the no-gap puzzle can win the same way, by reading edges.

The paper ran a controlled version of this on real networks. With normalization and color jitter fixed, it compared two settings. With no gap between tiles, jigsaw accuracy was 98% and detection on the VOC benchmark scored 47.7 mAP. With a gap, jigsaw accuracy fell to 88% but detection rose to 52.6 mAP. Here mAP is mean average precision, a score for how well a detector finds and names objects; higher is better. Removing an easy cue hurt the puzzle score and improved the detection result. A better score on the pretext task does not mean better features.

Different tasks teach different features

TaskConstructed targetPossible useful evidence
Relative positionNeighbor classPart layout and scene geometry
RotationApplied angleObject orientation and upright structure
JigsawPatch permutationSpatial arrangement
ColorizationMissing color channelsObject and material context
InpaintingMissing image regionShape and contextual completion

The last two rows are the next chapter. Notice the habit this table builds: for any pretext task, name its target, and then ask what evidence the network would need to hit that target.

Adding a gap between jigsaw tiles lowered puzzle accuracy from 98% to 88% and raised detection from 47.7 to 52.6 mAP. What does this show?

Chapter 5

Predicting What Is Missing

Fill in colors, holes and noisy values, and see why the loss you pick decides what kind of guess you get

Hand a child a black-and-white photo of a toy car and a box of crayons. What color is the car? She might pick red. She might pick blue. The photo does not say. It only says “car”, and cars come in many colors.

Now ask a network to do this job, millions of times, and use the job to teach it what cars, grass and skies look like. This is the idea of predicting missing data: remove part of the input, and train the network to put it back. The next chapters use it at scale. First we need to see what the choice of loss does to the answer.

Colorization: the same gray, several colors

Zhang and colleagues (2016) trained a network to color gray photos. The training pairs cost nothing: take any color photo, drop its color channels to get the input, and keep the original colors as the target. The lecture shows good results and then a failure. A gray object can honestly be several colors. The same gray evidence can support several plausible colors.

To learn the object and its material, the network must use context: grass is probably green, sky is probably blue, a banana is probably yellow. That is the pretext task’s gift. The lecture lists the likely evidence as object and material context.

Inpainting: fill a hole from its surroundings

Pathak and colleagues (2016) cut a hole out of the middle of a photo and ask a context encoder to fill it. The missing center must be inferred from the surrounding evidence: the shape of the car edge on both sides, the lines of the street. The lecture shows two outputs of the same system, one trained with a squared-error (L2) loss and one trained with L2 plus an adversarial loss, a second network that tries to tell fills from real photos. Its comment is brief and important: the loss changes which errors the reconstruction is encouraged to avoid.

Why would the loss matter? Think of the car again, with two possible colors. The device below makes the problem concrete.

Average, pick, or draw

The true car is orange or blue. Slide the chance that it is blue. Each row is one way to guess the color: the average (what a squared-error loss learns), the likelier color, or a random draw from the chances.

50%

Toy with one number for color: 0 is orange and 1 is blue; the car is blue with the chance you set. Expected squared error is q(1−q) for the average, min(q, 1−q) for the pick, and 2q(1−q) for a draw. A guess “is a real color” when it lies within 0.15 of one of the two true colors.

At 50% the squared-error guess is a muddy color halfway between orange and blue. It has the lowest average error (0.25 against 0.50 for the others). Yet it matches no real car. Squared error is minimized by the mean of the possible answers, so it prefers a blend that is never correct over a pick that is sometimes wrong. That is why L2 fills tend to look blurry. The adversarial loss punishes a fill that looks fake, so it pushes toward one sharp, real-looking answer.

Denoising: corrupt the input, keep the clean target

The oldest recipe is the denoising autoencoder (DAE) of Vincent and colleagues (2008). Corrupt the input x by removing some values, to get x̃. Encode it, y = fφ(x̃), to get features. Decode them, z = gθ(y), to predict the clean input. Compare against the clean x, not the corrupted one.

LDAE
The loss, averaged over many clean inputs x and many random corruptions x̃ of each.
ℓ
Any measure of mismatch, such as squared error.
gθ
The decoder: it turns features back into an input-sized guess. Training updates it.
fφ
The encoder, the part other tasks can reuse. Training updates it too.
x
The clean original: the target the corruption never touched.

Here is a shape-level trace. Take an image x of 28 × 28 = 784 values. Zero out about a third of them at random to get x̃, also 784 values. The encoder maps 784 numbers to, say, 128 features. The decoder maps 128 back to 784. The loss compares those 784 outputs with the clean x. Copying the corrupted values is not enough, because a third of them are wrong. The network must lean on dependencies in the data: neighboring pixels of a stroke agree, so a missing pixel can be guessed from its neighbors.

pythondef dae_step(x, enc, dec, p=0.33):
    keep = (rand_like(x) > p)            # 1 where we keep the value
    x_tilde = x * keep                     # corrupt: some values removed
    z = dec(enc(x_tilde))                  # features, then a guess at the clean input
    loss = ((z - x) ** 2).mean()        # compare with the CLEAN x
    return loss

What did the code lose?

One more check from the lecture. A world model can decode its features back into a picture, which shows what the code kept. The lecture’s example, from Ha and Schmidhuber (2018), puts an observed frame next to a reconstruction made through the visual code. The two look alike, but fine details are gone. Its challenge is practical: pick one task-relevant detail, then propose a measurement beyond visual similarity. For the pushed block, that detail might be the contact point. A measurement could be a probe that reads it from the code, as in Chapter 1.

Here is the failure in the field. The symptom: colorized photos look washed out, with brown grass and gray skies. The measurement: compare each predicted color with the colors that really occur for that object; if the distance is large for every real color, the model has predicted a blend. The fix: change the loss so a sharp answer wins, for example an adversarial term, or let the network predict a distribution over colors and then draw from it.

A car is orange or blue with equal chance. A network trained with squared error predicts a muddy in-between color. Why?

Chapter 6

Masked Autoencoders

Hide most of the image, encode only what remains, and score the guess on the hidden part

Imagine a jigsaw puzzle with three quarters of the pieces missing. You can still tell what the picture is: a clock face, a pair of animals, a street. Your eyes use the few pieces you have to guess the whole. A network can learn to do the same, and in doing so it must learn what clocks, animals and streets look like.

This is the masked autoencoder (MAE) of He and colleagues (2022). It combines two older ideas from the last chapter: hide part of the input, and reconstruct the hidden part. Its trick is in who sees what.

Hide most patches, encode only the rest

First, cut the image into a grid of patches: equal square pieces. The lecture uses a grid of 16 patches (4 by 4) and shows 4 of them visible. The method then has three steps.

  1. Encoder. Only the visible patches enter, here 4 of 16. The encoder is the big, expensive network, so it does the least work.
  2. Decoder. It receives the visible features, plus a mask token (a placeholder vector that means “a patch is missing here”) for every hidden position, plus the position of every patch.
  3. Pixel prediction. The decoder predicts pixels for all patches. We score only the 12 masked ones.

The lecture’s note on cost is the key design choice. Mask tokens enter the lightweight decoder after the expensive encoder. Why does that matter? A token is one patch as the encoder sees it. The encoder compares every token with every other token, a cost that grows with the square of the token count. With 4 tokens that is 4 × 4 = 16 comparisons. With all 16 it would be 16 × 16 = 256. Hiding most of the image makes the encoder 16 times cheaper in that step.

As shapes, the visible tokens are a table of 4 × d numbers going into the encoder. The decoder sees a table of 16 × d: 4 real feature rows plus 12 copies of one learned mask-token row, each with its position added. It outputs a table of 16 × p, where p is the pixel count of one patch.

pythondef random_mask(tokens, ratio=0.75, rng=np.random):
    n = tokens.shape[0]                    # 16 patches
    keep = int(n * (1 - ratio))            # 4 stay visible
    perm = rng.permutation(n)
    vis, hid = np.sort(perm[:keep]), np.sort(perm[keep:])
    return tokens[vis], vis, hid               # encoder gets only tokens[vis]

Only the masked patches are scored

Which patches count in the loss? Only the hidden ones. The lecture works a one-number-per-patch example to make this clear. Four patches have target values 1, 2, 3, 4. The encoder sees patches 1 and 3. The decoder predicts all four values. The scored set is T = {2, 4}.

L
The reconstruction loss for one image.
1/|T|
Divide by the number of scored patches, to get an average.
∑k∈T
Add up over the masked patches only. Visible patches are skipped.
x̂k
The decoder’s predicted pixels for patch k.
xktarget
The true pixels of patch k, taken from the original image.

Now put numbers in. Say the decoder predicts (1.1, 1.8, 3.2, 4.5). Patches 1 and 3 are visible, so they are not scored, even though the decoder is close on them. Patch 2: (2 − 1.8)² = 0.04. Patch 4: (4 − 4.5)² = 0.25. The loss is (0.04 + 0.25) / 2 = 0.145. The visible patches supply context. The masked patches supply the scored answers. Scoring the visible ones too would let the decoder win points by copying.

How much to hide, and where

The mask ratio is the fraction of patches hidden. It is a design choice you test, not a given. The lecture notes that its picture examples use 80% masking, while the main training recipe uses 75%. It then plots two curves against the ratio: one when the whole encoder is fine-tuned on the task, and one when the encoder is frozen and a linear readout is fit. It asks where the useful range is and why the two curves differ. Its lesson in one line: a strong result after fine-tuning does not imply equally accessible frozen features.

The geometry of the mask matters too. The lecture compares a scattered random mask at 75%, one missing block at 50%, and a regular grid at 75%. The same fraction of hidden pixels can leave very different spatial evidence. A block can remove a whole object. A grid leaves a hint everywhere. Try three masks on a toy picture.

Hide patches, rebuild the picture

Left: the picture, with pink patches hidden. Right: a simple rebuild that fills each hidden patch by blending nearby visible pixels. Pick a mask style, then change how many of the 16 patches are hidden.

12 of 16

Toy: a 24 × 24 picture of a smooth background and a bright T, cut into 4 × 4 patches of 6 × 6 pixels. The rebuild is a distance-weighted blend of visible pixels, not a trained decoder, so the error numbers show how much evidence a mask leaves, not what MAE scores. The 16-patch grid and the 16-versus-256 count follow the lecture’s example.

At 12 hidden, the random mask leaves four scattered patches, and the blend rebuilds the picture with an error of about 0.18. The block style leaves the visible patches bunched in one corner, so the far side has nothing near it, and the error doubles to about 0.38. The grid style spreads the four visible patches evenly and lands near 0.20. Same count, very different evidence. What counts is not how many pixels you hide, but how much evidence remains near each hidden one.

A plausible picture is not the same picture

The lecture ends this part with a warning. Look at triplets of masked input, reconstruction and original: the reconstruction often looks right and still differs from the original in detail. Recovering a plausible scene and preserving the exact observed detail are different tests. For a world model, the second test is the one that decides whether the contact point survives.

In MAE, why do the mask tokens enter after the encoder instead of before it?

Chapter 7

Reusing the Encoder and Predicting the Next Patch

Swap the decoder for a task head, then meet image models that learn by guessing the next patch

Training is over. You hold an encoder that learned from millions of masked pictures. Nobody has asked it a real question yet. A product team arrives and says: “Tell me what is in each photo, where it is, and which pixels belong to it.” How do you put the encoder to work?

Downstream tasks reuse the encoder

A downstream task is the task we want to solve after pretraining. The recipe is short: image, then pretrained encoder, then a task head (a small new network), then the task’s prediction. The reconstruction decoder is gone. In its place stands whatever head the task needs.

TaskWhat does the head predict?
ClassificationAn image category
Object detectionObject locations and categories
SegmentationLabels or masks for image pixels

There are two ways to train the head. Frozen encoder: keep the encoder fixed and train only the head. This is the linear or attentive probing of Chapter 1. Fine-tuning: train the head and also update the encoder. Fine-tuning usually scores higher, because everything adapts. Frozen scores tell you more about the features themselves. The lecture’s rule: replace the reconstruction decoder with the task head, and evaluate the task you need.

AIM: predict the next patch

MAE hides a random subset of patches. A second family follows the idea behind language models. It reads patches in a fixed order and predicts each next patch from the ones before. AIM, from El-Nouby and colleagues (2024), is an image-only autoregressive model: it predicts its own next piece, one step after another.

Read the lecture’s figure from the bottom up. Split the image into patches and choose an order: 1, 2, 3, and so on. The input is the patches in that order. The target is the same list shifted by one position. The feature at patch i predicts patch i + 1. A small prediction head, an MLP, turns that feature into the next patch’s pixel values.

x̂i+1
The predicted pixels of the next patch.
Hθ
The prediction head: an MLP (a small stack of linear layers) with trainable settings θ.
ei
The feature the transformer made at patch i, using only patches up to i.

One rule keeps the game fair: the scored target must stay hidden from its predicting position. If patch i could see patch i + 1, it would copy it. A causal mask enforces this: each position may look only at itself and earlier positions. AIM adds one wrinkle, a prefix. The first S patches may see each other, like the context in a question. Their next-patch targets are then visible to them, so those predictions are not scored. The device shows the mask for 5 patches.

Who sees whom in AIM

Left: the scan order on a 3 × 3 image. The 5th patch visited is highlighted, with the one before it. Right: the mask for 5 tokens. Filled cells are what each output row may see. Slide the prefix S.

2

The mask rule and the S = 2 table follow the lecture: rows up to S see the whole prefix, later rows see themselves and earlier tokens, and a row’s target is scored only if it is hidden from that row. The raster and spiral orders follow the lecture’s picture; the checkerboard and random orders are our own examples.

At S = 2, rows 1 and 2 see {x1, x2}. Row 1’s target is x2, which it can see, so that row is excluded. Row 2’s target is x3, hidden, so it is scored. Rows 3 and 4 are scored too, and row 5 has nothing left to predict. That gives 3 scored rows out of 5.

The scan order is a training choice. Raster reads left to right, top to bottom. A spiral works outward from the center. A checkerboard visits alternate cells first. A fixed random order shuffles them. Ask of each: which patch comes before patch 5? The answers differ. The lecture’s reminder is plain: autoregressive order in an image is not physical time. Nothing in a photo happened first.

AIMv2: image patches and caption words together

AIMv2, from Fini and colleagues (2024, CVPR 2025), adds text. One sequence holds the image patches first and the caption tokens after them. At image positions the model predicts the next patch’s pixels, scored with squared error. At text positions it predicts the next caption token, scored with cross-entropy. The two losses add, with a weight α on the image part.

L = Ltext + α Limg

The image comes first so that text prediction can use its visual features. Both losses train the vision encoder that is used downstream. The lecture also reads an ablation. A caption-only baseline, called Cap, is compared with the version that also predicts image patches, with the encoder frozen and an attentive probe on top. The advice is the lesson of the whole chapter: compare objectives within a controlled setup before you compare headline scores.

Does reconstruction learn the features we need?

The lecture closes the family with three honest doubts.

The training target is missing pixels. The downstream target might be an object’s category or location. Reconstruction alone does not evaluate its features. So reconstruction can learn strong features, but its objective does not directly name every downstream need. That leads to the next family, which asks a different question: which views should have similar features?

In AIM with prefix S = 2, why is the prediction from the first token not scored?

Chapter 8

Contrastive Learning with SimCLR

Turn “which view belongs to which image” into a classification problem, and read its loss number by number

You are shown a photo of a tower, then two more photos: a second shot of the same tower from another angle, and a giraffe. Which one belongs with the first? You answer at once. You never needed a label. You only needed to know that two of the photos share a source.

That is contrastive learning. The training answer is not a category or a missing pixel. It is the pairing: which views belong together. SimCLR, from Chen and colleagues (2020), is the cleanest version, and Lilian Weng’s post is a good map of its relatives.

Three steps: pair, compare, prefer

The lecture states the mechanism in three beats. First, construct the pairing: sample two views (altered copies) of one example, and call them an anchor A1 and a positive A2. Second, encode and compare: map each view to an embedding (a vector that stands for it) and compute a similarity. Third, favor the positive over the negatives. A negative is a view from a different image, such as B1 or C1. We want the matched view to score higher than every mismatch.

s(A1, A2) > s(A1, B1), s(A1, C1)

Notice what is missing: no pixels are rebuilt. The loss compares embeddings, not reconstructed pictures. Here are the data-flow shapes for one view: an image goes through the backbone Eφ to a feature e, then through a projector gψ to a vector u, and the loss compares u. The lecture adds a note that is easy to miss: the loss compares u, but downstream readouts commonly use e. The projector is a small helper that lives only during pretraining.

Views must differ, but not too easily

Two crops of one image supply a positive pair. How you alter the views decides what the encoder must learn. The main SimCLR recipe combines crop-and-resize, color distortion and blur. (The lecture’s figure surveys more candidates; the main recipe does not use every one.) Weng’s summary of the paper agrees: the composition of random cropping and random color distortion is crucial.

Why color distortion? The lecture builds a small trap. Suppose you only crop. The anchor is a tower with a blue sky. Candidate 1 is another crop of the same tower. Candidate 2 is a giraffe in front of a green field. A rule that says “pick the blue candidate” gets this right without recognizing the tower. Now change each crop’s colors separately. The same tower crop turns orange, and the giraffe crop turns blue. The same color rule now picks the wrong one. Independent color changes make color alone an unreliable clue, so the network has to look at shapes.

A caution from the lecture’s own question: which transformation would harm an orientation-sensitive or color-sensitive task? If you must read the block’s angle, a rotation augmentation tells the encoder to throw the angle away. The views you choose are the facts you ask the encoder to ignore.

Each row is a five-way classification

Take three source images A, B and C. Two views each give six views: A1, A2, B1, B2, C1, C2. Anchor A1 now chooses among five candidates: A2, B1, B2, C1, C2. A1 itself is excluded. Exactly one candidate, A2, is the positive. Four are negatives.

Lay the six views along both sides of a 6 × 6 table. The diagonal is the self-pair, crossed out. Each row has one positive, from the same source, and four negatives. Both A1 to A2 and A2 to A1 count as training terms, so the table holds six five-way problems. In general, N images give 2N views, and each anchor has 2(N − 1) negatives, as Weng writes.

From similarities to a loss

The cosine similarity of two vectors is their dot product after each is scaled to length 1. It runs from −1 (opposite directions) through 0 (unrelated) to 1 (the same direction). We turn a row of similarities into probabilities with a softmax (the step that makes scores positive and sum to 1, as in a language model) and then take the negative log of the paired view’s probability.

ℓi,j
The loss for anchor i whose positive is view j. Small when the positive gets most of the probability.
pij
The probability the model gives the paired view j, among all candidates.
exp(sij / τ)
A positive weight for the pair, where sij = cos(ui, uj) is the similarity and τ > 0 sets the scale. τ is the temperature.
∑k≠i
Add the weights of every candidate except the anchor itself. The sum includes the positive j.

This is a classification task whose correct class comes from how we sampled the views. (The SimCLR paper names this loss NT-Xent, a normalized temperature-scaled cross-entropy.) Now work one row by hand, using the lecture’s example. The positive has similarity 1 and the four negatives have similarity 0, at temperature τ = 1. The positive’s weight is e1 = 2.718. Each negative’s weight is e0 = 1. The sum is 2.718 + 4 = 6.718. So p+ = 2.718 / 6.718 = 0.405, each negative gets 1 / 6.718 = 0.149, and the loss is −ln 0.405 = 0.905. The device runs this exact row. Move the knobs.

One row of the contrastive loss

One anchor, one positive, four negatives. Bars show each candidate’s probability. The number under each bar is its gradient: the push on its similarity. Start at the lecture’s setting, then lower the temperature and raise the similar negative.

1.00
1.00
0.00

The lecture’s fixed example: positive similarity 1, four negatives at 0, τ = 1, giving probabilities 0.405 and 0.149 and a loss of 0.905. The sliders move the positive and negative 1 only; the other three negatives stay at 0. The gradient is the lecture’s formula, (pk − 1[k = +]) / τ; a negative number means the similarity is pushed up, a positive number means it is pushed down.

Read the gradient row. The positive carries p+ − 1 = −0.595 (at τ = 1), a push up. Each negative carries its own probability, 0.149, a push down. Higher-probability negatives get a stronger push, which is why a negative that looks too much like the anchor is pushed hardest. Now set τ to 0.1, the positive’s similarity to 0.6 and the similar negative to 0.9. That negative takes about 95% of the row and the loss jumps past 3. Lower τ sharpens whatever the current preference is, even when it is wrong. These gradients flow back through the projector and the encoder, which is how the features change.

Here is a second row with made-up scores, to practice. At τ = 0.5, say the positive scores 0.9 and the negatives score 0.1, 0.2, −0.1 and 0.0. The weights are e1.8 = 6.05, e0.2 = 1.22, e0.4 = 1.49, e−0.2 = 0.82 and e0 = 1.00. Their sum is 10.58. So p+ = 0.572 and the loss is 0.559.

pythonfrom scipy.special import logsumexp
def nt_xent(u, tau=0.5):               # u: (2N, d); rows 2k and 2k+1 are two views of image k
    u = u / np.linalg.norm(u, axis=1, keepdims=True)
    s = (u @ u.T) / tau                  # (2N, 2N) scaled cosine scores
    np.fill_diagonal(s, -np.inf)          # a view never competes with itself
    pos = np.arange(len(u)) ^ 1            # partner of row i: 0<->1, 2<->3, ...
    logp = s - logsumexp(s, axis=1, keepdims=True)
    return -logp[np.arange(len(u)), pos].mean()
At τ = 1, one positive has similarity 1 and four negatives have similarity 0. What is the positive’s probability, and why?

Chapter 9

Negatives, CLIP and SigLIP

See why the loss needs a push as well as a pull, what goes wrong with false negatives, and how two big systems cope

Suppose a lazy student gets this rule: “Make two views of the same picture look alike.” The student finds a perfect answer in one second. Ignore the picture and write the same word on every page. All views look alike. The score is perfect, and the student has learned nothing.

A network can do the same. This chapter is about the part of the loss that stops it: the negatives. It also covers what goes wrong with them, what they cost at scale, and how CLIP and SigLIP, the two big image-and-text systems in the lecture, use them.

Pulling is not enough: collapse

Take the simplest loss that agrees with “two views should match”. It measures how far apart the two views’ vectors are: ℓpair = ∥u(1) − u(2)∥2. The encoder can reach ℓ = 0 by sending every input to the same vector c. That failure is called collapse. The loss is perfect and the features say nothing, because every picture gets the same description.

Now ask the positive to beat four negatives instead. If the encoder collapses, all five candidates tie. Each gets probability 1/5, and the loss is −log (1/5) = log 5 = 1.61. Pushing the negatives away from the anchor, so that the positive stands out, lowers the loss. So the negatives give the loss a reason to keep different pictures apart. The device trains a tiny encoder both ways, so you can see collapse happen.

Watch an encoder collapse

Eight pictures, each a colored point in a plane. A tiny encoder (a 2 × 2 matrix plus an offset) places them. Choose the loss, then press Train. The line below traces how far apart the eight points sit.

Toy, trained live in your browser with finite-difference gradients: 8 inputs spread around a circle, 2 noisy views each, embedding u = W x + b, 300 steps at rate 0.3. “Pull only” is the squared distance between the two views’ vectors. “Pull and push” is the contrastive loss of Chapter 8 at τ = 0.5, on cosine similarity. The spread is the average distance between the 8 clean points. Not a measured result from the course.

In the pull-only run the points slide together until the spread reaches 0.01. Eight different pictures now have one description. In the push-and-pull run the spread stays near 1.5, the loss settles at about 1.5, and the pictures stay apart. The lecture puts it in one line: simply deleting the negatives is insufficient. It also says that learning without negatives is possible, because there are other ways to prevent collapse (Thursday’s subject, with BYOL as the famous example).

Where negatives come from, and a false negative

In SimCLR the negatives are the other pictures in the same batch, the group of examples one training step handles together. That is cheap, but it rests on an assumption: “a different image ID means a different thing.” The lecture breaks that assumption with a robot example. A batch holds two nearby frames of one video. The anchor is frame A. The positive is another crop of A. A negative is frame B, a moment later, where the T-block is in nearly the same place.

The training rule says: lower A’s similarity to B, because B has a different image ID. But what we wanted was the opposite. Similar object configurations should stay easy to recognize as similar. A false negative is a pair that the sampler separates even though the pair shares information we want to keep. More negatives mean more comparisons, but also more false negatives, more score computation and more communication. Weng’s post meets the same problem under the name of sampling bias, and describes debiased and hard-negative-weighted losses as fixes.

How 100 GPUs share their negatives

More negatives usually help. The lecture shows how a big run gets them. Take 100 GPUs, 256 images on each, and 2 views of each image: 100 × 256 × 2 = 51,200 vectors. Each GPU holds 256 × 2 = 512 of them.

  1. All-gather. The GPUs exchange vectors, so each one now has all 51,200.
  2. Score. Each GPU compares its 512 local anchors with all 51,200 vectors: 512 × 51,200 = 26,214,400 similarities. Other GPUs compute their own rows.
  3. Pick the match. For each anchor, exclude itself and keep 1 positive. The other 51,198 candidates are negatives.
  4. Backpropagate and synchronize. Send gradients back through the gathered vectors, then average the model gradients across GPUs.

The exchange carries feature vectors, not images. For a sense of size, suppose each vector has 128 numbers stored in 2 bytes each (an assumption, not the lecture’s figure). Each GPU receives 50,688 vectors from its peers, about 13 megabytes per step. That is the “communication” cost in the lecture’s list.

CLIP: the caption is the positive

CLIP (Radford and colleagues, 2021) applies the same idea to two kinds of data. The positive is an observed image and its caption. The other captions in the batch act as negatives. Two encoders, one for images and one for text, form an image-by-text similarity matrix. A softmax along each row picks the right caption for an image. A softmax along each column picks the right image for a caption. Weng describes it as maximizing the N correct pairs and minimizing the N(N − 1) incorrect pairs of a batch, and notes the 400 million image and text pairs it was trained on.

After training you can classify without class labels. Write each class as a prompt, such as “a photo of a dog”, and pick the prompt closest to the image. The lecture’s caution for a world model: text supplies semantic supervision at scale, but captions may omit the geometry or motion that a world model needs. A caption says “a T on a table” and not “the stem is 12 degrees clockwise”.

SigLIP: each pair is its own yes or no

CLIP’s softmax compares one caption with all competing captions in its row. That needs the whole batch at once. SigLIP (Zhai and colleagues, 2023) changes the loss: score every image and text pair on its own, as a yes-or-no question.

ℓij
The loss for image i and text j. One term per pair.
yij
The label: +1 for an observed pair, −1 for every other pair.
sij
The cosine similarity of the image vector and the text vector.
a, b
Two learned numbers, with a > 0: a sets the scale and b the offset. σ is the sigmoid, which squeezes any number into 0 to 1.

For an image of a tower beside a river, the caption “A tower beside a river” has label +1, so the loss encourages a higher score. The caption “A pair of giraffes” has label −1, so the loss encourages a lower score. There is no batch-wide softmax denominator, so pair losses can be computed and added up block by block. The lecture keeps the honesty: SigLIP still uses negative pairs. Cross-device negatives still need communication, and false negatives remain.

A loss only asks the two views of each image to have the same embedding. What can the encoder do to score perfectly?

Chapter 10

Field Guide

Keep the numbers, the checklist and the core code, then carry the open question into Lecture 6

You can now reproduce the first half of the week from memory. A representation is a description of an observation in a chosen form. A probe tests it with a tiny readout on a frozen encoder, and a probe score belongs to the pair of encoder and probe. Good features keep what the later question needs, and what that is depends on the question. Self-supervised learning lets the data make its own target. Reconstruction hides or alters the input and recovers it: rotation, jigsaw, colors, holes, noise, masked patches, next patches. Contrastive learning asks which views belong together and needs negatives to avoid collapse. Each method makes the features keep different facts.

The numbers that matter

WhatNumberWhy it matters
Linear probey = W e + b; train only W and bThe encoder stays frozen, so the score tests the features
Mean poolingweight 1/K per patchDilutes one useful patch among K
Relative position8 offsets, chance 12.5%The sampler knows the answer
Rotation4 angles, chance 25%Forces orientation cues into the features
Jigsaw, gap or no gap98% puzzle, 47.7 mAP vs 88% puzzle, 52.6 mAPA harder puzzle transferred better
MAE patches4 of 16 visible; 75% in the main recipe, 80% in the examplesEncoder cost falls with the visible count
MAE lossmean over masked patches onlyVisible patches would allow copying
AIM prefix S = 23 of 5 rows scoredA target must be hidden from its predicting position
AIMv2 lossL = Ltext + α LimgOne sequence trains one encoder on two targets
SimCLR toy batch3 images, 6 views, 5 candidates per anchorEach row is a classification problem
One loss row at τ = 1p+ = 0.405, loss 0.905Gradient (pk − 1[k = +]) / τ
All candidates tiedp+ = 1/5, loss log 5 = 1.61What collapse costs once negatives exist
Big-batch negatives100 × 256 × 2 = 51,200 vectors; 51,198 negatives per anchorNegatives come from the batch, shared by all-gather

The four questions, answered for each method

The lecture ends its map with one habit: for any method, name the visible input, the target, the loss and the reused features. Here is the table, filled in.

MethodVisible inputTargetLossReused
RotNetTurned imageThe angleCross-entropy, 4 classesEncoder
JigsawShuffled tilesThe permutationCross-entropyEncoder
Denoising autoencoderCorrupted inputClean inputReconstruction errorEncoder
MAEVisible patchesPixels of masked patchesSquared error on maskedEncoder
AIMEarlier patchesNext patch pixelsSquared errorEncoder
SimCLRTwo cropsWhich view matchesSoftmax over candidatesBackbone feature e
CLIP, SigLIPImages and captionsWhich caption matchesSoftmax or sigmoidThe encoders

A checklist for any representation

  1. Name the question. What will the features be used for: position, contact, motion, identity, the next frame?
  2. Name the shortcut. What cheap clue could solve the pretext task without learning what you want?
  3. Check what the views ignore. Every augmentation tells the encoder to throw something away.
  4. Probe fairly. Same labels, same features, same probe type, same test scenes. Say which probe.
  5. Test frozen and fine-tuned. A strong fine-tuned score does not imply easy-to-read frozen features.
  6. Decode and look. A plausible reconstruction is not the observed image.
  7. Count the cost. Encoder tokens, negatives, and the bytes the chips exchange.

The core as code

pythonimport numpy as np
from scipy.special import logsumexp

# 1. A linear probe on frozen features e: (n, d) -> angle labels as (cos, sin)
def fit_probe(e, angle, lam=1e-3):
    y = np.stack([np.cos(angle), np.sin(angle)], 1)              # (n, 2)
    X = np.hstack([e, np.ones((len(e), 1))])                    # add the bias column
    return np.linalg.solve(X.T @ X + lam * np.eye(X.shape[1]), X.T @ y)  # (d+1, 2): W and b

# 2. The MAE loss: score only the masked patches
def mae_loss(pred, target, masked):                          # pred, target: (16, p); masked: indices
    return ((pred[masked] - target[masked]) ** 2).mean()

# 3. The SimCLR loss (NT-Xent): u is (2N, d), rows 2k and 2k+1 are views of image k
def nt_xent(u, tau=0.5):
    u = u / np.linalg.norm(u, axis=1, keepdims=True)
    s = (u @ u.T) / tau
    np.fill_diagonal(s, -np.inf)
    pos = np.arange(len(u)) ^ 1
    logp = s - logsumexp(s, axis=1, keepdims=True)
    return -logp[np.arange(len(u)), pos].mean()

# 4. The SigLIP loss: every image-text pair is its own yes or no
def sigmoid_loss(zi, zt, a, b):                             # zi, zt: (n, d), unit length; a > 0
    s = zi @ zt.T                                          # (n, n) cosine scores
    y = 2 * np.eye(len(zi)) - 1                          # +1 on the diagonal, -1 elsewhere
    return np.logaddexp(0, -y * (a * s + b)).mean()   # -log sigmoid(z) = log(1 + e^-z)

What these targets miss, and Thursday’s question

The lecture closes by asking what each family leaves out, using the world-model test: does the target expose position, contact and motion?

ReconstructionContrastive learning
The training answerOriginal content or structure (DAE, MAE, AIM, AIMv2)Which examples should match (SimCLR, CLIP, SigLIP)
What can go wrongEasy clues solve the task; pixel targets reward irrelevant detail or have several plausible answersPositive views can erase useful differences; false negatives separate similar cases; large pools cost computation and communication
The world-model testDoes recovering the target expose position, contact and motion?Do the pair labels preserve the differences later predictions need?

Thursday’s lecture, Lecture 6, turns to self-distillation and prediction in feature space. DINO matches a teacher’s outputs. data2vec predicts contextual features. The JEPA family predicts features of hidden image regions or video. The question it inherits: if the targets are learned too, how do we keep them informative without negatives?

Keep going

The takeaway. A representation is a promise about which questions will be easy. Reconstruction promises the features can rebuild the input. Contrast promises matching views stay close and other things stay apart. Neither promise names position, contact or motion, so test the promise you need with a fair probe.

Credit: this lesson re-teaches Lecture 5 of CIS 6280 World Models (University of Pennsylvania, Fall 2026), taught by Jiatao Gu, with Lilian Weng’s post on contrastive representation learning as the reading. Read the course’s own slides and the course site. The explanations, devices and toy numbers here are ours, and each toy is labelled as one.

Which statement matches the lecture’s test for a self-supervised method on a world-model problem?

Now press Present or Teach and explain one training method out loud, from memory: what is visible, what the target is, who makes it, and what the encoder must keep to score well. Then go back to the two rings at the top, set the ring blur to 0.6, and say why polar coordinates still label about 96% right.

CIS 6280 Lecture 5: Representation Learning I
The course site · Lecture slides · Contrastive Representation Learning (Weng) · Back to Gleams