Representation Learning II

Ask a network to make two views of one picture agree, and it finds a cheat: give every picture the same feature, and the loss reads zero. Train the toy below, then stop the cheat three ways.

Learn how a model can match views and predict learned features instead of pixels, why that tends to collapse, and what keeps it from collapsing: negatives, a slow teacher, and explicit rules on the spread of features.

Pick Views only and press Train. Watch sixteen features melt into one point. Then pick another rule and train again. After that we build every piece from zero: contrastive scores, DINO, JEPA, VICReg and LeJEPA.

You need the idea of a vector (a short list of numbers) and a feel for probabilities (numbers from 0 to 1 that add up to 1). We build the rest from zero.

Based on Lecture 6 of Penn CIS 6280: World Models (Fall 2026), taught by Jiatao Gu, and its reading, Rohit Bandaru’s Deep Dive into Yann LeCun’s JEPA. Lecture slides. Every idea is re-taught here in our own words and devices; it follows Lecture 5, Representation Learning I.

Train an encoder to match views

Toy model, not the lecture’s: 16 inputs of 4 numbers each, a linear encoder of 8 weights that makes 2 features, and two views made by adding fresh noise (0.3 spread) to each input. “Spread” is the average standard deviation of the two features across the 16 images. The negatives rule is the InfoNCE score of Chapter 1 at temperature 0.2; the floor rule is the variance and covariance penalty of Chapter 8 with floor 1. Gradients are computed numerically, in your browser, from the same seeded start every time.

Chapter 0

Why Pixels Are a Costly Answer

See what a reconstruction loss rewards, why it can overvalue a wall and undervalue a fingertip, and why that opens the door to other targets

A robot arm pushes a block across a striped table. The camera sends one small image: 9,100 pixels. Of these, 9,000 show the table. Only 100 show the spot where the gripper touches the block. That small spot decides whether the push succeeds.

Now train a network to rebuild this image from a damaged copy. Which pixels will it care about most? The answer is the first lesson of today, and it explains why the rest of the lecture exists.

Where Lecture 5 stopped

Last lecture gave us a vocabulary. A representation is another way to write down an observation: a vector z = f(o) that an encoder f computes from the observation o, such as a camera image. Training rewards the encoder for keeping some information and lets it drop the rest. To test the result, we freeze the encoder and train a small readout (also called a probe) on top of it. If a tiny readout can answer a question from the frozen features, the features hold the answer.

Lecture 5 ended with the masked autoencoder (MAE): hide patches of an image, then predict the hidden pixels. Today starts one step later, with a close cousin called AIM. Then it asks the bigger question in the lecture’s first sentence: what if the target is something other than pixels? One short note on the course itself: this is Lecture 6 of CIS 6280 at Penn, taught by Jiatao Gu on Thursday, 10 September 2026. The slide deck is linked in the fine print above.

Reconstruction in one line

Every reconstruction method fits one pattern. Damage the input, encode what is left, decode a guess, and compare the guess with the original. The lecture writes this as one line.

T
Corrupt the input: hide patches, or add noise. This is the damage.
E
The encoder. It turns the damaged input into features. It is the part we keep.
D
The decoder. It rebuilds the original data from those features. We throw it away after training.
ℓ
The loss that compares the rebuilt image with the original x. For MAE it counts error only at the hidden patches.
Lrec
The number training pushes down. To push it down, the encoder must learn features that help recover what T removed.

AIM: predict the next patch

MAE hides random patches. AIM (an autoregressive image model) hides the future instead. It cuts an image into patches (small squares) and lines them up like words in a sentence. Then it plays the language-model game: read the patches so far, predict the next one. The loss sums the squared pixel error over the patches it scores.

Look at who sees whom. Take five patches and a prefix of two, written S = 2. The first two patches see each other. Every later patch sees only itself and the patches before it. So the output at patch 2 predicts patch 3 without ever looking at patch 3. The targets inside the prefix are never scored, because the prefix patches already see each other, and that would be cheating.

MAEAIM
Visible contextRandomly kept patchesThe prefix and earlier patches
Prediction targetThe masked patchesThe next patch in a chosen scan order
What is scoredOriginal pixel valuesOriginal pixel values
TransferReuse the encoderReuse the encoder

The lecture’s summary: the target is similar, but the evidence available to the model differs. Both can learn useful context. Neither one ever names “contact”, “pose” or “velocity”. Those words matter to a robot, and the loss does not know them.

The weight of a wall

Here is why that matters. The loss for pixels is mean squared error (MSE): for each pixel, take the gap between the guess and the truth, square it, and add up. Squaring means a big gap costs far more than a small one. It also means every pixel gets a vote, and a pixel is a pixel whether it shows a wall or a fingertip.

Now work the table scene. Suppose the guess is off by the same amount e at every pixel. Each pixel adds e². The 9,000 table pixels add 9,000 e². The 100 contact pixels add 100 e². The table pays 90 times more into the loss. A network with limited capacity learns to cut the biggest cost first: a smooth, predictable stripe texture. The small cue that decides the task is easy to leave blurry.

Play with it. Set an error for the table and an error for the contact spot, and watch where the loss comes from.

Weigh the pixels

Two bars show how much each region adds to the loss. Drag the table error and the contact error. Press Same error everywhere to see the 90 to 1 ratio.

0.20
0.50

The pixel counts (9,000 background, 100 contact) are the lecture’s. The errors are illustrative numbers on a scale from 0 to 1. Each region adds (number of pixels) × (error)² to the loss, which is the lecture’s sum of squared gaps without the 1/P average.

With the defaults, the table adds 360 to the loss and the contact spot adds 25. The wall owns 93.5% of the cost, even though each contact pixel is off by 2.5 times as much. That is a mismatch between weights and task value. It is not a rule that MSE always ignores objects. A network with enough capacity fixes both. But capacity is limited, and the cheapest way to lower the loss is not always the way that serves the task.

When two answers are both right

There is a second catch. Suppose a block sits either on the left or on the right of a box, 50% each, and the hidden part of the image shows it. What single picture should the network output?

Under squared error the best deterministic answer is the conditional mean, written E[x | c]: the average of all outcomes that fit the visible context c. Here the mean is a block in the middle, and the middle is neither answer. Work it with positions −1 and +1. Guess 0: the error is 1 either way, so the expected squared error is 1. Guess −1: the error is 0 half the time and 4 the other half, so the expected squared error is 2. The blurry middle wins the score and never happens.

Left or right, one guess

The block is at −1 or +1, half the time each. Drag your single guess and read the expected squared error.

−1.00

Toy numbers (positions −1 and +1, probability 0.5 each). The expected squared error of a guess g is 0.5(g + 1)² + 0.5(g − 1)² = g² + 1.

A generative target avoids the trap. Instead of one answer, the model learns a whole distribution, p(xT | xC), the chance of every hidden part xT given the visible part xC. A distribution can give 50% to the left and 50% to the right. DiffMAE does this with denoising: visible patches guide a sequence of steps that turn noise into a picture, and one sampled run picks one coherent completion. (The course covers diffusion in later lectures.) Its encoder also scores well on recognition tests. The lecture adds a sober line: realism alone does not certify features.

So there are two problems with pixels: the loss may weigh the wrong things, and a single answer averages away real uncertainty. The next chapters change the question. Instead of “what are the missing pixels?” we ask “which views belong together?” and, later, “what would a good feature of the missing part look like?”

  1. Which views belong together: positive pairs, scores and the InfoNCE loss.
  2. When negatives mislead: false negatives, CLIP and SigLIP.
  3. The JEPA blueprint: predict in feature space, with a latent z.
  4. DINO: a student chases its teacher: output slots, cross-entropy and gradients.
  5. DINO: how it avoids collapse: centering, sharpening, EMA.
  6. BYOL, data2vec and I-JEPA: continuous targets and mask design.
  7. Video: V-JEPA: masked completion versus forecasting.
  8. Rules on the spread of features: VICReg and what moments miss.
  9. SIGReg and LeJEPA: a Gaussian check by projection.
  10. Three families and JEPA variants: how to judge a method.
  11. Field guide: numbers, checklist, code and links onward.
At equal error per pixel, 9,000 table pixels and 100 contact pixels share a squared-error loss. What follows?

Chapter 1

Which Views Belong Together

Learn features by picking the matching view out of a crowd, and see why that loss needs other images to push against

You hold two photos of the same tall clock tower. One is taken from the bridge, one from the river bank. The colours differ, the angle differs, a bus blocks half of one. You still know they show the same tower. Now someone hands you a third photo of a giraffe. You know that one is different.

Notice what you did. You never rebuilt a single pixel. You judged a relationship between pictures: these two belong together, that one does not. The second family of methods in this lecture trains a network on exactly this skill.

Views, embeddings and a score

A view is one version of an example: a crop, a colour shift, a blur. Two views of the same photo form a positive pair. A view from a different photo is a negative for that pair. The network turns each view into an embedding, a vector of numbers that stands for the view. Then it scores how alike two embeddings are.

The usual score is a dot product after normalising each vector to length 1. For unit vectors the dot product is the cosine of the angle between them. It equals 1 when the vectors point the same way, 0 when they are at a right angle, and −1 when they point opposite ways. High score means “these two look alike to the network”.

The lecture states the whole idea in three steps. First, construct the pairing: sample two views of one example. Second, encode and compare: map each view to an embedding and compute a score. Third, favour the positive: train so that the matched view scores higher than the others. For an anchor A1, its partner A2 and two outsiders B1 and C1, the wish is s(A1, A2) > s(A1, B1) and s(A1, C1). The loss compares embeddings, never reconstructed pixels.

One concrete batch

Take three images, A, B and C, and make two views of each. That gives six views: A1, A2, B1, B2, C1, C2. Choose A1 as the anchor, the view we are scoring. Cross out the anchor itself, because it matches itself trivially. What remains is five candidates: A2 is the one positive, and the other four (B1, B2, C1, C2) are negatives. With N images there are 2N views, so every anchor faces one positive and 2N − 2 negatives.

InfoNCE, line by line

The loss that does this is called InfoNCE. It is a softmax, the same function that turned scores into next-word probabilities in language models. Think of a quiz with 2N − 1 answer choices. The anchor must pick its partner. The loss is the surprise at the correct choice.

ui
The unit-length embedding of view i. The encoder f makes a feature e; a small projector g maps it to u for the loss. Later tasks usually reuse e, not u.
sik
The score of view i against view k: a dot product, from −1 to 1.
τ
The temperature. It divides every score before the softmax, so it sets how sharply the scores are compared.
pij
The probability that candidate j is the partner of anchor i. The sum runs over every view except the anchor, and the positive is inside it.
ℓi
The loss for anchor i: minus the log of the probability given to the true partner j(i), which the sampler chose.

Work the lecture’s example. The positive scores 1 and each of the four negatives scores 0. At τ = 1 the exponentials are e1 = 2.718 for the positive and 1 for each negative. The total is 2.718 + 4 = 6.718, so the positive gets 2.718 / 6.718 = 40.5% and each negative gets 14.9%. The loss is −ln 0.405 = 0.904.

Now cool the temperature to τ = 0.2. The scores divided by τ become 5 and 0. The positive’s exponential is e5 = 148.4, the total is 152.4, and the positive gets 97.4%. The loss drops to 0.026. Same scores, very different probabilities. A small τ magnifies score gaps, so the best candidate takes almost everything. Try it.

Pick the partner among five

One anchor, one positive, four negatives. Drag the temperature and the two scores. Then press Make every score equal to see what a collapsed encoder gets.

1.00
1.00
0.00

The values 1 and 0 and the two temperatures (1.0 and 0.2) are the lecture’s example for one anchor with four negatives; the 40% and 97.4% are its numbers. The softmax and the loss are computed exactly, live in your browser.

What the gradient does

The lecture also gives the slope of the loss with respect to each score: ∂ℓ / ∂sk = (pk − 1[k = +]) / τ. The symbol 1[k = +] is 1 for the positive and 0 for the others. Read it as a rule. For the positive, p − 1 is negative, so gradient descent raises its score. For each negative, the slope is pk / τ, which is positive, so descent lowers that score, and it lowers most the negatives that currently look most like the anchor. At τ = 1 the first step pulls the positive with weight 0.595 and pushes each negative with 0.149.

The cheat, and the number that exposes it

Why do we need negatives at all? Try the simplest loss: make the two views of one image agree, L+ = ‖f(v1) − f(v2)‖². A network can drive this to exactly zero with a trick: ignore the input and output the same constant c for every image and crop. Every positive pair agrees perfectly. And every image looks identical in feature space. This is representation collapse, and it is what you watched in the hero at the top of the page.

The symptom is a loss near zero and features that cannot tell images apart. The measurement is the spread of features across a batch: it falls to zero. InfoNCE gives a second measurement. If an encoder gives every view the same embedding, all scores tie, each of the 2N − 1 candidates gets probability 1 / (2N − 1), and the loss is log(2N − 1). For our three-image batch that is ln 5 = 1.609, and it cannot go lower until the encoder separates the images. The fix is the negatives themselves: they turn a constant output into a loss the network cannot ignore.

An encoder outputs the same unit vector for every view in a batch of 3 images (6 views). What is the InfoNCE loss for each anchor?

Chapter 2

When Negatives Mislead, and Words Join the Pair

Count the negatives that secretly share a class, see how big batches are split across chips, and compare CLIP’s softmax with SigLIP’s independent votes

Picture a photo app that trains on pet pictures with one rule: two crops of the same photo are a positive pair, and anything else is a negative. The anchor is a crop of photo A, a black-and-white cat. Photo B is a ginger cat. Photo C is a dog. By the rule, the ginger cat and the dog are both negatives. Training pushes the anchor away from both.

The task we care about later is “cat or dog?” The rule has just told the network to separate two cats. Hold that thought. It is the main weakness of the whole contrastive family, and the lecture names it.

False negatives

A false negative is a negative that shares the very class (or meaning) we want the features to recognise. The training rule cannot tell: it only knows which photo each view came from. The more images share a class, the more of the pushing is wrong.

How much? Take a batch of N images and C equally common classes. The anchor’s own other view is the positive. The other 2N − 2 views come from N − 1 other images, and about 1 in C of them are in the anchor’s class. The chance of each is small, but the count adds up when batches are large. Slide the numbers.

Count the false negatives

Each square is one negative view. Red squares share the anchor’s class. Drag the batch size and the number of classes.

256
10

Illustrative arithmetic, not a measurement: it assumes every class is equally common and every image is drawn independently. The expected count is 2(N − 1) / C negative views in the anchor’s class. The lecture states the failure (another cat is a negative under this rule) without giving numbers.

With 256 images and 10 classes, about 51 of the 510 negatives are the same class as the anchor: 10%. Push them all away and the class stops being a direction in feature space. Reference work on debiased contrastive learning studies the fix; the lecture only flags the problem.

Splitting a batch across chips

More negatives usually help, so people want batches of thousands. One chip cannot hold them. The lecture shows how several GPUs share the job. Each GPU keeps its own images and encodes them locally. Then the GPUs run an all-gather: every GPU receives every other GPU’s feature vectors (not the images). Each GPU now computes the loss only for its own anchors, scoring them against all the gathered candidates. The data flow has this shape: with G GPUs and B images per GPU, each GPU holds a B × d block of features and gathers a (G · B) × d table.

The backward pass reverses it. Each feature receives gradients from the loss on every GPU that used it. Those gradients are summed at the GPU that owns the feature, backpropagated into that GPU’s encoder, and then ordinary data-parallel training (DDP) averages the parameter gradients so every copy of the model stays the same. The lecture’s summary: share candidates, split the loss, combine gradients for one model update. The optional reading, MoCo (momentum contrast), takes a different route to more negatives: a queue of recent features, kept consistent by a slowly updated key encoder. Its versions add a stronger recipe (v2) and vision transformers with an online predictor (v3). MoCo stays contrastive: negatives still do the separating.

CLIP: the pair is an image and its caption

Now change what a “view” is. CLIP trains two encoders at once, one for images and one for text. A positive pair is an image and its own caption. Every other caption in the batch is a negative. Both encoders receive gradients. The lecture notes that language changes what is shared: captions stress some visual properties and skip others.

The payoff is a readout that needs no extra training. For zero-shot classification, write one sentence per class, such as “a photo of a giraffe” and “a photo of a boat”. Encode the sentences and the image. The answer is the sentence with the highest similarity: k̂ = arg maxk sim(fimage(x), ftext(tk)). The candidate descriptions are the decision rule. A useful readout, the lecture says, is concrete: inputs, candidate labels and a score.

SigLIP: a separate vote for every pair

CLIP scores one image against all captions together with a softmax, so the probabilities must add to 1. SigLIP asks a different question for each image and caption pair: do these two match, yes or no? It uses a sigmoid, a function that squeezes any score into a number between 0 and 1, so each pair gets its own independent answer. The loss for pair (i, j) is ℓij = −log σ(yij sij), where y is +1 for a matching pair and −1 for a mismatch.

Take the lecture’s giraffe photo. The caption “a photo of a giraffe” is a match and scores 0.88. The caption “a photo of a boat” is a mismatch and scores 0.27. These do not have to add to 1. The loss on the match is −ln 0.88 = 0.128. The loss on the mismatch asks for a low score, so it is −ln(1 − 0.27) = 0.315. Each pair is judged alone.

Softmax or independent votes

One giraffe photo, two captions. Drag the two raw scores, then switch the rule. Watch what happens to one caption when only the other score moves.

2.0
−1.0

The 0.88 and 0.27 are the lecture’s illustrative sigmoid scores; they equal σ(2.0) and σ(−1.0), so the sliders start there. The CLIP column applies a softmax to the same two scores. The photo is a giraffe, so the giraffe caption is the match.

What contrastive learning buys, and what it costs

Contrastive learningThe limit that remains
TargetRelations between views, not pixelsThe chosen views decide which information is shared
Feature emphasisNo need to reproduce every pixelColour, pose or location cues can be erased by augmentation
Collapse controlPositive against negativesFalse negatives, and large candidate sets

Read the middle row twice. If an augmentation always changes colour, two views of one photo disagree about colour, so the loss teaches the network to ignore colour. That is fine for “cat or dog” and fatal for a robot that must tell a red cup from a blue one. The lecture ends this part with the cheat from Chapter 1: positive agreement alone does not rule out f(v) = c. The next chapters ask for a way to keep features alive without negatives.

Training rule: only crops of the same photo form a positive pair. Why can a second photo of a cat hurt a cat-versus-dog readout?

Chapter 3

The JEPA Blueprint

Predict a learned feature of the missing part, let a hidden choice z cover the branches, and see why z must not carry too much

A car drives toward a fork in the road. Next second it will go left or right, and nothing in the picture so far says which. You want a model that watches many such cars and learns what can happen. If you ask for the next picture pixel by pixel, you get Chapter 0’s blurry middle: half a car on each road.

Yann LeCun’s answer, in his 2022 position paper A Path Towards Autonomous Machine Intelligence, has three parts. Predict a summary of the future, not its pixels. Give the predictor a hidden switch for the choices the past cannot decide. And add something that stops the whole scheme from collapsing. The lecture’s third family of methods is built on this idea. The lecture also recommends Rohit Bandaru’s Deep Dive into Yann LeCun’s JEPA, and we draw on it here.

The third family, and its one question

The reconstruction family used the original data as the answer. The contrastive family used a relationship between views. The third family, learned targets, uses a representation of another view, produced by a network that is itself learning. The target moves as training goes on. This raises the same trap as before. If the target is just “whatever the other branch outputs”, then matching it is trivial when every input gets one output. So the lecture asks the same pair of questions of every method that follows: what is the target, and what prevents a constant representation?

It sorts the answers into two kinds. Self-distillation uses teacher targets and unequal updates between branches (DINO, BYOL, I-JEPA). Distribution regularisation puts explicit rules on how features vary (VICReg, SIGReg, LeJEPA). The ingredients can be mixed. Each method needs both a useful matching task and a way to resist collapse.

One picture for the whole family

The lecture then draws one umbrella over them, the joint-embedding predictive architecture (JEPA): encode both observations, then match or predict one representation from the other. DINO and BYOL match or predict outputs across two views. data2vec, I-JEPA and V-JEPA predict target features from partial context. LeJEPA matches views and adds a rule on the spread of features. Direct matching is the simple case. A separate predictor and a latent variable are optional extras. Treat “JEPA” as one way of looking at these methods, not as one recipe.

The original blueprint, symbol by symbol

Now the original drawing from LeCun’s paper. There are two related observations: x (the past, say) and y (what follows). Each goes through an encoder, giving representations sx and sy. A predictor tries to guess sy from sx. The distance between the guess and the real sy is the energy: low when x and y are compatible, high when they are not.

sx
The encoded context, for example the car approaching the fork.
sy
The encoded target, produced by the second encoder from the real y.
z
A latent variable: a choice among the possibilities that x alone cannot settle. Different z values stand for turning left or turning right.
s̃y
The predicted representation of y. Sweep z over its allowed set Z and you get a whole region of predictions.
E
The energy: the distance D between the real and the predicted representation. The lecture writes it for one z; the reading takes the lowest over all z.

Work a one-number toy. Say the fork means the predictor, from sx alone, outputs 0. Turning left has sy = −1 and turning right has sy = +1. With no z, the energy is (−1 − 0)² = 1 for either turn, and the predictor sits at the blurry average again. Now let z shift the guess by any amount up to ±1. For left, pick z = −1: the guess is −1 and the energy is 0. For right, pick z = +1: energy 0. Both real outcomes become cheap. Good.

The catch: a z that can carry everything

Now let z shift the guess by up to ±3. Left and right still cost 0. But so does an outcome that cannot happen, such as sy = 2.5, a car hanging in the air. With enough room, z simply contains the answer, and the predictor stops needing x. Then every pair of x and y has low energy, and the energy tells you nothing. That is collapse in the energy picture. The lecture says it in one line: low prediction error is not enough. Keep information in the features, and limit what z can carry. Try the dial.

How much may z carry?

The curve is the energy of each possible outcome sy. The band is the set of predictions z can reach. Drag the range of z, and see when an impossible outcome becomes cheap.

1.00

Toy numbers, not from the lecture: a predictor that outputs 0 from the fork, left and right outcomes at −1 and +1, and an impossible outcome at 2.5. With z free in [−c, c] the lowest energy is max(0, |sy| − c)². The idea (low energy for compatible pairs, collapse when z holds too much) is LeCun’s, as summarised in the lecture and in Bandaru’s reading.

At c = 1 the two real turns cost 0 and the impossible outcome costs (2.5 − 1)² = 2.25. The model knows what is possible. At c = 3 the impossible outcome costs 0 as well. The reading names two ways to stop that. Contrastive methods raise the energy of negative examples by hand, but the number of negatives needed grows very fast with the size of the representation. Regularised methods shrink the region where energy can be low, for example by limiting z. They avoid that blow-up, which is why the rest of this lecture leans on them.

The reading adds four goals for training a JEPA without negatives: keep sx informative about x, keep sy informative about y, make sy predictable from sx, and limit the information in z. Notice the tension. A summary that keeps everything is hard to predict. A summary that is easy to predict may keep nothing. The reading’s hierarchical version (H-JEPA) uses coarse, abstract summaries for long-range prediction and finer ones for short-range. We keep to the one-level case.

In the JEPA blueprint, what is the job of the latent variable z, and what goes wrong if z can carry too much?

Chapter 4

DINO: A Student Chases Its Teacher

Turn each view into a vote over learned slots, derive the loss and its slope by hand, and see the two ways the votes can go dead

A student sits in a classroom and sees only a small crop of a photo of a clock tower. The teacher next to her sees the whole view. Both are asked the same question: “Which of these K labelled bins does this picture belong in?” The student’s job is to give the same answer the teacher gives. The teacher is not a person with an answer key. The teacher is a slowly changing copy of the student herself.

That is DINO in one scene. It is a method from the contrastive-looking family with one big difference: it uses no negatives at all.

Self-distillation

Distillation means a student learns from the outputs of a teacher. In self-distillation the teacher is built from the student. At the start the teacher is an exact copy. During training the teacher’s parameters follow the student’s slowly (Chapter 5 shows how). The loss for the current step changes only the student. The two views supply the data relationship: two crops of one image. The changing teacher supplies today’s target.

An output vocabulary with no words

What do the two networks output? Each ends in a head with K learned output slots. The lecture calls it a “visual vocabulary”: K bins whose meanings are not preset. No one names slot 7 “dog”. The slots start random and get meaning only because training makes them useful. The head produces K scores (logits), and a softmax turns them into K probabilities that add to 1. The teacher’s probabilities are q and the student’s are p.

We use K = 3, as the lecture does, to keep every number visible. Suppose the teacher says q = (0.7, 0.2, 0.1) for one image, and the student, shown another crop, says p = (0.4, 0.4, 0.2). The student is too low on slot 1 and too high on slots 2 and 3.

The loss, derived

How far is p from q? Use cross-entropy: L = −∑k qk log pk. It is the average surprise the student feels when the teacher’s slots are drawn according to q. It is smallest when p = q. The teacher’s q is fixed for this update: no gradient flows into the teacher.

qk
The teacher’s probability for slot k, a fixed target during this update.
pk
The student’s probability for slot k: softmax of sk / τS.
sk
The student’s raw score (logit) for slot k.
τS
The student’s temperature. It plays the role τ played in Chapter 1.
L
The loss. The second form uses ∑qk = 1 to pull the log of the softmax apart: log pk = sk/τS − log ∑j esj/τS.

Work the numbers with q = (0.7, 0.2, 0.1) and p = (0.4, 0.4, 0.2). Natural logs: ln 0.4 = −0.916 and ln 0.2 = −1.609. So L = 0.7 × 0.916 + 0.2 × 0.916 + 0.1 × 1.609 = 0.641 + 0.183 + 0.161 = 0.986. Even a perfect student, with p = q, would not reach zero. Its floor is the teacher’s own entropy, −∑ q ln q = 0.250 + 0.322 + 0.230 = 0.802.

Now the slope. Differentiate the second form with respect to sk. The first term gives −qk/τS. The derivative of the log-sum-exp term is pk/τS. So ∂L / ∂sk = (pk − qk) / τS. For our example (before dividing by τS) the three slopes are −0.30, +0.20 and +0.10. Gradient descent moves against the slope: it raises logit 1 and lowers logits 2 and 3. The learning signal is the gap between two views’ probability vectors. It vanishes exactly when they agree.

Match the teacher’s vote

Purple bars: the teacher. Warm bars: the student. Drag the student’s three logits or press Take a step and watch the loss fall toward its floor. Then change what the teacher says.

0.69
0.69
0.00

The teacher (0.7, 0.2, 0.1), the student (0.4, 0.4, 0.2), the three slopes and K = 3 are the lecture’s. Here τS = 1 and each step changes every logit by 0.5 × (p − q). The loss is natural-log cross-entropy; its floor is the entropy of the teacher’s vote.

Two ways the votes go dead

Look at the other two buttons of the selector. What if the teacher says the same one-hot vector (1, 0, 0) for every image? The student can match it exactly: the loss reaches 0.000. The vote is confident, yet it says nothing about which image it saw. What if the teacher says the same uniform vector (1/3 each) for every image? The student can match that too, with a floor of ln 3 = 1.099. The vote is uncertain, and again it carries no information about the image.

The symptom is a loss that falls nicely. The measurement is whether the teacher’s vote differs between images, which a single image’s loss never shows. The lecture’s sentence is exact: matching the teacher is not enough if the teacher’s target stops depending on the image. The fix is to shape the teacher’s vote, and that is the next chapter.

For a student with logits s and temperature τS, the slope of DINO’s loss on logit k is (pk − qk) / τS. With q = (0.7, 0.2, 0.1) and p = (0.4, 0.4, 0.2), what does a gradient descent step do?

Chapter 5

DINO: How the Votes Stay Alive

Remove a slot that wins for every image, sharpen what is left, and slow the teacher down, then read the six lines of one training step

A class votes on a favourite colour, for three different pictures. One student always shouts “red” first, so for every picture the class says red. The vote exists, but it is useless: it says nothing about the picture. To fix it, you first subtract the shout that happens for everyone. Then you ask the class to be braver about what is left.

Those two moves are DINO’s centering and sharpening. A third move, a slow-moving teacher, keeps the target steady. Together they keep the teacher’s vote from dying in the two ways Chapter 4 showed.

Centering: subtract the shout everyone makes

Look at the lecture’s teacher scores. Three images, three slots. Image A scores (8, 1, 0), image B scores (7, 3, 1), and image C scores (6, 2, 2). Slot 1 has the highest raw score for every image. That is the collapse risk: a permanently favoured slot. Average each column across the images and you get the centre c = (7, 2, 1). Subtract it from each row and the table becomes A: (1, −1, −1), B: (0, 1, 0), C: (−1, 0, 1). The favourite slot now differs by image: slot 1 for A, slot 2 for B, slot 3 for C.

c
The running centre: a smoothed average of the teacher’s raw scores over recent batches. It is updated from the raw teacher logits, and it takes no gradient.
q
The teacher’s vote after the centre is removed and the temperature is applied.
τT
The teacher’s temperature. It is kept low, which is the sharpening step.

The lecture is careful about the limit. Centering counters a slot that is favoured for every image. On its own it can still push every vote toward uniform. That is why a second move comes next.

Sharpening: be braver about what is left

Take image B, centred to (0, 1, 0). At τT = 1 the softmax gives (0.212, 0.576, 0.212): slot 2 leads but is not sure. At τT = 0.2 the scores become (0, 5, 0), and the softmax gives (0.007, 0.987, 0.007). Same preferred slot, much stronger preference. Sharpening cannot invent a preference: exactly tied scores stay uniform at any temperature. Centering removes the part shared across images, and sharpening strengthens the part that belongs to one image. Try both.

Centre and sharpen the teacher

Each row is one image’s vote over three slots. Switch centering on and off, and drag the teacher temperature. Watch whether the three images still choose different slots.

0.20

The raw scores (8,1,0), (7,3,1), (6,2,2) and the centre (7,2,1) are the lecture’s illustration, with the running centre already at the column means. The votes are exact softmaxes of the scores divided by the temperature. The default 0.2 is the lecture’s sharp example.

With raw scores, every image votes for slot 1 at any temperature in the dial: the vote is the same for all three images. After centring, A votes slot 1, B votes slot 2 and C votes slot 3, and a low temperature makes each vote clear. Push the temperature up to 2 and the three votes still differ but they flatten: the second guard weakens.

A teacher that moves slowly

The third move is about time. If the teacher were an exact copy of the student at every step, the target would shift with every update, and the student would chase its own tail. DINO instead updates the teacher with an exponential moving average (EMA): the new teacher is mostly the old teacher plus a small piece of the student. The slide writes it as θT ← m θT + (1 − m) θS.

Work one number. With m = 0.9, a teacher weight of 1.0 and a student weight of 2.0 give 0.9 × 1.0 + 0.1 × 2.0 = 1.2. The teacher moves a tenth of the way. A rough rule: the teacher averages over the last 1 / (1 − m) steps, so about 10 steps at m = 0.9 and 100 at m = 0.99. The student takes gradient steps. The teacher takes no gradient steps; it changes only through the average.

Slow the teacher down

The thin line is one student parameter over 80 updates. The thick line is the teacher’s average of it. Drag the momentum m.

0.90

Illustrative: the student curve is a made-up wandering parameter with noise (seeded, so it is the same each visit), not a real training run. The EMA rule and m = 0.9 are the lecture’s.

Does the slow teacher help? The lecture shows DINO’s own ablation (its Figure 6), measured by k-NN accuracy on learned features. The teacher’s features beat the contemporaneous student’s under this recipe. The table beside it compares teacher choices by top-1 accuracy: a plain copy of the student gives 0.1, the previous iteration gives 0.1, the previous epoch gives 66.6, and the momentum teacher gives 72.8. Training with no lag fails; a lagged or averaged teacher works. Remember the lecture’s warning: EMA alone is not a guarantee against collapse. It works together with centering and sharpening.

One training step, line by line

Put it together. The lecture’s pseudocode has six lines. Notice what flows where: the teacher takes the large global crop; the student takes a different crop.

python# one DINO step (conceptual; the real center update uses the raw teacher logits)
q = teacher(global_view).center().sharpen()   # [K] votes, no gradient
p = student(other_view).softmax()               # [K] votes
loss = cross_entropy(q.detach(), p)             # -sum(q * log(p))
# update the student by backpropagation
# update the teacher: theta_T = m * theta_T + (1 - m) * theta_S
# update the center from the teacher logits

Three different things get updated, each in its own way: the student by gradients, the teacher by the average, the centre by its running mean. The detach call is a stop-gradient: it tells backpropagation to treat q as a fixed number. Without it, the loss could lower itself by moving the teacher toward the student, which is the cheat again.

Many crops, one target

The lecture adds multi-crop: the teacher sees a large global view, while the student sees many crops, small local ones included. A small crop must predict the vote of the big one. This rewards features that carry information about the whole from a part. The lecture’s note for later: for a world model, test whether pose and contact survive, even when object identity stays stable. DINO’s attention maps (attention from the image-level token to image patches, trained with no segmentation masks) light up objects. The lecture calls that evidence to inspect, not a result: a downstream probe decides what can be read out.

With raw teacher scores A = (8, 1, 0), B = (7, 3, 1), C = (6, 2, 2), the centre is c = (7, 2, 1). What does subtracting c do?

Chapter 6

BYOL, data2vec and I-JEPA: Predict a Description, Not a Picture

Regress on a slow teacher’s features instead of pixels, then design the hidden region so that the missing part is worth predicting

A friend covers one corner of a photo and asks you to describe what is under it. You do not need to paint it. You say “the rest of the dog’s back, brown fur.” A good description keeps what matters and drops the exact pixels. The next three methods train a network to write such descriptions, and the descriptions come from a teacher network that is still learning.

BYOL: predict the other view’s features

DINO matched probabilities over slots. BYOL (Bootstrap Your Own Latent) predicts continuous feature vectors instead. It has two branches. The online branch has an extra small network on top called a predictor, qθ. The target branch has no predictor; its weights are an EMA of the online weights, and its output goes through a stop-gradient. There are no negative pairs.

The loss normalises both vectors to length 1 and then measures the squared distance: L = ‖qθ(zθ) / ‖qθ(zθ)‖ − sg(z′ξ / ‖z′ξ‖)‖². The method also trains with the two views swapped. For two unit vectors the squared distance equals 2 − 2 cos(angle). So a prediction 60° off the target costs 2 − 2 × 0.5 = 1.0. A perfect prediction costs 0, a right angle costs 2, and the opposite direction costs 4. The lecture’s summary of why it holds together: a predictor on one branch, a stopped and slowly updated target on the other.

data2vec: predict masked positions

data2vec keeps BYOL’s continuous targets and changes the task. The student sees a masked input. The teacher sees the whole input. The student must reproduce the teacher’s features at the masked positions, by regression. The target is the average of normalised features from several upper layers of the teacher, and each position in those layers already holds context from the full input. The same recipe covers images, speech and text. A short comparison with DINO:

DINOdata2vec
Input relationshipDifferent crops of one imageMasked student input, unmasked teacher input
Target and lossSoftmax distribution, cross-entropyContinuous teacher features, regression at masked positions

I-JEPA: the same principle, a cleaner design

I-JEPA shares data2vec’s principle: masked feature prediction, full-input teacher targets, an EMA teacher. It changes the machinery. The context encoder sees only the visible patches. A separate predictor, a small Transformer, is told where to predict through position tokens. The hidden region is not random: it is several large target blocks, and one informative context block is kept. The teacher is the target encoder with its output taken at the target positions.

hC
Features of the visible context patches xC, from the student encoder fθ.
ŷT
The prediction at the hidden positions T. The predictor gφ gets the context features and the positions to fill.
yT
The target: the EMA teacher fθ̄ encodes the full image, and we keep the normalised features at the positions in T.
L
The average squared feature error over the requested positions. sg is a stop-gradient: the teacher gets no gradient from this loss.

Follow the shapes with an illustrative case. A 224 × 224 image cut into 16 × 16 patches gives 14 × 14 = 196 patches. Say the context keeps 49 patches, the four target blocks cover 96 others, and the remaining 51 are unused. The context encoder maps 49 patches to a 49 × D table. The predictor receives it plus 96 position tokens, and returns a 96 × D table. The teacher encodes all 196 patches and we keep 96 rows. The loss compares two 96 × D tables. D is whatever feature width the encoder has.

Design the hole

The lecture spends a slide on a design choice that matters a great deal: the mask. A tiny scattered hole can be filled from nearby texture, like a missing pixel in a lawn. A larger coherent block can only be filled by knowing what the surrounding object is. The context must also stay large enough to make the prediction informative. Hardness alone is not the goal: the missing content must be inferable from useful context.

Hide patches four ways

Dark squares are hidden. Pick a recipe. The read-out says how close the hidden patches sit to a visible one, a rough sign of whether nearby texture could fill them in.

Toy grid of 12 × 12 patches, seeded. The recipes follow the rows of the I-JEPA paper’s Table 6 in spirit (random 60%, three quadrants, one block of 60%, four target blocks); the real masks differ in detail. The paper’s top-1 accuracies for those four rows are 17.6, 15.5, 20.2 and 54.2. Distances are in patches and are not from the paper.

The paper’s own ablation (Table 6) holds the setting fixed and changes only the mask. The multi-block recipe, with four target blocks of 15% to 20% of the image and a context of 85% to 100% (minus the targets), reaches 54.2 top-1. Rasterised quadrants reach 15.5, one big block 20.2 and random patches 17.6. Use the table to judge a masking recipe, the lecture says, not to claim that one geometric change explains everything.

One more detail. The lecture’s feature visualisations, which look like pictures, come from a separately trained decoder. I-JEPA itself predicts features and never paints pixels. A pretty reconstruction is a probe of the features, not part of the method.

What is the main difference in what I-JEPA and a masked autoencoder (MAE) predict at the hidden positions?

Chapter 7

Video: Filling a Hole Is Not Forecasting

Keep motion in the features, then check whether “predict” means filling a gap or looking ahead

An air hockey puck sits at the centre of the table. You see one photo: the puck, the table, nothing else. Did the puck just arrive from the left, or from the right? The photo cannot say. Now you are given the three frames before it. The answer is at once clear.

The lecture builds a controlled version of this scene. Two clips share the same object, the same camera and the same background. Only the position over time differs: clip A moves left to centre, clip B moves right to centre. The final images are identical. So a representation of a single frame cannot hold the direction. For video the goal is to keep the motion direction visible in the features from the ordered frames.

Same last frame, different motion

Choose a clip, then choose what the readout may look at. Does the answer change when you switch clips?

A controlled toy in the spirit of the lecture’s classroom illustration: three frames, one puck, positions only. “50%” is chance for two directions. It is not a measured result.

V-JEPA: complete the features of a masked clip

V-JEPA carries Chapter 6’s recipe from images to video. The student sees a clip with some regions masked. The teacher sees the full clip. The student predicts the teacher’s features at the masked places, and the loss is an average L1 distance: L = (1 / |T|) ∑k∈T ‖ŷk − sg(yk)‖1. L1 adds up the absolute gaps without squaring them. Two errors of 3 and 0.1 cost 3.1 under L1, but 9.0 + 0.01 = 9.01 under squared error, where the big gap dominates. The target for each hidden location holds spatiotemporal context: information from nearby places and from nearby times.

Look at the data flow, as in Chapter 6. A clip is a stack of frames cut into small space-time blocks of patches. The student encoder reads only the visible blocks. A predictor receives those features plus the positions of the hidden blocks and returns one feature vector per hidden block. The teacher encodes the whole clip. The loss compares the two sets of vectors.

Two tasks that both say “predict”

Here the lecture makes its sharpest point. Masked completion fills a hole: the predictor sees regions before and after the hole, and it predicts features at the missing places. Causal forecasting looks ahead: the predictor sees the history up to time t, plus any actions that were supplied, and it predicts features after t. A masked clip can contain future context even while some frames are hidden. Filling a hole in the middle of a clip is closer to interpolation than to prediction.

This matters for world models, because a world model must forecast: given what has happened and what I will do, what comes next? A feature-completion task does not automatically give you that. The lecture’s advice is short: inspect the attention mask and the conditioning, not just the word “predict”. The device below lets you inspect both.

What can the predictor see?

Eight frames in time. Warm squares are what the predictor sees, purple squares are what it must predict. Pick the task, then drag where the target starts.

4
2

A schematic of the lecture’s table (masked completion sees regions before and after a hole; causal forecasting sees history up to t plus actions). The 8-frame clip is illustrative. Actions are not drawn.

A worked case, with shapes

Count tokens for an illustrative clip. Take 8 frames, each cut into a 14 × 14 grid of patches. Group two frames together into one space-time block, called a tubelet, and you get 4 × 14 × 14 = 784 tokens. Say the mask hides 600 of them. The student encoder reads the 184 visible tokens. The predictor gets those 184 feature vectors plus 600 position tokens and returns a 600 × D table. The teacher reads all 784 tokens and we keep its 600 rows at the hidden places. The L1 loss compares two 600 × D tables.

Now ask what the student could see. If the hidden tokens are scattered blocks in space and time, the visible tokens include frames from after some of the hidden ones. The model can use the end of the clip to explain the middle. That is useful for learning features. It is not what a controller needs, because a controller never sees the end of a clip it has not yet acted out.

So the failure mode has a clear shape. The symptom: a model trained by masked completion looks excellent on filling holes, and a team calls it a forecaster. The measurement: print the attention mask and count how many visible tokens come later in time than a target token. The fix: train or test with a causal mask, so each prediction sees only the past, and supply the actions as inputs. That is the task in the lecture’s second table row, and Chapter 10 returns to it.

Reading V-JEPA 2.1’s numbers carefully

The lecture closes the video section with a table from the V-JEPA 2.1 paper, and the sentences around it are a lesson in reading results. The table ablates a ViT-L model on two readouts. SSv2 measures action classification (what happened in the clip). ADE20K measures image segmentation, scored by mean intersection over union (mIoU, higher is better). Neither is a control-success score.

ViT-L ablationSSv2 accuracyADE20K mIoU
V-JEPA 2 baseline72.822.2
+ context loss62.533.8
+ multi-level prediction72.138.6

Read the rows as a trade. Adding a context loss lifts segmentation from 22.2 to 33.8 and drops action classification from 72.8 to 62.5. Adding multi-level prediction then wins most of the action score back (72.1) and lifts segmentation again to 38.6. The lecture’s lesson: where you ask for targets, and from which layers, changes what is easy to read out. A single number never says what a feature “understands”. It says what one readout could extract.

Which statement about V-JEPA’s masked clip task is correct?

Chapter 8

Rules on the Spread of Features

Stop collapse by measuring the whole batch, learn VICReg’s variance and covariance terms by hand, and find the cloud that fools them

A teacher collects a stack of answer sheets. Every sheet has the same answer, in the same handwriting. She cannot grade one sheet against another, and she would not trust any single sheet. So she adds a rule that looks at the whole stack: the answers must differ from each other. DINO built that rule into the teacher with centering and sharpening. This chapter writes the rule down as a plain penalty on a batch of features.

What the methods so far rely on

Here is the lecture’s own recap of how each earlier method resists collapse. MAE and AIM take targets from the original data, so a useful prediction must depend on the input. SimCLR, CLIP and SigLIP compare positives with negatives, so identical outputs cannot separate matching from non-matching pairs. DINO uses a slow EMA teacher with stop-gradient, plus centering and sharpening. BYOL uses EMA, stop-gradient and an online-only predictor. data2vec uses normalised teacher features for stable regression targets. I-JEPA and V-JEPA use EMA, stop-gradient and an asymmetric predictor, with recipe-specific normalisation. The lecture’s caution: learned-target methods depend on the complete recipe, and no listed ingredient is a universal guarantee.

An explicit penalty on the batch

The second route is direct. Write the loss as agreement between views plus a penalty R on the batch of features Z: L = Lagreement + λ R(Z). Here Z holds the embeddings of different images, and λ sets how strongly the penalty counts. Views of one image can agree in either picture, whether the batch is one point or a cloud. But a constant Z must pay a nonzero price. The open question is what to constrain, and which collapsed solutions the constraint rules out. You saw one answer in the hero: the variance floor.

VICReg: three terms with three jobs

VICReg stands for Variance, Invariance, Covariance Regularisation. Each letter is one job. Variance works across images: each feature coordinate must vary. Invariance works within one image: its two views must agree. Covariance works across coordinates: they should not repeat each other. The lecture also names VCReg, the V and C terms alone, and mentions Barlow Twins as related reading (it pushes the cross-correlation between views toward the identity matrix).

sj
The spread (standard deviation) of feature coordinate j across the images in the batch. In the VICReg paper it is √(variance + ε), where ε is a tiny number that keeps the arithmetic stable.
γ
The floor: the smallest spread we accept. The hinge max(0, ·) costs nothing once sj reaches γ.
RV
The variance penalty: the average shortfall of the spread below the floor. A constant feature has sj near 0, so it pays about γ.
Cij
The covariance of coordinates i and j across images. It is large when two coordinates rise and fall together.
RC
The covariance penalty: squares the off-diagonal covariances and averages over the d coordinates. It discourages redundant coordinates; the sum skips i = j.

One batch by hand

Take the lecture’s tiny batch: B = 3 images and d = 2 features. Z has rows (1, 2), (2, 4) and (3, 6). The column means are 2 and 4. Subtract them to centre Z, then compute C = Zc⊤ Zc / (B − 1). In code: Zc = Z - Z.mean(dim=0) and C = Zc.T @ Zc / (B - 1). The centred rows are (−1, −2), (0, 0) and (1, 2), so C = [[1, 2], [2, 4]]. The diagonal holds the variances, 1 and 4, which feed the variance term. The off-diagonal holds the covariance, 2: feature 2 is exactly twice feature 1, so the two columns vary together.

Score it with γ = 1. The spreads are 1 and 2, both at or above the floor, so RV = 0. The covariance penalty squares and sums the off-diagonal entries (2² + 2² = 8) and divides by d = 2, so RC = 4. The batch passes the floor and fails the redundancy check. Both terms use a batch of images; the covariance matrix compares feature columns.

The cloud that fools them

Variance and covariance are second-order statistics: they summarise spread and linear co-movement, nothing more. The lecture builds a counterexample. Take 200 image embeddings in two dimensions. Put 196 at the origin. Put the other four at (±a, 0) and (0, ±a), with a = √(199 / 2) = 9.975. The mean is 0. Each coordinate has squared sum 2a² = 199, and dividing by B − 1 = 199 gives variance exactly 1. The covariance is 0. So the sample covariance is the identity matrix, RV = RC = 0, and VCReg is perfectly happy. Yet 98% of the images share one feature vector. Try the four clouds.

Can the checks be fooled?

Choose a cloud of 200 embeddings. The device measures the spread, the covariance and the two penalties live, then counts how many images share one point.

The 196 + 4 construction and its a = √(199/2) are the lecture’s. The other three clouds are illustrative, seeded samples, standardised to unit spread where noted. Penalties use γ = 1 and d = 2 as in the lecture’s batch example; the small stabiliser ε is left out.

With the spread cloud both penalties are about 0. The redundant cloud pays RC but passes the floor. The identical cloud pays the full floor, RV = 1 (both spreads are 0), so it is caught. The outlier cloud pays nothing, and that is the problem. The symptom is a batch that passes every check. The measurement is a direct look at the distribution: how many images share one point? The fix is a check that sees the shape of the cloud, which is the next chapter.

Shape and cost

There is a second problem. The covariance matrix has d × d entries. For d = 8,192 that is about 67 million entries, or 256 MiB in 32-bit numbers, for the matrix alone. So we want a check that sees the shape of the distribution and stays practical as d grows. Both would still use batch statistics.

In the lecture’s 200-point example (196 at the origin, 4 outliers at distance a = √(199/2)), what are RV and RC, and what does that show?

Chapter 9

SIGReg and LeJEPA: A Bell-Curve Check by Projection

Look at a cloud of features along a few random directions, ask whether each profile is a bell, and combine that check with view matching

You want to know whether a crowd of 10,000 people is evenly spread over a field. Counting people in every square metre is too much work. Instead you walk a few straight lines across the field and tally who stands near each line, then ask: does each tally look like a smooth bell? If the crowd is bunched in one corner, some line will show a spike. A few lines are cheap, and they expose bunching that an average cannot.

That is the idea of SIGReg, the regulariser at the heart of LeJEPA. It answers the two worries from Chapter 8: the check must see the shape of the feature cloud, and it must stay cheap as the features grow wide.

Three words, one idea

The name is a recipe. Sketched: use a few sampled one-dimensional projections instead of the full space. Isotropic Gaussian: the reference is a bell-shaped cloud, centred at zero, with the same spread in every direction (“isotropic” means the same in all directions). Regularisation: add a loss that grows with the mismatch. A Gaussian cloud has a useful property. Look at it along any single direction and you see a bell curve. So a hard question about a cloud in thousands of dimensions turns into many easy questions about bell curves on a line.

Shapes first

Follow the lecture’s pseudocode with the tensor shapes. The encoder and projector turn B images into Z of shape [B, d]: one row per image, one column per feature. Draw K random unit directions, stored as A with shape [d, K]. Multiply: U = Z @ A, shape [B, K]. Read one column of U. It holds B numbers, one per image, each the position of that image along one direction. Compare those B numbers, as a group, with a standard Gaussian using a differentiable test. Repeat for all K columns and average the penalties.

python# conceptual SIGReg, one view (the lecture's pseudocode; gaussian_test is the paper's own test)
Z = encoder_projector(images)              # [B, d]  B images, d features each
A = random_unit_directions(d, K)           # [d, K]  K sampled unit directions
U = Z @ A                                  # [B, K]  one projected value per image and direction
penalties = gaussian_test(U, axis=0)       # each test uses the B values in one column
regularizer = penalties.mean()
# add it to the view-matching loss; backpropagate through U and Z

One tiny case. Take the batch from Chapter 8, rows (1, 2), (2, 4), (3, 6), and one direction a = (0.6, 0.8). The three projected values are 1×0.6 + 2×0.8 = 2.2, then 4.4, then 6.6. These three numbers are compared with a standard bell curve centred at 0. Their centre sits at 4.4, far from 0, so this direction reports a large mismatch. The key point is the axis: we project away the feature axis and keep the batch axis. The test never looks at the d coordinates of one image alone.

Now try it on the clouds from the last chapter. The device projects 200 embeddings onto directions you choose and measures how far the profile is from a bell with a simple stand-in test (the largest gap between the cumulative curves).

Is the profile a bell?

Bars: how many images fall at each position along one direction. Curve: the Gaussian reference. Pick a cloud, drag the direction, and raise the number of directions that are averaged.

30
8

Illustrative: the distance here is the largest gap between the cloud’s cumulative curve and a standard Gaussian’s (0 is a perfect match, 0.5 is the worst a collapsed cloud can do). It is a stand-in for the paper’s own differentiable test. The 196 + 4 cloud is the lecture’s counterexample; the others are seeded toy samples. The K directions are fixed, evenly spaced angles.

The Gaussian cloud scores near 0 at every angle. The identical cloud scores the maximum, 0.50. The stretched cloud fails along its long axis. And the 196 + 4 cloud, which VCReg accepted with zero penalty, scores about 0.49 from every direction: 196 of 200 points pile up at one spot, a giant spike a bell curve never has. Averaging K directions gives the regulariser.

LeJEPA: two terms along two axes

LeJEPA (“Le” for latent-Euclidean, “JEPA” as before) glues two terms together. Within one image, bring the embeddings of its views together. Across different images, keep the embeddings spread like a Gaussian. The minimal recipe learns both sides directly, with no EMA teacher. Compare VICReg: view agreement plus variance and covariance penalties. LeJEPA: view agreement plus SIGReg.

μn
The average embedding over the V views of image n.
Lmatch
How far each view lies from its image’s average, averaged over B images, V views and d features. It is zero when the views agree.
RSIGReg
The Gaussian check, applied to the B images at one view index v: z1:B,v is the batch of embeddings of every image, at view v.
λ
The balance between the two terms. The two terms act along different axes of the batch.

Work Lmatch for one image (B = 1) with V = 2 views and d = 2 features: z1 = (1, 0) and z2 = (0, 1). The average is μ = (0.5, 0.5). Each view is 0.5² + 0.5² = 0.5 away, so Lmatch = (0.5 + 0.5) / (1 × 2 × 2) = 0.25. A constant encoder makes this term 0. It also makes the distribution term fail, because every image would sit at one point instead of a Gaussian cloud. That is the whole guard: each term blocks the other’s cheat.

Many chips, one test

A global batch can be split across GPUs, say B = 128 images as 64 on each of two. Each GPU projects its own 64 images on the same directions and computes compact sums and a count. The sums are added across GPUs and divided by the total count, giving the test on all 128 values. The communication must stay differentiable, so gradients return to each GPU’s encoder. Because the test is built from sums across images, no GPU needs to send whole feature vectors.

The cost of widths

The last device compares the memory a full covariance matrix needs against the projection that SIGReg needs. For video the feature of one clip has M = T · H · W · D numbers. The lecture’s illustrative clip is 8 × 14 × 14 × 512, so M = 802,816.

Covariance matrix or projections?

Drag the feature dimension d (it doubles with each step) and the number of directions K. Compare the two bars on a shared log scale.

8,192
256

Counting numbers in 32-bit floats (4 bytes each). The d = 8,192 case (about 67 million entries, 256 MiB) and the 802,816-number clip (about 2.6 TB) are the lecture’s. The choice K = 256 is illustrative. Sizes below 1,000 GiB use binary units (MiB, GiB); larger ones use decimal TB, as the lecture does.

At d = 8,192 the covariance matrix holds about 67 million entries: 256 MiB. A projection matrix with K = 256 directions holds 2.1 million entries: 8 MiB. For the dense clip with M = 802,816 the full covariance needs about 2.6 TB, while the same K = 256 directions need about 784 MiB. This is why the lecture gives the clip’s cost as O(BM2) for covariance and O(BMK) for SIGReg. It adds a warning: well-distributed token features can still be identical across space and time within each clip. A constraint applied to whole clips does not check what happens inside one.

Read the lecture’s own summary before moving on. DINO, BYOL and predictive JEPAs use teacher updates and asymmetry, and their safeguards depend on the recipe. VICReg uses variance and decorrelation, and second-order statistics can miss severe concentration. LeJEPA uses view agreement plus SIGReg, and finite batches and finite directions limit the check. All of them depend on which examples are measured together. And the last line is the one to remember: avoiding collapse does not show that the representation keeps useful information.

In SIGReg, Z has shape [B, d] and A has shape [d, K]. What does one column of U = Z @ A hold, and what is compared with a Gaussian?

Chapter 10

Three Families, and How to Judge a Claim

Compare the answers each family trains toward, match each claim to the readout that can test it, and follow the path toward world models

A colleague drops a box of frozen features on your desk and says, “These understand the world.” You should ask one question before anything else: understand what, and how would we check? A feature can hold an object’s category and lose its position. It can hold position and lose its motion. “Understands the world” is not a test. A test has an input, a candidate answer and a score.

The three families side by side

The lecture ends by putting all three families in one table. Each row asks the same four things: what counts as the answer, what the family is good at, and what can go wrong.

FamilyWhat counts as the answerMain opportunityWhat can go wrong
ReconstructionThe original data: the missing pixelsKeeps rich spatial detail, and can model conditional outcomesTask-irrelevant error can dominate; point estimates average
ContrastiveThe paired view among candidatesEmphasises information shared across the chosen viewsFalse negatives; augmentation can erase useful differences
Learned targetsAnother encoding of a view or of hidden contentA more abstract prediction spaceCollapse; the target can erase useful distinctions

Match each risk to a chapter. “Task-irrelevant error can dominate” is the 9,000 table pixels of Chapter 0. “Point estimates average” is the blurry middle between left and right. “False negatives” are the second cat of Chapter 2. “Collapse” has been the villain since the hero, and “the target can erase useful distinctions” is the warning to remember: a slow teacher with centering can avoid collapse and still throw away the contact spot. The lecture’s advice is to choose the target, the visible evidence and the collapse control together. No family guarantees a world-model-ready representation.

Match the claim to the readout

Back to the box of features. The lecture lists four things a world model might need, each with a concrete test. Object category is tested by a frozen global linear probe: one vector per image, one linear layer on top. Object location or contact needs frozen local features and a small spatial head, because a single global vector has thrown position away. Motion direction needs frozen clip features and a direction classifier, as in Chapter 7. Future consequences are for later: an action-conditioned predictor, then control. The rule is short: specify the input and the readout before you say “these features understand the world”.

Can this test settle that claim?

Pick a claim, then pick a test. The device says whether the test reads the kind of information the claim needs.

The four pairs (category with a frozen global linear probe, location or contact with frozen local features and a small spatial head, motion direction with frozen clip features and a direction classifier, future consequences with action-conditioned prediction later) are the lecture’s table. The off-diagonal verdicts are our reading of what each test can see.

The variants, and one rule for reading them

New JEPA-style papers keep appearing. The lecture’s table gives four examples and one rule. I-JEPA and V-JEPA predict hidden contextual image or video features, with an EMA target and recipe-specific masking and normalisation. LeJEPA matches the embeddings of views of the same image under a SIGReg constraint. VL-JEPA predicts text embeddings conditioned on visual input, and its objective is a bidirectional InfoNCE, which means it uses negative comparisons. LLM-JEPA predicts latents alongside language learning, and the ordinary token cross-entropy stays in the objective. So the name “JEPA” does not say whether negatives, a teacher or a regulariser is doing the guarding. To assess a new variant, read its target, its conditioning and its complete objective, not only its name.

The lecture adds that prediction and generation are not rivals. DiffMAE shows denoising reconstruction can support representation learning. D-JEPA combines denoising or generative objectives with feature learning. VarJEPA represents predictive uncertainty, with evidence that depends on the domains tested. In the lecture’s words, a distribution can preserve alternatives, and a learned feature target can reduce detail; these are compatible design choices.

From features to a world model

Two last slides turn the page. LeWorldModel reuses the same regulariser for a different task. The predictor reads the past: ẑt+1 = pφ(z≤t, a≤t). Here z≤t are the encoded observations so far and a≤t are the actions taken. The loss is L = Lfuture feature + λ RSIGReg. Compare this with Chapter 7’s table: it is causal forecasting, conditioned on history and actions, with a guard against collapse on the features.

The V-JEPA 2 robot demonstration closes the evidence chain. A representation benchmark measures accessible information. A robot result also depends on action-conditioned training, prediction, planning and execution; V-JEPA 2’s control system includes additional action-conditioned training. So keep the chain explicit: features, then learned dynamics, then the evaluated control system. A good link at the start does not make the rest of the chain good.

The next lecture, Predictors and Generative Models, moves from what information to represent to how to predict with it. It covers predictor design (architectures, conditioning, prediction structure) and the foundations of generative models: conditional distributions, several possible outcomes, and sampling. The test changes too: from useful features for downstream tasks to accurate conditional predictions and plausible outcomes. The slide’s last line: the representation sets the prediction space, and the predictor models relationships and uncertainty within it.

You want to check that frozen video features keep which way an object moves. Which test fits, and why does a frozen global linear probe on a single frame not?

Chapter 11

The Whole Lecture on One Page

Review the numbers, keep a six-question checklist, read the core as code, and see where the series goes next

You can now rebuild Lecture 6 from memory. Lecture 5 gave the vocabulary: a representation, a frozen encoder, a small readout. This lecture asked what to train the encoder toward, and answered with three families. Everything below is that answer on one page.

The one-paragraph summary

Reconstruction trains toward the original data (MAE, AIM): it keeps detail, but a pixel loss can overweight a wall and a single answer averages the possible ones. Contrastive learning trains toward the matching view among candidates (SimCLR, CLIP, SigLIP): the InfoNCE softmax needs negatives, because positive agreement alone lets a constant feature win, and negatives can be false. Learned targets train toward another network’s feature of a view or of hidden content, which is the JEPA family. DINO, BYOL, data2vec, I-JEPA and V-JEPA keep the target stable with a slow EMA teacher, a stop-gradient and recipe-specific tricks such as centering and sharpening. VICReg and LeJEPA instead put explicit rules on the batch of features. SIGReg checks the shape of the cloud through random projections. Whatever the family, ask what the target is, what the predictor sees, what blocks a constant, and which readout tests the result.

The numbers that matter

WhatNumberWhy it matters
Pixel loss, equal error9,000 background pixels vs 100 contact pixels: 90 to 1The loss can weigh the wall over the task cue
Two equally likely answersGuess the middle: error 1; guess one side: error 2The conditional mean is neither answer
InfoNCE, one anchorPositive 40% at τ = 1, 97.4% at τ = 0.2; all tied: ln 5 = 1.609Temperature sharpens; a constant encoder has a known loss
SigLIP, illustrativeMatch 0.88, mismatch 0.27 (no need to add to 1)One yes or no per pair, not a softmax
DINO slopes, K = 3q = (0.7, 0.2, 0.1), p = (0.4, 0.4, 0.2): slopes −0.30, +0.20, +0.10The gradient is (p − q) / τS
Centering and sharpeningc = (7, 2, 1); slot 2 gets 57.6% at τT = 1 and 98.7% at 0.2Two guards against two kinds of dead vote
EMA teacherm = 0.9: 90% old teacher, 10% student; top-1 0.1 (copy) vs 72.8 (momentum)A slow target makes training work
I-JEPA masking, top-1Multi-block 54.2; block 20.2; random 17.6; rasterised 15.5The shape of the hidden region matters a lot
V-JEPA 2.1, ViT-LSSv2 / ADE20K: 72.8 / 22.2, then 62.5 / 33.8, then 72.1 / 38.6Targets and layers change what is readable
VCReg, B = 3, d = 2C = [[1, 2], [2, 4]], so RV = 0 and RC = 4 at γ = 1Redundant features are caught
The shortcut cloud196 of 200 at one point, covariance = identity, penalties 0Second-order checks can miss concentration
Covariance memoryd = 8,192: about 67 million entries, 256 MiBSIGReg avoids the d × d matrix
Dense clip8 × 14 × 14 × 512: M = 802,816, covariance about 2.6 TBCost O(BM²) vs O(BMK)

The checklist for any new method

  1. Target. What is the answer the loss compares with: pixels, a paired view, or another network’s feature?
  2. Evidence. What can the predictor see? Look at the mask and the conditioning, not the word “predict”.
  3. Constant guard. What stops one output for every input: negatives, a slow teacher, a rule on the batch?
  4. Batch dependence. Which examples are measured together, and does the guard depend on batch size?
  5. What is erased. Which information do the views, the mask or the target throw away?
  6. Readout. Which frozen features, which small head, which score? Say it before you say “understands”.

The core as code

Five short functions hold the lecture’s core. They are sketches in PyTorch style: sigreg stands for the paper’s own differentiable Gaussian test, which you should take from its linked implementation.

pythonimport torch, torch.nn.functional as F

def info_nce(u, tau):                       # u: [2N, d] unit vectors; row i and row i+N are views of one image (Ch 1)
    n2 = u.shape[0]; N = n2 // 2
    s = (u @ u.T) / tau                         # [2N, 2N] scores divided by temperature
    s.fill_diagonal_(float('-inf'))             # the anchor is not a candidate
    partner = (torch.arange(n2) + N) % n2
    return F.cross_entropy(s, partner)              # -log p(partner)

def dino_loss(s_logits, t_logits, center, tau_s, tau_t):   # [B, K] each (Ch 4, 5)
    q = F.softmax((t_logits - center) / tau_t, dim=-1).detach()  # centre, sharpen, stop-gradient
    logp = F.log_softmax(s_logits / tau_s, dim=-1)
    return -(q * logp).sum(-1).mean()

@torch.no_grad()
def ema_and_center(teacher, student, center, t_logits, m, m_c):
    for pt, ps in zip(teacher.parameters(), student.parameters()):
        pt.mul_(m).add_(ps, alpha=1 - m)          # theta_T = m theta_T + (1 - m) theta_S
    return m_c * center + (1 - m_c) * t_logits.mean(0)   # running centre from raw teacher logits

def vcreg(Z, gamma=1.0, eps=1e-4):                 # Z: [B, d], B > 1 (Ch 8)
    B, d = Z.shape
    Zc = Z - Z.mean(0)
    C = Zc.T @ Zc / (B - 1)                    # [d, d] covariance
    s = torch.sqrt(C.diag() + eps)
    r_v = F.relu(gamma - s).mean()
    off = C - torch.diag(C.diag())
    r_c = (off ** 2).sum() / d
    return r_v, r_c

def lejepa_loss(Z, lam, sigreg):                 # Z: [B, V, d] (Ch 9)
    match = ((Z - Z.mean(1, keepdim=True)) ** 2).mean()        # views of one image agree
    reg = sum(sigreg(Z[:, v]) for v in range(Z.shape[1])) / Z.shape[1]  # B images at each view index
    return (1 - lam) * match + lam * reg

Read it line by line and name the chapter behind each choice. The -inf on the diagonal is “exclude the anchor”. The detach is the stop-gradient. The two mul_ and add_ lines are the EMA. The Zc.T @ Zc line is the covariance of Chapter 8. And in lejepa_loss the two terms run along different axes of Z: the mean over views, then the test over images.

Where this sits

Lecture 5 defined representations and ended at the masked autoencoder. This lecture finished the survey of what to train toward. Next comes prediction: how to model what happens next, with several possible outcomes. That is where the representation, which sets the space of things that can be predicted, meets a predictor and a generative model. This lesson, like the rest of the series, grows as the course proceeds.

Keep going

Course credit

This lesson follows Lecture 6, Representation Learning II, of CIS 6280: World Models (Fall 2026) at the University of Pennsylvania, taught by Jiatao Gu. The lecture slides and the course site are the source. The reading is Rohit Bandaru’s Deep Dive into Yann LeCun’s JEPA. Every explanation and device here is ours; any mistake is ours too. The primary papers behind each method are the ones the slides cite: MAE, AIM, DiffMAE, SimCLR, MoCo, CLIP, SigLIP, DINO, BYOL, data2vec, I-JEPA, V-JEPA, VICReg, LeJEPA and LeCun’s 2022 position paper.

The takeaway. A representation is only as good as the thing it was trained toward and the thing used to test it. Pixels, matched views and learned features each reward some information and drop the rest. Every method needs a guard against the constant answer, and every claim needs a readout that can actually see what the claim is about.
A new paper trains a JEPA-style model and reports that it never collapses. What must you still ask before you trust its features for a robot?

Now press Present or Teach and explain the three families from memory: what each one trains toward, how each one can collapse, and which guard stops it. Then go back to the toy at the top, pick Views only, train it, and say in your own words why the loss reads zero while the features are useless.

CIS 6280 Lecture 6: Representation Learning II
The course site · Lecture slides · Deep Dive into Yann LeCun’s JEPA · Back to Gleams