JEPA

Chapter 1: Three Paths to learn without labels

Self-supervised learning — learning from raw data with no human labels — comes in three broad flavors. JEPA is the third, and you can only appreciate it by seeing the first two and what each gets wrong.

Path 1 — Generative / reconstructive

Corrupt the input (mask it, add noise), then rebuild the original input. Examples: autoencoders, MAE, the “predict the next pixel” family. The decoder lives in input space. As Chapter 0 showed, that forces the model to spend capacity on unpredictable detail and rewards blur. It does work — MAE learns useful features — but it is paying a tax for modeling noise.

Path 2 — Joint-embedding / contrastive

Take two augmented views of the same image, encode both, and pull their embeddings together — while pushing apart embeddings of different images (the “negatives”). Examples: SimCLR, MoCo, DINO. This learns in feature space (good!), but it has its own costs: it needs carefully hand-designed augmentations to define “same,” and it often needs many negative examples or clever tricks to avoid the model cheating by ignoring the input. It learns what is invariant, but throws away where and how things relate.

Path 3 — JEPA (joint-embedding predictive)

The synthesis. Like Path 2, work in feature space, not pixel space. But like Path 1, the signal is prediction: from the features of one part of the input, predict the features of another part. Encode a context region, encode a target region, and train a small predictor to map context-features → target-features. No pixels to rebuild (so no blur tax). No negatives or augmentations required (so no hand-tuned invariances). You predict meaning, conditioned on where you’re predicting it.

Chapter 2: Predict Meaning, not pixels

Here is the central trick, and it is worth slowing down for. Both the context and the target pass through an encoder that turns raw input into an abstract feature vector. The encoder is free to throw away whatever is not worth keeping — exact textures, noise, the precise shade of a pixel. What survives is the gist: “there is fur here, a leg shape there, grass below.” We then predict in that space.

Why latent prediction dodges the blur

Recall the masked dog. In pixel space there were thousands of valid completions, and their average was mush. But ask: what is the feature vector of all those completions? They mostly agree — they all encode “dog leg on grass.” The detail that differed (exact blades of grass) was discarded by the encoder. So in feature space the target is nearly a single point, and predicting a single point is easy and sharp. The unpredictability didn’t vanish — it was filtered out before the loss ever saw it.

Worked example by hand

Suppose a masked region has two equally valid pixel completions, A and B:

In pixel space, no matter what single image you output, you eat an error of about 0.125 forever — the irreducible cost of guessing one of two sharp answers, which is why you hedge to the blur. But the encoder maps both completions to the same feature [1.0]. So if you predict the feature [1.0], your loss is zero — and you were never asked which exact pixels appear. The objective only graded you on the part that was actually knowable. That is the entire advantage, in one table.

But this freedom hides a danger. If the encoder can choose what to keep, what stops it from keeping nothing — mapping every input to the same boring vector, making prediction trivially perfect? That catastrophe has a name, and it is the subject of the next chapter.

Chapter 3: Collapse — the trivial cheat

We are training an encoder and a predictor to make predicted features match target features. The loss is the distance between them. Let us think like a lazy optimizer: what is the easiest way to drive that distance to zero?

Make the encoder ignore its input and output a constant vector — say, all zeros — for every image. Then the target feature is always zero. The predictor learns to output zero. The distance is exactly zero. Loss minimized, training “converged.” And the representations are completely useless — they contain no information about the input at all. This is representational collapse, and it is the central hazard of every joint-embedding method.

This is why you cannot naively train both encoders by gradient descent on the same prediction loss. Gradient descent is delighted to find the constant solution — it is a deep, wide basin of zero loss. Contrastive methods avoid it with negatives (“push different images apart, so you can’t map everything to one point”). JEPA avoids it differently, with an elegant asymmetry we’ll meet next chapter.

Chapter 4: The Fix — a target that won’t chase

The collapse happened because the target encoder was free to move toward the easy constant. The fix is to break the symmetry between the two encoders. JEPA uses two ideas together, borrowed from BYOL and MoCo.

Idea 1 — stop-gradient on the target

When we compute the loss, we treat the target features as a fixed goal: no gradient flows back into the target encoder through the loss. The predictor and context encoder must move to hit the target; the target does not get to move toward them. This alone removes the “both sides race to zero” dynamic.

Idea 2 — the target encoder is an EMA of the context encoder

So how does the target encoder ever learn? Not by gradients, but by slowly copying the context encoder. After each step we nudge the target weights a tiny bit toward the context weights — an exponential moving average (EMA):

With m near 1, the target encoder is a slow, lagging echo of the context encoder. It is always a moving target that the context encoder is chasing — but a target that moves gently and is always grounded in a slightly older, stable version of the network. This lag is exactly what prevents the runaway collapse: the system can never instantly agree on the constant solution, because the target is always behind.

Worked example: how slow is the EMA?

Let m = 0.996, a target weight currently at 0.50, and the context weight has jumped to 0.90. The update: 0.996×0.50 + 0.004×0.90 = 0.498 + 0.0036 = 0.5016. The target barely moved — one ten-thousandth of the way per step. It takes hundreds of steps to catch up. That sluggishness is the feature, not a bug: a stable anchor the fast network learns against.

python — the JEPA training step (the whole idea in 9 lines)
z_ctx = context_encoder(context_patches)          # [B, N_ctx, D]  — trainable
with torch.no_grad():                            # stop-gradient on the target
    z_tgt = target_encoder(all_patches)[tgt_idx]  # [B, N_tgt, D] — the fixed goal
pred  = predictor(z_ctx, target_positions)        # [B, N_tgt, D]
loss  = ((pred - z_tgt)**2).mean()             # L2 in feature space
loss.backward(); opt.step()                       # updates context + predictor only
for p, pt in zip(context_encoder.parameters(), target_encoder.parameters()):
    pt.data = m * pt.data + (1 - m) * p.data    # EMA — no gradients here

Chapter 5: The Predictor & mask tokens

We keep saying “predict the target features from the context features.” But there is a missing piece: the predictor must know which target it is predicting. The features of a dog’s leg and the features of the sky above it are different — the predictor needs to be told where it is aiming. That is the job of mask tokens.

How the predictor is conditioned

The predictor (a small transformer) takes two things as input:

1. The context features — the encoder’s output for the visible patches, each carrying its own positional embedding so the predictor knows where the evidence sits.
2. One mask token per target location. A mask token is a single learned vector (the same one reused everywhere) plus a positional embedding for the specific patch it stands in for. The learned vector says “something goes here, predict it”; the positional embedding says “here, at row 7, column 3.”

The predictor attends from the mask tokens to the context features and outputs a predicted feature vector for each target location. So the same context can be queried for many different targets just by feeding different positional embeddings — “given what I can see, what’s the feature at this spot? and this one?” The positional embedding is the “question” and the predicted feature is the “answer.”

A crucial design choice: the predictor is kept small and narrow relative to the encoders. We want the heavy lifting — understanding the image — to happen in the encoder, so that the encoder’s features are rich and reusable downstream. If the predictor were huge, it could memorize shortcuts and let the encoder stay lazy. A small predictor forces the encoder to produce features that are genuinely predictable from one another.

Chapter 6: Masking — what to hide matters enormously

JEPA’s power depends heavily on how you choose the context and target. Hide the wrong thing and the task becomes either trivial or impossible. I-JEPA (the image version, 2023) introduced a specific multi-block masking strategy that is worth understanding, because it encodes a belief about what makes a good learning signal.

The recipe

• Target = several large blocks. Sample about four sizeable rectangular blocks as the things to predict. Large blocks (not single patches) force the model to predict semantic structure — a whole object part — not just interpolate a missing pixel from its neighbors.
• Context = a large block, minus the targets. Take one big region as the visible context, then remove any patches that overlap the targets. This is critical: if the context contained even a sliver of a target, the model could cheat by copying instead of predicting.
• Informative scale. Blocks are sampled to be big enough that prediction requires understanding the scene, but the context is rich enough that prediction is possible.

Why blocks, not random patches?

If you masked single scattered patches (like some pixel methods), each hole could be filled by copying a neighbor — low-level texture interpolation, no understanding needed. By masking large contiguous blocks, the nearest evidence is far away, so the only way to predict the target features is to reason about the whole object: “the context shows a dog’s head and front legs, so the hidden block is probably the body.” The masking strategy is how you dial in the difficulty to force semantic learning.

Chapter 7: The Full I-JEPA Loop (showcase)

Now assemble every piece into the complete training loop and run it. An image is split into patches. Context patches go through the context encoder. All patches go through the target encoder (the EMA copy, stop-gradient), and we keep the features at the target locations. The predictor, fed context features plus mask tokens, predicts those target features. The L2 distance is the loss — gradients update the context encoder and predictor; the target encoder is nudged by EMA.

The simulator below runs this loop on toy data and tracks the two numbers that matter: the prediction loss (should fall) and the representation variance (must stay high — if it crashes to zero, the model collapsed). Use it to break JEPA and see the failure modes for yourself.

Play until you have seen both outcomes: a healthy run (loss down, variance up) and a collapse (loss down, variance to zero). That contrast is the lesson — in self-supervised learning, a falling loss alone tells you nothing; you must watch what the representations are doing. The genius of I-JEPA is that, with the asymmetry in place, the easy way to lower the loss is to actually understand the image.

Chapter 8: JEPA as a World Model

Zoom all the way out. JEPA is not just an image trick — for Yann LeCun it is a blueprint for how a machine should model the world at all. His argument: an intelligent agent needs a world model that can predict the consequences of actions, so it can plan. And that model should predict in abstract state space, not pixels — for exactly the Chapter 0 reason.

Why pixel world-models fail at planning

Suppose you want to plan by imagining the future: “if I push this object, what happens next?” A pixel-predicting video model has to render every future frame in full detail. The future is uncertain, so — just like the masked dog — it blurs. Worse, planning means rolling the prediction forward many steps, and each blurry step feeds the next, so errors compound into useless fog within a few frames. You cannot plan in a hallucinated blur.

A JEPA world model predicts the next abstract state from the current abstract state and an action. Because the state already dropped unpredictable detail, the prediction stays sharp, and rolling it forward stays stable — you can imagine many steps ahead in concept space and pick the action sequence that reaches your goal. That is the substrate LeCun argues planning agents need.

The bigger family

• Action-conditioned JEPA: feed an action alongside the context so the predictor models “state + action → next state” — a controllable world model for robotics.
• Hierarchical JEPA (H-JEPA): stack JEPAs that predict at different time scales — low levels predict the next instant, high levels predict abstract long-horizon outcomes — so a plan can be coarse-to-fine.
• V-JEPA: the video instantiation, learning intuitive physics and motion from unlabeled footage by predicting masked spacetime regions in feature space.

Chapter 9: Cheat Sheet & Connections

The whole architecture in one breath, then the family comparison.

The self-supervised family

JEPA’s distinctive position: it predicts in feature space (so no blur), uses prediction rather than comparison (so it captures spatial relationships, not just invariance), and needs neither negatives nor hand-crafted augmentations (the masking is the task). The EMA-target trick it shares with BYOL/DINO is the load-bearing piece that keeps it from collapsing.

When to reach for JEPA

• Self-supervised pre-training where you want semantic features without designing augmentations — especially at scale and on a compute budget.
• World models for planning / robotics — predict abstract future states, not pixels.
• Reach for a generative model instead when you actually need to produce the output (image synthesis, inpainting) — JEPA learns to understand, not to draw.

Keep exploring

→ Contrastive Learning — InfoNCE, BYOL, DINO, and the collapse problem in depth
→ CLIP — joint-embedding across image and text
→ Vision Transformer — the patch encoder JEPA is built on
→ World Models — learning to imagine and plan
→ Diffusion — the generative counterpart that does model pixels