Latent-Variable and Adversarial Models

A model that gives one answer averages the possibilities into a blur. Latent variables and adversarial games let a model draw many sharp answers, and each way can fail in its own way.

Learn how a variational autoencoder and a GAN each turn simple noise into data, what the ELBO is, and how each one collapses.

Pick a model, then press Draw 50. The data has two outcomes, minus one and plus one, in equal shares. Watch where each model puts its draws. Then we build, piece by piece, the hidden cause, the bound, the encoder, the game, and the 2018 system that used them to dream.

You need a rough idea of what a probability is and a little calculus (what a slope is). We build the rest from zero.

Based on Lecture 7 of Penn CIS 6280: World Models (Fall 2026), taught by Jiatao Gu, and its two readings: Lilian Weng, From Autoencoder to Beta-VAE, and Carl Doersch, Tutorial on Variational Autoencoders. Lecture slides. Every idea is re-taught here in our own words and devices; it follows Lecture 6, Representation Learning II.

Fit two outcomes

Toy target from the lecture’s constructed examples (slide 8: two equally likely values, minus 1 and plus 1; slide 40: a 50% and 50% data split). The latent decoder is x = tanh(4z) plus a little noise with z drawn from a standard normal; the collapsed GAN is a generator stuck at plus 1. Both are illustrative, and every draw is a real random draw in your browser.

Chapter 0

Why One Answer Is Not Enough

See why a model that gives a single prediction blurs the future, and why that sends us to models that can draw many answers

A small car drives toward a fork in the road. In the next video frame it will be on the left branch or on the right branch. Both futures are equally likely, and you cannot tell which from the frame you have.

You train a model to predict the next frame. Your loss is the usual one: the squared difference between the predicted pixels and the real pixels. After training, you look at its prediction. The car is gone from both branches. A pale ghost car sits in the middle of the wall between them.

The model did not fail. It did exactly what the loss asked. This chapter shows why, and it is the reason the whole lecture exists.

Where we are in the course

This is Lecture 7 of Penn CIS 6280, World Models. A world model is a learned model of an environment that can say what happens next, so an agent can plan inside it. Jiatao Gu teaches the course in Fall 2026. Lecture 5 and Lecture 6 asked a quieter question: when an encoder (a network that squeezes an input into a short list of numbers called a representation) is trained, what does the training signal make it keep?

Three signals came up. Reconstruction hides part of the input and asks the network to rebuild it, as the masked autoencoder MAE does. Contrastive learning shows two views of one image and asks the network to match them, while pushing other images away, as SimCLR does. Predicting learned features asks one network to predict what another network sees, as I-JEPA does. The slides stress one idea over and over: a lower loss does not mean a better representation until you say what the downstream task needs. A tiny traffic light may matter more than a large patch of road texture.

Slide 8 then makes a point that opens today. With a fixed target and squared error, the best single prediction is the conditional mean: the average of every target that could follow this input. The slide builds the smallest example. Two targets are equally likely, minus 1 and plus 1. The mean is 0, and 0 is not a possible target.

Try it. Drag the share of plus-1 outcomes, then drag your own guess along the loss curve.

Find the best single answer

The target is +1 with the chance you pick, otherwise minus 1. The yellow curve is the average squared error of one fixed answer. Drag your guess and look for the bottom of the bowl.

50%
0.60

Exact: for a guess g the average squared error is p(g − 1)² + (1 − p)(g + 1)², and its minimum sits at g = 2p − 1, the mean of the targets (slide 8). The two-value target is the lecture’s constructed example.

Slide the chance to 50% and the bottom of the bowl lands on 0.00. No guess can do better, and every sharp guess pays. A guess of +1 costs 2.00 on average, because half the time it is two units away. The ghost car in the middle is the bottom of that bowl.

Does a better target space fix it?

Not by itself. The slide gives two target spaces. In pixels, averaging two different pictures gives a blur. In learned features, averaging still happens if the features tell the two futures apart. A feature map can also throw away an unpredictable detail, such as the exact leaf texture, and then the average is harmless. You must decide whether that detail matters.

The slide ends with a careful question: must every predictor sample all possible outcomes? Only if the task needs those alternatives. A point prediction is fine when the uncertainty that matters is small. A world model that plans a drive is not in that case, because the fork is the whole point.

What today is about

Today’s models do not give one answer. They give a distribution and let you draw from it. The lecture builds two ways to do that. A latent-variable model, the variational autoencoder (VAE), draws a hidden cause and then decodes it. A generative adversarial network (GAN) learns a sampler by playing against a critic. At the end, a 2018 system called World Models uses a VAE as its eyes. The hero above has already shown the three outcomes you will meet: a blur, a healthy spread, and a collapse onto one mode.

Slide 13 is a four-minute clip of Yann LeCun, 21:06 to 25:08 of an ETH Zurich talk, with two questions to keep in mind. Which details should a predictive representation keep or ignore? And what does a reconstruction objective ask z to retain? We will answer the second one with numbers in Chapters 5 and 6.

One paragraph on the course itself. Assignment 1 came out on Sep 10 and is due Sep 24 on Canvas. The Sep 15 session was tentatively virtual on Zoom, and the Sep 17 session was tentatively canceled, so check Canvas. The lecture plan gave 20 minutes to the recap and the LeCun video, 43 minutes to the VAE and its link to state-space models, 5 minutes to GANs and 15 minutes to World Models. Lecture 8 on Sep 22 covers autoregressive models.

The road

  1. Hidden causes: draw a latent, then an observation; why the likelihood is hard to score.
  2. Posterior and variational inference: approximate the hidden cause, then share the work with an encoder.
  3. The ELBO: one identity that gives a bound you can train.
  4. The VAE as code: Gaussian posteriors, the KL, the reparameterization trick, a training step.
  5. What the KL charges: input information, aggregate matching and posterior collapse.
  6. Rate and distortion: the beta knob, and why a decoder returns a mean.
  7. Latents for prediction: the conditional VAE and the four objects.
  8. Adversarial learning: the GAN game, contrastive learning, mode collapse.
  9. World Models, part one: vision, memory and a mixture of futures.
  10. Imagination and exploitation: control, dreaming, and where a model can be fooled.
  11. The field guide: numbers, code, and what comes next.
Two futures are equally likely, minus 1 and plus 1. A network trained with squared error outputs one number. What does it learn to output?

Chapter 1

Hidden Causes and the Decoder

Draw a short hidden code first, decode it into an observation, and see why scoring a real observation is the hard part

Look at one photo of a face. The photo has a million pixels, but you can describe it with a few numbers: how the head turns, how the light falls, whether the mouth smiles. Change one of those numbers and a whole region of pixels changes together. The few numbers explain the many.

Those numbers are a latent variable: a hidden quantity that you never see directly, but that causes what you do see. Last week an encoder squeezed an image into features. This week we flip the arrow. We write down how a hidden cause generates an observation, and then ask how to find the cause from the observation.

The state-space recipe, with one step

You know this shape from Lecture 3 and Lecture 4. A state-space model has a hidden state s and an observation o. The state starts from a prior, p(s0), and each observation is drawn given its state, p(ot | st). Slide 15 keeps only the first time step. It then renames the state z and the observation x. What is left is a model of one image.

pθ(z, x)
The probability of a hidden cause z and an observation x together. The subscript theta marks the numbers the network learns.
p(z)
The prior: how the hidden cause is drawn before you see anything. The lecture picks a standard normal, N(0, I): a round cloud centred on zero.
pθ(x | z)
The decoder: given a hidden cause, a distribution over observations. A neural network sets its parameters.

Generation is two steps. First draw a latent, z ~ p(z). Then draw an observation, x ~ pθ(x | z). Slide 16 states the payoff in one line: the decoder turns a simple latent distribution into a flexible distribution over observations. Doersch’s tutorial makes the same point: you can build almost any shape of data by pushing normal noise through a complicated enough function. Draw a few points and watch it happen.

Draw a latent, then an observation

The purple bell on top is the prior over one latent number z. The decoder maps z to a point on the warm curve, then adds noise. Press Draw 1 to see one full draw, or Draw 100 to see the shape of the data.

0.15

Toy decoder, not from the lecture: the mean of x is (z, 0.4z² − 0.4) and each coordinate gets Gaussian noise of the size you set. The prior is the lecture’s N(0, I) in one dimension. Draws are random in your browser.

Notice the two jobs. The prior is boring on purpose: one simple cloud that is easy to sample. All the interesting structure lives in the decoder. With noise at 0.00 every draw lands exactly on the curve. With more noise the points form a fuzzy band. That band is a distribution over observations that no simple formula could have written down directly.

What the decoder promises: a likelihood

A decoder does not output a picture. It outputs the parameters of a distribution over pictures. Slide 17 gives the common case, a Gaussian with a mean μθ(z) from the network and a fixed variance σx². Take the negative log of that Gaussian and the squared error appears.

−log pθ(x | z)
The reconstruction loss: how surprised the decoder is by the real x. Smaller is better.
‖x − μθ(z)‖² / 2σx²
The squared distance between the real pixels and the decoder’s mean, scaled by how much error the decoder expects.
C
A constant that does not change with theta or z, so training ignores it.

A worked case with two pixels. The real image is x = (0.8, 0.2). The decoder’s mean is μ = (0.6, 0.5). The squared distance is 0.2² + 0.3² = 0.04 + 0.09 = 0.13. With σx² = 0.5 the loss is 0.13 / (2 × 0.5) = 0.13. With σx² = 0.1 the same mistake costs 0.13 / 0.2 = 0.65. A small assumed variance makes every error hurt more.

So squared error is a choice, not a law. It says: errors are Gaussian, all pixels share one variance, and the decoder only needs a mean. For binary pixels the matching choice is a Bernoulli, and the loss becomes binary cross-entropy: if the true pixel is 1 and the decoder says 0.8, the loss is −ln 0.8 = 0.22; if the true pixel is 0, the same 0.8 costs −ln 0.2 = 1.61. A richer distribution gives a different loss and a different assumption. The slide’s summary: reconstruction is a modeling choice about errors, noise and unresolved variation.

Maximum likelihood, and the integral that blocks it

How should we train? Slide 18 gives the classic answer: choose theta to make the real data likely. For one real observation x, the model’s probability for it is every way z could have produced it, added up.

log pθ(x)
The log-likelihood of the real observation. Maximum likelihood tries to push this up, averaged over the data.
∫ p(z) pθ(x | z) dz
Add up, over every possible hidden cause, the chance of that cause times the chance it produced x.

Sampling is easy: draw z, decode. Scoring is hard: that sum runs over all z. The slide asks the natural question. Could we just average pθ(x | z) over many prior samples? In principle yes. In practice, most prior samples explain a given x badly. The next device counts how fast that goes wrong as the latent grows.

Count the useful prior samples

Say an observation needs every one of its d latent numbers to land within 0.5 of 1.0. The curve is the exact chance that one prior draw does. Drag d, then press Draw 10,000.

4

Illustrative toy of the slide’s warning, not the lecture’s numbers. Per coordinate the chance of landing in (0.5, 1.5) under a standard normal is 0.2417; all d coordinates must land, so the chance is 0.2417d. The sampled count is a real random run in your browser.

At d = 1 about a quarter of prior samples help. At d = 6, about 1 in 5,000. At d = 10 the chance is about 6.8 in ten million, so ten thousand samples are expected to hit only 0.007 times. Real latents have tens to hundreds of numbers. A scorer built on blind prior samples would need astronomically many draws. That is the sentence on the slide: an easy sampler does not give an easy likelihood.

A latent-variable model makes new data in two steps. What are they?

Chapter 2

Inferring the Hidden Cause

Run the generator backwards, replace an impossible calculation with a fitting problem, and then train a network to do the fitting for you

You walk into an office and see a wet umbrella by the door. You did not see the weather. You can still ask: how likely is it that it rained? You start from what you believed before (it rains on 3 days in 10), and you update with what you saw (a wet umbrella is likely after rain, and unlikely after sun).

That update is posterior inference: from the observation back to the hidden cause. The posterior is the belief about the cause after you have seen the evidence.

Bayes’ rule, with the umbrella

Say the chance of a wet umbrella is 0.8 after rain and 0.1 after sun. Multiply each prior by its likelihood: rain gives 0.3 × 0.8 = 0.24, and sun gives 0.7 × 0.1 = 0.07. Add them: 0.31. That sum is the probability of the evidence itself. Divide each product by it. The posterior chance of rain is 0.24 / 0.31 = 0.77.

pθ(z | x)
The posterior: which hidden causes could explain this particular observation, and how strongly.
p(z) pθ(x | z)
The prior times the decoder’s likelihood. This is the joint, and we can evaluate it for any pair (z, x).
pθ(x)
The evidence: the 0.31 above, but now an integral over every z. This is the Chapter 1 integral, and it is the blocker.

With two weather states the evidence is one addition. With a neural decoder and a latent of 64 numbers it is a 64-dimensional integral. In Lecture 3 and Lecture 4 you met a case where this worked: for a linear Gaussian state-space model, Gaussian conditioning gives the posterior in closed form, and that is the Kalman filter. A neural decoder breaks the closed form. The slide states the position plainly: we still want a distribution over plausible latents, but we cannot write it down.

Variational inference: fit a simple shape instead

The slide 20 move is to change the problem. Do not compute the posterior. Fit a simple distribution to it. First choose a family that is easy to handle, such as Gaussians with an adjustable mean μ and variance eℓ. Write qλ(z) with λ = (μ, ℓ). Then find the member of the family closest to the posterior.

“Closest” needs a ruler. The ruler is the KL divergence, KL(q ‖ p) = Eq[log q(z) − log p(z)]. It is the extra surprise, on average, when you believe p but the world follows q. It is never negative, and it is zero only when the two are the same. A worked case: let q = (0.5, 0.5) and p = (0.9, 0.1) over two outcomes. Then KL = 0.5 ln(0.5 / 0.9) + 0.5 ln(0.5 / 0.1) = 0.5 × (−0.588) + 0.5 × 1.609 = 0.51 nats. A nat is the unit you get from the natural logarithm.

The slide holds the model θ and the observation x fixed, then minimises over λ. Since this is an ordinary optimisation, you can run it for each x separately, and you do not need a neural network. Try it on a posterior that is not a Gaussian.

Fit a Gaussian to a posterior

The blue shape is a posterior that is not a Gaussian. The pink bell is your q. Move its mean and its variance until the yellow KL number is as small as you can make it, or press Optimise.

1.00
1.00

Illustrative posterior, not from the lecture: 65% of N(−1.0, 0.45²) plus 35% of N(1.4, 0.35²). KL(q ‖ p) is computed numerically on a fine grid in your browser; Optimise searches a grid of (μ, eℓ) for the smallest value.

Two things appear. First, the best Gaussian settles on the taller bump and leaves the other bump uncovered. That is how this ruler behaves: KL(q ‖ p) punishes q for putting mass where p has none, so q hides in one mode. Second, the best KL is not zero. A Gaussian cannot be a two-bump shape. The family sets a floor on how good the approximation can be.

Amortizing: one network for every x

Per-input fitting has a cost. Each new observation starts its own optimisation loop. For 1,000 images and 100 steps each, that is 100,000 gradient steps before you have one useful code. Slide 21 replaces the loop with a shared inference network, the encoder qφ(z | x). It reads x and predicts the parameters in one forward pass: λφ(x) = (μφ(x), ℓφ(x)). Weng’s post uses the same names: qφ(z | x) is the “probabilistic encoder” and pθ(x | z) the “probabilistic decoder”.

The word for this is amortized: you pay the fitting cost once, during training, and spread it over every future input. The training objective is the same KL, now averaged over data and minimised over the encoder weights φ. For a batch of B inputs the data flow is plain: x of shape [B, ...] goes in, and two tensors of shape [B, d] come out, one of means and one of log-variances.

The price is the amortization gap: a shared network may fit one particular x worse than a dedicated loop would. This toy has an exact answer, so you can see the gap.

Close the amortization gap

Blue is the exact answer for each x. Pink is what your encoder, a straight line, says. Drag its slope and offset to match, or press Train the encoder.

1.10
0.30

Illustrative linear-Gaussian toy with exact algebra: z ~ N(0, 1) and x | z ~ N(z, 0.25). The true posterior is N(0.8x, 0.2). The encoder predicts the mean as slope × x + offset and uses the exact variance 0.2, so the average KL is ((slope − 0.8)² × 1.25 + offset²) / 0.4 nats for x drawn from N(0, 1.25). The 1,000 inputs and 100 steps are made-up round numbers.

At the best setting (slope 0.80, offset 0.00) the gap is exactly zero, and the read-out shows what the shared network costs at inference: one pass per input, with no loop at all. A real encoder is not a line and a real posterior is not this tidy, so the gap there is never exactly zero. It is the price you accept for speed.

Why does the lecture replace a separate optimisation per observation with an encoder network?

Chapter 3

The ELBO: A Bound You Can Train

Derive one exact identity, read it as a reconstruction term plus a prior term, and see what makes the bound tight

You want to raise log pθ(x), but you cannot compute it. A builder in this spot has one good trick: find a number you can compute that is never larger than the target. Push that number up, and the target is pushed up too, because it sits above.

A number that always lies below a target is a lower bound. This chapter builds one, and it is the objective every VAE trains.

Step 1: the exact identity (slide 22)

Start with the quantity we already know how to talk about: the KL between the encoder’s qφ(z | x) and the true posterior. Write it as an average under q.

KL(q ‖ pθ(z | x)) = Eq[ log qφ(z | x) − log pθ(z | x) ].

We cannot evaluate the posterior inside the bracket. But Bayes’ rule from Chapter 2 says pθ(z | x) = pθ(x, z) / pθ(x). Take the log: log pθ(z | x) = log pθ(x, z) − log pθ(x). Put that in. The hard term log pθ(x) does not depend on z, so the average leaves it alone and it comes outside:

KL = Eq[ log qφ(z | x) − log pθ(x, z) ] + log pθ(x).

Give the first bracket a name, with its sign flipped. It is the ELBO, the evidence lower bound. (“Evidence” is the old word for pθ(x).) Rearranging gives the identity of the slide.

log pθ(x)
The log-likelihood we want to raise but cannot compute.
ℒELBO(x)
Eq[ log pθ(x, z) − log qφ(z | x) ]. Every piece is something we can evaluate: the joint, and the encoder’s own density.
KL( q ‖ pθ(z | x) )
The gap: how far the encoder is from the true posterior. It is never negative.

Because the gap is never negative, ELBO ≤ log pθ(x). That is the bound. And the gap is exactly the Chapter 2 quantity, so two goals line up. Raising the ELBO over θ raises the likelihood, and raising it over φ closes the gap.

Step 2: reading it as two forces (slide 23)

Split the joint: log pθ(x, z) = log p(z) + log pθ(x | z). The ELBO becomes Eq[log p(z) + log pθ(x | z) − log qφ(z | x)]. Group the first and third terms and you meet a KL again:

ℒELBO(x)
The score we maximise. Training minimises its negative.
Eq log pθ(x | z)
The reconstruction likelihood: after inferring a code from x, how well does the decoder explain x? This rewards keeping information.
KL( q ‖ p(z) )
The prior KL: how far the inferred code strays from the prior. This penalises codes that the prior would never produce.

Two forces, then. One says: fit the observation. The other says: stay compatible with the generative prior, so that sampling from the prior later still makes sense. If your last lectures used a squared-error reconstruction loss, it now sits inside a proper likelihood objective. The next device uses a toy where every term has a closed form, so you can check the identity with your own hands.

Check the identity

One observation x, and a Gaussian q you choose. The bar is the negative ELBO, split in two. The blue line is the true negative log-likelihood, the best the bar can do. Slide q onto the posterior and the gap shuts.

1.00
0.30
1.00

Illustrative linear-Gaussian toy with exact closed forms: z ~ N(0, 1), x | z ~ N(z, 0.25), so log p(x) = −½ ln(2π × 1.25) − x² / 2.5 and the posterior is N(0.8x, 0.2). For q = N(m, s²): the reconstruction term is −½ ln(2π × 0.25) − ((x − m)² + s²) / 0.5, and the prior KL is ½(m² + s² − 1 − ln s²). The gap is computed from log p(x) − ELBO and equals the posterior KL.

Try x = 1.00 and snap q to the posterior (mean 0.80, spread 0.45, so variance 0.20). Negative ELBO and negative log-likelihood both read 1.43 nats, and the gap is 0.00. Of that 1.43, about 0.71 is reconstruction cost and 0.72 is prior KL. Notice what the prior KL is at this perfect fit: not zero.

Step 3: what “tight” means (slide 24)

The slide separates two quantities that students tend to merge. The training penalty is KL(q ‖ p(z)): the encoder against the prior. The gap to the log-likelihood is KL(q ‖ pθ(z | x)): the encoder against the true posterior. When q equals the true posterior the second one vanishes and ELBO = log pθ(x).

Does a tight bound need the prior KL to be zero? No. The true posterior for this x is centred at 0.80, not at 0. It can sit well away from the prior, and the best q follows it. The slide’s takeaway: a tight ELBO means accurate inference for the current generative model. It says nothing about whether that model is good.

When it goes wrong. The symptom: the training loss falls, yet the model explains the data poorly. The measurement: split the negative ELBO into its parts and compare it with the likelihood, if you can. In the device, set x = 1.00 and leave q at mean 0.30 and spread 1.00. The negative ELBO is 3.25 nats against a true 1.43, a gap of 1.82. The prior KL is only 0.05, so the encoder is hugging the prior and ignoring x. The fix is to give the encoder what it needs to move: more capacity, or less pressure from the prior (Chapter 6).

Why does maximising the ELBO over the encoder weights also close the gap to the log-likelihood?

Chapter 4

The VAE as Code

Turn the ELBO into a network you can train: Gaussian codes, a closed-form KL, the reparameterization trick and one training step

You now have a score to maximise, the ELBO. To train a network on it you need three practical things. The encoder must output a distribution, not a single code. The KL term must have a formula you can differentiate. And the random draw in the middle must not block the gradient. This chapter fixes each one.

An autoencoder that outputs a distribution

A plain autoencoder has an encoder that squeezes x to a code z and a decoder that rebuilds x from z. Weng’s post walks through that family first, with its denoising, sparse and contractive variants. The variational autoencoder (VAE), slide 25, changes two things. The encoder outputs a distribution over z, and the decoder outputs a likelihood over x. Then it trains both by minimising the negative ELBO, an approximate maximum likelihood.

Slide 26 picks the simplest encoder distribution: a Gaussian with a diagonal spread. For d latent coordinates the encoder outputs two lists of d numbers, the means μ and the log-variances ℓ. The variance is eℓ. Why log-variance? A raw variance must stay positive, and a network output can be any number. The exponential makes any output a legal variance.

With the prior N(0, I), the prior KL has a closed form, one term per coordinate.

KL( q ‖ p(z) )
The prior KL from Chapter 3, for one input x.
½
A constant that comes out of the Gaussian algebra.
∑j=1d
Add the cost of each of the d latent coordinates. They are independent, so the costs add.
μj²
The cost of a mean far from zero.
eℓj
The spread of the code. Together with the next two terms it is smallest when the spread is exactly 1.
− 1 − ℓj
The shift that makes the whole bracket zero when μ = 0 and ℓ = 0.

Worked example. One coordinate with μ = 1 and ℓ = 0 (variance 1): ½(1 + 1 − 1 − 0) = 0.50 nats. A tighter code with μ = 0 and ℓ = −2 (variance 0.135): ½(0 + 0.135 − 1 + 2) = 0.57 nats. Both cost something. Only μ = 0, ℓ = 0 costs nothing, and the slide asks what the KL prefers alone: exactly that, so every input would share the prior N(0, I). Move the two knobs and see.

Price a Gaussian code

Purple is the prior N(0, 1). Pink is the encoder’s code for one coordinate. The yellow number is its KL. Then raise d to see all the coordinates add up.

1.00
0.0
8

Exact: the closed form on the slide, ½(μ² + eℓ − 1 − ℓ) per coordinate, with every coordinate given the same μ and ℓ so the total is d times one coordinate.

The blocked gradient and the fix (slide 27)

The reconstruction term needs a sampled code: draw z from the encoder’s Gaussian, decode, compare. But a random draw is a dead end for backpropagation. The network can tell the decoder how to change, yet there is no path from the loss back through “pick a random number” to the encoder’s μ and ℓ. Weng’s post says it directly: sampling is a stochastic process, so we cannot backpropagate through it.

The reparameterization trick moves the randomness out of the way. Draw parameter-free noise ε ~ N(0, I) first. Then build the code as a plain formula of the encoder’s outputs: z = μ + eℓ/2 ⊙ ε. The result has exactly the same distribution as drawing from q. But now z is a differentiable function of μ and ℓ, and the noise is just an input. The slide draws the gradient path: reconstruction loss, to z, to (μ, ℓ), to φ.

Push gradient through a draw

The loss is (z − 3)². Set the encoder’s μ and ℓ, then press Draw 1,000 noise values. Each draw gives z = μ + eℓ/2ε and a gradient. The averages should match the exact bars.

1.00
0.0

Illustrative toy loss L = (z − 3)², chosen because its exact expectation is known: E[L] = (μ − 3)² + eℓ, so dE/dμ = 2(μ − 3) and dE/dℓ = eℓ. Each draw’s gradient uses the chain rule through z: dL/dμ = 2(z − 3) and dL/dℓ = 2(z − 3) × ½ eℓ/2ε. The draws are random in your browser.

A single draw gives a noisy gradient. A thousand draws, averaged, land close to the exact answer. A real VAE uses one draw per example per step and relies on the many examples in a batch to average the noise out. That is slide 28’s remark: one latent sample per example gives a Monte Carlo estimate of the reconstruction expectation.

One training step

Slide 28 shows the whole step in a few lines. The comments give the tensor shapes. B is the batch size and d the latent length.

python# x: [B, ...]; encoder outputs: [B, d]
mu, logvar = encoder(x)                 # two tensors, each [B, d]
eps = randn_like(mu)                    # parameter-free noise, [B, d]
z = mu + exp(0.5 * logvar) * eps      # reparameterized code, [B, d]

nll = -decoder_distribution(z).log_prob(x).sum_over_event_dims()   # [B]
kl  = 0.5 * (mu**2 + exp(logvar) - 1 - logvar).sum(-1)      # [B]
loss = (nll + kl).mean()                              # one number

optimizer.zero_grad(); loss.backward(); optimizer.step()

Read it in four beats. The encoder reads the batch. The code is built from noise. The loss adds a reconstruction cost and a prior cost: the first sums over every pixel of the observation, the second over every latent coordinate, and the batch averages the two together. Then the usual backward pass updates both networks at once, because the noise is an input and not a block.

Two ways to use a trained VAE (slide 29)

A trained VAE can run in two modes, and they test different things.

OperationWhere z comes fromWhat it tests
Reconstruct an observed xz ~ qφ(z | x), the encoderCan a code inferred from x explain x?
Generate a new xz ~ p(z), the priorDo prior samples decode into plausible observations?

Excellent reconstruction does not guarantee excellent prior samples. The encoder may place real images in regions of z that the prior visits rarely or differently, so decoding a fresh prior draw lands in a hole that training never covered. Evaluate prior generation separately.

When it goes wrong. The symptom: the KL term jumps to huge values, or the loss stops being a number at all, early in training. The measurement: print the log-variance. If an encoder output reaches ℓ = 10, then e10 = 22,026 and the KL for that one coordinate is ½(22,026 − 1 − 10), about 11,008 nats, enough to swamp every other term. One common guard is to bound ℓ to a modest range before the exponential. The deeper point is the one the formula shows: the KL punishes both a large mean and a spread far from 1.

Why do we write z = mu + exp(l/2) times epsilon with epsilon drawn from N(0, I), instead of drawing z straight from q?

Chapter 5

What the KL Charges For

Split the prior KL into a price for information and a price for shape, and see how a code can look perfect and say nothing

Imagine two students who must each send you a code that describes a photo. The first sends a number drawn at random, with no link to the photo. The second sends a number that really depends on the photo. Seen from far away, both pile up in the same bell-shaped heap, because both numbers look like standard normal noise.

Only one of them told you anything. A good objective must be able to tell them apart. That is the question of this chapter.

The cousin from last week: SIGReg

Last week, LeJEPA used a term called SIGReg to keep learned features spread out and not collapsed. Slide 30 compares it with the VAE’s prior KL, because both try to organise the distribution of codes. Both pair a task loss with a latent regulariser: task loss plus latent distribution regularisation.

SIGReg in representation learningPrior KL in a VAE
Task signalPredict or agree across related viewsReconstruct the observed input
LatentsOne feature vector per viewA distribution qφ(z | x) per input
Gaussian matchingAcross inputs: the distribution of feature vectorsFor each input: its conditional distribution against N(0, I)

The slide calls the link close. VAE regularisation includes aggregate distribution matching, the same kind of thing SIGReg does. But the VAE’s per-input KL does a second job: it also penalises information that the code carries about x. We can pull those two jobs apart.

Two bills in one KL (slide 31)

Define the aggregate posterior qφ(z) as the average of all the per-input codes: qφ(z) = Ex qφ(z | x). It is the heap you see from far away. Then the average prior KL splits exactly:

Ex KL( qφ(z | x) ‖ p(z) )
The average prior KL: the “rate” the VAE pays during training.
Iq(X; Z)
The mutual information between the input and the code: on average, how much knowing z shrinks your uncertainty about x, counted in nats. Zero means z says nothing about x.
KL( qφ(z) ‖ p(z) )
Aggregate matching: how close the far-away heap is to the prior.

The slide builds a one-dimensional example in which the second bill is zero for both students, so only the first bill differs. In the first case the code ignores x: q(z | x) = N(0, 1) for every x. In the second, the code follows x: its mean varies with x (variance 3/4 across inputs), and each code has variance 1/4. Both produce the same aggregate, N(0, 1). The average VAE KL is 0 in the first case and log 2 = 0.69 nats in the second. That 0.69 nats is the price of the information the second code carries.

The next device parametrises both cases with one slider, a. The code is z = a · x + (noise). Choose the noise so z always keeps unit variance. Then a = 0 is the first student, and a = 0.866 (so that a² = 3/4) is the second.

Same heap, different information

Each dot is one input x (across) and its code z (up). The curve at the right is the heap of z. Drag a from 0 toward 1 and watch the dots line up while the heap stays put.

0.00

Built from the slide’s example: x ~ N(0, 1) and z | x ~ N(a x, 1 − a²), so the aggregate of z is N(0, 1) for every a. The average KL is −½ ln(1 − a²) nats, which is 0 at a = 0 and log 2 at a² = 3/4, matching the slide. The 300 points are a fixed seeded sample; the correlation and variance are measured on them.

At a = 0 the dots form a cloud with no pattern, the correlation is near 0, and the VAE KL is 0.00. At a² = 3/4 the dots follow a clear line, the heap of z is still a standard bell, and the KL is 0.69 nats. A test that only looks at the shape of the heap would give both a perfect score. The KL sees the difference, because it charges for information.

Posterior collapse (slide 32)

Now run training in the first student’s direction. If the encoder gives the same code N(0, 1) for every input, then qφ(z | x) ≈ p(z) and the mutual information is close to 0. This is posterior collapse. The code carries almost nothing about the input.

The decoder can then learn to ignore z. If pθ(x | z) = pθ(x), the model’s true posterior also equals the prior, pθ(z | x) = p(z), and the encoder that copies the prior is exactly right. The prior KL is zero, the gap is zero, and the model never uses its latent.

Why would training choose that? Look back at slide 26 and at Chapter 4: the KL alone is happiest at μ = 0, ℓ = 0. The reconstruction term must reward information strongly enough to pay for it. If the decoder can already model x well without z, there is nothing to pay for, and the cheap path wins. A decoder that is strong enough to explain x by itself makes that path easy.

A useful warning from the slide: can z still have nonzero variance during posterior collapse? Yes. z stays a Gaussian draw. The slide’s summary is exact: posterior collapse is loss of input information, not necessarily a constant latent vector. So checking that z is not constant does not rule it out.

Two diagnostic questions come from the slides. Does qφ(z | x) actually depend on x? And does the decoder use z? Measuring the average KL is the quick first check: a value near 0 nats, for a model that should be using its code, is the warning sign.

This also answers the question from the LeCun clip. A reconstruction objective asks z to keep whatever lowers the reconstruction cost by more than its KL price. It asks nothing about the traffic light or the texture. It only counts pixels.

Two codes both have an aggregate posterior of N(0, 1). The first ignores x; the second follows x. What does the average VAE KL report?

Chapter 6

Rate, Distortion and the Beta Knob

Read the VAE loss as a budget for a message, turn one weight to trade detail for compactness, and learn what a decoder’s picture really shows

You must describe a photo to a friend over a phone line that charges by the word. If you send many words, your friend can redraw the photo well, but the call is expensive. If you send few words, the call is cheap, and your friend draws a rough sketch. At some price, you stop describing the photo at all and say only “a photo”.

A VAE makes exactly this deal, and slide 33 writes it with two letters.

Two costs, one weight

Distortion, D, is how badly the decoder redraws x from the code: D = E[−log pθ(x | z)], the reconstruction cost from Chapter 3. Rate, R, is the price of the message: R = Ex KL(qφ(z | x) ‖ p(z)), the prior KL. A rate of 0 nats is the empty message. The VAE minimises D + βR, with a weight β on the rate.

ChoicePressure on the solution
Higher βA larger penalty for information and prior mismatch.
Lower βMore emphasis on reconstructing observations.
β = 1The standard negative ELBO for the specified likelihood.

Weng’s post calls the weight on the KL term a hyperparameter and says that for β > 1 it applies a stronger constraint on the latent bottleneck and limits the capacity of z. It adds that a higher β may create a trade-off between reconstruction quality and the extent of disentanglement. The slide gives a sharper warning: choosing a bottleneck does not identify the right semantics automatically. What is useful for reconstruction may differ from what is useful for a task.

A toy where we can find the best answer

To see the whole curve, we need a model small enough to solve. Take data x ~ N(0, 1). Let the encoder send z = a·x plus noise of spread s, and let the decoder be the best straight-line guess of x from z. For a decoder that expects a noise variance σx² = 0.1, the formulas are exact. The rate is ½(a² + s² − 1 − ln s²). The distortion is [ s² / (a² + s²) ] / (2σx²), up to a constant. The device finds the best a and s for each β by searching a grid.

Turn the beta knob

Each point on the curve is the best encoder for one β. Left is a cheap, poor message. Right is a rich, costly one. Drag β and watch the marker slide.

1.0

Illustrative linear-Gaussian toy, not the lecture’s numbers: x ~ N(0, 1), z = a x + s ε, decoder variance 0.1, best linear decoder. Rate and distortion are exact formulas; the best (a, s) for each β comes from a grid search in your browser. In this toy the optimum collapses at about β = 1 / 0.1 = 10.

At β = 1, the standard ELBO, the best encoder spends 1.15 nats of rate and pays a distortion of 0.50. At β = 4 it spends 0.46 nats and pays 2.00. The curve slopes down and to the right: more rate, less distortion, always a trade, never a free lunch.

Near β = 10 something sharp happens. The rate drops to 0.00 and the distortion jumps to 5.00, which is what you pay when the decoder ignores z entirely (1 / (2 × 0.1) = 5). That is posterior collapse from Chapter 5, now with a cause you can see: β made information too expensive. The same collapse can be reached from the other side. A decoder that expects large noise (a big σx²) makes each unit of distortion cheap, which is the same as raising β.

This explains a common practical move. If a VAE collapses, builders lower the weight on the KL, so information is cheap enough to keep. If they want a tighter, more compact code, they raise it.

What a decoder’s picture really shows (slide 34)

Most VAE pictures you have seen are blurry. Slide 34 asks whether blur is a law of VAEs. The answer is no. To see why, separate three things that sound alike.

  1. The conditional mean: for a fixed-variance Gaussian decoder, the network’s output μθ(z) is the best guess of x given z, the average of every x that could have produced that z, μ*(z) = E[X | Z = z]. If one z leaves several appearances possible, this average blends them.
  2. A conditional sample: draw x from pθ(x | z). The mean plus the decoder’s noise.
  3. A sample from the full model: draw z from the prior first, then x. This is what generation means.

Pictures in papers are usually the first kind, because the mean looks cleaner than the noisy sample. That is a display choice. Blur is also not forced. It depends on how much information z carries (a latent that tells the two futures apart removes the averaging), on the likelihood family, on the architecture, and on optimisation. Chapter 7 puts this to work: a richer latent lets the decoder keep two futures apart.

When it goes wrong. The symptom: reconstructions look soft and lose fine detail. The measurement: report distortion next to rate. In the toy, β = 1 gives D = 0.50 at a rate of 1.15 nats. Lower β to 0.2 and the rate rises to 1.96 nats while D falls to 0.10, five times smaller. The fix is a choice, not a trick: spend more rate, or give the decoder a likelihood that expects less noise. Which one is right depends on what the downstream task needs from the code.

In the toy, what happens to the best encoder as beta grows past the point where one unit of rate costs more than the distortion it saves?

Chapter 7

Latents for Prediction

Condition the VAE on what you know, keep the future out of the prediction path, and name the four objects that make up the model

Back to the car at the fork. You do not want a random picture. You want a picture of what happens next, given what you have seen so far and the action you plan to take. Your model must also keep both futures alive, left and right, and let you draw either.

A plain VAE has no input slot for “what I know”. This chapter adds one, and then shows the one trap the slide warns about: a model can score perfectly in training by using information it will not have in use.

The conditional VAE (slide 35)

Let c be the information available when you make a prediction. In the lecture’s example, c is the history plus a candidate action. Let x be the target, such as the next observation. A conditional variational autoencoder (CVAE) puts c into every part of the model.

ℒ(x, c)
The ELBO for the target x, given the context c.
Eq log pθ(x | z, c)
The decoder now sees both the code z and the context c.
KL( qφ(z | x, c) ‖ pθ(z | c) )
The encoder, which sees the answer, is pulled toward a prior that sees only the context.

The structure is the Chapter 3 ELBO with “given c” written everywhere. Doersch’s tutorial treats this case too: condition on an input to predict an output. Data flow, for a batch of B: the context c is [B, k] and the target x is [B, m]. The encoder reads both and outputs [B, d] means and log-variances. A second, smaller network reads only c and outputs the prior’s means and log-variances, also [B, d]. Per coordinate, the KL between two Gaussians is ½[ ln(σp² / σq²) + (σq² + (μq − μp)²) / σp² − 1 ]. With μp = 0 and σp = 1 it reduces to Chapter 4’s formula.

The leak: two different circuits

The slide splits the model into two moments, and the difference is the whole point.

MomentWhere z comes from
During trainingInfer z using both c and the observed target x. Example: c is the history and a candidate action.
At prediction timeSample z using c alone. The future target x is unavailable. Generate possible next observations conditioned on c.

Training may peek at the answer, because the encoder is only a tool for scoring the decoder. Prediction must not. Try the three ways to produce a future below. One of them cheats.

Predict the car at the fork

A hidden true future is chosen: left or right. Pick how the model predicts, then press Roll 20 futures. Compare where the warm paths land with the two real roads.

Toy scene, not the lecture’s data: two equally likely futures, each ending at its own road end. The prior-sampling mode draws z as a fair coin given c; the leak mode gives the encoder the hidden target. Rolls are random in your browser.

In Decoder mean mode every path ends at the exact middle, between the roads, on no road at all. That is Chapter 0’s ghost car again. In Sample z from c mode the paths split, roughly half to each road, and both are real futures. In Encoder sees x mode every path hits the true road, a perfect score, and it is worth nothing: that code was built from the answer. At prediction time there is no answer to give it.

The sound fix is a discipline in code. Use qφ(z | x, c) only to compute the training loss. At prediction time, sample z from pθ(z | c) and nothing else. If you change which one a test uses, state it, because the test is then answering a different question.

Latents inside a world model (slide 36)

Now connect to the course. In a world model each time step has an observation ot. The VAE gives a code for the present frame, zt ~ qφ(zt | ot). A recurrent network Fψ then summarises the earlier codes and actions into a memory, ht = Fψ(z<t, a<t). The code zt describes the present frame. The memory ht describes the past. The slide’s warning is that neither is automatically a sufficient state.

The slide gives a driving scene with three items. Roadside texture may account for many pixels, yet has little task relevance. The color of a traffic light is a small region of the image, yet can decide whether to stop. Vehicle velocity is not determined by a single position-only frame, but a history of frames can reveal motion. A reconstruction loss counts pixels. It cannot see which of these items matters. Static compression becomes a world-model component only when it is tied to temporal prediction.

The four objects (slide 37)

ObjectRole
p(z)Generate a latent before seeing an observation.
pθ(x | z)Specify an observation likelihood.
pθ(z | x)The true posterior induced by the generative model.
qφ(z | x)A tractable learned approximation to that posterior.

Which network do you need to generate a new, unconditional observation? The decoder and a prior sampler. The encoder is used for inference from observed data. The slide closes the VAE half of the lecture with a hand-off: next, learn a sampler through discrimination, without an explicit likelihood at all.

When it goes wrong. The symptom: a prediction model scores far better in your report than in use. The measurement: rerun the same test with z sampled from p(z | c). In the fork toy the leaking test matched the true future 20 times out of 20. The honest test matched about 10 out of 20, because both futures are valid and the context does not say which one comes. That is not a failure of the model. The fix is to score the sampled futures against the set of real futures, and to keep the encoder out of the test path.

A conditional VAE for next-frame prediction scores very well when its encoder is given the true next frame at test time. Why is that score misleading?

Chapter 8

Learning a Sampler by Playing a Game

Skip the likelihood, train a forger against a detective, and learn the one failure that sample quality alone cannot show

A forger paints fake banknotes. A detective inspects each note and says real or fake. At first the forger is bad, and the detective is never fooled. The forger sees which notes got caught and improves. The detective sees which fakes slipped through and sharpens. If the game goes well, the forger ends up painting notes that look exactly like the real ones.

Nobody in this story wrote down a probability. The forger never learned a formula for “how likely is this note”. It only learned to make notes the detective cannot reject. That is a generative adversarial network, a GAN.

Explicit versus implicit generation

The VAE half of the lecture is explicit: it has a likelihood pθ(x | z), and it trains on a bound of the likelihood. The GAN is implicit: it learns only a way to draw samples. Slide 38 writes the sampler in one line: z ~ p(z), x = Gθ(z). The generator G is a plain function from noise to data. It has no noise of its own and no density.

Two networks play. The discriminator D reads an x and outputs the probability that it is real. It learns to tell data samples from generated samples. The generator G changes its samples so that D is less able to tell. Goodfellow and colleagues wrote the game as one objective.

minG
The generator wants the whole expression as small as possible.
maxD
The discriminator wants it as large as possible. The two pull in opposite directions.
E log D(x)
Reward for calling a real sample real: the larger D(x) is on real data, the closer this term gets to 0.
E log(1 − D(G(z)))
Reward for calling a fake sample fake: the smaller D is on generated data, the closer this term gets to 0.

A worked number. Suppose D gives 0.9 on a real sample and 0.2 on a fake. The score is ln 0.9 + ln(1 − 0.2) = −0.105 − 0.223 = −0.33. That is a good score for D. If the fakes get better and D can only say 0.5 for everything, the score is ln 0.5 + ln 0.5 = −1.39, which is the value of the game when G is perfect. The slide’s training rule is plain: alternate D and G updates. And the basic formulation supplies a sampler but no posterior encoder q(z | x).

The ideal game, with a perfect detective

Real GANs train two networks by gradient steps. To see the shape of the game, we can idealise one side. Suppose the detective is always perfect for the current generator. For two known distributions the best D has a formula: D*(x) = pdata(x) / (pdata(x) + pg(x)), the share of the density that belongs to real data at that point. Put that D* into the objective, and what remains depends only on how different pg is from pdata.

Play against a perfect detective

Blue is the data, a standard bell. Warm is your generator. Pink is the best possible detective D*. Move the generator until the pink line goes flat at one half, or press Let G learn.

2.00
0.50

Idealised, not how a real GAN trains: the data is N(0, 1), the generator is N(mean, spread²), and the discriminator is assumed to be perfect at every step, so the value is the integral of pdata ln D* + pg ln(1 − D*), computed on a grid. It equals −ln 4 plus twice the Jensen-Shannon divergence. Let G learn follows the slope of that value.

With the generator far off, the detective can say “real” on one side and “fake” on the other, and the value is low. The pink curve swings between 0 and 1. As the generator slides onto the data, the pink curve flattens toward 0.5 and the value rises to −1.39. At a perfect match the best detective can do no better than a coin flip. That flat line is the sign that the forger has won.

Discrimination in two places (slide 39)

Both contrastive learning and GANs use a discrimination task, so slide 39 asks what differs: the label, and the final product.

Contrastive representation learningGAN training
Discrimination taskIdentify matching views among alternativesIdentify real versus generated samples
Trainable object of interestAn encoder that exposes useful featuresA generator that changes the sample distribution
Why alternatives matterThey shape invariance and separationThey provide a moving distribution to match
Typical evaluationFrozen features plus a specified downstream taskSample distribution, fidelity and coverage

Is the GAN’s discriminator then a good general-purpose encoder? It may learn useful features. The adversarial objective does not establish that by itself. The slide’s rule is worth keeping: a discrimination loss is a mechanism, and its labels and training loop decide its purpose.

The failure that looks like success (slide 40)

Now the GAN’s own way to fail. A generator can win each round by finding one kind of output that the detective cannot reject, and making only that. This is mode collapse: the generator produces convincing samples from too few parts of the data distribution.

The slide gives a constructed case. The data has red outcomes 50% of the time and blue outcomes 50% of the time. A collapsed generator gives red 100% of the time and blue 0%. Every red output might look valid, and half of the possible outcomes are still missing. The hero at the top of this page showed the same thing as a pile at +1.

Can a few good samples hide a collapse?

The data is half red, half blue. Set how often your generator makes red, and how many samples you check by eye. The curve shows the chance that every sample you check is red, so you would see nothing wrong.

100%
5

Exact arithmetic on the slide’s constructed example: data 50% red and 50% blue. The chance that n independent checked samples are all red is pn. The share of the data’s probability the generator covers is 1 − |p − 0.5|, which is 50% for the collapsed generator. The red outcomes are drawn in warm orange so that the red failure colour stays reserved for failure.

At 100% red, every sample passes any single-sample check, and you cannot tell by looking. Even at 90% red, checking 5 samples shows nothing wrong 59% of the time. The overlap with the data is 60%. So the slide’s lesson is that sample quality and distribution coverage are different axes. You need coverage tests, and for conditional models, tests of diversity given the same input.

A GAN generator makes only red outputs; the data is half red and half blue. Why does checking a handful of its samples by eye often miss the problem?

Chapter 9

World Models, Part One: Vision and Memory

Use a VAE as the eyes, a recurrent mixture network as the memory, and see why the memory must predict a spread of futures, not one

You are learning to drive a car in a racing game. Real laps are slow and every crash costs a restart. Now suppose you could shut your eyes and run the game in your head: you picture the track ahead, you picture what your steering will do, and you try ten thousand laps in your imagination for free.

To do that you need two skills. You must see the track in a form small enough to hold in your head, and you must know how that picture changes when you act. This chapter builds those two parts, and the lecture’s last section is about whether the dream can be trusted.

An old idea (slide 41)

The slide starts with Schmidhuber’s 1990 figure of a pair of networks. A model network predicts future sensory and reinforcement inputs. A control network uses the learned model to improve behaviour. The idea already ties together memory, prediction and action, the lecture says, and world modeling is a functional goal with a history beyond any one representation objective.

Three parts, V, M and C (slide 42)

The 2018 system by Ha and Schmidhuber, World Models, gives the pair three named parts. The slide lists them in one line each.

PartJob
V, VisionA VAE compresses each observation.
M, MemoryAn MDN-RNN predicts distributions over future latents.
C, ControllerA small policy acts from the learned features.

The slide adds a warning that is easy to miss: the VAE is one component of the system, not the whole world model.

V: the eyes (slide 43)

V is exactly the model you built in Chapters 1 to 4. Each frame ot goes through an encoder, zt ~ qφ(zt | ot), and a decoder pθ(ot | zt) rebuilds it. The slide shows the CarRacing game: a frame of green grass and a grey road, and the VAE’s reconstruction of it. V learns from collected observations with a reconstruction term and a prior KL, and nothing else.

The slide is honest about the cost. This objective does not directly ask which details affect future reward. Reconstruction creates a compact interface, and the slide says predictive usefulness remains an empirical question. It is the roadside-texture-versus-traffic-light worry of Chapter 7, now in a real system.

M: the memory (slides 44 and 45)

M must predict the next code, and the future is uncertain. A plain network trained with squared error would predict the average, the ghost car of Chapter 0. So M outputs a mixture density network (MDN): a small mixture of Gaussians. For each latent coordinate j, the network predicts K mixture weights, K means and K variances. A mixture is a distribution made by first choosing one of K bells by weight, then drawing from that bell.

pψ(zt+1,j | …)
The chance of the next value of coordinate j, given the current code zt, the action at and the recurrent memory ht.
∑k=1K
Add up the K bells.
πt,j,k
The weight of bell k: how likely the future is to follow that one. The weights add up to 1.
N(…; μ, σ²)
One bell, with its own centre and spread.

The recurrent memory is updated by ht+1 = fψ(ht, zt, at). For a batch of B and d latent coordinates, the network’s mixture output holds weights, means and variances for every coordinate and component: a tensor of shape [B, d, K, 3]. The slide notes that the released implementation models the coordinates independently given the recurrent context, so each of the d coordinates has its own small mixture. Try one coordinate.

Predict a spread of futures

The blue bars are 400 real next values from the same situation: usually near −1, sometimes near +1. Choose how many bells the memory may use, then press Sample 20 futures.

Toy data, not from the paper: 60% of values come from N(−1, 0.15²) and 40% from N(1, 0.15²), fixed seeded sample. The mixture is fitted by 60 steps of expectation-maximisation in your browser. The score is the average negative log-likelihood per value, in nats, lower is better. The sampled futures are random.

With K = 1 the single bell sits in the gap and spreads wide, a blurred average again. About half of its draws land between the peaks, where the data has almost no values. With K = 2 the bells sit on the peaks and the score drops sharply. A third bell has little left to explain. That is the memory M: not one answer, a shape of futures to sample from.

Three stages of training (slide 45)

StageTraining dataObjective
1 · VisionObservations collected from the environmentTrain the VAE.
2 · MemoryEncoded observations plus recorded actionsTrain the next-latent likelihood; freeze V.
3 · ControlInteraction using V and M featuresOptimise the task return; freeze V and M.

Stage 2 trains M with the loss LM = −∑t log pψ(zt+1 | zt, at, ht). V is frozen: its weights stop changing, so M always sees the same kind of codes. Stage 3 then freezes both. Each stage trains one small piece while the earlier ones stand still.

Look closely at the last line of the slide. Training histories contain observed latents. Imagined histories will contain model samples. A one-step likelihood objective must ultimately support repeated prediction. Chapter 10 is about what goes wrong when it does not.

When it goes wrong. The symptom: the predicted next frames look like a smear between two possible scenes. The measurement: score the memory with the negative log-likelihood. On the toy data one bell scored 1.41 nats per value and two scored 0.16, a nine-fold drop. A third bell also scored 0.16, so it bought nothing. The fix is to give M enough bells for the number of distinct futures you expect, and no more, because each extra bell adds parameters that the data cannot justify.

Why does the memory M output a mixture of Gaussians for each latent coordinate, instead of one predicted value?

Chapter 10

Imagination, Control and Exploitation

Let a controller act on the learned features, then train it inside the model’s dream, and learn where such a dream can lie

A chess player closes her eyes and plays out a plan: if I move here, he moves there, then I take the knight. She is not looking at a board any more. She is running a model of the game in her head. This works well as long as her mental model of the game is right. If she wrongly believes a piece can jump over another, she will love a plan that cannot work.

This chapter has the same two halves: what the learned model lets a controller do, and how it can be fooled.

Looking, or imagining (slide 46)

The slide compares two ways the system moves from one step to the next.

With a real observationInside the learned model
zt ~ qφ(zt | ot)zt+1 ~ pψ(· | zt, at, ht)
Infer a latent from a new sensor input.Sample a next latent from the transition model.
New observations can correct the history.No new observation corrects an erroneous rollout.

The controller takes the current code and the memory and picks an action: at = Cη(zt, ht). The memory then updates, ht+1 = fψ(ht, zt, at). A decoder can draw the imagined latents as pictures for a human to look at. The controller need not consume decoded pixels at all. It reads the latents directly.

The last row is the crux. When you look, the sensor pulls you back toward the truth every step. When you imagine, every small error feeds into the next prediction, and nothing pulls you back. Here is the smallest case that shows how fast this builds up.

Let a small error pile up

The real system carries 95% of its state forward each step. The learned model has one small, steady bias per step. Compare a model that looks each step with one that only imagines.

0.050
30

Illustrative linear toy, exact algebra, not the paper’s system: the true state follows s′ = 0.95 s + 0.5 a. The model adds a constant bias b every step. When the model starts each step from the real state, the error is b. When it starts from its own last guess, the error follows e′ = 0.95 e + b, so et = b (1 − 0.95t) / 0.05 and settles at 20 b.

With a bias of 0.050, the model that looks each step is off by exactly 0.050 at every step. The model that only imagines is off by 0.79 after 30 steps and heads for 1.00. The error is 20 times bigger in the long run. That is the slide’s remark made concrete: a one-step likelihood objective must ultimately support repeated prediction.

Control with latent features (slide 47)

The CarRacing experiment compares two controllers. One reads only zt. The other reads zt and the recurrent memory ht. The slide spells out what was done: the controller was optimised in the real environment, using the learned features. The paper’s optimiser was CMA-ES, a search method that tries many parameter settings and keeps the better ones. The slide is careful to add that this experiment is not online model-predictive control. The controller does not plan by rolling the model forward at every step. Its message: memory can help control even when the policy does not plan by explicit online rollouts.

Learning inside a dream (slide 48)

The VizDoom experiment goes further. Here the controller is trained inside a learned latent simulator, then tested in the real environment. M predicts the next latents and also whether the episode ends. Survival supplies the task objective. Observations and their VAE reconstructions look like game frames, and the controller that was trained in the dream is then transferred to the real VizDoom game. The slide names the stronger test: whether the behaviour learned inside the model works in the environment.

Where a dream can lie (slide 49)

A controller is optimised against the model it experiences. It will find the actions that make the model’s return as high as possible. If the model is wrong somewhere, those wrong places look like opportunities. The slide’s sentence: it may discover actions where that model is wrong, so good performance in an imagined environment can fail to transfer. The paper varied the sampling temperature to study robustness. The slide adds that more noise is not a general correction for model bias.

Fool the controller

Blue is the real return of each action. Warm is what the learned model says. The controller picks the highest warm point. Raise the model’s flaw and watch the pick jump to the flaw.

1.00

Illustrative, not the paper’s numbers: real return r(a) = 1 − (a − 1)², which peaks at 1.00 for a = 1. The model matches it except for a narrow bump of the size you set near a = 2.5, where it wrongly adds return. The controller takes the action with the highest imagined return.

With a flaw of 1.00 the controller still picks the good action near a = 1, and imagined and real returns agree. Push the flaw past about 2.25 and the pick jumps to a = 2.50: the imagined return reads high, while the real return there is −1.25. The slide’s rule follows directly: evaluate the model where the resulting policy takes it, not only on the original dataset. The dataset never visited that corner, so the usual test could not have found the flaw.

Three layers, three kinds of evidence (slide 50)

LayerQuestionEvidence
RepresentationWhat survives the encoder?Readouts under a stated task and budget.
DynamicsWhat follows this history and action?Conditional prediction and multi-step rollouts.
UtilityCan these predictions support behaviour?Planning or control in the environment.

JEPA-style features, VAE latents or other representations can all take part in a predictive system. Can you identify a world model from the encoder’s training loss alone? No. You must say how time and actions enter, what the system predicts, and how those predictions are used. World models are defined by predictive capability and use, and JEPA is one route to learning them.

When it goes wrong. The symptom: a controller that does well in the dream does badly outside it. The measurement: roll the model out along the controller’s own path and compare with the real environment, step by step. With a bias of 0.100 per step, the looking model stays at 0.10 while the imagining model drifts toward 2.00. The fix has two parts: refresh the model with real observations from the places the policy visits, and judge the controller by its return in the real environment, not by its return in the dream.

A controller trained inside a learned model scores very high in the dream and poorly in the real environment. Which explanation does the lecture give?

Chapter 11

The Field Guide

Review the lecture on one page: the three collapses, the numbers, a checklist, the core code, and what comes next

You can now say, from memory, what Lecture 7 argued. A model that gives one answer blurs the futures into their average. A latent-variable model draws a hidden cause first and decodes it, so it can produce many sharp answers. Scoring the data means summing over every cause, which is out of reach, so a variational encoder approximates the posterior and the ELBO turns the whole thing into a loss: reconstruction plus a KL to the prior. The reparameterization trick lets that loss train the encoder. The KL splits into a price for information and a price for shape. A GAN skips the likelihood and learns a sampler by playing a detective. A world model uses a VAE as its eyes and a mixture network as its memory, and its dream must be tested where the policy goes.

Three collapses, one habit (slide 51)

The lecture ties three failures together. They look alike from far away, but each loses something different, and each has its own test.

ContextWhat collapses?A diagnostic question
Noncontrastive representation learningDifferent inputs receive indistinguishable features.Do features vary and preserve useful information?
VAE posterior collapseThe latent carries little information about its input.Does q(z | x) depend on x? Does the decoder use z?
GAN mode collapseGenerated samples cover too little of the target distribution.Which valid outcomes are missing?

Is checking latent variance enough to rule out all three? No. Variance, input information and output coverage are different properties. The slide’s habit is the one to keep: name the random variable, the distribution, and the information that was lost. Try the habit on six cases.

Name the failure

Each case lists what a model’s diagnostics show. Pick the failure. One case is a trap: nothing has collapsed.

All six cases are made up to practise the habit; the numbers in them are illustrative, not measurements of any real model.

The numbers that matter

WhatNumber or formWhy it matters
Best single answer, targets −1 and +10 (slide 8)Squared error averages; the average may be no possible target.
Generationz ~ p(z), x ~ pθ(x | z)The decoder turns a simple latent into a flexible distribution.
Evidence∫ p(z) pθ(x | z) dzEasy to sample, hard to score.
The identitylog pθ(x) = ELBO + KL(q ‖ posterior)The ELBO is a lower bound; the gap is the inference error.
ELBO, two forcesEq log pθ(x | z) − KL(q ‖ p(z))Fit the data, stay near the prior.
Gaussian KL per coordinate½(μ² + eℓ − 1 − ℓ)Zero only when μ = 0 and ℓ = 0.
Reparameterizationz = μ + eℓ/2 ⊙ εMoves the randomness out of the gradient path.
KL splitI(X; Z) + KL(aggregate ‖ prior); 0 vs log 2 = 0.69 nats (slide 31)Same Gaussian heap, different information.
Rate and distortionmin D + βR; β = 1 is the ELBOHigher β squeezes the code.
GAN gameminG maxD E log D(x) + E log(1 − D(G(z)))A sampler with no likelihood and no encoder.
Mode collapse exampledata 50% / 50%, generator 100% / 0% (slide 40)Good samples, missing outcomes.
World Models, 2018V (VAE), M (MDN-RNN), C (controller)Three small parts, trained in three stages.
Toy results from this lesson (illustrative)ELBO 1.43 nats; β = 1 gives R 1.15, D 0.50; collapse near β = 10Worked on exact toy models, not the lecture’s data.

A checklist for any generative model

The core, as code

This sketch gathers the three trainable pieces of the lecture. decoder_distribution, D, G, normal_logpdf and logsumexp stand for whatever your library provides.

python# 1. VAE loss, Ch 4: x [B, ...] -> mu, logvar [B, d]
mu, logvar = encoder(x)
z = mu + exp(0.5 * logvar) * randn_like(mu)
nll = -decoder_distribution(z).log_prob(x).sum_over_event_dims()   # distortion D
kl  = 0.5 * (mu**2 + exp(logvar) - 1 - logvar).sum(-1)         # rate R
loss = (nll + beta * kl).mean()                                  # beta = 1 is the ELBO

# 2. GAN game, Ch 8: alternate the two updates
d_loss = -(log(D(x_real)) + log(1 - D(G(z).detach()))).mean()   # D maximises the game
g_loss = log(1 - D(G(z))).mean()                                # G minimises it

# 3. Memory M, Ch 9: logits, mean, logvar [B, d, K]; z_next [B, d]
logp = log_softmax(logits, -1) + normal_logpdf(z_next[..., None], mean, logvar)
loss_M = -logsumexp(logp, -1).sum(-1).mean()                     # next-latent likelihood, V frozen

Where this sits, and what comes next (slide 52)

Today’s result in three lines, from the slide: representations preserve selected information; VAEs and GANs learn distributions; World Models links prediction to control. The next steps it lists are Lecture 8, autoregressive prediction and rollout, then Lecture 11, the RSSM with its recurrent state, prior and inference, and evaluation of long-horizon behaviour. The closing slide previews the factorisation we will use: for a fixed candidate action sequence, p(o1:T | a1:T−1) = p(o1) ∏ p(ot+1 | o≤t, a≤t). One exit question: what does your model know at time t, and what can it predict under an action? A plausible next sample is only the beginning of a useful rollout.

Lecture 4’s state-space models and Lecture 6’s learned features both meet here. The state-space chain gave us p(z) p(x | z), and the representation lectures told us to ask what the code keeps.

Keep going

Course credit. This lesson re-teaches Lecture 7 of Penn CIS 6280 World Models (Fall 2026), taught by Jiatao Gu, with its readings by Lilian Weng and Carl Doersch. The slides are linked at the top of this page and in the footer. The devices and the toy models are ours, and each one says so in its fine print.

The takeaway. A world model must keep many futures alive, and every way of doing so can lose something quietly: information in the code, outcomes in the sampler, truth in the dream. Name what was lost, then measure it.
A model’s latent has variance 1.00 on every input, its samples all pass inspection, and its decoder output does not change when z changes. Which statement is right?

Now press Teach and explain, from memory, why a squared-error predictor blurs, how the ELBO is built, what the KL charges for, and how a GAN and a world model can fail. Then go back to the hero and watch the three models again.

CIS 6280 Lecture 7: Latent-Variable and Adversarial Models
The course site · Lecture slides · Weng, From Autoencoder to Beta-VAE · Doersch, Tutorial on VAEs · Back to Gleams