A model that gives one answer averages the possibilities into a blur. Latent variables and adversarial games let a model draw many sharp answers, and each way can fail in its own way.
Learn how a variational autoencoder and a GAN each turn simple noise into data, what the ELBO is, and how each one collapses.
Pick a model, then press Draw 50. The data has two outcomes, minus one and plus one, in equal shares. Watch where each model puts its draws. Then we build, piece by piece, the hidden cause, the bound, the encoder, the game, and the 2018 system that used them to dream.
You need a rough idea of what a probability is and a little calculus (what a slope is). We build the rest from zero.
Based on Lecture 7 of Penn CIS 6280: World Models (Fall 2026), taught by Jiatao Gu, and its two readings: Lilian Weng, From Autoencoder to Beta-VAE, and Carl Doersch, Tutorial on Variational Autoencoders. Lecture slides. Every idea is re-taught here in our own words and devices; it follows Lecture 6, Representation Learning II.
Toy target from the lecture’s constructed examples (slide 8: two equally likely values, minus 1 and plus 1; slide 40: a 50% and 50% data split). The latent decoder is x = tanh(4z) plus a little noise with z drawn from a standard normal; the collapsed GAN is a generator stuck at plus 1. Both are illustrative, and every draw is a real random draw in your browser.
Chapter 0
See why a model that gives a single prediction blurs the future, and why that sends us to models that can draw many answers
A small car drives toward a fork in the road. In the next video frame it will be on the left branch or on the right branch. Both futures are equally likely, and you cannot tell which from the frame you have.
You train a model to predict the next frame. Your loss is the usual one: the squared difference between the predicted pixels and the real pixels. After training, you look at its prediction. The car is gone from both branches. A pale ghost car sits in the middle of the wall between them.
The model did not fail. It did exactly what the loss asked. This chapter shows why, and it is the reason the whole lecture exists.
This is Lecture 7 of Penn CIS 6280, World Models. A world model is a learned model of an environment that can say what happens next, so an agent can plan inside it. Jiatao Gu teaches the course in Fall 2026. Lecture 5 and Lecture 6 asked a quieter question: when an encoder (a network that squeezes an input into a short list of numbers called a representation) is trained, what does the training signal make it keep?
Three signals came up. Reconstruction hides part of the input and asks the network to rebuild it, as the masked autoencoder MAE does. Contrastive learning shows two views of one image and asks the network to match them, while pushing other images away, as SimCLR does. Predicting learned features asks one network to predict what another network sees, as I-JEPA does. The slides stress one idea over and over: a lower loss does not mean a better representation until you say what the downstream task needs. A tiny traffic light may matter more than a large patch of road texture.
Slide 8 then makes a point that opens today. With a fixed target and squared error, the best single prediction is the conditional mean: the average of every target that could follow this input. The slide builds the smallest example. Two targets are equally likely, minus 1 and plus 1. The mean is 0, and 0 is not a possible target.
Try it. Drag the share of plus-1 outcomes, then drag your own guess along the loss curve.
The target is +1 with the chance you pick, otherwise minus 1. The yellow curve is the average squared error of one fixed answer. Drag your guess and look for the bottom of the bowl.
Exact: for a guess g the average squared error is p(g − 1)² + (1 − p)(g + 1)², and its minimum sits at g = 2p − 1, the mean of the targets (slide 8). The two-value target is the lecture’s constructed example.
Slide the chance to 50% and the bottom of the bowl lands on 0.00. No guess can do better, and every sharp guess pays. A guess of +1 costs 2.00 on average, because half the time it is two units away. The ghost car in the middle is the bottom of that bowl.
Not by itself. The slide gives two target spaces. In pixels, averaging two different pictures gives a blur. In learned features, averaging still happens if the features tell the two futures apart. A feature map can also throw away an unpredictable detail, such as the exact leaf texture, and then the average is harmless. You must decide whether that detail matters.
The slide ends with a careful question: must every predictor sample all possible outcomes? Only if the task needs those alternatives. A point prediction is fine when the uncertainty that matters is small. A world model that plans a drive is not in that case, because the fork is the whole point.
Today’s models do not give one answer. They give a distribution and let you draw from it. The lecture builds two ways to do that. A latent-variable model, the variational autoencoder (VAE), draws a hidden cause and then decodes it. A generative adversarial network (GAN) learns a sampler by playing against a critic. At the end, a 2018 system called World Models uses a VAE as its eyes. The hero above has already shown the three outcomes you will meet: a blur, a healthy spread, and a collapse onto one mode.
Slide 13 is a four-minute clip of Yann LeCun, 21:06 to 25:08 of an ETH Zurich talk, with two questions to keep in mind. Which details should a predictive representation keep or ignore? And what does a reconstruction objective ask z to retain? We will answer the second one with numbers in Chapters 5 and 6.
One paragraph on the course itself. Assignment 1 came out on Sep 10 and is due Sep 24 on Canvas. The Sep 15 session was tentatively virtual on Zoom, and the Sep 17 session was tentatively canceled, so check Canvas. The lecture plan gave 20 minutes to the recap and the LeCun video, 43 minutes to the VAE and its link to state-space models, 5 minutes to GANs and 15 minutes to World Models. Lecture 8 on Sep 22 covers autoregressive models.
Chapter 1
Draw a short hidden code first, decode it into an observation, and see why scoring a real observation is the hard part
Look at one photo of a face. The photo has a million pixels, but you can describe it with a few numbers: how the head turns, how the light falls, whether the mouth smiles. Change one of those numbers and a whole region of pixels changes together. The few numbers explain the many.
Those numbers are a latent variable: a hidden quantity that you never see directly, but that causes what you do see. Last week an encoder squeezed an image into features. This week we flip the arrow. We write down how a hidden cause generates an observation, and then ask how to find the cause from the observation.
You know this shape from Lecture 3 and Lecture 4. A state-space model has a hidden state s and an observation o. The state starts from a prior, p(s0), and each observation is drawn given its state, p(ot | st). Slide 15 keeps only the first time step. It then renames the state z and the observation x. What is left is a model of one image.
Generation is two steps. First draw a latent, z ~ p(z). Then draw an observation, x ~ pθ(x | z). Slide 16 states the payoff in one line: the decoder turns a simple latent distribution into a flexible distribution over observations. Doersch’s tutorial makes the same point: you can build almost any shape of data by pushing normal noise through a complicated enough function. Draw a few points and watch it happen.
The purple bell on top is the prior over one latent number z. The decoder maps z to a point on the warm curve, then adds noise. Press Draw 1 to see one full draw, or Draw 100 to see the shape of the data.
Toy decoder, not from the lecture: the mean of x is (z, 0.4z² − 0.4) and each coordinate gets Gaussian noise of the size you set. The prior is the lecture’s N(0, I) in one dimension. Draws are random in your browser.
Notice the two jobs. The prior is boring on purpose: one simple cloud that is easy to sample. All the interesting structure lives in the decoder. With noise at 0.00 every draw lands exactly on the curve. With more noise the points form a fuzzy band. That band is a distribution over observations that no simple formula could have written down directly.
A decoder does not output a picture. It outputs the parameters of a distribution over pictures. Slide 17 gives the common case, a Gaussian with a mean μθ(z) from the network and a fixed variance σx². Take the negative log of that Gaussian and the squared error appears.
A worked case with two pixels. The real image is x = (0.8, 0.2). The decoder’s mean is μ = (0.6, 0.5). The squared distance is 0.2² + 0.3² = 0.04 + 0.09 = 0.13. With σx² = 0.5 the loss is 0.13 / (2 × 0.5) = 0.13. With σx² = 0.1 the same mistake costs 0.13 / 0.2 = 0.65. A small assumed variance makes every error hurt more.
So squared error is a choice, not a law. It says: errors are Gaussian, all pixels share one variance, and the decoder only needs a mean. For binary pixels the matching choice is a Bernoulli, and the loss becomes binary cross-entropy: if the true pixel is 1 and the decoder says 0.8, the loss is −ln 0.8 = 0.22; if the true pixel is 0, the same 0.8 costs −ln 0.2 = 1.61. A richer distribution gives a different loss and a different assumption. The slide’s summary: reconstruction is a modeling choice about errors, noise and unresolved variation.
How should we train? Slide 18 gives the classic answer: choose theta to make the real data likely. For one real observation x, the model’s probability for it is every way z could have produced it, added up.
Sampling is easy: draw z, decode. Scoring is hard: that sum runs over all z. The slide asks the natural question. Could we just average pθ(x | z) over many prior samples? In principle yes. In practice, most prior samples explain a given x badly. The next device counts how fast that goes wrong as the latent grows.
Say an observation needs every one of its d latent numbers to land within 0.5 of 1.0. The curve is the exact chance that one prior draw does. Drag d, then press Draw 10,000.
Illustrative toy of the slide’s warning, not the lecture’s numbers. Per coordinate the chance of landing in (0.5, 1.5) under a standard normal is 0.2417; all d coordinates must land, so the chance is 0.2417d. The sampled count is a real random run in your browser.
At d = 1 about a quarter of prior samples help. At d = 6, about 1 in 5,000. At d = 10 the chance is about 6.8 in ten million, so ten thousand samples are expected to hit only 0.007 times. Real latents have tens to hundreds of numbers. A scorer built on blind prior samples would need astronomically many draws. That is the sentence on the slide: an easy sampler does not give an easy likelihood.
Chapter 2
Run the generator backwards, replace an impossible calculation with a fitting problem, and then train a network to do the fitting for you
You walk into an office and see a wet umbrella by the door. You did not see the weather. You can still ask: how likely is it that it rained? You start from what you believed before (it rains on 3 days in 10), and you update with what you saw (a wet umbrella is likely after rain, and unlikely after sun).
That update is posterior inference: from the observation back to the hidden cause. The posterior is the belief about the cause after you have seen the evidence.
Say the chance of a wet umbrella is 0.8 after rain and 0.1 after sun. Multiply each prior by its likelihood: rain gives 0.3 × 0.8 = 0.24, and sun gives 0.7 × 0.1 = 0.07. Add them: 0.31. That sum is the probability of the evidence itself. Divide each product by it. The posterior chance of rain is 0.24 / 0.31 = 0.77.
With two weather states the evidence is one addition. With a neural decoder and a latent of 64 numbers it is a 64-dimensional integral. In Lecture 3 and Lecture 4 you met a case where this worked: for a linear Gaussian state-space model, Gaussian conditioning gives the posterior in closed form, and that is the Kalman filter. A neural decoder breaks the closed form. The slide states the position plainly: we still want a distribution over plausible latents, but we cannot write it down.
The slide 20 move is to change the problem. Do not compute the posterior. Fit a simple distribution to it. First choose a family that is easy to handle, such as Gaussians with an adjustable mean μ and variance eℓ. Write qλ(z) with λ = (μ, ℓ). Then find the member of the family closest to the posterior.
“Closest” needs a ruler. The ruler is the KL divergence, KL(q ‖ p) = Eq[log q(z) − log p(z)]. It is the extra surprise, on average, when you believe p but the world follows q. It is never negative, and it is zero only when the two are the same. A worked case: let q = (0.5, 0.5) and p = (0.9, 0.1) over two outcomes. Then KL = 0.5 ln(0.5 / 0.9) + 0.5 ln(0.5 / 0.1) = 0.5 × (−0.588) + 0.5 × 1.609 = 0.51 nats. A nat is the unit you get from the natural logarithm.
The slide holds the model θ and the observation x fixed, then minimises over λ. Since this is an ordinary optimisation, you can run it for each x separately, and you do not need a neural network. Try it on a posterior that is not a Gaussian.
The blue shape is a posterior that is not a Gaussian. The pink bell is your q. Move its mean and its variance until the yellow KL number is as small as you can make it, or press Optimise.
Illustrative posterior, not from the lecture: 65% of N(−1.0, 0.45²) plus 35% of N(1.4, 0.35²). KL(q ‖ p) is computed numerically on a fine grid in your browser; Optimise searches a grid of (μ, eℓ) for the smallest value.
Two things appear. First, the best Gaussian settles on the taller bump and leaves the other bump uncovered. That is how this ruler behaves: KL(q ‖ p) punishes q for putting mass where p has none, so q hides in one mode. Second, the best KL is not zero. A Gaussian cannot be a two-bump shape. The family sets a floor on how good the approximation can be.
Per-input fitting has a cost. Each new observation starts its own optimisation loop. For 1,000 images and 100 steps each, that is 100,000 gradient steps before you have one useful code. Slide 21 replaces the loop with a shared inference network, the encoder qφ(z | x). It reads x and predicts the parameters in one forward pass: λφ(x) = (μφ(x), ℓφ(x)). Weng’s post uses the same names: qφ(z | x) is the “probabilistic encoder” and pθ(x | z) the “probabilistic decoder”.
The word for this is amortized: you pay the fitting cost once, during training, and spread it over every future input. The training objective is the same KL, now averaged over data and minimised over the encoder weights φ. For a batch of B inputs the data flow is plain: x of shape [B, ...] goes in, and two tensors of shape [B, d] come out, one of means and one of log-variances.
The price is the amortization gap: a shared network may fit one particular x worse than a dedicated loop would. This toy has an exact answer, so you can see the gap.
Blue is the exact answer for each x. Pink is what your encoder, a straight line, says. Drag its slope and offset to match, or press Train the encoder.
Illustrative linear-Gaussian toy with exact algebra: z ~ N(0, 1) and x | z ~ N(z, 0.25). The true posterior is N(0.8x, 0.2). The encoder predicts the mean as slope × x + offset and uses the exact variance 0.2, so the average KL is ((slope − 0.8)² × 1.25 + offset²) / 0.4 nats for x drawn from N(0, 1.25). The 1,000 inputs and 100 steps are made-up round numbers.
At the best setting (slope 0.80, offset 0.00) the gap is exactly zero, and the read-out shows what the shared network costs at inference: one pass per input, with no loop at all. A real encoder is not a line and a real posterior is not this tidy, so the gap there is never exactly zero. It is the price you accept for speed.
Chapter 3
Derive one exact identity, read it as a reconstruction term plus a prior term, and see what makes the bound tight
You want to raise log pθ(x), but you cannot compute it. A builder in this spot has one good trick: find a number you can compute that is never larger than the target. Push that number up, and the target is pushed up too, because it sits above.
A number that always lies below a target is a lower bound. This chapter builds one, and it is the objective every VAE trains.
Start with the quantity we already know how to talk about: the KL between the encoder’s qφ(z | x) and the true posterior. Write it as an average under q.
KL(q ‖ pθ(z | x)) = Eq[ log qφ(z | x) − log pθ(z | x) ].
We cannot evaluate the posterior inside the bracket. But Bayes’ rule from Chapter 2 says pθ(z | x) = pθ(x, z) / pθ(x). Take the log: log pθ(z | x) = log pθ(x, z) − log pθ(x). Put that in. The hard term log pθ(x) does not depend on z, so the average leaves it alone and it comes outside:
KL = Eq[ log qφ(z | x) − log pθ(x, z) ] + log pθ(x).
Give the first bracket a name, with its sign flipped. It is the ELBO, the evidence lower bound. (“Evidence” is the old word for pθ(x).) Rearranging gives the identity of the slide.
Because the gap is never negative, ELBO ≤ log pθ(x). That is the bound. And the gap is exactly the Chapter 2 quantity, so two goals line up. Raising the ELBO over θ raises the likelihood, and raising it over φ closes the gap.
Split the joint: log pθ(x, z) = log p(z) + log pθ(x | z). The ELBO becomes Eq[log p(z) + log pθ(x | z) − log qφ(z | x)]. Group the first and third terms and you meet a KL again:
Two forces, then. One says: fit the observation. The other says: stay compatible with the generative prior, so that sampling from the prior later still makes sense. If your last lectures used a squared-error reconstruction loss, it now sits inside a proper likelihood objective. The next device uses a toy where every term has a closed form, so you can check the identity with your own hands.
One observation x, and a Gaussian q you choose. The bar is the negative ELBO, split in two. The blue line is the true negative log-likelihood, the best the bar can do. Slide q onto the posterior and the gap shuts.
Illustrative linear-Gaussian toy with exact closed forms: z ~ N(0, 1), x | z ~ N(z, 0.25), so log p(x) = −½ ln(2π × 1.25) − x² / 2.5 and the posterior is N(0.8x, 0.2). For q = N(m, s²): the reconstruction term is −½ ln(2π × 0.25) − ((x − m)² + s²) / 0.5, and the prior KL is ½(m² + s² − 1 − ln s²). The gap is computed from log p(x) − ELBO and equals the posterior KL.
Try x = 1.00 and snap q to the posterior (mean 0.80, spread 0.45, so variance 0.20). Negative ELBO and negative log-likelihood both read 1.43 nats, and the gap is 0.00. Of that 1.43, about 0.71 is reconstruction cost and 0.72 is prior KL. Notice what the prior KL is at this perfect fit: not zero.
The slide separates two quantities that students tend to merge. The training penalty is KL(q ‖ p(z)): the encoder against the prior. The gap to the log-likelihood is KL(q ‖ pθ(z | x)): the encoder against the true posterior. When q equals the true posterior the second one vanishes and ELBO = log pθ(x).
Does a tight bound need the prior KL to be zero? No. The true posterior for this x is centred at 0.80, not at 0. It can sit well away from the prior, and the best q follows it. The slide’s takeaway: a tight ELBO means accurate inference for the current generative model. It says nothing about whether that model is good.
When it goes wrong. The symptom: the training loss falls, yet the model explains the data poorly. The measurement: split the negative ELBO into its parts and compare it with the likelihood, if you can. In the device, set x = 1.00 and leave q at mean 0.30 and spread 1.00. The negative ELBO is 3.25 nats against a true 1.43, a gap of 1.82. The prior KL is only 0.05, so the encoder is hugging the prior and ignoring x. The fix is to give the encoder what it needs to move: more capacity, or less pressure from the prior (Chapter 6).
Chapter 4
Turn the ELBO into a network you can train: Gaussian codes, a closed-form KL, the reparameterization trick and one training step
You now have a score to maximise, the ELBO. To train a network on it you need three practical things. The encoder must output a distribution, not a single code. The KL term must have a formula you can differentiate. And the random draw in the middle must not block the gradient. This chapter fixes each one.
A plain autoencoder has an encoder that squeezes x to a code z and a decoder that rebuilds x from z. Weng’s post walks through that family first, with its denoising, sparse and contractive variants. The variational autoencoder (VAE), slide 25, changes two things. The encoder outputs a distribution over z, and the decoder outputs a likelihood over x. Then it trains both by minimising the negative ELBO, an approximate maximum likelihood.
Slide 26 picks the simplest encoder distribution: a Gaussian with a diagonal spread. For d latent coordinates the encoder outputs two lists of d numbers, the means μ and the log-variances ℓ. The variance is eℓ. Why log-variance? A raw variance must stay positive, and a network output can be any number. The exponential makes any output a legal variance.
With the prior N(0, I), the prior KL has a closed form, one term per coordinate.
Worked example. One coordinate with μ = 1 and ℓ = 0 (variance 1): ½(1 + 1 − 1 − 0) = 0.50 nats. A tighter code with μ = 0 and ℓ = −2 (variance 0.135): ½(0 + 0.135 − 1 + 2) = 0.57 nats. Both cost something. Only μ = 0, ℓ = 0 costs nothing, and the slide asks what the KL prefers alone: exactly that, so every input would share the prior N(0, I). Move the two knobs and see.
Purple is the prior N(0, 1). Pink is the encoder’s code for one coordinate. The yellow number is its KL. Then raise d to see all the coordinates add up.
Exact: the closed form on the slide, ½(μ² + eℓ − 1 − ℓ) per coordinate, with every coordinate given the same μ and ℓ so the total is d times one coordinate.
The reconstruction term needs a sampled code: draw z from the encoder’s Gaussian, decode, compare. But a random draw is a dead end for backpropagation. The network can tell the decoder how to change, yet there is no path from the loss back through “pick a random number” to the encoder’s μ and ℓ. Weng’s post says it directly: sampling is a stochastic process, so we cannot backpropagate through it.
The reparameterization trick moves the randomness out of the way. Draw parameter-free noise ε ~ N(0, I) first. Then build the code as a plain formula of the encoder’s outputs: z = μ + eℓ/2 ⊙ ε. The result has exactly the same distribution as drawing from q. But now z is a differentiable function of μ and ℓ, and the noise is just an input. The slide draws the gradient path: reconstruction loss, to z, to (μ, ℓ), to φ.
The loss is (z − 3)². Set the encoder’s μ and ℓ, then press Draw 1,000 noise values. Each draw gives z = μ + eℓ/2ε and a gradient. The averages should match the exact bars.
Illustrative toy loss L = (z − 3)², chosen because its exact expectation is known: E[L] = (μ − 3)² + eℓ, so dE/dμ = 2(μ − 3) and dE/dℓ = eℓ. Each draw’s gradient uses the chain rule through z: dL/dμ = 2(z − 3) and dL/dℓ = 2(z − 3) × ½ eℓ/2ε. The draws are random in your browser.
A single draw gives a noisy gradient. A thousand draws, averaged, land close to the exact answer. A real VAE uses one draw per example per step and relies on the many examples in a batch to average the noise out. That is slide 28’s remark: one latent sample per example gives a Monte Carlo estimate of the reconstruction expectation.
Slide 28 shows the whole step in a few lines. The comments give the tensor shapes. B is the batch size and d the latent length.
python# x: [B, ...]; encoder outputs: [B, d] mu, logvar = encoder(x) # two tensors, each [B, d] eps = randn_like(mu) # parameter-free noise, [B, d] z = mu + exp(0.5 * logvar) * eps # reparameterized code, [B, d] nll = -decoder_distribution(z).log_prob(x).sum_over_event_dims() # [B] kl = 0.5 * (mu**2 + exp(logvar) - 1 - logvar).sum(-1) # [B] loss = (nll + kl).mean() # one number optimizer.zero_grad(); loss.backward(); optimizer.step()
Read it in four beats. The encoder reads the batch. The code is built from noise. The loss adds a reconstruction cost and a prior cost: the first sums over every pixel of the observation, the second over every latent coordinate, and the batch averages the two together. Then the usual backward pass updates both networks at once, because the noise is an input and not a block.
A trained VAE can run in two modes, and they test different things.
| Operation | Where z comes from | What it tests |
|---|---|---|
| Reconstruct an observed x | z ~ qφ(z | x), the encoder | Can a code inferred from x explain x? |
| Generate a new x | z ~ p(z), the prior | Do prior samples decode into plausible observations? |
Excellent reconstruction does not guarantee excellent prior samples. The encoder may place real images in regions of z that the prior visits rarely or differently, so decoding a fresh prior draw lands in a hole that training never covered. Evaluate prior generation separately.
When it goes wrong. The symptom: the KL term jumps to huge values, or the loss stops being a number at all, early in training. The measurement: print the log-variance. If an encoder output reaches ℓ = 10, then e10 = 22,026 and the KL for that one coordinate is ½(22,026 − 1 − 10), about 11,008 nats, enough to swamp every other term. One common guard is to bound ℓ to a modest range before the exponential. The deeper point is the one the formula shows: the KL punishes both a large mean and a spread far from 1.
Chapter 5
Split the prior KL into a price for information and a price for shape, and see how a code can look perfect and say nothing
Imagine two students who must each send you a code that describes a photo. The first sends a number drawn at random, with no link to the photo. The second sends a number that really depends on the photo. Seen from far away, both pile up in the same bell-shaped heap, because both numbers look like standard normal noise.
Only one of them told you anything. A good objective must be able to tell them apart. That is the question of this chapter.
Last week, LeJEPA used a term called SIGReg to keep learned features spread out and not collapsed. Slide 30 compares it with the VAE’s prior KL, because both try to organise the distribution of codes. Both pair a task loss with a latent regulariser: task loss plus latent distribution regularisation.
| SIGReg in representation learning | Prior KL in a VAE | |
|---|---|---|
| Task signal | Predict or agree across related views | Reconstruct the observed input |
| Latents | One feature vector per view | A distribution qφ(z | x) per input |
| Gaussian matching | Across inputs: the distribution of feature vectors | For each input: its conditional distribution against N(0, I) |
The slide calls the link close. VAE regularisation includes aggregate distribution matching, the same kind of thing SIGReg does. But the VAE’s per-input KL does a second job: it also penalises information that the code carries about x. We can pull those two jobs apart.
Define the aggregate posterior qφ(z) as the average of all the per-input codes: qφ(z) = Ex qφ(z | x). It is the heap you see from far away. Then the average prior KL splits exactly:
The slide builds a one-dimensional example in which the second bill is zero for both students, so only the first bill differs. In the first case the code ignores x: q(z | x) = N(0, 1) for every x. In the second, the code follows x: its mean varies with x (variance 3/4 across inputs), and each code has variance 1/4. Both produce the same aggregate, N(0, 1). The average VAE KL is 0 in the first case and log 2 = 0.69 nats in the second. That 0.69 nats is the price of the information the second code carries.
The next device parametrises both cases with one slider, a. The code is z = a · x + (noise). Choose the noise so z always keeps unit variance. Then a = 0 is the first student, and a = 0.866 (so that a² = 3/4) is the second.
Each dot is one input x (across) and its code z (up). The curve at the right is the heap of z. Drag a from 0 toward 1 and watch the dots line up while the heap stays put.
Built from the slide’s example: x ~ N(0, 1) and z | x ~ N(a x, 1 − a²), so the aggregate of z is N(0, 1) for every a. The average KL is −½ ln(1 − a²) nats, which is 0 at a = 0 and log 2 at a² = 3/4, matching the slide. The 300 points are a fixed seeded sample; the correlation and variance are measured on them.
At a = 0 the dots form a cloud with no pattern, the correlation is near 0, and the VAE KL is 0.00. At a² = 3/4 the dots follow a clear line, the heap of z is still a standard bell, and the KL is 0.69 nats. A test that only looks at the shape of the heap would give both a perfect score. The KL sees the difference, because it charges for information.
Now run training in the first student’s direction. If the encoder gives the same code N(0, 1) for every input, then qφ(z | x) ≈ p(z) and the mutual information is close to 0. This is posterior collapse. The code carries almost nothing about the input.
The decoder can then learn to ignore z. If pθ(x | z) = pθ(x), the model’s true posterior also equals the prior, pθ(z | x) = p(z), and the encoder that copies the prior is exactly right. The prior KL is zero, the gap is zero, and the model never uses its latent.
Why would training choose that? Look back at slide 26 and at Chapter 4: the KL alone is happiest at μ = 0, ℓ = 0. The reconstruction term must reward information strongly enough to pay for it. If the decoder can already model x well without z, there is nothing to pay for, and the cheap path wins. A decoder that is strong enough to explain x by itself makes that path easy.
A useful warning from the slide: can z still have nonzero variance during posterior collapse? Yes. z stays a Gaussian draw. The slide’s summary is exact: posterior collapse is loss of input information, not necessarily a constant latent vector. So checking that z is not constant does not rule it out.
Two diagnostic questions come from the slides. Does qφ(z | x) actually depend on x? And does the decoder use z? Measuring the average KL is the quick first check: a value near 0 nats, for a model that should be using its code, is the warning sign.
This also answers the question from the LeCun clip. A reconstruction objective asks z to keep whatever lowers the reconstruction cost by more than its KL price. It asks nothing about the traffic light or the texture. It only counts pixels.
Chapter 6
Read the VAE loss as a budget for a message, turn one weight to trade detail for compactness, and learn what a decoder’s picture really shows
You must describe a photo to a friend over a phone line that charges by the word. If you send many words, your friend can redraw the photo well, but the call is expensive. If you send few words, the call is cheap, and your friend draws a rough sketch. At some price, you stop describing the photo at all and say only “a photo”.
A VAE makes exactly this deal, and slide 33 writes it with two letters.
Distortion, D, is how badly the decoder redraws x from the code: D = E[−log pθ(x | z)], the reconstruction cost from Chapter 3. Rate, R, is the price of the message: R = Ex KL(qφ(z | x) ‖ p(z)), the prior KL. A rate of 0 nats is the empty message. The VAE minimises D + βR, with a weight β on the rate.
| Choice | Pressure on the solution |
|---|---|
| Higher β | A larger penalty for information and prior mismatch. |
| Lower β | More emphasis on reconstructing observations. |
| β = 1 | The standard negative ELBO for the specified likelihood. |
Weng’s post calls the weight on the KL term a hyperparameter and says that for β > 1 it applies a stronger constraint on the latent bottleneck and limits the capacity of z. It adds that a higher β may create a trade-off between reconstruction quality and the extent of disentanglement. The slide gives a sharper warning: choosing a bottleneck does not identify the right semantics automatically. What is useful for reconstruction may differ from what is useful for a task.
To see the whole curve, we need a model small enough to solve. Take data x ~ N(0, 1). Let the encoder send z = a·x plus noise of spread s, and let the decoder be the best straight-line guess of x from z. For a decoder that expects a noise variance σx² = 0.1, the formulas are exact. The rate is ½(a² + s² − 1 − ln s²). The distortion is [ s² / (a² + s²) ] / (2σx²), up to a constant. The device finds the best a and s for each β by searching a grid.
Each point on the curve is the best encoder for one β. Left is a cheap, poor message. Right is a rich, costly one. Drag β and watch the marker slide.
Illustrative linear-Gaussian toy, not the lecture’s numbers: x ~ N(0, 1), z = a x + s ε, decoder variance 0.1, best linear decoder. Rate and distortion are exact formulas; the best (a, s) for each β comes from a grid search in your browser. In this toy the optimum collapses at about β = 1 / 0.1 = 10.
At β = 1, the standard ELBO, the best encoder spends 1.15 nats of rate and pays a distortion of 0.50. At β = 4 it spends 0.46 nats and pays 2.00. The curve slopes down and to the right: more rate, less distortion, always a trade, never a free lunch.
Near β = 10 something sharp happens. The rate drops to 0.00 and the distortion jumps to 5.00, which is what you pay when the decoder ignores z entirely (1 / (2 × 0.1) = 5). That is posterior collapse from Chapter 5, now with a cause you can see: β made information too expensive. The same collapse can be reached from the other side. A decoder that expects large noise (a big σx²) makes each unit of distortion cheap, which is the same as raising β.
This explains a common practical move. If a VAE collapses, builders lower the weight on the KL, so information is cheap enough to keep. If they want a tighter, more compact code, they raise it.
Most VAE pictures you have seen are blurry. Slide 34 asks whether blur is a law of VAEs. The answer is no. To see why, separate three things that sound alike.
Pictures in papers are usually the first kind, because the mean looks cleaner than the noisy sample. That is a display choice. Blur is also not forced. It depends on how much information z carries (a latent that tells the two futures apart removes the averaging), on the likelihood family, on the architecture, and on optimisation. Chapter 7 puts this to work: a richer latent lets the decoder keep two futures apart.
When it goes wrong. The symptom: reconstructions look soft and lose fine detail. The measurement: report distortion next to rate. In the toy, β = 1 gives D = 0.50 at a rate of 1.15 nats. Lower β to 0.2 and the rate rises to 1.96 nats while D falls to 0.10, five times smaller. The fix is a choice, not a trick: spend more rate, or give the decoder a likelihood that expects less noise. Which one is right depends on what the downstream task needs from the code.
Chapter 7
Condition the VAE on what you know, keep the future out of the prediction path, and name the four objects that make up the model
Back to the car at the fork. You do not want a random picture. You want a picture of what happens next, given what you have seen so far and the action you plan to take. Your model must also keep both futures alive, left and right, and let you draw either.
A plain VAE has no input slot for “what I know”. This chapter adds one, and then shows the one trap the slide warns about: a model can score perfectly in training by using information it will not have in use.
Let c be the information available when you make a prediction. In the lecture’s example, c is the history plus a candidate action. Let x be the target, such as the next observation. A conditional variational autoencoder (CVAE) puts c into every part of the model.
The structure is the Chapter 3 ELBO with “given c” written everywhere. Doersch’s tutorial treats this case too: condition on an input to predict an output. Data flow, for a batch of B: the context c is [B, k] and the target x is [B, m]. The encoder reads both and outputs [B, d] means and log-variances. A second, smaller network reads only c and outputs the prior’s means and log-variances, also [B, d]. Per coordinate, the KL between two Gaussians is ½[ ln(σp² / σq²) + (σq² + (μq − μp)²) / σp² − 1 ]. With μp = 0 and σp = 1 it reduces to Chapter 4’s formula.
The slide splits the model into two moments, and the difference is the whole point.
| Moment | Where z comes from |
|---|---|
| During training | Infer z using both c and the observed target x. Example: c is the history and a candidate action. |
| At prediction time | Sample z using c alone. The future target x is unavailable. Generate possible next observations conditioned on c. |
Training may peek at the answer, because the encoder is only a tool for scoring the decoder. Prediction must not. Try the three ways to produce a future below. One of them cheats.
A hidden true future is chosen: left or right. Pick how the model predicts, then press Roll 20 futures. Compare where the warm paths land with the two real roads.
Toy scene, not the lecture’s data: two equally likely futures, each ending at its own road end. The prior-sampling mode draws z as a fair coin given c; the leak mode gives the encoder the hidden target. Rolls are random in your browser.
In Decoder mean mode every path ends at the exact middle, between the roads, on no road at all. That is Chapter 0’s ghost car again. In Sample z from c mode the paths split, roughly half to each road, and both are real futures. In Encoder sees x mode every path hits the true road, a perfect score, and it is worth nothing: that code was built from the answer. At prediction time there is no answer to give it.
The sound fix is a discipline in code. Use qφ(z | x, c) only to compute the training loss. At prediction time, sample z from pθ(z | c) and nothing else. If you change which one a test uses, state it, because the test is then answering a different question.
Now connect to the course. In a world model each time step has an observation ot. The VAE gives a code for the present frame, zt ~ qφ(zt | ot). A recurrent network Fψ then summarises the earlier codes and actions into a memory, ht = Fψ(z<t, a<t). The code zt describes the present frame. The memory ht describes the past. The slide’s warning is that neither is automatically a sufficient state.
The slide gives a driving scene with three items. Roadside texture may account for many pixels, yet has little task relevance. The color of a traffic light is a small region of the image, yet can decide whether to stop. Vehicle velocity is not determined by a single position-only frame, but a history of frames can reveal motion. A reconstruction loss counts pixels. It cannot see which of these items matters. Static compression becomes a world-model component only when it is tied to temporal prediction.
| Object | Role |
|---|---|
| p(z) | Generate a latent before seeing an observation. |
| pθ(x | z) | Specify an observation likelihood. |
| pθ(z | x) | The true posterior induced by the generative model. |
| qφ(z | x) | A tractable learned approximation to that posterior. |
Which network do you need to generate a new, unconditional observation? The decoder and a prior sampler. The encoder is used for inference from observed data. The slide closes the VAE half of the lecture with a hand-off: next, learn a sampler through discrimination, without an explicit likelihood at all.
When it goes wrong. The symptom: a prediction model scores far better in your report than in use. The measurement: rerun the same test with z sampled from p(z | c). In the fork toy the leaking test matched the true future 20 times out of 20. The honest test matched about 10 out of 20, because both futures are valid and the context does not say which one comes. That is not a failure of the model. The fix is to score the sampled futures against the set of real futures, and to keep the encoder out of the test path.
Chapter 8
Skip the likelihood, train a forger against a detective, and learn the one failure that sample quality alone cannot show
A forger paints fake banknotes. A detective inspects each note and says real or fake. At first the forger is bad, and the detective is never fooled. The forger sees which notes got caught and improves. The detective sees which fakes slipped through and sharpens. If the game goes well, the forger ends up painting notes that look exactly like the real ones.
Nobody in this story wrote down a probability. The forger never learned a formula for “how likely is this note”. It only learned to make notes the detective cannot reject. That is a generative adversarial network, a GAN.
The VAE half of the lecture is explicit: it has a likelihood pθ(x | z), and it trains on a bound of the likelihood. The GAN is implicit: it learns only a way to draw samples. Slide 38 writes the sampler in one line: z ~ p(z), x = Gθ(z). The generator G is a plain function from noise to data. It has no noise of its own and no density.
Two networks play. The discriminator D reads an x and outputs the probability that it is real. It learns to tell data samples from generated samples. The generator G changes its samples so that D is less able to tell. Goodfellow and colleagues wrote the game as one objective.
A worked number. Suppose D gives 0.9 on a real sample and 0.2 on a fake. The score is ln 0.9 + ln(1 − 0.2) = −0.105 − 0.223 = −0.33. That is a good score for D. If the fakes get better and D can only say 0.5 for everything, the score is ln 0.5 + ln 0.5 = −1.39, which is the value of the game when G is perfect. The slide’s training rule is plain: alternate D and G updates. And the basic formulation supplies a sampler but no posterior encoder q(z | x).
Real GANs train two networks by gradient steps. To see the shape of the game, we can idealise one side. Suppose the detective is always perfect for the current generator. For two known distributions the best D has a formula: D*(x) = pdata(x) / (pdata(x) + pg(x)), the share of the density that belongs to real data at that point. Put that D* into the objective, and what remains depends only on how different pg is from pdata.
Blue is the data, a standard bell. Warm is your generator. Pink is the best possible detective D*. Move the generator until the pink line goes flat at one half, or press Let G learn.
Idealised, not how a real GAN trains: the data is N(0, 1), the generator is N(mean, spread²), and the discriminator is assumed to be perfect at every step, so the value is the integral of pdata ln D* + pg ln(1 − D*), computed on a grid. It equals −ln 4 plus twice the Jensen-Shannon divergence. Let G learn follows the slope of that value.
With the generator far off, the detective can say “real” on one side and “fake” on the other, and the value is low. The pink curve swings between 0 and 1. As the generator slides onto the data, the pink curve flattens toward 0.5 and the value rises to −1.39. At a perfect match the best detective can do no better than a coin flip. That flat line is the sign that the forger has won.
Both contrastive learning and GANs use a discrimination task, so slide 39 asks what differs: the label, and the final product.
| Contrastive representation learning | GAN training | |
|---|---|---|
| Discrimination task | Identify matching views among alternatives | Identify real versus generated samples |
| Trainable object of interest | An encoder that exposes useful features | A generator that changes the sample distribution |
| Why alternatives matter | They shape invariance and separation | They provide a moving distribution to match |
| Typical evaluation | Frozen features plus a specified downstream task | Sample distribution, fidelity and coverage |
Is the GAN’s discriminator then a good general-purpose encoder? It may learn useful features. The adversarial objective does not establish that by itself. The slide’s rule is worth keeping: a discrimination loss is a mechanism, and its labels and training loop decide its purpose.
Now the GAN’s own way to fail. A generator can win each round by finding one kind of output that the detective cannot reject, and making only that. This is mode collapse: the generator produces convincing samples from too few parts of the data distribution.
The slide gives a constructed case. The data has red outcomes 50% of the time and blue outcomes 50% of the time. A collapsed generator gives red 100% of the time and blue 0%. Every red output might look valid, and half of the possible outcomes are still missing. The hero at the top of this page showed the same thing as a pile at +1.
The data is half red, half blue. Set how often your generator makes red, and how many samples you check by eye. The curve shows the chance that every sample you check is red, so you would see nothing wrong.
Exact arithmetic on the slide’s constructed example: data 50% red and 50% blue. The chance that n independent checked samples are all red is pn. The share of the data’s probability the generator covers is 1 − |p − 0.5|, which is 50% for the collapsed generator. The red outcomes are drawn in warm orange so that the red failure colour stays reserved for failure.
At 100% red, every sample passes any single-sample check, and you cannot tell by looking. Even at 90% red, checking 5 samples shows nothing wrong 59% of the time. The overlap with the data is 60%. So the slide’s lesson is that sample quality and distribution coverage are different axes. You need coverage tests, and for conditional models, tests of diversity given the same input.
Chapter 9
Use a VAE as the eyes, a recurrent mixture network as the memory, and see why the memory must predict a spread of futures, not one
You are learning to drive a car in a racing game. Real laps are slow and every crash costs a restart. Now suppose you could shut your eyes and run the game in your head: you picture the track ahead, you picture what your steering will do, and you try ten thousand laps in your imagination for free.
To do that you need two skills. You must see the track in a form small enough to hold in your head, and you must know how that picture changes when you act. This chapter builds those two parts, and the lecture’s last section is about whether the dream can be trusted.
The slide starts with Schmidhuber’s 1990 figure of a pair of networks. A model network predicts future sensory and reinforcement inputs. A control network uses the learned model to improve behaviour. The idea already ties together memory, prediction and action, the lecture says, and world modeling is a functional goal with a history beyond any one representation objective.
The 2018 system by Ha and Schmidhuber, World Models, gives the pair three named parts. The slide lists them in one line each.
| Part | Job |
|---|---|
| V, Vision | A VAE compresses each observation. |
| M, Memory | An MDN-RNN predicts distributions over future latents. |
| C, Controller | A small policy acts from the learned features. |
The slide adds a warning that is easy to miss: the VAE is one component of the system, not the whole world model.
V is exactly the model you built in Chapters 1 to 4. Each frame ot goes through an encoder, zt ~ qφ(zt | ot), and a decoder pθ(ot | zt) rebuilds it. The slide shows the CarRacing game: a frame of green grass and a grey road, and the VAE’s reconstruction of it. V learns from collected observations with a reconstruction term and a prior KL, and nothing else.
The slide is honest about the cost. This objective does not directly ask which details affect future reward. Reconstruction creates a compact interface, and the slide says predictive usefulness remains an empirical question. It is the roadside-texture-versus-traffic-light worry of Chapter 7, now in a real system.
M must predict the next code, and the future is uncertain. A plain network trained with squared error would predict the average, the ghost car of Chapter 0. So M outputs a mixture density network (MDN): a small mixture of Gaussians. For each latent coordinate j, the network predicts K mixture weights, K means and K variances. A mixture is a distribution made by first choosing one of K bells by weight, then drawing from that bell.
The recurrent memory is updated by ht+1 = fψ(ht, zt, at). For a batch of B and d latent coordinates, the network’s mixture output holds weights, means and variances for every coordinate and component: a tensor of shape [B, d, K, 3]. The slide notes that the released implementation models the coordinates independently given the recurrent context, so each of the d coordinates has its own small mixture. Try one coordinate.
The blue bars are 400 real next values from the same situation: usually near −1, sometimes near +1. Choose how many bells the memory may use, then press Sample 20 futures.
Toy data, not from the paper: 60% of values come from N(−1, 0.15²) and 40% from N(1, 0.15²), fixed seeded sample. The mixture is fitted by 60 steps of expectation-maximisation in your browser. The score is the average negative log-likelihood per value, in nats, lower is better. The sampled futures are random.
With K = 1 the single bell sits in the gap and spreads wide, a blurred average again. About half of its draws land between the peaks, where the data has almost no values. With K = 2 the bells sit on the peaks and the score drops sharply. A third bell has little left to explain. That is the memory M: not one answer, a shape of futures to sample from.
| Stage | Training data | Objective |
|---|---|---|
| 1 · Vision | Observations collected from the environment | Train the VAE. |
| 2 · Memory | Encoded observations plus recorded actions | Train the next-latent likelihood; freeze V. |
| 3 · Control | Interaction using V and M features | Optimise the task return; freeze V and M. |
Stage 2 trains M with the loss LM = −∑t log pψ(zt+1 | zt, at, ht). V is frozen: its weights stop changing, so M always sees the same kind of codes. Stage 3 then freezes both. Each stage trains one small piece while the earlier ones stand still.
Look closely at the last line of the slide. Training histories contain observed latents. Imagined histories will contain model samples. A one-step likelihood objective must ultimately support repeated prediction. Chapter 10 is about what goes wrong when it does not.
When it goes wrong. The symptom: the predicted next frames look like a smear between two possible scenes. The measurement: score the memory with the negative log-likelihood. On the toy data one bell scored 1.41 nats per value and two scored 0.16, a nine-fold drop. A third bell also scored 0.16, so it bought nothing. The fix is to give M enough bells for the number of distinct futures you expect, and no more, because each extra bell adds parameters that the data cannot justify.
Chapter 10
Let a controller act on the learned features, then train it inside the model’s dream, and learn where such a dream can lie
A chess player closes her eyes and plays out a plan: if I move here, he moves there, then I take the knight. She is not looking at a board any more. She is running a model of the game in her head. This works well as long as her mental model of the game is right. If she wrongly believes a piece can jump over another, she will love a plan that cannot work.
This chapter has the same two halves: what the learned model lets a controller do, and how it can be fooled.
The slide compares two ways the system moves from one step to the next.
| With a real observation | Inside the learned model |
|---|---|
| zt ~ qφ(zt | ot) | zt+1 ~ pψ(· | zt, at, ht) |
| Infer a latent from a new sensor input. | Sample a next latent from the transition model. |
| New observations can correct the history. | No new observation corrects an erroneous rollout. |
The controller takes the current code and the memory and picks an action: at = Cη(zt, ht). The memory then updates, ht+1 = fψ(ht, zt, at). A decoder can draw the imagined latents as pictures for a human to look at. The controller need not consume decoded pixels at all. It reads the latents directly.
The last row is the crux. When you look, the sensor pulls you back toward the truth every step. When you imagine, every small error feeds into the next prediction, and nothing pulls you back. Here is the smallest case that shows how fast this builds up.
The real system carries 95% of its state forward each step. The learned model has one small, steady bias per step. Compare a model that looks each step with one that only imagines.
Illustrative linear toy, exact algebra, not the paper’s system: the true state follows s′ = 0.95 s + 0.5 a. The model adds a constant bias b every step. When the model starts each step from the real state, the error is b. When it starts from its own last guess, the error follows e′ = 0.95 e + b, so et = b (1 − 0.95t) / 0.05 and settles at 20 b.
With a bias of 0.050, the model that looks each step is off by exactly 0.050 at every step. The model that only imagines is off by 0.79 after 30 steps and heads for 1.00. The error is 20 times bigger in the long run. That is the slide’s remark made concrete: a one-step likelihood objective must ultimately support repeated prediction.
The CarRacing experiment compares two controllers. One reads only zt. The other reads zt and the recurrent memory ht. The slide spells out what was done: the controller was optimised in the real environment, using the learned features. The paper’s optimiser was CMA-ES, a search method that tries many parameter settings and keeps the better ones. The slide is careful to add that this experiment is not online model-predictive control. The controller does not plan by rolling the model forward at every step. Its message: memory can help control even when the policy does not plan by explicit online rollouts.
The VizDoom experiment goes further. Here the controller is trained inside a learned latent simulator, then tested in the real environment. M predicts the next latents and also whether the episode ends. Survival supplies the task objective. Observations and their VAE reconstructions look like game frames, and the controller that was trained in the dream is then transferred to the real VizDoom game. The slide names the stronger test: whether the behaviour learned inside the model works in the environment.
A controller is optimised against the model it experiences. It will find the actions that make the model’s return as high as possible. If the model is wrong somewhere, those wrong places look like opportunities. The slide’s sentence: it may discover actions where that model is wrong, so good performance in an imagined environment can fail to transfer. The paper varied the sampling temperature to study robustness. The slide adds that more noise is not a general correction for model bias.
Blue is the real return of each action. Warm is what the learned model says. The controller picks the highest warm point. Raise the model’s flaw and watch the pick jump to the flaw.
Illustrative, not the paper’s numbers: real return r(a) = 1 − (a − 1)², which peaks at 1.00 for a = 1. The model matches it except for a narrow bump of the size you set near a = 2.5, where it wrongly adds return. The controller takes the action with the highest imagined return.
With a flaw of 1.00 the controller still picks the good action near a = 1, and imagined and real returns agree. Push the flaw past about 2.25 and the pick jumps to a = 2.50: the imagined return reads high, while the real return there is −1.25. The slide’s rule follows directly: evaluate the model where the resulting policy takes it, not only on the original dataset. The dataset never visited that corner, so the usual test could not have found the flaw.
| Layer | Question | Evidence |
|---|---|---|
| Representation | What survives the encoder? | Readouts under a stated task and budget. |
| Dynamics | What follows this history and action? | Conditional prediction and multi-step rollouts. |
| Utility | Can these predictions support behaviour? | Planning or control in the environment. |
JEPA-style features, VAE latents or other representations can all take part in a predictive system. Can you identify a world model from the encoder’s training loss alone? No. You must say how time and actions enter, what the system predicts, and how those predictions are used. World models are defined by predictive capability and use, and JEPA is one route to learning them.
When it goes wrong. The symptom: a controller that does well in the dream does badly outside it. The measurement: roll the model out along the controller’s own path and compare with the real environment, step by step. With a bias of 0.100 per step, the looking model stays at 0.10 while the imagining model drifts toward 2.00. The fix has two parts: refresh the model with real observations from the places the policy visits, and judge the controller by its return in the real environment, not by its return in the dream.
Chapter 11
Review the lecture on one page: the three collapses, the numbers, a checklist, the core code, and what comes next
You can now say, from memory, what Lecture 7 argued. A model that gives one answer blurs the futures into their average. A latent-variable model draws a hidden cause first and decodes it, so it can produce many sharp answers. Scoring the data means summing over every cause, which is out of reach, so a variational encoder approximates the posterior and the ELBO turns the whole thing into a loss: reconstruction plus a KL to the prior. The reparameterization trick lets that loss train the encoder. The KL splits into a price for information and a price for shape. A GAN skips the likelihood and learns a sampler by playing a detective. A world model uses a VAE as its eyes and a mixture network as its memory, and its dream must be tested where the policy goes.
The lecture ties three failures together. They look alike from far away, but each loses something different, and each has its own test.
| Context | What collapses? | A diagnostic question |
|---|---|---|
| Noncontrastive representation learning | Different inputs receive indistinguishable features. | Do features vary and preserve useful information? |
| VAE posterior collapse | The latent carries little information about its input. | Does q(z | x) depend on x? Does the decoder use z? |
| GAN mode collapse | Generated samples cover too little of the target distribution. | Which valid outcomes are missing? |
Is checking latent variance enough to rule out all three? No. Variance, input information and output coverage are different properties. The slide’s habit is the one to keep: name the random variable, the distribution, and the information that was lost. Try the habit on six cases.
Each case lists what a model’s diagnostics show. Pick the failure. One case is a trap: nothing has collapsed.
All six cases are made up to practise the habit; the numbers in them are illustrative, not measurements of any real model.
| What | Number or form | Why it matters |
|---|---|---|
| Best single answer, targets −1 and +1 | 0 (slide 8) | Squared error averages; the average may be no possible target. |
| Generation | z ~ p(z), x ~ pθ(x | z) | The decoder turns a simple latent into a flexible distribution. |
| Evidence | ∫ p(z) pθ(x | z) dz | Easy to sample, hard to score. |
| The identity | log pθ(x) = ELBO + KL(q ‖ posterior) | The ELBO is a lower bound; the gap is the inference error. |
| ELBO, two forces | Eq log pθ(x | z) − KL(q ‖ p(z)) | Fit the data, stay near the prior. |
| Gaussian KL per coordinate | ½(μ² + eℓ − 1 − ℓ) | Zero only when μ = 0 and ℓ = 0. |
| Reparameterization | z = μ + eℓ/2 ⊙ ε | Moves the randomness out of the gradient path. |
| KL split | I(X; Z) + KL(aggregate ‖ prior); 0 vs log 2 = 0.69 nats (slide 31) | Same Gaussian heap, different information. |
| Rate and distortion | min D + βR; β = 1 is the ELBO | Higher β squeezes the code. |
| GAN game | minG maxD E log D(x) + E log(1 − D(G(z))) | A sampler with no likelihood and no encoder. |
| Mode collapse example | data 50% / 50%, generator 100% / 0% (slide 40) | Good samples, missing outcomes. |
| World Models, 2018 | V (VAE), M (MDN-RNN), C (controller) | Three small parts, trained in three stages. |
| Toy results from this lesson (illustrative) | ELBO 1.43 nats; β = 1 gives R 1.15, D 0.50; collapse near β = 10 | Worked on exact toy models, not the lecture’s data. |
This sketch gathers the three trainable pieces of the lecture. decoder_distribution, D, G, normal_logpdf and logsumexp stand for whatever your library provides.
python# 1. VAE loss, Ch 4: x [B, ...] -> mu, logvar [B, d] mu, logvar = encoder(x) z = mu + exp(0.5 * logvar) * randn_like(mu) nll = -decoder_distribution(z).log_prob(x).sum_over_event_dims() # distortion D kl = 0.5 * (mu**2 + exp(logvar) - 1 - logvar).sum(-1) # rate R loss = (nll + beta * kl).mean() # beta = 1 is the ELBO # 2. GAN game, Ch 8: alternate the two updates d_loss = -(log(D(x_real)) + log(1 - D(G(z).detach()))).mean() # D maximises the game g_loss = log(1 - D(G(z))).mean() # G minimises it # 3. Memory M, Ch 9: logits, mean, logvar [B, d, K]; z_next [B, d] logp = log_softmax(logits, -1) + normal_logpdf(z_next[..., None], mean, logvar) loss_M = -logsumexp(logp, -1).sum(-1).mean() # next-latent likelihood, V frozen
Today’s result in three lines, from the slide: representations preserve selected information; VAEs and GANs learn distributions; World Models links prediction to control. The next steps it lists are Lecture 8, autoregressive prediction and rollout, then Lecture 11, the RSSM with its recurrent state, prior and inference, and evaluation of long-horizon behaviour. The closing slide previews the factorisation we will use: for a fixed candidate action sequence, p(o1:T | a1:T−1) = p(o1) ∏ p(ot+1 | o≤t, a≤t). One exit question: what does your model know at time t, and what can it predict under an action? A plausible next sample is only the beginning of a useful rollout.
Lecture 4’s state-space models and Lecture 6’s learned features both meet here. The state-space chain gave us p(z) p(x | z), and the representation lectures told us to ask what the code keeps.
Course credit. This lesson re-teaches Lecture 7 of Penn CIS 6280 World Models (Fall 2026), taught by Jiatao Gu, with its readings by Lilian Weng and Carl Doersch. The slides are linked at the top of this page and in the footer. The devices and the toy models are ours, and each one says so in its fine print.
Now press Teach and explain, from memory, why a squared-error predictor blurs, how the ELBO is built, what the KL charges for, and how a GAN and a world model can fail. Then go back to the hero and watch the three models again.