Score Matching & Guidance — Flow Matching & Diffusion (MIT 6.S184)

Chapter 0: Why — the Missing Object

The whole lecture in one sentence. In Lectures 1–2 you learned to transport Gaussian noise into data by following a vector field — a wind map of arrows. This lecture introduces a second wind map: the score, which points toward higher data density. That single extra object lets you (1) add controlled noise back into sampling (the SDE), (2) train your model a second way (score matching, the original "denoising" recipe), and (3) bend the model toward a prompt (guidance). Almost every image and video you have ever seen from a generative model was made with guidance built on the score.

Quick recap of where Lectures 1 and 2 left us. A flow model is an ODE: start at a random Gaussian point X₀, follow a learned vector field u^θ_t(x) — the arrows — until time t = 1, and the endpoint X₁ looks like a data sample. Lecture 2 told us how to learn those arrows: pick a Gaussian conditional probability path that drags a clean data point z toward noise, and regress the network onto the conditional vector field u_t(x|z). The Gaussian path is

p_t(x | z) = N(x; α_t z, β_t² I_d)

Read that as: "at time t, given the clean data point z, the noisy sample x is Gaussian, centered at a shrunken copy α_tz of the data, with spread β_t." The schedules satisfy α₀ = 0, α₁ = 1 (start with no data, end at full data) and β₀ = 1, β₁ = 0 (start at full noise, end with none). At t = 1 the sample is exactly z; at t = 0 it is pure N(0, I).

So what's missing? The vector field tells you which way to move. It does not, by itself, tell you which way the probability mass rises. Those are different questions. "Move toward t = 1" is dynamics; "move toward where data is dense" is geometry of the distribution. The object that answers the second question is the score function — the gradient of the log-density. This lecture is the story of that object and the three superpowers it grants.

Here is the roadmap, mapped onto the slides:

The three jobs of this lecture.

Score & score matching (Sec. 4). Define the score, find its closed form for the Gaussian path, and learn it the same way we learned the vector field. We discover the network is secretly predicting the noise that corrupted the data — the "denoising" in denoising diffusion.
SDE sampling (Sec. 4.2). The score is the key that turns a deterministic ODE into a noisy SDE: add a drift of (σ²/2) times the score, plus Brownian noise. Same destination, more robust journey.
Guidance (Sec. 5). How to make the model obey a prompt y. Vanilla (classifier) guidance adds the gradient of a classifier; classifier-free guidance — the workhorse of real systems — combines a conditional and unconditional model.

We will end by building a guided 2D sampler with a guidance-scale slider, watching samples snap onto a target cluster — and over-saturate when you crank the scale too high, exactly the artifact you see in real over-guided images.

Lecture 1 gave us the vector field (which way to move) and a way to sample. What new object does this lecture introduce, and why do we need it?

A faster GPU; the math is unchanged. The score — the gradient of the log-density, pointing toward higher data density — which enables SDE sampling, a second training recipe, and guidance. A bigger dataset; the score is just more data. A replacement for the vector field that makes it obsolete.

Chapter 1: The Score — Arrows Toward Density

Forget dynamics for a moment. Take any probability distribution with density q(x) over points x in space. The score function is defined as the gradient of its log-density:

score(x) = ∇_x log q(x)

Let's unpack every piece, because the score is one of those objects that sounds abstract and is actually dead simple once you picture it.

Reading the symbols. q(x) is the density — how much probability mass sits at point x (high where data is common, near zero in the wasteland). log q(x) is just its logarithm — same peaks and valleys, gentler slopes. ∇_x (the gradient, "nabla") collects the partial derivatives with respect to each coordinate of x; it is a vector pointing in the direction of steepest increase. So the score, ∇_x log q(x), is an arrow at every point that points uphill toward higher (log-)density — toward where the data is.

The analogy that makes it stick: imagine the density as a mountain range, with peaks where data clusters and flat plains where there is nothing. Stand anywhere and ask "which way is uphill?" That answer — a direction and a steepness — is the score. On a steep slope near a cluster's edge it is a big arrow pointing inward; on a flat plain or right at a peak it is tiny.

Why the log, not the density itself? Two reasons. First, the gradient of the log strips off the overall normalization: since log(c·q) = log c + log q and log c is a constant, ∇ log q does not care whether q is properly normalized. That is enormous — we almost never know the normalizing constant of a real distribution, and the score lets us ignore it. Second, densities span many orders of magnitude (a data peak can be astronomically more likely than the plains); the log compresses that range into something a network can actually represent.

The score points the way a sampler wants to walk

Here is the punchline that motivates the rest of the lecture. If you are a particle and you want to drift toward where data lives, you follow the score uphill. That is exactly the noise-injecting dynamics (Langevin dynamics) we will build in Chapter 4. The score is the compass needle of every diffusion sampler.

The score field of a two-cluster density

The shaded heat is a density q(x) — a mixture of two Gaussian clusters (brighter = more data). The arrows are the score ∇ log q(x): at every point they point toward the nearest cluster center (uphill in density), shrinking to nothing exactly at the peaks. Drag cluster gap to pull the clusters apart and watch the arrows in the valley flip allegiance. This is the compass a sampler follows to find data.

cluster gap1.30

Now connect this to our generative setting. Lecture 2 worked with a time-indexed family of densities: the conditional probability path p_t(x|z) (the noised version of one data point z) and the marginal probability path p_t(x) (the noised version of the whole dataset). Each is a density, so each has a score:

conditional score = ∇_x log p_t(x | z) marginal score = ∇_x log p_t(x)

And just as the marginal vector field is an average of conditional vector fields over the data, the marginal score is a (posterior-weighted) average of conditional scores — the same averaging structure. That parallel is what makes everything in Chapter 3 work: the trick that let us learn the vector field will, line for line, let us learn the score.

What does the score ∇ log q(x) represent geometrically, and why use the log?

The height of the density; the log just makes it positive. The time-derivative of the trajectory; the log speeds up sampling. An arrow at every point pointing uphill toward higher density (toward data); the log strips off the unknown normalizing constant and compresses the dynamic range. The number of data points; the log counts them.

Chapter 2: The Conditional Score, Solved Exactly

Abstract scores are nice, but we can only train what we can write down. For the Gaussian path — the path that powers essentially every diffusion model — the conditional score has a clean closed form. Let's derive it from scratch, because the derivation is short and it makes the formula impossible to forget.

Recall the conditional path is a Gaussian centered at the shrunken data point with spread β_t:

p_t(x | z) = N(x; α_t z, β_t² I_d)

Step 1: write down the Gaussian log-density

A d-dimensional isotropic Gaussian with mean μ and variance β_t² in each coordinate has density proportional to exp of a negative squared distance. Its log is:

log p_t(x | z) = −¹⁄_{2β_t²} · ||x − α_t z||² + C

where ||x − α_tz||² is the squared distance from x to the mean α_tz, and C collects everything that does not depend on x (the normalizing constant and the log of β_t). We are about to take a gradient in x, so C is going to vanish — this is exactly the "score ignores normalization" property from Chapter 1, in action.

Step 2: take the gradient in x

Differentiate with respect to x. The constant C drops. For the quadratic, recall the one-line rule ∇_x ||x − a||² = 2(x − a) (the multivariable analogue of d⁄dx (x − a)² = 2(x − a)). With a = α_tz:

∇_x log p_t(x | z) = −¹⁄_{2β_t²} · 2(x − α_t z) = −^{(x − α_t z)}⁄_{β_t²}

The conditional score of a Gaussian path.

∇_x log p_t(x | z) = −^{(x − α_t z)}⁄_{β_t²}

Read it as a force: the arrow points from the noisy sample x back toward the mean α_tz (the shrunken clean data), and its strength is the displacement divided by the variance β_t². It is a restoring spring pulling you toward where the data is. The two ingredients you must get right: the minus sign (toward the mean, not away) and the division by β_t² (a tighter cloud pulls harder).

Sanity-check the two extremes

Sign: if x is to the right of the mean (x − α_tz > 0), the score is negative — it pushes you left, back toward the center. Correct: density is higher near the mean. Scale: early in the path β_t is large (lots of noise), so the spring is weak — everything is vague. Late in the path β_t → 0, so 1⁄β_t² blows up — the score becomes a razor-sharp pull onto the exact data point. (That blow-up is real and will bite us numerically in Chapter 3.)

The hidden gift: vector field and score are the same information. In Lecture 2 the conditional vector field for this same Gaussian path was also a linear function of x and z. Two linear functions of the same variables can always be rearranged into each other. Concretely, for the Gaussian path there are scalars a_t, b_t (built from α_t, β_t and their time-derivatives) with

u_t(x | z) = a_t · ∇_x log p_t(x | z) + b_t · x

So once you have learned the vector field, you have also learned the score — for free — and vice versa. This "conversion formula" (Proposition 1 in the notes) is why early diffusion models could learn the score and later flow models could learn the vector field and end up at the same place. They are two coordinates on one object.

Build this formula with your hands and verify it numerically against the definition (finite differences of the log-density) in the lab below.

For the Gaussian path, the conditional score is −(x − α_tz)/β_t². What are the two non-negotiable features of this formula?

It is positive and grows with x. It points away from the mean and ignores β_t. The minus sign (it points toward the mean α_tz) and the division by β_t² (a tighter cloud pulls harder); it vanishes exactly at the mean. It is constant in x, equal to z.

Chapter 3: Score Matching = Predicting the Noise

We want a network s^θ_t(x) that outputs the marginal score ∇ log p_t(x) at any noisy x and time t. We cannot regress onto the marginal score directly — we never have a formula for it (it averages over the whole unknown data distribution). But Lecture 2 already taught us the escape hatch, and it transfers here unchanged.

The same trick as flow matching. The marginal score is a posterior-weighted average of conditional scores, which we do know in closed form. So instead of regressing onto the unknowable marginal score, regress onto the known conditional score for a freshly sampled data point z. The two losses below differ only in which score you target:

L_SM(θ) = E [ || s^θ_t(x) − ∇ log p_t(x) ||² ]

L_CSM(θ) = E [ || s^θ_t(x) − ∇ log p_t(x | z) ||² ]

The remarkable theorem (Theorem 22): these two losses have identical gradients — they differ only by a constant independent of θ. So minimizing the tractable conditional one (L_CSM, "denoising score matching") drives the network to the true marginal score. Same proof as the flow-matching theorem, with the score swapped in for the vector field.

Plug in the Gaussian score and a beautiful thing happens

Here is the step that gave "denoising diffusion models" their name. We never sample x out of thin air — we build it from a clean data point z and a fresh noise draw ε, following the Gaussian path:

x = α_t z + β_t ε, ε ∼ N(0, I_d)

Now substitute this x into the conditional score from Chapter 2. Watch the cancellation:

∇ log p_t(x | z) = −^{(x − α_t z)}⁄_{β_t²} = −^{(α_t z + β_t ε − α_t z)}⁄_{β_t²} = −^{β_t ε}⁄_{β_t²} = −^ε⁄_{β_t}

The target score is just the noise that corrupted the data, divided by −β_t. The data point z cancelled out completely — what remains is ε, the exact noise vector you injected. So to match the score, the network must learn to recover the noise you added. That is literally "denoising." Many implementations skip the 1⁄β_t scale and train a noise predictor ε^θ_t(x) directly with the loss || ε^θ_t(α_tz + β_tε) − ε ||². Score, noise-prediction, and denoiser are three reparameterizations of one quantity.

The numerical landmine. Notice the target is −ε/β_t. As t → 1, β_t → 0 and that target blows up — the score becomes enormous and the regression becomes numerically unstable. This is exactly why early denoising diffusion models (DDPM) dropped the 1⁄β_t² weight and predicted the raw noise ε instead: the noise ε is a tame unit-variance target, while the score it implies is not. "Predict the noise" is not just elegant — it is numerically the right thing to regress.

The training loop, in five lines of pseudocode

# Denoising score matching (Algorithm 4)
for each minibatch:
    z = sample_data()                 # a clean data point z ~ p_data
    t = uniform(0, 1)                 # a random time
    eps = sample_gaussian()           # the noise, eps ~ N(0, I)
    x = alpha_t * z + beta_t * eps    # the corrupted sample
    loss = || eps_theta(x, t) - eps ||^2   # predict the noise you added
    backprop_and_step(loss)

That is the entire training procedure for a denoising diffusion model. No SDEs, no scores explicitly — just "corrupt a data point with known noise, and train a network to guess that noise." Build it and confirm that a fitted noise-predictor lowers the loss and that its implied score matches the analytic one.

When you plug x = α_tz + β_tε into the Gaussian conditional score, what does the network end up having to predict, and why is that name "denoising"?

The clean data z; it memorizes the dataset. The injected noise ε (the score equals −ε/β_t); recovering the noise that corrupted the sample is denoising. The time t; it learns a clock. The constant C from the log-density.

Chapter 4: From Flow to SDE — the Score Is the Key

Lecture 1 sampled with a deterministic ODE (a flow). But diffusion models are stochastic — they inject fresh noise at every step. Why would you want noise, and how do you add it without destroying the result? The answer is the score, and it is one of the most elegant facts in the whole subject.

The SDE extension trick (Theorem 17). Take the ODE that follows the marginal probability path, with marginal vector field u_t(x). For any diffusion coefficient σ_t ≥ 0 you choose, you can convert it into an SDE that lands on the same distribution p_t at every time — provided you add a correction to the drift equal to (σ_t²/2) times the marginal score:

dX_t = [ u_t(X_t) + ^σ_t²⁄₂ ∇ log p_t(X_t) ] dt + σ_t dW_t

The square brackets are the new drift; σ_t dW_t is the Brownian noise (Lecture 1). Set σ_t = 0 and the correction and noise both vanish — you are back to the plain ODE. The score is exactly the extra ingredient that lets you dial in noise.

Why this particular correction? The intuition

Adding raw Brownian noise to an ODE would smear the distribution — it would diffuse mass outward, making p_t fatter than intended. The score correction is a counter-force: it pushes mass back uphill toward where the density should be dense, precisely cancelling the smearing. Noise spreads things out; the score (pointing toward density) pulls them back in. Balance the two and the marginal distribution is preserved exactly — you get extra randomness in which path a sample takes, with no change in where the samples end up.

Where this comes from (the Fokker–Planck equation, in one breath). An ODE's distribution obeys the continuity equation (mass is conserved as it flows). An SDE's distribution obeys the Fokker–Planck equation, which adds one term: a Laplacian (σ_t²/2)Δp_t — the same diffusion term that appears in the physics of heat spreading through a metal bar. The score correction is exactly what you must add to the drift so that the flow term and the new heat-diffusion term cancel, leaving p_t unchanged. You do not need to prove it to use it — just remember: noise term in, score correction to cancel it.

The famous special case: Langevin dynamics

Take the SDE trick and freeze the target — let the probability path be a single fixed distribution p (so u_t = 0, nothing to flow toward in time). What's left is pure score-following plus noise:

dX_t = ^σ²⁄₂ ∇ log p(X_t) dt + σ dW_t

Langevin dynamics. A particle drifts uphill along the score (toward high density) while being jostled by noise. Run it long enough and the distribution of particles converges to p itself — p is the stationary distribution. This is the engine behind molecular-dynamics simulations and a huge family of Markov-chain Monte Carlo methods across the sciences. For us it is the cleanest picture of what the score is for: it is the compass that, combined with noise, lets you wander into — and sample from — any distribution whose score you know. (The Ornstein–Uhlenbeck process from Lecture 1 is the special case where p is a Gaussian.)

You'll watch Langevin dynamics pull a cloud of particles into a two-cluster target in the showcase. The takeaway for now: the score converts a flow into a diffusion. ODE and SDE are two ways to traverse the same probability path; the score is the bridge between them.

Langevin dynamics: score + noise fills a target distribution

Particles (dots) start clumped at the origin and evolve by the Langevin SDE toward a two-cluster target (shaded). Each step drifts uphill along the score and adds a Brownian kick. Drag noise σ: too little and particles get stuck in one cluster; just right and they explore and fill both clusters in the correct proportions. Press Run to re-seed. This is the SDE trick with the vector field switched off — pure score-following.

noise σ0.70

"ODE and SDE traverse the same path, the score is the bridge" is not a slogan — for a Gaussian path it is an exact algebraic identity. Prove it: rebuild the velocity field from the score alone.

To turn the ODE dX = u_t(X)dt into an SDE that follows the same probability path p_t, what must you add, and what is its purpose?

Nothing — you just add noise σdW and the path is automatically preserved. A larger learning rate, to compensate for the noise. A drift correction (σ²/2)∇ log p_t — the score pushes mass back toward high density to exactly cancel the smearing that the Brownian noise would otherwise cause. A second neural network trained from scratch.

Chapter 5: Sampling With SDEs — and Why Noise Helps

We now have a sampler. Recall the Euler–Maruyama step from Lecture 1: take the deterministic drift step and a scaled Brownian kick. For diffusion sampling the drift is the corrected drift from Chapter 4:

drift_t(x) = u^θ_t(x) + ^σ_t²⁄₂ s^θ_t(x)

X_t+h = X_t + h · drift_t(X_t) + σ_t · √h · ε, ε ∼ N(0, I_d)

Here u^θ_t is the learned flow field and s^θ_t is the learned score. For a Gaussian path you don't even need two networks: by the conversion formula of Chapter 2 you can build the score from the vector field (or vice versa) and plug it into the same step. The √h on the noise term is the Brownian-motion scaling from Lecture 1 — variance grows like elapsed time, so the per-step standard deviation grows like the square root of the step.

Algorithm: SDE sampling (Algorithm 2 with score)

# Sample a diffusion model with Euler-Maruyama
# Input: flow net u_theta, score net s_theta, noise schedule sigma_t, steps n
t = 0.0
h = 1.0 / n
x = sample_gaussian()                       # X_0 ~ p_init = N(0, I)
for i in range(n):
    drift = u_theta(x, t) + 0.5*sigma_t**2 * s_theta(x, t)
    x = x + h*drift + sigma_t*sqrt(h)*sample_gaussian()   # one EM step
    t = t + h
return x                                   # X_1 ~ p_data (approximately)

Set σ_t = 0 and this collapses to the Lecture-1 ODE sampler: no score correction, no noise, just the flow field. Crank σ_t up and you inject more stochasticity per step.

So why add noise at all, if every σ_t reaches the same distribution in theory? Because in practice we are never perfect:

Training error. The network only approximates the true vector field / score. The ODE faithfully follows your imperfect field and can drift off into low-density regions with no way back.
Simulation error. Finite step size h leaves discretization error every step, and it accumulates.
Self-correction. The noise term keeps re-randomizing the sample, and the score correction keeps pulling it back toward high density — so small errors get washed out rather than compounding. SDE sampling can "heal" mistakes the ODE would lock in.

The honest caveat — noise is an option, not a mandate. Bigger σ_t also amplifies simulation error (you need smaller steps to integrate a noisier SDE accurately), so there is an empirically optimal σ_t for a given trained model and compute budget. The existence of a "best σ" is an artifact of imperfect training and finite steps, not a deep truth about the continuous dynamics. The practical wisdom from the course: ODE sampling (σ = 0) often gives the best results; SDE sampling is a tool you reach for when you need diversity, robustness, or stochastic search — not a default.

You already built the Euler–Maruyama step in Lecture 1 and the score in Chapter 2 — the diffusion sampler is just those two snapped together with the (σ²/2)·score correction. The showcase in Chapter 8 lets you watch ODE-vs-SDE side by side; for now, lock in the structure.

Build the simplest member of the family: Langevin dynamics — the SDE sampler with the vector field switched off (u = 0), so a known score alone drives the sampling.

In theory every diffusion coefficient σ_t ≥ 0 samples the same data distribution. Why might a practitioner still add noise (σ_t > 0)?

Noise makes the network train faster. Because of training and simulation error, the noise + score correction can self-correct errors that a deterministic ODE would lock in — improving robustness/diversity in practice. Noise is required by the Fokker–Planck equation; σ = 0 is illegal. Larger σ always gives strictly better samples.

Chapter 6: Conditioning on a Prompt — Vanilla Guidance

Everything so far generates some object — "make an image." But you usually want a specific kind of object — "make an image of a cat baking a cake." That extra information is a prompt y (a class label, a text caption). We want to sample from the guided data distribution p_data(· | y): images that fit the prompt.

A terminology landmine, defused. We have been overloading the word "conditional." In Lecture 2, "conditional" meant conditioning on a clean data point z (the conditional probability path). Now we condition on a prompt y. To keep them straight, the notes reserve the word guided for conditioning on the prompt y. So: conditional = on z (the path); guided = on y (the prompt).

The obvious approach: just feed the prompt in

The simplest idea works, in principle. Give the network the prompt as an extra input: a guided vector field u^θ_t(x | y) that takes x, t, and y. Train it exactly like before but with paired (image, prompt) data; at sampling time, fix y and run the ODE/SDE. This is vanilla guidance. Clean, and theoretically it samples p_data(· | y).

The decomposition that reveals the knob

To see why vanilla guidance underperforms — and how to fix it — rewrite the guided score using Bayes' rule. Bayes says the guided density factors as p_t(x | y) ∝ p_t(x) · p_t(y | x). Take the gradient of the log (the score), and because the gradient is in x, any term that does not depend on x drops:

∇ log p_t(x | y) = ∇ log p_t(x) + ∇ log p_t(y | x)

Read the two pieces. The first term, ∇ log p_t(x), is the unguided score — "make it look like a real image." The second, ∇ log p_t(y | x), is the gradient of a classifier: p_t(y | x) asks "given this noisy image x, how likely is the prompt y?", so its gradient points in the direction that makes x match the prompt better. Guided generation = realism term + prompt-matching term. Converting scores back to vector fields (via the Chapter-2 formula) gives the same split for the vector field:

u_t(x | y) = u_t(x) + a_t · ∇ log p_t(y | x)

The fix: turn up the prompt term

People noticed the prompt-matching term was too weak — images came out only loosely related to the prompt. The natural hack: multiply that term by a guidance scale w > 1 to amplify it:

ũ_t(x | y) = u_t(x) + w · a_t · ∇ log p_t(y | x) (classifier guidance)

This is classifier guidance. The term ∇ log p_t(y | x) is just the gradient of a classifier trained on noisy images (it must classify x at every noise level), so you can learn it by ordinary supervised learning — then add w times its gradient to the drift at sampling time.

The honest admission: this is a heuristic. For any w ≠ 1, the amplified field ũ_t(x | y) is no longer the true guided vector field — you are deliberately distorting the dynamics to push harder toward the prompt. It works empirically, but you have left the land of "sampling the true distribution." Hold that thought: the same w ≠ 1 distortion in the next chapter is what causes over-saturation at high guidance.

Classifier guidance has two real costs that doomed it in practice: (1) you must train and store a second network (the noisy-image classifier) alongside your generator, and (2) for high-dimensional prompts like free text, p_t(y | x) is brutally hard to model and its gradient hard to obtain. The next chapter removes the classifier entirely — that's why classifier-free guidance won.

Classifier guidance writes the guided score as ∇ log p_t(x) + ∇ log p_t(y | x), then scales the second term by w > 1. What is that second term, and what is the catch?

It is the data distribution; scaling it is exact. It is the gradient of a classifier p_t(y|x) (the prompt-matching direction); the catch is it needs a separate classifier on noisy data, and for w ≠ 1 you no longer sample the true guided distribution. It is the step size; scaling it speeds up sampling. It is Gaussian noise; scaling it adds diversity for free.

Chapter 7: Classifier-Free Guidance — the Workhorse

Classifier-free guidance (CFG) achieves the same amplification as classifier guidance without ever training a classifier. It is the single most important practical trick in this lecture: virtually every AI image or video you have seen was sampled with CFG, typically at a scale around w ≈ 4.

The algebra: eliminate the classifier

Start from classifier guidance and substitute the Bayes identity for the classifier gradient, ∇ log p_t(y | x) = ∇ log p_t(x | y) − ∇ log p_t(x). After the dust settles (full derivation in the notes), the scaled guided vector field becomes a simple linear combination of the guided and unguided fields — no classifier in sight:

ũ_t(x | y) = (1 − w) · u_t(x | ∅) + w · u_t(x | y)

Reading the formula. u_t(x | y) is the conditional field (with the prompt). u_t(x | ∅) is the unconditional field, where ∅ means "no prompt." Equivalently, writing the guidance strength as a multiplier on the difference: you push away from the unconditional field and toward the conditional one, exaggerating the direction the prompt adds. A common engineering form of the exact same line is

u_cfg = (1 + w) · u_cond − w · u_uncond

— you walk from the unconditional prediction toward the conditional one and then keep going past it by a factor controlled by w. Both forms say the same thing: amplify the prompt's contribution.

The "wait, two models?" objection — and the fix

The formula needs both a conditional field u_t(x | y) and an unconditional one u_t(x | ∅). Sounds like two networks again. The beautiful trick: train one network to be both. Add a special null token ∅ to your label set meaning "no prompt," and during training randomly drop the real label — replace y with ∅ with some probability η (say 10–35%). The network learns the conditional behavior on the kept labels and the unconditional behavior on the dropped ones.

# CFG training (Algorithm 5) -- one network does both jobs
for each minibatch:
    z, y = sample_paired_data()       # image + its prompt
    t   = uniform(0, 1)
    eps = sample_gaussian()
    x   = alpha_t*z + beta_t*eps      # corrupt the image
    if random() < eta:                 # drop the label with prob eta
        y = NULL                      # the empty token, no prompt
    loss = || u_theta(x, t, y) - u_target(x | z) ||^2
    backprop_and_step(loss)

At sampling time you make two calls to the same network — one with y, one with ∅ — and combine them with the CFG formula above. One model, two passes, a slider w.

CFG does NOT sample the data distribution — and that's the point. For w > 1, the samples are not distributed like p_data(· | y). You have deliberately gone beyond the true conditional distribution — pushing into the region where the prompt signal is strongest, past where real data lives. It is a heuristic, justified almost entirely by how good it looks. Turn w up and prompt adherence sharpens dramatically; turn it up too far and you get the telltale over-guidance artifacts: blown-out colors, over-saturation, loss of diversity, unnatural contrast. The next chapter lets you watch exactly this trade-off happen.

Implement the CFG combination yourself and measure how guidance pulls samples toward the prompt's cluster — and how far past it you go.

Classifier-free guidance uses ũ = (1−w)·u(x|∅) + w·u(x|y). How does it avoid training a separate classifier, and what does w > 1 do?

It trains the classifier secretly inside the loss; w sets its learning rate. It uses two separate generators; w blends their pixels. One network learns both conditional and unconditional behavior by randomly dropping the label to a null token ∅ during training; w > 1 amplifies the prompt's contribution, sharpening adherence (and over-saturating if too high). It removes the prompt entirely; w controls image resolution.

Chapter 8: Showcase — Guided 2D Sampling

This is the payoff. Below, a flow model has been (hand-)set up over a 2D toy with several data clusters, and one of them is the "prompt" cluster you want. Drag the guidance scale w and watch the generated samples respond exactly as the theory predicts — and then misbehave.

What to look for, in order.

w = 0 (unguided): samples spread across all the clusters — the model generates "any object," ignoring the prompt.
w = 1: vanilla guidance — the true conditional. Samples concentrate on the prompt cluster but still respect its natural spread.
w = 2–4: classic CFG. Samples sharpen hard onto the prompt cluster's center — strong prompt adherence, the sweet spot real systems use.
w > 6: over-guidance. Samples collapse into a tight overshooting blob past the cluster — the 2D analogue of the over-saturated, low-diversity, blown-out look you get when you crank guidance too high on a real image generator.

Classifier-free guidance: the scale slider, live

Faint shaded blobs are the data clusters; the teal ring marks the prompt cluster. Orange dots are generated samples after running the guided field u_cfg = (1+w)·u_cond − w·u_uncond. At w = 0 the field is purely conditional, so samples land right on the prompt. Raise guidance scale w and the prompt's pull is amplified — samples sharpen and then overshoot past the target into a saturated blob (dots pile up at the frame edge). Press Resample for fresh noise. The readout reports how far past the prompt the samples land.

guidance w2.00

This single slider is the most-tuned knob in generative AI. Every Stable Diffusion / Midjourney / Sora pipeline exposes some form of it ("CFG scale," "guidance"). Too low and the output ignores your prompt; too high and it turns crispy and over-cooked. Stable Diffusion 3 ships around w ≈ 4. You just built the intuition for why that number lives where it does: it is the trade-off between prompt adherence (rises with w) and naturalness/diversity (falls with w) that you are watching on screen.

In the showcase, what happens as you push the guidance scale w from 4 up past 8, and what real-world artifact does it model?

Samples spread out to cover more clusters — modeling more diversity. Samples collapse into a tight blob that overshoots past the prompt cluster — the 2D analogue of over-saturated, low-diversity, blown-out over-guided images. Nothing changes; w only affects training. Samples return exactly to the unguided distribution.

Chapter 9: Connections & Cheat Sheet

You now hold the second master object of the course. With the vector field (Lectures 1–2) and the score (this lecture), you can train, sample deterministically or stochastically, and obey a prompt. Here is the whole lecture on one page.

The spine. The score ∇ log p_t(x) is the arrow toward higher data density. For the Gaussian path the conditional score is −(x − α_tz)/β_t², which when you plug in x = α_tz + β_tε equals −ε/β_t — so learning the score = predicting the injected noise (denoising). The score converts an ODE into an SDE by adding (σ²/2)·score to the drift (the SDE extension trick); with the field switched off this is Langevin dynamics. Guidance makes the model obey a prompt: classifier guidance adds w·∇ log p_t(y|x); classifier-free guidance combines (1−w)·unconditional + w·conditional from one network trained with label-dropout.

Cheat sheet

Score function. score(x) = ∇_x log q(x) = arrow uphill toward density; ignores normalization (gradient of log).

Gaussian conditional score. p_t(x|z) = N(α_tz, β_t²I) ⇒ ∇ log p_t(x|z) = −(x − α_tz)/β_t². Vanishes at the mean; pull strength 1/β_t².

Conversion (free lunch). u_t(x|z) = a_t·∇ log p_t(x|z) + b_t·x. Learn one, get the other.

Denoising score matching. x = α_tz + β_tε ⇒ target score = −ε/β_t; predict the noise: loss = ||ε^θ_t(x) − ε||². Stable; the raw score blows up as β_t → 0.

SDE extension trick. dX = [u_t(X) + (σ_t²/2)∇ log p_t(X)]dt + σ_tdW. Same p_t for any σ_t ≥ 0; σ = 0 recovers the ODE.

Langevin dynamics. dX = (σ²/2)∇ log p(X)dt + σdW → stationary distribution p. Score + noise = sampler.

Classifier guidance. ũ_t(x|y) = u_t(x) + w·a_t·∇ log p_t(y|x). Needs a noisy-data classifier; heuristic for w ≠ 1.

Classifier-free guidance (CFG). ũ_t(x|y) = (1−w)·u_t(x|∅) + w·u_t(x|y) = (1+w′)·u_cond − w′·u_uncond. One network, label-dropout training. The workhorse; w ≈ 4 in practice; over-saturates if too high.

What's next in this series

Lecture 2 — Flow Matching (the twin). The training recipe for the vector field, the partner of this lecture's score matching — same proof, different target.
Lecture 4 — Architectures & Latent Spaces. The networks that are u^θ and s^θ: U-Nets, Diffusion Transformers, and CFG inside a VAE latent space (Stable Diffusion 3, Movie Gen).
Lecture 5 — Discrete Diffusion. Carrying these ideas to text and language.

Related lessons on this site

Lecture 1 — Flow & Diffusion Models — ODEs, SDEs, Euler–Maruyama (the prerequisites for this lecture).
Flow Matching (Gleam) — the conditional-path training target this lecture builds on.
Diffusion Models (Gleam) — the DDPM noise-prediction view of score matching.
Diffusion (Architecture Atlas) — bird's-eye overview, including guidance.
SDEs & Continuous Diffusion (Deep-Dive) — the rigorous Fokker–Planck treatment.

The Feynman test. Open the Lecture 3 slides. Can you, from memory: define the score and say why it uses the log; derive the Gaussian conditional score and explain its sign and 1/β_t²; show that denoising score matching means predicting the injected noise; state the SDE extension trick and why the score correction cancels the noise's smearing; and write the CFG combination, explain the label-dropout trick, and predict what happens as the guidance scale goes too high? If yes — you own Lecture 3, the heart of how real generative systems obey a prompt.

One sentence: how is score matching related to flow matching, and what does the score additionally unlock?

They are unrelated; the score replaces the vector field entirely. Score matching trains a classifier; flow matching trains the generator. Score matching is the twin of flow matching (same proof, regressing onto the conditional score instead of the conditional vector field — and for Gaussian paths the two are interconvertible); the score additionally unlocks SDE sampling and guidance. They differ only in image resolution.