Understand Diffusion
Models

Chapter 1: The Forward Process

The key insight of diffusion models: destruction is easy, creation is hard. The forward process takes a real image and gradually adds Gaussian noise over T steps (typically T=1000). At each step, the image gets a little noisier, until at step T it's indistinguishable from pure static.

Mathematically, at each step t we mix the current image with a little noise: q(x_t | x_t-1) = N(x_t; √(1-β_t) x_t-1, β_t I). The noise schedule β_t controls how fast the signal is destroyed.

The Exact Tensor Operation

In code, the forward process is a single line. Given a clean image x₀ and noise ε ~ N(0, I), both tensors of shape [batch, C, H, W]:

Let's plug in real numbers. With a cosine schedule and T=1000:

python
# The entire forward process in PyTorch
def forward_process(x_0, t, alpha_bar):
    """x_0: [B, C, H, W], t: [B], alpha_bar: [T]"""
    a_bar = alpha_bar[t].view(-1, 1, 1, 1)  # broadcast
    eps = torch.randn_like(x_0)            # noise ~ N(0, I)
    x_t = torch.sqrt(a_bar) * x_0 + torch.sqrt(1 - a_bar) * eps
    return x_t, eps

python
def forward_diffusion(x_0, t, alpha_bar):
    a_bar = alpha_bar[t].view(-1, 1, 1, 1)   # [B] -> [B,1,1,1]
    eps = torch.randn_like(x_0)              # sample noise
    x_t = torch.sqrt(a_bar) * x_0 + torch.sqrt(1 - a_bar) * eps
    return x_t, eps

Chapter 2: The Reverse Process

If we could reverse the forward process — undo each noise step — we'd have a generative model! Start from pure noise x_T ~ N(0, I) and iteratively denoise to get a clean image x₀. The problem: the exact reverse q(x_t-1|x_t) requires knowing the full data distribution, which is what we're trying to learn.

Solution: train a neural network ε_θ(x_t, t) to approximate the reverse step. This network takes in a noisy image and the timestep, and predicts the noise that was added. Given the predicted noise, we can estimate the slightly-less-noisy image.

What Goes In, What Comes Out

The denoiser is typically a U-Net: an encoder-decoder with skip connections. Let's be precise about the data flow:

Chapter 3: Training the Denoiser

Training is surprisingly simple. For each training step: (1) pick a random image x₀ from the dataset, (2) pick a random timestep t, (3) sample noise ε ~ N(0,I), (4) create the noisy image x_t = √ᾱ_t x₀ + √(1-ᾱ_t) ε, and (5) train the network to predict ε from x_t and t.

That's it. Plain MSE loss between the true noise and the predicted noise. No adversarial training, no mode collapse, no training instability. This simplicity is a huge reason diffusion models won.

The Complete Training Loop

Here's the full training loop. Read it carefully — there are no hidden steps:

python
for x_0 in dataloader:                     # 1. Sample x_0 from data
    t = torch.randint(0, T, (B,))         # 2. Sample t ~ Uniform{0..T-1}
    eps = torch.randn_like(x_0)            # 3. Sample epsilon ~ N(0, I)

    # 4. Compute x_t (the noisy version)
    a = alpha_bar[t].view(-1,1,1,1)
    x_t = torch.sqrt(a) * x_0 + torch.sqrt(1-a) * eps

    # 5. Predict noise, compute loss
    eps_hat = unet(x_t, t)                 # [B, C, H, W] -> [B, C, H, W]
    loss = F.mse_loss(eps_hat, eps)         # That's it. MSE on noise.

    loss.backward()
    optimizer.step()

Chapter 4: The Math

The theoretical foundation of diffusion models rests on three pillars. You don't need them to use diffusion models, but understanding them reveals why the simple training objective actually works.

Pillar 1: The ELBO

We want to maximize log p(x₀), but it's intractable. Instead we maximize a lower bound (the Evidence Lower Bound). The ELBO decomposes into a sum of KL divergences — one per timestep — each comparing the true reverse step to our learned approximation.

Pillar 2: KL Divergence

KL divergence measures how different two distributions are. Since both q and p_θ are Gaussian, the KL has a closed form. It reduces to comparing means — which becomes the simple MSE loss.

Pillar 3: Score Function

The score is ∇_x log p(x) — a vector pointing toward higher-density regions. Denoising is equivalent to estimating the score: ε_θ(x_t, t) ∝ -∇_x log p(x_t). This connection to score matching is why diffusion models are sometimes called "score-based generative models."

Chapter 5: Sampling

Once trained, we generate images by starting from noise and iteratively denoising. The original DDPM sampler uses all T=1000 steps — faithful to the theory but painfully slow (~1 minute per image).

DDIM (Denoising Diffusion Implicit Models) noticed that the forward process can be made deterministic, allowing you to skip steps. With just 50 steps, quality is nearly identical. DPM-Solver treats sampling as solving an ODE and uses higher-order methods (like Runge-Kutta) to achieve great quality in 10-25 steps.

The Reverse Step in Detail

DDPM sampling computes x_t-1 from x_t using the predicted noise ε̂:

where z ~ N(0, I) is fresh noise added at each step. This stochastic term is what makes DDPM slow — each step adds randomness, so you can't skip ahead.

DDIM removes this stochastic term (sets σ_t = 0), making the process deterministic. A deterministic ODE can be evaluated at any subset of timesteps. Instead of [1000, 999, 998, ...], DDIM evaluates at [1000, 980, 960, ...] — 50 evenly spaced steps instead of 1000.

python
# DDPM: 1000 sequential steps (slow)
for t in range(T, 0, -1):
    eps_hat = unet(x_t, t)
    x_t = reverse_step(x_t, eps_hat, t) + sigma[t] * torch.randn_like(x_t)

# DDIM: skip timesteps (fast, deterministic)
steps = [1000, 980, 960, ..., 20, 0]  # 50 steps
for t, t_prev in zip(steps[:-1], steps[1:]):
    eps_hat = unet(x_t, t)
    x_t = ddim_step(x_t, eps_hat, t, t_prev)  # no stochastic term

Chapter 6: Latent Diffusion

Diffusing directly in pixel space is expensive: a 512×512 image has ~786K dimensions. Latent Diffusion Models (LDMs) first encode the image into a compact latent space using a pretrained VAE (Variational Autoencoder), then run diffusion there.

The VAE encoder compresses the image by 8x in each spatial dimension: 512×512 → 64×64 latent. The diffusion model learns to denoise in this 64×64 space (much cheaper!), then the VAE decoder reconstructs the final image. This is exactly what Stable Diffusion does.

The Numbers That Matter

Let's count dimensions to see why this is such a huge win:

Every U-Net forward pass operates on a 16K-dimensional tensor instead of a 786K-dimensional one. Since the U-Net runs once per sampling step (and there are 20-50 steps), this is a 48× speedup per step. Multiply that across all steps and training iterations, and you understand why latent diffusion made Stable Diffusion practical.

Chapter 7: Conditioning

Unconditional generation is impressive but not useful. We want to say "a cat wearing a top hat" and get that image. Conditioning injects a text prompt into the denoising process.

The pipeline: (1) A text encoder (typically CLIP) converts the prompt into an embedding vector. (2) This embedding is injected into the U-Net via cross-attention layers: the noisy image attends to the text tokens. The network learns to denoise differently depending on the text.

Classifier-Free Guidance (CFG)

During training, the text condition is randomly dropped (replaced with empty text) some percentage of the time. At inference, we compute both the conditional and unconditional noise predictions, then amplify the difference:

The guidance scale w (typically 7-12) controls how strongly the model follows the prompt. Higher w = more adherence to text but less diversity and potential artifacts.

How CFG Actually Works

During training, the text condition is randomly replaced with an empty string ∅ about 10-20% of the time. This teaches the network to predict noise both with and without the prompt. At inference, every denoising step runs the U-Net twice:

python
# Every sampling step runs TWO forward passes
eps_uncond = unet(x_t, t, text="")      # unconditional prediction
eps_cond   = unet(x_t, t, text=prompt)   # conditional prediction

# Amplify the difference
eps_guided = eps_uncond + w * (eps_cond - eps_uncond)

# w=1.0: just use conditional (no guidance)
# w=7.5: standard for text-to-image (SD 1.5, SDXL)
# w=15+: very strong adherence, but artifacts/saturation

Chapter 8: ControlNet & Adapters

Text alone is often insufficient. You might want to specify a precise pose, edge map, or depth layout. ControlNet adds a parallel encoder that takes a spatial condition (like a Canny edge image) and injects it into the U-Net's skip connections.

The genius: the original Stable Diffusion weights are frozen. ControlNet trains a copy of the encoder that learns to translate spatial signals. This preserves the base model's quality while adding precise spatial control.

Chapter 9: The Diffusion Ecosystem

Diffusion models have evolved rapidly. Here's a map of the landscape as of mid-2025:

Consistency Models

A radical departure: instead of iterating T steps, learn to jump directly from any noisy x_t to x₀ in a single step. Consistency models (Song et al., 2023) enforce that all points on the same denoising trajectory map to the same output. The result: 1-2 step generation with quality approaching multi-step diffusion.