Denoising Diffusion Probabilistic Models

Introduction

In 2015, Sohl-Dickstein et al. proposed a provocative idea borrowed from non-equilibrium statistical mechanics: what if we could generate data by learning to reverse a diffusion process? The idea was theoretically intriguing but produced mediocre samples. Five years later, Ho, Jain & Abbeel (2020) revisited the approach with modern neural networks and a crucial simplification of the training objective. The result — Denoising Diffusion Probabilistic Models (DDPMs) — produced image quality rivaling GANs while being far more stable to train.

DDPMs work in two phases. The forward process takes a clean data sample and progressively corrupts it by adding Gaussian noise over T timesteps, until only pure noise remains. The reverse process starts from pure noise and learns to iteratively denoise, one step at a time, recovering a clean data sample. The neural network only needs to solve a modest task at each step — predict and remove a small amount of noise — rather than generating an entire image from scratch.

This article derives the DDPM framework from first principles: the forward Markov chain, the reverse posterior, the variational bound, and the elegant noise-prediction simplification that makes training as simple as a single line of code.

The Forward Process

The forward process is a Markov chain that adds Gaussian noise to data over T timesteps. Starting from a clean data sample x0 ~ q(x0), each step slightly corrupts the previous state:

Here β1, ..., βT is the noise schedule — a sequence of small positive values (typically 10^-4 to 0.02) that control how much noise is added at each step. The scaling factor √(1 - βt) slightly shrinks the signal while βt controls the noise variance. Together they ensure the total variance doesn't blow up.

At each step, we can use the reparameterization trick to write:

After enough steps (typically T = 1000), the signal is completely drowned by noise: xT ≈ 𝒩(0, I). The data has been destroyed.

Closed-form sampling at any timestep

A crucial mathematical property: we don't need to run all T steps sequentially to get xt. Define αt = 1 - βt and ᾱt = ∏s=1t αs (the cumulative product of all signal-retention factors up to step t). Then:

This closed-form is essential for training efficiency. Instead of sequentially noising an image through all 1000 steps, we can jump directly to any timestep t by sampling ε and computing xt in one operation. During training, we sample a random t for each example in the batch — no sequential processing needed.

The Reverse Process

The forward process destroys data. The reverse process learns to undo the destruction. Starting from pure noise xT ~ 𝒩(0, I), we want to iteratively recover xT-1, xT-2, ..., x0 until we arrive at a clean data sample.

The reverse process is parameterized as:

At each step, the neural network takes the current noisy state xt and the timestep t, and predicts the parameters of a Gaussian distribution over the previous (less noisy) state. The key insight from Feller (1949) and later formalized in the diffusion model context: when β_t is small enough, the reverse of a Gaussian diffusion step is also approximately Gaussian. So a Gaussian parameterization for each reverse step is well-justified.

The reverse posterior q(x_t-1 | x_t, x₀)

There is a remarkable mathematical gift in the DDPM framework. While the true reverse distribution q(xt-1 | xt) is intractable (it requires integrating over all possible x0), the reverse posterior conditioned on x₀ is tractable and Gaussian:

This tractable posterior is the key that unlocks DDPM training. We know the true distribution we want each reverse step to approximate (given that we could peek at x₀). We can measure how well our learned reverse step matches this target using KL divergence — and the KL between two Gaussians has a closed-form expression.

The Training Objective

The variational lower bound

Following the ELBO framework from Article 01, we can decompose the variational lower bound on log p(x₀) into a sum of terms — one per timestep:

Where each Lt-1 is the KL divergence between the true reverse posterior and our learned approximation:

Since both distributions are Gaussian, this KL has a closed-form expression that reduces to comparing their means (the variances can be fixed). The entire training objective becomes: make the predicted mean μ_θ(x_t, t) match the true posterior mean μ̃_t.

The noise prediction reparameterization

Ho et al. (2020) made a pivotal choice. Instead of directly predicting the mean μ_θ, they reparameterized the network to predict the noise ε that was added to create x_t from x₀.

Recall that xt = √ᾱt · x0 + √(1 - ᾱt) · ε. If we know ε, we can recover x₀:

And then express the mean in terms of the predicted noise:

The simplified loss

After plugging the noise parameterization into the VLB and dropping the weighting terms that depend on the schedule (which Ho et al. found empirically harmful), we arrive at the beautifully simple training objective:

The training algorithm in pseudocode:

while training:
    x0 = sample_from_data()             # clean data
    t = randint(1, T)                    # random timestep
    eps = torch.randn_like(x0)           # random noise
    xt = sqrt(alpha_bar[t]) * x0 + sqrt(1 - alpha_bar[t]) * eps   # noisy data
    loss = mse(model(xt, t), eps)        # predict the noise
    loss.backward()
    optimizer.step()

This simplicity is not an approximation — it's a reweighted version of the full variational bound. Ho et al. found that uniform weighting across timesteps produces better sample quality than the theoretically derived weights, even though the latter give a tighter bound on the log-likelihood.

Noise Schedules

The noise schedule β1, ..., βT controls the rate of noise addition. This choice significantly affects sample quality.

Linear schedule (Ho et al., 2020): β_t increases linearly from β₁ = 10^-4 to β_T = 0.02 over T = 1000 steps. Simple and effective, but the signal-to-noise ratio drops too quickly in early steps, wasting capacity on nearly-destroyed images.

Cosine schedule (Nichol & Dhariwal, 2021): designed so that ᾱ_t follows a cosine curve. This produces a more gradual noise increase, with more timesteps spent at intermediate noise levels where the denoising task is neither trivial nor hopeless. The cosine schedule consistently improves FID scores, especially at higher resolutions.

The signal-to-noise ratio (SNR) at timestep t is SNR(t) = ᾱt / (1 - ᾱt). A good noise schedule distributes the SNR decrease evenly in log-space across timesteps, giving the network a smooth curriculum from easy denoising (high SNR) to hard denoising (low SNR).

Sampling (The Reverse Pass)

Once trained, generating new data is straightforward:

1. Sample xT ~ 𝒩(0, I) — start from pure noise.
2. For t = T, T-1, ..., 1, compute one reverse step:
   x_t-1 = (1/√α_t) · (x_t - (β_t/√(1-ᾱ_t)) · ε_θ(x_t, t)) + σ_t · z
   where z ~ 𝒩(0, I) for t > 1, and z = 0 for t = 1.
3. Return x0 — a generated sample.

The variance σt2 can be set to β_t (Ho et al.) or β̃_t (both work, giving slightly different sample characteristics). The stochastic noise z at each step introduces diversity — different z sequences produce different samples from the same starting noise x_T.

The obvious limitation: sampling requires T forward passes through the neural network. At T = 1000 and ~0.1s per pass on a GPU, a single image takes ~100 seconds. This motivated the entire field of fast samplers covered in Article 07.

Putting It Together

Let's zoom out and see the complete DDPM picture:

Training: repeat until convergence — sample clean data x₀, sample random t ∈ {1,...,T}, sample noise ε ~ 𝒩(0,I), compute noisy x_t, predict ε_θ(x_t, t), take gradient step on ‖ε - ε_θ‖².

Sampling: sample x_T ~ 𝒩(0,I). For t = T down to 1: predict noise, compute mean, add stochastic noise (if t > 1). Return x₀.

The mathematical elegance is remarkable. The forward process is fixed (no parameters). The reverse process is a single neural network ε_θ shared across all timesteps, conditioned on t. The training loss is plain MSE. The sampling is a simple iterative loop.

DDPMs opened the floodgates. Within two years, diffusion models overtook GANs on every major image generation benchmark. But understanding why they work so well requires the score function perspective — which is where we're headed next.

References

Seminal papers and key works referenced in this article.

1. Ho et al. "Denoising Diffusion Probabilistic Models." NeurIPS, 2020. arXiv
2. Sohl-Dickstein et al. "Deep Unsupervised Learning using Nonequilibrium Thermodynamics." ICML, 2015. arXiv
3. Song et al. "Denoising Diffusion Implicit Models." ICLR, 2021. arXiv
4. Nichol & Dhariwal. "Improved Denoising Diffusion Probabilistic Models." ICML, 2021. arXiv