Diffusion Models

Chapter 1: The Forward Process

The forward process (also called the diffusion process or noising process) is a Markov chain that gradually adds Gaussian noise to data over T steps. At each step, we mix the current signal with a small amount of random noise.

One Step of the Forward Process

Starting from data x₀, the transition from step t−1 to step t is:

Let's unpack this. At each step:

In code, one forward step is:

python
def forward_step(x_prev, beta_t):
    noise = np.random.randn(*x_prev.shape)    # sample ε ~ N(0, I)
    scale = np.sqrt(1 - beta_t)               # shrink factor
    x_t = scale * x_prev + np.sqrt(beta_t) * noise
    return x_t

The Noise Schedule

The values β₁, β₂, ..., β_T form the noise schedule. Typical choices:

Usually β_t ranges from β₁ ≈ 10⁻⁴ (tiny noise at start) to β_T ≈ 0.02 (more noise at end). With T = 1000 steps, the cumulative effect turns any data distribution into approximately N(0, I).

Chapter 2: Closed-Form Noising

A magical property of Gaussian noise: we don't need to run all t steps sequentially to get x_t from x₀. There's a closed-form shortcut.

The Reparameterization

Define α_t = 1 − β_t (the "keep" fraction at step t). Define the cumulative product:

Then we can jump directly from x₀ to x_t:

In code:

python
def noise_at_t(x_0, alpha_bar_t):
    """Jump directly to noise level t."""
    eps = np.random.randn(*x_0.shape)  # ε ~ N(0, I)
    x_t = np.sqrt(alpha_bar_t) * x_0 + np.sqrt(1 - alpha_bar_t) * eps
    return x_t, eps  # return both — we'll need ε for training!

What ᾱ_t Tells Us

Chapter 3: The Reverse Process

The forward process is fixed — no learned parameters. It just adds noise. The reverse process is where all the learning happens: given noisy data x_t, estimate the slightly less noisy version x_t−1.

The Reverse Transition

We parameterize the reverse process as:

The neural network θ takes x_t and the time step t as input, and outputs the predicted mean μ_θ. The variance σ_t² is typically set to β_t (fixed, not learned).

Denoising = Reverse Diffusion

Generation starts from pure noise x_T ~ N(0, I) and iterates:

Each step, the network removes a little noise. After T steps, we have a clean sample from the data distribution. The neural network has learned the inverse of the noising process.

Chapter 4: Training: Predict the Noise

We could train the network to directly predict μ_θ. But Ho et al. (2020) discovered something elegant: it's easier to train the network to predict the noise ε that was added, rather than the clean signal.

The Reparameterization Trick

Recall the closed-form noising: x_t = √ᾱ_t · x₀ + √(1 − ᾱ_t) · ε. If we know ε, we can recover x₀:

And from x₀, we can compute the exact reverse mean. So predicting ε is equivalent to predicting the mean, just reparameterized.

The Training Algorithm (DDPM)

python
# DDPM training loop (simplified)
for x_0 in dataloader:
    t = torch.randint(1, T+1, (batch_size,))     # random timestep
    eps = torch.randn_like(x_0)                    # true noise
    alpha_bar = alpha_bar_schedule[t]
    x_t = sqrt(alpha_bar) * x_0 + sqrt(1-alpha_bar) * eps
    eps_pred = model(x_t, t)                       # network predicts noise
    loss = F.mse_loss(eps, eps_pred)
    loss.backward()
    optimizer.step()

Chapter 5: Score Matching

There's an alternative (and mathematically deeper) way to think about diffusion models: through score functions. This perspective connects diffusion to a rich literature in statistics and physics.

What Is the Score?

The score function of a probability distribution p(x) is the gradient of its log-density:

The score points in the direction of increasing probability. If you're at a low-density point, the score tells you "move this way to reach higher density." It's a vector field that flows toward the modes (peaks) of the distribution.

Score Matching = Noise Prediction

Here's the punchline. For the noisy distribution at time t, the score is:

The noise ε that was added is (up to a scale factor) the negative of the score! So when our network predicts ε, it's implicitly learning the score function. This is why the two perspectives — noise prediction and score matching — are equivalent.

Langevin Dynamics

Once you have the score, you can generate samples using Langevin dynamics: a physics-inspired sampling method that follows the score field with added noise:

This is like a ball rolling uphill in the probability landscape (the score term), with random jiggling (the noise term). Over many steps, the samples converge to the target distribution.

Chapter 6: Waveform Generation

Diffusion models aren't just for images. WaveGrad (Chen et al., 2020) applies the same diffusion framework to generate raw audio waveforms — the 1D signal that represents sound pressure over time.

Why Waveforms Are Hard

Audio waveforms are sampled at 16,000–48,000 Hz. A 1-second clip has 16,000+ samples. Each sample must be predicted with high precision — even small errors create audible artifacts (clicks, pops, distortion). Previous approaches (WaveNet) used autoregressive generation: predict one sample at a time. At 24 kHz, that's 24,000 sequential neural network calls per second. Way too slow for real-time.

WaveGrad Architecture

WaveGrad conditions the denoising on a mel spectrogram (a compact time-frequency representation) and refines the waveform over a small number of steps:

Conditioning on Mel Spectrograms

The neural network receives two inputs: (1) the current noisy waveform x_t, and (2) the target mel spectrogram. The mel spectrogram tells the network what the audio should sound like (frequency content over time). The denoising process figures out how to realize that as a specific waveform.

Chapter 7: Denoising Showcase

This is the payoff. Below, you'll watch a full diffusion reverse process unfold on a 1D signal. Start from pure noise. Step through the denoising. Watch structure emerge from chaos. Adjust the noise schedule, number of steps, and signal type to see how the process behaves.