Diffusion & Flow Math

Exact numerical computation gives ~0.536. At 25% of the way through the schedule, about half the signal remains. The approximation log(1-β) ≈ -β is accurate because each β is small (< 0.02).

Exact numerical value is ~0.00118. Our approximation overshoots because the log(1-β) ≈ -β approximation becomes less accurate for larger β values (near t=750, β ≈ 0.015). The point stands: at 75% through the schedule, the signal is essentially destroyed — less than 0.12% remains.

Compare with the linear schedule: ā₂₅₀ = 0.536 (linear) vs 0.847 (cosine). At t=250, the cosine schedule preserves 85% of the signal vs only 54% for linear. The cosine schedule is much more gentle early on.

Cosine: 14.6% signal at t=750. Linear: 0.12% signal at t=750. The cosine schedule still has significant signal at 75% of the way through — this gives the model useful gradients even in the later stages. The linear schedule has already destroyed everything by this point.

Both schedules end at ā_T ≈ 0 (pure noise). The difference is in the distribution of difficulty. The linear schedule spends most timesteps in either "almost clean" or "almost noise" regimes — the denoiser doesn't get enough practice on the hardest middle range. The cosine schedule's ā₅₀₀ ≈ 0.5 means exactly half the timesteps are above 50% signal and half below — a perfectly balanced curriculum.

javascript
function cosineAlphaBar(t, T) {
  const s = 0.008;
  const f = x => Math.cos((x / T + s) / (1 + s) * Math.PI / 2) ** 2;
  return f(t) / f(0);
}

In practice, the mean loss is used for optimization (typically around 0.01-0.1 during training). The sum loss grows with image size — a 256×256×3 image would have 196,608 elements and sum loss of 7,864 for the same per-element error.

This is the SNR reweighting. At high noise (ā_t small), the x₀ loss amplifies errors massively because 1/ā_t is huge. Noise prediction with L_simple implicitly downweights high-noise timesteps by a factor of ā_t/(1-ā_t), which is why it produces more stable training.

Line 4 is the bug. The noise coefficient should be sqrt(1 - alphabar_t), not sqrt(alphabar_t). The correct formula is x_t = √ā_t · x₀ + √(1-ā_t) · ε. Using √ā_t for both terms means at early timesteps (ā_t ≈ 1) you add too much noise, and at late timesteps (ā_t ≈ 0) you add too little. The model learns a corrupted noise distribution, producing blurry reconstructions.

v-prediction smoothly interpolates: at low noise it predicts ε (like DDPM), at high noise it predicts -x₀. This gives a numerically stable target at all timesteps — unlike ε-prediction which has high variance at low noise, or x₀-prediction which has high variance at high noise.

The score at x=2 is -2, pointing toward the mean at x=0. The score is always a "pull" toward high-density regions. At the mean itself, the score is zero.

The score is smaller (magnitude 0.5 vs 2). A wider distribution has a gentler gradient — being at x=2 is less "unusual" when σ=2, so the pull back toward the mean is weaker.

For a symmetric mixture p(x) = ½ N(-3, σ²) + ½ N(+3, σ²), the density p(x) is symmetric around 0, so its log is also symmetric, and the derivative at x=0 is exactly 0. The leftward pull from the left mode and rightward pull from the right mode cancel perfectly at the midpoint.

javascript
function gaussianScore(x, mu, sigmaSquared) {
  return -(x - mu) / sigmaSquared;
}

Each reverse step t → t-1 requires one forward pass of ε_θ(x_t, t) to predict the noise. With T=1000 steps (from t=1000 down to t=1), that's exactly 1000 forward passes. If each pass takes ~50ms on a GPU, generation takes ~50 seconds — far slower than GANs (one pass) or VAEs (one pass).

DDIM with 50 steps visits timesteps {1000, 980, 960, ...20}. Each DDIM step jumps 20 timesteps at once. This gives a 20× speedup over DDPM (50 forward passes vs 1000) with only moderate quality loss — typically imperceptible for 50+ steps.

Exact computation: x̂₀ = (1.0 - 0.70711×0.4)/0.70711 = 0.71716/0.70711 = 1.01421. x_t-1 = 0.77460×1.01421 + 0.63246×0.4 = 0.78560 + 0.25298 = 1.039. Note: DDIM with no noise is deterministic — run it again with the same x_T and you get the exact same x₀.

DDIM-50 is 20× faster than DDPM with minimal quality loss. DDIM-20 pushes under 1 second but quality drops noticeably. Modern schedulers (DPM-Solver, UniPC) can match DDIM-50 quality with only 15-20 steps by using higher-order ODE solvers.

Notice how the guided noise prediction (1.6, 4.6) is much larger than the conditional prediction (0.3, 0.7). CFG amplifies the difference between conditional and unconditional, pushing the sample harder toward what the prompt describes. At w=7.5, the second element goes from 0.7 to 4.6 — a 6.6× amplification.

At w=0, you get the unconditional model — it generates random images with no regard for the text prompt. At w=1, ε̃ = ε_uncond + ε_cond - ε_uncond = ε_cond, which is just the standard conditional output. Values w > 1 push beyond the conditional prediction, amplifying the prompt's influence.

CFG exactly doubles the cost compared to unguided DDIM-50 (2.25s). In practice, the two predictions can be batched (concatenate the cond and uncond inputs), so wall-clock time is less than 2× if the GPU has headroom. Still, eliminating one of the two passes via distillation is worth a lot.

Higher w amplifies the conditional signal more aggressively. At moderate w (5-10), images closely match the prompt with good quality. At very high w (15+), the amplification pushes pixel values to extremes — colors oversaturate, edges become unnaturally sharp, and artifacts appear. It's like turning up the volume on a speaker: at first it's clearer, then it distorts. The sweet spot for Stable Diffusion is w=7-9.

Actually, the full attention FLOPs are O(n² d) where d is the channel dimension. Pixel space: 262144² tokens. Latent space: 4096² tokens. Ratio = (262144/4096)² = 64² = 4096. But the U-Net doesn't use attention at every resolution — it uses attention only at lower resolutions (32×32, 16×16, 8×8) in both pixel and latent models. The 256× figure comes from the spatial sequence length ratio at the resolution where attention is applied: in latent space, the highest-resolution attention operates on (64/2)² = 1024 tokens vs pixel space (512/16)² = 1024 tokens at the same U-Net depth. The key saving is that the latent model can afford attention at HIGHER resolutions (64×64 = 4096 tokens is feasible, 512×512 = 262K is not).

32 MB vs 128 GB — that's a 4096× difference. Full-resolution self-attention in pixel space at 512×512 requires 128 GB for a single attention matrix of a single head of a single layer. This is why pixel-space diffusion either avoids attention or restricts it to tiny resolutions.

The SDXL latent at 128×128 has 4× the spatial tokens of SD 1.5's 64×64 latent. This 4× increase in sequence length makes self-attention 16× more expensive, which is why SDXL uses a larger U-Net with more efficient attention patterns and requires substantially more VRAM.

The autoencoder runs exactly twice during generation: once at the very end (decoder converts final latent z₀ to pixels) and once during training (encoder converts training images to latents). The entire iterative denoising process — all 50+ sampling steps — runs in the 64×64×4 latent space. This is why it's called latent diffusion.

At t=0 we're at x₀=[1,2] (data), at t=1 we're at x₁=[5,8] (noise), and at t=0.3 we're 30% of the way along the straight line between them. Note: in some flow matching papers, the convention is reversed (t=0 is noise, t=1 is data). Always check the convention!

The target velocity is constant — it doesn't depend on t at all. This is a key simplification of flow matching: the velocity along a straight path is just the displacement vector from data to noise. The model learns to approximate this constant velocity at each point in space-time.

Both methods train a neural network with MSE loss on paired inputs and targets. DDPM: input is noised image x_t = √ā_tx₀ + √(1-ā_t)ε, target is ε. Flow matching: input is interpolated point ψ_t = (1-t)x₀ + tx₁, target is velocity x₁-x₀. The architecture, dataset, and optimizer can be identical — only the parameterization of the interpolation path and prediction target differ.

javascript
function flowMatchStep(x, v, dt) {
  return x + v * dt;
}

Correct order: (1) Sample x₀ from data, (2) Sample x₁ ~ N(0,I), (3) Compute interpolation ψ_t, (4) Forward pass v_θ(ψ_t, t), (5) MSE loss vs target velocity (x₁ - x₀). This is the entire training loop — simpler than DDPM because there's no noise schedule (α, ā) to manage.

The true solution of dx/dt = -x is x(t) = 3e^-t, so x(0.3) = 3e^-0.3 = 2.222. Our Euler approximation gives 2.187 — an error of ~1.6%. The Euler method multiplies by (1-Δt) at each step, so x_n = 3(0.9)ⁿ. This converges to the true exponential as Δt → 0.

With 30 steps (Δt = 0.01) the error would drop to ~0.15%. With 3 steps (Δt = 0.1) it's 1.6%. This is why diffusion sampling quality degrades with fewer steps — each step introduces discretization error, and it compounds. Higher-order solvers (Heun, RK4, DPM-Solver) reduce error per step, allowing fewer steps for the same quality.

With 5 steps, Δt = 0.2 per step. The Euler method approximates curves with straight line segments — larger steps mean more error where the velocity field curves. The accumulated error means the sample ends up in a low-probability region, producing blurry averages of multiple modes rather than sharp samples. This is why modern schedulers (DPM-Solver++, UniPC) use higher-order methods that better follow the ODE trajectory.

ODE sampling is deterministic: no randomness is injected, so the same x_T always maps to the same x₀. SDE sampling adds Gaussian noise at each step (the dW term), so even with the same starting point, each run follows a different trajectory and produces a different image. Both are valid samples from the learned distribution, but ODE gives you a reproducible mapping while SDE gives you stochastic diversity.

Near t=T, the velocity field is relatively smooth (noisy samples all look similar). Near t=0, the model is resolving fine spatial details — small errors get amplified into visible artifacts. Adaptive solvers automatically take smaller steps where the velocity field has high curvature (near t=0) and larger steps where it's smooth (near t=T). This optimal step allocation is why adaptive solvers outperform fixed-step schedules.

At 1024×1024, pixel-space attention is impossible: 1M² = 10¹² attention scores per head per layer. Even without attention, the U-Net would process 1M spatial tokens at the highest resolution. Latent diffusion compresses to 128×128 = 16K tokens, making self-attention feasible (16K² = 256M scores — large but manageable). Every modern high-resolution model (SDXL, FLUX, Imagen 3) uses latent or cascaded approaches.

82.7M is just the convolution parameters. A real U-Net also has: self-attention layers at each level (adding ~4×C² per attention block), cross-attention for text conditioning, time embedding MLPs, skip connections, and group norms. The full SD 1.5 U-Net is ~860M parameters; SDXL's is ~2.6B. Our rough ResNet estimate captures the scaling pattern: the deepest levels (highest C) dominate the parameter count — level 3 alone is 63% of total.

520 TFLOPs for one image. For comparison, a single GPT-4 token generation requires ~1.8 TFLOPs (for a ~1.8T parameter model with 2 FLOPs/param). So generating one SDXL image costs roughly as much as generating 289 GPT-4 tokens — about a paragraph of text.

~4.2 seconds per image meets our <5 second target. In practice, SDXL on an A100 with optimized inference (TensorRT, FlashAttention) generates 1024×1024 images in 3-5 seconds, matching our estimate well.

863 images/hour falls short of the 1000+ target. Options: (1) reduce steps from 50 to 40 (gives 1079/hr), (2) use a distilled model that needs fewer steps, (3) batch 2 images per pass if VRAM allows, (4) use DDIM-25 with a higher-order solver (DPM-Solver++) for comparable quality. Option 3 is the most practical: batch=2 nearly doubles throughput to ~1700/hr.

Correct order: (1) Encode text prompt, (2) Sample noise in latent space, (3) Iterative denoising loop, (4) VAE decode to pixels, (5) Post-process, (6) Save/return. The text encoding happens first (and only once per prompt). The denoising loop is 98% of the compute. The VAE decoder runs once at the end. Post-processing (clamping values, converting to uint8) is negligible.