Stable Diffusion 3

Chapter 1: Rectified Flow

All diffusion-style models share the same core idea: define a forward process that turns data into noise, then learn to reverse it. The question is what path the forward process takes.

The forward process

Given a data sample x₀ and noise ε ~ N(0, I), define the noisy sample at time t as:

At t = 0, z₀ = x₀ (pure data). At t = 1, z₁ = ε (pure noise). For any t in between, z_t is a linear interpolation between data and noise. This is the defining property of rectified flow — the path from data to noise is a straight line.

Why straight lines matter

Compare this to DDPM, which uses a variance-preserving schedule: z_t = α_t x₀ + σ_t ε, where α_t and σ_t follow specific curved schedules. The DDPM path from data to noise is curved — the signal and noise don't mix linearly.

Why does this matter? Because at inference time, you need to follow these paths backwards. A curved path requires many small steps to trace accurately (like driving on a winding mountain road). A straight path can theoretically be traversed in a single step (like a highway).

The training objective

The network v_Θ learns the velocity field by minimizing the conditional flow matching loss:

The target velocity ε − x₀ is constant for a given (x₀, ε) pair — it doesn't depend on t. This makes the learning problem simpler than DDPM's time-dependent noise prediction.

Chapter 2: The MMDiT Architecture

Previous text-to-image models (Stable Diffusion 1/2, SDXL) use a U-Net backbone with cross-attention: the image features attend to text features, but text features are frozen — they never get to see what the image is doing. This is a one-way street.

SD3 replaces the U-Net with a Multimodal Diffusion Transformer (MMDiT), which treats text and image as two parallel streams that interact through joint attention.

Two streams, one attention

The MMDiT block has two independent pathways:

1. Image stream: Processes patch embeddings of the noisy latent z_t
2. Text stream: Processes token embeddings from the text encoders

Each stream has its own LayerNorm, linear projections for Q/K/V, and MLP. But for the attention operation, the two sequences are concatenated and attention is computed jointly. This means every image patch can attend to every text token, and every text token can attend to every image patch.

Modulation via timestep

Both streams are modulated by the diffusion timestep t and a pooled text embedding y (from CLIP). These modulation parameters (α, β, γ) scale and shift the hidden states — identical to DiT's adaptive layer norm (adaLN). Each stream gets its own set of modulation parameters, so the same timestep signal can affect text and image processing differently.

Scaling the model

The model size is controlled by a single parameter: depth d (number of MMDiT blocks). The hidden dimension is 64d, the MLP expands to 4 × 64d, and there are d attention heads. This gives a clean scaling axis: d=15 is ~500M params, d=24 is ~2B, d=38 is ~8B.

The full forward pass with shapes (1024×1024 generation, d=38)

QK-Normalization

When finetuning on high resolutions, the attention logits can grow uncontrollably — the largest attention values explode, causing entropy collapse (all attention weight concentrates on one token). SD3 applies RMSNorm to Q and K before computing attention, which stabilizes training and allows efficient bf16 mixed-precision even at 1024×1024.

Chapter 3: Text Conditioning

SD3 doesn't rely on a single text encoder. It uses three pretrained text models, each capturing different aspects of the prompt:

CLIP-L/14

OpenAI's CLIP with a ViT-L/14 image encoder. Produces a 77-token sequence embedding and a pooled 768-dim vector. Good at visual-semantic alignment — it knows what words "look like" because it was trained on image-text pairs.

CLIP-G/14

A larger CLIP variant (ViT-bigG/14). Also produces 77 tokens + a pooled vector. The larger model captures more nuanced visual concepts.

T5-XXL

Google's 4.7B parameter text-to-text transformer. Unlike CLIP (which is trained on image-text pairs), T5 is a pure language model. It understands complex sentence structure, negation, spatial relationships, and counting — precisely the things CLIP struggles with.

How they combine

The two CLIP models provide the pooled text embedding y (concatenated, then projected), which drives the timestep modulation in every MMDiT block. The sequence outputs from all three encoders are concatenated into the context sequence c (77 + 77 tokens from CLIP, plus T5 tokens), which forms the text stream in MMDiT's joint attention.

Exact data flow with shapes

Chapter 4: Training

Two key training innovations separate SD3 from vanilla rectified flow: logit-normal timestep sampling and resolution-dependent shifting.

Logit-normal timestep sampling

In standard rectified flow, timesteps t are sampled uniformly from [0, 1]. But not all timesteps are equally informative. At t ≈ 0, z_t is almost pure data — easy to predict. At t ≈ 1, z_t is almost pure noise — the prediction target is roughly the dataset mean. The hardest (most informative) predictions are at intermediate timesteps.

SD3 samples timesteps from a logit-normal distribution:

where logit(t) = log(t / (1−t)). With m = 0 and s = 1, this puts most sampling weight on intermediate timesteps while still covering the endpoints with some probability.

Resolution-dependent timestep shifting

Higher resolution images have more pixels, so they need more noise to destroy the signal. Think of it this way: a 1024×1024 image has 16× more pixels than 256×256. If you average the noisy pixels (as a rough estimate of the clean image), the law of large numbers gives you a much better estimate with more pixels. So the same amount of noise is "less noisy" at higher resolutions.

SD3 addresses this with a resolution-dependent shift. Given a model pretrained at resolution n and finetuned at resolution m, the timestep mapping is:

For 1024×1024 finetuning from 256×256, a shift value of α = √(m/n) = 3.0 is used. This effectively pushes the noise schedule to add more noise at each timestep, compensating for the higher resolution.

Frozen vs. trained components

Total system: ~14.7B parameters. Of those, only the 8B MMDiT is trained for the diffusion task. The 6.7B in text encoders and VAE are pretrained and frozen — they provide the "language" and "pixel" expertise, while MMDiT learns to compose them.

Why rectified flow instead of DDPM?

Three concrete reasons:

• Fewer steps needed: SD3-8B with 50 Euler steps achieves competitive quality. With DDPM, you'd need 200+ steps for similar quality. This is 4x faster at inference.
• Simpler loss: The velocity prediction target v = ε − x₀ is constant for each (data, noise) pair — it doesn't depend on the timestep. DDPM's noise prediction target is timestep-independent too, but the SNR weighting makes some timesteps harder. Rectified flow's logit-normal sampling explicitly addresses this.
• Better scaling: Larger models learn straighter trajectories (lower path length), so they need even fewer steps. The 8B model only loses 2.71% quality going from 50 to 5 steps — a property unique to rectified flow.

Chapter 5: Scaling Laws

One of SD3's most important contributions is demonstrating that MMDiT follows predictable scaling trends, just like language models.

Validation loss scales with compute

The paper trains MMDiT models at depths d = 15, 18, 21, 24, 30, and 38 (roughly 500M to 8B parameters) for 500k steps each on 256² images. The validation loss decreases smoothly and predictably as model size increases. Crucially, there are no diminishing returns — the curves don't flatten out, suggesting even larger models would continue to improve.

Lower loss = better images

This sounds obvious but isn't guaranteed. Many generative models show poor correlation between training loss and sample quality. SD3 demonstrates a strong, monotonic relationship: lower validation loss consistently correlates with better CLIP scores, better FID, and better human preference ratings.

Larger models are more sample-efficient

The 8B model (d=38) loses only 2.71% CLIP score when reducing from 50 to 5 sampling steps. The 500M model (d=15) loses 4.30%. Larger models learn straighter trajectories, so they tolerate aggressive step reduction better. This means bigger models are not only better but also faster per quality level.

Chapter 6: Results

SD3's 8B model achieves state-of-the-art performance across multiple benchmarks, outperforming both open-source and closed-source alternatives.

GenEval benchmark

GenEval tests compositional text-to-image generation: can the model render the right number of objects, in the right colors, in the right positions? SD3 (d=38, 1024², DPO-aligned) scores 0.74 overall, compared to DALL-E 3's 0.67 and SDXL's 0.55.

The breakdown is striking:

• Two objects: 0.94 (DALL-E 3: 0.87, SDXL: 0.74)
• Color attribution: 0.60 (DALL-E 3: 0.45, SDXL: 0.23)
• Counting: 0.72 (DALL-E 3: 0.47, SDXL: 0.39)

Human preference evaluation

In head-to-head human evaluations on PartiPrompts, SD3 wins against SDXL Turbo, SDXL, Pixart-α, and DALL-E 3 across visual quality, prompt following, and typography. The typography advantage is particularly large — a direct benefit of MMDiT's joint attention allowing the model to "see" text tokens while generating image patches.

What degrades and when

• Drop T5-XXL (save 4.7B params + ~10GB VRAM): GenEval drops from 0.74 to ~0.62. Counting fails (0.72 → ~0.50), color attribution degrades (0.60 → ~0.40). Typography quality drops significantly. Visual quality is largely preserved.
• Fewer MMDiT blocks: d=15 (500M) vs d=38 (8B) — validation loss increases from 0.40 to 0.62. The 500M model can still generate coherent images but fails on complex compositions.
• Fewer sampling steps: 50 steps → 5 steps: the 8B model only loses 2.71% CLIP score. The 500M model loses 4.30%. Below 5 steps, quality degrades sharply for all sizes.
• 4-channel VAE instead of 16-channel: Reconstruction FID jumps from 1.06 to 2.41 — fine details (textures, small text, thin lines) become blurry regardless of MMDiT quality.
• No DPO alignment: GenEval drops by ~0.03-0.05. Human preference ratings decrease noticeably. DPO adds negligible inference cost but requires human preference data.

Chapter 7: Comparison with SDXL & DALL-E 3

Understanding what changed from SDXL to SD3 reveals why each design choice matters.

SDXL vs SD3: Architecture

DALL-E 3 vs SD3

DALL-E 3 (OpenAI) uses a U-Net with cross-attention and a two-stage approach: a prior model generates CLIP embeddings from text, then a diffusion model generates images from those embeddings. SD3 eliminates this two-stage pipeline entirely — text encoders feed directly into MMDiT.

DALL-E 3's key innovation was training on highly descriptive synthetic captions. SD3 adopts this insight (50/50 original + CogVLM synthetic captions) while also advancing the architecture and noise formulation.

Step efficiency comparison

This is where rectified flow shines. SD3's performance degrades gracefully with fewer steps, while DDPM-based models collapse. At 5 steps, rectified flow formulations still produce coherent images, whereas traditional formulations produce blurry messes.

Complete training and inference budget

The VRAM requirement is dominated by the text encoders at inference: T5-XXL alone needs ~9.4 GB in bf16. For memory-constrained deployment, T5 can be dropped (with the compositional quality degradation noted above) to save nearly 10 GB, or quantized to int8 (~4.7 GB).

Chapter 8: The Autoencoder

Like its predecessors, SD3 operates in a compressed latent space, not in pixel space. A pretrained autoencoder maps images to and from this latent representation.

Why latent diffusion?

A 1024×1024 RGB image has ~3.1M values. Running a transformer with attention over 3.1M tokens is computationally infeasible. The autoencoder compresses the image by a factor of 8× in each spatial dimension: 1024×1024×3 → 128×128×16 — about 97% fewer values. The diffusion model works in this compact space, then the decoder reconstructs the final image.

16-channel latent space

This is a key upgrade from SDXL (4 channels) and previous Stable Diffusion versions. More channels = richer latent representation = higher reconstruction quality. The paper shows this systematically:

The 16-channel autoencoder reduces reconstruction FID by 56% compared to 4-channel. This is crucial because the autoencoder's reconstruction quality is an upper bound on the final image quality — the diffusion model can never generate images better than the autoencoder can reconstruct.

Patching

The 128×128×16 latent is further divided into 2×2 patches, yielding 64×64 = 4096 tokens, each of dimension 16×4 = 64. These tokens form the image sequence in MMDiT. This patch size matches DiT's design and keeps the sequence length manageable for attention.

VAE training (separate stage)

The 16-channel autoencoder is trained independently before any diffusion training begins. It uses a combination of:

• Reconstruction loss: L1 + perceptual (LPIPS) + adversarial (PatchGAN discriminator)
• Regularization: KL divergence toward N(0, I) in the latent space

Once trained, the VAE is frozen for all subsequent diffusion training. This two-stage approach is inherited from the original Latent Diffusion paper (Rombach et al., 2022). The VAE has ~80M parameters — small compared to the 8B MMDiT it supports.

Why the VAE is a bottleneck: The autoencoder's reconstruction quality is an absolute ceiling on final image quality. No matter how perfect the MMDiT becomes, it cannot generate details the VAE cannot reconstruct. This is why upgrading from 4 to 16 channels matters so much — it lifts the ceiling. Fine details like thin lines, small text, and fabric textures that were permanently lost in a 4-channel latent are now preserved.

Chapter 9: Connections

SD3 sits at the intersection of several research threads. Understanding these connections clarifies both where SD3 came from and where the field is going.

DiT (Peebles & Xie, 2023)

MMDiT is a direct descendant of DiT, which first showed that replacing U-Net with a transformer backbone for diffusion models works and scales. DiT used class-conditional generation with adaLN modulation. MMDiT extends this to text-to-image by adding the text stream and joint attention. In fact, DiT with concatenated text+image tokens is a special case of MMDiT with shared weights.

DDPM & Diffusion Models (Ho et al., 2020)

DDPM established the modern diffusion framework: define a forward noising process, train a network to reverse it. SD3's rectified flow is a different choice of forward process (— straight lines instead of curved variance-preserving paths —) but the same fundamental idea: learn to undo the noise.

Flow Matching (Lipman et al., 2023)

Rectified flow is a special case of the flow matching framework. Flow matching provides the theoretical foundation: you can train velocity fields by matching conditional vector fields along specified probability paths. Rectified flow simply chooses the straight-line path z_t = (1−t)x₀ + tε. SD3's contribution is showing this works at scale with the right timestep sampling.

Flux (Black Forest Labs, 2024)

Several SD3 authors later founded Black Forest Labs and created Flux, which builds on the same MMDiT foundation but introduces a hybrid architecture: early layers use MMDiT-style joint attention, while later layers use single-stream attention (text and image tokens are fully merged). Flux also drops the T5 encoder's contribution to the pooled embedding and simplifies the modulation mechanism.

Latent Diffusion (Rombach et al., 2022)

SD3 inherits the core LDM insight: train the diffusion model in a compressed latent space, not in pixel space. The key upgrade is moving from 4 to 16 latent channels and using a transformer backbone instead of a U-Net.

What each upgrade contributed