Understand Generative
Adversarial Networks

Chapter 1: Generator & Discriminator

The generator takes random noise z (sampled from a simple distribution like a Gaussian) and transforms it into a data sample (e.g., an image). It's a neural network mapping from noise space to data space: G(z) → fake image.

The discriminator takes a data sample (real or generated) and outputs a single number: the probability that the input is real. It's a binary classifier: D(x) → [0, 1]. Architecturally, it's typically a CNN that downsamples the image through strided convolutions until a final dense layer produces the scalar output.

Concrete Data Flow (64×64 Image GAN)

Let's trace the exact tensor shapes through a DCGAN generating 64×64 RGB images:

Each layer in the generator uses: TransposedConv2d (stride=2, padding=1, kernel=4) → BatchNorm → ReLU. Each layer in the discriminator uses: Conv2d (stride=2, padding=1, kernel=4) → BatchNorm → LeakyReLU(0.2). The final generator layer uses tanh (output range [−1, 1]), and the final discriminator layer uses sigmoid (output range [0, 1]). Real images are normalized to [−1, 1] to match.

Chapter 2: The Min-Max Game

The GAN training objective is a minimax game. The discriminator tries to maximize V(D,G), the generator tries to minimize it. At the Nash equilibrium, the generator produces perfect fakes and the discriminator outputs 0.5 for everything ("I can't tell").

Breaking this down: D wants to maximize log D(x) (output 1 for real data) AND maximize log(1 − D(G(z))) (output 0 for fakes). G wants to minimize log(1 − D(G(z))) (make D output 1 for fakes).

Why the Non-Saturating Loss

Early in training, D easily rejects all fakes: D(G(z)) ≈ 0. The gradient of log(1 − D(G(z))) at D(G(z))=0 is nearly zero — G gets no learning signal. But the gradient of −log(D(G(z))) at D(G(z))=0 is very large — G gets a strong push to improve. Same equilibrium, much better gradient landscape.

The non-saturating loss is the default in virtually every modern GAN implementation. It's such a universal trick that many papers don't even mention it — it's just assumed.

Chapter 3: Training Dynamics

GANs are trained by alternating updates. One step for D, one step for G, repeat. On each D step, we show it a batch of real data (label: 1) and a batch of fake data from G (label: 0). On each G step, we generate fakes and update G to make D's output on them closer to 1.

This works — sometimes. GAN training is notoriously unstable. If D gets too good too fast, G gets no useful gradient signal. If G gets too good, D can't learn. The balance is delicate.

The Training Loop in Code

One iteration looks like this: (1) sample a real batch from your dataset, (2) sample z and generate a fake batch through G, (3) train D on both (real label=1, fake label=0) with binary cross-entropy, (4) sample new z, generate fakes, and train G to make D output 1. Two separate Adam optimizers — one for D, one for G — with learning rate balance being critical (typically 2e-4 for both, β₁=0.5).

python
# One training iteration
real_batch = next(dataloader)           # [B, 3, 64, 64]
z = torch.randn(B, 128)               # noise vector
fake_batch = G(z)                      # [B, 3, 64, 64]

# Train D: maximize log(D(real)) + log(1-D(fake))
loss_D = BCE(D(real_batch), 1) + BCE(D(fake_batch.detach()), 0)
opt_D.step()

# Train G: maximize log(D(G(z)))
z2 = torch.randn(B, 128)
loss_G = BCE(D(G(z2)), 1)             # fool D
opt_G.step()

python
def train_step(G, D, opt_G, opt_D, real_batch, z_dim=128):
    B = real_batch.size(0)
    ones = torch.ones(B, 1).to(real_batch.device)
    zeros = torch.zeros(B, 1).to(real_batch.device)
    criterion = nn.BCEWithLogitsLoss()

    # ---- Train D ----
    opt_D.zero_grad()
    # Real
    d_real = D(real_batch)
    loss_real = criterion(d_real, ones)
    # Fake (detach! don't update G)
    z = torch.randn(B, z_dim).to(real_batch.device)
    fake = G(z).detach()
    d_fake = D(fake)
    loss_fake = criterion(d_fake, zeros)
    d_loss = loss_real + loss_fake
    d_loss.backward()
    opt_D.step()

    # ---- Train G ----
    opt_G.zero_grad()
    z2 = torch.randn(B, z_dim).to(real_batch.device)
    fake2 = G(z2)           # fresh fakes, no detach
    d_fake2 = D(fake2)
    g_loss = criterion(d_fake2, ones)  # non-saturating: fool D
    g_loss.backward()
    opt_G.step()

    return d_loss.item(), g_loss.item()

Chapter 4: Mode Collapse

The generator's worst failure mode: it finds one thing that fools the discriminator and repeats it forever. If the real data has many modes (faces with glasses, without glasses, smiling, frowning), the generator might only learn to produce one type of face. This is mode collapse.

Why does it happen? The generator optimizes to fool D, not to cover all modes. If one particular output consistently gets high D scores, G concentrates there. D eventually catches on, but G just jumps to a new single mode — creating a cycle called mode hopping.

Detection and Fixes

Detection: Generate 1000 samples, compute pairwise distances. If the standard deviation of distances is near zero, you have collapse. You can also track the Frechet Inception Distance (FID) — it measures both quality and diversity. A low FID with high intra-class similarity screams mode collapse.

Fixes: Spectral normalization constrains D's Lipschitz constant per-layer, preventing it from becoming too sharp. Progressive growing starts training at 4×4 resolution and slowly adds layers, giving G time to learn at each scale. Minibatch discrimination gives D statistics about the whole batch, so it can detect when all samples look identical.

Chapter 5: WGAN & Stabilization

The original GAN loss has a fundamental problem: when D is perfect, the gradients for G vanish. The Wasserstein GAN (WGAN) fixes this by replacing the JS divergence with the Wasserstein distance (Earth Mover's Distance) — a metric that provides useful gradients even when distributions don't overlap.

The WGAN critic (no longer called a "discriminator" since it doesn't classify) outputs an unbounded score, not a probability. No sigmoid on the final layer. The score difference between real and fake data estimates the Wasserstein distance. But the critic must satisfy a Lipschitz constraint: its output can't change too fast as the input changes. Without this constraint, the critic could assign arbitrarily large scores, making the optimization meaningless.

Gradient Penalty: The Implementation

Weight clipping is crude — it pushes weights to the edges of [−c, c], wasting capacity. The gradient penalty (WGAN-GP) is much better. It samples random interpolations between real and fake data, then penalizes deviations from unit gradient norm:

This enforces the 1-Lipschitz constraint: D's output can change by at most 1 unit per unit change in input. The interpolation trick samples points between the real and fake distributions — exactly where the gradient matters most. Typical λ = 10. The total critic loss becomes: E[D(fake)] - E[D(real)] + λ * GP.

Chapter 6: StyleGAN

StyleGAN (Karras et al., 2019) revolutionized image generation with two key ideas: style mixing (inject different noise vectors at different layers to control coarse vs fine features) and progressive growing (start training at low resolution and gradually increase). The result: photorealistic 1024×1024 faces that fooled humans ~50% of the time.

The generator architecture uses a mapping network (8-layer MLP) to transform z into a style vector w, which is then injected into each layer via adaptive instance normalization (AdaIN). Different layers control different scales: early layers = pose, face shape; later layers = hair color, fine texture.

StyleGAN by the Numbers

In StyleGAN2 (1024×1024 faces): z is 512-dim, the mapping network is 8 fully-connected layers (512→512 each with LeakyReLU), producing w (also 512-dim). The synthesis network has 18 layers (2 per resolution from 4×4 to 1024×1024). Style w is injected at every layer via weight demodulation (StyleGAN2 replaced AdaIN). Total parameters: ~30M for the mapping network + ~30M for the synthesis network.

Chapter 7: Conditional GANs

Vanilla GANs generate random samples. Conditional GANs generate samples conditioned on some input: a class label, a text prompt, or even another image. The condition is fed to both G and D.

pix2pix translates one image to another (sketch → photo, satellite → map). CycleGAN does unpaired translation (horses ↔ zebras) using a cycle consistency loss. The PatchGAN discriminator classifies each N×N patch as real or fake, rather than the whole image.

How Conditioning Works

In a class-conditional GAN, the condition (e.g., "cat" = class 5) is embedded into a vector and concatenated with z before entering G. In D, the class embedding is concatenated with the feature vector before the final classification. For image-conditional GANs like pix2pix, the source image is literally concatenated channel-wise with the input: D receives a 6-channel input (3 from the source, 3 from the generated/real target).

The projection discriminator (Miyato & Koyama, 2018) is more elegant: instead of concatenation, it takes the inner product between the class embedding and D's feature vector. This gives a per-class real/fake score and trains more stably than concatenation, especially with many classes.

Chapter 8: GANs Today

For pure image quality and diversity, diffusion models have overtaken GANs. Models like Stable Diffusion, DALL-E 3, and Imagen produce higher-quality, more diverse outputs with more stable training. GANs had their era — roughly 2014-2021 — as the undisputed kings of generative modeling.

But GANs aren't dead. They remain dominant where speed matters: real-time face filters (Snapchat, TikTok), game asset generation, super-resolution (Real-ESRGAN), video prediction, and fast inference. A GAN generates an image in one forward pass; diffusion models need 20–50 denoising steps. For applications requiring <100ms latency, GANs are still the only game in town.

The Speed Gap

A StyleGAN2 generator produces a 1024×1024 image in ~50ms on a single GPU. Stable Diffusion takes ~3 seconds (50 steps at ~60ms each) for a 512×512 image. That's a 60x speed gap. For real-time applications (face filters at 30fps, game asset generation, live video), GANs remain the only viable option. Diffusion models need consistency distillation or adversarial distillation to close this gap.