Distributed PyTorch: DDP and FSDP from the Inside

Eight GPUs can compute exactly the gradient of one GPU with an eight times bigger batch. The price is communication: in a ring all-reduce every GPU sends 2(N−1)/N times the gradient, 1.75× for 8 GPUs, and never more than 2× however many GPUs join.

Learn what DistributedDataParallel and FSDP do inside: processes and ranks, all-reduce on a ring, gradient buckets that overlap backward, no_sync, SyncBatchNorm, and ZeRO's sharded memory, with the bugs that hang a cluster.

Slide the number of GPUs and watch the chunks travel round the ring. Then switch to FSDP: each GPU keeps one shard of the weights, gathers the rest just before use, and scatters the gradients back. Watch memory and traffic per GPU.

You need the training loop (part 5) and the memory budget of training (part 7). We build everything else from zero.

Round the ring

Ready
4
1.3B

Memory counts model state only, in mixed precision with AdamW: 16 bytes per parameter (2 for bf16 weights, 2 for bf16 gradients, 12 for the fp32 master copy and Adam's two moments), as in the ZeRO paper. Activations are left out. Traffic assumes bf16 (2 bytes) on the wire. FSDP here shards everything, like ZeRO stage 3; the drawing shows one FSDP unit for clarity.

Chapter 0

One GPU is not enough

A training run hits two walls: it takes too long, or it does not fit. Find out which wall you hit before you pick a tool

You want to train a 7-billion-parameter language model. You have one 80 GB GPU. You type the training loop from part 5, press run, and get CUDA out of memory before the first step.

So you try a 1.3-billion-parameter model instead. It fits. The progress bar says six days. You have 64 GPUs in the cluster, and 63 of them are idle.

These are the two walls of training. The memory wall: the model's state does not fit on one GPU. The time wall: it fits, but one GPU is too slow. Each wall has its own tool, and this lesson builds both from the inside.

Ψ
The number of parameters in the model.
memory
The model state: weights, gradients and optimizer state. With AdamW in mixed precision it is 16 bytes per parameter. Part 7 derives the 16; activations come on top.
D
The number of training tokens (or samples) the run processes.
compute
Floating-point operations for the whole run: about 2Ψ per token forward and 4Ψ backward (Kaplan et al., 2020).

Worked, 7B parameters on 140B tokens (20 tokens per parameter, the Chinchilla ratio): memory 16 × 7 × 109 = 112 GB, which does not fit in 80 GB. Compute 6 × 7 × 109 × 1.4 × 1011 = 5.9 × 1021 FLOPs. At 400 TFLOP/s that is 1.5 × 107 s, about 170 days on one GPU.

Which wall do you hit?

Slide the model size and the number of GPUs. The left bars are model-state memory per GPU against an 80 GB card, copied on every GPU (DDP) or sharded across them (FSDP). The lower bars are the time to train on 1 GPU and on N.

7.0B
8

Illustrative assumptions: 20 tokens per parameter, 400 TFLOP/s achieved per GPU, perfect scaling with N (real runs lose some to communication), 16 bytes per parameter of model state. Activations are left out; they make the memory wall come sooner.

Data parallelism attacks the time wall. Every GPU holds a full copy of the model and works on a different slice of the batch. DistributedDataParallel (DDP) is PyTorch's version. It needs the model to fit on one GPU.

Sharding attacks the memory wall. Each GPU keeps only a slice of the weights, gradients and optimizer state, and borrows the rest for a moment when it needs them. Fully Sharded Data Parallel (FSDP) is PyTorch's version of this idea, which the ZeRO paper named.

The road map of this lesson

  1. Same maths as one big batch: why averaging gradients works, and when it does not. Why it works
  2. All-reduce on a ring: the collective under DDP, and the 2(N−1)/N. Build it
  3. Processes, ranks and torchrun: a real DDP script, line by line. Build it in PyTorch
  4. Buckets and overlap: how DDP hides communication behind backward. Build it
  5. Accumulation and no_sync: bigger batches without more traffic. Build it
  6. BatchNorm, seeds and SyncBatchNorm: the parts that do not average. When not
  7. ZeRO: who stores what: the memory maths of stages 1, 2 and 3. Why it works
  8. FSDP from the inside: gather before use, scatter after, fully_shard. Build it in PyTorch
  9. Tensor and pipeline parallelism: splitting the model itself, at a glance. Where else
  10. When it goes wrong: hangs, timeouts, duplicate logs and silent drift. Did it work?
  11. Connections: the cheat sheet, a decision table and the rest of the series.

PyTorch, Zero to Expert

This is part 9 of an eleven-part series. Each lesson has Code Labs that run the PyTorch API in your browser.

  1. PyTorch Tensors: Storage, Strides and Views
  2. Autograd: the Graph Behind backward()
  3. nn.Module Inside Out
  4. Datasets and DataLoaders
  5. The Training Loop, Line by Line
  6. Debugging Training: When the Loss Won't Go Down
  7. Memory and Mixed Precision
  8. Making PyTorch Fast
  9. Distributed PyTorch: DDP and FSDP from the Inside (this lesson)
  10. PyTorch Internals: From a Python Call to a Kernel
  11. Build PyTorch from Scratch
A 3B-parameter model trains with AdamW in mixed precision on 80 GB GPUs. You move from 1 GPU to 8 with DDP. What happens to memory per GPU?

Chapter 1

Same maths as one big batch

Average the gradients of N shards and you get the gradient of the whole batch, exactly. Find out why, and the three cases where it breaks

Think of a class of 32 students who must find the average height of a group. One student can measure all 32 people. Or four students can each measure 8 people and report their own averages. The average of the four reports is the same number, if each student measured the same number of people.

Training works the same way. The loss of a batch is a mean over its examples. The gradient of a mean is the mean of the per-example gradients. So the batch gradient is just one more average, and averages can be split.

g
The gradient of the mean loss over the whole batch: what one big GPU would compute.
B
The global batch size: the examples in one optimizer step, across all GPUs.
N
The number of GPUs (processes), each with a full copy of the model.
gr
The local gradient on GPU r: the mean over its own shard of b examples.

Worked, four per-example gradients 2, 4, 6, 8 on two GPUs: the shard means are 3 and 7, their average is 5, and the mean of all four is 5. Now split them 3 + 1: shard means 4 and 8 average to 6, which is wrong. Weighting by shard size fixes it: (3 × 4 + 1 × 8) / 4 = 5.

That is the whole idea of data parallelism. Start every GPU with the same weights. Give each GPU a different shard. Average the gradients. Every GPU then takes the same optimizer step and stays identical. The averaging step is called an all-reduce: every GPU contributes a tensor, and every GPU receives the sum (DDP then divides by N).

Split the batch, average the gradients

Each pink bar is one example's gradient for a single weight. Choose how many GPUs share the batch and how the batch is cut. Then choose how to combine the GPUs' answers. The green line is the big-batch gradient.

Per-example gradients are fixed illustrative numbers for one weight of a linear model. DistributedSampler pads the dataset so every rank gets the same count; the short shard here is what you get with a plain loader, or with drop_last=False on the dataset's last batch.

The proof needs three facts to be true. Each one fails in a real case.

Two GPUs hold 6 and 2 examples. Their local mean gradients are 1.0 and 3.0. What gradient does one GPU with all 8 examples compute?

Chapter 2

All-reduce on a ring

Every GPU must end with the sum of everyone's gradients. Derive why a ring sends only 2(N−1)/N of the data per GPU, and why that cannot be beaten

Eight GPUs each hold a 2 GB gradient. Every GPU needs the sum. The obvious plan: everyone sends to GPU 0, GPU 0 adds, then sends the sum back. GPU 0 must receive 7 × 2 = 14 GB and send another 14 GB. Its link does all the work while the other links sit idle.

A collective is a communication step that every process in a group joins. The ones you will meet in this lesson:

The ring algorithm uses every link at once. Put the N GPUs in a circle. Cut each gradient into N chunks. It runs in two phases of N−1 steps each. In the first phase, a reduce-scatter, every GPU sends one chunk to its right-hand neighbour, which adds it to its own copy and passes the running sum on next step. After N−1 steps each GPU holds one chunk that contains everyone's sum. In the second phase, an all-gather, the finished chunks travel round the ring once more, and each GPU copies them.

M
The size of the tensor being reduced, in bytes (for DDP, one bucket of gradients).
N
The number of GPUs in the ring.
sent
Bytes each GPU sends: 2(N−1) steps, one chunk of M/N per step. Each GPU receives the same amount.
α
The latency: the fixed cost of one message, whatever its size.
β
The bandwidth of one link, in bytes per second.

Worked, M = 2 GB, N = 8, β = 100 GB/s (illustrative): each GPU sends 2 × 7/8 × 2 = 3.5 GB, which takes 35 ms. The send-to-GPU-0 plan pushes 28 GB through one link: 280 ms. With 1,000 GPUs the ring sends 2 × 999/1000 × 2 ≈ 4 GB per GPU: almost no more.

Why can no algorithm do much better? Each GPU must receive the other N−1 GPUs' contribution to its share of the result, and must send its own contribution out. Patarasuk and Yuan (2009) proved that the ring's bandwidth term is the lowest possible for an all-reduce. NVIDIA's nccl-tests use the same factor: for all-reduce, the "bus bandwidth" they report is the algorithm bandwidth times 2(n−1)/n.

Ring against one root

Slide the message size and the number of GPUs. The curves are the time of one all-reduce as N grows (log scales). Find the size where the ring's latency term, 2(N−1)α, starts to matter.

200 MB
8

The α–β model with illustrative α = 10 µs per message and β = 100 GB/s per link. Real libraries also use tree algorithms for small messages, where latency dominates, and hierarchical rings across machines.

A ring all-reduce of a 1 GB gradient runs on 4 GPUs. How much does each GPU send?

Chapter 3

Processes, ranks and torchrun

DDP runs one Python process per GPU. Learn how each process finds out who it is, finds the others, and gets its own slice of the data

Picture a relay team that has never met. Before the race, each runner needs three facts: how many runners there are, which number they wear, and where the team meets. Then they need a way to share the baton. DDP's processes need the same facts.

DDP starts one process per GPU. Each process runs the same script, holds its own copy of the model and its own optimizer, and talks to the others only through collectives. One process per GPU avoids Python's global interpreter lock: each GPU gets a whole Python interpreter feeding it.

node
The machine's index, 0 to nodes − 1 (torchrun's --node_rank).
G
Processes per machine, usually one per GPU (--nproc_per_node).
local_rank
The process's index on its machine; it picks cuda:local_rank.
rank
The global index. Rank and local rank are equal only on the first machine.
world
The world size: the total number of processes.

Worked, 2 machines with 4 GPUs each: the process on machine 1, GPU 2 has rank 1 × 4 + 2 = 6, local rank 2, world size 8. A bug that uses rank to pick the GPU asks for cuda:6 on a 4-GPU machine and crashes.

You do not start the processes yourself. The launcher torchrun starts G copies of your script on each machine and sets environment variables for each: RANK, LOCAL_RANK, WORLD_SIZE, LOCAL_WORLD_SIZE, MASTER_ADDR and MASTER_PORT. The master address and port are where the processes meet. Then dist.init_process_group(backend) reads them, waits until all processes have joined, and builds the process group that collectives run on.

Who is who

Set the cluster. Then tap a GPU (or move the last slider) to pick a process. You see the environment torchrun gives it, and the dataset indices its DistributedSampler hands it in epoch 0.

2
4
6

The dataset has 20 samples. The sampler uses PyTorch's rule: shuffle with seed + epoch, pad to a multiple of the world size, then rank r takes every world-th index from r. The shuffle here is a stand-in permutation, not torch's random stream.

Here is a complete DDP script. Read it line by line; each line answers one of the questions above.

Run this on a machine with PyTorch and 2+ GPUs: torchrun --nproc_per_node=2 train_ddp.py

# train_ddp.py  -- launch: torchrun --nproc_per_node=2 train_ddp.py
import os, torch, torch.nn as nn, torch.distributed as dist
from torch.nn.parallel import DistributedDataParallel as DDP
from torch.utils.data import DataLoader, TensorDataset
from torch.utils.data.distributed import DistributedSampler

def main():
    rank = int(os.environ["RANK"])              # 0 .. world-1, across all machines
    local_rank = int(os.environ["LOCAL_RANK"])  # 0 .. GPUs-per-machine - 1
    world = int(os.environ["WORLD_SIZE"])
    if torch.cuda.is_available():
        torch.cuda.set_device(local_rank)       # before any CUDA work
        device, backend = torch.device("cuda", local_rank), "nccl"
    else:
        device, backend = torch.device("cpu"), "gloo"   # lets you try it on a laptop
    dist.init_process_group(backend)            # reads MASTER_ADDR/PORT set by torchrun

    g = torch.Generator().manual_seed(0)        # the SAME data on every rank
    X = torch.randn(4096, 32, generator=g)
    y = (X @ torch.randn(32, 4, generator=g)).argmax(1)
    torch.manual_seed(1234 + rank)              # but different dropout/augment per rank
    sampler = DistributedSampler(TensorDataset(X, y), shuffle=True, seed=0)
    loader = DataLoader(TensorDataset(X, y), batch_size=64, sampler=sampler)

    model = nn.Sequential(nn.Linear(32, 128), nn.ReLU(), nn.Linear(128, 4)).to(device)
    model = DDP(model, device_ids=[local_rank] if device.type == "cuda" else None)
    opt = torch.optim.AdamW(model.parameters(), lr=1e-3)

    for epoch in range(3):
        sampler.set_epoch(epoch)                # a new shuffle every epoch
        for xb, yb in loader:
            xb, yb = xb.to(device), yb.to(device)
            loss = nn.functional.cross_entropy(model(xb), yb)
            opt.zero_grad(set_to_none=True)
            loss.backward()                     # buckets all-reduce during backward
            opt.step()
        if rank == 0:                           # one process logs and saves
            print(f"epoch {epoch}: loss {loss.item():.3f} on {world} processes")
            torch.save(model.module.state_dict(), "ckpt.pt")
    dist.barrier()                              # nobody exits before the save
    dist.destroy_process_group()

if __name__ == "__main__":
    main()
A job runs on 3 machines with 8 GPUs each. Which values does the process on machine 2, GPU 5 get?

Chapter 4

Buckets and overlap

DDP starts communicating before backward has finished. See how hooks, buckets and a reversed order hide most of the all-reduce

A plain approach would wait for backward() to finish, then all-reduce every gradient. The network would sit idle during backward, and the GPU would sit idle during the all-reduce. The DDP paper measured that communication was more than half of the backward time on its 32-GPU tests.

Look at the order in which backward produces gradients. It starts at the loss and walks back, so the last layer's gradients are finished first. Those can be on the wire while the GPU is still computing the earlier layers' gradients. That is overlap.

But one all-reduce per parameter tensor would be slow: every call pays the latency α, and a large model has hundreds of small tensors. So DDP packs gradients into buckets. Its design, from the paper (Li et al., 2020):

Tbwd
Time for the backward computation alone.
Tcomm
Time for all the bucket all-reduces, including α per bucket.
Tlast
The tail: the last bucket can only start when backward ends, so its all-reduce is never hidden.
Toverlap
The step's backward-plus-sync time when buckets overlap with computation.

Worked, illustrative: Tbwd = 200 ms, Tcomm = 150 ms in buckets, last bucket 10 ms. Serial: 350 ms. Overlapped: about 200 + 10 = 210 ms, 40% less. The paper measured 38.0% (ResNet50) and 35.2% (BERT) speed-ups from overlap with NCCL on 32 GPUs.

Hide the all-reduce behind backward

The top lane is backward, last layer first. The lower lane is the all-reduce of each bucket. Slide the bucket size from tiny to one giant bucket, and switch overlap off to see the plain approach.

25 MB
1 ms

An illustrative model of a 100M-parameter network (400 MB of fp32 gradients, 100 equal layers): backward takes 200 ms; an all-reduce takes α plus bytes × 2(N−1)/N / β with N = 8 and β = 4 GB/s. The paper found 10–25 MB best for ResNet50 and 50 MB for BERT with NCCL on 16 GPUs.

Two constructor options change this machinery. find_unused_parameters=True walks the autograd graph after each forward to mark parameters that got no gradient, so their buckets do not wait forever; it costs a graph traversal per step, so leave it off unless you need it. gradient_as_bucket_view=True makes each .grad a view into its bucket, which saves one copy of the gradients in memory.

Why does DDP fill its buckets in the reverse order of model.parameters()?

Chapter 5

Accumulation and no_sync

Build a bigger batch from K small ones, communicate once instead of K times, and avoid the loss that is K times too big

You want a global batch of 1,024, but only 32 examples fit per GPU, and you have 8 GPUs. Eight GPUs give 256 per step. Gradient accumulation runs K = 4 forward and backward passes before each optimizer step. .grad adds up across them, so the step sees all 1,024 examples.

With DDP there is a cost. Every backward() fires DDP's hooks, and every hook launches all-reduces. Four micro-batches mean four full all-reduces per step. Three of them are wasted: averaging is linear, so you can sum locally and average once.

The context manager model.no_sync() switches DDP's hooks off. Backward passes inside it only accumulate into the local .grad. The first backward outside it syncs everything accumulated so far.

K
The number of micro-batches per optimizer step on each GPU.
Lk
The mean loss over micro-batch k. Dividing by K makes the sum of K means equal the mean over all K micro-batches.
g
The gradient after K backward calls: the gradient of the mean over the whole accumulated batch.
traffic
Bytes each GPU sends per optimizer step, without and with no_sync; M is the gradient size.

Worked, a 1B-parameter model with bf16 gradients (M = 2 GB), N = 8, K = 4: syncing every micro-batch sends 4 × 3.5 = 14 GB per GPU per step; with no_sync, 3.5 GB. The DDP paper measured 38% (NCCL) and 57% (gloo) speed-ups for ResNet50 on 256 GPUs when syncing every 8 iterations.

Sync once per step

Each pink block is one micro-batch's forward and backward; each purple block is an all-reduce. Switch the two fixes on and off. The bar on the right is the size of the final gradient compared with the true one.

4

Illustrative times: 60 ms per micro-batch, 35 ms per all-reduce that is not hidden by overlap. Gradient size is relative: 1.0 is the gradient of the mean over all K micro-batches.

The pattern in real code. The function works for any DDP-wrapped model; with real DDP on 2 processes it gives the same weights as one big batch.

Run this inside a DDP script such as the one in Chapter 3

import contextlib
import torch.nn.functional as F

def train_step(model, opt, micro_batches):
    """One optimizer step over K micro-batches. model is wrapped in DDP."""
    K = len(micro_batches)
    for i, (x, y) in enumerate(micro_batches):
        last = i == K - 1
        ctx = contextlib.nullcontext() if last else model.no_sync()
        with ctx:                                  # no all-reduce inside no_sync()
            loss = F.cross_entropy(model(x), y) / K  # each micro-batch is 1/K
            loss.backward()                        # .grad accumulates locally
    opt.step()                                     # grads were synced on the last one
    opt.zero_grad(set_to_none=True)
You accumulate K = 8 micro-batches with DDP, use no_sync on the first 7, but forget to divide the loss by 8. What happens?

Chapter 6

BatchNorm, seeds and SyncBatchNorm

Some parts of a model look across the batch or draw random numbers. Those do not average away. Learn which, and the fix for each

BatchNorm normalises each feature with the mean and variance of the current batch. Under DDP, "the current batch" is the GPU's own shard. With 256 examples per GPU that is fine. With 2 per GPU, as in detection and segmentation with large images, each GPU's statistics are mostly noise.

How noisy? The mean of b random values with spread σ has its own spread σ/√b. Pool the statistics over all N GPUs and the spread falls to σ/√(N·b).

σ
The true spread (standard deviation) of one feature across examples.
b
Examples per GPU.
N
The number of GPUs whose statistics are pooled.
spread of μ̂
How far the batch mean typically lands from the true mean. BatchNorm subtracts this estimate from every example.

Worked: b = 2 on N = 16 GPUs. Alone, each GPU's mean is off by about σ / 1.41 = 0.71σ. Synced over 32 examples: σ / 5.7 = 0.18σ. With b = 1 the variance is zero, and PyTorch refuses: ValueError: Expected more than 1 value per channel when training.

How wrong is the batch mean?

Each pink dot is one GPU's estimate of a feature's mean; the green line is the true mean. Shrink the per-GPU batch and watch the dots scatter. Switch to SyncBatchNorm: every GPU uses one pooled estimate.

2
16

Each GPU draws b values from a normal distribution with mean 0 and spread 1, with a fixed seed so the picture is stable. The shaded band is ±1 expected spread.

SyncBatchNorm fixes it. In training mode, each GPU computes its shard's per-channel mean and variance, and an all-gather shares them with the count. Every GPU then combines them into the global statistics and normalises with those. The backward pass syncs the matching gradient terms too. You turn it on with one line, before wrapping in DDP:

# inside the DDP script, after building the model, before DDP(...)
model = torch.nn.SyncBatchNorm.convert_sync_batchnorm(model)   # every BatchNorm*D -> SyncBatchNorm
model = DDP(model.to(device), device_ids=[local_rank])

Combining the statistics is the part to understand. Means combine by weighting with counts. Variances do not simply average: the spread between the GPUs' means is part of the total spread. Combine second moments instead.

Random numbers on many GPUs

Each process has its own random stream. Three rules keep them right:

Where else these collectives work

The same few collectives appear wherever many processes must agree:

Two GPUs each see 2 examples of one feature: GPU 0 has 0 and 2, GPU 1 has 4 and 6. Each GPU's own variance is 1. What variance does SyncBatchNorm use?

Chapter 7

ZeRO: who stores what

DDP keeps N identical copies of everything. Remove the copies one kind at a time and watch memory per GPU fall

Look again at DDP's memory. Eight GPUs each hold the same weights, the same averaged gradients, and the same Adam state. Seven of the eight copies are redundant. The ZeRO paper (Rajbhandari et al., 2019), short for Zero Redundancy Optimizer, removes them in three stages.

Start from the 16 bytes per parameter of mixed-precision AdamW, derived in Memory and Mixed Precision: 2 bytes of 16-bit weights, 2 bytes of 16-bit gradients, and K = 12 bytes of optimizer state (a 32-bit master copy of the weights, plus Adam's m and v).

Ψ
The number of parameters. Every term is in bytes per GPU.
K
Bytes of optimizer state per parameter: 12 for mixed-precision Adam.
Nd
The data-parallel degree: how many GPUs share the shards.
DDP
Every GPU holds everything: 16Ψ bytes.
ZeRO-1
Optimizer state sharded. Approaches 4Ψ for large Nd: a 4× cut.
ZeRO-2
Gradients sharded too. Approaches 2Ψ: an 8× cut.
ZeRO-3
Everything sharded. Falls as 1/Nd without limit.

Worked, the paper's example (Figure 1): Ψ = 7.5B, Nd = 64. DDP: 16 × 7.5 = 120 GB. ZeRO-1: 30 + 90/64 = 31.4 GB. ZeRO-2: 15 + 105/64 = 16.6 GB. ZeRO-3: 120/64 = 1.9 GB.

Shard one kind at a time

Each bar is the model state on one GPU, split into weights (orange), gradients (blue) and optimizer state (yellow). The dashed line is an 80 GB GPU. Find the fewest GPUs that let each stage fit.

7.5B
64

The ZeRO paper's formulas for model state, in GB of 109 bytes. Activations, temporary buffers and fragmentation come on top. Communication is in parameters' worth of data per step: 2Ψ for DDP, ZeRO-1 and ZeRO-2, 3Ψ for ZeRO-3 (paper, Section 7).

What does sharding cost in traffic? Section 7 of the paper counts it. DDP's all-reduce is a reduce-scatter plus an all-gather: 2Ψ of data. ZeRO-2 does a reduce-scatter of gradients, updates its shard, and all-gathers the updated weights: still 2Ψ. ZeRO-3 must all-gather the weights twice, once for forward and once for backward, plus the reduce-scatter: 3Ψ, 1.5× DDP.

A 13B-parameter model trains with mixed-precision AdamW on 8 GPUs of 80 GB. Which setup leaves at least 40 GB per GPU free for activations?

Chapter 8

FSDP from the inside

Gather a layer's weights just before it runs, free them just after, and scatter its gradients back. Then write it with fully_shard

Think of a library with one copy of every book, spread over N branches. Each branch owns one N-th of the shelves. To read a chapter you borrow the missing pages from the other branches, read, and hand them back. That is Fully Sharded Data Parallel: ZeRO stage 3 as a PyTorch API.

FSDP works on units: groups of layers you choose, usually one transformer block each. For one training step, unit by unit:

  1. Forward: all-gather the unit's weights from all ranks; run its forward; free the gathered copy (reshard), keeping only this rank's shard.
  2. Backward: all-gather the unit's weights again; compute its gradients; free the gathered weights.
  3. Reduce-scatter the unit's gradients: rank r keeps the averaged gradient of its own shard only.
  4. Optimizer step: each rank updates its own shard, with its own shard of the optimizer state. No more communication.
N
GPUs sharing the shards.
2Ψunit
Bytes of one unit's gathered 16-bit weights. A smaller unit means a smaller temporary copy, and more, smaller collectives.
f
Units gathered at once: about 2, the one running plus the one being prefetched.
activations
Saved for backward, as without FSDP. Sharding does not touch them.
mempeak
The most memory one GPU needs during the step.

Worked, 7B parameters in 32 blocks on 8 GPUs: sharded state 16 × 7 / 8 = 14 GB. One block is 7B / 32 ≈ 0.22B parameters, 0.44 GB in bf16; two in flight add 0.9 GB. Model state about 15 GB per GPU, against 112 GB with DDP.

Memory through one FSDP step

The curve is the gathered-weight memory on one GPU through forward (left) and backward (right), above the sharded base. Each purple tick is one all-gather or reduce-scatter. Switch resharding off and count the collectives you save.

8
8

An illustrative 7B-parameter model with equal blocks, mixed-precision AdamW, no activations drawn. One unit in flight plus one prefetched. Collective counts are per step.

In PyTorch 2.x, FSDP's current API is FSDP2: the function fully_shard, imported from torch.distributed.fsdp. You call it on each block, then on the whole model, bottom-up. Each parameter becomes a DTensor sharded on its first dimension: rank r stores a slice of rows. The script below runs as written, on GPUs with NCCL or on a CPU with gloo:

Run this on a machine with PyTorch and 2+ GPUs: torchrun --nproc_per_node=2 train_fsdp2.py

# train_fsdp2.py  -- launch: torchrun --nproc_per_node=2 train_fsdp2.py
import os, torch, torch.nn as nn, torch.distributed as dist
from torch.distributed.fsdp import fully_shard, MixedPrecisionPolicy
from torch.distributed.device_mesh import init_device_mesh
from torch.distributed.checkpoint.state_dict import get_model_state_dict, StateDictOptions

class Block(nn.Module):
    def __init__(self, d):
        super().__init__()
        self.norm, self.ff = nn.LayerNorm(d), nn.Sequential(nn.Linear(d, 4 * d), nn.GELU(), nn.Linear(4 * d, d))
    def forward(self, x):
        return x + self.ff(self.norm(x))

def main():
    local_rank = int(os.environ["LOCAL_RANK"])
    cuda = torch.cuda.is_available()
    if cuda:
        torch.cuda.set_device(local_rank)
    dist.init_process_group("nccl" if cuda else "gloo")
    rank, dev = dist.get_rank(), torch.device("cuda" if cuda else "cpu")
    mesh = init_device_mesh(dev.type, (dist.get_world_size(),))   # 1-D: all ranks shard

    model = nn.Sequential(*[Block(256) for _ in range(4)], nn.Linear(256, 10)).to(dev)
    mp = MixedPrecisionPolicy(param_dtype=torch.bfloat16, reduce_dtype=torch.float32) if cuda else MixedPrecisionPolicy()
    for block in model[:4]:
        fully_shard(block, mesh=mesh, mp_policy=mp)        # one FSDP unit per block
    fully_shard(model, mesh=mesh, mp_policy=mp)           # the root: whatever is left over
    w = model[0].ff[0].weight                   # now a DTensor: this rank's rows only
    print(f"rank {rank}: {type(w).__name__} local {tuple(w.to_local().shape)} of {tuple(w.shape)}")

    opt = torch.optim.AdamW(model.parameters(), lr=1e-3)   # state is sharded too
    g = torch.Generator().manual_seed(rank)                 # each rank its own data
    for step in range(5):
        x = torch.randn(32, 256, generator=g).to(dev)
        y = torch.randint(0, 10, (32,), generator=g).to(dev)
        loss = nn.functional.cross_entropy(model(x), y)     # all-gather per block
        loss.backward()                                     # reduce-scatter per block
        opt.step(); opt.zero_grad()
    full = get_model_state_dict(model, options=StateDictOptions(full_state_dict=True, cpu_offload=True))
    if rank == 0:
        print("loss", round(loss.item(), 3), "| full checkpoint tensors:", len(full))
        torch.save(full, "fsdp_full.pt")
    dist.destroy_process_group()

if __name__ == "__main__":
    main()
With FSDP and reshard_after_forward=True, how many collectives does one unit take part in per training step?

Chapter 9

Tensor and pipeline parallelism, at a glance

Data parallelism splits the batch. The other two split the model: across the width of each layer, or across its depth

FSDP still runs each layer whole on every GPU, just with borrowed weights. Two other methods split the computation of the model itself. This chapter shows the core of each. The CS336 lessons on data and tensor parallelism and on pipeline parallelism and FSDP, and the distributed training lesson, take them further.

Tensor parallelism splits each weight matrix across GPUs. Megatron-LM (Shoeybi et al., 2019) does it for a transformer's MLP, Y = GeLU(XA), Z = YB. Split A by columns: GPU t computes Yt = GeLU(XAt), and GeLU can run on each slice alone. Split B by rows: GPU t computes YtBt, a partial sum of Z. One all-reduce adds the partial sums. The paper reports one all-reduce in the forward pass and one in the backward pass per MLP, and the same for attention.

Pipeline parallelism splits the layers into p consecutive stages, one per GPU. Only activations cross the boundaries. The problem is waiting: stage 4 has nothing to do until stage 1 has finished. Cutting the batch into m micro-batches lets the stages work on different micro-batches at once. The idle time left over is the bubble.

p
The number of pipeline stages (GPUs in the pipeline).
m
The number of micro-batches per optimizer step.
bubble
The fraction of time each GPU is idle in a GPipe-style schedule (Huang et al., 2019).

Worked: p = 4 stages, m = 4 micro-batches: bubble 3/7 = 43%. With m = 16: 3/19 = 16%. GPipe found the bubble "negligible when M ≥ 4 × K", where K is the number of partitions: m ≥ 16 here.

Split the width, or split the depth

Tensor parallel: slide the number of GPUs and see each one's slice of A and B, and the single all-reduce. Pipeline: slide stages and micro-batches and watch the red bubble shrink.

4
4

Tensor view: an MLP with hidden size d = 4096 and 4d inner size, bf16, batch of 8 sequences of 2048 tokens; numbers illustrative. Pipeline view: GPipe schedule, all forwards then all backwards, backward taking twice a forward.

Which to use? Tensor parallelism sends activations at every layer, so it needs the fastest links: GPUs inside one machine. Pipeline parallelism sends little, at stage boundaries only, so it can cross machines; but it needs many micro-batches to hide the bubble. Large runs combine all three: tensor parallel within a machine, pipeline across a few machines, and data parallel (often FSDP) across the rest. PyTorch has both built in: torch.distributed.tensor.parallel (ColwiseParallel, RowwiseParallel) and torch.distributed.pipelining (ScheduleGPipe, Schedule1F1B).

A pipeline has 8 stages. How many micro-batches keep the GPipe bubble under 10%?

Chapter 10

When it goes wrong

A distributed bug often shows as nothing at all: a job that stops moving. Learn the usual causes, the checks that find them, and the fixes

Collectives are a meeting. Every rank must arrive, in the same order, with tensors of the same shape. If one rank takes a different path through the code, the others wait at the meeting point. Nothing crashes. GPU use drops to zero, and the job sits there until a timeout ends it.

The default timeout is long. The PyTorch documentation gives 10 minutes for NCCL and 30 minutes for other backends, and you can set it with init_process_group(timeout=...). When it fires, NCCL's watchdog stops the process with a timeout error that names the collective that was waiting.

seqr
The ordered list of collectives rank r calls, with their tensor shapes and types.
rank r
Any process in the group. One mismatch anywhere stops the whole job.

Worked: 4 ranks, 1,000 steps, one all-reduce per step. If rank 0 alone runs an extra all_reduce for logging on step 100, the four sequences differ at position 101. All four ranks stop at step 100 and wait; with NCCL the job dies 10 minutes later. Worse, a mismatched call can pair with a different call on another rank and mix data silently.

Find the hang

Each row is one rank; each block is one call. Purple blocks are collectives. Pick a bug and watch the ranks run. Red marks the first collective where they no longer match.

A schematic of the call sequences, not a timing model.

The bugs experts look for first:

Two helpers that prevent and detect the worst of these. With real DDP on 2 processes they return 0.5 for ranks 0 and 1, and catch a drift planted on rank 1.

Run this inside a DDP script such as the one in Chapter 3

import torch, torch.distributed as dist

def mean_over_ranks(x: torch.Tensor) -> float:
    """EVERY rank must call this: a collective that one rank skips hangs the rest."""
    x = x.detach().clone()
    dist.all_reduce(x)                       # sum over ranks, same call on all of them
    return x.item() / dist.get_world_size()

def replicas_in_sync(model, tol=1e-6) -> bool:
    """Compare a fingerprint of the weights across ranks; DDP replicas must match."""
    v = torch.nn.utils.parameters_to_vector(model.parameters()).detach().float()
    fp = torch.stack([v.sum(), v.square().sum()])
    every = [torch.zeros_like(fp) for _ in range(dist.get_world_size())]
    dist.all_gather(every, fp)               # every rank gets every fingerprint
    return all(torch.allclose(e, every[0], atol=tol) for e in every)

# in the training loop:
#   val = mean_over_ranks(loss)              # all ranks call it ...
#   if dist.get_rank() == 0: print(val)      # ... only rank 0 prints
#   assert replicas_in_sync(model.module)    # cheap check every few hundred steps
A 64-GPU job stops making progress at the end of epoch 3, with every GPU at 0% use and no error. What do you check first?

Chapter 11

Connections

Pick a strategy for your model, keep the one-screen cheat sheet, know the limits, and go on to PyTorch's internals

Choosing a strategy comes down to two numbers: does the model state fit on one GPU, and how fast are the links between your GPUs? Set your model and cluster below for a first answer.

Pick a strategy

Slide the model size and the number of GPUs. The card suggests a starting strategy, the model-state memory per GPU it gives, and why. It is a rule of thumb, not a planner.

1.3B
8

Rules of thumb: model state 16 bytes per parameter; keep at most half of 80 GB for it, the rest for activations; a single machine has 8 GPUs with fast links. Real choices also depend on sequence length, batch size and interconnect.

The cheat sheet

PartIn one line
Data parallelSame weights, different shards, average gradients: g = (1/N) Σ gr, exact for equal shards and per-example losses.
Ring all-reduceReduce-scatter + all-gather, 2(N−1) steps; each GPU sends 2(N−1)/N · M; time 2(N−1)α + 2(N−1)M/(Nβ).
Launchtorchrun --nproc_per_node=G; RANK, LOCAL_RANK, WORLD_SIZE; set_device(local_rank) then init_process_group.
DDPBroadcast at construction; hooks + 25 MiB buckets in reverse parameter order; async all-reduce overlaps backward.
DataDistributedSampler, same seed on every rank, set_epoch(epoch) every epoch; dropout seeds differ by rank.
AccumulationLoss / K; no_sync() on the first K−1; global batch = N · K · b.
BatchNormPer-GPU stats when b is small: SyncBatchNorm.convert_sync_batchnorm, or LayerNorm / GroupNorm.
ZeRO16Ψ → 4Ψ + 12Ψ/N → 2Ψ + 14Ψ/N → 16Ψ/N; traffic 2Ψ, 2Ψ, 2Ψ, 3Ψ.
FSDP2fully_shard each block, then the root; all-gather before use, reshard after, reduce-scatter grads; DTensor dim-0 shards.
TP / PPColumn then row split, one all-reduce per MLP; pipeline bubble (p−1)/(m+p−1).
HangsSame collectives, same order, same shapes on every rank; TORCH_DISTRIBUTED_DEBUG=DETAIL.

When to use which

MethodUse it whenAvoid it whenUse instead
DDPModel state plus activations fit on one GPU; you want speed.16Ψ is over about half the GPU.ZeRO-1 or FSDP.
ZeRO-1 / -2The weights fit but Adam's state does not.The 16-bit weights alone do not fit.FSDP (ZeRO-3).
FSDPThe model does not fit; one layer does.Links are slow and the model fits: you pay 1.5× traffic for nothing.DDP.
Tensor parallelOne layer is too big or too slow; GPUs share fast links.Across machines.Pipeline or FSDP across machines.
PipelineVery deep models across machines, with many micro-batches.Few micro-batches: the bubble eats the gain.FSDP.

Did it work?

Honest limits

Where this sits, and where to go next

The takeaway. Data parallelism is one average: same weights, different shards, an all-reduce of gradients. DDP makes that average cheap with a ring and hides it behind backward with buckets. When the model stops fitting, stop copying it: shard the optimizer state, the gradients, and finally the weights, gathering each layer only for the moment it runs. And keep every rank at the same meeting points, in the same order.
A 2B-parameter model trains on 8 GPUs of 80 GB in one machine, with long sequences that need 45 GB of activations per GPU. Which strategy is the best first try?

Now press Present or Teach and explain, out loud and from memory, why averaging gradients over shards gives the big-batch gradient, and why a ring sends 2(N−1)/N of the data. Then go back to the hero, switch to FSDP, and find the GPU count where a 13B model's state drops under 40 GB per GPU.

Distributed PyTorch: DDP and FSDP from the Inside, part 9 of PyTorch, Zero to Expert, in The Builder's Toolkit. Sources: S. Li, Y. Zhao, R. Varma, O. Salpekar, P. Noordhuis, T. Li, A. Paszke, J. Smith, B. Vaughan, P. Damania and S. Chintala, "PyTorch Distributed: Experiences on Accelerating Data Parallel Training", VLDB 2020 (arXiv 2006.15704): the design of buckets, hooks, overlap and no_sync (Sections 3–4) and the measurements quoted here (Section 5). S. Rajbhandari, J. Rasley, O. Ruwase and Y. He, "ZeRO: Memory Optimizations Toward Training Trillion Parameter Models", 2019 (arXiv 1910.02054): memory formulas (Section 5, Figure 1, Table 1) and communication volume (Section 7). P. Patarasuk and X. Yuan, "Bandwidth optimal all-reduce algorithms for clusters of workstations", Journal of Parallel and Distributed Computing, 2009. NVIDIA's nccl-tests performance notes (bus bandwidth). M. Shoeybi et al., "Megatron-LM", 2019 (arXiv 1909.08053, Section 3). Y. Huang et al., "GPipe", 2019 (arXiv 1811.06965, Section 2.3). J. Kaplan et al., "Scaling Laws for Neural Language Models", 2020 (arXiv) for 6ΨD; J. Hoffmann et al., 2022 (arXiv) for 20 tokens per parameter; P. Goyal et al., 2017 (arXiv) for learning-rate scaling. PyTorch 2.10 APIs and defaults (DDP's bucket_cap_mb and first bucket, DistributedSampler, fully_shard, MixedPrecisionPolicy, init_process_group, error messages) were checked by running PyTorch 2.10; both scripts ran with torchrun on 2 processes. Timing numbers on devices are illustrative and labelled so.
Back to Gleams