Eight GPUs can compute exactly the gradient of one GPU with an eight times bigger batch. The price is communication: in a ring all-reduce every GPU sends 2(N−1)/N times the gradient, 1.75× for 8 GPUs, and never more than 2× however many GPUs join.
Learn what DistributedDataParallel and FSDP do inside: processes and ranks, all-reduce on a ring, gradient buckets that overlap backward, no_sync, SyncBatchNorm, and ZeRO's sharded memory, with the bugs that hang a cluster.
Slide the number of GPUs and watch the chunks travel round the ring. Then switch to FSDP: each GPU keeps one shard of the weights, gathers the rest just before use, and scatters the gradients back. Watch memory and traffic per GPU.
You need the training loop (part 5) and the memory budget of training (part 7). We build everything else from zero.
Memory counts model state only, in mixed precision with AdamW: 16 bytes per parameter (2 for bf16 weights, 2 for bf16 gradients, 12 for the fp32 master copy and Adam's two moments), as in the ZeRO paper. Activations are left out. Traffic assumes bf16 (2 bytes) on the wire. FSDP here shards everything, like ZeRO stage 3; the drawing shows one FSDP unit for clarity.
Chapter 0
A training run hits two walls: it takes too long, or it does not fit. Find out which wall you hit before you pick a tool
You want to train a 7-billion-parameter language model. You have one 80 GB GPU. You type the training loop from part 5, press run, and get CUDA out of memory before the first step.
So you try a 1.3-billion-parameter model instead. It fits. The progress bar says six days. You have 64 GPUs in the cluster, and 63 of them are idle.
These are the two walls of training. The memory wall: the model's state does not fit on one GPU. The time wall: it fits, but one GPU is too slow. Each wall has its own tool, and this lesson builds both from the inside.
Worked, 7B parameters on 140B tokens (20 tokens per parameter, the Chinchilla ratio): memory 16 × 7 × 109 = 112 GB, which does not fit in 80 GB. Compute 6 × 7 × 109 × 1.4 × 1011 = 5.9 × 1021 FLOPs. At 400 TFLOP/s that is 1.5 × 107 s, about 170 days on one GPU.
Slide the model size and the number of GPUs. The left bars are model-state memory per GPU against an 80 GB card, copied on every GPU (DDP) or sharded across them (FSDP). The lower bars are the time to train on 1 GPU and on N.
Illustrative assumptions: 20 tokens per parameter, 400 TFLOP/s achieved per GPU, perfect scaling with N (real runs lose some to communication), 16 bytes per parameter of model state. Activations are left out; they make the memory wall come sooner.
Data parallelism attacks the time wall. Every GPU holds a full copy of the model and works on a different slice of the batch. DistributedDataParallel (DDP) is PyTorch's version. It needs the model to fit on one GPU.
Sharding attacks the memory wall. Each GPU keeps only a slice of the weights, gradients and optimizer state, and borrows the rest for a moment when it needs them. Fully Sharded Data Parallel (FSDP) is PyTorch's version of this idea, which the ZeRO paper named.
This is part 9 of an eleven-part series. Each lesson has Code Labs that run the PyTorch API in your browser.
Chapter 1
Average the gradients of N shards and you get the gradient of the whole batch, exactly. Find out why, and the three cases where it breaks
Think of a class of 32 students who must find the average height of a group. One student can measure all 32 people. Or four students can each measure 8 people and report their own averages. The average of the four reports is the same number, if each student measured the same number of people.
Training works the same way. The loss of a batch is a mean over its examples. The gradient of a mean is the mean of the per-example gradients. So the batch gradient is just one more average, and averages can be split.
Worked, four per-example gradients 2, 4, 6, 8 on two GPUs: the shard means are 3 and 7, their average is 5, and the mean of all four is 5. Now split them 3 + 1: shard means 4 and 8 average to 6, which is wrong. Weighting by shard size fixes it: (3 × 4 + 1 × 8) / 4 = 5.
That is the whole idea of data parallelism. Start every GPU with the same weights. Give each GPU a different shard. Average the gradients. Every GPU then takes the same optimizer step and stays identical. The averaging step is called an all-reduce: every GPU contributes a tensor, and every GPU receives the sum (DDP then divides by N).
Each pink bar is one example's gradient for a single weight. Choose how many GPUs share the batch and how the batch is cut. Then choose how to combine the GPUs' answers. The green line is the big-batch gradient.
Per-example gradients are fixed illustrative numbers for one weight of a linear model. DistributedSampler pads the dataset so every rank gets the same count; the short shard here is what you get with a plain loader, or with drop_last=False on the dataset's last batch.
The proof needs three facts to be true. Each one fails in a real case.
Chapter 2
Every GPU must end with the sum of everyone's gradients. Derive why a ring sends only 2(N−1)/N of the data per GPU, and why that cannot be beaten
Eight GPUs each hold a 2 GB gradient. Every GPU needs the sum. The obvious plan: everyone sends to GPU 0, GPU 0 adds, then sends the sum back. GPU 0 must receive 7 × 2 = 14 GB and send another 14 GB. Its link does all the work while the other links sit idle.
A collective is a communication step that every process in a group joins. The ones you will meet in this lesson:
The ring algorithm uses every link at once. Put the N GPUs in a circle. Cut each gradient into N chunks. It runs in two phases of N−1 steps each. In the first phase, a reduce-scatter, every GPU sends one chunk to its right-hand neighbour, which adds it to its own copy and passes the running sum on next step. After N−1 steps each GPU holds one chunk that contains everyone's sum. In the second phase, an all-gather, the finished chunks travel round the ring once more, and each GPU copies them.
Worked, M = 2 GB, N = 8, β = 100 GB/s (illustrative): each GPU sends 2 × 7/8 × 2 = 3.5 GB, which takes 35 ms. The send-to-GPU-0 plan pushes 28 GB through one link: 280 ms. With 1,000 GPUs the ring sends 2 × 999/1000 × 2 ≈ 4 GB per GPU: almost no more.
Why can no algorithm do much better? Each GPU must receive the other N−1 GPUs' contribution to its share of the result, and must send its own contribution out. Patarasuk and Yuan (2009) proved that the ring's bandwidth term is the lowest possible for an all-reduce. NVIDIA's nccl-tests use the same factor: for all-reduce, the "bus bandwidth" they report is the algorithm bandwidth times 2(n−1)/n.
Slide the message size and the number of GPUs. The curves are the time of one all-reduce as N grows (log scales). Find the size where the ring's latency term, 2(N−1)α, starts to matter.
The α–β model with illustrative α = 10 µs per message and β = 100 GB/s per link. Real libraries also use tree algorithms for small messages, where latency dominates, and hierarchical rings across machines.
Chapter 3
DDP runs one Python process per GPU. Learn how each process finds out who it is, finds the others, and gets its own slice of the data
Picture a relay team that has never met. Before the race, each runner needs three facts: how many runners there are, which number they wear, and where the team meets. Then they need a way to share the baton. DDP's processes need the same facts.
DDP starts one process per GPU. Each process runs the same script, holds its own copy of the model and its own optimizer, and talks to the others only through collectives. One process per GPU avoids Python's global interpreter lock: each GPU gets a whole Python interpreter feeding it.
cuda:2.--node_rank).--nproc_per_node).cuda:local_rank.Worked, 2 machines with 4 GPUs each: the process on machine 1, GPU 2 has rank 1 × 4 + 2 = 6, local rank 2, world size 8. A bug that uses rank to pick the GPU asks for cuda:6 on a 4-GPU machine and crashes.
You do not start the processes yourself. The launcher torchrun starts G copies of your script on each machine and sets environment variables for each: RANK, LOCAL_RANK, WORLD_SIZE, LOCAL_WORLD_SIZE, MASTER_ADDR and MASTER_PORT. The master address and port are where the processes meet. Then dist.init_process_group(backend) reads them, waits until all processes have joined, and builds the process group that collectives run on.
Set the cluster. Then tap a GPU (or move the last slider) to pick a process. You see the environment torchrun gives it, and the dataset indices its DistributedSampler hands it in epoch 0.
The dataset has 20 samples. The sampler uses PyTorch's rule: shuffle with seed + epoch, pad to a multiple of the world size, then rank r takes every world-th index from r. The shuffle here is a stand-in permutation, not torch's random stream.
Here is a complete DDP script. Read it line by line; each line answers one of the questions above.
Run this on a machine with PyTorch and 2+ GPUs: torchrun --nproc_per_node=2 train_ddp.py
# train_ddp.py -- launch: torchrun --nproc_per_node=2 train_ddp.py import os, torch, torch.nn as nn, torch.distributed as dist from torch.nn.parallel import DistributedDataParallel as DDP from torch.utils.data import DataLoader, TensorDataset from torch.utils.data.distributed import DistributedSampler def main(): rank = int(os.environ["RANK"]) # 0 .. world-1, across all machines local_rank = int(os.environ["LOCAL_RANK"]) # 0 .. GPUs-per-machine - 1 world = int(os.environ["WORLD_SIZE"]) if torch.cuda.is_available(): torch.cuda.set_device(local_rank) # before any CUDA work device, backend = torch.device("cuda", local_rank), "nccl" else: device, backend = torch.device("cpu"), "gloo" # lets you try it on a laptop dist.init_process_group(backend) # reads MASTER_ADDR/PORT set by torchrun g = torch.Generator().manual_seed(0) # the SAME data on every rank X = torch.randn(4096, 32, generator=g) y = (X @ torch.randn(32, 4, generator=g)).argmax(1) torch.manual_seed(1234 + rank) # but different dropout/augment per rank sampler = DistributedSampler(TensorDataset(X, y), shuffle=True, seed=0) loader = DataLoader(TensorDataset(X, y), batch_size=64, sampler=sampler) model = nn.Sequential(nn.Linear(32, 128), nn.ReLU(), nn.Linear(128, 4)).to(device) model = DDP(model, device_ids=[local_rank] if device.type == "cuda" else None) opt = torch.optim.AdamW(model.parameters(), lr=1e-3) for epoch in range(3): sampler.set_epoch(epoch) # a new shuffle every epoch for xb, yb in loader: xb, yb = xb.to(device), yb.to(device) loss = nn.functional.cross_entropy(model(xb), yb) opt.zero_grad(set_to_none=True) loss.backward() # buckets all-reduce during backward opt.step() if rank == 0: # one process logs and saves print(f"epoch {epoch}: loss {loss.item():.3f} on {world} processes") torch.save(model.module.state_dict(), "ckpt.pt") dist.barrier() # nobody exits before the save dist.destroy_process_group() if __name__ == "__main__": main()
torch.cuda.set_device(local_rank) comes first, so every later CUDA call uses this process's GPU. Without it, all processes pile onto cuda:0.DDP(model, device_ids=[local_rank]) does three things at construction. It broadcasts rank 0's parameters and buffers to every rank, so all replicas start identical even if their random seeds differ. It assigns gradients to buckets. It registers one autograd hook per parameter (Chapter 4).sampler.set_epoch(epoch) changes the shuffle. The sampler seeds its generator with seed + epoch, and every rank must compute the same permutation, or the slices overlap.model.module is the plain model inside the DDP wrapper, so the checkpoint loads without DDP.Chapter 4
DDP starts communicating before backward has finished. See how hooks, buckets and a reversed order hide most of the all-reduce
A plain approach would wait for backward() to finish, then all-reduce every gradient. The network would sit idle during backward, and the GPU would sit idle during the all-reduce. The DDP paper measured that communication was more than half of the backward time on its 32-GPU tests.
Look at the order in which backward produces gradients. It starts at the loss and walks back, so the last layer's gradients are finished first. Those can be on the wire while the GPU is still computing the earlier layers' gradients. That is overlap.
But one all-reduce per parameter tensor would be slow: every call pays the latency α, and a large model has hundreds of small tensors. So DDP packs gradients into buckets. Its design, from the paper (Li et al., 2020):
model.parameters(), as a guess at the order backward will finish them. The default cap is bucket_cap_mb=25 (25 MiB). The first bucket is kept small, 1 MiB in PyTorch 2.x, so the first all-reduce can start early..grad.Worked, illustrative: Tbwd = 200 ms, Tcomm = 150 ms in buckets, last bucket 10 ms. Serial: 350 ms. Overlapped: about 200 + 10 = 210 ms, 40% less. The paper measured 38.0% (ResNet50) and 35.2% (BERT) speed-ups from overlap with NCCL on 32 GPUs.
The top lane is backward, last layer first. The lower lane is the all-reduce of each bucket. Slide the bucket size from tiny to one giant bucket, and switch overlap off to see the plain approach.
An illustrative model of a 100M-parameter network (400 MB of fp32 gradients, 100 equal layers): backward takes 200 ms; an all-reduce takes α plus bytes × 2(N−1)/N / β with N = 8 and β = 4 GB/s. The paper found 10–25 MB best for ResNet50 and 50 MB for BERT with NCCL on 16 GPUs.
Two constructor options change this machinery. find_unused_parameters=True walks the autograd graph after each forward to mark parameters that got no gradient, so their buckets do not wait forever; it costs a graph traversal per step, so leave it off unless you need it. gradient_as_bucket_view=True makes each .grad a view into its bucket, which saves one copy of the gradients in memory.
Chapter 5
Build a bigger batch from K small ones, communicate once instead of K times, and avoid the loss that is K times too big
You want a global batch of 1,024, but only 32 examples fit per GPU, and you have 8 GPUs. Eight GPUs give 256 per step. Gradient accumulation runs K = 4 forward and backward passes before each optimizer step. .grad adds up across them, so the step sees all 1,024 examples.
With DDP there is a cost. Every backward() fires DDP's hooks, and every hook launches all-reduces. Four micro-batches mean four full all-reduces per step. Three of them are wasted: averaging is linear, so you can sum locally and average once.
The context manager model.no_sync() switches DDP's hooks off. Backward passes inside it only accumulate into the local .grad. The first backward outside it syncs everything accumulated so far.
no_sync; M is the gradient size.Worked, a 1B-parameter model with bf16 gradients (M = 2 GB), N = 8, K = 4: syncing every micro-batch sends 4 × 3.5 = 14 GB per GPU per step; with no_sync, 3.5 GB. The DDP paper measured 38% (NCCL) and 57% (gloo) speed-ups for ResNet50 on 256 GPUs when syncing every 8 iterations.
Each pink block is one micro-batch's forward and backward; each purple block is an all-reduce. Switch the two fixes on and off. The bar on the right is the size of the final gradient compared with the true one.
Illustrative times: 60 ms per micro-batch, 35 ms per all-reduce that is not hidden by overlap. Gradient size is relative: 1.0 is the gradient of the mean over all K micro-batches.
The pattern in real code. The function works for any DDP-wrapped model; with real DDP on 2 processes it gives the same weights as one big batch.
Run this inside a DDP script such as the one in Chapter 3
import contextlib
import torch.nn.functional as F
def train_step(model, opt, micro_batches):
"""One optimizer step over K micro-batches. model is wrapped in DDP."""
K = len(micro_batches)
for i, (x, y) in enumerate(micro_batches):
last = i == K - 1
ctx = contextlib.nullcontext() if last else model.no_sync()
with ctx: # no all-reduce inside no_sync()
loss = F.cross_entropy(model(x), y) / K # each micro-batch is 1/K
loss.backward() # .grad accumulates locally
opt.step() # grads were synced on the last one
opt.zero_grad(set_to_none=True)
Chapter 6
Some parts of a model look across the batch or draw random numbers. Those do not average away. Learn which, and the fix for each
BatchNorm normalises each feature with the mean and variance of the current batch. Under DDP, "the current batch" is the GPU's own shard. With 256 examples per GPU that is fine. With 2 per GPU, as in detection and segmentation with large images, each GPU's statistics are mostly noise.
How noisy? The mean of b random values with spread σ has its own spread σ/√b. Pool the statistics over all N GPUs and the spread falls to σ/√(N·b).
Worked: b = 2 on N = 16 GPUs. Alone, each GPU's mean is off by about σ / 1.41 = 0.71σ. Synced over 32 examples: σ / 5.7 = 0.18σ. With b = 1 the variance is zero, and PyTorch refuses: ValueError: Expected more than 1 value per channel when training.
Each pink dot is one GPU's estimate of a feature's mean; the green line is the true mean. Shrink the per-GPU batch and watch the dots scatter. Switch to SyncBatchNorm: every GPU uses one pooled estimate.
Each GPU draws b values from a normal distribution with mean 0 and spread 1, with a fixed seed so the picture is stable. The shaded band is ±1 expected spread.
SyncBatchNorm fixes it. In training mode, each GPU computes its shard's per-channel mean and variance, and an all-gather shares them with the count. Every GPU then combines them into the global statistics and normalises with those. The backward pass syncs the matching gradient terms too. You turn it on with one line, before wrapping in DDP:
# inside the DDP script, after building the model, before DDP(...) model = torch.nn.SyncBatchNorm.convert_sync_batchnorm(model) # every BatchNorm*D -> SyncBatchNorm model = DDP(model.to(device), device_ids=[local_rank])
Combining the statistics is the part to understand. Means combine by weighting with counts. Variances do not simply average: the spread between the GPUs' means is part of the total spread. Combine second moments instead.
Each process has its own random stream. Three rules keep them right:
seed must be the same on every rank, or the slices overlap (lab in Chapter 3).torch.manual_seed(0), all ranks draw the same dropout masks and the same crops. The N shards then share their noise, and you lose part of the benefit of N different views. Seed with base + rank, as the Chapter 3 script does.The same few collectives appear wherever many processes must agree:
Chapter 7
DDP keeps N identical copies of everything. Remove the copies one kind at a time and watch memory per GPU fall
Look again at DDP's memory. Eight GPUs each hold the same weights, the same averaged gradients, and the same Adam state. Seven of the eight copies are redundant. The ZeRO paper (Rajbhandari et al., 2019), short for Zero Redundancy Optimizer, removes them in three stages.
Start from the 16 bytes per parameter of mixed-precision AdamW, derived in Memory and Mixed Precision: 2 bytes of 16-bit weights, 2 bytes of 16-bit gradients, and K = 12 bytes of optimizer state (a 32-bit master copy of the weights, plus Adam's m and v).
Worked, the paper's example (Figure 1): Ψ = 7.5B, Nd = 64. DDP: 16 × 7.5 = 120 GB. ZeRO-1: 30 + 90/64 = 31.4 GB. ZeRO-2: 15 + 105/64 = 16.6 GB. ZeRO-3: 120/64 = 1.9 GB.
Each bar is the model state on one GPU, split into weights (orange), gradients (blue) and optimizer state (yellow). The dashed line is an 80 GB GPU. Find the fewest GPUs that let each stage fit.
The ZeRO paper's formulas for model state, in GB of 109 bytes. Activations, temporary buffers and fragmentation come on top. Communication is in parameters' worth of data per step: 2Ψ for DDP, ZeRO-1 and ZeRO-2, 3Ψ for ZeRO-3 (paper, Section 7).
What does sharding cost in traffic? Section 7 of the paper counts it. DDP's all-reduce is a reduce-scatter plus an all-gather: 2Ψ of data. ZeRO-2 does a reduce-scatter of gradients, updates its shard, and all-gathers the updated weights: still 2Ψ. ZeRO-3 must all-gather the weights twice, once for forward and once for backward, plus the reduce-scatter: 3Ψ, 1.5× DDP.
Chapter 8
Gather a layer's weights just before it runs, free them just after, and scatter its gradients back. Then write it with fully_shard
Think of a library with one copy of every book, spread over N branches. Each branch owns one N-th of the shelves. To read a chapter you borrow the missing pages from the other branches, read, and hand them back. That is Fully Sharded Data Parallel: ZeRO stage 3 as a PyTorch API.
FSDP works on units: groups of layers you choose, usually one transformer block each. For one training step, unit by unit:
Worked, 7B parameters in 32 blocks on 8 GPUs: sharded state 16 × 7 / 8 = 14 GB. One block is 7B / 32 ≈ 0.22B parameters, 0.44 GB in bf16; two in flight add 0.9 GB. Model state about 15 GB per GPU, against 112 GB with DDP.
The curve is the gathered-weight memory on one GPU through forward (left) and backward (right), above the sharded base. Each purple tick is one all-gather or reduce-scatter. Switch resharding off and count the collectives you save.
An illustrative 7B-parameter model with equal blocks, mixed-precision AdamW, no activations drawn. One unit in flight plus one prefetched. Collective counts are per step.
In PyTorch 2.x, FSDP's current API is FSDP2: the function fully_shard, imported from torch.distributed.fsdp. You call it on each block, then on the whole model, bottom-up. Each parameter becomes a DTensor sharded on its first dimension: rank r stores a slice of rows. The script below runs as written, on GPUs with NCCL or on a CPU with gloo:
Run this on a machine with PyTorch and 2+ GPUs: torchrun --nproc_per_node=2 train_fsdp2.py
# train_fsdp2.py -- launch: torchrun --nproc_per_node=2 train_fsdp2.py import os, torch, torch.nn as nn, torch.distributed as dist from torch.distributed.fsdp import fully_shard, MixedPrecisionPolicy from torch.distributed.device_mesh import init_device_mesh from torch.distributed.checkpoint.state_dict import get_model_state_dict, StateDictOptions class Block(nn.Module): def __init__(self, d): super().__init__() self.norm, self.ff = nn.LayerNorm(d), nn.Sequential(nn.Linear(d, 4 * d), nn.GELU(), nn.Linear(4 * d, d)) def forward(self, x): return x + self.ff(self.norm(x)) def main(): local_rank = int(os.environ["LOCAL_RANK"]) cuda = torch.cuda.is_available() if cuda: torch.cuda.set_device(local_rank) dist.init_process_group("nccl" if cuda else "gloo") rank, dev = dist.get_rank(), torch.device("cuda" if cuda else "cpu") mesh = init_device_mesh(dev.type, (dist.get_world_size(),)) # 1-D: all ranks shard model = nn.Sequential(*[Block(256) for _ in range(4)], nn.Linear(256, 10)).to(dev) mp = MixedPrecisionPolicy(param_dtype=torch.bfloat16, reduce_dtype=torch.float32) if cuda else MixedPrecisionPolicy() for block in model[:4]: fully_shard(block, mesh=mesh, mp_policy=mp) # one FSDP unit per block fully_shard(model, mesh=mesh, mp_policy=mp) # the root: whatever is left over w = model[0].ff[0].weight # now a DTensor: this rank's rows only print(f"rank {rank}: {type(w).__name__} local {tuple(w.to_local().shape)} of {tuple(w.shape)}") opt = torch.optim.AdamW(model.parameters(), lr=1e-3) # state is sharded too g = torch.Generator().manual_seed(rank) # each rank its own data for step in range(5): x = torch.randn(32, 256, generator=g).to(dev) y = torch.randint(0, 10, (32,), generator=g).to(dev) loss = nn.functional.cross_entropy(model(x), y) # all-gather per block loss.backward() # reduce-scatter per block opt.step(); opt.zero_grad() full = get_model_state_dict(model, options=StateDictOptions(full_state_dict=True, cpu_offload=True)) if rank == 0: print("loss", round(loss.item(), 3), "| full checkpoint tensors:", len(full)) torch.save(full, "fsdp_full.pt") dist.destroy_process_group() if __name__ == "__main__": main()
init_device_mesh(dev.type, (world,)) describes the GPUs as a 1-D grid that shards everything. A 2-D mesh lets you shard within a machine and replicate across machines (hybrid sharding).fully_shard(block) makes each block one unit: one all-gather and one reduce-scatter per block. Calling it on the root last collects what is left (here the final Linear).MixedPrecisionPolicy(param_dtype=torch.bfloat16, reduce_dtype=torch.float32) gathers and computes in bf16 but reduces gradients in fp32. The sharded master weights stay fp32.fully_shard, so it sees the sharded parameters and its state is sharded too.get_model_state_dict(..., full_state_dict=True, cpu_offload=True) gathers a normal checkpoint for rank 0 to save. For big models, save sharded with torch.distributed.checkpoint instead.Chapter 9
Data parallelism splits the batch. The other two split the model: across the width of each layer, or across its depth
FSDP still runs each layer whole on every GPU, just with borrowed weights. Two other methods split the computation of the model itself. This chapter shows the core of each. The CS336 lessons on data and tensor parallelism and on pipeline parallelism and FSDP, and the distributed training lesson, take them further.
Tensor parallelism splits each weight matrix across GPUs. Megatron-LM (Shoeybi et al., 2019) does it for a transformer's MLP, Y = GeLU(XA), Z = YB. Split A by columns: GPU t computes Yt = GeLU(XAt), and GeLU can run on each slice alone. Split B by rows: GPU t computes YtBt, a partial sum of Z. One all-reduce adds the partial sums. The paper reports one all-reduce in the forward pass and one in the backward pass per MLP, and the same for attention.
Pipeline parallelism splits the layers into p consecutive stages, one per GPU. Only activations cross the boundaries. The problem is waiting: stage 4 has nothing to do until stage 1 has finished. Cutting the batch into m micro-batches lets the stages work on different micro-batches at once. The idle time left over is the bubble.
Worked: p = 4 stages, m = 4 micro-batches: bubble 3/7 = 43%. With m = 16: 3/19 = 16%. GPipe found the bubble "negligible when M ≥ 4 × K", where K is the number of partitions: m ≥ 16 here.
Tensor parallel: slide the number of GPUs and see each one's slice of A and B, and the single all-reduce. Pipeline: slide stages and micro-batches and watch the red bubble shrink.
Tensor view: an MLP with hidden size d = 4096 and 4d inner size, bf16, batch of 8 sequences of 2048 tokens; numbers illustrative. Pipeline view: GPipe schedule, all forwards then all backwards, backward taking twice a forward.
Which to use? Tensor parallelism sends activations at every layer, so it needs the fastest links: GPUs inside one machine. Pipeline parallelism sends little, at stage boundaries only, so it can cross machines; but it needs many micro-batches to hide the bubble. Large runs combine all three: tensor parallel within a machine, pipeline across a few machines, and data parallel (often FSDP) across the rest. PyTorch has both built in: torch.distributed.tensor.parallel (ColwiseParallel, RowwiseParallel) and torch.distributed.pipelining (ScheduleGPipe, Schedule1F1B).
Chapter 10
A distributed bug often shows as nothing at all: a job that stops moving. Learn the usual causes, the checks that find them, and the fixes
Collectives are a meeting. Every rank must arrive, in the same order, with tensors of the same shape. If one rank takes a different path through the code, the others wait at the meeting point. Nothing crashes. GPU use drops to zero, and the job sits there until a timeout ends it.
The default timeout is long. The PyTorch documentation gives 10 minutes for NCCL and 30 minutes for other backends, and you can set it with init_process_group(timeout=...). When it fires, NCCL's watchdog stops the process with a timeout error that names the collective that was waiting.
Worked: 4 ranks, 1,000 steps, one all-reduce per step. If rank 0 alone runs an extra all_reduce for logging on step 100, the four sequences differ at position 101. All four ranks stop at step 100 and wait; with NCCL the job dies 10 minutes later. Worse, a mismatched call can pair with a different call on another rank and mix data silently.
Each row is one rank; each block is one call. Purple blocks are collectives. Pick a bug and watch the ranks run. Red marks the first collective where they no longer match.
A schematic of the call sequences, not a timing model.
The bugs experts look for first:
if rank == 0. Logging, validation or saving that calls all_reduce, barrier or a DDP forward on one rank only. Fix: every rank calls the collective; only rank 0 prints.drop_last, or the model.join() context manager, which shadows the missing collectives.RuntimeError: Expected to have finished reduction in the prior iteration before starting a new one. Fix: find_unused_parameters=True, or make every parameter take part.if rank == 0, followed by dist.barrier() so no rank races ahead and reads a half-written file.base + rank.Two helpers that prevent and detect the worst of these. With real DDP on 2 processes they return 0.5 for ranks 0 and 1, and catch a drift planted on rank 1.
Run this inside a DDP script such as the one in Chapter 3
import torch, torch.distributed as dist
def mean_over_ranks(x: torch.Tensor) -> float:
"""EVERY rank must call this: a collective that one rank skips hangs the rest."""
x = x.detach().clone()
dist.all_reduce(x) # sum over ranks, same call on all of them
return x.item() / dist.get_world_size()
def replicas_in_sync(model, tol=1e-6) -> bool:
"""Compare a fingerprint of the weights across ranks; DDP replicas must match."""
v = torch.nn.utils.parameters_to_vector(model.parameters()).detach().float()
fp = torch.stack([v.sum(), v.square().sum()])
every = [torch.zeros_like(fp) for _ in range(dist.get_world_size())]
dist.all_gather(every, fp) # every rank gets every fingerprint
return all(torch.allclose(e, every[0], atol=tol) for e in every)
# in the training loop:
# val = mean_over_ranks(loss) # all ranks call it ...
# if dist.get_rank() == 0: print(val) # ... only rank 0 prints
# assert replicas_in_sync(model.module) # cheap check every few hundred steps
Chapter 11
Pick a strategy for your model, keep the one-screen cheat sheet, know the limits, and go on to PyTorch's internals
Choosing a strategy comes down to two numbers: does the model state fit on one GPU, and how fast are the links between your GPUs? Set your model and cluster below for a first answer.
Slide the model size and the number of GPUs. The card suggests a starting strategy, the model-state memory per GPU it gives, and why. It is a rule of thumb, not a planner.
Rules of thumb: model state 16 bytes per parameter; keep at most half of 80 GB for it, the rest for activations; a single machine has 8 GPUs with fast links. Real choices also depend on sequence length, batch size and interconnect.
| Part | In one line |
|---|---|
| Data parallel | Same weights, different shards, average gradients: g = (1/N) Σ gr, exact for equal shards and per-example losses. |
| Ring all-reduce | Reduce-scatter + all-gather, 2(N−1) steps; each GPU sends 2(N−1)/N · M; time 2(N−1)α + 2(N−1)M/(Nβ). |
| Launch | torchrun --nproc_per_node=G; RANK, LOCAL_RANK, WORLD_SIZE; set_device(local_rank) then init_process_group. |
| DDP | Broadcast at construction; hooks + 25 MiB buckets in reverse parameter order; async all-reduce overlaps backward. |
| Data | DistributedSampler, same seed on every rank, set_epoch(epoch) every epoch; dropout seeds differ by rank. |
| Accumulation | Loss / K; no_sync() on the first K−1; global batch = N · K · b. |
| BatchNorm | Per-GPU stats when b is small: SyncBatchNorm.convert_sync_batchnorm, or LayerNorm / GroupNorm. |
| ZeRO | 16Ψ → 4Ψ + 12Ψ/N → 2Ψ + 14Ψ/N → 16Ψ/N; traffic 2Ψ, 2Ψ, 2Ψ, 3Ψ. |
| FSDP2 | fully_shard each block, then the root; all-gather before use, reshard after, reduce-scatter grads; DTensor dim-0 shards. |
| TP / PP | Column then row split, one all-reduce per MLP; pipeline bubble (p−1)/(m+p−1). |
| Hangs | Same collectives, same order, same shapes on every rank; TORCH_DISTRIBUTED_DEBUG=DETAIL. |
| Method | Use it when | Avoid it when | Use instead |
|---|---|---|---|
| DDP | Model state plus activations fit on one GPU; you want speed. | 16Ψ is over about half the GPU. | ZeRO-1 or FSDP. |
| ZeRO-1 / -2 | The weights fit but Adam's state does not. | The 16-bit weights alone do not fit. | FSDP (ZeRO-3). |
| FSDP | The model does not fit; one layer does. | Links are slow and the model fits: you pay 1.5× traffic for nothing. | DDP. |
| Tensor parallel | One layer is too big or too slow; GPUs share fast links. | Across machines. | Pipeline or FSDP across machines. |
| Pipeline | Very deep models across machines, with many micro-batches. | Few micro-batches: the bubble eats the gain. | FSDP. |
torch.distributed. The algorithms and numbers are the real ones; the two scripts marked "Run this" use the real APIs, checked on PyTorch 2.10.Now press Present or Teach and explain, out loud and from memory, why averaging gradients over shards gives the big-batch gradient, and why a ring sends 2(N−1)/N of the data. Then go back to the hero, switch to FSDP, and find the GPU count where a 13B model's state drops under 40 GB per GPU.