Every PyTorch model learns inside the same dozen lines. Each line has one job and one place where it must sit. Drop or move a single line and training breaks, often without an error.
Learn why each line exists: the optimizer maths, schedules and warmup, clipping, accumulation, honest evaluation, checkpoints that resume exactly, and the logs that prove a run is healthy. Then write it yourself in seven Code Labs that run real PyTorch code in your browser.
Switch a line off and watch the run respond. The network trains in your browser, right now. Start with opt.zero_grad(). Then raise the learning rate and switch off clipping.
You need nn.Module and DataLoader (the two lessons before this one) and the idea of a gradient. We build the rest from zero.
Where sched.step() runs:
A 2→32→32→2 network with Dropout 0.3, trained by SGD with momentum 0.9 on 120 two-moon points plus 3 corrupted rows far from the data. Batch 16, 40 epochs (320 steps), 30 warmup steps then cosine decay. Validation: 400 fresh points. The faint dashed curve is the correct loop at the same learning rate. Everything is computed live; the numbers are illustrative.
Chapter 0
Nothing crashes, the loss goes down, and the model is still wrong. Start with one weight and watch what each line changes in memory
You copy a training loop from a blog post and run it. The loss goes down, so you ship the model. In use, it does worse than your validation number promised. A week later you resume a run from a checkpoint, and the loss jumps back up.
Nothing crashed. Each of these failures comes from one line that is missing or in the wrong place. PyTorch warns about almost none of them. The loop is yours to get right, and nobody checks it for you.
So we start as small as possible: one weight, one example, and the real loop. A training step is one pass through the inner loop: clear old gradients, compute the loss, compute new gradients, and move the weights. Run it one line at a time and watch the memory.
Press Run next line. The highlighted line runs, and the box it changes lights up. After two full steps, switch off zero_grad, reset, and compare the gradient on step 2.
Model: pred = w · x with x = 2, target y = 3, start w = 0.5. Optimizer: SGD, lr 0.05, momentum 0.9. Scheduler: ExponentialLR with gamma 0.9 (the learning rate shrinks by 10% per step). These are exact numbers for this toy.
The loss is the squared error. Its gradient tells the weight which way to move, and how hard:
nn.Parameter: a tensor that wants a gradient.loss.backward() computes it and stores it in w.grad.Step 1: pred = 0.5 × 2 = 1, L = (1 − 3)2 = 4, grad = 2 × 2 × (−2) = −8. The momentum buffer starts as the gradient, −8, so w = 0.5 − 0.05 × (−8) = 0.9.
Five kinds of state live through the loop, and each line touches only some of them. The parameters are the weights. The gradients sit in each parameter's .grad. The optimizer state holds extra numbers per weight, such as a momentum buffer. The schedule state is a step counter that sets the learning rate. The autograd graph is the record of the forward pass that backward() walks. It is built by the forward pass and freed by backward().
Each chapter takes one line, or one habit, of a production loop. The tags say which question it answers: why it works, how to build it, or how to tell that it worked.
This is part 5 of an eleven-part series inside The Builder's Toolkit.
loss.backward(), and opt.step() has not run yet. What has changed?Chapter 1 How
The canonical loop, what each line changes, what breaks without it, and the one rule that decides where zero_grad can go
Here is the loop that almost every PyTorch project runs. Read it once now. The rest of the lesson explains each line.
for epoch in range(num_epochs):
model.train() # Dropout on, BatchNorm uses batch statistics
for x, y in train_loader: # x: (B, ...), y: (B,)
opt.zero_grad(set_to_none=True) # forget last step's gradients
logits = model(x) # forward: builds the autograd graph
loss = loss_fn(logits, y) # one number, the mean over the batch
loss.backward() # adds dloss/dp into p.grad, frees the graph
torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0) # cap the step
opt.step() # move the weights using p.grad
sched.step() # next learning rate (a per-step schedule)
model.eval() # Dropout off, BatchNorm uses running stats
with torch.no_grad(): # build no graph while we measure
val_loss, val_acc = evaluate(model, val_loader)
| Line | What it changes | If it is missing |
|---|---|---|
model.train() | The training flag on every submodule | After the first eval(), Dropout stays off for the rest of training |
zero_grad() | Every p.grad, to None | Gradients add up across steps; the run swings or blows up |
model(x), loss_fn | Activations and the autograd graph | Nothing to learn from |
backward() | Adds into every p.grad; frees the graph | step() sees no gradient; no weight moves |
clip_grad_norm_ | Scales all p.grad down if their norm is too big | One bad batch can throw the weights far away |
opt.step() | The weights, and the optimizer state | The loss is computed and nothing learns |
sched.step() | The learning rate in opt.param_groups | The learning rate never warms up or decays |
eval() + no_grad() | Layer behaviour; no graph is kept | Noisy, biased metrics; wasted memory |
The surprise is backward(). It does not overwrite .grad. It accumulates: it adds the new gradient to whatever is already there.
None; the first backward() creates it.backward() adds it, it never replaces.From Chapter 0: step 1 leaves w.grad = −8. On step 2 the true gradient is −4.8. Without zero_grad, w.grad becomes −8 + (−4.8) = −12.8, so step 2 pushes 2.7 times too hard.
Adding is a feature, not a bug. It lets two losses share one backward pass, lets a weight used twice collect both gradients, and lets you build a big batch from small ones (Chapter 7). The price is that you must clear the slate yourself. The rule is short: gradients must be cleared after the last step() and before the next backward(). Anywhere in that window works. Try the four places below.
Same one-weight problem, plain SGD, learning rate 0.05, 30 steps. The target weight is 1.5. Pick where zero_grad() sits.
Exact numbers for pred = w · x, x = 2, y = 3, w starts at 0.5, SGD without momentum.
set_to_none. In PyTorch 2.x, zero_grad() sets each .grad to None by default instead of filling it with zeros. It frees the memory and skips a write. It also changes what the optimizer does: a parameter whose gradient is None is skipped entirely on the next step().
loss.backward(), then opt.zero_grad(), then opt.step(). What happens to plain SGD?Chapter 2 Did it work
What model.eval() switches, what no_grad and inference_mode save, and why the mean of batch means is the wrong number
A mode is a flag that some layers read. model.train() and model.eval() walk every submodule and set its training attribute to True or False. Most layers ignore the flag. Two common ones do not.
Forget model.eval() and both go wrong. Dropout makes every metric noisy: evaluate the same model twice and you get two answers. BatchNorm is worse. It keeps updating its running mean from your validation batches. With PyTorch's default momentum of 0.1, one validation batch with mean 5 moves a running mean of 0 to 0.5. Your validation data has now leaked into the model.
Gradients are a separate switch. model.eval() does not stop the graph from being built. torch.no_grad() does: inside it, results do not require gradients, so no activations are kept for backward(). torch.inference_mode() goes further. It also skips the bookkeeping that tracks in-place changes, so it is a little faster. The price: a tensor made inside it can never join a graph later. PyTorch then raises "Inference tensors cannot be saved for backward."
Now the metric itself. A validation loader gives batches. The easy code averages the accuracy of each batch, then averages those averages. That is the mean of batch means, and it is wrong whenever the batches differ in size. The last batch is usually short, so its few examples get the same vote as a full batch.
reduction="sum").Worked: three batches of 50 with 45 correct (90%) and a last batch of 10 with 5 correct (50%). Mean of means: (0.9 + 0.9 + 0.9 + 0.5) / 4 = 80.0%. True: (135 + 5) / 160 = 87.5%.
1,000 validation predictions. Each bar is a batch: its width is its size, its height its accuracy. Slide the batch size. Then sort the set so the hard examples come last, as an unshuffled loader often does.
A fixed set of 1,000 right-or-wrong predictions, 85.6% correct overall, with errors more common among hard examples. Illustrative.
The fix is to accumulate sums. Ask the loss for reduction="sum", count correct answers, add up the batch sizes, and divide once at the end. Put the model back in the mode you found it in, so the training loop is not surprised.
@torch.no_grad() # no graph inside this function def evaluate(model, loader): was_training = model.training model.eval() # Dropout off, BatchNorm frozen loss_sum, correct, n = 0.0, 0, 0 for x, y in loader: logits = model(x) # (B, C) loss_sum += F.cross_entropy(logits, y, reduction="sum").item() correct += (logits.argmax(dim=1) == y).sum().item() n += y.numel() model.train(was_training) # restore the caller's mode return loss_sum / n, correct / n
Chapter 3 Why
SGD, momentum and Adam as three update rules, raced on a narrow valley, with the numbers PyTorch really uses
Picture the loss as a landscape and the weights as a ball on it. The gradient points uphill. Every optimizer moves the ball downhill, but they differ in how they use the slope. Three rules cover most real training.
SGD (stochastic gradient descent) takes a step straight down the slope it sees now. Momentum keeps a running sum of past gradients, like a heavy ball that keeps rolling. Adam keeps two running averages per weight: one of the gradient, and one of the gradient squared. It divides the first by the root of the second, so every weight takes steps of a similar size.
p.grad).Adam's first step, t = 1: m = 0.1g, v = 0.001g2, so m̂ = g and v̂ = g2. The step is η · g/|g| = η, whatever the size of g. On PyTorch 2.10, gradients of 1,000 and of 0.001 both moved their weight by 0.001.
That last line is the key to Adam. Its step size is set by the learning rate, not by the size of the gradient. A weight whose gradient is always tiny still moves. A weight whose gradient is huge does not jump. That is why Adam is forgiving across layers of very different scale, and why its learning rate means something different from SGD's.
The valley is 20 times steeper across than along. Press Race. Then scale every learning rate up and watch SGD zigzag across the valley while Adam keeps its step.
Loss 0.5(0.06x2 + 1.2y2), start (−8, 2.2), 80 steps. At 1× the learning rates are SGD 0.8, momentum (0.9) 0.15, Adam 0.25. Exact maths; the valley is illustrative.
Read the trace. SGD crawls along the valley floor, because the slope there is gentle. Raise its learning rate to go faster and it starts to bounce from wall to wall: the steep direction limits how big a step SGD can take. Momentum builds speed along the floor, where gradients agree from step to step, so it arrives first. It is a heavy ball, though: it overshoots and swings across the valley before it settles. Adam rescales each direction by its own gradient size, so it heads down both at a similar pace and curves in smoothly.
Chapter 4 Why How
Why weight decay inside Adam's gradient goes weak exactly where you need it, how AdamW fixes it, and which weights should not decay at all
Weight decay pulls every weight a little toward zero on each step. It keeps weights small, which usually helps the model generalise. With plain SGD there are two ways to write it, and they are the same thing.
The first way adds a penalty ½λ‖p‖2 to the loss. This is L2 regularisation. Its gradient is λp, so the step becomes p −= η(g + λp). The second way shrinks the weight directly: p ← (1 − ηλ)p, then the usual step. Expand both and they match.
Adam breaks the match. If you add λp to the gradient, Adam divides it by √v̂ along with everything else. A weight with large, noisy gradients has a large √v̂, so its decay is divided by a large number and nearly vanishes. Loshchilov and Hutter put it this way: with Adam, L2 regularisation "leads to weights with large historic parameter and/or gradient amplitudes being regularized less than they would be when using weight decay" AdamW · §1. Their fix, decoupled weight decay, shrinks the weight outside Adam's division. That is AdamW.
torch.optim.Adam(weight_decay=λ) does. The decay passes through m and v, so it is divided by √v̂.torch.optim.AdamW does. In PyTorch 2.10 it is Adam with decoupled_weight_decay=True; the default λ is 0.01.Worked: η = 10−3, λ = 0.01 → each step multiplies the weight by 1 − 10−5. After 10,000 steps: e−0.1 = 0.905, with no gradient at all. After 100,000 steps: e−1 = 0.368.
Two weights start at 1. Their gradients are pure noise: one large, one small. Only the decay pulls them toward zero. Switch between the two optimizers and change the decay.
Learning rate 0.01, Adam defaults otherwise. Gradient noise has standard deviation 10 for the "large" weight and 0.1 for the "small" one. The dashed curve is pure decay, (1 − ηλ)t. Each curve averages 40 runs. Illustrative.
Weight decay is meant for the big matrices that do the mixing: linear layers, convolutions, attention projections. Two kinds of parameter should usually be spared.
Both kinds are 1-D tensors, while the matrices are 2-D or more. So the usual split is one line: decay a parameter if p.ndim >= 2. Embedding tables are 2-D and are decayed by this rule; some recipes exclude them too.
decay, no_decay = [], []
for name, p in model.named_parameters():
if not p.requires_grad:
continue # frozen weights stay out of the optimizer
(decay if p.ndim >= 2 else no_decay).append(p) # matrices decay; biases, norm gains do not
opt = torch.optim.AdamW([
{"params": decay, "weight_decay": 0.1},
{"params": no_decay, "weight_decay": 0.0},
], lr=3e-4, betas=(0.9, 0.95)) # a common LLM setting, not the default
torch.optim.Adam(..., weight_decay=0.1), which weights get the least decay?Chapter 5 Why How
Warmup then cosine, OneCycle, the step-counting bug, and why Adam's first steps are the dangerous ones
A learning-rate schedule changes η as training goes on. Early in training the weights are far from good, and big steps make fast progress. Late in training the weights sit near a minimum, and big steps bounce around it. So most schedules end low. Many also start low, for a reason we will measure below.
The most common modern schedule is linear warmup then cosine decay. The rate climbs in a straight line from zero to its peak over W steps, then follows half a cosine down to a floor.
Worked: ηmax = 3×10−4, W = 1,000, T = 10,000, ηmin = 0. Step 500: 1.5×10−4. Step 5,500: s = 0.5, cos(π/2) = 0, so 1.5×10−4. Step 10,000: 0.
OneCycleLR is a close cousin, from Smith's work on "super-convergence". PyTorch's defaults: start at max/25, climb along a cosine to the max over the first 30% of steps, then fall to max/25/104. It also moves momentum the other way, from 0.95 down to 0.85 at the peak and back. The idea behind it: "large learning rates regularize the training" Smith & Topin · Abstract, and a long low tail then settles the weights.
A run of 10 epochs, 100 optimizer steps each. Pick a shape and set the warmup and floor. Then pick a wrong way to call sched.step() and see the schedule your optimizer really gets.
The shapes follow PyTorch's LambdaLR, MultiStepLR (gamma 0.1 at 50% and 75%) and OneCycleLR defaults. Exact maths.
The stepping bugs are the common way schedules go wrong, because PyTorch cannot know what unit you meant. A scheduler counts calls to step(), nothing else. If you build it with T = 1,000 steps and call it once per epoch, it sees 10 calls and never leaves warmup. If you build it for 10 epochs and call it per batch, it finishes in the first 10 steps and sits at the floor for the rest. Also call sched.step() after opt.step(). If you call it first, PyTorch warns "Detected call of `lr_scheduler.step()` before `optimizer.step()`" and your run skips the first value of the schedule.
Warmup earned its place in large-batch SGD. Goyal and colleagues trained ResNet-50 on ImageNet with batches of 8,192 by scaling the learning rate with the batch and ramping it up over the first 5 epochs Goyal 2017 · §2.2. Without warmup, top-1 error was 24.84% ± 0.37; with gradual warmup it was 23.74% ± 0.09, close to the small-batch baseline of 23.60% ± 0.12 Table 1.
For Adam there is a second reason, and you can see it in Adam's own maths. Recall Chapter 3: on step 1, every weight moves by exactly η, whatever its gradient. If that gradient is mostly noise, the weight still takes a full step in a random direction. Only later, when m and v have averaged many gradients, does the step shrink to match how much signal there really is. Liu and colleagues traced warmup's benefit to this: the adaptive learning rate "has undesirably large variance in the early stage" RAdam · §1. They report that removing warmup from a Transformer translation model raised the training loss from about 3 to around 10.
Each weight's gradient is a small true signal plus noise. The curve is the size of Adam's step, in units of η. Lower the signal and look at the first steps. Then switch on warmup.
Gradient g = μ + σξ with ξ standard normal, σ = 1, μ set by the slider. Adam defaults (β1 0.9, β2 0.999). The curve averages 400 independent weights; the band spans the 10th to 90th percentile. Exact simulation.
LambdaLR with 500 warmup steps out of 5,000, but call sched.step() once per epoch, and an epoch is 100 steps. What does training see?Chapter 6 How
One steep cliff can throw the weights across the landscape. Clipping by norm caps the step and keeps its direction
Most of a loss landscape is gentle. Some places are not: a sharp wall where the loss changes a lot over a tiny distance. Pascanu, Mikolov and Bengio described such cliffs in recurrent networks. Near one, the gradient is huge, and a normal step throws the weights far away, undoing many steps of progress.
Gradient clipping by norm puts a ceiling on the gradient's length. Treat all the gradients of the model as one long vector. If its length is above a threshold c, scale the whole vector down to length c. The direction stays the same; only the size is capped.
.grad). clip_grad_norm_ returns this number, measured before clipping.max_norm. 1.0 is a common default for transformers. The 10−6 avoids dividing by zero; it is in PyTorch's code.Worked: g = (3, 4), ‖g‖ = 5, c = 1 → g becomes (0.6, 0.8): same direction, length 1. Clipping each value to [−1, 1] instead would give (1, 1): length 1.41, and a new direction.
The ball starts on a high plateau. Between it and the minimum is a steep drop. Run it without clipping, then with clipping, then raise the learning rate.
Loss 0.025x2 + 0.3y2 plus a smooth wall of height 6 across the path. Plain gradient descent, 80 steps. Exact maths; the landscape is illustrative.
Where it goes. After backward(), because the gradients must exist. Before opt.step(), because the step must use the clipped values. With mixed precision and a GradScaler, the gradients are still multiplied by the loss scale at that point. Call scaler.unscale_(opt) first, or you compare a scaled norm with an unscaled threshold.
scaler.scale(loss).backward() scaler.unscale_(opt) # .grad now holds true values gnorm = torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0) scaler.step(opt) # skips the step if any grad is inf/nan scaler.update() log["grad_norm"] = gnorm.item() # the PRE-clip norm: log it
Choosing c. Pascanu and colleagues suggest looking "at statistics on the average norm over a sufficiently large number of updates", and found training "not very sensitive" to the exact value Pascanu 2013 · §3.2. In practice: log the norm for a while, then set c a little above its typical value, so clipping only catches the spikes.
clip_grad_norm_(params, max_norm=5). What are the gradients after, and what does the call return?Chapter 7 How
Build the gradient of a batch that does not fit in memory from several small ones, and divide the loss the right way
Say the recipe calls for batches of 256, but only 64 examples fit in memory at once. Gradient accumulation runs 4 micro-batches of 64, calls backward() on each, and lets the gradients add up in .grad, the feature from Chapter 1. Then it takes one optimizer step and clears the gradients.
The catch is the scale. A loss with the default reduction="mean" is the mean over its own micro-batch. Four backward calls add four means, so the accumulated gradient is 4 times the gradient of the big batch. Divide each micro-batch loss by K, the number of micro-batches.
backward() adds for it.Worked: K = 4 equal micro-batches with mean gradients 2, 4, 6, 8. Without dividing, .grad = 20. Dividing each loss by 4: 0.5 + 1 + 1.5 + 2 = 5, the big-batch mean.
Each arrow is one micro-batch's gradient, added tip to tail into .grad. The dashed arrow is the big-batch gradient. Change K. Switch off the division. Then make the last micro-batch short.
A 2-D slice of a gradient for 64 examples, so the arrows can be drawn. Per-example gradients are fixed random vectors. "Short" makes the last micro-batch hold 4 examples. Exact sums.
K = 4 # micro-batches per optimizer step opt.zero_grad(set_to_none=True) for i, (x, y) in enumerate(train_loader): # each x is a micro-batch loss = loss_fn(model(x), y) / K # scale BEFORE backward loss.backward() # adds into .grad if (i + 1) % K == 0 or i + 1 == len(train_loader): torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0) # on the FULL gradient opt.step() sched.step() # one tick per optimizer step opt.zero_grad(set_to_none=True)
Three details hide in those lines. Clip only after the last micro-batch, because clipping a partial sum caps the wrong vector. Step the scheduler once per optimizer step, not per micro-batch. And if the last group of an epoch is short, dividing by K under-weights it. When micro-batches differ in size, or a language model's micro-batches hold different numbers of real tokens, use reduction="sum" and divide by the total count for the whole step.
Chapter 8 How Did it work
Average the weights over recent steps to smooth out the noise, and stop when validation stops improving, keeping the best copy
With a noisy gradient, the weights never sit still. They jitter around a good region, and each step lands somewhere a little different. Validation then jumps from epoch to epoch. One cure is to evaluate not the latest weights but a running average of recent ones.
An exponential moving average (EMA) of the weights is a second copy of the model. After each optimizer step, it moves a small fraction of the way toward the live weights. Old steps fade out smoothly. It costs one extra copy of the weights and no extra gradients.
opt.step().Worked: β = 0.999 → a window of about 1,000 steps. A step 1,000 steps back still has weight 0.9991000 = 0.368 of the newest one. With β = 0.99 the window is 100 steps.
Early stopping watches the validation loss after each epoch. When it has not improved for a number of epochs called the patience, training stops, and you keep the weights from the best epoch, not the last one. It guards against overfitting: the model keeps fitting the training set while getting worse on new data.
A large network on 60 noisy points. It fits the training set (warm) while validation (green) gets worse. Turn on an EMA and pick its decay. Set the patience and see where training stops and which epoch is kept.
A 2→48→48→2 network, no Dropout, SGD with momentum 0.9 at a constant learning rate of 0.1, batch 10, 100 epochs on 60 two-moon points with noise 0.3. Validation: 500 fresh points. Trained live in your browser; the numbers are illustrative.
from torch.optim.swa_utils import AveragedModel, get_ema_multi_avg_fn
ema = AveragedModel(model, multi_avg_fn=get_ema_multi_avg_fn(0.999),
use_buffers=True) # also track BatchNorm running stats
# in the loop, right after opt.step():
ema.update_parameters(model)
# at evaluation time:
val_loss, val_acc = evaluate(ema.module, val_loader)
best, bad, patience = float("inf"), 0, 5
# after each epoch:
if val_loss < best - 1e-4: # min_delta: ignore tiny wiggles
best, bad = val_loss, 0
torch.save(ema.module.state_dict(), "best.pt") # a copy on disk, frozen in time
else:
bad += 1
if bad >= patience:
break # then load best.pt for testing
When EMA helps. It helps most when the learning rate stays high and the gradients are noisy: long runs, small batches, and diffusion models, where an EMA of 0.9999 is standard. With a schedule that decays to zero, the live weights settle by themselves, and the EMA adds less. Match the window to the run: if β = 0.999 but the run has only 300 steps, the EMA still holds mostly the random starting weights. In a check run for this lesson, a 280-step run scored 65% with β = 0.999 and 96% with β = 0.99.
Chapter 9 How
What a checkpoint must hold so that a restart is invisible, and the knobs that make a run repeat
Long runs stop: a machine is reclaimed, a job hits its time limit, a node fails. A checkpoint is a file that lets the run carry on. The test of a good one is strict: the resumed run should produce the same losses as a run that never stopped.
The weights are not enough. Everything the next step depends on must be saved:
model.state_dict()): parameters and buffers, such as BatchNorm statistics.opt.state_dict()): Adam's m, v and step count for every weight, plus each group's current learning rate.sched.state_dict()): how many steps it has counted.Worked: P = 1 billion → 4 GB of weights + 8 GB of Adam state = 12 GB per checkpoint. Two thirds of it is the part people forget to save.
An AdamW run with warmup and cosine decay. The run stops at the teal line and resumes. Leave pieces out of the checkpoint and look for the seam in the loss, the learning rate and Adam's step size.
A network like the hero's (2→32→32→2, Dropout 0.3) on 128 two-moon points, AdamW (lr 0.01, weight decay 0.01 on matrices only), 40 epochs of 8 steps, 24 warmup steps then cosine. The model weights are always restored. Computed live; illustrative.
Each missing piece leaves its own mark. Without the optimizer state, Adam restarts with empty moments: its first steps move every weight by the full learning rate (Chapter 3), several times its settled step size. In this small model the loss only bumps; in large models such a jolt can cause a loss spike. Without the scheduler state, a fresh scheduler starts its warmup again, so the learning rate drops to zero and climbs back to the peak just when it should be low. Without the random states, the data comes in a new order and Dropout draws new masks. The curve looks healthy, but it is a different run, and you can no longer compare it with the old one.
import os, random
import numpy as np, torch
def save_checkpoint(path, model, opt, sched, scaler, loader_gen, epoch, step):
state = {"model": model.state_dict(), "opt": opt.state_dict(),
"sched": sched.state_dict(), "scaler": scaler.state_dict(),
"epoch": epoch, "step": step,
"rng": {"torch": torch.get_rng_state(), "loader": loader_gen.get_state(),
"python": random.getstate(), "numpy": np.random.get_state(),
"cuda": torch.cuda.get_rng_state_all() if torch.cuda.is_available() else []}}
torch.save(state, path + ".tmp")
os.replace(path + ".tmp", path) # atomic: a crash never leaves half a file
def load_checkpoint(path, model, opt, sched, scaler, loader_gen):
ck = torch.load(path, map_location="cpu", weights_only=False) # our own file only
model.load_state_dict(ck["model"]) # model already on its device
opt.load_state_dict(ck["opt"]) # moves Adam's state to the params' device
sched.load_state_dict(ck["sched"]); scaler.load_state_dict(ck["scaler"])
r = ck["rng"]
torch.set_rng_state(r["torch"]); loader_gen.set_state(r["loader"])
random.setstate(r["python"]); np.random.set_state(r["numpy"])
if r["cuda"]: torch.cuda.set_rng_state_all(r["cuda"])
return ck["epoch"] + 1, ck["step"] # resume at the NEXT epoch
Exact resume and exact repeat use the same ideas. To make a run repeat bit for bit, control every source of randomness and every source of nondeterminism.
| Knob | What it fixes | Cost |
|---|---|---|
torch.manual_seed, random.seed, np.random.seed | Initial weights, Dropout, augmentations. torch.manual_seed seeds every device. | None |
DataLoader(generator=g) + worker_init_fn | The shuffle order, and random numbers inside worker processes | None |
torch.use_deterministic_algorithms(True) | Errors out on kernels that are not deterministic, such as some scatter-adds on GPU | Some ops slower; some raise |
torch.backends.cudnn.benchmark = False | cuDNN picks the same convolution algorithm each run | Possibly slower convolutions |
CUBLAS_WORKSPACE_CONFIG=:4096:8 | Deterministic cuBLAS matrix multiplies, needed with the deterministic flag on CUDA | A little memory |
| Same hardware, versions, batch size and GPU count | Floating-point sums happen in the same order | Flexibility |
The last row is the one people forget. Floating-point addition is not associative: (a + b) + c can differ from a + (b + c) in the last bits. Change the GPU, the library version or the number of processes, and sums happen in a different order. Tiny differences then grow over thousands of steps. Bit-for-bit repeats are possible only on a fixed setup.
Chapter 10 Did it work How
Four numbers that diagnose most broken runs, a production-shaped loop that puts every chapter together, and when to use a library instead
A loss curve alone hides most problems. Log four numbers every few steps and most broken runs give themselves away.
sched.get_last_lr(). It catches every scheduler bug from Chapter 5 at a glance.clip_grad_norm_ returns. Spikes come before loss blow-ups; a slow rise warns of trouble.Add validation metrics at each evaluation, and peak memory (torch.cuda.max_memory_allocated()) on a GPU. Then learn the shapes of trouble below.
Each run has one problem. Pick a run and find the panel that shows it. The readout names the cause and the fix.
Synthetic logs, drawn to show each failure's typical signature. Illustrative.
Worked: B = 256, tcompute = 60 ms. With tdata = 40 ms: 256 / 0.1 s = 2,560 examples/s, busy 60%. With tdata = 150 ms: 256 / 0.21 s = 1,219 examples/s, busy 29%. More DataLoader workers buy that back.
Here is every chapter in one loop, in three parts. It runs as written on a CPU or a GPU, on stand-in data. Swap in your own dataset and model. The setup builds the data, the parameter groups, the schedule, the scaler and the EMA.
import math, os, random, time
import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.utils.data import DataLoader, TensorDataset
from torch.optim.swa_utils import AveragedModel, get_ema_multi_avg_fn
def seed_everything(seed):
random.seed(seed); np.random.seed(seed); torch.manual_seed(seed) # also seeds every GPU
device = "cuda" if torch.cuda.is_available() else "cpu"
seed_everything(0)
X = torch.randn(4096, 20); y = X[:, :3].argmax(dim=1) # stand-in data with a pattern
train_ds, val_ds = TensorDataset(X[:3584], y[:3584]), TensorDataset(X[3584:], y[3584:])
g = torch.Generator().manual_seed(0) # owns the shuffle order
train_loader = DataLoader(train_ds, batch_size=64, shuffle=True, generator=g, drop_last=True)
val_loader = DataLoader(val_ds, batch_size=256)
model = nn.Sequential(nn.Linear(20, 128), nn.GELU(), nn.LayerNorm(128), nn.Dropout(0.1),
nn.Linear(128, 3)).to(device)
decay = [p for p in model.parameters() if p.ndim >= 2]
no_decay = [p for p in model.parameters() if p.ndim < 2]
opt = torch.optim.AdamW([{"params": decay, "weight_decay": 0.05},
{"params": no_decay, "weight_decay": 0.0}], lr=1e-3)
EPOCHS, ACCUM = 10, 2 # 2 micro-batches per optimizer step
total = EPOCHS * len(train_loader) // ACCUM
warm = int(0.05 * total)
def lr_lambda(s):
if s < warm:
return s / max(1, warm)
return 0.5 * (1 + math.cos(math.pi * (s - warm) / max(1, total - warm)))
sched = torch.optim.lr_scheduler.LambdaLR(opt, lr_lambda)
scaler = torch.amp.GradScaler(device, enabled=(device == "cuda"))
ema = AveragedModel(model, multi_avg_fn=get_ema_multi_avg_fn(0.99), use_buffers=True)
Evaluation and checkpoints, from Chapters 2 and 9:
@torch.no_grad()
def evaluate(m, loader):
was = m.training; m.eval()
loss_sum, correct, n = 0.0, 0, 0
for xb, yb in loader:
xb, yb = xb.to(device), yb.to(device)
logits = m(xb)
loss_sum += F.cross_entropy(logits, yb, reduction="sum").item()
correct += (logits.argmax(1) == yb).sum().item(); n += yb.numel()
m.train(was)
return loss_sum / n, correct / n
def save_checkpoint(path, epoch, step, best):
state = {"model": model.state_dict(), "ema": ema.state_dict(), "opt": opt.state_dict(),
"sched": sched.state_dict(), "scaler": scaler.state_dict(), "epoch": epoch, "step": step, "best": best,
"rng": {"torch": torch.get_rng_state(), "loader": g.get_state(),
"python": random.getstate(), "numpy": np.random.get_state(),
"cuda": torch.cuda.get_rng_state_all() if device == "cuda" else []}}
torch.save(state, path + ".tmp")
os.replace(path + ".tmp", path) # atomic: never half a checkpoint
def load_checkpoint(path):
ck = torch.load(path, map_location=device, weights_only=False) # our own file: trusted
for obj, key in [(model, "model"), (ema, "ema"), (opt, "opt"), (sched, "sched"), (scaler, "scaler")]:
obj.load_state_dict(ck[key])
r = ck["rng"]
torch.set_rng_state(r["torch"]); g.set_state(r["loader"])
random.setstate(r["python"]); np.random.set_state(r["numpy"])
if device == "cuda": torch.cuda.set_rng_state_all(r["cuda"])
return ck["epoch"] + 1, ck["step"], ck["best"]
The loop: accumulation, mixed precision, clipping, scheduling, EMA, logging, checkpoints and early stopping. Each line sits where the earlier chapters said it must.
start, best, patience, bad, step = 0, float("inf"), 3, 0, 0
if os.path.exists("last.pt"):
start, step, best = load_checkpoint("last.pt")
for epoch in range(start, EPOCHS):
model.train(); t0 = time.perf_counter(); seen = 0
opt.zero_grad(set_to_none=True)
for i, (xb, yb) in enumerate(train_loader):
xb, yb = xb.to(device, non_blocking=True), yb.to(device, non_blocking=True)
with torch.autocast(device_type=device, dtype=torch.float16, enabled=(device == "cuda")):
loss = F.cross_entropy(model(xb), yb) / ACCUM
scaler.scale(loss).backward()
seen += yb.numel()
if (i + 1) % ACCUM == 0:
scaler.unscale_(opt)
gnorm = torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
scaler.step(opt); scaler.update()
sched.step(); ema.update_parameters(model)
opt.zero_grad(set_to_none=True); step += 1
if step % 20 == 0:
print(f"step {step} loss {loss.item() * ACCUM:.3f} lr {sched.get_last_lr()[0]:.2e} "
f"gnorm {gnorm.item():.2f} {seen / (time.perf_counter() - t0):.0f} ex/s")
val_loss, val_acc = evaluate(ema.module, val_loader)
print(f"epoch {epoch} val loss {val_loss:.4f} acc {val_acc:.3f}")
save_checkpoint("last.pt", epoch, step, min(best, val_loss))
if val_loss < best - 1e-4:
best, bad = val_loss, 0
torch.save(ema.module.state_dict(), "best.pt")
else:
bad += 1
if bad >= patience:
print("early stop"); break
Run this on a machine with PyTorch. On a CPU, autocast and the GradScaler switch themselves off. Tested on PyTorch 2.10.
| Field | What changes in the loop |
|---|---|
| Image classification | Random augmentation on the CPU workers; BatchNorm, so mind eval() |
| Language-model pretraining | Count steps in tokens; divide by real tokens, not micro-batches; AdamW with β2 = 0.95 and clip 1.0 are common |
| Fine-tuning a pretrained backbone | Parameter groups with a smaller backbone learning rate; short warmup; frozen parts left out of the optimizer |
| Audio and sensor time series | Variable-length batches, so masked losses summed over real frames; heavy class imbalance in the metric |
| Diffusion and generative models | A random noise level per example in the loss; EMA weights with β near 0.9999 are the ones you sample from |
What stays the same in every row: zero_grad, forward, loss, backward, clip, step and schedule; evaluation under eval() and no_grad() with summed metrics; checkpoints that hold every piece of state.
| Use a hand-written loop when | Use a library instead when | Library |
|---|---|---|
| You are learning, debugging, or the step is unusual (GANs, RL, several optimizers) | You want standard multi-GPU, mixed precision and checkpoints with little code | Hugging Face Accelerate: keeps your loop, wraps the devices |
| You need to see every line to trust a result | The task is a standard fine-tune of a Hugging Face model | Hugging Face Trainer |
| The project is small and short-lived | A team shares many experiments and needs callbacks and logging hooks | PyTorch Lightning |
Every one of these libraries runs the loop from this lesson inside. When their output looks wrong, the bugs are the same ones, so knowing the lines is what lets you debug them.
Chapter 11
Rebuild the loop from memory, keep the one-screen cheat sheet, know the limits, and follow the series onward
One last test before the cheat sheet. The lines of a loop are scrambled below. Rebuild the inner loop in order, from memory.
Tap the lines in the order they run, starting at the top of the epoch. A wrong pick flashes red and says why.
Some orders are equally correct (zero_grad may also sit right after the step); this device accepts the canonical order.
| Part | In one line |
|---|---|
| The block | train() → [zero_grad → forward → loss → backward → clip → step → sched] × batches → eval() + no_grad → evaluate → checkpoint. |
| Shapes | x: B×… → logits B×C → loss: one number (mean) → every p.grad: same shape as p. |
| Maths in one line | AdamW: p ← (1−ηλ)p − η·m̂/(√v̂+ε); clip: g·min(1, c/‖g‖); accumulate: Σ (nk/N)ḡk; EMA: βθema + (1−β)θ. |
| Knobs | Peak LR, warmup (a few % of steps), decay shape and floor, weight decay (matrices only), clip c (above the typical norm), accumulation K, EMA β (window « run), patience. |
| Bugs to spot | zero_grad between backward and step; no eval() at eval; mean of batch means; scheduler counted in the wrong unit or called before step; Adam(weight_decay) mistaken for AdamW; decaying norms and biases; loss not divided under accumulation; clipping before unscale_; best = state_dict() by reference; checkpoint without optimizer, scheduler or RNG. |
| Checks | Evaluate twice, get the same number; resume and compare the next loss exactly; accumulated grad equals big-batch grad; plot the LR you actually got; log the pre-clip grad norm and throughput. |
| Alternatives | Accelerate (keep your loop), Lightning (callbacks), HF Trainer (standard fine-tunes). Same lines inside. |
Now press Present or Teach and explain, out loud and from memory, why each line of the loop sits where it does. Then go back to the hero, switch off two lines at once, and predict what the curves will do before you look.