Debugging Training

A broken training run rarely crashes. It trains, it prints numbers, and it is quietly wrong. Debugging it is a craft with a method: read the symptoms, run cheap tests that each rule causes in or out, and fix one thing at a time.

Learn a step-by-step recipe, the bugs experts have learned to spot, and the tests that catch them: loss at step zero, overfit one batch, gradient norms per layer, NaN hunting, the learning-rate range test, and unit tests for models.

Here is a patient: a run that went wrong. Run the tests, then name the cause. The right diagnosis applies the fix and retrains it, live, in your browser.

You need the training loop, line by line: forward, loss, backward, step. We build everything else from zero.

The patient

Ready

Run a test

Name the cause

Each patient is a real 2 → 16 → 16 → 3 ReLU network trained in your browser on a three-armed spiral (240 training and 120 validation points), with one planted bug, trained by SGD with momentum. Train loss is warm, validation loss blue; gradient norms are per layer, on a log scale. All numbers are illustrative.

Chapter 0

A bug that trains anyway

Normal code crashes when it is wrong. Training code usually keeps going and gets a little worse. That is why it needs its own method

You flip your training images left to right, to get more data for free. You forget one detail: some labels say "left" and "right". A flipped picture of a left turn now carries the label "left" but shows a right turn.

Nothing crashes. The loss goes down. The run finishes and prints a number. The number is a bit worse than it could be, and nobody knows why. Andrej Karpathy uses almost this example in his 2019 essay A Recipe for Training Neural Networks Karpathy 2019 · 2) fails silently.

In ordinary programs, a mistake usually throws an exception: an error that stops the program and names the line. In training code, he writes, the possible errors are "logical (as opposed to syntactic)". A misconfigured network "will throw exceptions only if you're lucky; Most of the time it will train but silently work a bit worse."

Try it. Below, two models train on the same three classes: left, right and middle. One flips half its training points without fixing the labels. Watch both loss curves. Then guess which run has the bug before you reveal what it cost.

Two runs, one bug

Both runs train live. Pick how often the buggy run flips a point. Then press "Reveal what it cost" to see what the loss was hiding.

Synthetic 2-D data, three classes, 300 training and 300 validation points; a 2 → 16 → 3 network trained with Adam for 400 steps. Which run is buggy is chosen at random each time you change the setting. Numbers are illustrative.

Why this needs a method

At 30% flips, both curves fall. The buggy one settles higher, but you only know that because you have the clean run beside it. In real work there is no clean twin. You see one curve and must decide if it is good.

So we need tests that give a clear yes or no. Each test in this lesson is cheap, runs in seconds, and rules a family of bugs in or out. Karpathy's summary is the motto of the whole lesson: be "thorough, defensive, paranoid, and obsessed with visualizations of basically every possible thing."

The road map of this lesson

  1. The recipe: Karpathy's order of work, and why you add one thing at a time.
  2. Check the loss at step zero: it should be ln C. A Code Lab fixes a bad init.
  3. Overfit one batch: the most useful single test, built in a Code Lab.
  4. The bug zoo: six symptoms, their causes, and the test for each.
  5. Shapes and silent broadcasting: misaligned labels, mixed batches, and a gradient test.
  6. Softmax twice, eval mode, BatchNorm: three bugs in plain sight.
  7. Gradients by layer: dead ReLUs, exploding and vanishing gradients, hooks.
  8. Hunting NaNs: anomaly mode and the log-sum-exp trick.
  9. The learning-rate range test: one run that maps good and bad rates.
  10. Tests for data and models: leaks, noisy labels, and a find-the-bug lab.
  11. Connections: the triage card, the cheat sheet, honest limits.

PyTorch, Zero to Expert

  1. PyTorch Tensors: Storage, Strides and Views
  2. Autograd: the Graph Behind backward()
  3. nn.Module Inside Out
  4. Datasets and DataLoaders
  5. The Training Loop, Line by Line
  6. Debugging Training (this lesson)
  7. Memory and Mixed Precision
  8. Making PyTorch Fast
  9. Distributed PyTorch: DDP and FSDP from the Inside
  10. PyTorch Internals: From a Python Call to a Kernel
  11. Build PyTorch from Scratch
Why can a flip-augmentation bug survive for weeks?

Chapter 1 The method

The recipe

Build from simple to complex, make a guess about every step before you run it, and never add two unchecked things at once

Karpathy's recipe has six stages Karpathy 2019 · The recipe. The first two do the most to prevent bugs, and they come before any fancy model.

  1. Become one with the data. Do not touch model code yet. Look at thousands of examples for hours. He once found duplicate examples this way, and another time corrupted images and labels. Look for imbalance, bias and noisy labels.
  2. Set up the end-to-end skeleton and get dumb baselines. Train a model you "couldn't possibly have screwed up", such as a linear classifier. Fix the random seed. Turn off data augmentation. Verify the loss at step zero. Overfit one batch. Train an input-independent baseline.
  3. Overfit. Get a model big enough to fit the training set. "Don't be a hero": copy the simplest architecture from the most related paper. Adam with a learning rate of 3e-4 is a safe start.
  4. Regularize. Give up some training loss to gain validation loss: more data first, then augmentation, then smaller inputs, dropout, weight decay, early stopping.
  5. Tune. Random search beats grid search when some settings matter more than others.
  6. Squeeze out the juice. Ensembles, and leave it training longer than feels right.

One rule runs through all six. He builds "from simple to complex" and at every step makes "concrete hypotheses about what will happen". Then he either confirms the guess with an experiment or digs until he finds the issue. And he says to plug signals in "one by one", checking each time that you get the boost you expected.

Why one at a time? Suppose each change you add has a small chance p of hiding a bug. Add k changes at once, and the chance that none of them is broken is:

p
The chance that one new change (a loss, an augmentation, a schedule) carries a bug. Assumed the same for each and independent.
k
How many unchecked changes go into one run.
P(clean)
The chance the run has no bug at all.

Worked: p = 0.1. One change: 0.91 = 90% clean. Eight changes at once: 0.98 = 43% clean. Worse, when that run disappoints, you do not know which of the eight to blame.

One change at a time

Each chip is a change you want to add. Some hide a bug (you cannot see which). Press a strategy and count the runs it takes until you know exactly which changes are broken.

8
0.10

"Bisect" means: when a group fails, split it in half and run each half on its own. The bug chance and the change names are illustrative. Real bugs can also interact, which makes all-at-once even harder.

With few bugs, all-at-once plus bisection can be cheap. But look at what each failed run tells you. In the one-at-a-time column, every red run points at exactly one change. In the all-at-once column, a red run only says "something here". And real bugs interact: two bugs can hide each other, so bisection can mislead you.

You plan to add a new loss term, mixup, and a cosine schedule. Following the recipe, what do you do?

Chapter 2 Test 1

Check the loss at step zero

Before any training you can say exactly what the loss should be. If the number is wrong, something is wrong before the first step

Picture a model with C classes that knows nothing yet. The fairest guess gives every class the same probability, 1/C. Cross-entropy is minus the log of the probability given to the true class. So before training, the loss should be:

C
The number of classes.
L0
The cross-entropy of a uniform guess, in nats (natural-log units, what PyTorch reports).

Worked: C = 10 → ln 10 = 2.303. C = 100 → 4.605. C = 3 → 1.099. CS231n gives the same check: on CIFAR-10 "we would expect the initial loss to be 2.302".

Karpathy lists it as "verify loss @ init" Karpathy 2019 · stage 2. If you start far above ln C, the last layer is too confident: its weights are too large, so the logits (the raw scores before softmax) are spread wide. The model is sure of random answers, and sure-and-wrong costs a lot.

How much? If the logits at the start are random with spread s, the expected loss for many classes is roughly:

s
The standard deviation of the logits at step zero.
E[L0]
The average loss over many examples. It comes from the average of ez for a normal z, which is es²/2.

Worked: C = 10, s = 2 → 2.303 + 2 = 4.30. A real check in PyTorch: a 10-class MLP whose last layer was scaled up 10 times started at 4.31.

Where should the loss start?

Pick the number of classes. Then slide the spread of the starting logits, which is what a large last-layer init does. The bars are the losses of 2,000 random examples; the teal line is ln C.

1.00

Logits are drawn as independent normals with spread s; the true class is random. The average is computed from the same 2,000 samples. The approximation ln C + s²/2 is good for many classes and overestimates for few.

Fix the init, not just the number

The cure is to make the last layer start small. Scale its weights down and set its bias on purpose. Karpathy's rule for the bias is "init well": if your data is 1 positive to 10 negatives, set the bias so the network predicts 0.1 at the start. For a sigmoid output the bias is the log-odds of the positive rate π:

π
The fraction of positive examples in the training data.
b
The starting bias of the output. Then sigmoid(b) = π.

Worked: π = 0.1 → b = ln(0.1 / 0.9) = −2.197. For regression with targets around 50, start the bias at 50.

Without this, the curve has a "hockey stick": the first few hundred steps only learn the bias, and the loss drops steeply before real learning starts. It wastes steps and can hide other problems.

A 100-class model starts training at a loss of 9.8. What does that tell you?

Chapter 3 Test 2

Overfit one batch

A healthy model must be able to memorise a handful of examples. If it cannot, there is a bug, and you have found it in seconds

Take one tiny batch, say eight examples. Train on it over and over. A working model with enough capacity will memorise it, and the loss will fall to almost zero. Karpathy: "Overfit a single batch of only a few examples (e.g. as little as two)... If they do not, there is a bug somewhere and we cannot continue to the next stage." Karpathy 2019 · stage 2

CS231n gives the same rule with 20 examples, and adds a warning. Turn regularisation off for this test, or it can stop you reaching zero. And passing is not proof: if a bug makes your features random, you can still memorise a small set and never generalise.

Why is this test so strong? It removes almost every excuse. There is no data volume, no overfitting worry, and no learning-rate schedule. What is left is: can the forward pass, the loss, the backward pass and the optimiser work together at all?

When it fails, the place where it gets stuck is a clue. A model that cannot tell its inputs apart can at best predict how often each label appears. Its loss then stops at the entropy of the labels:

πc
The fraction of the batch (or data set) whose label is class c.
H
The lowest cross-entropy any input-blind model can reach. For C equal classes it is ln C, the step-zero loss.

Worked: 8 examples, 4 classes, 2 of each → H = ln 4 = 1.386. Labels 90% class 0, 10% class 1 → H = 0.9 · 0.105 + 0.1 · 2.303 = 0.325.

So a loss stuck exactly at H means "my model ignores its input". That points at a short list: inputs not reaching the model, a dead layer, or a batch-mixing bug that makes every example look the same.

The one-batch test

Pick a planted bug. The network trains on the same 8 examples for 300 steps. A pass means the loss drops below 0.01. The dashed teal line is the input-blind floor H.

A 2 → 32 → 4 ReLU network, Adam at 0.03 unless the bug says otherwise, 8 points with labels 0,1,2,3,0,1,2,3. "Labels reshuffled" draws a new random order of the labels every step, as a loader bug would. Numbers are illustrative.

Read the curves. "Batch averaged" lands exactly on H. "Labels reshuffled" also cannot fall: the target moves every step. "Softmax twice" stalls above a floor of its own (Chapter 6 explains it). A tiny learning rate barely moves in 300 steps. A missing zero_grad makes every update the sum of all past gradients. That acts like a huge momentum, so the curve swings instead of settling.

On one batch with 3 equal classes, your model's loss stops at 1.0986 and will not move. What is the most likely family of causes?

Chapter 4 Symptoms

The bug zoo: read the symptoms

Six things a broken run shows you, the bugs that usually cause each one, and the cheapest test that tells them apart

A doctor does not test for every disease. The symptoms narrow the list, and a test decides. Training is the same. Each symptom below points at a short list of causes. Each cause has a test that takes under a minute.

One symptom has a number you can predict, so learn it first. A model that always predicts the most common class gets the majority-class accuracy. A model that predicts the label frequencies gets the loss H from Chapter 3. Karpathy calls this an input-independent baseline: train with the inputs set to zero, and your real model must beat it.

πc
How often class c appears.
accmaj
The accuracy of always answering the most common class.
Lblind
The best loss of any model that ignores its input: the label entropy.

Worked: a fraud set with 90% normal → accmaj = 90%, Lblind = 0.325. A model that "reaches 90% accuracy" there has learned nothing yet.

Six symptoms

Pick a symptom. Each picture is a real small run, trained in your browser with the bug planted. The text lists the usual causes and the test that splits them.

Spiral data with 3 classes (for the majority symptom: 2 classes, 90/10). "Great val, bad test" uses a 1-nearest-neighbour memoriser with 30% label noise and a validation set that shares 60% of its rows with training. All numbers are illustrative.

SymptomUsual causesTest that decides
Loss flat from the startNo step (missing opt.step(), params not in the optimiser), labels misaligned, dead layer, LR far too lowOverfit one batch; print grad norms per layer; look at a batch
Loss becomes NaN or infLR too high, log(0), division by a zero std, fp16 overflow, bad input valuestorch.isfinite on inputs and loss; detect_anomaly; LR range test
Val worse than train from step 0Train and val preprocessed differently; dropout or BatchNorm left in train mode for eval; different label mapsEvaluate the training set with the val code path; compare input statistics
Accuracy stuck at the majority classModel ignores input (dead layer, input zeroed); heavy imbalance with no reweightingCompare loss with H(π); input-independent baseline; per-class accuracy
Loss falls but the metric does notMetric computed against labels in another order; argmax on the wrong dim; metric on the wrong splitCompute the metric on the training batch you just fit; unit-test the metric
Great val, terrible testDuplicates across splits, leaked statistics, tuning to val too many times, test from a shifted distributionHash rows to find duplicates; split by source (patient, user, day); keep a never-touched test set
Train loss falls nicely, but validation accuracy stays at 33% for 3 equal classes from the first epoch to the last. Which test do you run first?

Chapter 5 Bug family

Shapes, silent broadcasting and mixed batches

The commonest bugs do not change whether code runs. They change which numbers meet which. A gradient test makes the wiring visible

Broadcasting lets PyTorch combine tensors of different shapes by stretching size-1 dimensions. It is useful. It is also the source of a classic silent bug. Your model outputs predictions of shape (B, 1). Your targets have shape (B,). Subtract them, and the result is not (B,). It is (B, B): every prediction minus every target.

import torch, torch.nn.functional as F
pred = torch.randn(32, 1)          # model output: (B, 1)
y = torch.randn(32)                # targets: (B,)
print((pred - y).shape)            # torch.Size([32, 32])  -- every pair!
loss = F.mse_loss(pred, y)          # runs, with a UserWarning:
# "Using a target size (torch.Size([32])) that is different to the input size
#  (torch.Size([32, 1])). This will likely lead to incorrect results due to
#  broadcasting. Please ensure they have the same size."
loss = F.mse_loss(pred.squeeze(-1), y)   # the fix: (B,) vs (B,)

F.mse_loss warns you; a hand-written ((pred - y) ** 2).mean() does not. What does the wrong loss reward? Average over all pairs:

pi
The prediction for example i.
yj
The target of example j. The bug compares every prediction with every target.
ȳ
The mean target of the batch.
Var(y)
The spread of the targets. No prediction can remove it.

Worked: y = (1, 2, 3). A perfect model, p = (1, 2, 3), scores 0.667 + 0.667 = 1.333. The constant guess p = (2, 2, 2) scores 0 + 0.667 = 0.667. The bug pays the model to predict the mean and ignore the input.

The (B, B) trap

A linear model learns y = 2x + noise by gradient descent. Switch the target shape. Left: which prediction-target pairs the loss compares. Right: the fitted line after training.

B = 16 points, x uniform in [−1, 1], noise standard deviation 0.2; 300 steps of gradient descent with learning rate 0.1. Numbers are illustrative.

Misaligned labels

A second family: inputs and labels that drift apart. You shuffle the inputs with one permutation and the labels with another. Or two files are sorted differently. Or an augmentation flips the image but not the box. The shapes are perfect. The loss sits near ln C, because the labels are now noise. Karpathy's cure is to "visualize just before the net": decode the exact tensors that enter model(x) and look at them with their labels.

perm = torch.randperm(N)
X, Y = X[perm], Y[torch.randperm(N)]   # BUG: two different orders
X, Y = X[perm], Y[perm]                # fix: one permutation for both
# better: keep them together in one Dataset and let DataLoader(shuffle=True) shuffle indices

Batch mixing, and a test that sees it

A third family is the sneakiest: an operation that lets examples in a batch talk to each other. Softmax or normalisation over dim=0 instead of dim=1. A view where a permute was needed. Karpathy notes the network "will typically still train okay because it will learn to ignore data from the other examples."

His test uses autograd as a probe Karpathy 2019 · chart dependencies. Make the loss depend on example i only. Backpropagate to the input. Only row i of the input's gradient may be non-zero. Any other non-zero row is a leak between examples. The same test checks that a causal model at time t sees only times before t.

Your regression model's predictions collapse to almost the same value for every input, and the hand-written MSE loss stalls near the variance of the targets. What should you check first?

Chapter 6 Bug family

Softmax twice, eval mode and BatchNorm

Three bugs that hide in code that looks right: the loss fed probabilities, a model left in the wrong mode, and normalisation over a batch that is too small

Softmax twice

nn.CrossEntropyLoss and F.cross_entropy expect raw logits. Inside, they apply log_softmax themselves. If your model already ends in nn.Softmax, the loss applies softmax a second time, to numbers that are already between 0 and 1.

The code runs and the loss falls. But it can never fall far. Even a perfect one-hot probability vector, fed in as logits, gives the true class only a small edge after the second softmax:

C
The number of classes.
e
e1 = 2.718: the best "logit" the true class can have is a probability of 1, and the others 0.
Lfloor
The lowest loss the buggy model can ever report.

Worked: C = 3 → ln(1 + 2/2.718) = 0.551. C = 10 → ln(4.311) = 1.461. C = 100 → 3.622. A 10-class loss stuck near 1.46 is a fingerprint.

The gradient suffers too. It passes through the softmax's slope twice, so it shrinks. In PyTorch, for logits (4, 0, 0) and true class 0, the gradient on the true logit is −0.035 from the right loss and −0.022 through the double softmax.

Logits in, or probabilities in?

Slide the margin: how far the true class's logit sits above the others. The warm curve is the correct loss; the red curve feeds softmax outputs into the loss.

4.0

All non-true logits are equal, so the curves are exact formulas. Gradients are slopes with respect to the margin.

The same trap has relatives. nn.NLLLoss expects log-probabilities, so it needs log_softmax before it, not softmax. BCEWithLogitsLoss expects logits; BCELoss expects probabilities. Pick the "with logits" versions: they are also safer for numbers.

Eval mode

Two common layers behave differently in training and in evaluation. Dropout zeroes random units in training and does nothing in evaluation. BatchNorm normalises with the current batch's mean and variance in training, and with stored running averages in evaluation. model.eval() and model.train() flip a flag on every submodule to choose.

BatchNorm with tiny batches

In training mode, BatchNorm turns each feature into a z-score using the batch's own statistics:

μB
The mean of this feature over the batch.
σB2
The batch variance (the biased one, divided by B).
ε
A small constant, 1e-5 by default, to avoid dividing by zero.
x̂i
The normalised value of example i, before the learned scale and shift.

Worked, B = 2: inputs (2, 5) → μ = 3.5, σ2 = 2.25 → outputs (−1, +1). Any two different numbers give −1 and +1. PyTorch 2.10 prints exactly tensor([-1.0000, 1.0000]).

So with two examples per batch, the layer erases each feature's value and keeps only "which of the two is bigger". With one example it cannot work at all, and PyTorch says so: ValueError: Expected more than 1 value per channel when training, got input size torch.Size([1, 10]).

How small is too small?

Each dot is one example. Across: its value normalised with the true mean and spread (what eval mode does with good running statistics). Up: what training mode gives with this batch size. On the green diagonal, the two agree.

2

One feature, 256 samples from a normal distribution with mean 3 and spread 2, split into batches of the chosen size. The error is the average absolute gap between the two values.

Fixes, in order of effort: use a bigger batch for BatchNorm layers; or switch to GroupNorm or LayerNorm, which normalise within one example; or freeze BatchNorm in eval mode when fine-tuning with small batches. The running averages are updated with momentum 0.1 by default: new = 0.9 · old + 0.1 · batch value. Read the real thing: torch/nn/modules/batchnorm.py.

A 10-class classifier trains for days and its loss never goes below about 1.46. What is the first thing to check?

Chapter 7 Test 3

Gradients, layer by layer

One loss number hides what each layer is doing. Log the gradient and the update of every layer, and dead, exploding and vanishing layers show themselves

The loss is one number for the whole network. When it is flat, you cannot tell whether every layer is learning slowly or one layer is not learning at all. So log one number per layer. The gradient norm of a weight tensor is the length of its gradient, √(sum of squares). A layer with norm 0 is not learning.

Why gradients explode or vanish

Each layer multiplies the signal by a weight matrix. If each layer makes the signal a little bigger, the product over many layers grows like a power. For ReLU layers with weights drawn with variance g2 · 2 / n, the variance of the activations obeys:

g
The init gain: how much larger than the Kaiming scale √(2/n) the weights are. Kaiming init is g = 1.
L
The number of layers.
Var(hL)
The spread of the activations after L layers (strictly, their mean square). The gradients flowing back follow the same rule.

Worked, L = 10: g = 1.2 → 1.4410 = 38 times the variance (6.2 times the spread). g = 0.8 → 0.6410 = 0.012 (the spread shrinks to 0.11). Ten layers turn a 20% error into a factor of six.

A dead ReLU is a unit whose input is negative for every example. It outputs 0 and passes back a gradient of 0, so it never recovers. A large negative bias, or one huge step from a too-high learning rate, can kill a whole layer. Then every layer below it gets zero gradient too.

Ten layers, one batch

A real forward and backward pass through ten layers of width 64. Raise or lower the gain and watch the bars grow or shrink with depth. Push the bias negative to kill ReLUs.

1.00
0.0

Batch of 64 standard-normal inputs, width 64, biases 0.1 except layer 5; the loss is half the mean squared distance to fixed random targets. Spread and gradient norm are drawn on a log scale from 1e-3 to 1e+3. Weights are re-drawn with a fixed seed whenever you change a setting.

The update-to-weight ratio

A gradient norm alone does not say if a layer moves too fast or too slow: it depends on the size of the weights and on the learning rate. CS231n suggests tracking "the ratio of the update magnitudes to the value magnitudes", for each set of parameters CS231n · Babysitting the learning process:

ΔW
The actual change of the weights in one step. For plain SGD it is the learning rate times the gradient; for Adam, measure it: the weights after the step minus before.
W
The weights before the step.
r
The relative size of one step. The notes call 1e-3 "a rough heuristic": far below means the rate may be too low, far above means too high.

Worked, SGD: learning rate 0.01, ‖grad‖ = 0.5, ‖W‖ = 5 → r = 0.005 / 5 = 1e-3. A layer at 1e-6 is barely moving; a layer at 1e-1 rewrites itself every ten steps.

Run this on a machine with PyTorch

import torch

def step_with_stats(model, opt, loss):
    """One training step that also returns per-layer grad norms and update ratios."""
    opt.zero_grad()
    loss.backward()
    before = {n: p.detach().clone() for n, p in model.named_parameters()}
    gnorm = {n: p.grad.norm().item() for n, p in model.named_parameters()
             if p.grad is not None}
    total = torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)  # returns the norm BEFORE clipping
    opt.step()
    ratio = {n: ((p.detach() - before[n]).norm() / (before[n].norm() + 1e-12)).item()
             for n, p in model.named_parameters()}
    return gnorm, ratio, total.item()     # log all three, every N steps

Two more tools work without touching the training loop. A forward hook, module.register_forward_hook(fn), runs after a module's forward and sees its output: log the fraction of ReLU outputs above zero. A tensor hook, p.register_hook(fn), runs when backward computes p's gradient. Both return a handle; call handle.remove() when done. Read the real thing: torch/nn/modules/module.py and torch/utils/hooks.py.

Your per-layer log shows gradient norms of exactly 0 for layers 0 to 3, and normal values for layers 4 to 6. What happened?

Chapter 8 Test 4

Hunting NaNs

One NaN anywhere spreads to everything it touches. Find the first one, and you have found the bug

NaN means "not a number". It comes from operations with no answer: 0/0, ∞ − ∞, 0 × ∞, the log of a negative number. inf comes from numbers too big for the format, or from log(0) = −∞. Any arithmetic with a NaN gives NaN. So one bad value in one step spreads through the loss, every gradient and every weight. Once a weight is NaN, the run is dead.

The usual sources, roughly in order of how often they bite:

Why exp overflows, and the fix

float32 can hold numbers up to about 3.4 × 1038. That is e88.72. So e89 is already inf. A softmax written as exp(z) / exp(z).sum() breaks as soon as any logit passes 88.7. The cure is the log-sum-exp trick: subtract the largest logit first. It changes nothing in exact maths, and keeps every exponent at or below 0:

m
The largest logit. After subtracting it, the biggest exponent is e0 = 1, so nothing can overflow.
ln Σ ez
The log-sum-exp. Then log py = zy − logsumexp(z): no exp of a large number and no log of zero.

Worked: z = (1000, 0). Naive: e1000 = inf, inf / inf = NaN. Stable: m = 1000 → 1000 + ln(1 + e−1000) = 1000, so log p2 = 0 − 1000 = −1000, finite.

PyTorch's torch.softmax already subtracts the max, so it does not overflow. It can still underflow: a probability below about 1.4 × 10−45 rounds to exactly 0, and log of that is −∞. That is why F.log_softmax and F.cross_entropy exist.

Where does the first NaN appear?

The same loss, written three ways, on logits (z, 0, 0) for true class 2. Push z up. Each box is one step of the computation, forward along the top and backward along the bottom; red means inf or NaN.

20

Every value is rounded to float32 at each step, as on a CPU. The backward row shows the gradient of the loss with respect to each step's output, for the true class; the true class is a non-max logit, the hardest case.

The tools

Assert early. assert torch.isfinite(loss) after the forward pass costs almost nothing. Add it for inputs as they load, too.

Forward hooks find the first module whose output goes bad:

def add_nan_guards(model):
    def guard(name):
        def hook(module, inputs, out):
            if isinstance(out, torch.Tensor) and not torch.isfinite(out).all():
                raise RuntimeError(f"non-finite output from {name} ({type(module).__name__})")
        return hook
    return [m.register_forward_hook(guard(n)) for n, m in model.named_modules()]
# handles = add_nan_guards(model); ...; for h in handles: h.remove()

Anomaly mode checks the backward pass. Inside with torch.autograd.detect_anomaly():, every backward function's output is checked. The first one that returns NaN raises RuntimeError: Function 'SqrtBackward0' returned nan values in its 0th output. A warning also prints the traceback of the forward line that created that function. It is slow, so switch it on only to hunt. Read the real thing: torch/autograd/anomaly_mode.py.

Your custom loss computes torch.log(torch.softmax(z, dim=1)) and the run turns NaN once the logits grow large. What is the right fix?

Chapter 9 Test 5

The learning-rate range test

One short run with a growing learning rate shows where training starts to work and where it blows up. It replaces a dozen guesses

The learning rate is the setting most likely to be wrong. Too low, and the loss crawls; you think the model is weak. Too high, and the loss jumps around or turns NaN; you think the code is broken. Leslie Smith's LR range test maps the whole range in one run Smith 2017 · §3.3.

His words: "run your model for several epochs while letting the learning rate increase linearly between low and high LR values." Then plot accuracy against the rate. "Note the learning rate value when the accuracy starts to increase and when the accuracy slows, becomes ragged, or starts to fall." In his CIFAR-10 example that gave a low bound of 0.001 and a high bound of 0.006.

His 2018 report adds what happens at the top Smith 2018 · §4.1: as the rate grows, "it eventually becomes too large and causes the test/validation loss to increase and the accuracy to decrease." The rate at that turn is the largest usable for a cyclical schedule; "a smaller value will be necessary when choosing a constant learning rate".

The popular version in libraries changes two details. fastai's lr_find grows the rate exponentially, from 1e-7 to 10 over 100 steps by default, and plots the training loss. It stops early when the loss diverges. Its minimum suggestion is "one-tenth the minimum before divergence". An exponential sweep gives every factor of ten the same number of steps:

ηmin
The starting rate, far too small to do anything.
ηmax
The final rate, far too big.
k
The step, from 0 to K − 1.
K
The number of steps in the test, typically 100 to a few hundred mini-batches.
ηk
The rate used at step k. Each step multiplies it by the same factor.

Worked: ηmin = 1e-4, ηmax = 100, K = 120 → factor (106)1/119 = 100.0504 = 1.123 per step, so about 20 steps for each factor of ten.

Sweep, then pick

Left: a real range test on a spiral classifier (SGD with momentum). Slide your pick along the rate axis. Right: a fresh model trained for 300 steps at your pick.

1e-3

2 → 32 → 32 → 3 ReLU network, batch 32, SGD with momentum 0.9; rates from 1e-4 to 100 over at most 140 steps, loss smoothed with an exponential average (0.9) with bias correction, stopped at 4× the best loss. Numbers are illustrative.

Read the curve from left to right. At first the loss is flat: the rate is too small to move anything. Then it falls: the model learns. It reaches a valley. Then it turns and shoots up: the steps are now so big they overshoot. The best constant rate sits on the falling slope, well left of the valley's bottom. Rates near the bottom are already close to unstable.

Your range test's smoothed loss is lowest at a rate of 0.3 and explodes above 1. Which constant rate is the safest first choice?

Chapter 10 Automate

Tests for data and models

Turn every check in this lesson into code that runs on every change: duplicates and leaks in the data, noisy labels, and unit tests for the model

Check the data

Karpathy's first stage is to look at the data, and he names what he found: "duplicate examples" and "corrupted images / labels". Two data bugs matter most for debugging, because they make your numbers lie.

Duplicates across splits. If some test rows also appear in training, a model that memorises scores well on them for free. Your measured accuracy becomes a mix:

d
The fraction of evaluation rows that also appear in training.
accdup
Accuracy on those rows: near 1 for a model that memorises.
accnew
Accuracy on truly new rows: what you will get in deployment.

Worked: d = 0.3, accdup = 1.0, accnew = 0.60 → 0.3 + 0.42 = 0.72. You report 72% and ship 60%.

Find exact duplicates by hashing each row: hashlib.sha1(x.numpy().tobytes()).hexdigest(), then intersect the sets. Find near-duplicates (a photo resized, a sentence with one word changed) by nearest neighbours on embeddings. And split by the right unit: by patient, user, day or recording, not by row, so the same source never sits on both sides.

Label noise. Some labels are just wrong. A simple way to find them: train, then sort the training examples by their loss. Wrong labels crowd the top of the list, because the model keeps disagreeing with them. Look at the top hundred by eye.

Leaks and noisy labels

Pick a data bug. For duplicates, slide how many test rows were copied from training and compare the reported accuracy with the true one. For noise, slide the noise rate and see how many of the flipped labels a loss ranking finds.

0.30

Duplicates: a 1-nearest-neighbour model (it memorises) on two overlapping 2-D classes, 400 training and 400 test rows. Noise: 400 points, a logistic-regression model trained for 200 steps, labels flipped at the chosen rate; "found" counts flipped labels in the top-k by loss, where k is the number flipped. Numbers are illustrative.

Unit tests for models

Each test in this lesson becomes a few lines of pytest. Run them on every change, before any long job. They take seconds and catch the bugs that otherwise cost days.

Run this on a machine with PyTorch (pip install pytest; pytest test_model.py)

# test_model.py -- seconds to run, days saved
import math, torch, torch.nn as nn, torch.nn.functional as F
B, D, C = 8, 20, 5

def make():   # swap in your own model factory
    torch.manual_seed(0)
    return nn.Sequential(nn.Linear(D, 64), nn.ReLU(), nn.Dropout(0.1), nn.Linear(64, C))

def test_output_shape():
    assert make()(torch.randn(B, D)).shape == (B, C)

def test_loss_at_init():
    m = make(); x, y = torch.randn(512, D), torch.randint(0, C, (512,))
    assert abs(F.cross_entropy(m(x), y).item() - math.log(C)) < 0.3

def test_every_param_gets_a_gradient():
    m = make(); F.cross_entropy(m(torch.randn(B, D)), torch.randint(0, C, (B,))).backward()
    for name, p in m.named_parameters():
        assert p.grad is not None and p.grad.abs().sum() > 0, name

def test_batch_independence():
    m = make().eval(); x = torch.randn(B, D, requires_grad=True)
    (m(x)[3] * torch.randn(C)).sum().backward()
    rows = x.grad.abs().sum(dim=1)
    assert rows[3] > 0 and rows.sum() == rows[3]

def test_eval_is_deterministic():
    m = make().eval(); x = torch.randn(B, D)
    with torch.no_grad():
        assert torch.equal(m(x), m(x))

def test_overfits_one_batch():
    m = make(); x, y = torch.randn(B, D), torch.arange(B) % C
    opt = torch.optim.Adam(m.parameters(), lr=1e-2)
    for _ in range(300):
        loss = F.cross_entropy(m(x), y); opt.zero_grad(); loss.backward(); opt.step()
    assert loss.item() < 0.05

Notice what each test is for. The shape test catches wiring. The step-zero test catches init and softmax-twice. The gradient test catches parameters left out of the graph or behind a dead layer. The independence test catches batch mixing. The eval test catches a missing eval() path. The one-batch test catches nearly everything else. PyTorch's own torch.testing.assert_close and torch.autograd.gradcheck are the tools for comparing numbers and checking hand-written gradients.

A model scores 95% on validation and 71% on a fresh test set collected a month later. Which check comes first?

Chapter 11

Connections

The triage card for your next broken run, the cheat sheet, what these tests cannot tell you, and where to go next

When a run goes wrong, do not start by changing things. Start by naming the symptom, then run the cheapest test that splits its causes. Pick a symptom below for the order of checks.

Triage card

Pick what you see. The card lists the checks in the order to run them: cheapest and most likely first.

An ordering for common cases, not a proof. Fingerprint numbers assume C equal classes.

The cheat sheet

CheckIn one line
The recipeData first; a dumb end-to-end skeleton; overfit; regularise; tune. One change at a time, each with a prediction.
Step-zero lossln C for C classes (2.303 for 10). Far above: last layer too big. Binary bias = ln(π/(1 − π)).
Overfit one batch2 to 20 examples, regularisation off, loss must reach about 0. Stuck at H(π): the model ignores its input.
Fingerprintsln C: input-blind. ln(1 + (C − 1)/e): softmax twice. Var(y): broadcast MSE. Majority accuracy: nothing learned.
ShapesAssert shapes; (B,1) − (B,) is (B,B); one permutation for X and Y; gradient test for batch independence.
ModesLogits into cross_entropy; eval() + no_grad() to evaluate, train() after; BatchNorm needs batches well above 2.
GradientsLog grad norm and update/weight ratio (about 1e-3) per layer; zeros below a layer = dead ReLUs; growth with depth = bad init.
NaNisfinite asserts, forward hooks, detect_anomaly; log_softmax, eps in sqrt and division; lower the LR.
Learning rateRange test: exponential sweep, stop at 4× best; pick on the falling slope, about one tenth of the minimum.
DataHash for duplicates; split by source; fit statistics on train only; sort by loss to find bad labels.

Honest limits

Where this sits, and where to go next

This is part 6 of PyTorch, Zero to Expert, inside The Builder's Toolkit. It builds on the training loop and autograd (hooks and graphs). Next comes memory and mixed precision, where a new family of NaNs lives.

The takeaway. Training code fails silently, so you must make it fail loudly. Predict every number before you see it: the step-zero loss, the one-batch floor, the gradient norm of each layer. When a number disagrees, the gap is your bug. Add one change at a time, and turn every check into a test.
A new model's loss is flat from the first step. Which order of checks follows this lesson?

Now press Present or Teach and explain, out loud and from memory, why a loss stuck at exactly ln C means the model ignores its input. Then go back to the patients and diagnose all four with as few tests as you can.

Debugging Training, part 6 of PyTorch, Zero to Expert (The Builder's Toolkit). Sources: A. Karpathy, "A Recipe for Training Neural Networks" (blog post, 25 April 2019, karpathy.github.io); Stanford CS231n course notes, "Neural Networks Part 3: Learning and Evaluation" (sanity checks; ratio of weights to updates, cs231n.github.io); L. N. Smith, "Cyclical Learning Rates for Training Neural Networks" (WACV 2017; arXiv 2015, v6 2017, arXiv); L. N. Smith, "A Disciplined Approach to Neural Network Hyper-Parameters: Part 1" (US Naval Research Laboratory technical report, 2018, arXiv); fastai documentation, lr_find (docs.fast.ai). PyTorch behaviour (error messages, defaults, numbers) was checked against PyTorch 2.10.
Back to Gleams