When you run a PyTorch expression, autograd quietly writes down every step as a node. loss.backward() then walks those nodes in reverse. Each node multiplies the incoming gradient by its own local derivative and passes it on. That is the whole trick.
Learn how the graph is recorded, why only leaves get .grad, why gradients add up, how to switch autograd off, why in-place ops fail, how to hook, extend and check it, and why reverse mode is the right tool for one scalar loss.
Pick an expression and move the inputs. The purple boxes are the real grad_fn nodes PyTorch records. Press backward() and watch the gradient travel edge by edge, each step multiplied by one local derivative.
You need PyTorch tensors and backpropagation: the chain rule and what a gradient is. We build everything else from zero.
Node names are the ones PyTorch 2.10 prints for these expressions. Moving a slider runs a fresh forward pass (and a fresh graph). Pressing backward twice on one graph gives PyTorch's real error: the graph was freed after the first call. The finite-difference check uses a step of 10−6 in double precision.
Chapter 0
One line of loss code, a billion weights, and three ways to get the slope for every one of them
Here is one line of a training script:
loss = ((w * x + b).tanh() - t).pow(2).mean()
To train, you need the slope of loss with respect to w and b. In a real model, w is not one number. It is a billion numbers. You need all billion slopes on every step, thousands of times.
The vector of all those slopes is the gradient. There are three ways to get it.
The nudging method is easy to understand, so start there. For one weight wi:
Worked: L(w) = tanh(1.5w − 0.4) at w = 0.8. True slope = 1.5 × (1 − tanh²0.8) = 0.83858. With h = 10−3: (tanh(0.8015) − tanh(0.8)) / 0.001 = 0.83775. Close, but one run per weight.
Count the cost. With N weights, nudging needs N + 1 runs of the model. Autograd needs one forward run and one backward run, whatever N is. Slide N below. Then slide the step h and watch a second problem: rounding.
Left: runs of the model needed for one full gradient. Right: the error of one finite difference as you shrink the step h. Too big a step bends; too small a step drowns in rounding.
The error curves are computed live for L(w) = tanh(1.5w − 0.4) at w = 0.8: float64 in JavaScript's own numbers, float32 by rounding every step with Math.fround. Run times assume 10 ms per forward pass and a backward pass of about 2 forward passes (Chapter 10 explains the 2); they are illustrative.
Two lessons hide in the right panel. First, nudging is never exact: a large h measures a chord, not a tangent, and a tiny h subtracts two nearly equal numbers. Second, the best step depends on the number format. In float32 the error never gets below about 10−4. That is why PyTorch's own gradient checker (Chapter 8) works in float64.
grad_fn and next_functions..grad, and retain_grad().no_grad, inference_mode, detach, requires_grad_(False).autograd.Function, a straight-through estimator, a stable function, gradcheck.create_graph, torch.autograd.grad, a gradient penalty.This is lesson 2 of an eleven-part series inside The Builder's Toolkit. Each lesson has Code Labs that run a copy of the PyTorch API in your browser.
Chapter 1
Every operation knows only its own local derivative; backward multiplies a vector by it and passes the result on
Take the hero's first expression with x = 1.5, w = 0.8, b = −0.4. Forward, it runs in two steps: u = w·x + b = 0.8, then L = tanh(u) = 0.664.
The chain rule says: the slope of L with respect to w is the slope of L with respect to u, times the slope of u with respect to w. Each factor belongs to one step. The tanh step knows its own slope, 1 − tanh²(u). The multiply step knows its slope with respect to w, which is x. Neither needs to know about the other.
That is the design of autograd. Every operation carries a small backward rule. Backward starts with a 1 at the loss, then each node multiplies what arrives by its local slope and hands the product to its inputs.
Real tensors hold many numbers. An operation y = f(x) with n inputs and m outputs has m × n local slopes. Laid out as a table, they form the Jacobian J: row i, column j holds ∂yi/∂xj.
Write the gradient of the loss with respect to any tensor with a bar on top: ȳ is ∂L/∂y, with the same shape as y. Then the chain rule for one node is one product:
Worked (hero, first expression): L̄ = 1 → ū = 1 × (1 − 0.664²) = 0.559 → w̄ = ū × x = 0.559 × 1.5 = 0.839, b̄ = ū × 1 = 0.559.
Why never build J? For tanh on a million numbers, J is a million by a million: a trillion numbers, almost all zero. Each output depends only on its own input, so J is diagonal. The VJP is then just ȳ times (1 − y²), element by element. Every PyTorch operation ships a rule like this. Compare the two below.
Pick an operation and a size. The grid is the full Jacobian. The warm row under it is ȳTJ, computed two ways: by the full table, and by the operation's VJP rule. They always agree; only the cost differs.
Inputs x and the upstream gradient ȳ are fixed random numbers. Cell shade shows the size of each entry; the sign is printed when there is room. "Numbers stored" counts float32 values; the 106-input row is the same operation at a real activation size.
| Forward | Backward (the VJP) | What it saves |
|---|---|---|
y = a + b | ā = ȳ, b̄ = ȳ | nothing |
y = a * b | ā = ȳ · b, b̄ = ȳ · a | both inputs |
y = x.tanh() | x̄ = ȳ · (1 − y²) | its output y |
Y = X @ W | X̄ = Ȳ WT, W̄ = XT Ȳ | X and W |
y = x.sum() | x̄ = ȳ copied to every place | only the shape |
y = x.mean() | x̄ = ȳ / n in every place | only the shape |
Y = X + b (b broadcast over rows) | b̄ = the sum of Ȳ over the rows | only the shapes |
Read the matmul row with shapes. If X is (32, 64) and W is (64, 10), then Y and Ȳ are (32, 10). W̄ = XTȲ is (64, 32) times (32, 10) = (64, 10): the same shape as W, as every gradient must be. The broadcast row is the same idea: the bias was copied to 32 rows, so its gradient is summed back over 32 rows.
Chapter 2
Each tracked operation leaves a node behind; grad_fn points at it and next_functions links it to its inputs
PyTorch has no separate "build the graph" step. The graph is recorded while your Python code runs. This style is called define-by-run. Each time an operation runs on a tensor that requires a gradient, PyTorch creates a backward node for it.
Three attributes make the graph visible.
t.grad_fn is the node that made tensor t. Its name says which rule it runs: MulBackward0, TanhBackward0, MeanBackward0.node.next_functions is a tuple with one pair (node, input_nr) per input. It points at the nodes of the inputs, so it points backwards, toward the leaves.AccumulateGrad. Its job is to add the gradient that arrives into the leaf's .grad. An input that needs no gradient appears as None.Step through the program below one line at a time and watch the graph grow.
Move the line slider. Each line that operates on a tensor with requires_grad=True adds one purple node. Below the picture, the readout prints what PyTorch itself prints for that line.
Names and the structure of next_functions match PyTorch 2.10, which prints the leaf node as torch::autograd::AccumulateGrad. Object addresses are left out.
Look at the hero's third expression, w·x + sin(w). The weight w feeds two nodes, so two gradients flow back to it. The engine adds them before it calls w's node. In general, the gradient of a tensor is the sum over all the places that used it:
Worked: L = w·x + sin(w) with w = 0.8, x = 1.5. From the multiply: 1.5. From the sine: cos(0.8) = 0.697. Total w̄ = 2.197. Try it in the hero.
This is also why the engine runs nodes in dependency order. It counts how many gradients each node still waits for. A node runs only when all of them have arrived and been added. Then it runs once, with the total.
Control flow is just Python. An if or a loop in your forward pass is not part of the graph. Only the operations that actually ran are recorded. A model with an if records a different graph on each branch. Each forward pass makes a fresh graph.
Now walk the graph yourself, with real next_functions, the same way the engine does.
x = torch.randn(3) (no gradient) and w = torch.randn(3, requires_grad=True). Then y = (w * x).sum(). What is y.grad_fn.next_functions[0][0].next_functions?Chapter 3
.gradWhich tensors are leaves, why the gradients of everything else are thrown away, and the one-character bug that makes .grad None
A leaf is a tensor with no grad_fn: nobody made it with a tracked operation. You created it. Weights, biases and input data are leaves. Everything computed from a tensor that requires a gradient is a non-leaf.
Whether a result needs a gradient follows one rule:
grad_fn and is a non-leaf.Worked: x (False) × w (True) → True, with grad_fn=<MulBackward0>. x (False) × 2 → False: no node is recorded at all.
After loss.backward(), only leaves with requires_grad=True get their .grad filled in. The gradients of non-leaves are computed on the way, used, and dropped. Keeping them would cost as much memory as all the activations again. If you want one, call h.retain_grad() before backward.
Here is the trap. A line that looks like "make a weight" can make a non-leaf. Try each line below.
Pick a line. The picture shows where the tensor sits in the graph. The readout gives is_leaf, grad_fn and what .grad holds after loss.backward(). Predict before you look.
Every row was run on PyTorch 2.10. ".to(float64)" stands in for ".cuda()" or ".to('cuda')", which behave the same way on a GPU machine: the copy is a non-leaf made by ToCopyBackward0.
The fix for rows 2 and 3 is to finish building the tensor first, and only then ask for gradients: (torch.randn(3) * 0.01).requires_grad_(), or create it on the right device directly with torch.randn(3, device="cuda", requires_grad=True). Inside a model, use nn.Parameter, which is always a leaf.
w = torch.randn(3, requires_grad=True).cuda() and then torch.optim.SGD([w], lr=0.1). What happens?Chapter 4
Gradients are added into .grad, never written over it; and the graph's saved tensors are freed as backward goes
Run the hero's backward on fresh graphs twice without resetting, and w.grad doubles. That is not a bug. The AccumulateGrad node adds:
Worked (hero, first expression): first backward: w.grad = 0.839. A second forward and backward without a reset: 0.839 + 0.839 = 1.677.
Why add instead of write? Because sometimes you want the sum. The second main use is gradient accumulation: your batch of 64 does not fit in memory, so you run 4 micro-batches of 16 and call backward() on each. The four gradients add up in .grad, and you step once.
For the sum to equal the full-batch gradient, scale each micro-batch loss. If the full loss is a mean over N examples, split into k equal chunks:
(loss / k).backward().Worked: N = 64, k = 4 chunks of 16. Each chunk mean weighs each example by 1/16. Times 1/4 gives 1/64: exactly the full-batch weight.
The flip side: in an ordinary loop you must clear .grad every step with optimizer.zero_grad(). Forget it, and each step uses the sum of all past gradients. Try every mode below.
One weight w learns the slope of y = 3x by SGD (learning rate 0.9). Pick a loop and press Train. Top: w over the steps (the target 3 is dashed). Bottom: what .grad held when the optimizer stepped.
Loss: mean of (w·x − y)² over 64 points with x evenly spread on [−1, 1] and y = 3x, start w = 0. Every number is computed in your browser; nothing is drawn by hand.
optimizer.zero_grad() in PyTorch 2.x has set_to_none=True by default. It sets each .grad to None instead of filling it with zeros. That frees the memory until the next backward, and the next backward then writes instead of adding. Code that reads p.grad right after zero_grad() must expect None.
Backward has a second side effect. Many nodes saved tensors for their backward rule: the tanh node saved its output, the multiply node saved its inputs. These saved tensors are the activations that make training use so much memory. As backward passes each node, it frees that node's saved tensors. When backward ends, the graph is an empty shell.
So a second backward() through the same graph fails with: "Trying to backward through the graph a second time (or directly access saved tensors after they have already been freed)". backward(retain_graph=True) keeps the saved tensors, at the cost of keeping their memory.
Press forward, then backward, then backward again. Turn on retain_graph and repeat. The bar on the right is memory held by saved tensors.
A 3-layer MLP at batch 256 and width 4,096 in float32: every saved activation is 256 × 4,096 × 4 bytes = 4 MiB. Linear layers (AddmmBackward0) save their input; for layers 2 and 3 that is the ReLU output the ReLU node already saved, so it is one buffer, counted once. The MSE loss saves its input. Weights are referenced, not copied, so they are not counted. Sizes are illustrative.
The classic way to hit this error is a recurrent model trained on a long stream. You carry the hidden state from one batch into the next. Its history reaches into the previous batch's graph, which backward already freed. The fix is not retain_graph=True. The fix is h = h.detach() at each batch boundary: keep the value, cut the history.
Chapter 5
no_grad, inference_mode, detach and requires_grad_(False) all stop gradients, in four different places
Recording has a cost. Every node saves tensors, and those tensors hold memory until backward. When you only evaluate, or serve a model, you want no recording at all. When you train only part of a model, you want recording for some tensors and not others. PyTorch has four tools, and they are not the same.
with torch.no_grad(): turns recording off inside the block. Outputs have requires_grad=False and no grad_fn. They are ordinary tensors you can use later.with torch.inference_mode(): is no_grad and more. It also skips the version counter and view tracking, so it is a little faster. Its outputs are inference tensors: they can never be saved for a later backward, and you cannot change them in place outside the block.y = x.detach() cuts one tensor out of the graph. y shares x's storage (and its version counter) but has no history. Gradients stop at y. The rest of the graph still records.w.requires_grad_(False) is a flag on a leaf. w gets no gradient. But if w's layer gets an input that requires grad, the layer is still recorded, and gradients still flow through it to the layers before it.How much memory does recording hold? Roughly the saved activations:
Worked (illustrative): L = 24, s = 8, B·T·D = 8 × 1,024 × 1,024, bfloat16 → 24 × 8 × 8.4 M × 2 B = 3.2 GB that an evaluation loop under no_grad never allocates.
h comes from an earlier trainable layer, so it requires grad. W is this layer's weight. Pick a way to run loss = tanh(W @ h).sum(), then press backward. Grey dashed edges were not recorded.
The results (which .grad is set, the error text) are what PyTorch 2.10 does for this program. With h a 3-vector and W a 3 × 3 matrix, the matmul node is MvBackward0.
| Tool | Scope | Records a graph? | Output usable in autograd later? | Typical use |
|---|---|---|---|---|
torch.no_grad() | a block of code | no | yes, as a constant | evaluation loops; the optimizer's own update |
torch.inference_mode() | a block of code | no, and no version tracking | no: inference tensors cannot be saved for backward | serving; batch prediction |
t.detach() | one tensor | yes, for everything else | yes, as a constant that shares t's memory | targets, logging, carrying RNN state, stop-gradient tricks |
p.requires_grad_(False) | one leaf, until you switch it back | yes, if other inputs require grad | yes | freezing a pretrained backbone |
Two more you will see. torch.set_grad_enabled(flag) is no_grad with an on-off switch, handy in one function that serves both training and evaluation. And all of these work as decorators: @torch.no_grad() above a function turns recording off for its whole body.
requires_grad_(False)) sits between two trainable layers. Does the first trainable layer still get gradients?Chapter 6
Why "modified by an inplace operation" happens, how PyTorch detects it, and the one bypass that gives silently wrong gradients
An in-place operation changes a tensor's numbers in its existing memory instead of making a new tensor. Its name ends in an underscore: add_, mul_, clamp_. Indexed assignment such as x[0] = 5 and x += 1 are in place too.
Now remember Chapter 4: nodes save tensors for backward. The sigmoid node saves its own output, because its derivative is s(1 − s). If you then change that output in place, backward would use the wrong numbers.
PyTorch catches this with a version counter. Every tensor's storage has one. Each in-place change adds 1. When a node saves a tensor, it also writes down the version. At backward time, it checks:
Worked: y = sigmoid(z) is saved at version 0. y.mul_(2) makes it version 1. Backward compares 1 with 0 and stops: "…is at version 1; expected version 0 instead."
The program is y = torch.sigmoid(z) with z = (−1, 0.5, 2), then one change to y, then y.sum().backward(). Pick the change, then press backward.
All gradients and the error text were checked on PyTorch 2.10. The .data case really does give z.grad = 2s(1 − 2s), not the correct 2s(1 − s).
It depends only on what the earlier nodes saved. Addition saves nothing, so y = x + 1; y.add_(1) is fine. Multiplication saves both inputs. Exp, tanh, sigmoid and ReLU save their output. So changing the output of a sigmoid in place fails, and changing the output of an addition does not. nn.ReLU(inplace=True) works because ReLU's backward needs only where its output is positive, which its own in-place write keeps.
Leaves have their own rule. w.add_(1) on a weight that requires grad fails with "a leaf Variable that requires grad is being used in an in-place operation." Recording a change to a leaf would make the leaf a non-leaf. The way to update weights is to switch recording off first, which is what every optimizer does inside step():
with torch.no_grad(): w -= lr * w.grad # in place, not recorded: w stays a leaf
Now trigger the error yourself, read it, and fix it.
Chapter 7
Read or change a gradient while backward runs, and find the forward line that made a NaN
Backward runs in C++, out of sight. Two tools let you see into it.
A tensor hook is a function you attach with t.register_hook(fn). When backward computes the gradient for t, it calls fn(grad). If fn returns None, the gradient goes on unchanged. If fn returns a tensor, that tensor replaces the gradient for everything upstream. register_hook returns a handle; call handle.remove() to take the hook off.
Uses: log the gradient norm at each layer to find where gradients vanish; clip one tensor's gradient; flip a gradient's sign, as in the gradient reversal layer used for domain adaptation. A common clipping rule:
Worked: g = (3, 4), ‖g‖ = 5, c = 1 → g′ = (3, 4) × 0.2 = (0.6, 0.8), length 1.
Six layers hk+1 = tanh(a·hk). Bars show the size of the gradient at each layer, loss on the right. Put a hook on layer 3. Then inject a NaN and compare backward with and without anomaly mode.
Input h0 = 0.3. Gradients are computed exactly in your browser. The NaN case adds sqrt(0·h) into layer 2: the forward value is fine (0), but the slope of sqrt at 0 is infinite, and infinity times 0 is NaN. The anomaly message is PyTorch 2.10's text for this case.
A hook on a weight runs before the gradient is added into .grad, so what it returns is what gets added. To act after .grad is final, use p.register_post_accumulate_grad_hook(fn): fn receives the parameter itself. Some memory-saving training loops call the optimizer step for each parameter there, then free its gradient at once.
On a module, register_full_backward_hook(fn) calls fn(module, grad_input, grad_output): the gradients with respect to the module's inputs and outputs. Not its weights. For weights, hook the parameters.
A NaN in a gradient tells you nothing about where it came from. with torch.autograd.detect_anomaly(): changes two things. In forward, every node records the Python stack that created it. In backward, every node's output is checked for NaN. The first bad node raises an error that names it, for example "Function 'SqrtBackward0' returned nan values in its 0th output.", and prints the forward line that made it. Anomaly mode is slow. Turn it on to find a bug, then off.
g * 0. Which weights get a zero gradient?Chapter 8
autograd.FunctionGive an operation a backward rule you choose: a straight-through estimator, a numerically stable formula, and the gradient checker that proves you got it right
Sometimes the automatic rule is not the rule you want. Four common reasons:
round, sign and a hard clamp are flat almost everywhere, so their gradient is 0 and nothing behind them learns.A custom Function is a class with two static methods:
import torch class Cube(torch.autograd.Function): @staticmethod def forward(ctx, x): # x: any shape ctx.save_for_backward(x) # keep what backward needs (version-checked) return x ** 3 @staticmethod def backward(ctx, g): # g: dL/d(output), same shape as the output x, = ctx.saved_tensors return g * 3 * x ** 2 # one gradient per forward input x = torch.tensor([2.0], requires_grad=True) Cube.apply(x).sum().backward() # call it with .apply, never Cube()(x) print(x.grad) # tensor([12.])
Rules worth knowing by heart. backward returns one value per input of forward, in order; return None for inputs that are not tensors or need no gradient. ctx.needs_input_grad tells you which inputs want one, so you can skip work. Save tensors with ctx.save_for_backward, not as ctx.x = x: the official path checks the version counter and frees memory on time.
A quantized network rounds its weights to a few levels in forward. The rounding step has slope 0, so the full-precision weights behind it would never move. The straight-through estimator (STE) keeps the rounding in forward and pretends, in backward, that it was the identity: backward returns g unchanged. Bengio, Léonard and Courville name it and credit the idea to a 2012 lecture by Hinton. They call it "clearly a biased estimator" Bengio 2013 · §4. Biased, and it works. Try it.
The loss is (q(w) − 0.73)², where q rounds w to the nearest level. Pick the backward rule and press Train. Left: the staircase q(w) and the weight. Right: the loss over the steps.
Start w = −0.6, plain gradient descent with learning rate 0.1. The target 0.73 lies between levels on purpose: the best q can do is the nearest level. Numbers are computed live.
torch.log(torch.sigmoid(x)) is correct maths. But for x below about −88, the float32 sigmoid on a CPU rounds to 0 and the log returns −inf. The automatic backward then produces NaN. Rewrite it so no step can overflow:
Worked: x = −200: min = −200, log1p(e−200) ≈ 10−87 → −200, slope σ(200) = 1. The naive form gives −inf and a slope of 0.
A hand-written backward is a hand-written bug waiting to happen. torch.autograd.gradcheck(fn, inputs) builds the Jacobian twice: once from your backward, once by nudging each input with a central difference. It fails if any entry differs by more than atol + rtol × |numerical|. Its defaults are eps = 10−6, atol = 10−5, rtol = 10−3.
Worked: f = x³ at x = 2: exact 12. Central difference error from the step: about ε² = 10−12. From float64 rounding: about 10−16/10−6 = 10−10. Both far below atol. In float32 the rounding term would be 10−7/10−6 = 0.1, which is why gradcheck wants float64 inputs.
gradgradcheck does the same for the gradient of your gradient (Chapter 9). Run gradcheck on small, random, float64 inputs, away from kinks such as 0 for abs.
Chapter 9
torch.autograd.grad against backward(), create_graph=True, and a loss with a gradient inside it
So far every gradient landed in .grad. torch.autograd.grad(outputs, inputs) is the other door. It returns the gradients as a tuple and does not touch .grad at all.
loss.backward() | torch.autograd.grad(loss, xs) | |
|---|---|---|
| Result goes to | added into .grad of every leaf | returned as a tuple, one per input in xs |
| Which tensors | all leaves that require grad (or inputs=) | exactly xs, leaf or not |
| Unused input | its .grad stays as it was | error, unless allow_unused=True (then None) |
| Typical use | a training step | a gradient inside a loss; per-input gradients; meta-learning |
A gradient is a tensor computed by the backward nodes. Normally those computations are not recorded, so the result has no history. Pass create_graph=True and backward records itself. Then the gradient has a grad_fn, and you can differentiate it again.
y = x**3 - 2*x.g, = torch.autograd.grad(y, x, create_graph=True). g has grad_fn=<AddBackward0>.torch.autograd.grad(g, x): the gradient of the gradient.Worked, x = 2: f = 8 − 4 = 4, f′ = 12 − 2 = 10, f″ = 12. Without create_graph, the second call fails: "element 0 of tensors does not require grad and does not have a grad_fn".
Drag x. The blue curve is f, the warm curve f′ (the first backward), the dashed warm curve f″ (the second). Turn create_graph off and the second backward has nothing to differentiate.
Values are computed from the formulas; on PyTorch 2.10 the two autograd.grad calls return the same numbers.
WGAN-GP trains a critic D that scores real and fake samples. The theory wants D's slope with respect to its input to have length 1. So the loss adds a gradient penalty on points mixed between real and fake samples Gulrajani 2017 · §4:
torch.autograd.grad(D(x_hat).sum(), x_hat, create_graph=True), shape (B, features).Worked: one slope g = (0.6, 1.2) → ‖g‖ = 1.342 → (0.342)² = 0.117 → × 10 = 1.17.
The .sum() is a trick worth understanding. Each example's score depends only on its own input, so the gradient of the sum with respect to x̂ gives every example its own slope, in one backward pass. The penalty then needs the gradient of those slopes with respect to the weights: a second backward through the first. Without create_graph=True the penalty is a constant and trains nothing, with no error at all.
w.grad exactly as it was?Chapter 10
One reverse pass gives one row of the Jacobian; one forward-mode pass gives one column; a loss has exactly one row
Take any function f from n inputs to m outputs. Its full Jacobian is an m × n table. There are two ways to fill it.
Worked: a loss with n = 109 weights, m = 1: reverse mode 1 pass; forward mode 109 passes. A simulator with 3 knobs and 10,000 outputs: forward mode 3 passes; reverse mode 10,000.
Why is one pass "a small multiple" of a forward pass? Look at a matmul Y = XW. Forward is one matmul. Backward needs X̄ = ȲWT and W̄ = XTȲ: two matmuls of the same size. So a backward pass costs about two forward passes, and a training step about three. That is the 2 in Chapter 0.
Set n and m, pick a mode and press Fill. Each pass fills one row (reverse) or one column (forward). Set m = 1, the shape of a loss, and compare.
Pass counts are exact. Real cost per pass depends on the model; the cost bars use 3 forward-pass units for a reverse pass (forward plus a backward of about 2) and about 3 for a forward-mode pass, as an illustration.
torch.autograd.functional has jacobian(f, x), hessian(f, x), vjp(f, x, v), jvp(f, x, u) and hvp. torch.func has the composable versions: jacrev (rows by reverse mode), jacfwd (columns by forward mode), vmap to run many passes as one batched pass, and hessian, which is jacfwd(jacrev(f)): forward over reverse.
A Hessian of a loss is n × n. For n = 109 that is 1018 numbers: impossible to store. But a Hessian-vector product H u costs about one gradient plus one more pass. Second-order methods and curvature studies use only those.
Run this on a machine with PyTorch
import torch from torch.func import jvp, jacrev, jacfwd A = torch.randn(2, 6) f = lambda x: torch.tanh(A @ x) # R^6 -> R^2 x = torch.randn(6) J1 = jacrev(f)(x) # (2, 6): 2 reverse passes, one per row J2 = jacfwd(f)(x) # (2, 6): 6 forward passes, one per column out, col0 = jvp(f, (x,), (torch.eye(6)[0],)) # true forward mode print(torch.allclose(J1, J2), torch.allclose(col0, J1[:, 0])) # True True
Forward mode is not in the browser's mini-torch, so the lab below uses torch.autograd.functional.jvp, which builds a JVP out of two reverse passes. The count of passes is what the lab is about, and it does not change.
Chapter 11
A bug finder for the errors you will actually see, the cheat sheet, honest limits, and where to go next
Almost every autograd bug shows up as one of eight symptoms. Pick the one you have. The card gives the cause, the fix, and the chapter that explains it.
Pick a symptom. The card shows the usual cause, the one-line fix and the chapter to reread.
Error texts are PyTorch 2.10's.
| Idea | In one line |
|---|---|
| The mechanism | Forward records one node per op; backward walks them in reverse, each computing x̄ = JTȳ. |
| See the graph | t.grad_fn, node.next_functions, node.name(); leaves have AccumulateGrad. |
| Leaves | No grad_fn. Only leaves with requires_grad get .grad; retain_grad() for others. |
| Accumulation | .grad is added to: zero_grad() every step; (loss / k).backward() to accumulate. |
| Freeing | Backward frees saved tensors; retain_graph=True keeps them; detach carried state. |
| Off switches | no_grad (block), inference_mode (faster, final), detach (one tensor), requires_grad_(False) (freeze a leaf). |
| In place | Version counter must match what was saved; update weights under no_grad; never .data. |
| Hooks | t.register_hook(fn): read or replace a gradient; detect_anomaly() finds the first NaN node. |
| Custom op | autograd.Function: forward(ctx, ...), backward(ctx, g) → one grad per input; prove it with gradcheck in float64. |
| Higher order | torch.autograd.grad(..., create_graph=True) makes a gradient you can differentiate again. |
| Cost | Full Jacobian: m reverse passes or n forward passes. A loss has m = 1, so one reverse pass. |
torch.compile captures the graph ahead of time to remove it (Making PyTorch Fast).AccumulateGrad where PyTorch prints torch::autograd::AccumulateGrad. The lab code runs unchanged on real PyTorch.Reverse-mode differentiation is older than deep learning: Seppo Linnainmaa described it in 1970, and Rumelhart, Hinton and Williams made it the training method for neural networks in 1986 as backpropagation. PyTorch's contribution was to make it define-by-run: the graph is your ordinary Python program, recorded as it runs Paszke 2019.
Where to go next on this site:
running = running + loss for logging. What is the fix?Now press Present or Teach and explain, out loud and from memory, what happens between loss.backward() and w.grad. Then go back to the hero, pick w·x + sin(w), and predict w.grad before you press backward.