Autograd: the Graph Behind backward()

When you run a PyTorch expression, autograd quietly writes down every step as a node. loss.backward() then walks those nodes in reverse. Each node multiplies the incoming gradient by its own local derivative and passes it on. That is the whole trick.

Learn how the graph is recorded, why only leaves get .grad, why gradients add up, how to switch autograd off, why in-place ops fail, how to hook, extend and check it, and why reverse mode is the right tool for one scalar loss.

Pick an expression and move the inputs. The purple boxes are the real grad_fn nodes PyTorch records. Press backward() and watch the gradient travel edge by edge, each step multiplied by one local derivative.

You need PyTorch tensors and backpropagation: the chain rule and what a gradient is. We build everything else from zero.

The graph behind backward()

Ready
1.5
0.8
-0.4

Node names are the ones PyTorch 2.10 prints for these expressions. Moving a slider runs a fresh forward pass (and a fresh graph). Pressing backward twice on one graph gives PyTorch's real error: the graph was freed after the first call. The finite-difference check uses a step of 10−6 in double precision.

Chapter 0

Who computes the gradient?

One line of loss code, a billion weights, and three ways to get the slope for every one of them

Here is one line of a training script:

loss = ((w * x + b).tanh() - t).pow(2).mean()

To train, you need the slope of loss with respect to w and b. In a real model, w is not one number. It is a billion numbers. You need all billion slopes on every step, thousands of times.

The vector of all those slopes is the gradient. There are three ways to get it.

The nudging method is easy to understand, so start there. For one weight wi:

L
The loss: one number computed from all the weights w.
h
The step: a small number such as 10−3.
ei
A vector of zeros with a 1 in place i. It nudges only weight i.
∂L/∂wi
The partial derivative: how fast L changes when only wi moves.

Worked: L(w) = tanh(1.5w − 0.4) at w = 0.8. True slope = 1.5 × (1 − tanh²0.8) = 0.83858. With h = 10−3: (tanh(0.8015) − tanh(0.8)) / 0.001 = 0.83775. Close, but one run per weight.

Count the cost. With N weights, nudging needs N + 1 runs of the model. Autograd needs one forward run and one backward run, whatever N is. Slide N below. Then slide the step h and watch a second problem: rounding.

Nudging against recording

Left: runs of the model needed for one full gradient. Right: the error of one finite difference as you shrink the step h. Too big a step bends; too small a step drowns in rounding.

10⁶
10⁻³

The error curves are computed live for L(w) = tanh(1.5w − 0.4) at w = 0.8: float64 in JavaScript's own numbers, float32 by rounding every step with Math.fround. Run times assume 10 ms per forward pass and a backward pass of about 2 forward passes (Chapter 10 explains the 2); they are illustrative.

Two lessons hide in the right panel. First, nudging is never exact: a large h measures a chord, not a tangent, and a tiny h subtracts two nearly equal numbers. Second, the best step depends on the number format. In float32 the error never gets below about 10−4. That is why PyTorch's own gradient checker (Chapter 8) works in float64.

The road map of this lesson

  1. The chain rule, one node at a time: local vector-Jacobian products.
  2. The graph is recorded as you run: grad_fn and next_functions.
  3. Leaves: why only leaves get .grad, and retain_grad().
  4. What backward() leaves behind: gradients add up; the graph is freed.
  5. Switching autograd off: no_grad, inference_mode, detach, requires_grad_(False).
  6. In-place ops and the version counter: the most famous autograd error.
  7. Look inside backward: hooks and anomaly mode.
  8. Write your own backward: autograd.Function, a straight-through estimator, a stable function, gradcheck.
  9. Gradients of gradients: create_graph, torch.autograd.grad, a gradient penalty.
  10. Jacobians, and why reverse mode wins: VJP against JVP.
  11. Connections: a bug finder, the cheat sheet, limits and next steps.

PyTorch, Zero to Expert

This is lesson 2 of an eleven-part series inside The Builder's Toolkit. Each lesson has Code Labs that run a copy of the PyTorch API in your browser.

  1. PyTorch Tensors: storage, strides and views.
  2. Autograd (this lesson): the graph behind backward().
  3. nn.Module Inside Out.
  4. Datasets and DataLoaders.
  5. The Training Loop, Line by Line.
  6. Debugging Training.
  7. Memory and Mixed Precision.
  8. Making PyTorch Fast.
  9. Distributed PyTorch: DDP and FSDP.
  10. PyTorch Internals.
  11. Build PyTorch from Scratch.
A model has 10 million weights. How many runs of the model does a one-sided finite-difference gradient need?

Chapter 1

The chain rule, one node at a time

Every operation knows only its own local derivative; backward multiplies a vector by it and passes the result on

Take the hero's first expression with x = 1.5, w = 0.8, b = −0.4. Forward, it runs in two steps: u = w·x + b = 0.8, then L = tanh(u) = 0.664.

The chain rule says: the slope of L with respect to w is the slope of L with respect to u, times the slope of u with respect to w. Each factor belongs to one step. The tanh step knows its own slope, 1 − tanh²(u). The multiply step knows its slope with respect to w, which is x. Neither needs to know about the other.

That is the design of autograd. Every operation carries a small backward rule. Backward starts with a 1 at the loss, then each node multiplies what arrives by its local slope and hands the product to its inputs.

From numbers to vectors

Real tensors hold many numbers. An operation y = f(x) with n inputs and m outputs has m × n local slopes. Laid out as a table, they form the Jacobian J: row i, column j holds ∂yi/∂xj.

Write the gradient of the loss with respect to any tensor with a bar on top: ȳ is ∂L/∂y, with the same shape as y. Then the chain rule for one node is one product:

ȳ
The gradient arriving at the node's output: m numbers, one per output.
J
The node's local Jacobian, m × n. Autograd almost never builds it.
x̄
The gradient the node sends to its input: n numbers. This product is a vector-Jacobian product, or VJP.

Worked (hero, first expression): L̄ = 1 → ū = 1 × (1 − 0.664²) = 0.559 → w̄ = ū × x = 0.559 × 1.5 = 0.839, b̄ = ū × 1 = 0.559.

Why never build J? For tanh on a million numbers, J is a million by a million: a trillion numbers, almost all zero. Each output depends only on its own input, so J is diagonal. The VJP is then just ȳ times (1 − y²), element by element. Every PyTorch operation ships a rule like this. Compare the two below.

The Jacobian you never build

Pick an operation and a size. The grid is the full Jacobian. The warm row under it is ȳTJ, computed two ways: by the full table, and by the operation's VJP rule. They always agree; only the cost differs.

6

Inputs x and the upstream gradient ȳ are fixed random numbers. Cell shade shows the size of each entry; the sign is printed when there is room. "Numbers stored" counts float32 values; the 106-input row is the same operation at a real activation size.

The rules you will meet most

ForwardBackward (the VJP)What it saves
y = a + bā = ȳ, b̄ = ȳnothing
y = a * bā = ȳ · b, b̄ = ȳ · aboth inputs
y = x.tanh()x̄ = ȳ · (1 − y²)its output y
Y = X @ WX̄ = Ȳ WT, W̄ = XT ȲX and W
y = x.sum()x̄ = ȳ copied to every placeonly the shape
y = x.mean()x̄ = ȳ / n in every placeonly the shape
Y = X + b (b broadcast over rows)b̄ = the sum of Ȳ over the rowsonly the shapes

Read the matmul row with shapes. If X is (32, 64) and W is (64, 10), then Y and Ȳ are (32, 10). W̄ = XTȲ is (64, 32) times (32, 10) = (64, 10): the same shape as W, as every gradient must be. The broadcast row is the same idea: the bias was copied to 32 rows, so its gradient is summed back over 32 rows.

Y = X @ W with X of shape (32, 64) and W of shape (64, 10). What is W̄, the gradient of the loss with respect to W?

Chapter 2

The graph is recorded as you run

Each tracked operation leaves a node behind; grad_fn points at it and next_functions links it to its inputs

PyTorch has no separate "build the graph" step. The graph is recorded while your Python code runs. This style is called define-by-run. Each time an operation runs on a tensor that requires a gradient, PyTorch creates a backward node for it.

Three attributes make the graph visible.

Step through the program below one line at a time and watch the graph grow.

Watch the graph being recorded

Move the line slider. Each line that operates on a tensor with requires_grad=True adds one purple node. Below the picture, the readout prints what PyTorch itself prints for that line.

7

Names and the structure of next_functions match PyTorch 2.10, which prints the leaf node as torch::autograd::AccumulateGrad. Object addresses are left out.

When one tensor is used twice

Look at the hero's third expression, w·x + sin(w). The weight w feeds two nodes, so two gradients flow back to it. The engine adds them before it calls w's node. In general, the gradient of a tensor is the sum over all the places that used it:

k
One use of x: one node that took x as an input.
JkT ȳk
The VJP that use sends back to x.
x̄
The total gradient of the loss with respect to x.

Worked: L = w·x + sin(w) with w = 0.8, x = 1.5. From the multiply: 1.5. From the sine: cos(0.8) = 0.697. Total w̄ = 2.197. Try it in the hero.

This is also why the engine runs nodes in dependency order. It counts how many gradients each node still waits for. A node runs only when all of them have arrived and been added. Then it runs once, with the total.

Control flow is just Python. An if or a loop in your forward pass is not part of the graph. Only the operations that actually ran are recorded. A model with an if records a different graph on each branch. Each forward pass makes a fresh graph.

Now walk the graph yourself, with real next_functions, the same way the engine does.

x = torch.randn(3) (no gradient) and w = torch.randn(3, requires_grad=True). Then y = (w * x).sum(). What is y.grad_fn.next_functions[0][0].next_functions?

Chapter 3

Leaves, and why only leaves get .grad

Which tensors are leaves, why the gradients of everything else are thrown away, and the one-character bug that makes .grad None

A leaf is a tensor with no grad_fn: nobody made it with a tracked operation. You created it. Weights, biases and input data are leaves. Everything computed from a tensor that requires a gradient is a non-leaf.

Whether a result needs a gradient follows one rule:

inp.requires_grad
True if that input wants a gradient (it is a weight, or was computed from one).
out.requires_grad
True if any input wants one. Then the output gets a grad_fn and is a non-leaf.

Worked: x (False) × w (True) → True, with grad_fn=<MulBackward0>.   x (False) × 2 → False: no node is recorded at all.

After loss.backward(), only leaves with requires_grad=True get their .grad filled in. The gradients of non-leaves are computed on the way, used, and dropped. Keeping them would cost as much memory as all the activations again. If you want one, call h.retain_grad() before backward.

Here is the trap. A line that looks like "make a weight" can make a non-leaf. Try each line below.

Leaf or not?

Pick a line. The picture shows where the tensor sits in the graph. The readout gives is_leaf, grad_fn and what .grad holds after loss.backward(). Predict before you look.

Every row was run on PyTorch 2.10. ".to(float64)" stands in for ".cuda()" or ".to('cuda')", which behave the same way on a GPU machine: the copy is a non-leaf made by ToCopyBackward0.

The fix for rows 2 and 3 is to finish building the tensor first, and only then ask for gradients: (torch.randn(3) * 0.01).requires_grad_(), or create it on the right device directly with torch.randn(3, device="cuda", requires_grad=True). Inside a model, use nn.Parameter, which is always a leaf.

You write w = torch.randn(3, requires_grad=True).cuda() and then torch.optim.SGD([w], lr=0.1). What happens?

Chapter 4

What backward() leaves behind

Gradients are added into .grad, never written over it; and the graph's saved tensors are freed as backward goes

Run the hero's backward on fresh graphs twice without resetting, and w.grad doubles. That is not a bug. The AccumulateGrad node adds:

w.grad
The leaf's gradient buffer. It starts as None and is created on the first backward.
∂L/∂w
The gradient from this one backward call.

Worked (hero, first expression): first backward: w.grad = 0.839. A second forward and backward without a reset: 0.839 + 0.839 = 1.677.

Why add instead of write? Because sometimes you want the sum. The second main use is gradient accumulation: your batch of 64 does not fit in memory, so you run 4 micro-batches of 16 and call backward() on each. The four gradients add up in .grad, and you step once.

For the sum to equal the full-batch gradient, scale each micro-batch loss. If the full loss is a mean over N examples, split into k equal chunks:

∇ meanN(ℓ)
The gradient of the full-batch loss: the mean of the per-example losses ℓ.
∇ meanchunk(ℓ)
The gradient of one micro-batch's mean loss, from one backward call.
1/k
The scale you must apply yourself: (loss / k).backward().

Worked: N = 64, k = 4 chunks of 16. Each chunk mean weighs each example by 1/16. Times 1/4 gives 1/64: exactly the full-batch weight.

The flip side: in an ordinary loop you must clear .grad every step with optimizer.zero_grad(). Forget it, and each step uses the sum of all past gradients. Try every mode below.

Gradients add up

One weight w learns the slope of y = 3x by SGD (learning rate 0.9). Pick a loop and press Train. Top: w over the steps (the target 3 is dashed). Bottom: what .grad held when the optimizer stepped.

Loss: mean of (w·x − y)² over 64 points with x evenly spread on [−1, 1] and y = 3x, start w = 0. Every number is computed in your browser; nothing is drawn by hand.

optimizer.zero_grad() in PyTorch 2.x has set_to_none=True by default. It sets each .grad to None instead of filling it with zeros. That frees the memory until the next backward, and the next backward then writes instead of adding. Code that reads p.grad right after zero_grad() must expect None.

The graph is freed as backward runs

Backward has a second side effect. Many nodes saved tensors for their backward rule: the tanh node saved its output, the multiply node saved its inputs. These saved tensors are the activations that make training use so much memory. As backward passes each node, it frees that node's saved tensors. When backward ends, the graph is an empty shell.

So a second backward() through the same graph fails with: "Trying to backward through the graph a second time (or directly access saved tensors after they have already been freed)". backward(retain_graph=True) keeps the saved tensors, at the cost of keeping their memory.

Saved tensors fill memory, then backward frees them

Press forward, then backward, then backward again. Turn on retain_graph and repeat. The bar on the right is memory held by saved tensors.

A 3-layer MLP at batch 256 and width 4,096 in float32: every saved activation is 256 × 4,096 × 4 bytes = 4 MiB. Linear layers (AddmmBackward0) save their input; for layers 2 and 3 that is the ReLU output the ReLU node already saved, so it is one buffer, counted once. The MSE loss saves its input. Weights are referenced, not copied, so they are not counted. Sizes are illustrative.

The classic way to hit this error is a recurrent model trained on a long stream. You carry the hidden state from one batch into the next. Its history reaches into the previous batch's graph, which backward already freed. The fix is not retain_graph=True. The fix is h = h.detach() at each batch boundary: keep the value, cut the history.

You accumulate over 8 micro-batches with plain SGD and forget to divide each loss by 8. What happens?

Chapter 5

Switching autograd off

no_grad, inference_mode, detach and requires_grad_(False) all stop gradients, in four different places

Recording has a cost. Every node saves tensors, and those tensors hold memory until backward. When you only evaluate, or serve a model, you want no recording at all. When you train only part of a model, you want recording for some tensors and not others. PyTorch has four tools, and they are not the same.

How much memory does recording hold? Roughly the saved activations:

L
The number of layers.
s
Saved tensors of that size per layer. A transformer block saves several.
B·T·D
Batch size × sequence length × feature size: the numbers in one activation.
bytes
Bytes per number: 4 for float32, 2 for bfloat16.
Msaved
Memory held until backward. Under no_grad it is close to zero.

Worked (illustrative): L = 24, s = 8, B·T·D = 8 × 1,024 × 1,024, bfloat16 → 24 × 8 × 8.4 M × 2 B = 3.2 GB that an evaluation loop under no_grad never allocates.

Four off switches, four different cuts

h comes from an earlier trainable layer, so it requires grad. W is this layer's weight. Pick a way to run loss = tanh(W @ h).sum(), then press backward. Grey dashed edges were not recorded.

The results (which .grad is set, the error text) are what PyTorch 2.10 does for this program. With h a 3-vector and W a 3 × 3 matrix, the matmul node is MvBackward0.

ToolScopeRecords a graph?Output usable in autograd later?Typical use
torch.no_grad()a block of codenoyes, as a constantevaluation loops; the optimizer's own update
torch.inference_mode()a block of codeno, and no version trackingno: inference tensors cannot be saved for backwardserving; batch prediction
t.detach()one tensoryes, for everything elseyes, as a constant that shares t's memorytargets, logging, carrying RNN state, stop-gradient tricks
p.requires_grad_(False)one leaf, until you switch it backyes, if other inputs require gradyesfreezing a pretrained backbone

Two more you will see. torch.set_grad_enabled(flag) is no_grad with an on-off switch, handy in one function that serves both training and evaluation. And all of these work as decorators: @torch.no_grad() above a function turns recording off for its whole body.

A frozen layer (its weight has requires_grad_(False)) sits between two trainable layers. Does the first trainable layer still get gradients?

Chapter 6

In-place ops and the version counter

Why "modified by an inplace operation" happens, how PyTorch detects it, and the one bypass that gives silently wrong gradients

An in-place operation changes a tensor's numbers in its existing memory instead of making a new tensor. Its name ends in an underscore: add_, mul_, clamp_. Indexed assignment such as x[0] = 5 and x += 1 are in place too.

Now remember Chapter 4: nodes save tensors for backward. The sigmoid node saves its own output, because its derivative is s(1 − s). If you then change that output in place, backward would use the wrong numbers.

PyTorch catches this with a version counter. Every tensor's storage has one. Each in-place change adds 1. When a node saves a tensor, it also writes down the version. At backward time, it checks:

saved._version
The version written down when the node saved the tensor in the forward pass.
tensor._version
The version now: 0 plus one for every in-place change since it was made.
RuntimeError
"one of the variables needed for gradient computation has been modified by an inplace operation".

Worked: y = sigmoid(z) is saved at version 0. y.mul_(2) makes it version 1. Backward compares 1 with 0 and stops: "…is at version 1; expected version 0 instead."

The version counter at work

The program is y = torch.sigmoid(z) with z = (−1, 0.5, 2), then one change to y, then y.sum().backward(). Pick the change, then press backward.

All gradients and the error text were checked on PyTorch 2.10. The .data case really does give z.grad = 2s(1 − 2s), not the correct 2s(1 − s).

Which in-place ops are safe?

It depends only on what the earlier nodes saved. Addition saves nothing, so y = x + 1; y.add_(1) is fine. Multiplication saves both inputs. Exp, tanh, sigmoid and ReLU save their output. So changing the output of a sigmoid in place fails, and changing the output of an addition does not. nn.ReLU(inplace=True) works because ReLU's backward needs only where its output is positive, which its own in-place write keeps.

Leaves have their own rule. w.add_(1) on a weight that requires grad fails with "a leaf Variable that requires grad is being used in an in-place operation." Recording a change to a leaf would make the leaf a non-leaf. The way to update weights is to switch recording off first, which is what every optimizer does inside step():

with torch.no_grad():
    w -= lr * w.grad      # in place, not recorded: w stays a leaf

Now trigger the error yourself, read it, and fix it.

x requires grad. Which of these runs backward without an in-place error?

Chapter 7

Look inside backward: hooks and anomaly mode

Read or change a gradient while backward runs, and find the forward line that made a NaN

Backward runs in C++, out of sight. Two tools let you see into it.

A tensor hook is a function you attach with t.register_hook(fn). When backward computes the gradient for t, it calls fn(grad). If fn returns None, the gradient goes on unchanged. If fn returns a tensor, that tensor replaces the gradient for everything upstream. register_hook returns a handle; call handle.remove() to take the hook off.

Uses: log the gradient norm at each layer to find where gradients vanish; clip one tensor's gradient; flip a gradient's sign, as in the gradient reversal layer used for domain adaptation. A common clipping rule:

g
The gradient the hook receives.
c
The largest norm you allow.
g′
What the hook returns: g itself if it is short enough, else g shrunk to length c, same direction.

Worked: g = (3, 4), ‖g‖ = 5, c = 1 → g′ = (3, 4) × 0.2 = (0.6, 0.8), length 1.

A hook in the middle of backward

Six layers hk+1 = tanh(a·hk). Bars show the size of the gradient at each layer, loss on the right. Put a hook on layer 3. Then inject a NaN and compare backward with and without anomaly mode.

1.60
0.20

Input h0 = 0.3. Gradients are computed exactly in your browser. The NaN case adds sqrt(0·h) into layer 2: the forward value is fine (0), but the slope of sqrt at 0 is infinite, and infinity times 0 is NaN. The anomaly message is PyTorch 2.10's text for this case.

Hooks on leaves and on modules

A hook on a weight runs before the gradient is added into .grad, so what it returns is what gets added. To act after .grad is final, use p.register_post_accumulate_grad_hook(fn): fn receives the parameter itself. Some memory-saving training loops call the optimizer step for each parameter there, then free its gradient at once.

On a module, register_full_backward_hook(fn) calls fn(module, grad_input, grad_output): the gradients with respect to the module's inputs and outputs. Not its weights. For weights, hook the parameters.

Anomaly mode: who made the NaN?

A NaN in a gradient tells you nothing about where it came from. with torch.autograd.detect_anomaly(): changes two things. In forward, every node records the Python stack that created it. In backward, every node's output is checked for NaN. The first bad node raises an error that names it, for example "Function 'SqrtBackward0' returned nan values in its 0th output.", and prints the forward line that made it. Anomaly mode is slow. Turn it on to find a bug, then off.

A hook on a hidden activation h returns g * 0. Which weights get a zero gradient?

Chapter 8

Write your own backward: autograd.Function

Give an operation a backward rule you choose: a straight-through estimator, a numerically stable formula, and the gradient checker that proves you got it right

Sometimes the automatic rule is not the rule you want. Four common reasons:

A custom Function is a class with two static methods:

import torch

class Cube(torch.autograd.Function):
    @staticmethod
    def forward(ctx, x):              # x: any shape
        ctx.save_for_backward(x)          # keep what backward needs (version-checked)
        return x ** 3

    @staticmethod
    def backward(ctx, g):             # g: dL/d(output), same shape as the output
        x, = ctx.saved_tensors
        return g * 3 * x ** 2          # one gradient per forward input

x = torch.tensor([2.0], requires_grad=True)
Cube.apply(x).sum().backward()            # call it with .apply, never Cube()(x)
print(x.grad)                             # tensor([12.])

Rules worth knowing by heart. backward returns one value per input of forward, in order; return None for inputs that are not tensors or need no gradient. ctx.needs_input_grad tells you which inputs want one, so you can skip work. Save tensors with ctx.save_for_backward, not as ctx.x = x: the official path checks the version counter and frees memory on time.

The straight-through estimator

A quantized network rounds its weights to a few levels in forward. The rounding step has slope 0, so the full-precision weights behind it would never move. The straight-through estimator (STE) keeps the rounding in forward and pretends, in backward, that it was the identity: backward returns g unchanged. Bengio, Léonard and Courville name it and credit the idea to a 2012 lecture by Hinton. They call it "clearly a biased estimator" Bengio 2013 · §4. Biased, and it works. Try it.

Learning through a staircase

The loss is (q(w) − 0.73)², where q rounds w to the nearest level. Pick the backward rule and press Train. Left: the staircase q(w) and the weight. Right: the loss over the steps.

0.25

Start w = −0.6, plain gradient descent with learning rate 0.1. The target 0.73 lies between levels on purpose: the best q can do is the nearest level. Numbers are computed live.

A numerically stable function

torch.log(torch.sigmoid(x)) is correct maths. But for x below about −88, the float32 sigmoid on a CPU rounds to 0 and the log returns −inf. The automatic backward then produces NaN. Rewrite it so no step can overflow:

log σ(x)
The log of the sigmoid. The exponent −|x| is never positive, so e−|x| is at most 1 and never overflows.
d/dx
Its slope, 1 − σ(x), written as σ(−x) so no subtraction of near-equal numbers happens.

Worked: x = −200: min = −200, log1p(e−200) ≈ 10−87 → −200, slope σ(200) = 1. The naive form gives −inf and a slope of 0.

gradcheck: proof that a backward is right

A hand-written backward is a hand-written bug waiting to happen. torch.autograd.gradcheck(fn, inputs) builds the Jacobian twice: once from your backward, once by nudging each input with a central difference. It fails if any entry differs by more than atol + rtol × |numerical|. Its defaults are eps = 10−6, atol = 10−5, rtol = 10−3.

ε
The nudge, 10−6 by default. The two-sided form has error about ε² instead of ε.
∂f/∂xj
Column j of the numerical Jacobian, compared entry by entry with your backward's.

Worked: f = x³ at x = 2: exact 12. Central difference error from the step: about ε² = 10−12. From float64 rounding: about 10−16/10−6 = 10−10. Both far below atol. In float32 the rounding term would be 10−7/10−6 = 0.1, which is why gradcheck wants float64 inputs.

gradgradcheck does the same for the gradient of your gradient (Chapter 9). Run gradcheck on small, random, float64 inputs, away from kinks such as 0 for abs.

gradcheck fails for your straight-through clamp at x = 3, outside the range [−1, 1]. Is your backward wrong?

Chapter 9

Gradients of gradients

torch.autograd.grad against backward(), create_graph=True, and a loss with a gradient inside it

So far every gradient landed in .grad. torch.autograd.grad(outputs, inputs) is the other door. It returns the gradients as a tuple and does not touch .grad at all.

loss.backward()torch.autograd.grad(loss, xs)
Result goes toadded into .grad of every leafreturned as a tuple, one per input in xs
Which tensorsall leaves that require grad (or inputs=)exactly xs, leaf or not
Unused inputits .grad stays as it waserror, unless allow_unused=True (then None)
Typical usea training stepa gradient inside a loss; per-input gradients; meta-learning

A gradient is a tensor computed by the backward nodes. Normally those computations are not recorded, so the result has no history. Pass create_graph=True and backward records itself. Then the gradient has a grad_fn, and you can differentiate it again.

f
The forward value: y = x**3 - 2*x.
f′
g, = torch.autograd.grad(y, x, create_graph=True). g has grad_fn=<AddBackward0>.
f″
torch.autograd.grad(g, x): the gradient of the gradient.

Worked, x = 2: f = 8 − 4 = 4, f′ = 12 − 2 = 10, f″ = 12. Without create_graph, the second call fails: "element 0 of tensors does not require grad and does not have a grad_fn".

Differentiate twice

Drag x. The blue curve is f, the warm curve f′ (the first backward), the dashed warm curve f″ (the second). Turn create_graph off and the second backward has nothing to differentiate.

2.00

Values are computed from the formulas; on PyTorch 2.10 the two autograd.grad calls return the same numbers.

A loss with a gradient inside: the gradient penalty

WGAN-GP trains a critic D that scores real and fake samples. The theory wants D's slope with respect to its input to have length 1. So the loss adds a gradient penalty on points mixed between real and fake samples Gulrajani 2017 · §4:

x̂
A mixed point: ε·xreal + (1 − ε)·xfake, with ε uniform on [0, 1], one per example.
∇x̂ D(x̂)
The critic's slope with respect to its input: torch.autograd.grad(D(x_hat).sum(), x_hat, create_graph=True), shape (B, features).
λ
The penalty weight. The paper uses λ = 10 in all its experiments Algorithm 1; §4.

Worked: one slope g = (0.6, 1.2) → ‖g‖ = 1.342 → (0.342)² = 0.117 → × 10 = 1.17.

The .sum() is a trick worth understanding. Each example's score depends only on its own input, so the gradient of the sum with respect to x̂ gives every example its own slope, in one backward pass. The penalty then needs the gradient of those slopes with respect to the weights: a second backward through the first. Without create_graph=True the penalty is a constant and trains nothing, with no error at all.

Which call leaves w.grad exactly as it was?

Chapter 10

Jacobians, and why reverse mode wins for a loss

One reverse pass gives one row of the Jacobian; one forward-mode pass gives one column; a loss has exactly one row

Take any function f from n inputs to m outputs. Its full Jacobian is an m × n table. There are two ways to fill it.

n
The number of inputs. For training, every weight is an input: n can be 109.
m
The number of outputs. For a loss, m = 1.
passesreverse
Each costs a small multiple of one forward pass, and needs the saved activations.
passesforward
Each costs a small multiple of one forward pass, and saves nothing.

Worked: a loss with n = 109 weights, m = 1: reverse mode 1 pass; forward mode 109 passes. A simulator with 3 knobs and 10,000 outputs: forward mode 3 passes; reverse mode 10,000.

Why is one pass "a small multiple" of a forward pass? Look at a matmul Y = XW. Forward is one matmul. Backward needs X̄ = ȲWT and W̄ = XTȲ: two matmuls of the same size. So a backward pass costs about two forward passes, and a training step about three. That is the 2 in Chapter 0.

Rows or columns?

Set n and m, pick a mode and press Fill. Each pass fills one row (reverse) or one column (forward). Set m = 1, the shape of a loss, and compare.

12
1

Pass counts are exact. Real cost per pass depends on the model; the cost bars use 3 forward-pass units for a reverse pass (forward plus a backward of about 2) and about 3 for a forward-mode pass, as an illustration.

The tools

torch.autograd.functional has jacobian(f, x), hessian(f, x), vjp(f, x, v), jvp(f, x, u) and hvp. torch.func has the composable versions: jacrev (rows by reverse mode), jacfwd (columns by forward mode), vmap to run many passes as one batched pass, and hessian, which is jacfwd(jacrev(f)): forward over reverse.

A Hessian of a loss is n × n. For n = 109 that is 1018 numbers: impossible to store. But a Hessian-vector product H u costs about one gradient plus one more pass. Second-order methods and curvature studies use only those.

Run this on a machine with PyTorch

import torch
from torch.func import jvp, jacrev, jacfwd

A = torch.randn(2, 6)
f = lambda x: torch.tanh(A @ x)          # R^6 -> R^2
x = torch.randn(6)

J1 = jacrev(f)(x)                         # (2, 6): 2 reverse passes, one per row
J2 = jacfwd(f)(x)                         # (2, 6): 6 forward passes, one per column
out, col0 = jvp(f, (x,), (torch.eye(6)[0],))   # true forward mode
print(torch.allclose(J1, J2), torch.allclose(col0, J1[:, 0]))   # True True

Forward mode is not in the browser's mini-torch, so the lab below uses torch.autograd.functional.jvp, which builds a JVP out of two reverse passes. The count of passes is what the lab is about, and it does not change.

f takes 3 numbers and returns 10,000. You want its full Jacobian. Which mode needs fewer passes?

Chapter 11

Connections

A bug finder for the errors you will actually see, the cheat sheet, honest limits, and where to go next

Almost every autograd bug shows up as one of eight symptoms. Pick the one you have. The card gives the cause, the fix, and the chapter that explains it.

Autograd bug finder

Pick a symptom. The card shows the usual cause, the one-line fix and the chapter to reread.

Error texts are PyTorch 2.10's.

The cheat sheet

IdeaIn one line
The mechanismForward records one node per op; backward walks them in reverse, each computing x̄ = JTȳ.
See the grapht.grad_fn, node.next_functions, node.name(); leaves have AccumulateGrad.
LeavesNo grad_fn. Only leaves with requires_grad get .grad; retain_grad() for others.
Accumulation.grad is added to: zero_grad() every step; (loss / k).backward() to accumulate.
FreeingBackward frees saved tensors; retain_graph=True keeps them; detach carried state.
Off switchesno_grad (block), inference_mode (faster, final), detach (one tensor), requires_grad_(False) (freeze a leaf).
In placeVersion counter must match what was saved; update weights under no_grad; never .data.
Hookst.register_hook(fn): read or replace a gradient; detect_anomaly() finds the first NaN node.
Custom opautograd.Function: forward(ctx, ...), backward(ctx, g) → one grad per input; prove it with gradcheck in float64.
Higher ordertorch.autograd.grad(..., create_graph=True) makes a gradient you can differentiate again.
CostFull Jacobian: m reverse passes or n forward passes. A loss has m = 1, so one reverse pass.

Honest limits

Where this sits

Reverse-mode differentiation is older than deep learning: Seppo Linnainmaa described it in 1970, and Rumelhart, Hinton and Williams made it the training method for neural networks in 1986 as backpropagation. PyTorch's contribution was to make it define-by-run: the graph is your ordinary Python program, recorded as it runs Paszke 2019.

Where to go next on this site:

The takeaway. Autograd records one node per operation as your code runs. Backward walks the nodes in reverse, each multiplying the incoming gradient by its own local derivative, and adds the results into the leaves' .grad. Almost every autograd surprise is one of those words taken literally: recorded, reverse, local, adds, leaves.
Each training step, memory grows a little until the GPU runs out. The loop has running = running + loss for logging. What is the fix?

Now press Present or Teach and explain, out loud and from memory, what happens between loss.backward() and w.grad. Then go back to the hero, pick w·x + sin(w), and predict w.grad before you press backward.

Autograd: the Graph Behind backward(). Lesson 2 of PyTorch, Zero to Expert (The Builder's Toolkit). API behaviour, node names, defaults and error texts were checked against PyTorch 2.10 (Autograd mechanics, torch.autograd). Papers: A. Paszke et al., "PyTorch: An Imperative Style, High-Performance Deep Learning Library" (NeurIPS 2019, arXiv); A. G. Baydin, B. A. Pearlmutter, A. A. Radul, J. M. Siskind, "Automatic Differentiation in Machine Learning: a Survey" (JMLR 2018, arXiv); Y. Bengio, N. Léonard, A. Courville, "Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation" (2013, arXiv), §4 for the straight-through estimator; I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, A. Courville, "Improved Training of Wasserstein GANs" (NeurIPS 2017, arXiv), Algorithm 1 and §4 for the gradient penalty and λ = 10.
Back to Gleams