Making PyTorch Fast

On an NVIDIA A100, adding two bfloat16 tensors can reach at most 0.34 trillion operations per second: about 0.1% of the chip's 312. A large matrix multiply can reach all 312. The GPU is the same. What differs is how much maths each byte of memory buys.

Learn to measure speed honestly, to see where the time goes (Python overhead, memory traffic or arithmetic), and to fix each kind: fusion, torch.compile, CUDA graphs, bigger batches, tensor-core shapes and fewer syncs.

Pick an operation and read where it sits under the roof. Drag the intensity. Then pick the 5-op chain and turn on fusion: five round trips to memory become one, and the dot moves right.

You need memory and mixed precision (part 7) and a working training loop. We build everything else from zero.

Under the roof

Ready
0.17

Roofs from NVIDIA's A100 datasheet (A100 80GB SXM, Jun 2021): FP32 19.5 TFLOP/s, BF16 Tensor Core 312 TFLOP/s dense, memory bandwidth 2,039 GB/s. The roofline itself is Williams, Waterman and Patterson (2009). Elementwise ops act on 226 elements. LayerNorm counts about 8 FLOPs per element, and the chain counts tanh as 1 FLOP: rough counts. The float32 matmul uses the 19.5 TFLOP/s roof (TF32 off). Real kernels reach less than the roof.

Chapter 0

The step is slow. Why?

Before you change anything, find out which of three things limits the step: the CPU's overhead, the memory's bandwidth, or the arithmetic

You rent a GPU that can do 312 trillion operations per second. Your model's step takes 40 ms. You count its maths and get 1 trillion operations. At full speed that is 3 ms. So where did the other 37 ms go?

The answer is almost never "the GPU is slow". The GPU spends its time in one of three ways, and each needs a different fix. Guess the wrong one and you can work for a week with no gain.

A kernel is one function that runs on the GPU, such as "add these two tensors". Each PyTorch operation in eager mode launches one or more kernels. The CPU queues them and moves on; the GPU runs them in order.

c
The CPU's cost to issue the operation: Python, dispatch and the kernel launch. A few microseconds.
F
The number of floating-point operations (FLOPs) the kernel does. One multiply is 1 FLOP, one add is 1 FLOP.
P
The GPU's peak compute in FLOP per second: 312 × 1012 for bfloat16 on an A100.
Q
The bytes the kernel reads from and writes to GPU memory.
BW
The memory bandwidth in bytes per second: 2.039 × 1012 on an A100 80GB SXM.

Worked, a GELU on 256 × 4096 bfloat16 numbers: F ≈ 107 (about 10 per element) → 0.03 µs; Q = 4 MB → 2.1 µs; c = 10 µs (illustrative). The max is c: this op is overhead-bound. The GPU works 2 µs and waits 8.

The max is a simplification. The CPU's work and the GPU's work overlap, because the CPU runs ahead. The device below simulates that overlap for one block of a transformer layer: a LayerNorm, two matrix multiplies, a GELU and a residual add.

Three ways to be slow

Slide the tokens per step and pick a width. The top lane is the CPU issuing kernels; the bottom lane is the GPU running them. Red is the GPU waiting.

256

An ideal roofline model of an A100 in bfloat16: each kernel takes max(F/P, Q/BW) with the datasheet's P and BW. The CPU cost c = 10 µs per kernel is an illustrative round number. Kernels: LayerNorm, matmul d → 4d (bias fused), GELU, matmul 4d → d, residual add. Real kernels are slower than ideal, so real crossovers sit at larger sizes.

Slide the tokens down to 16. Every kernel is tiny, so the GPU lane is mostly red. Slide up to 64k tokens at width 4096 and the matrix multiplies fill the lane in purple: compute-bound. In between, at width 256, the blue memory-bound kernels take a large share.

The road map of this lesson

  1. Measure first: warm-up, synchronize, the median, and torch.utils.benchmark. Did it work?
  2. The overhead floor: why a tiny op costs as much as a big one. Why it works
  3. Memory or compute?: arithmetic intensity and the roofline. Why it works
  4. Fusion: why chains of elementwise ops are slow, and the fix. Build it
  5. torch.compile, part 1: Dynamo, guards and graph breaks. Build it
  6. torch.compile, part 2: Inductor, the modes, CUDA graphs, and when not to bother. When not
  7. Hidden syncs: .item(), printing and boolean masks. Build it
  8. Feed it and batch it: the input pipeline, batch size and convergence. Where else
  9. Tensor cores and shapes: precision, multiples of 8, channels_last. Build it
  10. Read the profiler: self time, total time, and the trace. Did it work?
  11. Connections: the order to try things, the cheat sheet, and the limits.

PyTorch, Zero to Expert

This is part 8 of an eleven-part series. Each lesson has Code Labs that run the PyTorch API in your browser.

  1. PyTorch Tensors: Storage, Strides and Views
  2. Autograd: the Graph Behind backward()
  3. nn.Module Inside Out
  4. Datasets and DataLoaders
  5. The Training Loop, Line by Line
  6. Debugging Training: When the Loss Won't Go Down
  7. Memory and Mixed Precision
  8. Making PyTorch Fast (this lesson)
  9. Distributed PyTorch: DDP and FSDP from the Inside
  10. PyTorch Internals: From a Python Call to a Kernel
  11. Build PyTorch from Scratch
A kernel does 1 million FLOPs and moves 4 MB. The CPU needs 10 µs to launch it. On an A100 (312 TFLOP/s, 2.039 TB/s), what limits it?

Chapter 1 Did it work?

Measure first, and measure right

CUDA runs asynchronously, first calls are slow, and timings are noisy. A correct timer warms up, synchronizes, repeats, and reports the median

Every speed-up claim is a pair of measurements. If the measurements are wrong, the claim is wrong. And the obvious timer is wrong on a GPU in three separate ways.

First, CUDA is asynchronous. When Python runs y = x @ w on a GPU tensor, PyTorch queues the kernel and returns at once. The multiply happens later. So time.perf_counter() around that line measures how long it took to queue the work: microseconds, not the milliseconds of the multiply. To measure the work, call torch.cuda.synchronize(), which waits until every queued kernel has finished.

Second, first calls are slow. The first call can include one-off work: creating the CUDA context, cuDNN trying several convolution algorithms when torch.backends.cudnn.benchmark = True, the memory allocator growing its cache, or torch.compile compiling. A warm-up runs the code a few times before you start timing.

Third, timings are noisy. Other processes, clock changes and the operating system add random delays. Most land on a few runs. The median (the middle value after sorting) ignores those few slow runs; the mean does not.

W
Warm-up calls that are run but not timed. Three to ten is common.
R
Timed repeats. More repeats give a steadier median.
tstart, tend
Clock reads, each taken right after torch.cuda.synchronize(), so the queue is empty when the clock starts and the work is done when it stops.
t̂
The estimate you report: the median time per call.

Worked, five timed runs in ms: 2.01, 1.98, 2.02, 2.80, 2.00. Mean = 2.16. Sorted: 1.98, 2.00, 2.01, 2.02, 2.80 → median = 2.01. One slow run moved the mean by 7% and the median by nothing.

Three ways a timer lies

The kernel truly takes 2.00 ms. Each bar is one timed call. Turn on synchronize and warm-up, then choose what to report. Only one setting gets close.

A simulation with illustrative numbers: queuing a kernel takes 0.02 ms of CPU time; the first call carries 180 ms of one-off set-up; each run has ±2% jitter, and runs 4 and 9 are hit by a background process (+40%).

Now the tool that does all of this for you. torch.utils.benchmark.Timer warms up, synchronizes, repeats, and reports the median. It times blocks of calls. Its docstring says why: "accelerator synchronization time is non-trivial (order single to low double digit microseconds)". Enough calls per block make that cost small.

Run this on a machine with PyTorch

import torch
from torch.utils import benchmark

dev = "cuda" if torch.cuda.is_available() else "cpu"
x = torch.randn(4096, 4096, device=dev)
w = torch.randn(4096, 4096, device=dev)

t = benchmark.Timer(stmt="x @ w", globals={"x": x, "w": w},
                    num_threads=torch.get_num_threads())   # default is 1 thread
m = t.blocked_autorange(min_run_time=1.0)   # warms up, syncs, repeats
print(m)                                         # median and IQR per call
print(f"{2 * 4096**3 / m.median / 1e12:.1f} TFLOP/s")

The last line turns a time into a rate. Rates are what you compare with the roofline in Chapter 3: a 4096 × 4096 matrix multiply is 2 × 40963 = 137 billion FLOPs.

You time y = model(x) on a GPU with time.perf_counter() and no synchronize. It reports 0.3 ms. What did you measure?

Chapter 2 Why it works

The overhead floor

Every eager operation pays a fixed CPU cost before the GPU sees it. Below a certain size, a bigger tensor is free and a smaller one saves nothing

Follow one line, y = a + b, from Python to the GPU. Python looks up __add__ and calls into PyTorch's C++ core. The dispatcher checks the dtypes and the device, works out whether autograd must record the op, and picks the CUDA kernel. The output tensor is allocated. Then the kernel is launched: put into the GPU's queue with its arguments.

Each of those steps is small, but together they cost microseconds, and they do not depend on the tensor's size. Call that fixed cost c. A 10-element add and a 1-million-element add pay the same c. The GPU's part, by contrast, grows with the size.

c
The fixed CPU cost per operation: Python, dispatch, allocation and launch.
12 n
Bytes moved by a float32 add of n elements: read two inputs and write one output, 4 bytes each.
BW
Memory bandwidth, 2.039 × 1012 bytes per second on an A100.
n*
The size where the GPU's work equals the overhead. Below it, the op costs c whatever its size.

Worked, c = 10 µs: n* = 10−5 × 2.039 × 1012 / 12 ≈ 1.7 million elements. An add of 4,096 elements needs 0.024 µs of memory time and pays 10 µs: the GPU is busy 0.24% of it.

The floor and the slope

Slide the tensor size. Left of the bend, the op costs the overhead (warm). Right of it, memory traffic (blue) sets the time. Then shrink the overhead, as CUDA graphs or compiled code do.

10,000
10

float32 add on an ideal A100 (2,039 GB/s). The overhead c is a free, illustrative parameter: on a real GPU it depends on the CPU, the Python version and the op. For scale, one a + 1.0 on a 1-element CPU tensor took 1.3 µs with PyTorch 2.10 on the laptop CPU used to write this lesson; a GPU launch adds more.

What this means for your code

A Python loop over rows, tokens or time steps turns one big op into thousands of small ones. Each pays c. The fix is vectorising: write the maths once for the whole batch, so one call does all the rows. Broadcasting and dim= arguments are how you say "for every row" without a loop. (Part 1, Tensors, covers broadcasting.)

When you cannot vectorise, because the work really is many small ops, two tools cut c itself. torch.compile fuses many small ops into a few kernels (Chapter 5). CUDA graphs record a whole sequence of launches once and replay it with one call (Chapter 6).

With c = 10 µs per op on an A100, which change makes a loop of 1,000 adds on 500-element tensors much faster?

Chapter 3 Why it works

Memory or compute? The roofline

Count the FLOPs and the bytes of a kernel. Their ratio, the arithmetic intensity, tells you which roof it hits and how fast it can ever go

Picture a kitchen. The cook (the arithmetic units) can chop very fast. But every ingredient must come from a store room down a long corridor (the memory). If a recipe needs one chop per ingredient, the cook spends the day waiting at the door. If it needs a thousand chops per ingredient, the corridor no longer matters.

The number that decides this is the arithmetic intensity: FLOPs done per byte moved between the GPU's memory and its chip. (Williams, Waterman and Patterson call it operational intensity.) Each GPU also has its own ratio: peak FLOP/s divided by bandwidth. Kernels below that ratio are memory-bound. Kernels above it are compute-bound.

F
FLOPs the kernel performs.
Q
Bytes it moves to and from GPU memory: inputs read once, outputs written once, in the ideal case.
I
Arithmetic intensity, in FLOP per byte.
P, BW
Peak compute and memory bandwidth from the datasheet. A100 80GB SXM: P = 312 TFLOP/s (BF16 Tensor Core) or 19.5 (FP32); BW = 2.039 TB/s.
BW
Memory bandwidth: the slope of the slanted roof.
attainable
The roofline: the fastest a kernel of intensity I can run, in FLOP/s.
ridge
The intensity where the two roofs meet. Williams and colleagues: it is "the minimum operational intensity required to achieve maximum performance".

Worked, A100: ridge = 312 / 2.039 = 153 FLOP/byte in bfloat16 (9.6 in FP32). A float32 add: 1 FLOP per 12 bytes, I = 0.083 → at most 0.083 × 2.039 = 0.17 TFLOP/s. A bfloat16 40963 matmul: I = 4096 / 3 = 1,365 > 153 → the full 312.

Counting a matrix multiply

An (M, K) @ (K, N) product makes M × N outputs. Each one is a dot product of length K: K multiplies and K adds. So F = 2MKN. It reads both inputs and writes the output: Q = s(MK + KN + MN) bytes, with s bytes per number. NVIDIA's matmul guide writes the same ratio for 2-byte types as MNK / (MK + NK + MN).

Two shapes show the range. A square N × N × N product in bfloat16 has I = N / 3: it grows with N, so big matmuls are compute-bound. A matrix-vector product (M = 1) has I close to 1 whatever its size: every weight is read once and used once. That is why generating text one token at a time is memory-bound.

Count a matmul

Pick a shape or set M, K and N. The boxes are the three matrices (side lengths on a log scale). Watch the dot cross the ridge as M grows from 1.

4096
4096
4096

Ideal counts: each input read once and the output written once, which tiled kernels approach for large shapes. Roofs from the A100 datasheet (FP32 19.5, BF16 312 TFLOP/s, 2,039 GB/s). Times are lower bounds: real kernels run below the roof.

The same counting works for every op. LayerNorm, softmax, GELU, dropout and residual adds all do a handful of FLOPs per element and so sit far left of the ridge: they are memory-bound on every modern GPU. Matrix multiplies and convolutions with large dimensions sit right of it. A transformer spends most of its FLOPs in the matmuls, but a surprising share of its time in the cheap ops, because each one makes a full trip through memory.

A bfloat16 matmul has M = 8 tokens, K = N = 4096. Its intensity is about 8. On an A100 (ridge 153), what bounds it?

Chapter 4 Build it

Fusion: one trip instead of five

Eager PyTorch runs each elementwise op as its own kernel, so a chain of them reads and writes memory again and again. A fused kernel reads once and writes once

Take a common line in a model: y = torch.tanh(torch.relu(x * w + b) * 0.5). In eager mode it runs five kernels: multiply, add, ReLU, multiply, tanh. Each kernel reads its inputs from GPU memory and writes its result back. The four results in the middle are written only to be read again by the next kernel.

Fusion merges the chain into one kernel. Each thread loads its elements of x, w and b once, does all five steps in its registers, and stores y once. The maths is the same. The memory traffic is a third.

Σk (ink + outk)
Full-tensor passes made by the separate kernels: for each kernel k, the tensors it reads plus the one it writes.
distinct inputs + outputs
Passes made by the fused kernel: each real input read once, each final output written once.
speed-up
Holds while both versions are memory-bound, which elementwise chains almost always are.

Worked, the chain above: eager = (2+1) + (2+1) + (1+1) + (1+1) + (1+1) = 12 passes. Fused = 3 inputs + 1 output = 4. Speed-up ≈ 3×. For 226 bfloat16 elements on an A100: 1.61 GB in 0.79 ms against 0.54 GB in 0.26 ms.

Fuse the chain

Turn ops of the chain on or off, then fuse. Blue arrows are trips to GPU memory (down = read, up = write). Purple chips are the maths, done on the chip.

226 bfloat16 elements per tensor (134 MB). Times = bytes / 2,039 GB/s, the ideal memory-bound time on an A100; launch overhead is left out.

What can fuse, and what cannot

You rarely write fused kernels by hand. Hand-written ones exist for the hottest patterns: F.scaled_dot_product_attention runs fused attention kernels such as FlashAttention, and the optimizers' fused=True option runs a whole AdamW step in one kernel. For everything else, torch.compile finds and fuses chains for you. The next two chapters show how.

A chain is ReLU, then × 2, then + 1, all on one tensor x (no other tensor inputs). How many memory passes does eager make, and fused?

Chapter 5 Build it

torch.compile, part 1: capturing the graph

TorchDynamo reads your function's Python bytecode as it runs, records the PyTorch ops into a graph, guards the assumptions it made, and breaks the graph where it cannot follow

To fuse ops, a compiler must see several of them at once. Eager PyTorch never does: it runs one op, returns to Python, runs the next. torch.compile solves this with two parts, described in the PyTorch 2 paper (Ansel and colleagues, ASPLOS 2024). TorchDynamo captures graphs. TorchInductor compiles them.

Dynamo works at the level of Python bytecode, the low-level instructions CPython runs. Python lets a program replace the function that evaluates each frame (each function call); this hook is PEP 523. Dynamo installs its own. When your compiled function is called, Dynamo walks its bytecode one instruction at a time. It does not run the tensor maths. It records every PyTorch op into an FX graph: a list of ops and the values that flow between them PyTorch 2 · §3.2, §3.4.

Recording makes assumptions: this argument is a tensor of shape (8, 64), this flag is True, this module has these layers. Dynamo writes each assumption down as a guard: a cheap check run on every later call. If the guards pass, the cached compiled code runs. If one fails, Dynamo compiles again. The paper counts "30 different types of guards" on tensor properties, Python types, constants, attributes and global state §3.3.

Run this on a machine with PyTorch

import torch

def f(x):
    y = torch.sin(x) * 2
    print("hi")                      # Python Dynamo cannot record
    if y.sum() > 0:                  # a branch on a tensor's VALUE
        return y.cos()
    return y + 1

ex = torch._dynamo.explain(f)(torch.randn(8))
print(ex.graph_count, ex.graph_break_count)   # 3 2
for b in ex.break_reasons:
    print(b.reason.splitlines()[0])
# Failed to trace builtin operator
# generic_jump TensorVariable()

That output is real, from PyTorch 2.10. When Dynamo meets something it cannot record, it makes a graph break. It compiles the graph so far, lets CPython run the awkward bytecode normally, and starts a new graph after it. The paper calls the generated pieces "continuation functions" §3.8. Your program stays correct, but each break splits a graph, and ops on either side cannot fuse.

breaks
Places where Dynamo handed control back to Python: explain(...).graph_break_count.
graphs
Compiled pieces per call: explain(...).graph_count. Fusion and other optimizations act inside one graph only.

Worked, the function above: a print and a branch on y.sum() > 0 → 2 breaks → 3 graphs, exactly what explain printed.

Where the graph breaks

Each button adds one line to the function. Coloured brackets are compiled graphs; red lines run in plain Python between them. The counts and reasons are what torch._dynamo.explain printed for each version.

Every combination was run through torch._dynamo.explain on PyTorch 2.10 (CPU); graph counts and the first line of each break reason are copied from its output.

Find your graph breaks

Two tools show them. torch._dynamo.explain(fn)(*args) returns the counts, the reasons and the user stack. The TORCH_LOGS environment variable logs breaks and recompiles while your real training runs:

Run this on a machine with PyTorch

# in the shell
TORCH_LOGS="graph_breaks,recompiles" python train.py

# what PyTorch 2.10 printed for the branch above (trimmed):
# Graph break in user code at train.py:5
# Graph Break Reason: Data-dependent branching
#   Hint: Use `torch.cond` to express dynamic control flow.
# and for a function g(x, k) called with k = 2, then k = 3:
# Recompiling function g ... triggered by the following guard failure(s):
#     - 2/0: k == 2

The second log shows a guard at work. The first call recorded k == 2 as a constant. The second call failed that guard and recompiled. By default Dynamo then treats the changed value as dynamic, so a third value reuses the second compile (checked on 2.10). Shapes behave the same way. Each function has a limit of 8 recompiles (torch._dynamo.config.recompile_limit); after that it falls back to eager.

The usual causes of breaks, in the paper's words: "usage of non-PyTorch libraries such as numpy; conversion to Python types such as tolist(); and data-dependent control flow operations" §6.1. The fixes are simple. Delete debugging prints and logging from the hot path. Keep values as tensors instead of calling .item(). Replace a branch on a value with torch.where or torch.cond. Wrap truly un-compilable parts in their own function, so they break once, cleanly.

A compiled function has a print, a .item() and a boolean-mask index. With no other problems, how many graphs does each call run?

Chapter 6 When not

torch.compile, part 2: Inductor, modes and CUDA graphs

TorchInductor turns each captured graph into fused Triton kernels on a GPU or C++ on a CPU. CUDA graphs remove launch overhead. None of it is free, and sometimes none of it helps

Dynamo hands each graph to a backend. The default is TorchInductor. It lowers PyTorch ops to a small loop-level form, then its scheduler decides "which operators get fused, what order kernels run in", and plans memory PyTorch 2 · §4.4. It ranks candidate fusions partly by the "estimated bytes of memory traffic saved", the same quantity you counted in Chapter 4. On GPUs it writes Triton kernels; on CPUs it writes C++ with OpenMP §4.5, §4.6.

Here is a measurement you can repeat. The 5-op chain of Chapter 4 on a 4096 × 4096 float32 tensor, timed with benchmark.Timer (one thread) on the laptop CPU used to write this lesson, PyTorch 2.10:

VersionMedian time per callNote
Eager61.9 ms5 separate kernels
torch.compile10.4 msone fused C++ loop; 6× faster
First compiled call4.9 stracing + code generation + C++ compile

That last row is the price. Compiling takes seconds to minutes for a large model. It pays off only when you run the compiled code many times.

Tcompile
One-off compile time (and again for every recompile).
teager, tcompiled
Steady-state time per call, measured after warm-up.
n*
Calls needed before compiling has saved more time than it cost.

Worked, the table: n* = 4.9 s / (61.9 − 10.4) ms = 95 calls. A training run of 100,000 steps repays it a thousand times over. A notebook cell you run three times does not.

The modes

torch.compile(model, mode=...) picks a preset of Inductor options. These are the presets PyTorch 2.10 lists (torch._inductor.list_mode_options()):

modeWhat it turns onUse it when
"default"Fusion and code generation; no CUDA graphsStart here. Fast to compile.
"reduce-overhead"triton.cudagraphs: replay kernel launches with CUDA graphsSmall batches or many small kernels: overhead-bound steps.
"max-autotune"Autotuning (benchmarks several kernels, incl. Triton matmul templates), coordinate-descent tuning, CUDA graphsLong runs where a long compile pays back.
"max-autotune-no-cudagraphs"The autotuning, without CUDA graphsInputs change in ways CUDA graphs cannot follow.

CUDA graphs: launch once, run many

Even fused, a model step can be hundreds of kernels, and each launch costs the CPU microseconds. A CUDA graph records a sequence of kernel launches, with their arguments and memory addresses, once. Afterwards one call replays the whole sequence; the GPU runs the kernels back to back. The paper: CUDA graphs record and replay "kernel launches at the CUDA driver level" and are "lower overhead than even the C++ wrapper code" §4.7.

Fewer launches, then one

The step is 24 small kernels. Each launch costs the CPU 10 µs; each kernel runs 3 µs. Fuse them, then capture them in a CUDA graph.

Illustrative numbers. Fusion turns 24 kernels into 6 that each do the work of 4 at about 0.7 of the cost (fewer memory passes). A CUDA graph replay is modelled as one 10 µs launch followed by back-to-back kernels with a 0.5 µs gap.

CUDA graphs have rules, because a replay repeats exactly what was recorded. The input tensors must live at the same addresses, so the data is copied into fixed buffers first. Shapes must stay the same. There can be no CPU work in the middle, and no syncs. The paper says Inductor uses them only when "safety requirements are met" and turns them off automatically "with dynamic shapes, non-CUDA tensors" and similar cases.

Run this on a machine with PyTorch and a GPU

import torch
model = torch.nn.Sequential(*[torch.nn.Linear(256, 256) for _ in range(12)]).cuda()
x = torch.randn(8, 256, device="cuda")          # a small batch: overhead-bound

fast = torch.compile(model, mode="reduce-overhead")   # fusion + CUDA graphs
with torch.no_grad():
    for _ in range(3):
        fast(x)                                  # compile, record, then replay
# by hand, without the compiler (inference shown; see the CUDA graphs docs for training):
g = torch.cuda.CUDAGraph()
static_x = x.clone()
with torch.no_grad(), torch.cuda.graph(g):
    static_y = model(static_x)                     # recorded, not run for real
static_x.copy_(torch.randn(8, 256, device="cuda"))   # new data, same address
g.replay()                                         # static_y now holds the result

When torch.compile does not help

The paper's headline is a geometric-mean speed-up of 2.27× for inference and 1.41× for training, over 180+ models on an A100 PyTorch 2 · Abstract. A mean hides the spread. Its ablation on HuggingFace models shows where the gain comes from Table 4. Without fusion, the inference speed-up falls from 1.91× to 1.68×. Without fusion and inlining, it becomes 0.80×: slower than eager. Without CUDA graphs, it is 1.81×. Compiling helps little when:

Is compiling worth it?

Defaults are the laptop measurement above. Change them to your model's numbers. The bars are total wall-clock time for the run.

61.9
10.4
4.9
100

Total = steps × time per step, plus the compile time for the compiled run. Recompiles would add more compile time.

Compiling takes 120 s. A step takes 50 ms eager and 40 ms compiled. After how many steps does compiling start to save time?

Chapter 7 Build it

Hidden syncs: when the CPU stops to wait

The CPU normally runs ahead of the GPU. Any line that needs a GPU value on the CPU stops it, drains the queue, and exposes all the CPU work that used to hide

Recall Chapter 1: the CPU queues kernels and moves on. In a healthy loop the CPU is a few kernels, or even a few steps, ahead of the GPU. Its Python work, the data loading and the launches all happen while the GPU is busy with earlier work. That overlap is free speed.

A host-device sync ends it. The host is the CPU; the device is the GPU. Whenever Python needs a value that lives on the GPU, the CPU must wait until the GPU has computed it. Then the GPU's queue is empty, and the GPU waits while the CPU prepares the next work. The lines that do this rarely look expensive:

Tcpu
CPU time per step: Python, dispatch, launches, data loading.
Tgpu
GPU time per step: the sum of its kernels.
Σ exposed CPU
After each sync, the CPU work done before the GPU has a kernel to run again. At worst it is all of Tcpu.
Tno sync, Tsync
Steady-state step time without and with syncs in the loop.

Worked: Tcpu = 6 ms, Tgpu = 20 ms. No sync: 20 ms, the CPU hides. If every bit of CPU work is exposed: up to 26 ms, 30% slower. The device shows the real answer lies in between, and depends on where the syncs fall.

The queue drains

Three steps of a training loop. Turn on sync points and watch red gaps open in the GPU lane. Then make the CPU side heavier, as with many small ops or slow data loading.

6.0

Illustrative step: GPU 20 ms (forward 6, backward 10, optimizer 4) in 46 kernels; the CPU's work is the next batch's loading (a third) plus one launch per kernel. Kernels start once launched and once the previous kernel ends. A sync makes the CPU wait for every queued kernel.

Notice what the slider does. With a light CPU side, a sync per step costs little: the GPU only waits for the next batch to load. With a heavy CPU side, the same syncs cost much more. This is why a sync that "never mattered" suddenly does after you speed up the GPU with torch.compile or bfloat16. The GPU's time shrank, so the CPU's exposed time is now a bigger share.

Find them, then remove them

Run this on a machine with PyTorch and a GPU

torch.cuda.set_sync_debug_mode("warn")      # or "error"; "default" turns it off
loss = model(x).mean()
v = loss.item()                               # with "warn", this line now prints a warning

# the fix: keep running totals on the GPU, read them rarely
running = torch.zeros((), device="cuda")
for step, (x, y) in enumerate(loader):
    loss = train_step(x, y)
    running += loss.detach()                  # stays on the GPU: no sync
    if step % 100 == 99:
        print("mean loss", (running / 100).item())   # one sync per 100 steps
        running.zero_()

The function's docstring warns that it is experimental and that "not all synchronizing operations will trigger warning or error". The profiler (Chapter 10) is the other witness: a sync shows up as a long cudaStreamSynchronize or cudaMemcpy on the CPU side, with an idle gap on the GPU. Two more fixes. Check for NaN every N steps, or keep torch.isfinite(loss).all() as a tensor until you read it. Replace x[mask] with torch.where or a multiplication by the mask, which keeps the shape fixed.

Which line in a training loop does NOT force a host-device sync?

Chapter 8 Where else

Feed it, then batch it

A fast step is useless if the data arrives late. Then pick the batch size for the real goal: the shortest time to a good model, not the most samples per second

Before you tune kernels, check the oldest bottleneck: the input pipeline. If the DataLoader cannot build batches as fast as the GPU eats them, the GPU waits at the start of every step, and nothing in Chapters 2 to 7 helps. Datasets and DataLoaders covers the fixes: enough workers, pinned memory, cheaper decoding, cached features.

The test takes a minute. Time one pass over the loader with no model at all. If it delivers fewer samples per second than training needs, the data path is the limit. In a profile, the sign is time inside the loader's __next__ at the start of each step, and an idle GPU while it runs.

Batch size and throughput

Every step pays fixed costs: launches, the optimizer's update of every weight, Python. A bigger batch shares them across more samples. While the step is overhead-bound (Chapter 2), doubling the batch barely changes the step time, so samples per second nearly double. Once the GPU is compute-bound, the step time grows in proportion and throughput levels off. Past that, memory runs out.

But throughput is not the goal. A bigger batch gives a less noisy gradient, so each step is worth more; yet beyond some size, extra samples per step stop helping. McCandlish and colleagues measured this trade-off across many tasks and fitted one simple curve (OpenAI, 2018):

B
The batch size.
S
Optimizer steps needed to reach a target loss.
E
Training examples needed: E = B · S.
Smin, Emin
The fewest steps (at a huge batch) and the fewest examples (at a tiny batch) that reach the target.
Bcrit
The critical batch size, Emin / Smin. The paper fits (S/Smin − 1)(E/Emin − 1) = 1 McCandlish 2018 · Eq. 2.11.

Worked: at B = Bcrit, both brackets are 2: twice the minimum steps and twice the minimum examples, as the paper notes. At B = Bcrit / 4: S = 5 Smin, E = 1.25 Emin. At B = 4 Bcrit: S = 1.25 Smin, E = 5 Emin: four times the data for a 4× smaller saving in steps.

Put the two together. Time to the target is steps times step time. Below Bcrit, a bigger batch cuts the steps a lot and costs little per step. Far above it, the steps barely fall while each step gets slower.

Fast steps or fast training?

Slide the batch size. The left chart is samples per second; the right is the time to reach the target loss. Change the critical batch size and watch the best batch move.

32

Illustrative model: step time = 4 ms + 0.03 ms × B; memory holds at most B = 2,048; Smin = 20,000 steps. The curve S(B) is the McCandlish fit; real Bcrit varies by task and grows as the loss falls during training. The learning rate is assumed re-tuned for each B.

Two practical rules follow. First, a larger batch needs a retuned learning rate; Goyal and colleagues (2017) scale it linearly with the batch, with a warm-up. Second, when memory caps the batch, gradient accumulation (part 5) gives the large batch's gradient, but not its speed: it runs the small steps one after another.

Your critical batch size is 256. You raise the batch from 256 to 2,048. By the McCandlish fit, what happens to the steps needed?

Chapter 9 Build it

Tensor cores, shapes and layouts

The biggest roof belongs to the tensor cores, and they are picky: low-precision numbers, dimensions in multiples of 8, and channels in the fast-moving position

A tensor core is a unit on the GPU that multiplies small matrix tiles in one instruction. It is where the large numbers on the datasheet come from. For the A100: 19.5 TFLOP/s for ordinary FP32, 156 for TF32 on tensor cores, and 312 for bfloat16 or FP16 on tensor cores (all dense, without sparsity) A100 datasheet.

So the first rule for compute-bound work is to use a tensor-core format. torch.autocast("cuda", dtype=torch.bfloat16) runs matmuls and convolutions in bfloat16. It keeps sensitive ops in float32; Memory and Mixed Precision shows how and why. For float32 models, torch.set_float32_matmul_precision("high") lets matmuls use TF32. In PyTorch 2.10 the default is "highest": TF32 off for matmuls, while torch.backends.cudnn.allow_tf32 is already True for convolutions (both checked).

For memory-bound ops, a 2-byte format moves half the bytes, so it can run up to twice as fast. For compute-bound ones on an A100, bfloat16's roof is 16× the FP32 roof. That gap is why mixed precision is the single largest speed-up for most training runs.

Shapes in multiples of 8

Tensor cores work on fixed tiles, and the libraries load data in aligned chunks. NVIDIA's matmul guide: performance "is better when equivalent matrix dimensions M, N, and K are aligned to multiples of 16 bytes (or 128 bytes on A100)". For FP16 that means multiples of 8 elements, or 64 on an A100; for TF32, multiples of 4, or 32 NVIDIA matmul guide · Table 1.

The second effect is tile quantization. A kernel covers the output with tiles, say 128 × 128. A dimension of 257 needs three tiles across, and the third is almost empty but costs as much as a full one. NVIDIA's example: a case that needs only 0.39% more operations runs 1.5× as many, because of the extra tiles.

N
A dimension of the output, such as the number of output features.
T
The tile size along that dimension: often 64, 128 or 256, chosen by the library.
useful
The fraction of the work spent on real outputs. The rest is padding inside the last tile.

Worked, T = 128: N = 256 → 2 tiles, useful 100%. N = 257 → 3 tiles, useful 257 / 384 = 67%. The fix is to pad 257 to a friendlier size, or trim it to 256.

Tiles and waste

Slide the output width N. Each column is one 128-wide tile; red is work spent on padding. Find the widths where nothing is wasted.

257

The 128 tile is one common choice, used in NVIDIA's illustration; libraries pick tile sizes per shape and often narrower tiles to reduce this waste. Alignment (multiples of 8 for 2-byte types) is a separate effect from tile waste.

A well-known example: Karpathy's nanoGPT sets vocab_size: int = 50304 with the comment "GPT-2 vocab_size of 50257, padded up to nearest multiple of 64 for efficiency". The extra 47 rows of the output layer are never used as targets. The lab below pads a vocabulary the same way, and fixes the trap that comes with it.

channels_last for convolutions

An image batch has four dimensions: N (batch), C (channels), H and W. PyTorch's default layout, NCHW, stores each channel's whole image before the next channel. NHWC, which PyTorch calls channels_last, stores all channels of one pixel together. NVIDIA's convolution guide: "convolutions implemented for Tensor Cores require NHWC layout"; given NCHW, they still run but "include some overhead due to automatic transpose operations".

Same tensor, two orders

A 1 × 3 × 2 × 2 tensor: three channels (colours) of a 2 × 2 image. The strip is memory, left to right. Switch the layout and read the strides.

The shape stays (1, 3, 2, 2) in both; only the strides change. Strides checked in PyTorch 2.10: (12, 4, 2, 1) and (12, 1, 6, 3).

Run this on a machine with PyTorch and a GPU

model = model.to(memory_format=torch.channels_last)        # conv weights to NHWC
for x, y in loader:
    x = x.cuda(non_blocking=True).to(memory_format=torch.channels_last)
    with torch.autocast("cuda", dtype=torch.bfloat16):  # tensor-core convs
        loss = F.cross_entropy(model(x), y)
    # x.shape is still (N, C, H, W); x.stride() shows the new order

The shape you index with does not change, so the rest of your code keeps working. Only the order in memory does. Most layers keep the format they receive, so converting the model and the input once is enough. Some ops (a view that assumes NCHW contiguity, some custom kernels) may convert back; the profiler shows extra copies if so.

You train in bfloat16 on an A100. Your MLP's hidden width is 3,000. NVIDIA's guide suggests what?

Chapter 10 Did it work?

Read the profiler

torch.profiler records every op and kernel with its start and end. Learn its table, the difference between self time and total time, and what to look for in the trace

Guessing where time goes is how weeks get lost. torch.profiler records each PyTorch op on the CPU and, with CUDA activity on, each kernel on the GPU. It then prints a table of averages, or saves a full timeline you can open in a trace viewer.

Run this on a machine with PyTorch

import torch
from torch.profiler import profile, ProfilerActivity, record_function, schedule

acts = [ProfilerActivity.CPU] + ([ProfilerActivity.CUDA] if torch.cuda.is_available() else [])
with profile(activities=acts, schedule=schedule(wait=1, warmup=2, active=3),
             record_shapes=True) as prof:
    for step, (x, y) in enumerate(loader):
        with record_function("train_step"):   # your own label in the table and trace
            train_step(x, y)
        prof.step()                             # advance the wait/warmup/active schedule
        if step == 6:
            break
key = "self_cuda_time_total" if torch.cuda.is_available() else "self_cpu_time_total"
print(prof.key_averages().table(sort_by=key, row_limit=15))
prof.export_chrome_trace("trace.json")        # open it at ui.perfetto.dev

The schedule skips one step, warms up for two (the profiler's own start-up is slow), and records three. Each row of the table is one op name, summed over all its calls. Two times matter most:

total(e)
End minus start of event e.
children(e)
Events that start and end inside e, with no event between them and e.
self(e)
Time inside e but in none of its children. Sort by this to find what to fix.

Worked, from the table below: Optimizer.step#AdamW.step has a total of 10.147 ms and a self time of 2.248 ms. The other 7.9 ms are its children (aten::div, aten::sqrt, aten::mul_, aten::lerp_, ...). The 2.2 ms is Python and bookkeeping in the optimizer itself.

A real table, read row by row

This table is real: five AdamW steps of a 512 → 2048 → 512 MLP with LayerNorm, batch 256, on a laptop CPU. Sort it, then step through the rows (or tap one).

Output of prof.key_averages().table(sort_by="self_cpu_time_total", row_limit=12), PyTorch 2.10, CPU only, after 3 warm-up steps; the model is described in the footer. "Self CPU time total: 40.453ms" for 5 steps. Times on your machine will differ.

What to look for

An op has a total time of 12 ms. Its direct children took 3 ms and 5 ms. What is its self time?

Chapter 11

Connections

Match a symptom to its cause, try fixes in the right order, keep the one-screen cheat sheet, and know what this lesson could not show

Speed work goes best in a fixed order: measure, find the bound, fix that bound, measure again. Pick a symptom below to see the usual causes, the check that tells them apart, and the fix.

From symptom to fix

Pick the symptom you see. The card lists causes from most to least common, the one check to run first, and the fix.

A field guide built from the chapters of this lesson, not an exhaustive list.

The cheat sheet

PartIn one line
MeasureWarm up, torch.cuda.synchronize() around the clock, repeat, report the median; or benchmark.Timer(...).blocked_autorange().
Three boundstop ≈ max(c, F/P, Q/BW): overhead, compute, memory. Fix the one that is largest.
OverheadA few µs per eager op; vectorise loops, raise the batch, compile, CUDA graphs (mode="reduce-overhead").
RooflineI = F/Q; attainable = min(P, I·BW); A100 ridge 153 FLOP/byte (BF16), 9.6 (FP32); square matmul I = N/3 (bf16); matvec I ≈ 1.
FusionSpeed-up ≈ Σ(ink+outk) / (inputs + outputs); torch.compile, scaled_dot_product_attention, fused=True optimizers.
DynamoBytecode → FX graph + guards; graphs = breaks + 1; find breaks with torch._dynamo.explain and TORCH_LOGS="graph_breaks,recompiles".
InductorFused Triton (GPU) or C++ (CPU) kernels; modes default / reduce-overhead / max-autotune; break-even n* = Tcompile / (teager − tcompiled).
Syncs.item(), print(t), if t, x[mask], .cpu(); find with torch.cuda.set_sync_debug_mode("warn"); accumulate on the GPU, read rarely.
BatchThroughput levels off; S = Smin(1 + Bcrit/B); optimise time to target, retune the learning rate.
Tensor coresbf16 autocast; set_float32_matmul_precision("high") for TF32; dims multiples of 8 (64 on A100); channels_last for convs.
Profilertorch.profiler.profile + schedule; sort by self time; self = total − children; look for GPU gaps in the trace.

Bugs experts have learned to spot

Honest limits

Where this sits, and where to go next

The takeaway. Measure with warm-up, synchronize and the median. Then find the bound. Overhead-bound: fewer, bigger ops (vectorise, batch, compile, CUDA graphs). Memory-bound: fewer bytes (fusion, 2-byte formats). Compute-bound: faster maths (tensor cores, good shapes). Keep the GPU fed and the CPU from waiting on it. And judge every change by time to a good model, not by a single benchmark.
A profile shows hundreds of tiny elementwise kernels, a GPU lane full of gaps, and CPU time per step larger than GPU time. What do you try first?

Now press Present or Teach and explain, out loud and from memory, why an add can never use more than a sliver of an A100 and what fusion changes. Then go back to the hero, pick the 5-op chain in float32, and work out by hand where the dot lands before you turn on fusion.

Making PyTorch Fast, part 8 of PyTorch, Zero to Expert, in The Builder's Toolkit. Sources: J. Ansel et al., "PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph Compilation", ASPLOS 2024 (doi:10.1145/3620665.3640366), cited by section and table; S. Williams, A. Waterman and D. Patterson, "Roofline: An Insightful Visual Performance Model for Multicore Architectures", Communications of the ACM 52(4), 2009 (doi:10.1145/1498765.1498785); NVIDIA A100 Tensor Core GPU datasheet, June 2021 (PDF); NVIDIA Deep Learning Performance guides on matrix multiplication and convolutions; S. McCandlish, J. Kaplan, D. Amodei et al., "An Empirical Model of Large-Batch Training", 2018 (arXiv:1812.06162); P. Goyal et al., "Accurate, Large Minibatch SGD", 2017 (arXiv:1706.02677); A. Karpathy, nanoGPT model.py. PyTorch behaviour (explain output, logs, modes, defaults, strides, the profiler table and the compile timings) was checked by running PyTorch 2.10 on a laptop CPU (Apple M5 Max); the profiled model is Sequential(Linear(512, 2048), GELU(), Linear(2048, 512), LayerNorm(512)) with AdamW, MSE loss and batch 256. Device numbers not taken from these sources are illustrative and labelled so.
Back to Gleams